1146 - Implementing the data-driven CLL-TIM algorithm into an EPIC-based EHR: proof of concept for automated decision support models

Background: Implementing large predictive models in routine clinical practice requires proof-of-concept examples of how such research algorithms, and their results, can be integrated safely and successfully in clinical practice. While Electronic Health Records (EHR) have increased the availability of health data for training predictive machine learning (ML) algorithms, the algorithms implemented in EHR are often rule-based models with few variables and lack continuous performance monitoring. This simplicity compromises prediction accuracy and fails to capture the complexity of patient-disease dynamics. Moreover, there is currently no proof-of-concept example of all the necessary processes and stakeholders involved in implementing complex decision-making algorithms in clinical practice. We here describe such a system exemplified by our implementation of an algorithm for Chronic Lymphocytic Leukemia (CLL) implemented as a stand-alone platform into a Danish EHR.

Aim: To implement a complex prognostic model into EHR and detail the steps and challenges required for such an implementation.

Methods: The CLL Treatment and Infection Model (CLL-TIM) was developed to identify high-risk CLL patients who may experience severe infections or require treatment within two years of diagnosis. CLL-TIM is currently being used to select patients for the PreVent-ACaLL clinical trial. We implemented CLL-TIM on a Python platform within the EHR system that extracts relevant data from the EHR on a daily basis. The platform provides CLL-TIM predictions, including individual patient risk factors and prediction confidence, as a view within the EHR. We present the process of data harmonization, validation between research and production environment data, benchmarking of predictions, architecture for automating predictions and the proposed setup for automatic monitoring of the algorithm.

Findings: To ascertain the success of the implementation, we benchmarked the implemented version of CLL-TIM on a blinded test set of CLL patients diagnosed with CLL after 2018. Compared to results obtained in the research environment, CLL-TIM demonstrated similar performance within the real-time production environment, achieving an MCC (Matthews correlation coefficient) of 0.64 under similar missing data conditions. Implementing CLL-TIM into clinical practice posed significant challenges particularly in the areas of 1) data validation and harmonization between research and production environments, 2) technical integration with the EHR, 3) strategic negotiation for presenting results within the patient context and 4) obtaining approval under the EU-based Medical Device Regulation.

Interpretation: Having highly accurate and trustworthy machine learning algorithms in clinical practice requires moving beyond simple rule-based models. In this study, we present a proof-of-concept with detailed steps involved in implementing a complex prognostic model as a standalone platform integrated with the EHR. Retrospective validation, continuous monitoring, and personalized predictions for all patients, regardless of missing data, were included, providing a pathway for others interested in implementing similar data-driven machine learning algorithms in clinical practice.