Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study

BACKGROUND

Pythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk.

METHODS AND FINDINGS

A curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis.

CONCLUSIONS

Extracting and curating a large, local institution's EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize this data flow for future machine learning efforts.

Presenting machine learning model information to clinical end users with model facts labels

There is tremendous enthusiasm surrounding the potential for machine learning to improve medical prognosis and diagnosis. However, there are risks to translating a machine learning model into clinical care and clinical end users are often unaware of the potential harm to patients. This perspective presents the "Model Facts" label, a systematic effort to ensure that front-line clinicians actually know how, when, how not, and when not to incorporate model output into clinical decisions. The "Model Facts" label was designed for clinicians who make decisions supported by a machine learning model and its purpose is to collate relevant, actionable information in 1-page. Practitioners and regulators must work together to standardize presentation of machine learning model information to clinical end users in order to prevent harm to patients. Efforts to integrate a model into clinical practice should be accompanied by an effort to clearly communicate information about a machine learning model with a "Model Facts" label.

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study

Background

Successful integrations of machine learning into routine clinical care are exceedingly rare, and barriers to its adoption are poorly characterized in the literature.

Objective

This study aims to report a quality improvement effort to integrate a deep learning sepsis detection and management platform, Sepsis Watch, into routine clinical care.

Methods

In 2016, a multidisciplinary team consisting of statisticians, data scientists, data engineers, and clinicians was assembled by the leadership of an academic health system to radically improve the detection and treatment of sepsis. This report of the quality improvement effort follows the learning health system framework to describe the problem assessment, design, development, implementation, and evaluation plan of Sepsis Watch.

Results

Sepsis Watch was successfully integrated into routine clinical care and reshaped how local machine learning projects are executed. Frontline clinical staff were highly engaged in the design and development of the workflow, machine learning model, and application. Novel machine learning methods were developed to detect sepsis early, and implementation of the model required robust infrastructure. Significant investment was required to align stakeholders, develop trusting relationships, define roles and responsibilities, and to train frontline staff, leading to the establishment of 3 partnerships with internal and external research groups to evaluate Sepsis Watch.

Conclusions

Machine learning models are commonly developed to enhance clinical decision making, but successful integrations of machine learning into routine clinical care are rare. Although there is no playbook for integrating deep learning into clinical care, learnings from the Sepsis Watch integration can inform efforts to develop machine learning technologies at other health care delivery systems.

Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality of Adults at Time of Admission

IMPORTANCE

The ability to accurately predict in-hospital mortality for patients at the time of admission could improve clinical and operational decision-making and outcomes. Few of the machine learning models that have been developed to predict in-hospital death are both broadly applicable to all adult patients across a health system and readily implementable. Similarly, few have been implemented, and none have been evaluated prospectively and externally validated.

OBJECTIVES

To prospectively and externally validate a machine learning model that predicts in-hospital mortality for all adult patients at the time of hospital admission and to design the model using commonly available electronic health record data and accessible computational methods.

DESIGN, SETTING, AND PARTICIPANTS

In this prognostic study, electronic health record data from a total of 43 180 hospitalizations representing 31 003 unique adult patients admitted to a quaternary academic hospital (hospital A) from October 1, 2014, to December 31, 2015, formed a training and validation cohort. The model was further validated in additional cohorts spanning from March 1, 2018, to August 31, 2018, using 16 122 hospitalizations representing 13 094 unique adult patients admitted to hospital A, 6586 hospitalizations representing 5613 unique adult patients admitted to hospital B, and 4086 hospitalizations representing 3428 unique adult patients admitted to hospital C. The model was integrated into the production electronic health record system and prospectively validated on a cohort of 5273 hospitalizations representing 4525 unique adult patients admitted to hospital A between February 14, 2019, and April 15, 2019.

MAIN OUTCOMES AND MEASURES

The main outcome was in-hospital mortality. Model performance was quantified using the area under the receiver operating characteristic curve and area under the precision recall curve.

RESULTS

A total of 75 247 hospital admissions (median [interquartile range] patient age, 59.5 [29.0] years; 45.9% involving male patients) were included in the study. The in-hospital mortality rates for the training validation; retrospective validations at hospitals A, B, and C; and prospective validation cohorts were 3.0%, 2.7%, 1.8%, 2.1%, and 1.6%, respectively. The area under the receiver operating characteristic curves were 0.87 (95% CI, 0.83-0.89), 0.85 (95% CI, 0.83-0.87), 0.89 (95% CI, 0.86-0.92), 0.84 (95% CI, 0.80-0.89), and 0.86 (95% CI, 0.83-0.90), respectively. The area under the precision recall curves were 0.29 (95% CI, 0.25-0.37), 0.17 (95% CI, 0.13-0.22), 0.22 (95% CI, 0.14-0.31), 0.13 (95% CI, 0.08-0.21), and 0.14 (95% CI, 0.09-0.21), respectively.

CONCLUSIONS AND RELEVANCE

Prospective and multisite retrospective evaluations of a machine learning model demonstrated good discrimination of in-hospital mortality for adult patients at the time of admission. The data elements, methods, and patient selection make the model implementable at a system level.

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study

Background

Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care.

Objective

This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows.

Methods

We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data.

Results

A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes.

Conclusions

This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Machine learning for early detection of sepsis: an internal and temporal validation study

OBJECTIVE

Determine if deep learning detects sepsis earlier and more accurately than other models. To evaluate model performance using implementation-oriented metrics that simulate clinical practice.

MATERIALS AND METHODS

We trained internally and temporally validated a deep learning model (multi-output Gaussian process and recurrent neural network [MGP-RNN]) to detect sepsis using encounters from adult hospitalized patients at a large tertiary academic center. Sepsis was defined as the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria, a blood culture order, and at least one element of end-organ failure. The training dataset included demographics, comorbidities, vital signs, medication administrations, and labs from October 1, 2014 to December 1, 2015, while the temporal validation dataset was from March 1, 2018 to August 31, 2018. Comparisons were made to 3 machine learning methods, random forest (RF), Cox regression (CR), and penalized logistic regression (PLR), and 3 clinical scores used to detect sepsis, SIRS, quick Sequential Organ Failure Assessment (qSOFA), and National Early Warning Score (NEWS). Traditional discrimination statistics such as the C-statistic as well as metrics aligned with operational implementation were assessed.

RESULTS

The training set and internal validation included 42 979 encounters, while the temporal validation set included 39 786 encounters. The C-statistic for predicting sepsis within 4 h of onset was 0.88 for the MGP-RNN compared to 0.836 for RF, 0.849 for CR, 0.822 for PLR, 0.756 for SIRS, 0.619 for NEWS, and 0.481 for qSOFA. MGP-RNN detected sepsis a median of 5 h in advance. Temporal validation assessment continued to show the MGP-RNN outperform all 7 clinical risk score and machine learning comparisons.

CONCLUSIONS

We developed and validated a novel deep learning model to detect sepsis. Using our data elements and feature set, our modeling approach outperformed other machine learning methods and clinical scores.

The Project Baseline Health Study: a step towards a broader mission to map human health

The Project Baseline Health Study (PBHS) was launched to map human health through a comprehensive understanding of both the health of an individual and how it relates to the broader population. The study will contribute to the creation of a biomedical information system that accounts for the highly complex interplay of biological, behavioral, environmental, and social systems. The PBHS is a prospective, multicenter, longitudinal cohort study that aims to enroll thousands of participants with diverse backgrounds who are representative of the entire health spectrum. Enrolled participants will be evaluated serially using clinical, molecular, imaging, sensor, self-reported, behavioral, psychological, environmental, and other health-related measurements. An initial deeply phenotyped cohort will inform the development of a large, expanded virtual cohort. The PBHS will contribute to precision health and medicine by integrating state of the art testing, longitudinal monitoring and participant engagement, and by contributing to the development of an improved platform for data sharing and analysis.

The impact of specialty pharmaceuticals as drivers of health care costs

The pharmaceutical industry is shifting its focus from blockbuster small molecules to specialty pharmaceuticals. Specialty pharmaceuticals are novel drugs and biologic agents that require special handling and ongoing monitoring, are administered by injection or infusion, and are sold in the marketplace by a small number of distributors. They are frequently identified by having a cost to payers and patients of $600 or more per treatment. The total costs of the new agents are likely to have a substantial impact on overall health care costs and on patients during the next decade, unless steps are taken to align competing interests. We examine the economic and policy issues related to specialty pharmaceuticals, taking care to consider the impact on patients. We assess the role of cost-sharing provisions, legislation that is promoting realignment within the market, the role of biosimilars in price competition, and the potential for novel drug development paradigms to help bend the cost curve. The economic aspects of this analysis highlight the need for a far-reaching discussion of potential novel approaches to innovation pathways in our quest for both affordability and new technology.

A framework for the oversight and local deployment of safe and high-quality prediction models

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.

Barriers to Achieving Economies of Scale in Analysis of EHR Data. A Cautionary Tale

Signed in 2009, the Health Information Technology for Economic and Clinical Health Act infused $28 billion of federal funds to accelerate adoption of electronic health records (EHRs). Yet, EHRs have produced mixed results and have even raised concern that the current technology ecosystem stifles innovation. We describe the development process and report initial outcomes of a chronic kidney disease analytics application that identifies high-risk patients for nephrology referral. The cost to validate and integrate the analytics application into clinical workflow was $217,138. Despite the success of the program, redundant development and validation efforts will require $38.8 million to scale the application across all multihospital systems in the nation. We address the shortcomings of current technology investments and distill insights from the technology industry. To yield a return on technology investments, we propose policy changes that address the underlying issues now being imposed on the system by an ineffective technology business model.

Isolation, characterization and expression of a GRA2 homologue from Neospora caninum

A cDNA library derived from mRNA of tachyzoites of Neospora caninum (NC-Liverpool strain) was screened with antisera from a cow naturally infected with N. caninum. The DNA sequence of 1 recombinant isolated predicted a significant protein sequence homology of the gene product to the 28 kDa (GRA2) antigen of Toxoplasma gondii. Studies on the N. caninum gene coding for this antigen demonstrated the presence of a single intron flanked by 2 exons; the gene was also highly expressed in culture-derived tachyzoites. The antigen was expressed in Escherichia coli; when injected into mice it stimulated the production of antibodies which detected a 29 kDa antigen of N. caninum. Secondary structure predictions made for the N. caninum protein showed support for several amphipathic helices separated by loops and turns. The available evidence indicates maintenance of protein secondary structure, and not DNA or amino acid sequence, has occurred during the evolution of GRA2 proteins in N. caninum and T. gondii.

Isolation of Neospora caninum genes detected during a chronic murine infection

In order to isolate genes coding for antigens of Neospora caninum which are recognised by the host immune system during a chronic murine infection, a cDNA library was immunoscreened with pooled sera from mice which survived three independent infections by N. caninum. Two new genes from N. caninum were isolated and expressed in Escherichia coli. The genes identified include one homologous to GRA1 of Toxoplasma gondii, plus another (NCP20) previously unknown in any taxon. Both genes encode small polypeptides which induced an IgG response in the mouse and were also recognised by IgG from a cow chronically infected with N. caninum. These results are consistent with the hypothesis that the polypeptides encoded by these genes are a target for the host immune system during chronic infections of N. caninum.

Development of a Human Immunodeficiency Virus Risk Prediction Model Using Electronic Health Record Data From an Academic Health System in the Southern United States

BACKGROUND

Human immunodeficiency virus (HIV) pre-exposure prophylaxis (PrEP) is underutilized in the southern United States. Rapid identification of individuals vulnerable to diagnosis of HIV using electronic health record (EHR)-based tools may augment PrEP uptake in the region.

METHODS

Using machine learning, we developed EHR-based models to predict incident HIV diagnosis as a surrogate for PrEP candidacy. We included patients from a southern medical system with encounters between October 2014 and August 2016, training the model to predict incident HIV diagnosis between September 2016 and August 2018. We obtained 74 EHR variables as potential predictors. We compared Extreme Gradient Boosting (XGBoost) versus least absolute shrinkage selection operator (LASSO) logistic regression models, and assessed performance, overall and among women, using area under the receiver operating characteristic curve (AUROC) and area under precision recall curve (AUPRC).

RESULTS

Of 998 787 eligible patients, 162 had an incident HIV diagnosis, of whom 49 were women. The XGBoost model outperformed the LASSO model for the total cohort, achieving an AUROC of 0.89 and AUPRC of 0.01. The female-only cohort XGBoost model resulted in an AUROC of 0.78 and AUPRC of 0.00025. The most predictive variables for the overall cohort were race, sex, and male partner. The strongest positive predictors for the female-only cohort were history of pelvic inflammatory disease, drug use, and tobacco use.

CONCLUSIONS

Our machine-learning models were able to effectively predict incident HIV diagnoses including among women. This study establishes feasibility of using these models to identify persons most suitable for PrEP in the South.

Social determinants of health data in solid organ transplantation: National data sources and future directions

Health equity research in transplantation has largely relied on national data sources, yet the availability of social determinants of health (SDOH) data varies widely among these sources. We sought to characterize the extent to which national data sources contain SDOH data applicable to end-stage organ disease (ESOD) and transplant patients. We reviewed 10 active national data sources based in the United States. For each data source, we examined patient inclusion criteria and explored strengths and limitations regarding SDOH data, using the National Institutes of Health PhenX toolkit of SDOH as a data collection instrument. Of the 28 SDOH variables reviewed, eight-core demographic variables were included in ≥80% of the data sources, and seven variables that described elements of social status ranged between 30 and 60% inclusion. Variables regarding identity, healthcare access, and social need were poorly represented (≤20%) across the data sources, and five of these variables were included in none of the data sources. The results of our review highlight the need for improved SDOH data collection systems in ESOD and transplant patients via: enhanced inter-registry collaboration, incorporation of standardized SDOH variables into existing data sources, and transplant center and consortium-based investigation and innovation.

Production and characterisation of monoclonal antibodies specific for chicken interleukin-12

Using genetic immunisation of mice, we produced antibodies against chicken interleukin-12p40 (chIL-12p40), also known as IL-12β. After a final injection with a recombinant chIL-12p40 protein, several stable hybridoma cell lines were established which secreted monoclonal antibodies (mAbs) to this component of the heterodimeric IL-12 cytokine. Specific binding of three of the mAbs to COS-7 cell-derived recombinant chIL-12p40 and the chIL-12p70 heterodimer was demonstrated in an indirect ELISA, and in dot blots. Two of the mAbs were used to develop a capture ELISA, suitable for detecting both recombinant protein (chIL-12p40 and the heterodimeric p70 protein) and native chIL-12. The mAbs were further characterised to show utility in immunocytochemistry.

Variation of health-care resource utilization according to GERD-associated complications

Complications associated with gastroesophageal reflux disease (GERD) can include esophageal stricture, Barrett's esophagus, gastrointestinal hemorrhage, and extraesophageal symptoms. The impact of GERD-associated complications on health-care utilization deserves further evaluation. We identified commercial enrollees 18-75 years old with claims for GERD (International Classification of Diseases, Ninth Revision, Clinical Modification Codes: 530.81 or 530.11) and subsequent usage of proton pump inhibitors from 01/01/05 to 06/30/09. The initial GERD diagnosis date was designated as the index date, and patients were studied for 6 months preindex and postindex. Eligible patients were subsequently stratified based on medical claims for GERD-associated complications as follows: stage A (GERD diagnosis, no other symptoms), stage B (GERD + extraesophageal symptoms), stage C (GERD + Barrett's esophagus), stage D (GERD + esophageal stricture), and stage E (GERD + iron-deficiency anemia or acute upper gastrointestinal hemorrhage). Patient characteristics, health-care utilization, and costs were compared between stage A and each stage with complicated GERD (B-D). Of the 174,597 patients who were eligible for analysis, 74% were classified as stage A, 20% stage B, 1% stage C, 2% stage D, and 3% stage E. Relative to stage A, patients in stages C, D, and E were significantly more likely to visit a gastroenterologist (13% vs. 68%, 71%, and 38%, respectively) and had higher rates of esophageal ulcers (0.3% vs. 8%, 5%, and 3%, respectively) and Nissen fundoplication (0.05% vs. 0.6%, 0.3%, and 0.2%, respectively). Six-month GERD-related costs ranged from $615/patient (stage A) to $1714/patient (stage D); all-cause costs ranged from $4195/patient (stage A) to $11,340/patient (stage E). Compared with stage A, all other cohorts had significantly higher all-cause and GERD-related costs (P < 0.0001 for all comparisons). While patients with more severe GERD represented a relatively small portion of the GERD cohort, they demonstrated significantly greater health-care costs and overall utilization than patients with uncomplicated GERD.

Budget impact of rosuvastatin initiation in high-risk hyperlipidemic patients from a US managed care perspective

INTRODUCTION

Statins reduce low-density lipoprotein cholesterol (LDL-C) levels, which, when elevated, represent a significant risk factor for cardiovascular (CV) disease. Hyperlipidemic patients at risk of CV events initiated on simvastatin or atorvastatin may be less likely to meet LDL-C goals (defined in National Cholesterol Education Program guidelines) and more likely to experience CV events than patients initiated on rosuvastatin. A 3-year budget impact model was developed to estimate the clinical impact and cost to a US managed care organization (MCO) with 1 million members of initiating high-risk hyperlipidemic patients on rosuvastatin rather than simvastatin or atorvastatin.

METHODS

A total of 1000 adult patients were assumed to initiate statins. The average baseline LDL-C level was 189 mg/dL. In scenario 1, all patients were initiated on simvastatin or atorvastatin and titrated to a higher dose, or switched to atorvastatin (if initiated on simvastatin) or rosuvastatin; in scenario 2, 50% of the 520 high-risk patients were initiated on rosuvastatin. Drug acquisition and administration costs were considered. Product labeling, clinical trial results, national prescription claims data, and published literature were used to populate the model.

RESULTS

Over 3 years, 75 additional patients reached their LDL-C goal in scenario 2, compared with scenario 1 (633 vs 558, respectively), at an increased cost of $240,628 ($1,415,516 vs $1,174,888, respectively). The additional per member per month (PMPM) cost of scenario 2 was $0.007.

LIMITATIONS

This analysis assumed that statin efficacy is the same in real life as in trials, and used titration and switching patterns not based on patients' goal attainment. However, sensitivity and scenario analyses showed that the model was less sensitive to these parameters than to cost-related parameters.

CONCLUSIONS

Initiating high-risk hyperlipidemic patients on rosuvastatin may increase the number of patients reaching LDL-C goal at a relatively modest increase in PMPM cost to an MCO.

Development of a refractory gastro-oesophageal reflux score using an administrative claims database

BACKGROUND

Approximately one-third of gastro-oesophageal reflux disease (GERD) patients demonstrate refractory symptoms following treatment with proton pump inhibitor (PPI) therapy.

AIM

To develop a refractory GERD score that can be applied to predict patients' healthcare utilisation.

METHODS

We enrolled adults (≥18 years) with a diagnosis of GERD. Refractory GERD was evaluated on an 8-point scale where 1 point was given for each of the following criteria: doubling, addition, or switching of GERD medication dose, receipt of a GERD-related endoscopic procedure or surgery, or ≥3 GERD-related outpatient visits. Refractory GERD was defined as the presence of two or more points.

RESULTS

A total of 135,139 GERD patients (44% male) were analysed with a mean (±s.d.) age of 52.9 ± 15 years. The mean overall refractory GERD score was 1.12 ± 1.2 (range 0-8 on an 8-point scale); 31% of patients had refractory GERD with a mean score of 2.56 ± 0.82. Among patients with refractory GERD, 31% doubled their GERD medication, 28% added a new GERD medication, 60% switched GERD medications, 54% had a GERD-related procedure and 1% had a GERD-related surgery. Patients with refractory GERD were more likely to be female (59% vs. 55%, P < 0.001) and had a higher co-morbidity score (0.78 vs. 0.56, P < 0.001). The overall mean costs for refractory patients during the study period were significantly higher compared with treatment-responsive patients ($18,088 ± $36,220 vs. $11,044 ± $22,955, P < 0.001).

CONCLUSIONS

Refractory GERD was present in approximately one-third of the GERD patients. We created a GERD refractory score that could define need for increased anti-reflux therapy and predict higher healthcare resource utilisation.

Assessment of Spanish Translation of Websites at Top-Ranked US Hospitals

This cross-sectional study examines whether Spanish translation of top US hospital websites is associated with hospitals with public medical schools, children’s hospitals, larger Latinx population, or local Immigration and Customs Enforcement activities.

A framework for understanding label leakage in machine learning for health care

INTRODUCTION

The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule."

FRAMEWORK

We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice.

RECOMMENDATIONS

Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.