Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review

BACKGROUND

External validation studies create evidence about a clinical prediction rule's (CPR's) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential.

METHODS

A forward citation search of the Framingham Wilson CHD risk rule's derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule.

RESULTS

We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data.

DISCUSSION

Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule's generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.

International Variation in Severe Exacerbation Rates in Patients With Severe Asthma

BACKGROUND

Exacerbation frequency strongly influences treatment choices in patients with severe asthma.

RESEARCH QUESTION

What is the extent of the variability of exacerbation rate across countries and its implications in disease management?

STUDY DESIGN AND METHODS

We retrieved data from the International Severe Asthma Registry, an international observational cohort of patients with a clinical diagnosis of severe asthma. We identified patients aged ≥ 18 years who did not initiate any biologics prior to baseline visit. A severe exacerbation was defined as the use of oral corticosteroids for ≥ 3 days or asthma-related hospitalization/ED visit. A series of negative binomial models were applied to estimate country-specific severe exacerbation rates during 365 days of follow-up, starting from a naive model with country as the only variable to an adjusted model with country as a random-effect term and patient and disease characteristics as independent variables.

RESULTS

The final sample included 7,510 patients from 17 countries (56% from the United States), contributing to 1,939 severe exacerbations (0.27/person-year). There was large between-country variation in observed severe exacerbation rate (minimum, 0.04 [Argentina]; maximum, 0.88 [Saudi Arabia]; interquartile range, 0.13-0.54), which remained substantial after adjusting for patient characteristics and sampling variability (interquartile range, 0.16-0.39).

INTERPRETATION

Individuals with similar patient characteristics but coming from different jurisdictions have varied severe exacerbation risks, even after controlling for patient and disease characteristics. This suggests unknown patient factors or system-level variations at play. Disease management guidelines should recognize such between-country variability. Risk prediction models that are calibrated for each jurisdiction will be needed to optimize treatment strategies.

[Data-driven intensive care: a lack of comprehensive datasets]

Intensive care units provide a data-rich environment with the potential to generate datasets in the realm of big data, which could be utilized to train powerful machine learning (ML) models. However, the currently available datasets are too small and exhibit too little diversity due to their limitation to individual hospitals. This lack of extensive and varied datasets is a primary reason for the limited generalizability and resulting low clinical utility of current ML models. Often, these models are based on data from single centers and suffer from poor external validity. There is an urgent need for the development of large-scale, multicentric, and multinational datasets. Ensuring data protection and minimizing re-identification risks pose central challenges in this process. The "Amsterdam University Medical Center database (AmsterdamUMCdb)" and the "Salzburg Intensive Care database (SICdb)" demonstrate that open access datasets are possible in Europe while complying with the data protection regulations of the General Data Protection Regulation (GDPR). Another challenge in building intensive care datasets is the absence of semantic definitions in the source data and the heterogeneity of data formats. Establishing binding industry standards for the semantic definition is crucial to ensure seamless semantic interoperability between datasets.

Geriatric End-of-Life Screening Tool Prediction of 6-Month Mortality in Older Patients

Importance

Emergency department (ED) visits by older adults with life-limiting illnesses are a critical opportunity to establish patient care end-of-life preferences, but little is known about the optimal screening criteria for resource-constrained EDs.

Objectives

To externally validate the Geriatric End-of-Life Screening Tool (GEST) in an independent population and compare it with commonly used serious illness diagnostic criteria.

Design, Setting, and Participants

This prognostic study assessed a cohort of patients aged 65 years and older who were treated in a tertiary care ED in Boston, Massachusetts, from 2017 to 2021. Patients arriving in cardiac arrest or who died within 1 day of ED arrival were excluded. Data analysis was performed from August 1, 2023, to March 27, 2024.

Exposure

GEST, a logistic regression algorithm that uses commonly available electronic health record (EHR) datapoints and was developed and validated across 9 EDs, was compared with serious illness diagnoses as documented in the EHR. Serious illnesses included stroke/transient ischemic attack, liver disease, cancer, lung disease, and age greater than 80 years, among others.

Main Outcomes and Measures

The primary outcome was 6-month mortality following an ED encounter. Statistical analyses included area under the receiver operating characteristic curve, calibration analyses, Kaplan-Meier survival curves, and decision curves.

Results

This external validation included 82 371 ED encounters by 40 505 unique individuals (mean [SD] age, 76.8 [8.4] years; 54.3% women, 13.8% 6-month mortality rate). GEST had an external validation area under the receiver operating characteristic curve of 0.79 (95% CI, 0.78-0.79) that was stable across years and demographic subgroups. Of included encounters, 53.4% had a serious illness, with a sensitivity of 77.4% (95% CI, 76.6%-78.2%) and specificity of 50.5% (95% CI, 50.1%-50.8%). Varying GEST cutoffs from 5% to 30% increased specificity (5%: 49.1% [95% CI, 48.7%-49.5%]; 30%: 92.2% [95% CI, 92.0%-92.4%]) at the cost of sensitivity (5%: 89.3% [95% CI, 88.8-89.9]; 30%: 36.2% [95% CI, 35.3-37.1]). In a decision curve analysis, GEST outperformed serious illness criteria across all tested thresholds. When comparing patients referred to intervention by GEST with serious illness criteria, GEST reclassified 45.1% of patients with serious illness as having low risk of mortality with an observed mortality rate 8.1% and 2.6% of patients without serious illness as having high mortality risk with an observed mortality rate of 34.3% for a total reclassification rate of 25.3%.

Conclusions and Relevance

The findings of this study suggest that both serious illness criteria and GEST identified older ED patients at risk for 6-month mortality, but GEST offered more useful screening characteristics. Future trials of serious illness interventions for high mortality risk in older adults may consider transitioning from diagnosis code criteria to GEST, an automatable EHR-based algorithm.

National Multi-Institutional Validation of a Surgical Transfusion Risk Prediction Model

BACKGROUND

Accurate estimation of surgical transfusion risk is important for many aspects of surgical planning, yet few methods for estimating are available for estimating such risk. There is a need for reliable validated methods for transfusion risk stratification to support effective perioperative planning and resource stewardship.

STUDY DESIGN

This study was conducted using the American College of Surgeons NSQIP datafile from 2019. S-PATH performance was evaluated at each contributing hospital, with and without hospital-specific model tuning. Linear regression was used to assess the relationship between hospital characteristics and area under the receiver operating characteristic (AUROC) curve.

RESULTS

A total of 1,000,927 surgical cases from 414 hospitals were evaluated. Aggregate AUROC was 0.910 (95% CI 0.904 to 0.916) without model tuning and 0.925 (95% CI 0.919 to 0.931) with model tuning. AUROC varied across individual hospitals (median 0.900, interquartile range 0.849 to 0.944), but no statistically significant relationships were found between hospital-level characteristics studied and model AUROC.

CONCLUSIONS

S-PATH demonstrated excellent discriminative performance, although there was variation across hospitals that was not well-explained by hospital-level characteristics. These results highlight the S-PATH's viability as a generalizable surgical transfusion risk prediction tool.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

Machine learning models in clinical practice for the prediction of postoperative complications after major abdominal surgery

Complications after surgery have a major impact on short- and long-term outcomes, and decades of technological advancement have not yet led to the eradication of their risk. The accurate prediction of complications, recently enhanced by the development of machine learning algorithms, has the potential to completely reshape surgical patient management. In this paper, we reflect on multiple issues facing the implementation of machine learning, from the development to the actual implementation of machine learning models in daily clinical practice, providing suggestions on the use of machine learning models for predicting postoperative complications after major abdominal surgery.

A guide to sharing open healthcare data under the General Data Protection Regulation

Sharing healthcare data is increasingly essential for developing data-driven improvements in patient care at the Intensive Care Unit (ICU). However, it is also very challenging under the strict privacy legislation of the European Union (EU). Therefore, we explored four successful open ICU healthcare databases to determine how open healthcare data can be shared appropriately in the EU. A questionnaire was constructed based on the Delphi method. Then, follow-up questions were discussed with experts from the four databases. These experts encountered similar challenges and regarded ethical and legal aspects to be the most challenging. Based on the approaches of the databases, expert opinion, and literature research, we outline four distinct approaches to openly sharing healthcare data, each with varying implications regarding data security, ease of use, sustainability, and implementability. Ultimately, we formulate seven recommendations for sharing open healthcare data to guide future initiatives in sharing open healthcare data to improve patient care and advance healthcare.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

External Validation of Mortality Prediction Models for Critical Illness Reveals Preserved Discrimination but Poor Calibration

OBJECTIVES

In a recent scoping review, we identified 43 mortality prediction models for critically ill patients. We aimed to assess the performances of these models through external validation.

DESIGN

Multicenter study.

SETTING

External validation of models was performed in the Simple Intensive Care Studies-I (SICS-I) and the Finnish Acute Kidney Injury (FINNAKI) study.

PATIENTS

The SICS-I study consisted of 1,075 patients, and the FINNAKI study consisted of 2,901 critically ill patients.

MEASUREMENTS AND MAIN RESULTS

For each model, we assessed: 1) the original publications for the data needed for model reconstruction, 2) availability of the variables, 3) model performance in two independent cohorts, and 4) the effects of recalibration on model performance. The models were recalibrated using data of the SICS-I and subsequently validated using data of the FINNAKI study. We evaluated overall model performance using various indexes, including the (scaled) Brier score, discrimination (area under the curve of the receiver operating characteristics), calibration (intercepts and slopes), and decision curves. Eleven models (26%) could be externally validated. The Acute Physiology And Chronic Health Evaluation (APACHE) II, APACHE IV, Simplified Acute Physiology Score (SAPS)-Reduced (SAPS-R)' and Simplified Mortality Score for the ICU models showed the best scaled Brier scores of 0.11' 0.10' 0.10' and 0.06' respectively. SAPS II, APACHE II, and APACHE IV discriminated best; overall discrimination of models ranged from area under the curve of the receiver operating characteristics of 0.63 (0.61-0.66) to 0.83 (0.81-0.85). We observed poor calibration in most models, which improved to at least moderate after recalibration of intercepts and slopes. The decision curve showed a positive net benefit in the 0-60% threshold probability range for APACHE IV and SAPS-R.

CONCLUSIONS

In only 11 out of 43 available mortality prediction models, the performance could be studied using two cohorts of critically ill patients. External validation showed that the discriminative ability of APACHE II, APACHE IV, and SAPS II was acceptable to excellent, whereas calibration was poor.

Stroke risk in older British men: Comparing performance of stroke-specific and composite-CVD risk prediction tools

Stroke risk is currently estimated as part of the composite risk of cardiovascular disease (CVD). We investigated if composite-CVD risk prediction tools QRISK3 and Pooled Cohort Equations-PCE, derived from middle-aged adults, are as good as stroke-specific Framingham Stroke Risk Profile-FSRP and QStroke for capturing the true risk of stroke in older adults. External validation for 10y stroke outcomes was performed in men (60-79y) of the British Regional Heart Study. Discrimination and calibration were assessed in separate validation samples (FSRP n = 3762, QStroke n = 3376, QRISK3 n = 2669 and PCE n = 3047) with/without adjustment for competing risks. Sensitivity/specificity were examined using observed and clinically recommended thresholds. Performance of FSRP, QStroke and QRISK3 was further compared head-to-head in 2441 men free of a range of CVD, including across age-groups. Observed 10y risk (/1000PY) ranged from 6.8 (hard strokes) to 11 (strokes/transient ischemic attacks). All tools discriminated weakly, C-indices 0.63-0.66. FSRP and QStroke overestimated risk at higher predicted probabilities. QRISK3 and PCE showed reasonable calibration overall with minor mis-estimations across the risk range. Performance worsened on adjusting for competing non-stroke deaths. However, in men without CVD, QRISK3 displayed relatively better calibration for stroke events, even after adjustment for competing deaths, including in oldest men. All tools displayed similar sensitivity (63-73 %) and specificity (52-54 %) using observed risks as cut-offs. When QRISK3 and PCE were evaluated using thresholds for CVD prevention, sensitivity for stroke events was 99 %, with false positive rate 97 % suggesting existing intervention thresholds may need to be re-examined to reflect age-related stroke burden.

Generalizability of Risk Stratification Algorithms for Exacerbations in COPD

BACKGROUND

Contemporary management of COPD relies on exacerbation history to risk-stratify patients for future exacerbations. Multivariate prediction models can improve the performance of risk stratification. However, the clinical usefulness of risk stratification can vary from one population to another.

RESEARCH QUESTION

How do two validated exacerbation risk prediction models (Acute COPD Exacerbation Prediction Tool [ACCEPT] and the Bertens model) compared with exacerbation history alone perform in different patient populations?

STUDY DESIGN AND METHODS

We used data from three clinical studies representing populations at different levels of moderate to severe exacerbation risk: the Study to Understand Mortality and Morbidity in COPD (SUMMIT; N = 2,421; annual risk, 0.22), the Long-term Oxygen Treatment Trial (LOTT; N = 595; annual risk, 0.38), and Towards a Revolution in COPD Health (TORCH; N = 1,091; annual risk, 0.52). We compared the area under the receiver operating characteristic curve (AUC) and net benefit (measure of clinical usefulness) among three risk stratification algorithms for predicting exacerbations in the next 12 months. We also evaluated the effect of model recalibration on clinical usefulness.

RESULTS

Compared with exacerbation history, ACCEPT showed better performance in all three samples (change in AUC, 0.08, 0.07, and 0.10, respectively; P ≤ .001 for all). The Bertens model showed better performance compared with exacerbation history in SUMMIT and TORCH (change in AUC, 0.10 and 0.05, respectively; P < .001 for both), but not in LOTT. No algorithm was superior in clinical usefulness across all samples. Before recalibration, the Bertens model generally outperformed the other algorithms in low-risk settings, whereas ACCEPT outperformed others in high-risk settings. All three algorithms showed the risk of harm (providing lower net benefit than not using any risk stratification). After recalibration, risk of harm was mitigated substantially for both prediction models.

INTERPRETATION

Exacerbation history alone is unlikely to provide clinical usefulness for predicting COPD exacerbations in all settings and could be associated with a risk of harm. Prediction models have superior predictive performance, but require setting-specific recalibration to confer higher clinical usefulness.

Prognostic models for COVID-19 needed updating to warrant transportability over time and space

BACKGROUND

Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time.

METHODS

We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit.

RESULTS

Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data.

CONCLUSIONS

NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

ACCEPT 2·0: Recalibrating and externally validating the Acute COPD exacerbation prediction tool (ACCEPT)

BACKGROUND

The Acute Chronic Obstructive Pulmonary Disease (COPD) Exacerbation Prediction Tool (ACCEPT) was developed for individualised prediction of COPD exacerbations. ACCEPT was well calibrated overall and had a high discriminatory power, but overestimated risk among individuals without recent exacerbations. The objectives of this study were to 1) fine-tune ACCEPT to make better predictions for individuals with a negative exacerbation history, 2) develop more parsimonious models, and 3) externally validate the models in a new dataset.

METHODS

We recalibrated ACCEPT using data from the Evaluation of COPD Longitudinally to Identify Predictive Surrogate End-points (ECLIPSE, a three-year observational study, 1,803 patients, 2,117 exacerbations) study by applying non-parametric regression splines to the predicted rates. We developed three reduced versions of ACCEPT by removing symptom score and/or baseline medications as predictors. We examined the discrimination, calibration, and net benefit of ACCEPT 2·0 in the placebo arm of the Towards a Revolution in COPD Health (TORCH, a three-year randomised clinical trial of inhaled therapies in COPD, 1,091 patients, 1,064 exacerbations) study. The primary outcome for prediction was the occurrence of ≥2 moderate or ≥1 severe exacerbation in the next 12 months; the secondary outcomes were prediction of the occurrence of any moderate/severe exacerbation or any severe exacerbation.

FINDINGS

ACCEPT 2·0 had an area-under-the-curve (AUC) of 0·76 for predicting the primary outcome. Exacerbation history alone (current standard of care) had an AUC of 0·68. The model was well calibrated in patients with positive or negative exacerbation histories. Changes in AUC in reduced versions were minimal for the primary outcome as well as for predicting the occurrence of any moderate/severe exacerbations (ΔAUC<0·011), but more substantial for predicting the occurrence of any severe exacerbations (ΔAUC<0·020). All versions of ACCEPT 2·0 provided positive net benefit over the use of exacerbation history alone for some range of thresholds.

INTERPRETATION

ACCEPT 2·0 showed good calibration regardless of exacerbation history, and predicts exacerbation risk better than current standard of care for a range of thresholds. Future studies need to investigate the utility of exacerbation prediction in various subgroups of patients.

FUNDING

This study was funded by a team grant from the Canadian Institutes of Health Research (PHT 178432).