How variation in predictor measurement affects the discriminative ability and transportability of a prediction model

External validation of the ARCH score in patients undergoing aortic arch reconstruction under circulatory arrest

BACKGROUND

Aortic arch surgery with hypothermic circulatory arrest (HCA) carries a higher risk of morbidity and mortality compared to routine cardiac surgical procedures. The newly developed ARCH (arch reconstruction under circulatory arrest with hypothermia) score has not been externally validated. We sought to externally validate this score in our local population.

METHODS

All consecutive open aortic arch surgeries with HCA performed between 2014 and 2023 were included. Univariable and multivariable analyses were performed. Model discrimination was assessed by the C-statistic with 95% confidence intervals as part of the receiver operating characteristic (ROC) curve analysis. Model performance was visualized by a calibration plot and quantified by the Brier score.

RESULTS

A total of 760 patients (38.3% females) were included. The mean age was 61 (±13.6) years, with 56.4% of patients' age >60 years. The procedures were carried out mostly emergently or urgently (59.6%). Total arch replacement was performed in 32.5% of the patients, and aortic root procedures were carried out in 74.6%. In-hospital death occurred in 64 patients (8.4%), and stroke occurred in 5.4%. The C-statistic revealed a low discriminatory ability for predicting in-hospital mortality (area under the ROC curve, 0.62; 95% confidence interval, 0.54-0.69; P = .002); however, model calibration was found to be excellent (Brier score of 0.07).

CONCLUSIONS

The ARCH score for in-hospital mortality showed low discriminatory ability in our local population, although with excellent ability for prediction of mortality.

Tutorial: dos and don'ts in clinical prediction research for venous thromboembolism

Clinical prediction modeling has become an increasingly popular domain of venous thromboembolism research in recent years. Prediction models can help healthcare providers make decisions regarding starting or withholding therapeutic interventions, or referrals for further diagnostic workup, and can form a basis for risk stratification in clinical trials. The aim of the current guide is to assist in the practical application of complicated methodological requirements for well-performed prediction research by presenting key dos and don'ts while expanding the understanding of predictive research in general for (clinical) researchers who are not specifically trained in the topic; throughout we will use prognostic venous thromboembolism scores as an exemplar.

Development and External Validation of Models to Predict Need for Nursing Home Level of Care in Community-Dwelling Older Adults With Dementia

Importance

Most older adults living with dementia ultimately need nursing home level of care (NHLOC).

Objective

To develop models to predict need for NHLOC among older adults with probable dementia using self-report and proxy reports to aid patients and family with planning and care management.

Design, Setting, and Participants

This prognostic study included data from 1998 to 2016 from the Health and Retirement Study (development cohort) and from 2011 to 2019 from the National Health and Aging Trends Study (validation cohort). Participants were community-dwelling adults 65 years and older with probable dementia. Data analysis was conducted between January 2022 and October 2023.

Exposures

Candidate predictors included demographics, behavioral/health factors, functional measures, and chronic conditions.

Main Outcomes and Measures

The primary outcome was need for NHLOC defined as (1) 3 or more activities of daily living (ADL) dependencies, (2) 2 or more ADL dependencies and presence of wandering/need for supervision, or (3) needing help with eating. A Weibull survival model incorporating interval censoring and competing risk of death was used. Imputation-stable variable selection was used to develop 2 models: one using proxy responses and another using self-responses. Model performance was assessed by discrimination (integrated area under the receiver operating characteristic curve [iAUC]) and calibration (calibration plots).

Results

Of 3327 participants with probable dementia in the Health and Retirement Study, the mean (SD) age was 82.4 (7.4) years and 2301 (survey-weighted 70%) were female. At the end of follow-up, 2107 participants (63.3%) were classified as needing NHLOC. Predictors for both final models included age, baseline ADL and instrumental ADL dependencies, and driving status. The proxy model added body mass index and falls history. The self-respondent model added female sex, incontinence, and date recall. Optimism-corrected iAUC after bootstrap internal validation was 0.72 (95% CI, 0.70-0.75) in the proxy model and 0.64 (95% CI, 0.62-0.66) in the self-respondent model. On external validation in the National Health and Aging Trends Study (n = 1712), iAUC in the proxy and self-respondent models was 0.66 (95% CI, 0.61-0.70) and 0.64 (95% CI, 0.62-0.67), respectively. There was excellent calibration across the range of predicted risk.

Conclusions and Relevance

This prognostic study showed that relatively simple models using self-report or proxy responses can predict need for NHLOC in community-dwelling older adults with probable dementia with moderate discrimination and excellent calibration. These estimates may help guide discussions with patients and families in future care planning.

Exploring the impact of missingness on racial disparities in predictive performance of a machine learning model for emergency department triage

Objective

To investigate how missing data in the patient problem list may impact racial disparities in the predictive performance of a machine learning (ML) model for emergency department (ED) triage.

Materials and Methods

Racial disparities may exist in the missingness of EHR data (eg, systematic differences in access, testing, and/or treatment) that can impact model predictions across racialized patient groups. We use an ML model that predicts patients' risk for adverse events to produce triage-level recommendations, patterned after a clinical decision support tool deployed at multiple EDs. We compared the model's predictive performance on sets of observed (problem list data at the point of triage) versus manipulated (updated to the more complete problem list at the end of the encounter) test data. These differences were compared between Black and non-Hispanic White patient groups using multiple performance measures relevant to health equity.

Results

There were modest, but significant, changes in predictive performance comparing the observed to manipulated models across both Black and non-Hispanic White patient groups; c-statistic improvement ranged between 0.027 and 0.058. The manipulation produced no between-group differences in c-statistic by race. However, there were small between-group differences in other performance measures, with greater change for non-Hispanic White patients.

Discussion

Problem list missingness impacted model performance for both patient groups, with marginal differences detected by race.

Conclusion

Further exploration is needed to examine how missingness may contribute to racial disparities in clinical model predictions across settings. The novel manipulation method demonstrated may aid future research.

Leveraging Data Science and Novel Technologies to Develop and Implement Precision Medicine Strategies in Critical Care

Precision medicine aims to identify treatments that are most likely to result in favorable outcomes for subgroups of patients with similar clinical and biological characteristics. The gaps for the development and implementation of precision medicine strategies in the critical care setting are many, but the advent of data science and multi-omics approaches, combined with the rich data ecosystem in the intensive care unit, offer unprecedented opportunities to realize the promise of precision critical care. In this article, the authors review the data-driven and technology-based approaches being leveraged to discover and implement precision medicine strategies in the critical care setting.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Cardiovascular Risk Prediction in Men and Women Aged Under 50 Years Using Routine Care Data

Background Prediction models for risk of cardiovascular events generally do not include young adults, and cardiovascular risk factors differ between women and men. Therefore, this study aimed to develop prediction models for first-ever cardiovascular event risk in men and women aged 30 to 49 years. Methods and Results We included patients aged 30 to 49 years without cardiovascular disease from a Dutch routine care database. Outcome was defined as first-ever cardiovascular event. Our reference models were sex-specific Cox proportional hazards models based on traditional cardiovascular predictors, which we compared with models using 2 predictor subsets with the 20 or 50 most important predictors based on the Cox elastic net model regularization coefficients. We assessed the C-index and calibration curve slopes at 10 years of follow-up. We stratified our analyses based on 30- to 39-year and 40- to 49-year age groups at baseline. We included 542 141 patients (mean age 39.7, 51% women). During follow-up, 10 767 cardiovascular events occurred. Discrimination of reference models including traditional cardiovascular predictors was moderate (women: C-index, 0.648 [95% CI, 0.645-0.652]; men: C-index, 0.661 [95%CI, 0.658-0.664]). In women and men, the Cox proportional hazard models including 50 most important predictors resulted in an increase in C-index (0.030 and 0.012, respectively), and a net correct reclassification of 3.7% of the events in women and 1.2% in men compared with the reference model. Conclusions Sex-specific electronic health record-derived prediction models for first-ever cardiovascular events in the general population aged <50 years have moderate discriminatory performance. Data-driven predictor selection leads to identification of nontraditional cardiovascular predictors, which modestly increase performance of models.

Replacing performance status with a simple patient-reported outcome in palliative radiotherapy prognostic modelling

Background and purpose

Prognostication is key to determining care in advanced incurable cancer. Although performance status (PS) has been shown to be a strong prognostic predictor, inter-rater reliability is limited, restricting models to specialist settings. This study assessed the extent to which a simple patient-reported outcome measure (PROM), the EQ-5D, may replace PS for prognosis of patients with bone metastases.

Materials and methods

Data from 1,011 patients in the Dutch Bone Metastasis Study were used. Cox proportional hazards models were developed to investigate the prognostic value of models incorporating PS alone, the EQ-5D SC dimension alone, all EQ-5D dimensions and EQ-VAS, and finally all dimensions and PS. Three prognostic groups were identified and performance assessed using the Harrell's C-index and Altman-Royston index of separation.

Results

Replacing performance status (PS) with the self-care (SC) dimension of the EQ-5D provides similar model performance. In our SC-based model, three groups are identified with median survival of 86 days (95 % CI 76-101), 174 days (95 % CI 145-213), and 483 days (95 % CI 431-539). Whilst not statistically significantly different, the C-index was 0.706 for the PS-only model, 0.718 for SC-only and 0.717 in our full model, suggesting patient-report outcome models perform as well as that based on PS.

Conclusion

Prognostic performance was similar across all models. The SC model provides prognostic value similar to that of PS, particularly where a prognosis of<6 months is considered. Larger, more contemporaneous studies are needed to assess the extent to which PROMs may be of prognostic value, particularly where specialist assessment is less feasible.

Predicting Chronic Opioid Use Among Patients With Osteoarthritis Using Electronic Health Record Data

OBJECTIVE

To estimate the risk of a patient with osteoarthritis (OA) developing chronic opioid use (COU) within 1 year of a new opioid prescription by using electronic health record (EHR) data and predictive models.

METHODS

We used EHR data from 13 health care organizations to identify patients with OA with an opioid prescription between March 1, 2017 and February 28, 2019 and no record of opioid use in the prior 6 months. We evaluated 4 machine learning models to estimate patients' risk of COU (≥3 prescriptions ≥84 days, maximum gap ≤60 days). We also estimated the transportability of models to organizations outside the training set.

RESULTS

The cohort consisted of 33,894 patients with OA, of whom 2,925 (8.6%) developed COU within 1 year. All models demonstrated good discrimination, with the best-performing model (random forest) achieving an area under the receiver operating characteristic curve (AUC) of 0.728 (95% CI 0.711-0.745), but the simplest regression model performed nearly as well (AUC 0.717 [95% CI 0.699-0.734]). Predicted risk deciles spanned a range of 2% risk for the 10th percentile to 18% risk for the 90th percentile for well-calibrated models. Models showed highly variable discrimination across organizations (AUC 0.571-0.842).

CONCLUSIONS

We found that EHR-based predictive models could estimate the risk of future COU among patients with OA to help inform care decisions. Black-box methods did not have significant advantages over more interpretable models. Care should be taken when extending all models into organizations not included in model training because of a high variability in performance across held-out organizations.

Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease

The medical field has seen a rapid increase in the development of artificial intelligence (AI)-based prediction models. With the introduction of such AI-based prediction model tools and software in cardiovascular patient care, the cardiovascular researcher and healthcare professional are challenged to understand the opportunities as well as the limitations of the AI-based predictions. In this article, we present 12 critical questions for cardiovascular health professionals to ask when confronted with an AI-based prediction model. We aim to support medical professionals to distinguish the AI-based prediction models that can add value to patient care from the AI that does not.

A Unified Framework on Generalizability of Clinical Prediction Models

To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs.

Quantitative prediction error analysis to investigate predictive performance under predictor measurement heterogeneity at model implementation

BACKGROUND

When a predictor variable is measured in similar ways at the derivation and validation setting of a prognostic prediction model, yet both differ from the intended use of the model in practice (i.e., "predictor measurement heterogeneity"), performance of the model at implementation needs to be inferred. This study proposed an analysis to quantify the impact of anticipated predictor measurement heterogeneity.

METHODS

A simulation study was conducted to assess the impact of predictor measurement heterogeneity across validation and implementation setting in time-to-event outcome data. The use of the quantitative prediction error analysis was illustrated using an example of predicting the 6-year risk of developing type 2 diabetes with heterogeneity in measurement of the predictor body mass index.

RESULTS

In the simulation study, calibration-in-the-large of prediction models was poor and overall accuracy was reduced in all scenarios of predictor measurement heterogeneity. Model discrimination decreased with increasing random predictor measurement heterogeneity.

CONCLUSIONS

Heterogeneity of predictor measurements across settings of validation and implementation reduced predictive performance at implementation of prognostic models with a time-to-event outcome. When validating a prognostic model, the targeted clinical setting needs to be considered and analyses can be conducted to quantify the impact of anticipated predictor measurement heterogeneity on model performance at implementation.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

A Methodological Appraisal of the HEART Score and Its Variants

We performed a methodological appraisal of the history, electrocardiogram, age, risk factors, and troponin (HEART) score and its variants in the context of Annals of Emergency Medicine's methodological standards for clinical decision rules. We note that this chest pain risk stratification tool was not formally derived, omits sex and other known predictors, has weak interrater reliability, and its 0, 1, and 2 score weightings do not align with their known predictivities. Its summary performance (pooled sensitivities of 96% to 97% with lower confidence interval bounds of 93% to 94%) is below that which emergency physicians state a willingness to accept, below the 98% sensitivity exhibited by baseline practice without the score, and below the 1% to 2% acceptable miss threshold specified by the American College of Emergency Physicians chest pain policy. Two variants (HEART Pathway, HEART-2) have the same inherent structural limitations and demonstrate slightly better but still suboptimal sensitivity. Although a simple prediction tool for chest pain outcomes is appealing, we believe that the widespread use of the HEART score and its variants should be reconsidered.

Prediction Models for Physical, Cognitive, and Mental Health Impairments After Critical Illness: A Systematic Review and Critical Appraisal

OBJECTIVES

Improved ability to predict impairments after critical illness could guide clinical decision-making, inform trial enrollment, and facilitate comprehensive patient recovery. A systematic review of the literature was conducted to investigate whether physical, cognitive, and mental health impairments could be predicted in adult survivors of critical illness.

DATA SOURCES

A systematic search of PubMed and the Cochrane Library (Prospective Register of Systematic Reviews ID: CRD42018117255) was undertaken on December 8, 2018, and the final searches updated on January 20, 2019.

STUDY SELECTION

Four independent reviewers assessed titles and abstracts against study eligibility criteria. Studies were eligible if a prediction model was developed, validated, or updated for impairments after critical illness in adult patients. Discrepancies were resolved by consensus or an independent adjudicator.

DATA EXTRACTION

Data on study characteristics, timing of outcome measurement, candidate predictors, and analytic strategies used were extracted. Risk of bias was assessed using the Prediction model Risk Of Bias Assessment Tool.

DATA SYNTHESIS

Of 8,549 screened studies, three studies met inclusion. All three studies focused on the development of a prediction model to predict (1) a mental health composite outcome at 3 months post discharge, (2) return-to-pre-ICU functioning and residence at 6 months post discharge, and (3) physical function 2 months post discharge. Only one model had been externally validated. All studies had a high risk of bias, primarily due to the sample size, and statistical methods used to develop and select the predictors for the prediction published model.

CONCLUSIONS

We only found three studies that developed a prediction model of any post-ICU impairment. There are several opportunities for improvement for future prediction model development, including the use of standardized outcomes and time horizons, and improved study design and statistical methodology.

The Zika Virus Individual Participant Data Consortium: A Global Initiative to Estimate the Effects of Exposure to Zika Virus during Pregnancy on Adverse Fetal, Infant, and Child Health Outcomes

This commentary describes the creation of the Zika Virus Individual Participant Data Consortium, a global collaboration to address outstanding questions in Zika virus (ZIKV) epidemiology through conducting an individual participant data meta-analysis (IPD-MA). The aims of the IPD-MA are to (1) estimate the absolute and relative risks of miscarriage, fetal loss, and short- and long-term sequelae of fetal exposure; (2) identify and quantify the relative importance of different sources of heterogeneity (e.g., immune profiles, concurrent flavivirus infection) for the risk of adverse fetal, infant, and child outcomes among infants exposed to ZIKV in utero; and (3) develop and validate a prognostic model for the early identification of high-risk pregnancies and inform communication between health care providers and their patients and public health interventions (e.g., vector control strategies, antenatal care, and family planning programs). By leveraging data from a diversity of populations across the world, the IPD-MA will provide a more precise estimate of the risk of adverse ZIKV-related outcomes within clinically relevant subgroups and a quantitative assessment of the generalizability of these estimates across populations and settings. The ZIKV IPD Consortium effort is indicative of the growing recognition that data sharing is a central component of global health security and outbreak response.

Missing data should be handled differently for prediction than for description or causal explanation

Missing data are much studied in epidemiology and statistics. Theoretical development and application of methods for handling missing data have mostly been conducted in the context of prospective research data and with a goal of description or causal explanation. However, it is now common to build predictive models using routinely collected data, where missing patterns may convey important information, and one might take a pragmatic approach to optimizing prediction. Therefore, different methods to handle missing data may be preferred. Furthermore, an underappreciated issue in prediction modeling is that the missing data method used in model development may not match the method used when a model is deployed. This may lead to overoptimistic assessments of model performance. For prediction, particularly with routinely collected data, methods for handling missing data that incorporate information within the missingness pattern should be explored and further developed. Where missing data methods differ between model development and model deployment, the implications of this must be explicitly evaluated. The trade-off between building a prediction model that is causally principled, and building a prediction model that maximizes the use of all available information, should be carefully considered and will depend on the intended use of the model.

Prediction of contralateral breast cancer: external validation of risk calculators in 20 international cohorts

BACKGROUND

Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in patients with invasive breast cancer (BC).

METHODS

We analyzed data of 132,756 patients (4682 CBC) from 20 international studies with a median follow-up of 8.8 years. Prediction performance included discrimination, quantified as a time-dependent Area-Under-the-Curve (AUC) at 5 and 10 years after diagnosis of primary BC, and calibration, quantified as the expected-observed (E/O) ratio at 5 and 10 years and the calibration slope.

RESULTS

The AUC at 10 years was: 0.58 (95% confidence intervals [CI] 0.57-0.59) for CBCrisk; 0.60 (95% CI 0.59-0.61) for the Manchester formula; 0.63 (95% CI 0.59-0.66) and 0.59 (95% CI 0.56-0.62) for PredictCBC-1A (for settings where BRCA1/2 mutation status is available) and PredictCBC-1B (for the general population), respectively. The E/O at 10 years: 0.82 (95% CI 0.51-1.32) for CBCrisk; 1.53 (95% CI 0.63-3.73) for the Manchester formula; 1.28 (95% CI 0.63-2.58) for PredictCBC-1A and 1.35 (95% CI 0.65-2.77) for PredictCBC-1B. The calibration slope was 1.26 (95% CI 1.01-1.50) for CBCrisk; 0.90 (95% CI 0.79-1.02) for PredictCBC-1A; 0.81 (95% CI 0.63-0.99) for PredictCBC-1B, and 0.39 (95% CI 0.34-0.43) for the Manchester formula.

CONCLUSIONS

Current CBC risk prediction tools provide only moderate discrimination and the Manchester formula was poorly calibrated. Better predictors and re-calibration are needed to improve CBC prediction and to identify low- and high-CBC risk patients for clinical decision-making.

Reflection on modern methods: five myths about measurement error in epidemiological research

Epidemiologists are often confronted with datasets to analyse which contain measurement error due to, for instance, mistaken data entries, inaccurate recordings and measurement instrument or procedural errors. If the effect of measurement error is misjudged, the data analyses are hampered and the validity of the study's inferences may be affected. In this paper, we describe five myths that contribute to misjudgments about measurement error, regarding expected structure, impact and solutions to mitigate the problems resulting from mismeasurements. The aim is to clarify these measurement error misconceptions. We show that the influence of measurement error in an epidemiological data analysis can play out in ways that go beyond simple heuristics, such as heuristics about whether or not to expect attenuation of the effect estimates. Whereas we encourage epidemiologists to deliberate about the structure and potential impact of measurement error in their analyses, we also recommend exercising restraint when making claims about the magnitude or even direction of effect of measurement error if not accompanied by statistical measurement error corrections or quantitative bias analysis. Suggestions for alleviating the problems or investigating the structure and magnitude of measurement error are given.

Fundamentals of Clinical Prediction Modeling for the Neurosurgeon

Clinical prediction models in neurosurgery are increasingly reported. These models aim to provide an evidence-based approach to the estimation of the probability of a neurosurgical outcome by combining 2 or more prognostic variables. Model development and model reporting are often suboptimal. A basic understanding of the methodology of clinical prediction modeling is needed when interpreting these models. We address basic statistical background, 7 modeling steps, and requirements of these models such that they may fulfill their potential for major impact for our daily clinical practice and for future scientific work.

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

OBJECTIVES

To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models.

DESIGN

Meta-epidemiological study.

DATA SOURCES AND STUDY SELECTION

On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model.

DATA EXTRACTION AND ANALYSES

Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance.

RESULTS

We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63).

CONCLUSIONS

Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.