Developing more generalizable prediction models from pooled studies and large clustered data sets

The potential benefit of statin prescription based on prediction of treatment responsiveness in older individuals: an application to the PROSPER randomized controlled trial

AIMS

Clinical guidelines often recommend treating individuals based on their cardiovascular risk. We revisit this paradigm and quantify the efficacy of three treatment strategies: (i) overall prescription, i.e. treatment to all individuals sharing the eligibility criteria of a trial; (ii) risk-stratified prescription, i.e. treatment only to those at an elevated outcome risk; and (iii) prescription based on predicted treatment responsiveness.

METHODS AND RESULTS

We reanalysed the PROSPER randomized controlled trial, which included individuals aged 70-82 years with a history of, or risk factors for, vascular diseases. We conducted the derivation and internal-external validation of a model predicting treatment responsiveness. We compared with placebo (n = 2913): (i) pravastatin (n = 2891); (ii) pravastatin in the presence of previous vascular diseases and placebo in the absence thereof (n = 2925); and (iii) pravastatin in the presence of a favourable prediction of treatment response and placebo in the absence thereof (n = 2890). We found an absolute difference in primary outcome events composed of coronary death, non-fatal myocardial infarction, and fatal or non-fatal stroke, per 10 000 person-years equal to: -78 events (95% CI, -144 to -12) when prescribing pravastatin to all participants; -66 events (95% CI, -114 to -18) when treating only individuals with an elevated vascular risk; and -103 events (95% CI, -162 to -44) when restricting pravastatin to individuals with a favourable prediction of treatment response.

CONCLUSION

Pravastatin prescription based on predicted responsiveness may have an encouraging potential for cardiovascular prevention. Further external validation of our results and clinical experiments are needed.

TRIAL REGISTRATION

ISRCTN40976937.

Why do probabilistic clinical models fail to transport between sites

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models.

Postpartum readmission for hypertension and pre-eclampsia: development and validation of a predictive model

OBJECTIVE

To develop a model for predicting postpartum readmission for hypertension and pre-eclampsia at delivery discharge and assess external validation or model transportability across clinical sites.

DESIGN

Prediction model using data available in the electronic health record from two clinical sites.

SETTING

Two tertiary care health systems from the Southern (2014-2015) and Northeastern USA (2017-2019).

POPULATION

A total of 28 201 postpartum individuals: 10 100 in the South and 18 101 in the Northeast.

METHODS

An internal-external cross validation (IECV) approach was used to assess external validation or model transportability across the two sites. In IECV, data from each health system were first used to develop and internally validate a prediction model; each model was then externally validated using the other health system. Models were fit using penalised logistic regression, and accuracy was estimated using discrimination (concordance index), calibration curves and decision curves. Internal validation was performed using bootstrapping with bias-corrected performance measures. Decision curve analysis was used to display potential cut points where the model provided net benefit for clinical decision-making.

MAIN OUTCOME MEASURES

The outcome was postpartum readmission for either hypertension or pre-eclampsia <6 weeks after delivery.

RESULTS

The postpartum readmission rate for hypertension and pre-eclampsia overall was 0.9% (0.3% and 1.2% by site, respectively). The final model included six variables: age, parity, maximum postpartum diastolic blood pressure, birthweight, pre-eclampsia before discharge and delivery mode (and interaction between pre-eclampsia × delivery mode). Discrimination was adequate at both health systems on internal validation (c-statistic South: 0.88; 95% confidence interval [CI] 0.87-0.89; Northeast: 0.74; 95% CI 0.74-0.74). In IECV, discrimination was inconsistent across sites, with improved discrimination for the Northeastern model on the Southern cohort (c-statistic 0.61 and 0.86, respectively), but calibration was not adequate. Next, model updating was performed using the combined dataset to develop a new model. This final model had adequate discrimination (c-statistic: 0.80, 95% CI 0.80-0.80), moderate calibration (intercept -0.153, slope 0.960, Emax 0.042) and provided superior net benefit at clinical decision-making thresholds between 1% and 7% for interventions preventing readmission. An online calculator is provided here.

CONCLUSIONS

Postpartum readmission for hypertension and pre-eclampsia may be accurately predicted but further model validation is needed. Model updating using data from multiple sites will be needed before use across clinical settings.

External validation of a United Kingdom primary-care Cushing's prediction tool in a population of referred Dutch dogs

BACKGROUND

A prediction tool was developed and internally validated to aid the diagnosis of Cushing's syndrome in dogs attending UK primary-care practices. External validation is an important part of model validation to assess model performance when used in different populations.

OBJECTIVES

To assess the original prediction model's transportability, applicability, and diagnostic performance in a secondary-care practice in the Netherlands.

ANIMALS

Two hundred thirty client-owned dogs.

METHODS

Retrospective observational study. Medical records of dogs under investigation of Cushing's syndrome between 2011 and 2020 were reviewed. Dogs diagnosed with Cushing's syndrome by the attending internists and fulfilling ALIVE criteria were defined as cases, others as non-cases. All dogs were scored using the aforementioned prediction tool. Dog characteristics and predictor-outcome effects in development and validation data sets were compared to assess model transportability. Calibration and discrimination were examined to assess model performance.

RESULTS

Eighty of 230 dogs were defined as cases. Significant differences in dog characteristics were found between UK primary-care and Dutch secondary-care populations. Not all predictors from the original model were confirmed to be significant predictors in the validation sample. The model systematically overestimated the probability of having Cushing's syndrome (a = -1.10, P < .001). Calibration slope was 1.35 and discrimination proved excellent (area under the receiver operating curve = 0.83).

CONCLUSIONS AND CLINICAL IMPORTANCE

The prediction model had moderate transportability, excellent discriminatory ability, and overall overestimated probability of having Cushing's syndrome. This study confirms its utility, though emphasizes that ongoing validation efforts of disease prediction tools are a worthwhile effort.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Diagnostic management of acute pulmonary embolism: a prediction model based on a patient data meta-analysis

AIMS

Risk stratification is used for decisions regarding need for imaging in patients with clinically suspected acute pulmonary embolism (PE). The aim was to develop a clinical prediction model that provides an individualized, accurate probability estimate for the presence of acute PE in patients with suspected disease based on readily available clinical items and D-dimer concentrations.

METHODS AND RESULTS

An individual patient data meta-analysis was performed based on sixteen cross-sectional or prospective studies with data from 28 305 adult patients with clinically suspected PE from various clinical settings, including primary care, emergency care, hospitalized and nursing home patients. A multilevel logistic regression model was built and validated including ten a priori defined objective candidate predictors to predict objectively confirmed PE at baseline or venous thromboembolism (VTE) during follow-up of 30 to 90 days. Multiple imputation was used for missing data. Backward elimination was performed with a P-value <0.10. Discrimination (c-statistic with 95% confidence intervals [CI] and prediction intervals [PI]) and calibration (outcome:expected [O:E] ratio and calibration plot) were evaluated based on internal-external cross-validation. The accuracy of the model was subsequently compared with algorithms based on the Wells score and D-dimer testing. The final model included age (in years), sex, previous VTE, recent surgery or immobilization, haemoptysis, cancer, clinical signs of deep vein thrombosis, inpatient status, D-dimer (in µg/L), and an interaction term between age and D-dimer. The pooled c-statistic was 0.87 (95% CI, 0.85-0.89; 95% PI, 0.77-0.93) and overall calibration was very good (pooled O:E ratio, 0.99; 95% CI, 0.87-1.14; 95% PI, 0.55-1.79). The model slightly overestimated VTE probability in the lower range of estimated probabilities. Discrimination of the current model in the validation data sets was better than that of the Wells score combined with a D-dimer threshold based on age (c-statistic 0.73; 95% CI, 0.70-0.75) or structured clinical pretest probability (c-statistic 0.79; 95% CI, 0.76-0.81).

CONCLUSION

The present model provides an absolute, individualized probability of PE presence in a broad population of patients with suspected PE, with very good discrimination and calibration. Its clinical utility needs to be evaluated in a prospective management or impact study.

REGISTRATION

PROSPERO ID 89366.

Prediction Models for Bronchopulmonary Dysplasia in Preterm Infants: A Systematic Review and Meta-Analysis

OBJECTIVE

To review systematically and assess the accuracy of prediction models for bronchopulmonary dysplasia (BPD) at 36 weeks of postmenstrual age.

STUDY DESIGN

Searches were conducted in MEDLINE and EMBASE. Studies published between 1990 and 2022 were included if they developed or validated a prediction model for BPD or the combined outcome death/BPD at 36 weeks in the first 14 days of life in infants born preterm. Data were extracted independently by 2 authors following the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (ie, CHARMS) and PRISMA guidelines. Risk of bias was assessed using the Prediction model Risk Of Bias ASsessment Tool (ie, PROBAST).

RESULTS

Sixty-five studies were reviewed, including 158 development and 108 externally validated models. Median c-statistic of 0.84 (range 0.43-1.00) was reported at model development, and 0.77 (range 0.41-0.97) at external validation. All models were rated at high risk of bias, due to limitations in the analysis part. Meta-analysis of the validated models revealed increased c-statistics after the first week of life for both the BPD and death/BPD outcome.

CONCLUSIONS

Although BPD prediction models perform satisfactorily, they were all at high risk of bias. Methodologic improvement and complete reporting are needed before they can be considered for use in clinical practice. Future research should aim to validate and update existing models.

Leveraging electronic health record data for endometriosis research

Endometriosis is a chronic, complex disease for which there are vast disparities in diagnosis and treatment between sociodemographic groups. Clinical presentation of endometriosis can vary from asymptomatic disease-often identified during (in)fertility consultations-to dysmenorrhea and debilitating pelvic pain. Because of this complexity, delayed diagnosis (mean time to diagnosis is 1.7-3.6 years) and misdiagnosis is common. Early and accurate diagnosis of endometriosis remains a research priority for patient advocates and healthcare providers. Electronic health records (EHRs) have been widely adopted as a data source in biomedical research. However, they remain a largely untapped source of data for endometriosis research. EHRs capture diverse, real-world patient populations and care trajectories and can be used to learn patterns of underlying risk factors for endometriosis which, in turn, can be used to inform screening guidelines to help clinicians efficiently and effectively recognize and diagnose the disease in all patient populations reducing inequities in care. Here, we provide an overview of the advantages and limitations of using EHR data to study endometriosis. We describe the prevalence of endometriosis observed in diverse populations from multiple healthcare institutions, examples of variables that can be extracted from EHRs to enhance the accuracy of endometriosis prediction, and opportunities to leverage longitudinal EHR data to improve our understanding of long-term health consequences for all patients.

Embracing cohort heterogeneity in clinical machine learning development: a step toward generalizable models

This study is a simple illustration of the benefit of averaging over cohorts, rather than developing a prediction model from a single cohort. We show that models trained on data from multiple cohorts can perform significantly better in new settings than models based on the same amount of training data but from just a single cohort. Although this concept seems simple and obvious, no current prediction model development guidelines recommend such an approach.

Prediction of model generalizability for unseen data: Methodology and case study in brain metastases detection in T1-Weighted contrast-enhanced 3D MRI

BACKGROUND AND PURPOSE

A medical AI system's generalizability describes the continuity of its performance acquired from varying geographic, historical, and methodologic settings. Previous literature on this topic has mostly focused on "how" to achieve high generalizability (e.g., via larger datasets, transfer learning, data augmentation, model regularization schemes), with limited success. Instead, we aim to understand "when" the generalizability is achieved: Our study presents a medical AI system that could estimate its generalizability status for unseen data on-the-fly.

MATERIALS AND METHODS

We introduce a latent space mapping (LSM) approach utilizing Fréchet distance loss to force the underlying training data distribution into a multivariate normal distribution. During the deployment, a given test data's LSM distribution is processed to detect its deviation from the forced distribution; hence, the AI system could predict its generalizability status for any previously unseen data set. If low model generalizability is detected, then the user is informed by a warning message integrated into a sample deployment workflow. While the approach is applicable for most classification deep neural networks (DNNs), we demonstrate its application to a brain metastases (BM) detector for T1-weighted contrast-enhanced (T1c) 3D MRI. The BM detection model was trained using 175 T1c studies acquired internally (from the authors' institution) and tested using (1) 42 internally acquired exams and (2) 72 externally acquired exams from the publicly distributed Brain Mets dataset provided by the Stanford University School of Medicine. Generalizability scores, false positive (FP) rates, and sensitivities of the BM detector were computed for the test datasets.

RESULTS AND CONCLUSION

The model predicted its generalizability to be low for 31% of the testing data (i.e., two of the internally and 33 of the externally acquired exams), where it produced (1) ∼13.5 false positives (FPs) at 76.1% BM detection sensitivity for the low and (2) ∼10.5 FPs at 89.2% BM detection sensitivity for the high generalizability groups respectively. These results suggest that the proposed formulation enables a model to predict its generalizability for unseen data.

Evaluating the validity of game-based assessments measuring cognitive function among children and adolescents: A systematic review and meta-analysis

Games offer advantages over traditional methods of assessing cognitive function among children and adolescents. However, the validity of game-based assessments has not been systematically evaluated. We conducted a systematic review and meta-analysis to assess the validity of game-based assessments measuring cognitive function among children and adolescents. We systematically searched several databases using pre-defined inclusion criteria. For papers that met the criteria, we extracted and analyzed the cognitive functions measured by each study, the correlation coefficients between game-based and traditional assessments, and factors that could influence the validity of game-based assessments. Our review identified 19 articles featuring 20 studies, 18 games, and 378 unique correlations between game-based and traditional assessments of cognitive function. Game-based assessments yielded significant correlations (n=282, 75%) with traditional assessments, over half of which were in the low to medium range in strength (r=0.3-0.69, n=227, 80%). Factors related to the child, such as age, gender, and prior gaming experience, may influence the validity of game-based assessments by modifying performance on game-based assessments. In addition, we found that game-based assessments that measured cognitive functions across more than one neurocognitive domain and used a prediction model for scoring were more likely to yield significant correlations. In contrast, including a narrative storyline in a game-based assessment was less likely to yield significant correlations. Most studies were of good quality, although the lack of sample size justification was a limiting factor. Further research is needed to elucidate the influence of identified factors on the validity of game-based assessment to justify the wide adoption of game-based assessments of cognitive function among children and adolescents.

Automated large-scale prediction of exudative AMD progression using machine-read OCT biomarkers

Age-related Macular Degeneration (AMD) is a major cause of irreversible vision loss in individuals over 55 years old in the United States. One of the late-stage manifestations of AMD, and a major cause of vision loss, is the development of exudative macular neovascularization (MNV). Optical Coherence Tomography (OCT) is the gold standard to identify fluid at different levels within the retina. The presence of fluid is considered the hallmark to define the presence of disease activity. Anti-vascular growth factor (anti-VEGF) injections can be used to treat exudative MNV. However, given the limitations of anti-VEGF treatment, as burdensome need for frequent visits and repeated injections to sustain efficacy, limited durability of the treatment, poor or no response, there is a great interest in detecting early biomarkers associated with a higher risk for AMD progression to exudative forms in order to optimize the design of early intervention clinical trials. The annotation of structural biomarkers on optical coherence tomography (OCT) B-scans is a laborious, complex and time-consuming process, and discrepancies between human graders can introduce variability into this assessment. To address this issue, a deep-learning model (SLIVER-net) was proposed, which could identify AMD biomarkers on structural OCT volumes with high precision and without human supervision. However, the validation was performed on a small dataset, and the true predictive power of these detected biomarkers in the context of a large cohort has not been evaluated. In this retrospective cohort study, we perform the largest-scale validation of these biomarkers to date. We also assess how these features combined with other EHR data (demographics, comorbidities, etc) affect and/or improve the prediction performance relative to known factors. Our hypothesis is that these biomarkers can be identified by a machine learning algorithm without human supervision, in a way that they preserve their predictive nature. The way we test this hypothesis is by building several machine learning models utilizing these machine-read biomarkers and assessing their added predictive power. We found that not only can we show that the machine-read OCT B-scan biomarkers are predictive of AMD progression, we also observe that our proposed combined OCT and EHR data-based algorithm outperforms the state-of-the-art solution in clinically relevant metrics and provides actionable information which has the potential to improve patient care. In addition, it provides a framework for automated large-scale processing of OCT volumes, making it possible to analyze vast archives without human supervision.

Developing prediction models when there are systematically missing predictors in individual patient data meta-analysis

Clinical prediction models are widely used in modern clinical practice. Such models are often developed using individual patient data (IPD) from a single study, but often there are IPD available from multiple studies. This allows using meta-analytical methods for developing prediction models, increasing power and precision. Different studies, however, often measure different sets of predictors, which may result to systematically missing predictors, that is, when not all studies collect all predictors of interest. This situation poses challenges in model development. We hereby describe various approaches that can be used to develop prediction models for continuous outcomes in such situations. We compare four approaches: a "restrict predictors" approach, where the model is developed using only predictors measured in all studies; a multiple imputation approach that ignores study-level clustering; a multiple imputation approach that accounts for study-level clustering; and a new approach that develops a prediction model in each study separately using all predictors reported, and then synthesizes all predictions in a multi-study ensemble. We explore in simulations the performance of all approaches under various scenarios. We find that imputation methods and our new method outperform the restrict predictors approach. In several scenarios, our method outperformed imputation methods, especially for few studies, when predictor effects were small, and in case of large heterogeneity. We use a real dataset of 12 trials in psychotherapies for depression to illustrate all methods in practice, and we provide code in R.

A multicenter prospective study validated a nomogram to predict individual risk of dependence in ambulation after rehabilitation

OBJECTIVES

To develop the Functional Risk Index for Dependence in Ambulation (FRIDA) score, a nomogram to predict individual risk of dependence in ambulation at discharge from postacute rehabilitation and validate its performance temporally and spatially.

STUDY DESIGN AND SETTING

We analyzed the database of a multicenter prospective observational quality cohort study conducted from January 2012 to March 2016, including data from 8,796 consecutive inpatients who underwent rehabilitation after stroke, hip fracture, lower limb joint replacement, debility, and other neurologic, orthopedic, or miscellaneous conditions.

RESULTS

A total of 3,026 patients (34.4%) were discharged dependent in ambulation. In the training set of 5,162 patients (58.7%), Lasso-regression selected advanced age, premorbid disability, and eight indicators of medical and functional adverse syndromes at baseline to establish the FRIDA score. At the temporal validation obtained on an external set of 3,234 patients (41.3%), meta-analyses showed that the FRIDA score had good and homogeneous discrimination (summary area under the curve 0.841, 95% confidence interval = 0.826-0.855, I² = 0.00%) combined with accurate calibration (summary Log O/E ratio 0.017, 95% confidence interval -0.155 to 0.190). These performances remained stable at spatial validation obtained on 3,626 patients, with substantial heterogeneity of estimates across nine facilities. Decision curve analyses showed that a FRIDA score-supported strategy far outperformed the usual "treat all" approach in each impairment categories.

CONCLUSION

The FRIDA score is a new clinically useful tool to predict an individual risk for dependence in ambulation at rehabilitation discharge in many different disabilities, and may also reflect well the case-mix composition of the rehabilitation facilities.

Measuring Health-Related Quality of Life With Multimodal Data: Viewpoint

The ability to objectively measure aspects of performance and behavior is a fundamental pillar of digital health, enabling digital wellness products, decentralized trial concepts, evidence generation, digital therapeutics, and more. Emerging multimodal technologies capable of measuring several modalities simultaneously and efforts to integrate inputs across several sources are further expanding the limits of what digital measures can assess. Experts from the field of digital health were convened as part of a multi-stakeholder workshop to examine the progress of multimodal digital measures in two key areas: detection of disease and the measurement of meaningful aspects of health relevant to the quality of life. Here we present a meeting report, summarizing key discussion points, relevant literature, and finally a vision for the immediate future, including how multimodal measures can provide value to stakeholders across drug development and care delivery, as well as three key areas where headway will need to be made if we are to continue to build on the encouraging progress so far: collaboration and data sharing, removal of barriers to data integration, and alignment around robust modular evaluation of new measurement capabilities.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.