Estimating real-world performance of a predictive model: a case-study in predicting mortality

Narrowing the gap: expected versus deployment performance

OBJECTIVES

Successful model development requires both an accurate a priori understanding of future performance and high performance on deployment. Optimistic estimations of model performance that are unrealized in real-world clinical settings can contribute to nonuse of predictive models. This study used 2 tasks, predicting ICU mortality and Bi-Level Positive Airway Pressure failure, to quantify: (1) how well internal test performances derived from different methods of partitioning data into development and test sets estimate future deployment performance of Recurrent Neural Network models and (2) the effects of including older data in the training set on models' performance.

MATERIALS AND METHODS

The cohort consisted of patients admitted between 2010 and 2020 to the Pediatric Intensive Care Unit of a large quaternary children's hospital. 2010-2018 data were partitioned into different development and test sets to measure internal test performance. Deployable models were trained on 2010-2018 data and assessed on 2019-2020 data, which was conceptualized to represent a real-world deployment scenario. Optimism, defined as the overestimation of the deployed performance by internal test performance, was measured. Performances of deployable models were also compared with each other to quantify the effect of including older data during training.

RESULTS, DISCUSSION, AND CONCLUSION

Longitudinal partitioning methods, where models are tested on newer data than the development set, yielded the least optimism. Including older years in the training dataset did not degrade deployable model performance. Using all available data for model development fully leveraged longitudinal partitioning by measuring year-to-year performance.

Prediction of visual field defects from macular optical coherence tomography in glaucoma using cluster analysis

Purpose

To assess the accuracy of cluster analysis‐based models in predicting visual field (VF) defects from macular ganglion cell‐inner plexiform layer (GCIPL) measurements in glaucomatous and healthy cohorts.

Methods

GCIPL measurements were extracted from posterior pole optical coherence tomography (OCT), from locations corresponding to central VF test grids. Models incorporating cluster analysis methods and corrections for age and fovea to optic disc tilt were developed from 493 healthy participants, and 5th and 1st percentile limits of GCIPL thickness were derived. These limits were compared with pointwise 5th and 1st percentile limits by calculating sensitivities and specificities in an additional 40 normal and 37 glaucomatous participants, as well as applying receiver operating characteristic (ROC) curve analyses to assess the accuracy of predicting VF results from co‐localised GCIPL measurements.

Results

Clustered models demonstrated globally low sensitivity, but high specificity in the glaucoma cohort (0.28–0.53 and 0.77–0.91, respectively), and high specificity in the healthy cohort (0.91–0.98). Clustered models showed similar sensitivities and superior specificities compared with pointwise methods (0.41–0.65 and 0.71–0.98, respectively). There were significant differences in accuracy between clusters, with relatively poor accuracy at peripheral macular locations (p < 0.0001 for all comparisons).

Conclusions

Cluster analysis‐based models incorporating age correction and holistic consideration of fovea to optic disc tilt demonstrated superior performance in predicting VF results to pointwise methods in both glaucomatous and healthy eyes. However, relatively low sensitivity and poorer performance at the peripheral macula indicate that OCT in isolation may be insufficient to predict visual function across the macula accurately. With modifications to criteria for abnormality, the concepts suggested by the described normative models may guide prioritisation of VF assessment requirements, with the potential to limit excessive VF testing.

Development, implementation, and prospective validation of a model to predict 60-day end-of-life in hospitalized adults upon admission at three sites

BACKGROUND

Automated systems that use machine learning to estimate a patient's risk of death are being developed to influence care. There remains sparse transparent reporting of model generalizability in different subpopulations especially for implemented systems.

METHODS

A prognostic study included adult admissions at a multi-site, academic medical center between 2015 and 2017. A predictive model for all-cause mortality (including initiation of hospice care) within 60 days of admission was developed. Model generalizability is assessed in temporal validation in the context of potential demographic bias. A subsequent prospective cohort study was conducted at the same sites between October 2018 and June 2019. Model performance during prospective validation was quantified with areas under the receiver operating characteristic and precision recall curves stratified by site. Prospective results include timeliness, positive predictive value, and the number of actionable predictions.

RESULTS

Three years of development data included 128,941 inpatient admissions (94,733 unique patients) across sites where patients are mostly white (61%) and female (60%) and 4.2% led to death within 60 days. A random forest model incorporating 9614 predictors produced areas under the receiver operating characteristic and precision recall curves of 87.2 (95% CI, 86.1-88.2) and 28.0 (95% CI, 25.0-31.0) in temporal validation. Performance marginally diverges within sites as the patient mix shifts from development to validation (patients of one site increases from 10 to 38%). Applied prospectively for nine months, 41,728 predictions were generated in real-time (median [IQR], 1.3 [0.9, 32] minutes). An operating criterion of 75% positive predictive value identified 104 predictions at very high risk (0.25%) where 65% (50 from 77 well-timed predictions) led to death within 60 days.

CONCLUSION

Temporal validation demonstrates good model discrimination for 60-day mortality. Slight performance variations are observed across demographic subpopulations. The model was implemented prospectively and successfully produced meaningful estimates of risk within minutes of admission.