Calibration: the Achilles heel of predictive analytics

Risk Stratification in Patients with Ischemic Stroke and Residual Cardiovascular Risk with Current Secondary Prevention

PURPOSE

Suboptimal secondary prevention in patients with stroke causes a remaining cardiovascular risk desirable to reduce. We have validated a prognostic model for secondary preventive settings and estimated future cardiovascular risk and theoretical benefit of reaching guideline recommended risk factor targets.

PATIENTS AND METHODS

The SMART-REACH (Secondary Manifestations of Arterial Disease-Reduction of Atherothrombosis for Continued Health) model for 10-year and lifetime risk of cardiovascular events was applied to 465 patients in the Norwegian Cognitive Impairment After Stroke (Nor-COAST) study, a multicenter observational study with two-year follow-up by linkage to national registries for cardiovascular disease and mortality. The residual risk when reaching recommended targets for blood pressure, low-density lipoprotein cholesterol, smoking cessation and antithrombotics was estimated.

RESULTS

In total, 11.2% had a new event. Calibration plots showed adequate agreement between estimated and observed 2-year prognosis (C-statistics 0.63, 95% confidence interval 0.55-0.71). Median estimated 10-year risk of recurrent cardiovascular events was 42% (Interquartile range (IQR) 32-54%) and could be reduced to 32% by optimal guideline-based therapy. The corresponding numbers for lifetime risk were 70% (IQR 63-76%) and 61%. We estimated an overall median gain of 1.4 (IQR 0.2-3.4) event-free life years if guideline targets were met.

CONCLUSION

Secondary prevention was suboptimal and residual risk remains elevated even after optimization according to current guidelines. Considerable interindividual variation in risk exists, with a corresponding variation in benefit from intensification of treatment. The SMART-REACH model can be used to identify patients with the largest benefit from more intensive treatment and follow-up.

Predicting the risk of cancer in adults using supervised machine learning: a scoping review

OBJECTIVES

The purpose of this scoping review is to: (1) identify existing supervised machine learning (ML) approaches on the prediction of cancer in asymptomatic adults; (2) to compare the performance of ML models with each other and (3) to identify potential gaps in research.

DESIGN

Scoping review using the population, concept and context approach.

SEARCH STRATEGY

PubMed search engine was used from inception to 10 November 2020 to identify literature meeting following inclusion criteria: (1) a general adult (≥18 years) population, either sex, asymptomatic (population); (2) any study using ML techniques to derive predictive models for future cancer risk using clinical and/or demographic and/or basic laboratory data (concept) and (3) original research articles conducted in all settings in any region of the world (context).

RESULTS

The search returned 627 unique articles, of which 580 articles were excluded because they did not meet the inclusion criteria, were duplicates or were related to benign neoplasm. Full-text reviews were conducted for 47 articles and a final set of 10 articles were included in this scoping review. These 10 very heterogeneous studies used ML to predict future cancer risk in asymptomatic individuals. All studies reported area under the receiver operating characteristics curve (AUC) values as metrics of model performance, but no study reported measures of model calibration.

CONCLUSIONS

Research gaps that must be addressed in order to deliver validated ML-based models to assist clinical decision-making include: (1) establishing model generalisability through validation in independent cohorts, including those from low-income and middle-income countries; (2) establishing models for all cancer types; (3) thorough comparisons of ML models with best available clinical tools to ensure transparency of their potential clinical utility; (4) reporting of model calibration performance and (5) comparisons of different methods on the same cohort to reveal important information about model generalisability and performance.

The Use of Machine Learning for the Care of Hypertension and Heart Failure

Machine learning (ML) is a branch of artificial intelligence that combines computer science, statistics, and decision theory to learn complex patterns from voluminous data. In the last decade, accumulating evidence has shown the utility of ML for prediction, diagnosis, and classification of hypertension and heart failure (HF). In addition, ML-enabled image analysis has potential value in assessing cardiac structure and function in an accurate, scalable, and efficient way. Considering the high burden of hypertension and HF in China and worldwide, ML may help address these challenges from different aspects. Indeed, prior studies have shown that ML can enhance each stage of patient care, from research and development, to daily clinical practice and population health. Through reviewing the published literature, the aims of the current systemic review are to summarize the utilities of ML for the care of those with hypertension and HF.

Minimum sample size for external validation of a clinical prediction model with a binary outcome

In prediction model research, external validation is needed to examine an existing model's performance using data independent to that for model development. Current external validation studies often suffer from small sample sizes and consequently imprecise predictive performance estimates. To address this, we propose how to determine the minimum sample size needed for a new external validation study of a prediction model for a binary outcome. Our calculations aim to precisely estimate calibration (Observed/Expected and calibration slope), discrimination (C-statistic), and clinical utility (net benefit). For each measure, we propose closed-form and iterative solutions for calculating the minimum sample size required. These require specifying: (i) target SEs (confidence interval widths) for each estimate of interest, (ii) the anticipated outcome event proportion in the validation population, (iii) the prediction model's anticipated (mis)calibration and variance of linear predictor values in the validation population, and (iv) potential risk thresholds for clinical decision-making. The calculations can also be used to inform whether the sample size of an existing (already collected) dataset is adequate for external validation. We illustrate our proposal for external validation of a prediction model for mechanical heart valve failure with an expected outcome event proportion of 0.018. Calculations suggest at least 9835 participants (177 events) are required to precisely estimate the calibration and discrimination measures, with this number driven by the calibration slope criterion, which we anticipate will often be the case. Also, 6443 participants (116 events) are required to precisely estimate net benefit at a risk threshold of 8%. Software code is provided.

Predicting mortality in cardiac care unit patients: external validation of the Mayo cardiac intensive care unit admission risk score

AIMS

The Mayo Cardiac Intensive Care Unit Admission Risk Score (M-CARS) had excellent performance in predicting in-hospital mortality in the US population. We sought to validate the M-CARS for in-hospital and post-discharge mortality in Asian patients admitted to the cardiac care unit (CCU).

METHODS AND RESULTS

Patients admitted to the CCU of a tertiary care centre between July 2015 and December 2019 were included into the study. Patients with intra-hospital transfer to the CCU due to intensive care unit overflow, postoperative cardiac surgery, or for monitoring after elective procedures were excluded. Cardiac arrest, cardiogenic shock, respiratory failure, Braden skin score, blood urea nitrogen, anion gap, and red cell distribution width, were used to calculate the M-CARS. Patients were stratified into three groups, according to the M-CARS (<2, 2-6, >6). Of 1988 patients in the study, 30.1% were female with a median age of 65 years. Prevalence of cardiogenic shock and respiratory failure at admission were 2.8% and 4.5%, respectively. One hundred and seventeen patients died during the admission (mortality rate of 5.9%). The in-hospital mortality rate in patients with M-CARS of <2, 2-6, and >6 was 1.1%, 9.8%, and 35.5%, respectively. C-statistic of M-CARS for in-hospital mortality was 0.840 (95% CI 0.805-0.873); whereas, it was 0.727 (95% CI 0.690-0.761) for 1-year post-discharge mortality. Calibration plot showed good agreement between predicted and observed in-hospital mortality in the majority of patients.

CONCLUSIONS

The M-CARS was useful in our study, in terms of discrimination and calibration. M-CARS identified high-risk patients in CCU, who had unacceptably high mortality rate during hospital stay and thereafter.

Validation and update of a multivariable prediction model for the identification and management of patients at risk for hepatocellular carcinoma

Background

A hepatocellular carcinoma (HCC) prediction model (ASAP), including age, sex, and the biomarkers alpha-fetoprotein and prothrombin induced by vitamin K absence-II, showed potential clinical value in the early detection of HCC. We validated and updated the model in a real-world cohort and promoted its transferability to daily clinical practice.

Methods

This retrospective cohort analysis included 1012 of the 2479 eligible patients aged 35 years or older undergoing surveillance for HCC. The data were extracted from the electronic medical records. Biomarker values within the test-to-diagnosis interval were used to validate the ASAP model. Due to its unsatisfactory calibration, three logistic regression models were constructed to recalibrate and update the model. Their discrimination, calibration, and clinical utility were compared. The performance statistics of the final updated model at several risk thresholds are presented. The outcomes of 855 non-HCC patients were further assessed during a median of 10.2 months of follow-up. Statistical analyses were performed using packages in R software.

Results

The ASAP model had superior discriminative performance in the validation cohort [C-statistic = 0.982, (95% confidence interval 0.972–0.992)] but significantly overestimated the risk of HCC (intercept − 3.243 and slope 1.192 in the calibration plot), reducing its clinical usefulness. Recalibration-in-the-large, which exhibited performance comparable to that of the refitted model revision, led to the retention of the excellent discrimination and substantial improvements in the calibration and clinical utility, achieving a sensitivity of 100% at the median prediction probability of the absence of HCC (1.3%). The probability threshold of 1.3% and the incidence of HCC in the cohort (15.5%) were used to stratify the patients into low-, medium-, and high-risk groups. The cumulative HCC incidences in the non-HCC patients significantly differed among the risk groups (log-rank test, p-value < 0.001). The 3-month, 6-month and 18-month cumulative incidences in the low-risk group were 0.6%, 0.9% and 0.9%, respectively.

Conclusions

The ASAP model is an accurate tool for HCC risk estimation that requires recalibration before use in a new region because calibration varies with clinical environments. Additionally, rational risk stratification and risk-based management decision-making, e.g., 3-month follow-up recommendations for targeted individuals, helped improve HCC surveillance, which warrants assessment in larger cohorts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12014-021-09326-w.

Preparing Laboratories for Interconnected Health Care

In an increasingly interconnected health care system, laboratory medicine can facilitate diagnosis and treatment of patients effectively. This article describes necessary changes and points to potential challenges on a technical, content, and organizational level. As a technical precondition, electronic laboratory reports have to become machine-readable and interpretable. Terminologies such as Logical Observation Identifiers Names and Codes (LOINC), Nomenclature for Properties and Units (NPU), Unified Code for Units of Measure (UCUM), and SNOMED-CT can lead to the necessary semantic interoperability. Even if only single “atomized” results of the whole report are extracted, the necessary information for correct interpretation must be available. Therefore, interpretive comments, e.g., concerns about an increased measurement uncertainty must be electronically attached to every affected measurement result. Standardization of laboratory analyses with traceable standards and reference materials will enable knowledge transfer and safe interpretation of laboratory analyses from multiple laboratories. In an interconnected health care system, laboratories should strive to transform themselves into a data hub that not only receives samples but also extensive information about the patient. On that basis, they can return measurement results enriched with high-quality interpretive comments tailored to the individual patient and unlock the full potential of laboratory medicine.

Chest radiograph-based artificial intelligence predictive model for mortality in community-acquired pneumonia

Background

Chest radiograph (CXR) is a basic diagnostic test in community-acquired pneumonia (CAP) with prognostic value. We developed a CXR-based artificial intelligence (AI) model (CAP AI predictive Engine: CAPE) and prospectively evaluated its discrimination for 30-day mortality.

Methods

Deep-learning model using convolutional neural network (CNN) was trained with a retrospective cohort of 2235 CXRs from 1966 unique adult patients admitted for CAP from 1 January 2019 to 31 December 2019. A single-centre prospective cohort between 11 May 2020 and 15 June 2020 was analysed for model performance. CAPE mortality risk score based on CNN analysis of the first CXR performed for CAP was used to determine the area under the receiver operating characteristic curve (AUC) for 30-day mortality.

Results

315 inpatient episodes for CAP occurred, with 30-day mortality of 19.4% (n=61/315). Non-survivors were older than survivors (mean (SD)age, 80.4 (10.3) vs 69.2 (18.7)); more likely to have dementia (n=27/61 vs n=58/254) and malignancies (n=16/61 vs n=18/254); demonstrate higher serum C reactive protein (mean (SD), 109 mg/L (98.6) vs 59.3 mg/L (69.7)) and serum procalcitonin (mean (SD), 11.3 (27.8) μg/L vs 1.4 (5.9) μg/L). The AUC for CAPE mortality risk score for 30-day mortality was 0.79 (95% CI 0.73 to 0.85, p<0.001); Pneumonia Severity Index (PSI) 0.80 (95% CI 0.74 to 0.86, p<0.001); Confusion of new onset, blood Urea nitrogen, Respiratory rate, Blood pressure, 65 (CURB-65) score 0.76 (95% CI 0.70 to 0.81, p<0.001), respectively. CAPE combined with CURB-65 model has an AUC of 0.83 (95% CI 0.77 to 0.88, p<0.001). The best performing model was CAPE incorporated with PSI, with an AUC of 0.84 (95% CI 0.79 to 0.89, p<0.001).

Conclusion

CXR-based CAPE mortality risk score was comparable to traditional pneumonia severity scores and improved its discrimination when combined.

Machine Learning Models for Classification of Cushing's Syndrome Using Retrospective Data

Accurate classification of Cushing's Syndrome (CS) plays a critical role in providing the early and correct diagnosis of CS that may facilitate treatment and improve patient outcomes. Diagnosis of CS is a complex process, which requires careful and concurrent interpretation of signs and symptoms, multiple biochemical test results, and findings of medical imaging by physicians with a high degree of specialty and knowledge to make correct judgments. In this article, we explore the state of the art machine learning algorithms to demonstrate their potential as a clinical decision support system to analyze and classify CS to facilitate the diagnosis, prognosis, and treatment of CS. Prominent algorithms are compared using nested cross-validation and various class comparison strategies including multiclass, one vs. all, and one vs. one binary classification. Our findings show that Random Forest (RF) algorithm is most suitable for the classification of CS. We demonstrate that the proposed approach can classify CS with an average accuracy of 92% and an average F1 score of 91.5%, depending on the class comparison strategy and selected features. RF-based one vs. all binary classification model achieves sensitivity of 97.6%, precision of 91.1%, and specificity of 87.1% to discriminate CS from non-CS on the test dataset. RF-based multiclass classification model achieves average per class sensitivity of 91.8%, average per class specificity of 97.1%, and average per class precision of 92.1% to classify different subtypes of CS on the test dataset. Clinical performance evaluation suggests that the developed models can help improve physicians' judgment in diagnosing CS.

A Multivariable Predictive Model for Success of External Cephalic Version

OBJECTIVE

To create a prediction model for external cephalic version (ECV) success using objective patient characteristics.

METHODS

This retrospective study included pregnant individuals of at least 18 years of age with a nonanomalous, singleton gestation who underwent an ECV attempt between 2006 and 2016 at a single quaternary care hospital. Variables assessed included maternal age, height, weight, body mass index (BMI), parity, fetal sex, gestational age, estimated fetal weight, type of fetal malpresentation, placental location, and amniotic fluid volume. Univariable and multivariable logistic regression models were used to determine the association of patient characteristics with ECV success. Estimated odds ratios and corresponding 95% CIs were calculated for each variable, and backward elimination and bootstrapping were used to find a parsimonious model for ECV success with the highest discriminatory capacity (as determined by the area under the receiver operating characteristic curve [AUC]). This model was evaluated with a calibration curve across deciles of success.

RESULTS

A total of 1,138 individuals underwent an ECV attempt and were included in this analysis. The overall ECV success frequency was 40.6%. Factors significantly associated with ECV success were maternal age, parity, placental location, estimated fetal weight, and type of fetal malpresentation. A final model with BMI, parity, placental location, and type of fetal malpresentation had the highest AUC (0.667 [95% CI 0.634-0.701]), resulted in good calibration, and is represented by the following equation: 1/[1+e-x] where x=1.1726-0.0314 (BMI)-0.9299 (nulliparity)+1.0218 (transverse or oblique presentation at ECV)-0.5113 (anterior placenta). An interactive version of this equation was created and can be accessed at www.ecvcalculator.com.

CONCLUSION

A prediction model that estimates the probability of ECV success was created and internally validated. This model incorporates easily obtainable and objective patient factors known before ECV and may be used in decision making and patient counseling about ECV.

Predicting hospital admissions from individual patient data (IPD): an applied example to explore key elements driving external validity

OBJECTIVE

To explore factors that potentially impact external validation performance while developing and validating a prognostic model for hospital admissions (HAs) in complex older general practice patients.

STUDY DESIGN AND SETTING

Using individual participant data from four cluster-randomised trials conducted in the Netherlands and Germany, we used logistic regression to develop a prognostic model to predict all-cause HAs within a 6-month follow-up period. A stratified intercept was used to account for heterogeneity in baseline risk between the studies. The model was validated both internally and by using internal-external cross-validation (IECV).

RESULTS

Prior HAs, physical components of the health-related quality of life comorbidity index, and medication-related variables were used in the final model. While achieving moderate discriminatory performance, internal bootstrap validation revealed a pronounced risk of overfitting. The results of the IECV, in which calibration was highly variable even after accounting for between-study heterogeneity, agreed with this finding. Heterogeneity was equally reflected in differing baseline risk, predictor effects and absolute risk predictions.

CONCLUSIONS

Predictor effect heterogeneity and differing baseline risk can explain the limited external performance of HA prediction models. With such drivers known, model adjustments in external validation settings (eg, intercept recalibration, complete updating) can be applied more purposefully.

TRIAL REGISTRATION NUMBER

PROSPERO id: CRD42018088129.

External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

BACKGROUND

There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated.

METHODS

A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data.

RESULTS

We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001).

CONCLUSIONS

Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

Assessing opioid overdose risk: a review of clinical prediction models utilizing patient-level data

Drug, and specifically opioid-related, overdoses remain a major public health problem in the United States. Multiple studies have examined individual risk factors associated with overdose risk, but research developing clinical risk prediction tools for overdose has only emerged in the last few years. We conducted a comprehensive review of the literature on patient-level factors associated with opioid-related overdose risk, with an emphasis on clinical risk prediction models for opioid-related overdose in the United States. Studies that developed and/or validated clinical prediction models were closely reviewed and evaluated to determine the state of the field. We identified 12 studies that reported risk prediction models for opioid-related overdose risk. Published models were developed from a variety of data sources, including Veterans Health Administration data, Medicare data, commercial insurance data, and statewide linked datasets. Studies reported model performance using measures of discrimination, usually at good-to-excellent levels, though they did not always assess calibration. C-statistics were better for models that included clinical predictors (c-statistics: 0.75-0.95) compared to models without them (c-statistics: 0.69-0.82). External validation of models was rare, and we found no studies evaluating implementation of models or risk prediction tools into clinical practice. A common feature of these models was a high rate of false positives, largely because opioid-related overdose is rare in the general population. Thus, efforts to implement prediction models into practice should take into account that published models overestimate overdose risk for many low-risk patients. Future prediction models assessing overdose risk should employ external validation and address model calibration. In order to translate findings from prediction models into clinical public health benefit, future studies should focus on developing clinical prediction tools based on prediction models, implementing these tools into clinical practice, and evaluating the impact of these models on treatment decisions, patient outcomes, and, ultimately, opioid overdose rates.

XGBoost Algorithm Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information

Background

The emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge.

Objective

This study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS).

Methods

To predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer–Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis.

Results

The AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction.

Conclusions

Our study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.

Recommendations for Statistical Reporting in Cardiovascular Medicine: A Special Report From the American Heart Association

Statistical analyses are a crucial component of the biomedical research process and are necessary to draw inferences from biomedical research data. The application of sound statistical methodology is a prerequisite for publication in the American Heart Association (AHA) journal portfolio. The objective of this document is to summarize key aspects of statistical reporting that might be most relevant to the authors, reviewers, and readership of AHA journals. The AHA Scientific Publication Committee convened a task force to inventory existing statistical standards for publication in biomedical journals and to identify approaches suitable for the AHA journal portfolio. The experts on the task force were selected by the AHA Scientific Publication Committee, who identified 12 key topics that serve as the section headers for this document. For each topic, the members of the writing group identified relevant references and evaluated them as a resource to make the standards summarized herein. Each section was independently reviewed by an expert reviewer who was not part of the task force. Expert reviewers were also permitted to comment on other sections if they chose. Differences of opinion were adjudicated by consensus. The standards presented in this report are intended to serve as a guide for high-quality reporting of statistical analyses methods and results.

Probabilistic Machine Learning for Healthcare

Machine learning can be used to make sense of healthcare data. Probabilistic machine learning models help provide a complete picture of observed data in healthcare. In this review, we examine how probabilistic machine learning can advance healthcare. We consider challenges in the predictive model building pipeline where probabilistic models can be beneficial, including calibration and missing data. Beyond predictive models, we also investigate the utility of probabilistic machine learning models in phenotyping, in generative models for clinical use cases, and in reinforcement learning.

Predicting critical state after COVID-19 diagnosis: model development using a large US electronic health record dataset

As the COVID-19 pandemic is challenging healthcare systems worldwide, early identification of patients with a high risk of complication is crucial. We present a prognostic model predicting critical state within 28 days following COVID-19 diagnosis trained on data from US electronic health records (IBM Explorys), including demographics, comorbidities, symptoms, and hospitalization. Out of 15753 COVID-19 patients, 2050 went into critical state or deceased. Non-random train-test splits by time were repeated 100 times and led to a ROC AUC of 0.861 [0.838, 0.883] and a precision-recall AUC of 0.434 [0.414, 0.485] (median and interquartile range). The interpretability analysis confirmed evidence on major risk factors (e.g., older age, higher BMI, male gender, diabetes, and cardiovascular disease) in an efficient way compared to clinical studies, demonstrating the model validity. Such personalized predictions could enable fine-graded risk stratification for optimized care management.

Review of Statistical Methods for Evaluating the Performance of Survival or Other Time-to-Event Prediction Models (from Conventional to Deep Learning Approaches)

The recent introduction of various high-dimensional modeling methods, such as radiomics and deep learning, has created a much greater diversity in modeling approaches for survival prediction (or, more generally, time-to-event prediction). The newness of the recent modeling approaches and unfamiliarity with the model outputs may confuse some researchers and practitioners about the evaluation of the performance of such models. Methodological literacy to critically appraise the performance evaluation of the models and, ideally, the ability to conduct such an evaluation would be needed for those who want to develop models or apply them in practice. This article intends to provide intuitive, conceptual, and practical explanations of the statistical methods for evaluating the performance of survival prediction models with minimal usage of mathematical descriptions. It covers from conventional to deep learning methods, and emphasis has been placed on recent modeling approaches. This review article includes straightforward explanations of C indices (Harrell's C index, etc.), time-dependent receiver operating characteristic curve analysis, calibration plot, other methods for evaluating the calibration performance, and Brier score.

Application of an individualized nomogram in first-trimester screening for trisomy 21

OBJECTIVES

To develop and validate a nomogram based on fetal nuchal translucency thickness (NT) and ultrasonographic facial markers for screening for trisomy 21 in the first trimester of pregnancy.

METHODS

This was a retrospective case-control study using stored two-dimensional midsagittal fetal profile images captured at 11 + 0 to 13 + 6 weeks' gestation in singleton pregnancies. We included images from 302 trisomy-21 pregnancies and 322 euploid pregnancies. Cases were divided into a training set (200 euploid + 200 with trisomy 21) and a validation set (122 euploid + 102 with trisomy 21) at a ratio of approximately 2:1. For each, the maternal age, gestational age, fetal NT and karyotype were noted, and 12 ultrasonographic fetal facial markers were measured. The least absolute shrinkage and selection operator (LASSO) method and multivariable analysis were used to select automatically the discriminative markers. Logistic regression was used to develop a LASSO model, based on the selected markers, to screen for trisomy 21 in the first trimester of pregnancy. Furthermore, 60 of the 624 images were selected randomly as a retest set to evaluate the model's robustness. The predictive performance of screening for trisomy 21 of a model based on fetal NT and maternal age and of the LASSO model was assessed using the area under the receiver-operating-characteristics curve (AUC). A nomogram was developed as an individualized tool to predict patient-specific probability for trisomy 21, which is a more visual presentation of the LASSO model. The performance of the nomogram was assessed using the C-index and calibration curve.

RESULTS

Into the LASSO model were incorporated eight markers, including fetal NT, prenasal-thickness-to-nasal-bone-length ratio, facial profile line, frontomaxillary facial angle, frontonasal facial angle, mandibulomaxillary facial angle, maxilla-nasion-mandible angle and d2 (distance between the anterior edge of the prefrontal skin and the mandibulomaxillary line) (all P < 0.05). The AUCs of the LASSO model for screening for trisomy 21 were 0.983 (95% CI, 0.971-0.994) in the training set and 0.979 (95% CI, 0.966-0.993) in the validation set, and these were higher than the AUCs of all eight individual ultrasonographic markers included in the model. The AUC of the LASSO model in the retest set was 0.997 (95% CI, 0.990-1.000), indicating good robustness of the LASSO model. The AUC of the LASSO model was significantly higher than that of the model based on fetal NT and maternal age in both training and validation sets (P < 0.001 for both). The nomogram of the LASSO model showed good discrimination of trisomy 21, with C-indices of 0.983 in the training set and 0.981 in the validation set.

CONCLUSIONS

We present an individualized nomogram which incorporates fetal NT and a series of ultrasonographic facial profile markers selected by the LASSO method and multivariable analysis. This nomogram can potentially be utilized as a convenient and effective tool in screening for trisomy 21 in the first trimester of pregnancy. © 2020 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Personalized fracture risk assessment: where are we at?

Introduction: Osteoporotic fracture imposes a significant health care burden globally. Personalized assessment of fracture risk can potentially guide treatment decisions. Over the past decade, a number of risk prediction models, including the Garvan Fracture Risk Calculator (Garvan) and FRAX®, have been developed and implemented in clinical practice. Areas covered: This article reviews recent development and validation results concerning the prognostic performance of the two tools. The main areas of review are the need for personalized fracture risk prediction, purposes of risk prediction, predictive performance in terms of discrimination and calibration, concordance between the Garvan and FRAX tools, genetic profiling for improving predictive performance, and treatment thresholds. In some validation studies, FRAX tended to underestimate fracture by as high as 50%. Studies have shown that the predicted risk from the Garvan tool is highly concordant with clinical decision. Expert opinion: Although there are some discrepancy in fracture risk prediction between Garvan and FRAX, both tools are valid and can aid patients and doctors communicate about risk and make informed decision. The ideal of personalized risk assessment for osteoporosis patients will be realized through the incorporation of genetic profiling into existing fracture risk assessment tools.

Machine Learning Prediction Models for Mechanically Ventilated Patients: Analyses of the MIMIC-III Database

Background: Mechanically ventilated patients in the intensive care unit (ICU) have high mortality rates. There are multiple prediction scores, such as the Simplified Acute Physiology Score II (SAPS II), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA), widely used in the general ICU population. We aimed to establish prediction scores on mechanically ventilated patients with the combination of these disease severity scores and other features available on the first day of admission.

Methods: A retrospective administrative database study from the Medical Information Mart for Intensive Care (MIMIC-III) database was conducted. The exposures of interest consisted of the demographics, pre-ICU comorbidity, ICU diagnosis, disease severity scores, vital signs, and laboratory test results on the first day of ICU admission. Hospital mortality was used as the outcome. We used the machine learning methods of k-nearest neighbors (KNN), logistic regression, bagging, decision tree, random forest, Extreme Gradient Boosting (XGBoost), and neural network for model establishment. A sample of 70% of the cohort was used for the training set; the remaining 30% was applied for testing. Areas under the receiver operating characteristic curves (AUCs) and calibration plots would be constructed for the evaluation and comparison of the models' performance. The significance of the risk factors was identified through models and the top factors were reported.

Results: A total of 28,530 subjects were enrolled through the screening of the MIMIC-III database. After data preprocessing, 25,659 adult patients with 66 predictors were included in the model analyses. With the training set, the models of KNN, logistic regression, decision tree, random forest, neural network, bagging, and XGBoost were established and the testing set obtained AUCs of 0.806, 0.818, 0.743, 0.819, 0.780, 0.803, and 0.821, respectively. The calibration curves of all the models, except for the neural network, performed well. The XGBoost model performed best among the seven models. The top five predictors were age, respiratory dysfunction, SAPS II score, maximum hemoglobin, and minimum lactate.

Conclusion: The current study indicates that models with the risk of factors on the first day could be successfully established for predicting mortality in ventilated patients. The XGBoost model performs best among the seven machine learning models.

Embryo selection with artificial intelligence: how to evaluate and compare methods?

Embryo selection within in vitro fertilization (IVF) is the process of evaluating qualities of fertilized oocytes (embryos) and selecting the best embryo(s) available within a patient cohort for subsequent transfer or cryopreservation. In recent years, artificial intelligence (AI) has been used extensively to improve and automate the embryo ranking and selection procedure by extracting relevant information from embryo microscopy images. The AI models are evaluated based on their ability to identify the embryo(s) with the highest chance(s) of achieving a successful pregnancy. Whether such evaluations should be based on ranking performance or pregnancy prediction, however, seems to divide studies. As such, a variety of performance metrics are reported, and comparisons between studies are often made on different outcomes and data foundations. Moreover, superiority of AI methods over manual human evaluation is often claimed based on retrospective data, without any mentions of potential bias. In this paper, we provide a technical view on some of the major topics that divide how current AI models are trained, evaluated and compared. We explain and discuss the most common evaluation metrics and relate them to the two separate evaluation objectives, ranking and prediction. We also discuss when and how to compare AI models across studies and explain in detail how a selection bias is inevitable when comparing AI models against current embryo selection practice in retrospective cohort studies.

External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb

INTRODUCTION

Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach.

METHODS

Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility.

RESULTS

Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model.

CONCLUSION

Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.

Using machine learning to create a decision tree model to predict outcomes of COVID-19 cases in the Philippines

OBJECTIVE

The aim of this study was to create a decision tree model with machine learning to predict the outcomes of COVID-19 cases from data publicly available in the Philippine Department of Health (DOH) COVID Data Drop.

METHODS

The study design was a cross-sectional records review of the DOH COVID Data Drop for 25 August 2020. Resolved cases that had either recovered or died were used as the final data set. Machine learning processes were used to generate, train and validate a decision tree model.

RESULTS

A list of 132 939 resolved COVID-19 cases was used. The notification rates and case fatality rates were higher among males (145.67 per 100 000 and 2.46%, respectively). Most COVID-19 cases were clustered among people of working age, and older cases had higher case fatality rates. The majority of cases were from the National Capital Region (590.20 per 100 000), and the highest case fatality rate (5.83%) was observed in Region VII. The decision tree model prioritized age and history of hospital admission as predictors of mortality. The model had high accuracy (81.42%), sensitivity (81.65%), specificity (81.41%) and area under the curve (0.876) but a poor F-score (16.74%).

DISCUSSION

The model predicted higher case fatality rates among older people. For cases aged > 51 years, a history of hospital admission increased the probability of COVID-19-related death. We recommend that more comprehensive primary COVID-19 data sets be used to create more robust prognostic models.

Predictive scoring model for inferior alveolar nerve injury after lower third molar removal based on features of cone-beam computed tomography image

INTRODUCTION

This study aimed to construct a predictive scoring system for inferior alveolar nerve injury (IANI) following lower third molar (LM3) surgery based on cone-beam computed tomography (CBCT) images.

MATERIAL AND METHODS

Of the 1573 patients who underwent LM3 removal following the CBCT, 39 with IANI and 457 randomly selected patients without IANI were enrolled. We collected information regarding the demographic characteristics of the patients, surgical situations, and inferior alveolar canal (IAC)-related CBCT factors. The association with IANI-risk was evaluated with a backward stepwise logistic regression model as per the Akaike information criterion. Scoring models' abilities of discrimination (area under the curve) and calibration (Hosmer-Lemeshow test and calibration plots) were assessed, followed by evaluation of the clinical usefulness using decision curve analysis.

RESULTS

As per the multivariate analysis, the coronal positioned IAC on the enlarged root (odds ratio [OR], 3.78; P = 0.001), the length of perforated IAC (>3.4 mm) (OR, 3.05; P = 0.012), lingual/inter-radicular position of the IAC (OR, 3.96; P = 0.001), multiple roots closed to the perforated IAC (OR, 2.78; P = 0.025), and age >30 y (OR, 2.31; P = 0.076) were identified in the extended scoring model ranging from 0 to 12. This model was compared with our previously constructed baseline model that involved the latter three variables mentioned above, resulting in superior performance than that of the baseline model.

CONCLUSION

The extended model would be a useful tool for reliable determination of the preoperative probability of IANI.

Modelling hospital outcome: problems with endogeneity

Background

Mortality modelling in the critical care paradigm traditionally uses logistic regression, despite the availability of estimators commonly used in alternate disciplines. Little attention has been paid to covariate endogeneity and the status of non-randomized treatment assignment. Using a large registry database, various binary outcome modelling strategies and methods to account for covariate endogeneity were explored.

Methods

Patient mortality data was sourced from the Australian & New Zealand Intensive Society Adult Patient Database for 2016. Hospital mortality was modelled using logistic, probit and linear probability (LPM) models with intensive care (ICU) providers as fixed (FE) and random (RE) effects. Model comparison entailed indices of discrimination and calibration, information criteria (AIC and BIC) and binned residual analysis. Suspect covariate and ventilation treatment assignment endogeneity was identified by correlation between predictor variable and hospital mortality error terms, using the Stata™ “eprobit” estimator. Marginal effects were used to demonstrate effect estimate differences between probit and “eprobit” models.

Results

The cohort comprised 92,693 patients from 124 intensive care units (ICU) in calendar year 2016. Patients mean age was 61.8 (SD 17.5) years, 41.6% were female and APACHE III severity of illness score 54.5(25.6); 43.7% were ventilated. Of the models considered in predicting hospital mortality, logistic regression (with or without ICU FE) and RE logistic regression dominated, more so the latter using information criteria indices. The LPM suffered from many predictions outside the unit [0,1] interval and both poor discrimination and calibration. Error terms of hospital length of stay, an independent risk of death score and ventilation status were correlated with the mortality error term. Marked differences in the ventilation mortality marginal effect was demonstrated between the probit and the "eprobit" models which were scenario dependent. Endogeneity was not demonstrated for the APACHE III score.

Conclusions

Logistic regression accounting for provider effects was the preferred estimator for hospital mortality modelling. Endogeneity of covariates and treatment variables may be identified using appropriate modelling, but failure to do so yields problematic effect estimates.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01251-8.

Risk Factors and Prediction Models for Venous Thromboembolism in Ambulatory Patients with Lung Cancer

Venous thromboembolism (VTE) is a significant cause of mortality in patients with lung cancer. Despite the availability of a wide range of anticoagulants to help prevent thrombosis, thromboprophylaxis in ambulatory patients is a challenge due to its associated risk of haemorrhage. As a result, anticoagulation is only recommended in patients with a relatively high risk of VTE. Efforts have been made to develop predictive models for VTE risk assessment in cancer patients, but the availability of a reliable predictive model for ambulate patients with lung cancer is unclear. We have analysed the latest information on this topic, with a focus on the lung cancer-related risk factors for VTE, and risk prediction models developed and validated in this group of patients. The existing risk models, such as the Khorana score, the PROTECHT score and the CONKO score, have shown poor performance in external validations, failing to identify many high-risk individuals. Some of the newly developed and updated models may be promising, but their further validation is needed.

Corticosteroid Randomization after Significant Head Injury and International Mission for Prognosis and Clinical Trialsin Traumatic Brain Injury Models Compared with a Machine Learning-Based Predictive Model from Tanzania

Hospitals in low- and middle-income countries (LMICs) could benefit from decision support technologies to reduce time to triage, diagnosis, and surgery for patients with traumatic brain injury (TBI). Corticosteroid Randomization after Significant Head Injury (CRASH) and International Mission for Prognosis and Clinical Trials in Traumatic Brain Injury (IMPACT) are robust examples of TBI prognostic models, although they have yet to be validated in Sub-Saharan Africa (SSA). Moreover, machine learning and improved data quality in LMICs provide an opportunity to develop context-specific, and potentially more accurate, prognostic models. We aim to externally validate CRASH and IMPACT on our TBI registry and compare their performances to that of the locally derived model (from the Kilimanjaro Christian Medical Center [KCMC]). We developed a machine learning-based prognostic model from a TBI registry collected at a regional referral hospital in Moshi, Tanzania. We also used the core CRASH and IMPACT online risk calculators to generate risk scores for each patient. We compared the discrimination (area under the curve [AUC]) and calibration before and after Platt scaling (Brier, Hosmer-Lemeshow Test, and calibration plots) for CRASH, IMPACT, and the KCMC model. The outcome of interest was unfavorable in-hospital outcome defined as a Glasgow Outcome Scale score of 1-3. There were 2972 patients included in the TBI registry, of whom 11% had an unfavorable outcome. The AUCs for the KCMC model, CRASH, and IMPACT were 0.919, 0.876, and 0.821, respectively. Prior to Platt scaling, CRASH was the best calibrated model (χ² = 68.1) followed by IMPACT (χ² = 380.9) and KCMC (χ² = 1025.6). We provide the first SSA validation of the core CRASH and IMPACT models. The KCMC model had better discrimination than either of these. CRASH had the best calibration, although all model predictions could be successfully calibrated. The top performing models, KCMC and CRASH, were both developed using LMIC data, suggesting that locally derived models may outperform imported ones from different contexts of care. Further work is needed to externally validate the KCMC model.

The need to separate the wheat from the chaff in medical informatics: Introducing a comprehensive checklist for the (self)-assessment of medical AI studies

This editorial aims to contribute to the current debate about the quality of studies that apply machine learning (ML) methodologies to medical data to extract value from them and provide clinicians with viable and useful tools supporting everyday care practices. We propose a practical checklist to help authors to self assess the quality of their contribution and to help reviewers to recognize and appreciate high-quality medical ML studies by distinguishing them from the mere application of ML techniques to medical data.

Prediction of Childhood Obesity from Nationwide Health Records

OBJECTIVE

To evaluate body mass index (BMI) acceleration patterns in children and to develop a prediction model targeted to identify children at high risk for obesity before the critical time window in which the largest increase in BMI percentile occurs.

STUDY DESIGN

We analyzed electronic health records of children from Israel's largest healthcare provider from 2002 to 2018. Data included demographics, anthropometric measurements, medications, diagnoses, and laboratory tests of children and their families. Obesity was defined as BMI ≥95th percentile for age and sex. To identify the time window in which the largest annual increases in BMI z score occurs during early childhood, we first analyzed childhood BMI acceleration patterns among 417 915 adolescents. Next, we devised a model targeted to identify children at high risk before this time window, predicting obesity at 5-6 years of age based on data from the first 2 years of life of 132 262 children.

RESULTS

Retrospective BMI analysis revealed that among adolescents with obesity, the greatest acceleration in BMI z score occurred between 2 and 4 years of age. Our model, validated temporally and geographically, accurately predicted obesity at 5-6 years old (area under the receiver operating characteristic curve of 0.803). Discrimination results on subpopulations demonstrated its robustness across the pediatric population. The model's most influential predictors included anthropometric measurements of the child and family. Other impactful predictors included ancestry and pregnancy glucose.

CONCLUSIONS

Rapid rise in the prevalence of childhood obesity warrant the development of better prevention strategies. Our model may allow an accurate identification of children at high risk of obesity.

Avoiding Unnecessary Biopsy: MRI-based Risk Models versus a PI-RADS and PSA Density Strategy for Clinically Significant Prostate Cancer

Background In validation studies, risk models for clinically significant prostate cancer (csPCa; Gleason score ≥3+4) combining multiparametric MRI and clinical factors have demonstrated poor calibration (over- and underprediction) and limited use in avoiding unnecessary prostate biopsies. Purpose MRI-based risk models following local recalibration were compared with a strategy that combined Prostate Imaging Data and Reporting System (PI-RADS; version 2) and prostate-specific antigen density (PSAd) to assess the potential reduction of unnecessary prostate biopsies. Materials and Methods This retrospective study included 385 patients without prostate cancer diagnosis who underwent multipara-metric MRI (PI-RADS category ≥3) and MRI-targeted biopsy between 2015 and 2019. Recalibration and selection of the best-performing MRI model (MRI-European Randomized Study of Screening for Prostate Cancer [ERSPC], van Leeuwen, Radtke, and Mehralivand models) were undertaken in cohort C1 (n = 242; 2015-2017). The impact on biopsy decisions was compared with an alternative strategy (no biopsy for PI-RADS category 3 plus PSAd < 0.1 ng/mL per milliliter) in cohort C2 (n = 143; 2018-2019). Discrimination, calibration, and clinical utility were assessed by using the area under the receiver operating characteristic curve (AUC), calibration plots, and decision curve analysis, respectively. Results The prevalence of csPCa was 38% (93 of 242 patients) and 45% (64 of 143 patients) in cohorts C1 and C2, respectively. Decision curve analysis demonstrated the highest net benefit for the van Leeuwen and Mehralivand models in C1. Used for biopsy decisions in C2, van Leeuwen (AUC, 0.84; 95% CI: 0.77, 0.9) and Mehralivand (AUC, 0.79; 95% CI: 0.72, 0.86) enabled no net benefit at a risk threshold of 10%. Up to a risk threshold of 15%, net benefit remained inferior to the PI-RADS plus PSAd strategy, which avoided biopsy in 63 per 1000 men, without missing csPCa. Without prior recalibration in C1, three of four models (MRIERSPC, Radtke, Mehralivand) were poorly calibrated and not clinically useful in C2. Conclusion The number of unnecessary prostate biopsies in men with positive MRI may be safely reduced by using a prostate-specific antigen density-based strategy. In a risk-averse scenario, this strategy enabled better biopsy decisions compared with MRI-based risk models. ©RSNA, 2021 Online supplemental material is available for this article.

A new approach for interpretability and reliability in clinical risk prediction: Acute coronary syndrome scenario

INTRODUCTION

The risk prediction of the occurrence of a clinical event is often based on conventional statistical procedures, through the implementation of risk score models. Recently, approaches based on more complex machine learning (ML) methods have been developed. Despite the latter usually have a better predictive performance, they obtain little approval from the physicians, as they lack interpretability and, therefore, clinical confidence. One clinical issue where both types of models have received great attention is the mortality risk prediction after acute coronary syndromes (ACS).

OBJECTIVE

We intend to create a new risk assessment methodology that combines the best characteristics of both risk score and ML models. More specifically, we aim to develop a method that, besides having a good performance, offers a personalized model and outcome for each patient, presents high interpretability, and incorporates an estimation of the prediction reliability which is not usually available. By combining these features in the same approach we expect that it can boost the confidence of physicians to use such a tool in their daily activity.

METHODS

In order to achieve the mentioned goals, a three-step methodology was developed: several rules were created by dichotomizing risk factors; such rules were trained with a machine learning classifier to predict the acceptance degree of each rule (the probability that the rule is correct) for each patient; that information was combined and used to compute the risk of mortality and the reliability of such prediction. The methodology was applied to a dataset of 1111 patients admitted with any type of ACS (myocardial infarction and unstable angina) in two Portuguese hospitals, to assess the 30-days all-cause mortality risk, being validated through a Monte-Carlo cross-validation technique. The performance was compared with state-of-the-art approaches: logistic regression (LR), artificial neural network (ANN), and clinical risk score model (namely the Global Registry of Acute Coronary Events - GRACE).

RESULTS

For the scenario being analyzed, the performance of the proposed approach and the comparison models was assessed through discrimination and calibration. The ability to rank the patients was evaluated through the area under the ROC curve (AUC), and the ability to stratify the patients into low or high-risk groups was determined using the geometric mean (GM) of specificity and sensitivity, the negative predictive value (NPV) and the positive predictive value (PPV). The validation calibration curves were also inspected. The proposed approach (AUC = 81%, GM = 74%, PPV = 17%, NPV = 99%) achieved testing results identical to the standard LR model (AUC = 83%, GM = 73%, PPV = 16%, NPV=99%), but offers superior interpretability and personalization; it also significantly outperforms the GRACE risk model (AUC = 79%, GM = 47%, PPV = 13%, NPV = 98%) and the standard ANN model (AUC = 78%, GM = 70%, PPV = 13%, NPV = 98%). The calibration curve also suggests a very good generalization ability of the obtained model as it approaches the ideal curve (slope = 0.96). Finally, the reliability estimation of individual predictions presented a great correlation with the misclassifications rate.

CONCLUSION

We developed and described a new tool that showed great potential to guide the clinical staff in the risk assessment and decision-making process, and to obtain their wide acceptance due to its interpretability and reliability estimation properties. The methodology presented a good performance when applied to ACS events, but those properties may have a beneficial application in other clinical scenarios as well.

Development and validation of a predictive model algorithm to identify anaphylaxis in adults with type 2 diabetes in U.S. administrative claims data

PURPOSE

To use medical record adjudication and predictive modeling methods to develop and validate an algorithm to identify anaphylaxis among adults with type 2 diabetes (T2D) in administrative claims.

METHODS

A conventional screening algorithm that prioritized sensitivity to identify potential anaphylaxis cases was developed and consisted of diagnosis codes for anaphylaxis or relevant signs and symptoms. This algorithm was applied to adults with T2D in the HealthCore Integrated Research Database (HIRD) from 2016 to 2018. Clinical experts adjudicated anaphylaxis case status from redacted medical records. We used confirmed case status as an outcome for predictive models developed using lasso regression with 10-fold cross-validation to identify predictors and estimate the probability of confirmed anaphylaxis.

RESULTS

Clinical adjudicators reviewed medical records with sufficient information from 272 adults identified by the anaphylaxis screening algorithm, which had an estimated Positive Predictive Value (PPV) of 65% (95% confidence interval [CI]: 60%-71%). The predictive model algorithm had a c-statistic of 0.95. The model's probability threshold of 0.60 excluded 89% (84/94) of false positives identified by the screening algorithm, with a PPV of 94% (95% CI: 91%-98%). The model excluded very few true positives (15 of 178), and identified 92% (95% CI: 87%-96%) of the cases selected by the screening algorithm.

CONCLUSIONS

Predictive modeling techniques yielded an accurate algorithm with high PPV and sensitivity for identifying anaphylaxis in administrative claims. This algorithm could be considered in future safety studies using similar claims data to reduce potential outcome misclassification.

Personalising lung cancer screening: An overview of risk-stratification opportunities and challenges

Randomised clinical trials have shown the efficacy of computed tomography lung cancer screening, initiating discussions on whether and how to implement population‐based screening programs. Due to smoking behaviour being the primary risk‐factor for lung cancer and part of the criteria for determining screening eligibility, lung cancer screening is inherently risk‐based. In fact, the selection of high‐risk individuals has been shown to be essential in implementing lung cancer screening in a cost‐effective manner. Furthermore, studies have shown that further risk‐stratification may improve screening efficiency, allow personalisation of the screening interval and reduce health disparities. However, implementing risk‐based lung cancer screening programs also requires overcoming a number of challenges. There are indications that risk‐based approaches can negatively influence the trade‐off between individual benefits and harms if not applied thoughtfully. Large‐scale implementation of targeted, risk‐based screening programs has been limited thus far. Consequently, questions remain on how to efficiently identify and invite high‐risk individuals from the general population. Finally, while risk‐based approaches may increase screening program efficiency, efficiency should be balanced with the overall impact of the screening program. In this review, we will address the opportunities and challenges in applying risk‐stratification in different aspects of lung cancer screening programs, as well as the balance between screening program efficiency and impact.

Development and Validation of a Machine Learning Model Using Administrative Health Data to Predict Onset of Type 2 Diabetes

Importance

Systems-level barriers to diabetes care could be improved with population health planning tools that accurately discriminate between high- and low-risk groups to guide investments and targeted interventions.

Objective

To develop and validate a population-level machine learning model for predicting type 2 diabetes 5 years before diabetes onset using administrative health data.

Design, Setting, and Participants

This decision analytical model study used linked administrative health data from the diverse, single-payer health system in Ontario, Canada, between January 1, 2006, and December 31, 2016. A gradient boosting decision tree model was trained on data from 1 657 395 patients, validated on 243 442 patients, and tested on 236 506 patients. Costs associated with each patient were estimated using a validated costing algorithm. Data were analyzed from January 1, 2006, to December 31, 2016.

Exposures

A random sample of 2 137 343 residents of Ontario without type 2 diabetes was obtained at study start time. More than 300 features from data sets capturing demographic information, laboratory measurements, drug benefits, health care system interactions, social determinants of health, and ambulatory care and hospitalization records were compiled over 2-year patient medical histories to generate quarterly predictions.

Main Outcomes and Measures

Discrimination was assessed using the area under the receiver operating characteristic curve statistic, and calibration was assessed visually using calibration plots. Feature contribution was assessed with Shapley values. Costs were estimated in 2020 US dollars.

Results

This study trained a gradient boosting decision tree model on data from 1 657 395 patients (12 900 257 instances; 6 666 662 women [51.7%]). The developed model achieved a test area under the curve of 80.26 (range, 80.21-80.29), demonstrated good calibration, and was robust to sex, immigration status, area-level marginalization with regard to material deprivation and race/ethnicity, and low contact with the health care system. The top 5% of patients predicted as high risk by the model represented 26% of the total annual diabetes cost in Ontario.

Conclusions and Relevance

In this decision analytical model study, a machine learning model approach accurately predicted the incidence of diabetes in the population using routinely collected health administrative data. These results suggest that the model could be used to inform decision-making for population health planning and diabetes prevention.

Development and Internal Validation of Supervised Machine Learning Algorithms for Predicting Clinically Significant Functional Improvement in a Mixed Population of Primary Hip Arthroscopy

PURPOSE

To (1) develop and validate a machine learning algorithm to predict clinically significant functional improvements after hip arthroscopy for femoroacetabular impingement syndrome and to (2) develop a digital application capable of providing patients with individual risk profiles to determine their propensity to gain clinically significant improvements in function.

METHODS

A retrospective review of consecutive hip arthroscopy patients who underwent cam/pincer correction, labral preservation, and capsular closure between January 2012 and 2017 from 1 large academic and 3 community hospitals operated on by a single high-volume hip arthroscopist was performed. The primary outcome was the minimal clinically important difference (MCID) for the Hip Outcome Score (HOS)-Activities of Daily Living (ADL) at 2 years postoperatively, which was calculated using a distribution-based method. A total of 21 demographic, radiographic, and patient-reported outcome measures were considered as potential covariates. An 80:20 random split was used to create training and testing sets from the patient cohort. Five supervised machine learning algorithms were developed using 3 iterations of 10-fold cross-validation on the training set and assessed by discrimination, calibration, Brier score, and decision curve analysis on an independent testing set of patients.

RESULTS

A total of 818 patients with a median (interquartile range) age of 32.0 (22.0-42.0) and 69.2% female were included, of whom 74.3% achieved the MCID for the HOS-ADL. The best-performing algorithm was the stochastic gradient boosting model (c-statistic = 0.84, calibration intercept = 0.20, calibration slope = 0.83, and Brier score = 0.13). Of the initial 21 candidate variables, the 8 most important features for predicting the MCID for the HOS-ADL included in model training were body mass index, age, preoperative HOS-ADL score, preoperative pain level, sex, Tönnis grade, symptom duration, and drug allergies. The algorithm was subsequently transformed into a digital application using local explanations to provide customized risk assessment: https://orthoapps.shinyapps.io/HPRG_ADL/.

CONCLUSIONS

The stochastic boosting gradient model conferred excellent predictive ability for propensity to gain clinically significant improvements in function after hip arthroscopy. An open-access digital application was created, which may augment shared decision-making and allow for preoperative risk stratification. External validation of this model is warranted to confirm the performance of these algorithms, as the generalizability is currently unknown.

LEVEL OF EVIDENCE

IV, Case series.

Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review

Background: Little is known about the role of artificial intelligence (AI) as a decisive technology in the clinical management of COVID-19 patients. We aimed to systematically review and critically appraise the current evidence on AI applications for COVID-19 in intensive care and emergency settings. Methods: We systematically searched PubMed, Embase, Scopus, CINAHL, IEEE Xplore, and ACM Digital Library databases from inception to 1 October 2020, without language restrictions. We included peer-reviewed original studies that applied AI for COVID-19 patients, healthcare workers, or health systems in intensive care, emergency, or prehospital settings. We assessed predictive modelling studies and critically appraised the methodology and key findings of all other studies. Results: Of fourteen eligible studies, eleven developed prognostic or diagnostic AI predictive models, all of which were assessed to be at high risk of bias. Common pitfalls included inadequate sample sizes, poor handling of missing data, failure to account for censored participants, and weak validation of models. Conclusions: Current AI applications for COVID-19 are not ready for deployment in acute care settings, given their limited scope and poor quality. Our findings underscore the need for improvements to facilitate safe and effective clinical adoption of AI applications, for and beyond the COVID-19 pandemic.

Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review

OBJECTIVE

To assess predictive performance of universal early warning scores (EWS) in disease subgroups and clinical settings.

DESIGN

Systematic review.

DATA SOURCES

Medline, CINAHL, Embase and Cochrane database of systematic reviews from 1997 to 2019.

INCLUSION CRITERIA

Randomised trials and observational studies of internal or external validation of EWS to predict deterioration (mortality, intensive care unit (ICU) transfer and cardiac arrest) in disease subgroups or clinical settings.

RESULTS

We identified 770 studies, of which 103 were included. Study designs and methods were inconsistent, with significant risk of bias (high: n=16 and unclear: n=64 and low risk: n=28). There were only two randomised trials. There was a high degree of heterogeneity in all subgroups and in national early warning score (I2=72%-99%). Predictive accuracy (mean area under the curve; 95% CI) was highest in medical (0.74; 0.74 to 0.75) and surgical (0.77; 0.75 to 0.80) settings and respiratory diseases (0.77; 0.75 to 0.80). Few studies evaluated EWS in specific diseases, for example, cardiology (n=1) and respiratory (n=7). Mortality and ICU transfer were most frequently studied outcomes, and cardiac arrest was least examined (n=8). Integration with electronic health records was uncommon (n=9).

CONCLUSION

Methodology and quality of validation studies of EWS are insufficient to recommend their use in all diseases and all clinical settings despite good performance of EWS in some subgroups. There is urgent need for consistency in methods and study design, following consensus guidelines for predictive risk scores. Further research should consider specific diseases and settings, using electronic health record data, prior to large-scale implementation.

PROSPERO REGISTRATION NUMBER

PROSPERO CRD42019143141.

Clinical prediction models: diagnosis versus prognosis

Clinical prediction models play an increasingly important role in contemporary clinical care, by informing healthcare professionals, patients and their relatives about outcome risks, with the aim to facilitate (shared) medical decision making and improve health outcomes. Diagnostic prediction models aim to calculate an individual's risk that a disease is already present, whilst prognostic prediction models aim to calculate the risk of particular heath states occurring in the future. This article serves as a primer for diagnostic and prognostic clinical prediction models, by discussing the basic terminology, some of the inherent challenges, and the need for validation of predictive performance and the evaluation of impact of these models in clinical care.

Prediction models and nomograms for 10-year risk of end-stage renal disease in Chinese type 2 diabetes mellitus patients in primary care

AIMS

To develop and validate 10-year risk prediction models, nomograms and charts for end-stage renal disease (ESRD) in Chinese patients with type 2 diabetes mellitus (T2DM) in primary care, in order to guide individualized treatment.

MATERIALS AND METHODS

This was a 10-year population-based observational cohort study. A total of 141 516 Chinese T2DM patients without history of cardiovascular disease or ESRD who were managed in public primary care clinics in 2008 were included and followed up until December 2017. Two-thirds of these patients were randomly selected to develop sex-specific ESRD risk prediction models using Cox regressions. The validity and accuracy of the models were tested on the remaining third of patients using Harrell's C-index. We selected variables based on their clinical and statistical importance to construct the nomograms and charts.

RESULTS

The median follow-up period was 9.75 years. The cumulative incidence of ESRD was 6.0% (men: 6.1%, women: 5.9%). Age, diabetes duration, systolic blood pressure (SBP), SBP variability, diastolic blood pressure, triglycerides, glycated haemoglobin (HbA1c), HbA1c variability, urine albumin to creatinine ratio (UACR), and estimated glomerular filtration rate (eGFR) were significant predictors for both sexes. Smoking and total cholesterol to HDL cholesterol ratio were additional significant predictors for men and women, respectively. The models showed Harrell's C-statistics of 0.889/0.889 (women/men). Age, eGFR, UACR, SBP and HbA1c were selected for both sexes to develop nomograms and charts.

CONCLUSIONS

Using routinely available variables, the 10-year ESRD risk of Chinese T2DM patients in primary care can be predicted with approximately 90% accuracy. We have developed different tools to facilitate routine ESRD risk prediction in primary care, so that individualized care can be provided to prevent or delay ESRD in T2DM patients.

Externally validated model predicting gait independence after stroke showed fair performance and improved after updating

OBJECTIVE

To externally validate recent prognostic models that predict independent gait following stroke.

STUDY DESIGN AND SETTING

A systematic search identified recent models (<10 years) that predicted independent gait in adult stroke patients, using easily obtainable predictors. Predictors from the original models were assigned proxies when required, and model performance was evaluated in the validation cohort (n = 957). Models were updated to determine if performance could be improved.

RESULTS

Three prognostic models met our criteria, all with high Risk of Bias. Validation data was only available for the Australian model. This model used National Institute of Health Stroke Scale (NIHSS) and age to predict independent gait, using Motor Assessment Scale (MAS) walking item. For validation, Scandinavian Stroke Scale (SSS) was a proxy for NIHSS, and Functional Independence Measure (FIM) locomotion item was a proxy for MAS. The Area Under the Curve was 0.77 (0.74-0.80) and had good calibration in the validation dataset. Adjustment of the intercept and regression coefficients slightly improved discrimination. By adding paretic leg strength, the model further improved (AUC 0.82).

CONCLUSION

External validation of the Australian model with proxies showed fair discrimination and good calibration. Updating the model by adding paretic leg strength further improved model performance.

Prospective Validation and Comparative Analysis of Coronary Risk Stratification Strategies Among Emergency Department Patients With Chest Pain

Background

Coronary risk stratification is recommended for emergency department patients with chest pain. Many protocols are designed as “rule‐out” binary classification strategies, while others use graded‐risk stratification. The comparative performance of competing approaches at varying levels of risk tolerance has not been widely reported.

Methods and Results

This is a prospective cohort study of adult patients with chest pain presenting between January 2018 and December 2019 to 13 medical center emergency departments within an integrated healthcare delivery system. Using an electronic clinical decision support interface, we externally validated and assessed the net benefit (at varying risk thresholds) of several coronary risk scores (History, ECG, Age, Risk Factors, and Troponin [HEART] score, HEART pathway, Emergency Department Assessment of Chest Pain Score Accelerated Diagnostic Protocol), troponin‐only strategies (fourth‐generation assay), unstructured physician gestalt, and a novel risk algorithm (RISTRA‐ACS). The primary outcome was 60‐day major adverse cardiac event defined as myocardial infarction, cardiac arrest, cardiogenic shock, coronary revascularization, or all‐cause mortality. There were 13 192 patient encounters included with a 60‐day major adverse cardiac event incidence of 3.7%. RISTRA‐ACS and HEART pathway had the lowest negative likelihood ratios (0.06, 95% CI, 0.03–0.10 and 0.07, 95% CI, 0.04–0.11, respectively) and the greatest net benefit across a range of low‐risk thresholds. RISTRA‐ACS demonstrated the highest discrimination for 60‐day major adverse cardiac event (area under the receiver operating characteristic curve 0.92, 95% CI, 0.91–0.94, P<0.0001).

Conclusions

RISTRA‐ACS and HEART pathway were the optimal rule‐out approaches, while RISTRA‐ACS was the best‐performing graded‐risk approach. RISTRA‐ACS offers promise as a versatile single approach to emergency department coronary risk stratification.

Registration

URL: https://www.clinicaltrials.gov; Unique identifier: NCT03286179.

Ventilator Weaning and Discontinuation Practices for Critically Ill Patients

Importance

Although most critically ill patients receive invasive mechanical ventilation (IMV), few studies have characterized how IMV is discontinued in practice.

Objective

To describe practice variation in IMV discontinuation internationally, associations between initial discontinuation events and outcomes, and factors associated with the use of select discontinuation strategies and failed initial spontaneous breathing trials (SBTs).

Design, Setting, and Participants

Prospective, multinational, observational study of critically ill adults who received IMV for at least 24 hours from 142 intensive care units (ICUs) in 19 countries within 6 regions (27 in Canada, 23 in India, 22 in the UK, 26 in Europe, 21 in Australia/New Zealand, and 23 in the US).

Exposures

Receiving IMV.

Main Outcomes and Measures

Primary analyses characterized types of initial IMV discontinuation events (extubation, SBT, or tracheostomy) and associations with clinical outcomes (including duration of ventilation, ICU and hospital mortality, and ICU and hospital length of stay). Secondary analyses examined the associations between SBT outcome and SBT timing and clinical outcomes.

Results

Among 1868 patients (median [interquartile range] age, 61.8 [48.9-73.1] years; 1173 [62.8%] men) 424 (22.7%) underwent direct extubation, 930 (49.8%) had an initial SBT (761 [81.8%] successful), 150 (8.0%) underwent direct tracheostomy, and 364 (19.5%) died before a weaning attempt. Across regions, there was variation in the use of written directives to guide care, daily screening, SBT techniques, ventilator modes, and the roles played by clinicians involved in weaning. Compared with initial direct extubation, patients who had an initial SBT had higher ICU mortality (20 [4.7%] vs 96 [10.3%]; absolute difference, 5.6% [95% CI, 2.6%-8.6%]), longer duration of ventilation (median of 2.9 vs 4.1 days; absolute difference, 1.2 days [95% CI, 0.7-1.6]), and longer ICU stay (median of 6.7 vs 8.1 days; absolute difference, 1.4 days [95% CI, 0.8-2.4]). Patients whose initial SBT failed (vs passed) had higher ICU mortality (29 [17.2%] vs 67 [8.8%]; absolute difference, 8.4% [95% CI, 2.0%-14.7%]), longer duration of ventilation (median of 6.1 vs 3.5 days; absolute difference, 2.6 days [95% CI, 1.6-3.6]), and longer ICU stay (median of 10.6 vs 7.7 days; absolute difference, 2.8 days [95% CI, 1.1-5.2]). Compared with patients who underwent early initial SBTs, patients who underwent late initial SBTs (>2.3 days after intubation) had longer duration of ventilation (median of 2.1 vs 6.1 days; absolute difference, 4.0 days [95% CI, 3.7-4.5]), longer ICU stay (median of 5.9 vs 10.8 days; absolute difference, 4.9 days [95% CI, 4.0-6.3]), and longer hospital stay (median of 14.3 vs 22.8 days; absolute difference, 8.5 days [95% CI, 6.0-11.0]).

Conclusions and Relevance

In this observational study of invasive mechanical ventilation discontinuation in 142 ICUs in Canada, India, the UK, Europe, Australia/New Zealand, and the US from 2013 to 2016, weaning practices varied internationally.

Trial Registration

ClinicalTrials.gov Identifier: NCT03955874.

Validation of the REVEAL Prognostic Models in Systemic Lupus Erythematosus-Associated Pulmonary Arterial Hypertension

No previous studies have investigated the predictive performance of the Registry to Evaluate Early and Long-term Pulmonary Arterial Hypertension Disease Management (REVEAL) prognostic equation and simplified risk score calculator in patients with systemic lupus erythematosus-associated pulmonary arterial hypertension (SLE-PAH). We aimed to validate these prediction tools in an external cohort of patients with SLE-PAH. In this study, the validation cohort consisted of patients with SLE-PAH registered in a prospective, multicenter, nationwide database between November 2006 and May2016. The follow-up of patients was censored at 1 year. Discrimination, calibration, model fit, and risk stratification of the REVEAL prognostic equation and simplified risk score calculator were validated. As a result, a total of 306 patients with SLE-PAH were included. The 1-year overall survival rate was 91.5%. The C-index of the prognostic equation was 0.736, demonstrating reasonably good discrimination, and it was greater than that for the simplified risk score calculator (0.710). The overall calibration slope was 0.83, and the Brier score was 0.079. The risk of renal insufficiency and World Health Organization Functional Class III (WHO FC III) were underestimated, and the risk assigned to a heart rate >92 bpm in the REVEAL prognostic models was not observed in our validation cohort. Both model discrimination and calibration were poor in the very high-risk group. In conclusion, the REVEAL models exhibit good discriminatory ability when predicting 1-year overall survival in patients with SLE-PAH. Findings from both models should be interpreted with caution in cases of renal insufficiency, WHO FC III, and heart rate >92 bpm.

Validation of multivariable lung cancer risk prediction models for the personalized assignment of optimal screening frequency: a retrospective analysis of data from the German Lung Cancer Screening Intervention Trial (LUSI)

BACKGROUND

Current guidelines for lung cancer screening via low-dose computed tomography recommend annual screening for all candidates meeting basic eligibility criteria. However, lung cancer risk of eligible screening participants can vary widely, and further risk stratification could be used to individually optimize screening intervals in view of expected benefits, possible harms and financial costs. To this effect, models have been developed in the US National Lung Screening Trial based on self-reported lung cancer risk factors and imaging data. We evaluated these models using data from an independent screening trial in Germany.

METHODS

We examined the Polynomial model by Schreuder et al., the Lung Cancer Risk Assessment Tool extended by CT characteristics (LCRAT + CT) by Robbins et al., and a criterion of presence vs. absence of pulmonary nodules ≥4 mm (Patz et al.), applied to sub-sets of screening participants according to eligibility criteria. Discrimination was evaluated via the receiver operating characteristic curve. Delayed diagnoses and false positive results were calculated at various thresholds of predicted risk. Model calibration was assessed by comparing mean predicted risk versus observed incidence.

RESULTS

One thousand five hundred and six participants were eligible for the validation of the LCRAT + CT model, and 1,889 for the validation of the Polynomial model and Patz criterion, yielding areas under the receiver operating characteristic curve of 0.73 (95% CI: 0.63, 0.82), 0.75 (0.67, 0.83), and 0.56 (0.53, 0.72) respectively. Skipping 50% annual screenings (participants within the 5 lowest risk deciles by LCRAT + CT in any round or by the Polynomial model; baseline screening round), would have avoided 75% (21.9%, 98.7%) and 40% (21.8%, 61.1%) false positive screen tests and delayed 10% (1.8%, 33.1%) or no (0%, 32.1%) diagnoses, respectively. Using the Patz criterion, referring 63.2% (61.0% to 65.4%) of participants to biennial screening would have avoided 4% (0.2% to 22.3%) of false positive screen tests but delayed 55% (24.6% to 81.9%) diagnoses.

CONCLUSIONS

In this German trial, the LCRAT + CT and Polynomial models showed useful discrimination of screening participants for one-year lung cancer risk following CT examination. Our results illustrate the remaining heterogeneity in risk within screening-eligible subjects and the trade-off between a low-frequency screening approach and delayed detection.