Calibration: the Achilles heel of predictive analytics

Calibrating variant-scoring methods for clinical decision making

Summary

Identifying pathogenic variants and annotating them is a major challenge in human genetics, especially for the non-coding ones. Several tools have been developed and used to predict the functional effect of genetic variants. However, the calibration assessment of the predictions has received little attention. Calibration refers to the idea that if a model predicts a group of variants to be pathogenic with a probability P, it is expected that the same fraction P of true positive is found in the observed set. For instance, a well-calibrated classifier should label the variants such that among the ones to which it gave a probability value close to 0.7, approximately 70% actually belong to the pathogenic class. Poorly calibrated algorithms can be misleading and potentially harmful for clinical decision making.

Avaliability and implementation

The dataset used for testing the methods is available through the DOI:10.5281/zenodo.4448197.

Supplementary information

Supplementary data are available at Bioinformatics online.

Introduction to Artificial Intelligence and Machine Learning for Pathology

CONTEXT.—

Recent developments in machine learning have stimulated intense interest in software that may augment or replace human experts. Machine learning may impact pathology practice by offering new capabilities in analysis, interpretation, and outcomes prediction using images and other data. The principles of operation and management of machine learning systems are unfamiliar to pathologists, who anticipate a need for additional education to be effective as expert users and managers of the new tools.

OBJECTIVE.—

To provide a background on machine learning for practicing pathologists, including an overview of algorithms, model development, and performance evaluation; to examine the current status of machine learning in pathology and consider possible roles and requirements for pathologists in local deployment and management of machine learning systems; and to highlight existing challenges and gaps in deployment methodology and regulation.

DATA SOURCES.—

Sources include the biomedical and engineering literature, white papers from professional organizations, government reports, electronic resources, and authors' experience in machine learning. References were chosen when possible for accessibility to practicing pathologists without specialized training in mathematics, statistics, or software development.

CONCLUSIONS.—

Machine learning offers an array of techniques that in recent published results show substantial promise. Data suggest that human experts working with machine learning tools outperform humans or machines separately, but the optimal form for this combination in pathology has not been established. Significant questions related to the generalizability of machine learning systems, local site verification, and performance monitoring remain to be resolved before a consensus on best practices and a regulatory environment can be established.

Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study

BACKGROUND

The National Early Warning Score (NEWS2) is currently recommended in the UK for the risk stratification of COVID-19 patients, but little is known about its ability to detect severe cases. We aimed to evaluate NEWS2 for the prediction of severe COVID-19 outcome and identify and validate a set of blood and physiological parameters routinely collected at hospital admission to improve upon the use of NEWS2 alone for medium-term risk stratification.

METHODS

Training cohorts comprised 1276 patients admitted to King's College Hospital National Health Service (NHS) Foundation Trust with COVID-19 disease from 1 March to 30 April 2020. External validation cohorts included 6237 patients from five UK NHS Trusts (Guy's and St Thomas' Hospitals, University Hospitals Southampton, University Hospitals Bristol and Weston NHS Foundation Trust, University College London Hospitals, University Hospitals Birmingham), one hospital in Norway (Oslo University Hospital), and two hospitals in Wuhan, China (Wuhan Sixth Hospital and Taikang Tongji Hospital). The outcome was severe COVID-19 disease (transfer to intensive care unit (ICU) or death) at 14 days after hospital admission. Age, physiological measures, blood biomarkers, sex, ethnicity, and comorbidities (hypertension, diabetes, cardiovascular, respiratory and kidney diseases) measured at hospital admission were considered in the models.

RESULTS

A baseline model of 'NEWS2 + age' had poor-to-moderate discrimination for severe COVID-19 infection at 14 days (area under receiver operating characteristic curve (AUC) in training cohort = 0.700, 95% confidence interval (CI) 0.680, 0.722; Brier score = 0.192, 95% CI 0.186, 0.197). A supplemented model adding eight routinely collected blood and physiological parameters (supplemental oxygen flow rate, urea, age, oxygen saturation, C-reactive protein, estimated glomerular filtration rate, neutrophil count, neutrophil/lymphocyte ratio) improved discrimination (AUC = 0.735; 95% CI 0.715, 0.757), and these improvements were replicated across seven UK and non-UK sites. However, there was evidence of miscalibration with the model tending to underestimate risks in most sites.

CONCLUSIONS

NEWS2 score had poor-to-moderate discrimination for medium-term COVID-19 outcome which raises questions about its use as a screening tool at hospital admission. Risk stratification was improved by including readily available blood and physiological parameters measured at hospital admission, but there was evidence of miscalibration in external sites. This highlights the need for a better understanding of the use of early warning scores for COVID.

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.

Machine Learning Within Studies of Early-Life Environmental Exposures and Child Health: Review of the Current Literature and Discussion of Next Steps

PURPOSE OF REVIEW

The goal of this article is to review the use of machine learning (ML) within studies of environmental exposures and children's health, identify common themes across studies, and provide recommendations to advance their use in research and practice.

RECENT FINDINGS

We identified 42 articles reporting upon the use of ML within studies of environmental exposures and children's health between 2017 and 2019. The common themes among the articles were analysis of mixture data, exposure prediction, disease prediction and forecasting, analysis of complex data, and causal inference. With the increasing complexity of environmental health data, we anticipate greater use of ML to address the challenges that cannot be handled by traditional analytics. In order for these methods to beneficially impact public health, the ML techniques we use need to be appropriate for our study questions, rigorously evaluated and reported in a way that can be critically assessed by the scientific community.

Right population, right resources, right algorithm: Using machine learning efficiently and effectively in surgical systems where data are a limited resource

There is a growing interest in using machine learning algorithms to support surgical care, diagnostics, and public health surveillance in low- and middle-income countries. From our own experience and the literature, we share several lessons for developing such models in settings where the data necessary for algorithm training and implementation is a limited resource. First, the training cohort should be as similar as possible to the population of interest, and recalibration can be used to improve risk estimates when a model is transported to a new context. Second, algorithms should incorporate existing data sources or data that is easily obtainable by frontline health workers or assistants in order to optimize available resources and facilitate integration into clinical practice. Third, the Super Learner ensemble machine learning algorithm can be used to define the optimal model for a given prediction problem while minimizing bias in the algorithm selection process. By considering the right population, right resources, and right algorithm, researchers can train prediction models that are both context-appropriate and resource-conscious. There remain gaps in data availability, affordable computing capacity, and implementation studies that hinder clinical algorithm development and use in low-resource settings, although these barriers are decreasing over time. We advocate for researchers to create open-source code, apps, and training materials to allow new machine learning models to be adapted to different populations and contexts in order to support surgical providers and health care systems in low- and middle-income countries worldwide.

Use of the Thrombolysis in Myocardial Infarction Risk Index for Elderly Patients With ST-Segment Elevation Myocardial Infarction

Background: Thrombolysis in Myocardial Infarction (TIMI) Risk Index (TRI) is a simple risk assessment tool for patients with ST-segment elevation myocardial infarction (STEMI). However, its applicability to elderly patients with STEMI undergoing percutaneous coronary intervention (PCI) is uncertain. Methods: This was a retrospective analysis of elderly (≥60 years) patients who underwent PCI for STEMI from January 2010 to April 2016. TRI was calculated on admission using the following formula: heart rate × (age/10)²/systolic blood pressure. Discrimination and calibration of TRI for in-hospital events and 1 year mortality were analyzed. Results: Totally 1,054 patients were divided into three groups according to the tertiles of the TRI: <27 (n = 348), 27-36 (n = 360) and >36 (n = 346). The incidence of acute kidney injury (AKI; 7.8 vs. 8.6 vs. 24.0%, p < 0.001), AHF (3.5 vs. 6.6 vs. 16.2%, p < 0.001), in-hospital death (0.6 vs. 3.3 vs. 11.6%, p < 0.001) and MACEs (5.2 vs. 5.8 vs. 15.9%, p < 0.001) was significantly higher in the third tertile. TRI showed good discrimination for in-hospital death [area under the curve (AUC) = 0.804, p < 0.001; Hosmer-Lemeshow p = 0.302], which was superior to its prediction for AKI (AUC = 0.678, p < 0.001; Hosmer-Lemeshow p = 0.121), and in-hospital MACEs (AUC = 0.669, p < 0.001; Hosmer-Lemeshow p = 0.077). Receiver-operation characteristics curve showed that TRI > 42.0 had a sensitivity of 64.8% and specificity of 82.2% for predicting in-hospital death. Kaplan-Meier analysis showed that patients with TRI > 42.0 had higher 1 year mortality (Log-rank = 79.2, p < 0.001). Conclusion: TRI is suitable for risk stratification in elderly patients with STEMI undergoing PCI, and is thus of continuing value for an aging population.

Comparison of Multivariable Logistic Regression and Machine Learning Models for Predicting Bronchopulmonary Dysplasia or Death in Very Preterm Infants

Bronchopulmonary dysplasia (BPD) is the most prevalent and clinically significant complication of prematurity. Accurate identification of at-risk infants would enable ongoing intervention to improve outcomes. Although postnatal exposures are known to affect an infant's likelihood of developing BPD, most existing BPD prediction models do not allow risk to be evaluated at different time points, and/or are not suitable for use in ethno-diverse populations. A comprehensive approach to developing clinical prediction models avoids assumptions as to which method will yield the optimal results by testing multiple algorithms/models. We compared the performance of machine learning and logistic regression models in predicting BPD/death. Our main cohort included infants <33 weeks' gestational age (GA) admitted to a Canadian Neonatal Network site from 2016 to 2018 (n = 9,006) with all analyses repeated for the <29 weeks' GA subcohort (n = 4,246). Models were developed to predict, on days 1, 7, and 14 of admission to neonatal intensive care, the composite outcome of BPD/death prior to discharge. Ten-fold cross-validation and a 20% hold-out sample were used to measure area under the curve (AUC). Calibration intercepts and slopes were estimated by regressing the outcome on the log-odds of the predicted probabilities. The model AUCs ranged from 0.811 to 0.886. Model discrimination was lower in the <29 weeks' GA subcohort (AUCs 0.699-0.790). Several machine learning models had a suboptimal calibration intercept and/or slope (k-nearest neighbor, random forest, artificial neural network, stacking neural network ensemble). The top-performing algorithms will be used to develop multinomial models and an online risk estimator for predicting BPD severity and death that does not require information on ethnicity.

Application of clinical prediction modeling in pediatric neurosurgery: a case study

There has been an increasing interest in articles reporting on clinical prediction models in pediatric neurosurgery. Clinical prediction models are mathematical equations that combine patient-related risk factors for the estimation of an individual's risk of an outcome. If used sensibly, these evidence-based tools may help pediatric neurosurgeons in medical decision-making processes. Furthermore, they may help to communicate anticipated future events of diseases to children and their parents and facilitate shared decision-making accordingly. A basic understanding of this methodology is incumbent when developing or applying a prediction model. This paper addresses this methodology tailored to pediatric neurosurgery. For illustration, we use original pediatric data from our institution to illustrate this methodology with a case study. The developed model is however not externally validated, and clinical impact has not been assessed; therefore, the model cannot be recommended for clinical use in its current form.

Predicting the Individual Treatment Effect of Neurosurgery for Patients with Traumatic Brain Injury in the Low-Resource Setting: A Machine Learning Approach in Uganda

Traumatic brain injury (TBI) disproportionately affects low- and middle-income countries (LMICs). In these low-resource settings, effective triage of patients with TBI-including the decision of whether or not to perform neurosurgery-is critical in optimizing patient outcomes and healthcare resource utilization. Machine learning may allow for effective predictions of patient outcomes both with and without surgery. Data from patients with TBI was collected prospectively at Mulago National Referral Hospital in Kampala, Uganda, from 2016 to 2019. One linear and six non-linear machine learning models were designed to predict good versus poor outcome near hospital discharge and internally validated using nested five-fold cross-validation. The 13 predictors included clinical variables easily acquired on admission and whether or not the patient received surgery. Using an elastic-net regularized logistic regression model (GLMnet), with predictions calibrated using Platt scaling, the probability of poor outcome was calculated for each patient both with and without surgery (with the difference quantifying the "individual treatment effect," ITE). Relative ITE represents the percent reduction in chance of poor outcome, equaling this ITE divided by the probability of poor outcome with no surgery. Ultimately, 1766 patients were included. Areas under the receiver operating characteristic curve (AUROCs) ranged from 83.1% (single C5.0 ruleset) to 88.5% (random forest), with the GLMnet at 87.5%. The two variables promoting good outcomes in the GLMnet model were high Glasgow Coma Scale score and receiving surgery. For the subgroup not receiving surgery, the median relative ITE was 42.9% (interquartile range [IQR], 32.7% to 53.5%); similarly, in those receiving surgery, it was 43.2% (IQR, 32.9% to 54.3%). We provide the first machine learning-based model to predict TBI outcomes with and without surgery in LMICs, thus enabling more effective surgical decision making in the resource-limited setting. Predicted ITE similarity between surgical and non-surgical groups suggests that, currently, patients are not being chosen optimally for neurosurgical intervention. Our clinical decision aid has the potential to improve outcomes.

Validation of APACHE II, APACHE III and SAPS II scores in in-hospital and one year mortality prediction in a mixed intensive care unit in Poland: a cohort study

BACKGROUND

There are several scores used for in-hospital mortality prediction in critical illness. Their application in a local scenario requires validation to ensure appropriate diagnostic accuracy. Moreover, their use in assessing post-discharge mortality in intensive care unit (ICU) survivors has not been extensively studied. We aimed to validate APACHE II, APACHE III and SAPS II scores in short- and long-term mortality prediction in a mixed adult ICU in Poland. APACHE II, APACHE III and SAPS II scores, with corresponding predicted mortality ratios, were calculated for 303 consecutive patients admitted to a 10-bed ICU in 2016. Short-term (in-hospital) and long-term (12-month post-discharge) mortality was assessed.

RESULTS

Median APACHE II, APACHE III and SAPS II scores were 19 (IQR 12-24), 67 (36.5-88) and 44 (27-56) points, with corresponding in-hospital mortality ratios of 25.8% (IQR 12.1-46.0), 18.5% (IQR 3.8-41.8) and 34.8% (IQR 7.9-59.8). Observed in-hospital mortality was 35.6%. Moreover, 12-month post-discharge mortality reached 17.4%. All the scores predicted in-hospital mortality (p < 0.05): APACHE II (AUC = 0.78; 95%CI 0.73-0.83), APACHE III (AUC = 0.79; 95%CI 0.74-0.84) and SAPS II (AUC = 0.79; 95%CI 0.74-0.84); as well as mortality after hospital discharge (p < 0.05): APACHE II (AUC = 0.71; 95%CI 0.64-0.78), APACHE III (AUC = 0.72; 95%CI 0.65-0.78) and SAPS II (AUC = 0.69; 95%CI 0.62-0.76), with no statistically significant difference between the scores (p > 0.05). The calibration of the scores was good.

CONCLUSIONS

All the scores are acceptable predictors of in-hospital mortality. In the case of post-discharge mortality, their diagnostic accuracy is lower and of borderline clinical relevance. Further studies are needed to create scores estimating the long-term prognosis of subjects successfully discharged from the ICU.

Detection of calibration drift in clinical prediction models to inform model updating

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

Predicting the risk of future depression among school-attending adolescents in Nigeria using a model developed in Brazil

Depression commonly emerges in adolescence and is a major public health issue in low- and middle-income countries where 90% of the world's adolescents live. Thus efforts to prevent depression onset are crucial in countries like Nigeria, where two-thirds of the population are aged under 24. Therefore, we tested the ability of a prediction model developed in Brazil to predict future depression in a Nigerian adolescent sample. Data were obtained from school students aged 14-16 years in Lagos, who were assessed in 2016 and 2019 for depression using a self-completed version of the Mini International Neuropsychiatric Interview for Children and Adolescents. Only the 1,928 students free of depression at baseline were included. Penalized logistic regression was used to predict individualized risk of developing depression at follow-up for each adolescent based on the 7 matching baseline sociodemographic predictors from the Brazilian model. Discrimination between adolescents who did and did not develop depression was better than chance (area under the curve = 0.62 (bootstrap-corrected 95% CI: 0.58-0.66). However, the model was not well-calibrated even after adjustment of the intercept, indicating poorer overall performance compared to the original Brazilian cohort. Updating the model with context-specific factors may improve prediction of depression in this setting.

Development and validation of a HOXB8 gene-based prognostic model and nomogram for colorectal cancer patients

BACKGROUND

At present, colorectal cancer (CRC) is still associated with a high rate of recurrence and distant metastasis with a poor prognosis. HOXB8 gene is related to the tumorigenesis and development in CRC.

AIM

To explore the prognostic value of HOXB8 gene in CRC patients, and provide a novel insight into the monitoring of disease progression and cancer recurrence in patients with high-risk CRC.

METHODS

The mRNA sequencing data of HOXB8 in CRC patients was downloaded from The Cancer Genome Atlas database. Then, we analyzed the relationship between HOXB8 expression and clinicopathologic features in CRC, and performed survival analysis based on HOXB8 expression. Univariate and multivariate Cox regression analyese were performed for identifying prognostic factors for CRC, and then a nomogram was established and evaluated by concordance index, calibration curve, and decision curve analysis (DCA).

RESULTS

HOXB8 mRNA expression was significantly correlated with CRC tumor tissue (P < 0.001), right-side CRC (P < 0.001), T stage (P = 0.024), and M stage (P = 0.0074). Survival analysis showed that overexpression of HOXB8 was associated with a poor progression-free survival (PFS) in CRC patients (P = 0.0019). Univariate and multivariate COX analyses suggested that the expression level of HOXB8 [HR: 1.539 (1.066-2.221), P = 0.021] and TNM stage were independent prognostic factors for PFS of CRC patients. A nomogram was established to predict 3- and 5-year PFS of CRC patients based on four factors including HOXB8 expression and TNM stage. The concordance index was 0.735, suggesting good discrimination; the calibration curve and DCA showed that the nomogram had good predictive power and clinical practicability.

CONCLUSION

The expression of HOXB8 is significantly related to the prognosis of CRC patients, and it has appreciated predictive ability for disease progression and cancer recurrence in CRC patients. HOXB8 could act as a potential biomarker to identify high-risk CRC patients and become a novel therapeutic target and prognostic indicator for CRC.

Predicting Subjective Recovery from Lower Limb Surgery Using Consumer Wearables

Introduction

A major challenge in the monitoring of rehabilitation is the lack of long-term individual baseline data which would enable accurate and objective assessment of functional recovery. Consumer-grade wearable devices enable the tracking of individual everyday functioning prior to illness or other medical events which necessitate the monitoring of recovery trajectories.

Methods

For 1,324 individuals who underwent surgery on a lower limb, we collected their Fitbit device data of steps, heart rate, and sleep from 26 weeks before to 26 weeks after the self-reported surgery date. We identified subgroups of individuals who self-reported surgeries for bone fracture repair (n = 355), tendon or ligament repair/reconstruction (n = 773), and knee or hip joint replacement (n = 196). We used linear mixed models to estimate the average effect of time relative to surgery on daily activity measurements while adjusting for gender, age, and the participant-specific activity baseline. We used a sub-cohort of 127 individuals with dense wearable data who underwent tendon/ligament surgery and employed XGBoost to predict the self-reported recovery time.

Results

The 1,324 study individuals were all US residents, predominantly female (84%), white or Caucasian (85%), and young to middle-aged (mean age 36.2 years). We showed that 12 weeks pre- and 26 weeks post-surgery trajectories of daily behavioral measurements (steps sum, heart rate, sleep efficiency score) can capture activity changes relative to an individual's baseline. We demonstrated that the trajectories differ across surgery types, recapitulate the documented effect of age on functional recovery, and highlight differences in relative activity change across self-reported recovery time groups. Finally, using a sub-cohort of 127 individuals, we showed that long-term recovery can be accurately predicted, on an individual level, only 1 month after surgery (AUROC 0.734, AUPRC 0.8). Furthermore, we showed that predictions are most accurate when long-term, individual baseline data are available.

Discussion

Leveraging long-term, passively collected wearable data promises to enable relative assessment of individual recovery and is a first step towards data-driven intervention for individuals.

SSP: Early prediction of sepsis using fully connected LSTM-CNN model

BACKGROUND

Sepsis is a life-threatening condition that occurs due to the body's reaction to infections, and it is a leading cause of morbidity and mortality in hospitals. Early prediction of sepsis onset facilitates early interventions that promote the survival of suspected patients. However, reliable and intelligent systems for predicting sepsis are scarce.

METHODS

This paper presents a novel technique called Smart Sepsis Predictor (SSP) to predict sepsis onset in patients admitted to an intensive care unit (ICU). SSP is a deep neural network architecture that encompasses long short-term memory (LSTM), convolutional, and fully connected layers to achieve early prediction of sepsis. SSP can work in two modes; Mode 1 uses demographic data and vital signs, and Mode 2 uses laboratory test results in addition to demographic data and vital signs. To evaluate SSP, we have used the 2019 PhysioNet/CinC Challenge dataset, which includes the records of 40,366 patients admitted to the ICU.

RESULTS

To compare SSP with existing state-of-the-art methods, we have measured the accuracy of the SSP in 4-, 8-, and 12-h prediction windows using publicly available data. Our results show that the SSP performance in Mode 1 and Mode 2 is much higher than existing methods, achieving an area under the receiver operating characteristic curve (AUROC) of 0.89 and 0.92, 0.88 and 0.87, and 0.86 and 0.84 for 4 h, 8 h, and 12 h before sepsis onset, respectively.

CONCLUSIONS

Using ICU data, sepsis onset can be predicted up to 12 h in advance. Our findings offer an early solution for mitigating the risk of sepsis onset.

Smoothed time-dependent receiver operating characteristic curve for right censored survival data

The prediction reliability is of primary concern in many clinical studies when the objective is to develop new predictive models or improve existing risk scores. In fact, before using a model in any clinical decision making, it is very important to check its ability to discriminate between subjects who are at risk of, for example, developing certain disease in a near future from those who will not. To that end, the time-dependent receiver operating characteristic (ROC) curve is the most commonly used method in practice. Several approaches have been proposed in the literature to estimate the ROC nonparametrically in the context of survival data. But, except one recent approach, all the existing methods provide a nonsmooth ROC estimator whereas, by definition, the ROC curve is smooth. In this article we propose and study a new nonparametric smooth ROC estimator based on a weighted kernel smoother. More precisely, our approach relies on a well-known kernel method used to estimate cumulative distribution functions of random variables with bounded supports. We derived some asymptotic properties for the proposed estimator. As bandwidth is the main parameter to be set, we present and study different methods to appropriately select one. A simulation study is conducted, under different scenarios, to prove the consistency of the proposed method and to compare its finite sample performance with a competitor. The results show that the proposed method performs better and appear to be quite robust to bandwidth choice. As for inference purposes, our results also reveal the good performances of a proposed nonparametric bootstrap procedure. Furthermore, we illustrate the method using a real data example.

Identification of undiagnosed atrial fibrillation patients using a machine learning risk prediction algorithm and diagnostic testing (PULsE-AI): Study protocol for a randomised controlled trial

Atrial fibrillation (AF) is associated with an increased risk of stroke, enhanced stroke severity, and other comorbidities. However, AF is often asymptomatic, and frequently remains undiagnosed until complications occur. Current screening approaches for AF lack either cost-effectiveness or diagnostic sensitivity; thus, there is interest in tools that could be used for population screening. An AF risk prediction algorithm, developed using machine learning from a UK dataset of 2,994,837 patients, was found to be more effective than existing models at identifying patients at risk of AF. Therefore, the aim of the trial is to assess the effectiveness of this risk prediction algorithm combined with diagnostic testing for the identification of AF in a real-world primary care setting. Eligible participants (aged ≥30 years and without an existing AF diagnosis) registered at participating UK general practices will be randomised into intervention and control arms. Intervention arm participants identified at highest risk of developing AF (algorithm risk score ≥ 7.4%) will be invited for a 12‑lead electrocardiogram (ECG) followed by two-weeks of home-based ECG monitoring with a KardiaMobile device. Control arm participants will be used for comparison and will be managed routinely. The primary outcome is the number of AF diagnoses in the intervention arm compared with the control arm during the research window. If the trial is successful, there is potential for the risk prediction algorithm to be implemented throughout primary care for narrowing the population considered at highest risk for AF who could benefit from more intensive screening for AF. Trial Registration: NCT04045639.

Best practices for authors of healthcare-related artificial intelligence manuscripts

Since its inception in 2017, npj Digital Medicine has attracted a disproportionate number of manuscripts reporting on uses of artificial intelligence. This field has matured rapidly in the past several years. There was initial fascination with the algorithms themselves (machine learning, deep learning, convoluted neural networks) and the use of these algorithms to make predictions that often surpassed prevailing benchmarks. As the discipline has matured, individuals have called attention to aberrancies in the output of these algorithms. In particular, criticisms have been widely circulated that algorithmically developed models may have limited generalizability due to overfitting to the training data and may systematically perpetuate various forms of biases inherent in the training data, including race, gender, age, and health state or fitness level (Challen et al. BMJ Qual. Saf. 28:231–237, 2019; O’neil. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Broadway Book, 2016). Given our interest in publishing the highest quality papers and the growing volume of submissions using AI algorithms, we offer a list of criteria that authors should consider before submitting papers to npj Digital Medicine.

A prognostic model predicted deterioration in health-related quality of life in older patients with multimorbidity and polypharmacy

OBJECTIVES

To develop and validate a prognostic model to predict deterioration in health-related quality of life (dHRQoL) in older general practice patients with at least one chronic condition and one chronic prescription.

STUDY DESIGN AND SETTING

We used individual participant data from five cluster-randomized trials conducted in the Netherlands and Germany to predict dHRQoL, defined as a decrease in EQ-5D-3 L index score of ≥5% after 6-month follow-up in logistic regression models with stratified intercepts to account for between-study heterogeneity. The model was validated internally and by using internal-external cross-validation (IECV).

RESULTS

In 3,582 patients with complete data, of whom 1,046 (29.2%) showed deterioration in HRQoL, and 12/87 variables were selected that were related to single (chronic) conditions, inappropriate medication, medication underuse, functional status, well-being, and HRQoL. Bootstrap internal validation showed a C-statistic of 0.71 (0.69 to 0.72) and a calibration slope of 0.88 (0.78 to 0.98). In the IECV loop, the model provided a pooled C-statistic of 0.68 (0.65 to 0.70) and calibration-in-the-large of 0 (-0.13 to 0.13). HRQoL/functionality had the strongest prognostic value.

CONCLUSION

The model performed well in terms of discrimination, calibration, and generalizability and might help clinicians identify older patients at high risk of dHRQoL.

REGISTRATION

PROSPERO ID: CRD42018088129.

Predicting the chance on live birth per cycle at each step of the IVF journey: external validation and update of the van Loendersloot multivariable prognostic model

OBJECTIVE

To study the performance of the 'van Loendersloot' prognostic model for our clinic's in vitro fertilisation (IVF) in its original version, the refitted version and in an adapted version replacing previous by current cycle IVF laboratory variables.

METHODS

This retrospective cohort study in our academic tertiary fertility clinic analysed 1281 IVF cycles of 591 couples, who completed at least one 2nd-6th IVF cycle with own fresh gametes after a previous IVF cycle with the same partner in our clinic between 2010 and 2018. The outcome of interest was the chance on a live birth after one complete IVF cycle (including all fresh and frozen embryo transfers from the same episode of ovarian stimulation). Model performance was expressed in terms of discrimination (c-statistics) and calibration (calibration model, comparison of prognosis to observed ratios of five disjoint groups formed by the quintiles of the IVF prognoses and a calibration plot).

RESULTS

A total of 344 live births were obtained (26.9%). External validation of the original van Loendersloot model showed a poor c-statistic of 0.64 (95% CI: 0.61 to 0.68) and an underestimation of IVF success. The refitted and the adapted models showed c-statistics of respectively 0.68 (95% CI: 0.65 to 0.71) and 0.74 (95% CI: 0.70 to 0.77). Similar c-statistics were found with cross-validation. Both models showed a good calibration model; refitted model: intercept=0.00 (95% CI: -0.23 to 0.23) and slope=1.00 (95% CI: 0.79 to 1.21); adapted model: intercept=0.00 (95% CI: -0.18 to 0.18) and slope=1.00 (95% CI: 0.83 to 1.17). Prognoses and observed success rates of the disjoint groups matched well for the refitted model and even better for the adapted model.

CONCLUSION

External validation of the original van Loendersloot model indicated that model updating was recommended. The good performance of the refitted and adapted models allows informing couples about their IVF prognosis prior to an IVF cycle and at the time of embryo transfer. Whether this has an impact on couple's expected success rates, distress and IVF discontinuation can now be studied.

A study protocol for the development of a multivariable model predicting 6- and 12-month mortality for people with dementia living in residential aged care facilities (RACFs) in Australia

BACKGROUND

For residential aged care facility (RACF) residents with dementia, lack of prognostic guidance presents a significant challenge for end of life care planning. In an attempt to address this issue, models have been developed to assess mortality risk for people with advanced dementia, predominantly using long-term care minimum data set (MDS) information from the USA. A limitation of these models is that the information contained within the MDS used for model development was not collected for the purpose of identifying prognostic factors. The models developed using MDS data have had relatively modest ability to discriminate mortality risk and are difficult to apply outside the MDS setting. This study will aim to develop a model to estimate 6- and 12-month mortality risk for people with dementia from prognostic indicators recorded during usual clinical care provided in RACFs in Australia.

METHODS

A secondary analysis will be conducted for a cohort of people with dementia from RACFs participating in a cluster-randomized trial of a palliative care education intervention (IMPETUS-D). Ten prognostic indicator variables were identified based on a literature review of clinical features associated with increased mortality for people with dementia living in RACFs. Variables will be extracted from RACF files at baseline and mortality measured at 6 and 12 months after baseline data collection. A multivariable logistic regression model will be developed for 6- and 12-month mortality outcome measures using backwards elimination with a fractional polynomial approach for continuous variables. Internal validation will be undertaken using bootstrapping methods. Discrimination of the model for 6- and 12-month mortality will be presented as receiver operating curves with c statistics. Calibration curves will be presented comparing observed and predicted event rates for each decile of risk as well as flexible calibration curves derived using loess-based functions.

DISCUSSION

The model developed in this study aims to improve clinical assessment of mortality risk for people with dementia living in RACFs in Australia. Further external validation in different populations will be required before the model could be developed into a tool to assist with clinical decision-making in the future.

Clinical features of COVID-19 mortality: development and validation of a clinical prediction model

Background

The COVID-19 pandemic has affected millions of individuals and caused hundreds of thousands of deaths worldwide. Predicting mortality among patients with COVID-19 who present with a spectrum of complications is very difficult, hindering the prognostication and management of the disease. We aimed to develop an accurate prediction model of COVID-19 mortality using unbiased computational methods, and identify the clinical features most predictive of this outcome.

Methods

In this prediction model development and validation study, we applied machine learning techniques to clinical data from a large cohort of patients with COVID-19 treated at the Mount Sinai Health System in New York City, NY, USA, to predict mortality. We analysed patient-level data captured in the Mount Sinai Data Warehouse database for individuals with a confirmed diagnosis of COVID-19 who had a health system encounter between March 9 and April 6, 2020. For initial analyses, we used patient data from March 9 to April 5, and randomly assigned (80:20) the patients to the development dataset or test dataset 1 (retrospective). Patient data for those with encounters on April 6, 2020, were used in test dataset 2 (prospective). We designed prediction models based on clinical features and patient characteristics during health system encounters to predict mortality using the development dataset. We assessed the resultant models in terms of the area under the receiver operating characteristic curve (AUC) score in the test datasets.

Findings

Using the development dataset (n=3841) and a systematic machine learning framework, we developed a COVID-19 mortality prediction model that showed high accuracy (AUC=0·91) when applied to test datasets of retrospective (n=961) and prospective (n=249) patients. This model was based on three clinical features: patient's age, minimum oxygen saturation over the course of their medical encounter, and type of patient encounter (inpatient vs outpatient and telehealth visits).

Interpretation

An accurate and parsimonious COVID-19 mortality prediction model based on three features might have utility in clinical settings to guide the management and prognostication of patients affected by this disease. External validation of this prediction model in other populations is needed.

Funding

National Institutes of Health.

Does the SORG algorithm generalize to a contemporary cohort of patients with spinal metastases on external validation?

BACKGROUND CONTEXT

The SORG machine-learning algorithms were previously developed for preoperative prediction of overall survival in spinal metastatic disease. On sub-group analysis of a previous external validation, these algorithms were found to have diminished performance on patients treated after 2010.

PURPOSE

The purpose of this study was to assess the performance of these algorithms on a large contemporary cohort of consecutive spinal metastatic disease patients.

STUDY DESIGN/SETTING

Retrospective study performed at a tertiary care referral center.

PATIENT SAMPLE

Patients of 18 years and older treated with surgery for metastatic spinal disease between 2014 and 2016.

OUTCOME MEASURES

Ninety-day and one-year mortality.

METHODS

Baseline patient and tumor characteristics of the validation cohort were compared to the development cohort using bivariate logistic regression. Performance of the SORG algorithms on external validation in the contemporary cohort was assessed with discrimination (c-statistic and receiver operating curve), calibration (calibration plot, intercept, and slope), overall performance (Brier score compared to the null-model Brier score), and decision curve analysis.

RESULTS

Overall, 200 patients were included with 90-day and 1-year mortality rates of 55 (27.6%) and 124 (62.9%), respectively. The contemporary external validation cohort and the developmental cohort differed significantly on primary tumor histology, presence of visceral metastases, American Spinal Injury Association impairment scale, and preoperative laboratory values. The SORG algorithms for 90-day and 1-year mortality retained good discriminative ability (c-statistic of 0.81 [95% confidence interval [CI], 0.74-0.87] and 0.84 [95% CI, 0.77-0.89]), overall performance, and decision curve analysis. The algorithm for 90-day mortality showed almost perfect calibration reflected in an overall calibration intercept of -0.07 (95% CI: -0.50, 0.35). The 1-year mortality algorithm underestimated mortality mainly for the lowest predicted probabilities with an overall intercept of 0.57 (95% CI: 0.18, 0.96).

CONCLUSIONS

The SORG algorithms for survival in spinal metastatic disease generalized well to a contemporary cohort of consecutively treated patients from an external institutional. Further validation in international cohorts and large, prospective multi-institutional trials is required to confirm or refute the findings presented here. The open-access algorithms are available here: https://sorg-apps.shinyapps.io/spinemetssurvival/.

Comparison of Conventional Statistical Methods with Machine Learning in Medicine: Diagnosis, Drug Development, and Treatment

Futurists have anticipated that novel autonomous technologies, embedded with machine learning (ML), will substantially influence healthcare. ML is focused on making predictions as accurate as possible, while traditional statistical models are aimed at inferring relationships between variables. The benefits of ML comprise flexibility and scalability compared with conventional statistical approaches, which makes it deployable for several tasks, such as diagnosis and classification, and survival predictions. However, much of ML-based analysis remains scattered, lacking a cohesive structure. There is a need to evaluate and compare the performance of well-developed conventional statistical methods and ML on patient outcomes, such as survival, response to treatment, and patient-reported outcomes (PROs). In this article, we compare the usefulness and limitations of traditional statistical methods and ML, when applied to the medical field. Traditional statistical methods seem to be more useful when the number of cases largely exceeds the number of variables under study and a priori knowledge on the topic under study is substantial such as in public health. ML could be more suited in highly innovative fields with a huge bulk of data, such as omics, radiodiagnostics, drug development, and personalized treatment. Integration of the two approaches should be preferred over a unidirectional choice of either approach.

Prognostic models for newly-diagnosed chronic lymphocytic leukaemia in adults: a systematic review and meta-analysis

BACKGROUND

Chronic lymphocytic leukaemia (CLL) is the most common cancer of the lymphatic system in Western countries. Several clinical and biological factors for CLL have been identified. However, it remains unclear which of the available prognostic models combining those factors can be used in clinical practice to predict long-term outcome in people newly-diagnosed with CLL.

OBJECTIVES

To identify, describe and appraise all prognostic models developed to predict overall survival (OS), progression-free survival (PFS) or treatment-free survival (TFS) in newly-diagnosed (previously untreated) adults with CLL, and meta-analyse their predictive performances.

SEARCH METHODS

We searched MEDLINE (from January 1950 to June 2019 via Ovid), Embase (from 1974 to June 2019) and registries of ongoing trials (to 5 March 2020) for development and validation studies of prognostic models for untreated adults with CLL. In addition, we screened the reference lists and citation indices of included studies.

SELECTION CRITERIA

We included all prognostic models developed for CLL which predict OS, PFS, or TFS, provided they combined prognostic factors known before treatment initiation, and any studies that tested the performance of these models in individuals other than the ones included in model development (i.e. 'external model validation studies'). We included studies of adults with confirmed B-cell CLL who had not received treatment prior to the start of the study. We did not restrict the search based on study design.

DATA COLLECTION AND ANALYSIS

We developed a data extraction form to collect information based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). Independent pairs of review authors screened references, extracted data and assessed risk of bias according to the Prediction model Risk Of Bias ASsessment Tool (PROBAST). For models that were externally validated at least three times, we aimed to perform a quantitative meta-analysis of their predictive performance, notably their calibration (proportion of people predicted to experience the outcome who do so) and discrimination (ability to differentiate between people with and without the event) using a random-effects model. When a model categorised individuals into risk categories, we pooled outcome frequencies per risk group (low, intermediate, high and very high). We did not apply GRADE as guidance is not yet available for reviews of prognostic models.

MAIN RESULTS

From 52 eligible studies, we identified 12 externally validated models: six were developed for OS, one for PFS and five for TFS. In general, reporting of the studies was poor, especially predictive performance measures for calibration and discrimination; but also basic information, such as eligibility criteria and the recruitment period of participants was often missing. We rated almost all studies at high or unclear risk of bias according to PROBAST. Overall, the applicability of the models and their validation studies was low or unclear; the most common reasons were inappropriate handling of missing data and serious reporting deficiencies concerning eligibility criteria, recruitment period, observation time and prediction performance measures. We report the results for three models predicting OS, which had available data from more than three external validation studies: CLL International Prognostic Index (CLL-IPI) This score includes five prognostic factors: age, clinical stage, IgHV mutational status, B2-microglobulin and TP53 status. Calibration: for the low-, intermediate- and high-risk groups, the pooled five-year survival per risk group from validation studies corresponded to the frequencies observed in the model development study. In the very high-risk group, predicted survival from CLL-IPI was lower than observed from external validation studies. Discrimination: the pooled c-statistic of seven external validation studies (3307 participants, 917 events) was 0.72 (95% confidence interval (CI) 0.67 to 0.77). The 95% prediction interval (PI) of this model for the c-statistic, which describes the expected interval for the model's discriminative ability in a new external validation study, ranged from 0.59 to 0.83. Barcelona-Brno score Aimed at simplifying the CLL-IPI, this score includes three prognostic factors: IgHV mutational status, del(17p) and del(11q). Calibration: for the low- and intermediate-risk group, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of four external validation studies (1755 participants, 416 events) was 0.64 (95% CI 0.60 to 0.67); 95% PI 0.59 to 0.68. MDACC 2007 index score The authors presented two versions of this model including six prognostic factors to predict OS: age, B2-microglobulin, absolute lymphocyte count, gender, clinical stage and number of nodal groups. Only one validation study was available for the more comprehensive version of the model, a formula with a nomogram, while seven studies (5127 participants, 994 events) validated the simplified version of the model, the index score. Calibration: for the low- and intermediate-risk groups, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of the seven external validation studies for the index score was 0.65 (95% CI 0.60 to 0.70); 95% PI 0.51 to 0.77.

AUTHORS' CONCLUSIONS

Despite the large number of published studies of prognostic models for OS, PFS or TFS for newly-diagnosed, untreated adults with CLL, only a minority of these (N = 12) have been externally validated for their respective primary outcome. Three models have undergone sufficient external validation to enable meta-analysis of the model's ability to predict survival outcomes. Lack of reporting prevented us from summarising calibration as recommended. Of the three models, the CLL-IPI shows the best discrimination, despite overestimation. However, performance of the models may change for individuals with CLL who receive improved treatment options, as the models included in this review were tested mostly on retrospective cohorts receiving a traditional treatment regimen. In conclusion, this review shows a clear need to improve the conducting and reporting of both prognostic model development and external validation studies. For prognostic models to be used as tools in clinical practice, the development of the models (and their subsequent validation studies) should adapt to include the latest therapy options to accurately predict performance. Adaptations should be timely.

An international prospective cohort study to validate two prediction rules for infections caused by 3rd-generation cephalosporin-resistant Enterobacterales

Background

The possibility of bloodstream infections caused by third-generation cephalosporin-resistant Enterobacterales (3GC-R-BSI) leads to a trade-off between empiric inappropriate treatment (IAT) and unnecessary carbapenem use (UCU). Accurately predicting 3GC-R-BSI could reduce IAT and UCU. We externally validate 2 previously derived prediction rules for community-onset (CO) and hospital-onset (HO) suspected bloodstream infections.

Methods

In 33 hospitals in 13 countries we prospectively enrolled 200 patients per hospital in whom blood cultures were obtained and intravenous antibiotics with coverage for Enterobacterales were empirically started. Cases were defined as 3GC-R-BSI or 3GC-R gram-negative infection (3GC-R-GNI) (analysis 2); all other outcomes served as a comparator. Model discrimination and calibration were assessed. Impact on carbapenem use was assessed at several cutoff points.

Results

4650 CO infection episodes were included and the prevalence of 3GC-R-BSI was 2.1% (n = 97). IAT occurred in 69 of 97 (71.1%) 3GC-R-BSI and UCU in 398 of 4553 non–3GC-R-BSI patients (8.7%). Model calibration was good, and the AUC was .79 (95% CI, .75–.83) for 3GC-R-BSI. The prediction rule potentially reduced IAT to 62% (60/97) while keeping UCU comparable at 8.4% or could reduce UCU to 6.3% (287/4553) while keeping IAT equal. IAT and UCU in all 3GC-R-GNIs (analysis 2) improved at similar percentages. 1683 HO infection episodes were included and the prevalence of 3GC-R-BSI was 4.9% (n = 83). Here model calibration was insufficient.

Conclusions

A prediction rule for CO 3GC-R infection was validated in an international cohort and could improve empirical antibiotic use. Validation of the HO rule yielded suboptimal performance.

Improving Caries Risk Prediction Modeling: A Call for Action

Dentistry has entered an era of personalized/precision care in which targeting care to groups, individuals, or even tooth surfaces based on their caries risk has become a reality to address the skewed distribution of the disease. The best approach to determine a patient's prognosis relies on the development of caries risk prediction models (CRPMs). A desirable model should be derived and validated to appropriately discriminate between patients who will develop disease from those who will not, and it should provide an accurate estimation of the patient's absolute risk (i.e., calibration). However, evidence suggests there is a need to improve the methodological standards and increase consistency in the way CRPMs are developed and evaluated. In fact, although numerous caries risk assessment tools are available, most are not routinely used in practice or used to influence treatment decisions, and choice is not commonly based on high-quality evidence. Research will propose models that will become more complex, incorporating new factors with high prognostic value (e.g., human genetic markers, microbial biomarkers). Big data and predictive analytic methods will be part of the new approaches for the identification of promising predictors with the ability to monitor patients' risk in real time. Eventually, the implementation of validated, accurate CRPMs will have to follow a user-centered design respecting the patient-clinician dynamic, with no disruption to the clinical workflow, and needs to operate at low cost. The resulting predictive risk estimate needs to be presented to the patient in an understandable way so that it triggers behavior change and effectively informs health care decision making, to ultimately improve caries outcomes. However, research on these later aspects is largely missing and increasingly needed in dentistry.

Early warning score validation methodologies and performance metrics: a systematic review

BACKGROUND

Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.

METHODS

A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.

RESULTS

The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.

CONCLUSIONS

Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

Adaptation of the prostate biopsy collaborative group risk calculator in patients with PSA less than 10 ng/ml improves its performance

PURPOSES

The prostate biopsy collaborative group risk calculator (PBCGRC) is a newly developed risk estimator for predicting prostate biopsy outcomes. However, its clinical usefulness is still unknown within the so-called gray area of PSA values. This study aimed to determine whether updating the PBCGRC improves its predictive performance for predicting any-grade and high-grade (HG), defined as biopsy Gleason score ≥ 7, prostate cancer (PCa) in patients with prostate-specific antigen (PSA) less than 10 ng/ml.

METHODS

The risk of any-grade and HGPCa was calculated using the PBCG risk calculation formulas updated by recalibration in the large, logistic recalibration and model revision. Predictive performances of the PBCGRC and the updated models were compared using discrimination, calibration, and clinical utility.

RESULTS

Within the study sample of 526 patients, PCa was detected in 193 (36.7%), and 78 (14.8%) of them had HGPCa. According to the calibration curves, the PBCGRC overestimated the risk of PCa. Predictive accuracy of the revised model was higher [the area under the receiver-operating characteristic curve (AUCs), 65.4% and 70.2%] than that of the PBCGRC (AUCs, 60.4% and 64.3%) for any-grade and HGPCa. The net benefit was greater for model revision in comparison with the original model.

CONCLUSION

The performance accuracy of PBCGRC for the prediction of any and HGPC in men undergoing prostate biopsy with PSA levels below 10 ng/ml is suboptimal. The model revision resulted with significant improvement in model performance. However, external validation of the revised model is necessary before its routine use in clinical practice.

When predictions are used to allocate scarce health care resources: three considerations for models in the era of Covid-19

BACKGROUND

The need for life-saving interventions such as mechanical ventilation may threaten to outstrip resources during the Covid-19 pandemic. Allocation of these resources to those most likely to benefit can be supported by clinical prediction models. The ethical and practical considerations relevant to predictions supporting decisions about microallocation are distinct from those that inform shared decision-making in ways important for model design.

MAIN BODY

We review three issues of importance for microallocation: (1) Prediction of benefit (or of medical futility) may be technically very challenging; (2) When resources are scarce, calibration is less important for microallocation than is ranking to prioritize patients, since capacity determines thresholds for resource utilization; (3) The concept of group fairness, which is not germane in shared decision-making, is of central importance in microallocation. Therefore, model transparency is important.

CONCLUSION

Prediction supporting allocation of life-saving interventions should be explicit, data-driven, frequently updated and open to public scrutiny. This implies a preference for simple, easily understood and easily applied prognostic models.

An Attitude Survey and Assessment of the Feasibility, Acceptability, and Usability of a Traumatic Brain Injury Decision Support Tool in Uganda

BACKGROUND

Traumatic brain injury (TBI) prognostic models are potential solutions to severe human and technical shortages. Although numerous TBI prognostic models have been developed, none are widely used in clinical practice, largely because of a lack of feasibility research to inform implementation. We previously developed a prognostic model and Web-based application for in-hospital TBI care in low-resource settings. In this study, we tested the feasibility, acceptability, and usability of the application with potential end-users.

METHODS

We performed our feasibility assessment with providers involved in TBI care at both a regional and national referral hospital in Uganda. We collected qualitative and quantitative data on decision support needs, application ease of use, and implementation design.

RESULTS

We completed 25 questionnaires on potential uses of the app and 11 semistructured feasibility interviews. Top-cited uses were informing the decision to operate, informing the decision to send the patient to intensive care, and counseling patients and relatives. Participants affirmed the potential of the application to support difficult triage situations, particularly in the setting of limited access to diagnostics and interventions, but were hesitant to use this technology with end-of-life decisions. Although all participants were satisfied with the application and agreed that it was easy to use, several expressed a need for this technology to be accessible by smartphone and offline.

CONCLUSIONS

We elucidated several potential uses for our app and important contextual factors that will support future implementation. This investigation helps address an unmet need to determine the feasibility of TBI clinical decision support systems in low-resource settings.

Precisión del pronóstico de la propagación del COVID-19 en Colombia

Introducción: el nuevo Coronavirus (COVID-19) ha sido clasificado por la Organización Mundial de la Salud como una emergencia en salud pública de importancia internacional (ESPII). Se han reconocido casos en todos los continentes. El 6 de marzo de 2020 se confirmó el primer caso en Colombia. Objetivo: presentar la precisión de un pronóstico de la dinámica de transmisión del COVID-19 en Colombia. Metodología: para desarrollar la investigación se utilizó la base de datos de las personas infectadas con el Covid-19, esta información corresponde al período 6 de marzo al 14 de abril de 2020.  Para su análisis de predicción se manejó el método modelo de Brown, utilizando el paquete estadístico SPSS v.25. Resultados: se apreció que el error de pronóstico fue muy bajo y correspondió al MAPE (error porcentual medio absoluto), con un 0,03%, seguido del MAD (desviación media absoluta), con un valor de 0,95, es decir que en ambos casos la predicción obtuvieron un alto grado de confiabilidad. Conclusiones: el uso de modelación matemática se ha desarrollado en grado representativo en las últimas décadas y son de gran impulso para ilustrar escenarios eficaces de prevención y control de enfermedades infectocontagiosas.

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

OBJECTIVE

To review and appraise the validity and usefulness of published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital with the disease.

DESIGN

Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group.

DATA SOURCES

PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020.

STUDY SELECTION

Studies that developed or validated a multivariable covid-19 related prediction model.

DATA EXTRACTION

At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

RESULTS

37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models.

CONCLUSION

Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

SYSTEMATIC REVIEW REGISTRATION

Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

READERS' NOTE

This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.