Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Digital Mental Health for Schizophrenia and Other Severe Mental Illnesses: An International Consensus on Current Challenges and Potential Solutions

BACKGROUND

Digital approaches may be helpful in augmenting care to address unmet mental health needs, particularly for schizophrenia and severe mental illness (SMI).

OBJECTIVE

An international multidisciplinary group was convened to reach a consensus on the challenges and potential solutions regarding collecting data, delivering treatment, and the ethical challenges in digital mental health approaches for schizophrenia and SMI.

METHODS

The consensus development panel method was used, with an in-person meeting of 2 groups: the expert group and the panel. Membership was multidisciplinary including those with lived experience, with equal participation at all stages and coproduction of the consensus outputs and summary. Relevant literature was shared in advance of the meeting, and a systematic search of the recent literature on digital mental health interventions for schizophrenia and psychosis was completed to ensure that the panel was informed before the meeting with the expert group.

RESULTS

Four broad areas of challenge and proposed solutions were identified: (1) user involvement for real coproduction; (2) new approaches to methodology in digital mental health, including agreed standards, data sharing, measuring harms, prevention strategies, and mechanistic research; (3) regulation and funding issues; and (4) implementation in real-world settings (including multidisciplinary collaboration, training, augmenting existing service provision, and social and population-focused approaches). Examples are provided with more detail on human-centered research design, lived experience perspectives, and biomedical ethics in digital mental health approaches for SMI.

CONCLUSIONS

The group agreed by consensus on a number of recommendations: (1) a new and improved approach to digital mental health research (with agreed reporting standards, data sharing, and shared protocols), (2) equal emphasis on social and population research as well as biological and psychological approaches, (3) meaningful collaborations across varied disciplines that have previously not worked closely together, (4) increased focus on the business model and product with planning and new funding structures across the whole development pathway, (5) increased focus and reporting on ethical issues and potential harms, and (6) organizational changes to allow for true communication and coproduction with those with lived experience of SMI. This study approach, combining an international expert meeting with patient and public involvement and engagement throughout the process, consensus methodology, discussion, and publication, is a helpful way to identify directions for future research and clinical implementation in rapidly evolving areas and can be combined with measurements of real-world clinical impact over time. Similar initiatives will be helpful in other areas of digital mental health and similarly fast-evolving fields to focus research and organizational change and effect improved real-world clinical implementation.

Computable phenotype for real-world, data-driven retrospective identification of relapse in ANCA-associated vasculitis

OBJECTIVE

ANCA-associated vasculitis (AAV) is a relapsing-remitting disease, resulting in incremental tissue injury. The gold-standard relapse definition (Birmingham Vasculitis Activity Score, BVAS>0) is often missing or inaccurate in registry settings, leading to errors in ascertainment of this key outcome. We sought to create a computable phenotype (CP) to automate retrospective identification of relapse using real-world data in the research setting.

METHODS

We studied 536 patients with AAV and >6 months follow-up recruited to the Rare Kidney Disease registry (a national longitudinal, multicentre cohort study). We followed five steps: (1) independent encounter adjudication using primary medical records to assign the ground truth, (2) selection of data elements (DEs), (3) CP development using multilevel regression modelling, (4) internal validation and (5) development of additional models to handle missingness. Cut-points were determined by maximising the F1-score. We developed a web application for CP implementation, which outputs an individualised probability of relapse.

RESULTS

Development and validation datasets comprised 1209 and 377 encounters, respectively. After classifying encounters with diagnostic histopathology as relapse, we identified five key DEs; DE1: change in ANCA level, DE2: suggestive blood/urine tests, DE3: suggestive imaging, DE4: immunosuppression status, DE5: immunosuppression change. F1-score, sensitivity and specificity were 0.85 (95% CI 0.77 to 0.92), 0.89 (95% CI 0.80 to 0.99) and 0.96 (95% CI 0.93 to 0.99), respectively. Where DE5 was missing, DE2 plus either DE1/DE3 were required to match the accuracy of BVAS.

CONCLUSIONS

This CP accurately quantifies the individualised probability of relapse in AAV retrospectively, using objective, readily accessible registry data. This framework could be leveraged for other outcomes and relapsing diseases.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting

OBJECTIVES

To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes.

STUDY DESIGN AND SETTING

This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation.

MAIN OUTCOME MEASURE

In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting.

RESULTS

All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71.

CONCLUSION

Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

Dynamic updating of clinical survival prediction models in a changing environment

BACKGROUND

Over time, the performance of clinical prediction models may deteriorate due to changes in clinical management, data quality, disease risk and/or patient mix. Such prediction models must be updated in order to remain useful. In this study, we investigate dynamic model updating of clinical survival prediction models. In contrast to discrete or one-time updating, dynamic updating refers to a repeated process for updating a prediction model with new data. We aim to extend previous research which focused largely on binary outcome prediction models by concentrating on time-to-event outcomes. We were motivated by the rapidly changing environment seen during the COVID-19 pandemic where mortality rates changed over time and new treatments and vaccines were introduced.

METHODS

We illustrate three methods for dynamic model updating: Bayesian dynamic updating, recalibration, and full refitting. We use a simulation study to compare performance in a range of scenarios including changing mortality rates, predictors with low prevalence and the introduction of a new treatment. Next, the updating strategies were applied to a model for predicting 70-day COVID-19-related mortality using patient data from QResearch, an electronic health records database from general practices in the UK.

RESULTS

In simulated scenarios with mortality rates changing over time, all updating methods resulted in better calibration than not updating. Moreover, dynamic updating outperformed ad hoc updating. In the simulation scenario with a new predictor and a small updating dataset, Bayesian updating improved the C-index over not updating and refitting. In the motivating example with a rare outcome, no single updating method offered the best performance.

CONCLUSIONS

We found that a dynamic updating process outperformed one-time discrete updating in the simulations. Bayesian updating offered good performance overall, even in scenarios with new predictors and few events. Intercept recalibration was effective in scenarios with smaller sample size and changing baseline hazard. Refitting performance depended on sample size and produced abrupt changes in hazard ratio estimates between periods.

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

An external validation of the Kidney Donor Risk Index in the UK transplant population in the presence of semi-competing events

BACKGROUND

Transplantation represents the optimal treatment for many patients with end-stage kidney disease. When a donor kidney is available to a waitlisted patient, clinicians responsible for the care of the potential recipient must make the decision to accept or decline the offer based upon complex and variable information about the donor, the recipient and the transplant process. A clinical prediction model may be able to support clinicians in their decision-making. The Kidney Donor Risk Index (KDRI) was developed in the United States to predict graft failure following kidney transplantation. The survival process following transplantation consists of semi-competing events where death precludes graft failure, but not vice-versa.

METHODS

We externally validated the KDRI in the UK kidney transplant population and assessed whether validation under a semi-competing risks framework impacted predictive performance. Additionally, we explored whether the KDRI requires updating. We included 20,035 adult recipients of first, deceased donor, single, kidney-only transplants between January 1, 2004, and December 31, 2018, collected by the UK Transplant Registry and held by NHS Blood and Transplant. The outcomes of interest were 1- and 5-year graft failure following transplantation. In light of the semi-competing events, recipient death was handled in two ways: censoring patients at the time of death and modelling death as a competing event. Cox proportional hazard models were used to validate the KDRI when censoring graft failure by death, and cause-specific Cox models were used to account for death as a competing event.

RESULTS

The KDRI underestimated event probabilities for those at higher risk of graft failure. For 5-year graft failure, discrimination was poorer in the semi-competing risks model (0.625, 95% CI 0.611 to 0.640;0.611, 95% CI 0.597 to 0.625), but predictions were more accurate (Brier score 0.117, 95% CI 0.112 to 0.121; 0.114, 95% CI 0.109 to 0.118). Calibration plots were similar regardless of whether the death was modelled as a competing event or not. Updating the KDRI worsened calibration, but marginally improved discrimination.

CONCLUSIONS

Predictive performance for 1-year graft failure was similar between death-censored and competing event graft failure, but differences appeared when predicting 5-year graft failure. The updated index did not have superior performance and we conclude that updating the KDRI in the present form is not required.

Risk factors for severe respiratory syncytial virus infection during the first year of life: development and validation of a clinical prediction model

BACKGROUND

Novel immunisation methods against respiratory syncytial virus (RSV) are emerging, but knowledge of risk factors for severe RSV disease is insufficient for optimal targeting of interventions against them. Our aims were to identify predictors for RSV hospital admission from registry-based data and to develop and validate a clinical prediction model to guide RSV immunoprophylaxis for infants younger than 1 year.

METHODS

In this model development and validation study, we studied all infants born in Finland between June 1, 1997, and May 31, 2020, and in Sweden between June 1, 2006, and May 31, 2020, along with the data for their parents and siblings. Infants were excluded if they died or were admitted to hospital for RSV within the first 7 days of life. The outcome was hospital admission due to RSV bronchiolitis during the first year of life. The Finnish study population was divided into a development dataset (born between June 1, 1997, and May 31, 2017) and a temporal hold-out validation dataset (born between June 1, 2017, and May 31, 2020). The development dataset was used for predictor discovery and selection in which we screened 1511 candidate predictors from the infants', parents', and siblings' data, and developed a logistic regression model with the 16 most important predictors. This model was then validated using the Finnish hold-out validation dataset and the Swedish dataset.

FINDINGS

In total, there were 1 124 561 infants in the Finnish development dataset, 130 352 infants in the Finnish hold-out validation dataset, and 1 459 472 infants in the Swedish dataset. In addition to known predictors such as severe congenital heart defects (adjusted odds ratio 2·89, 95% CI 2·28-3·65), we confirmed some less established predictors for RSV hospital admission, most notably oesophageal malformations (3·11, 1·86-5·19) and lower complexity congenital heart defects (1·43, 1·25-1·63). The prediction model's C-statistic was 0·766 (95% CI 0·742-0·789) in Finnish data and 0·737 (0·710-0·762) in Swedish validation data. The infants in the highest decile of predicted RSV hospital admission probability had 4·5 times higher observed risk compared with others. Calibration varied according to epidemic intensity. The model's performance was similar to a machine learning (XGboost) model using all 1511 candidate predictors (C-statistic in Finland 0·771, 95% CI 0·754-0·788). The prediction model showed clinical utility in decision curve analysis and in hypothetical number needed to treat calculations for immunisation, and its C-statistic was similar across different strata of parental income.

INTERPRETATION

The identified predictors and the prediction model can be used in guiding RSV immunoprophylaxis in infants, or as a basis for further immunoprophylaxis targeting tools.

FUNDING

Sigrid Jusélius Foundation, European Research Council, Pediatric Research Foundation, and Academy of Finland.

Opportunities and Challenges of Prognostic Models for Extremely Preterm Infants

Predicting the short- and long-term outcomes of extremely preterm infants remains a challenge. Multivariable prognostic models might be valuable tools for clinicians, parents, and policymakers for providing accurate outcome estimates. In this perspective, we discuss the opportunities and challenges of using prognostic models in extremely preterm infants at population and individual levels. At a population level, these models could support the development of guidelines for decisions about treatment limits and may support policy processes such as benchmarking and resource allocation. At an individual level, these models may enhance prenatal counselling conversations by considering multiple variables and improving transparency about expected outcomes. Furthermore, they may improve consistency in projections shared with parents. For the development of prognostic models, we discuss important considerations such as predictor and outcome measure selection, clinical impact assessment, and generalizability. Lastly, future recommendations for developing and using prognostic models are suggested. Importantly, the purpose of a prognostic model should be clearly defined, and integrating these models into prenatal counselling requires thoughtful consideration.

Automatic correction of performance drift under acquisition shift in medical image classification

Image-based prediction models for disease detection are sensitive to changes in data acquisition such as the replacement of scanner hardware or updates to the image processing software. The resulting differences in image characteristics may lead to drifts in clinically relevant performance metrics which could cause harm in clinical decision making, even for models that generalise in terms of area under the receiver-operating characteristic curve. We propose Unsupervised Prediction Alignment, a generic automatic recalibration method that requires no ground truth annotations and only limited amounts of unlabelled example images from the shifted data distribution. We illustrate the effectiveness of the proposed method to detect and correct performance drift in mammography-based breast cancer screening and on publicly available histopathology data. We show that the proposed method can preserve the expected performance in terms of sensitivity/specificity under various realistic scenarios of image acquisition shift, thus offering an important safeguard for clinical deployment.

Implications of the Use of Artificial Intelligence Predictive Models in Health Care Settings : A Simulation Study

BACKGROUND

Substantial effort has been directed toward demonstrating uses of predictive models in health care. However, implementation of these models into clinical practice may influence patient outcomes, which in turn are captured in electronic health record data. As a result, deployed models may affect the predictive ability of current and future models.

OBJECTIVE

To estimate changes in predictive model performance with use through 3 common scenarios: model retraining, sequentially implementing 1 model after another, and intervening in response to a model when 2 are simultaneously implemented.

DESIGN

Simulation of model implementation and use in critical care settings at various levels of intervention effectiveness and clinician adherence. Models were either trained or retrained after simulated implementation.

SETTING

Admissions to the intensive care unit (ICU) at Mount Sinai Health System (New York, New York) and Beth Israel Deaconess Medical Center (Boston, Massachusetts).

PATIENTS

130 000 critical care admissions across both health systems.

INTERVENTION

Across 3 scenarios, interventions were simulated at varying levels of clinician adherence and effectiveness.

MEASUREMENTS

Statistical measures of performance, including threshold-independent (area under the curve) and threshold-dependent measures.

RESULTS

At fixed 90% sensitivity, in scenario 1 a mortality prediction model lost 9% to 39% specificity after retraining once and in scenario 2 a mortality prediction model lost 8% to 15% specificity when created after the implementation of an acute kidney injury (AKI) prediction model; in scenario 3, models for AKI and mortality prediction implemented simultaneously, each led to reduced effective accuracy of the other by 1% to 28%.

LIMITATIONS

In real-world practice, the effectiveness of and adherence to model-based recommendations are rarely known in advance. Only binary classifiers for tabular ICU admissions data were simulated.

CONCLUSION

In simulated ICU settings, a universally effective model-updating approach for maintaining model performance does not seem to exist. Model use may have to be recorded to maintain viability of predictive modeling.

PRIMARY FUNDING SOURCE

National Center for Advancing Translational Sciences.

Fixing the Leaky Pipe: How to Improve the Uptake of Patient-Reported Outcomes-Based Prognostic and Predictive Models in Cancer Clinical Practice

PURPOSE

This discussion paper outlines challenges and proposes solutions for successfully implementing prediction models that incorporate patient-reported outcomes (PROs) in cancer practice.

METHODS

We organized a full-day multidisciplinary meeting of people with expertise in cancer care delivery, PRO collection, PRO use in prediction modeling, computing, implementation, and decision science. The discussions presented here focused on identifying challenges to the development, implementation and use of prediction models incorporating PROs, and suggesting possible solutions.

RESULTS

Specific challenges and solutions were identified across three broad areas. (1) Understanding decision making and implementation: necessitating multidisciplinary collaboration in the early stages and throughout; early stakeholder engagement to define the decision problem and ensure acceptability of PROs in prediction; understanding patient/clinician interpretation of PRO predictions and uncertainty to optimize prediction impact; striving for model integration into existing electronic health records; and early regulatory alignment. (2) Recognizing the limitations to PRO collection and their impact on prediction: incorporating validated, clinically important PROs to maximize model generalizability and clinical engagement; and minimizing missing PRO data (resulting from both structural digital exclusion and time-varying factors) to avoid exacerbating existing inequalities. (3) Statistical and modeling challenges: incorporating statistical methods to address missing data; ensuring predictive modeling recognizes complex causal relationships; and considering temporal and geographic recalibration so that model predictions reflect the relevant population.

CONCLUSION

Developing and implementing PRO-based prediction models in cancer care requires extensive multidisciplinary working from the earliest stages, recognition of implementation challenges because of PRO collection and model presentation, and robust statistical methods to manage missing data, causality, and calibration. Prediction models incorporating PROs should be viewed as complex interventions, with their development and impact assessment carried out to reflect this.

Subacute prognosis of oral nutrition (SPOON): Development of a multivariable prognostic model for complete oral intake in tube-fed subjects with acquired brain injury

BACKGROUND & AIMS

Enteral feeding is very common following an acquired brain injury. However, no prognostic models for oral food intake have been developed for subacute rehabilitation. The aim of this study was to develop a prognostic model and online tool, coined "subacute prognosis of oral nutrition" (SPOON), for complete oral intake in tube-fed subjects with acquired brain injury.

METHODS

The model was developed using routinely gathered clinical data from a cohort of 1233 adult patients who were tube-fed at admission for sub-acute inpatient rehabilitation. Candidate predictors were included based on scientific evidence and their availability in the medical records within the first days following admission. The outcome was time until achieving complete oral food intake without any tube-feeding supplements. Time until complete oral intake was analyzed by discrete time-to-event analysis with logit-link and presented as daily odds ratios (OR) with 95% confidence intervals (CI).

RESULTS

The following predictors of complete oral intake were included in the model: age, diagnosis, cuffed tracheostomy tube, days from injury to admission for rehabilitation, and the Early Functional Abilities (EFA) sum score. Multiple adjusted analyses were performed stratified by cuffed tracheostomy tube status. Some of the strongest predictors of complete oral intake were age 18-40 years, OR 1.99 (95%CI: 1.53; 2.59); 0-2 weeks since injury, OR 3.75 (95%CI: 2.72; 5.16); and EFA 61-100 (slight/no disturbance in function), OR 5.81 (95%CI: 4.47; 7.55). The online prognostic tool SPOON was evaluated in a usability study. Based on feedback from clinicians, the tool was further refined to enable extraction of data for prediction directly from medical records.

CONCLUSIONS

The objective of SPOON is to complement the planning of rehabilitation initiatives and inform discussions to determine if a percutaneous endoscopic gastrostomy (PEG) tube should be inserted. SPOON is being implemented locally, but external validation based on appropriate data modeling is warranted before further clinical implementation.

Into the Trees: Random Forests for Predicting Fusarium Head Blight Epidemics of Wheat in the United States

Constructing models that accurately predict Fusarium head blight (FHB) epidemics and are also amenable to large-scale deployment is a challenging task. In the United States, the emphasis has been on simple logistic regression (LR) models, which are easy to implement but may suffer from lower accuracies when compared with more complicated, harder-to-deploy (over large geographies) model frameworks such as functional or boosted regressions. This article examined the plausibility of random forests (RFs) for the binary prediction of FHB epidemics as a possible mediation between model simplicity and complexity without sacrificing accuracy. A minimalist set of predictors was also desirable rather than having the RF model use all 90 candidate variables as predictors. The input predictor set was filtered with the aid of three RF variable selection algorithms (Boruta, varSelRF, and VSURF), using resampling techniques to quantify the variability and stability of selected variable sets. Post-selection filtering produced 58 competitive RF models with no more than 14 predictors each. One variable representing temperature stability in the 20 days before anthesis was the most frequently selected predictor. This was a departure from the prominence of relative humidity-based variables previously reported in LR models for FHB. The RF models had overall superior predictive performance over the LR models and may be suitable candidates for use by the Fusarium Head Blight Prediction Center.

Can the application of machine learning to electronic health records guide antibiotic prescribing decisions for suspected urinary tract infection in the Emergency Department?

Urinary tract infections (UTIs) are a major cause of emergency hospital admissions, but it remains challenging to diagnose them reliably. Application of machine learning (ML) to routine patient data could support clinical decision-making. We developed a ML model predicting bacteriuria in the ED and evaluated its performance in key patient groups to determine scope for its future use to improve UTI diagnosis and thus guide antibiotic prescribing decisions in clinical practice. We used retrospective electronic health records from a large UK hospital (2011-2019). Non-pregnant adults who attended the ED and had a urine sample cultured were eligible for inclusion. The primary outcome was predominant bacterial growth ≥104 cfu/mL in urine. Predictors included demography, medical history, ED diagnoses, blood tests, and urine flow cytometry. Linear and tree-based models were trained via repeated cross-validation, re-calibrated, and validated on data from 2018/19. Changes in performance were investigated by age, sex, ethnicity, and suspected ED diagnosis, and compared to clinical judgement. Among 12,680 included samples, 4,677 (36.9%) showed bacterial growth. Relying primarily on flow cytometry parameters, our best model achieved an area under the ROC curve (AUC) of 0.813 (95% CI 0.792-0.834) in the test data, and achieved both higher sensitivity and specificity compared to proxies of clinician's judgement. Performance remained stable for white and non-white patients but was lower during a period of laboratory procedure change in 2015, in patients ≥65 years (AUC 0.783, 95% CI 0.752-0.815), and in men (AUC 0.758, 95% CI 0.717-0.798). Performance was also slightly reduced in patients with recorded suspicion of UTI (AUC 0.797, 95% CI 0.765-0.828). Our results suggest scope for use of ML to inform antibiotic prescribing decisions by improving diagnosis of suspected UTI in the ED, but performance varied with patient characteristics. Clinical utility of predictive models for UTI is therefore likely to differ for important patient subgroups including women <65 years, women ≥65 years, and men. Tailored models and decision thresholds may be required that account for differences in achievable performance, background incidence, and risks of infectious complications in these groups.

Six Externally Validated Prognostic Models Have Potential Clinical Value to Predict Patient Health Outcomes in the Rehabilitation of Musculoskeletal Conditions: A Systematic Review

OBJECTIVE

The purpose of this systematic review was to identify and appraise externally validated prognostic models to predict a patient's health outcomes relevant to physical rehabilitation of musculoskeletal (MSK) conditions.

METHODS

We systematically reviewed 8 databases and reported our findings according to Preferred Reporting Items for Systematic Reviews and Meta-Analysis 2020. An information specialist designed a search strategy to identify externally validated prognostic models for MSK conditions. Paired reviewers independently screened the title, abstract, and full text and conducted data extraction. We extracted characteristics of included studies (eg, country and study design), prognostic models (eg, performance measures and type of model) and predicted clinical outcomes (eg, pain and disability). We assessed the risk of bias and concerns of applicability using the prediction model risk of bias assessment tool. We proposed and used a 5-step method to determine which prognostic models were clinically valuable.

RESULTS

We found 4896 citations, read 300 full-text articles, and included 46 papers (37 distinct models). Prognostic models were externally validated for the spine, upper limb, lower limb conditions, and MSK trauma, injuries, and pain. All studies presented a high risk of bias. Half of the models showed low concerns for applicability. Reporting of calibration and discrimination performance measures was often lacking. We found 6 externally validated models with adequate measures, which could be deemed clinically valuable [ie, (1) STart Back Screening Tool, (2) Wallis Occupational Rehabilitation RisK model, (3) Da Silva model, (4) PICKUP model, (5) Schellingerhout rule, and (6) Keene model]. Despite having a high risk of bias, which is mostly explained by the very conservative properties of the PROBAST tool, the 6 models remain clinically relevant.

CONCLUSION

We found 6 externally validated prognostic models developed to predict patients' health outcomes that were clinically relevant to the physical rehabilitation of MSK conditions.

IMPACT

Our results provide clinicians with externally validated prognostic models to help them better predict patients' clinical outcomes and facilitate personalized treatment plans. Incorporating clinically valuable prognostic models could inherently improve the value of care provided by physical therapists.

Predicted probabilities of live birth following assisted reproductive technology using United States national surveillance data from 2016 to 2018

BACKGROUND

As the use of in vitro fertilization continues to increase in the United States, up-to-date models that estimate cumulative live birth rates after multiple oocyte retrievals and embryo transfers (fresh and frozen) are valuable for patients and clinicians weighing treatment options.

OBJECTIVE

This study aimed to develop models that generate predicted probabilities of live birth in individuals considering in vitro fertilization based on demographic and reproductive characteristics.

STUDY DESIGN

Our population-based cohort study used data from the National Assisted Reproductive Technology Surveillance System 2016 to 2018, including 196,916 women who underwent 207,766 autologous embryo transfer cycles and 25,831 women who underwent 36,909 donor oocyte transfer cycles. We used data on autologous in vitro fertilization cycles to develop models that estimate a patient's cumulative live birth rate after all embryo transfers (fresh and frozen) within 12 months after 1, 2, and 3 oocyte retrievals in new and returning patients. Among patients using donor oocytes, we estimated the cumulative live birth rate after their first, second, and third embryo transfers. Multinomial logistic regression models adjusted for age, prepregnancy body mass index (imputed for 18% of missing values), parity, gravidity, and infertility diagnoses were used to estimate the cumulative live birth rate.

RESULTS

Among new and returning patients undergoing autologous in vitro fertilization, female age had the strongest association with cumulative live birth rate. Other factors associated with higher cumulative live birth rates were lower body mass index and parity or gravidity ≥1, although results were inconsistent. Infertility diagnoses of diminished ovarian reserve, uterine factor, and other reasons were associated with a lower cumulative live birth rate, whereas male factor, tubal factor, ovulatory disorders, and unexplained infertility were associated with a higher cumulative live birth rate. Based on our models, a new patient who is 35 years old, with a body mass index of 25 kg/m2, no previous pregnancy, and unexplained infertility diagnoses, has a 48%, 69%, and 80% cumulative live birth rate after the first, second, and third oocyte retrieval, respectively. Cumulative live birth rates are 29%, 48%, and 62%, respectively, if the patient had diminished ovarian reserve, and 25%, 41%, and 52%, respectively, if the patient was 40 years old (with unexplained infertility). Very few recipient characteristics were associated with cumulative live birth rate in donor oocyte patients.

CONCLUSION

Our models provided estimates of cumulative live birth rate based on demographic and reproductive characteristics to help inform patients and providers of a woman's probability of success after in vitro fertilization.

Group B Streptococcus Early-Onset Disease: New Preventive and Diagnostic Tools to Decrease the Burden of Antibiotic Use

The difficulty in recognizing early-onset neonatal sepsis (EONS) in a timely manner due to non-specific symptoms and the limitations of diagnostic tests, combined with the risk of serious consequences if EONS is not treated in a timely manner, has resulted in a low threshold for starting empirical antibiotic treatment. New guideline strategies, such as the neonatal sepsis calculator, have been proven to reduce the antibiotic burden related to EONS, but lack sensitivity for detecting EONS. In this review, the potential of novel, targeted preventive and diagnostic methods for EONS is discussed from three different perspectives: maternal, umbilical cord and newborn perspectives. Promising strategies from the maternal perspective include Group B Streptococcus (GBS) prevention, exploring the virulence factors of GBS, maternal immunization and antepartum biomarkers. The diagnostic methods obtained from the umbilical cord are preliminary but promising. Finally, promising fields from the newborn perspective include biomarkers, new microbiological techniques and clinical prediction and monitoring strategies. Consensus on the definition of EONS and the standardization of research on novel diagnostic biomarkers are crucial for future implementation and to reduce current antibiotic overexposure in newborns.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

Severity of Illness Scores and Biomarkers for Prognosis of Patients with Coronavirus Disease 2019

The spectrum of disease severity and the insidiousness of clinical presentation make it difficult to recognize patients with coronavirus disease 2019 (COVID-19) at higher risk of worse outcomes or death when they are seen in the early phases of the disease. There are now well-established risk factors for worse outcomes in patients with COVID-19. These should be factored in when assessing the prognosis of these patients. However, a more precise prognostic assessment in an individual patient may warrant the use of predictive tools. In this manuscript, we conduct a literature review on the severity of illness scores and biomarkers for the prognosis of patients with COVID-19. Several COVID-19-specific scores have been developed since the onset of the pandemic. Some of them are promising and can be integrated into the assessment of these patients. We also found that the well-known pneumonia severity index (PSI) and CURB-65 (confusion, uremia, respiratory rate, BP, age ≥ 65 years) are good predictors of mortality in hospitalized patients with COVID-19. While neither the PSI nor the CURB-65 should be used for the triage of outpatient versus inpatient treatment, they can be integrated by a clinician into the assessment of disease severity and can be used in epidemiological studies to determine the severity of illness in patient populations. Biomarkers also provide valuable prognostic information and, importantly, may depict the main physiological derangements in severe disease. We, however, do not advocate the isolated use of severity of illness scores or biomarkers for decision-making in an individual patient. Instead, we suggest the use of these tools on a case-by-case basis with the goal of enhancing clinician judgment.

Taking shared decision making for prostate cancer to the next level: Requirements for a Dutch treatment decision aid with personalized risks on side effects

BACKGROUND

Different curative treatment modalities need to be considered in case of localized prostate cancer, all comparable in terms of survival and recurrence though different in side effects. To better inform patients and support shared decision making, the development of a web-based patient decision aid including personalized risk information was proposed. This paper reports on requirements in terms of content of information, visualization of risk profiles, and use in practice.

METHODS

Based on a Dutch 10-step guide about the setup of a decision aid next to a practice guideline, an iterative and co-creative design process was followed. In collaboration with various groups of experts (health professionals, usability and linguistic experts, patients and the general public), research and development activities were continuously alternated.

RESULTS

Content requirements focused on presenting information only about conventional treatments and main side effects; based on risk group; and including clear explanations about personalized risks. Visual requirements involved presenting general and personalized risks separately; through bar charts or icon arrays; and along with numbers or words, and legends. Organizational requirements included integration into local clinical pathways; agreement about information input and output; and focus on patients' numeracy and graph literacy skills.

CONCLUSIONS

The iterative and co-creative development process was challenging, though extremely valuable. The translation of requirements resulted in a decision aid about four conventional treatment options, including general or personalized risks for erection, urinary and intestinal problems that are communicated with icon arrays and numbers. Future implementation and validation studies need to inform about use and value in practice.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

BACKGROUND

Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models.

METHODS

We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers.

RESULTS

Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models.

CONCLUSION

Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

A hybrid adaptive approach for instance transfer learning with dynamic and imbalanced data

Machine learning has demonstrated success in clinical risk prediction modeling with complex electronic health record data. However, the evolving nature of clinical practices can dynamically change the underlying data distribution over time, leading to model performance drift. Adopting an outdated model is potentially risky and may result in unintentional losses. In this paper, we propose a novel Hybrid Adaptive Boosting approach (HA-Boost) for transfer learning. HA-Boost is characterized by the domain similarity-based and class imbalance-based adaptation mechanisms, which simultaneously address two critical limitations of the classical TrAdaBoost algorithm. We validated HA-Boost in predicting hospital-acquired acute kidney injury using real-world longitudinal electronic health records data. The experiment results demonstrate that HA-Boost stably outperforms the competing baselines in terms of both AUROC and AUPRC across a 7-year time span. This study has confirmed the effectiveness of transfer learning as a superior model updating approach in dynamic environment.

Identification of clinical factors related to prediction of alcohol use disorder from electronic health records using feature selection methods

BACKGROUND

High dimensionality in electronic health records (EHR) causes a significant computational problem for any systematic search for predictive, diagnostic, or prognostic patterns. Feature selection (FS) methods have been indicated to be effective in feature reduction as well as in identifying risk factors related to prediction of clinical disorders. This paper examines the prediction of patients with alcohol use disorder (AUD) using machine learning (ML) and attempts to identify risk factors related to the diagnosis of AUD.

METHODS

A FS framework consisting of two operational levels, base selectors and ensemble selectors. The first level consists of five FS methods: three filter methods, one wrapper method, and one embedded method. Base selector outputs are aggregated to develop four ensemble FS methods. The outputs of FS method were then fed into three ML algorithms: support vector machine (SVM), K-nearest neighbor (KNN), and random forest (RF) to compare and identify the best feature subset for the prediction of AUD from EHRs.

RESULTS

In terms of feature reduction, the embedded FS method could significantly reduce the number of features from 361 to 131. In terms of classification performance, RF based on 272 features selected by our proposed ensemble method (Union FS) with the highest accuracy in predicting patients with AUD, 96%, outperformed all other models in terms of AUROC, AUPRC, Precision, Recall, and F1-Score. Considering the limitations of embedded and wrapper methods, the best overall performance was achieved by our proposed Union Filter FS, which reduced the number of features to 223 and improved Precision, Recall, and F1-Score in RF from 0.77, 0.65, and 0.71 to 0.87, 0.81, and 0.84, respectively. Our findings indicate that, besides gender, age, and length of stay at the hospital, diagnosis related to digestive organs, bones, muscles and connective tissue, and the nervous systems are important clinical factors related to the prediction of patients with AUD.

CONCLUSION

Our proposed FS method could improve the classification performance significantly. It could identify clinical factors related to prediction of AUD from EHRs, thereby effectively helping clinical staff to identify and treat AUD patients and improving medical knowledge of the AUD condition. Moreover, the diversity of features among female and male patients as well as gender disparity were investigated using FS methods and ML techniques.

Prognostic models for COVID-19 needed updating to warrant transportability over time and space

BACKGROUND

Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time.

METHODS

We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit.

RESULTS

Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data.

CONCLUSIONS

NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings

As the implementation of artificial intelligence (AI)-enabled tools is realized across diverse clinical environments, there is a growing understanding of the need for ongoing monitoring and updating of prediction models. Dataset shift-temporal changes in clinical practice, patient populations, and information systems-is now well-documented as a source of deteriorating model accuracy and a challenge to the sustainability of AI-enabled tools in clinical care. While best practices are well-established for training and validating new models, there has been limited work developing best practices for prospective validation and model maintenance. In this paper, we highlight the need for updating clinical prediction models and discuss open questions regarding this critical aspect of the AI modeling lifecycle in three focus areas: model maintenance policies, performance monitoring perspectives, and model updating strategies. With the increasing adoption of AI-enabled tools, the need for such best practices must be addressed and incorporated into new and existing implementations. This commentary aims to encourage conversation and motivate additional research across clinical and data science stakeholders.

Risk scores in anaesthesia: the future is hard to predict

External validation helps to assess whether a given risk prediction model will perform well in a target population. Validation is an important step in maintaining the utility of risk prediction models, as their ability to provide reliable risk estimates will deteriorate over time (calibration drift).

ACCEPT 2·0: Recalibrating and externally validating the Acute COPD exacerbation prediction tool (ACCEPT)

BACKGROUND

The Acute Chronic Obstructive Pulmonary Disease (COPD) Exacerbation Prediction Tool (ACCEPT) was developed for individualised prediction of COPD exacerbations. ACCEPT was well calibrated overall and had a high discriminatory power, but overestimated risk among individuals without recent exacerbations. The objectives of this study were to 1) fine-tune ACCEPT to make better predictions for individuals with a negative exacerbation history, 2) develop more parsimonious models, and 3) externally validate the models in a new dataset.

METHODS

We recalibrated ACCEPT using data from the Evaluation of COPD Longitudinally to Identify Predictive Surrogate End-points (ECLIPSE, a three-year observational study, 1,803 patients, 2,117 exacerbations) study by applying non-parametric regression splines to the predicted rates. We developed three reduced versions of ACCEPT by removing symptom score and/or baseline medications as predictors. We examined the discrimination, calibration, and net benefit of ACCEPT 2·0 in the placebo arm of the Towards a Revolution in COPD Health (TORCH, a three-year randomised clinical trial of inhaled therapies in COPD, 1,091 patients, 1,064 exacerbations) study. The primary outcome for prediction was the occurrence of ≥2 moderate or ≥1 severe exacerbation in the next 12 months; the secondary outcomes were prediction of the occurrence of any moderate/severe exacerbation or any severe exacerbation.

FINDINGS

ACCEPT 2·0 had an area-under-the-curve (AUC) of 0·76 for predicting the primary outcome. Exacerbation history alone (current standard of care) had an AUC of 0·68. The model was well calibrated in patients with positive or negative exacerbation histories. Changes in AUC in reduced versions were minimal for the primary outcome as well as for predicting the occurrence of any moderate/severe exacerbations (ΔAUC<0·011), but more substantial for predicting the occurrence of any severe exacerbations (ΔAUC<0·020). All versions of ACCEPT 2·0 provided positive net benefit over the use of exacerbation history alone for some range of thresholds.

INTERPRETATION

ACCEPT 2·0 showed good calibration regardless of exacerbation history, and predicts exacerbation risk better than current standard of care for a range of thresholds. Future studies need to investigate the utility of exacerbation prediction in various subgroups of patients.

FUNDING

This study was funded by a team grant from the Canadian Institutes of Health Research (PHT 178432).

Predicting outcome of patients with prolonged disorders of consciousness using machine learning models based on medical complexity

Patients with severe acquired brain injury and prolonged disorders of consciousness (pDoC) are characterized by high clinical complexity and high risk to develop medical complications. The present multi-center longitudinal study aimed at investigating the impact of medical complications on the prediction of clinical outcome by means of machine learning models. Patients with pDoC were consecutively enrolled at admission in 23 intensive neurorehabilitation units (IRU) and followed-up at 6 months from onset via the Glasgow Outcome Scale-Extended (GOSE). Demographic and clinical data at study entry and medical complications developed within 3 months from admission were collected. Machine learning models were developed, targeting neurological outcomes at 6 months from brain injury using data collected at admission. Then, after concatenating predictions of such models to the medical complications collected within 3 months, a cascade model was developed. One hundred seventy six patients with pDoC (M: 123, median age 60.2 years) were included in the analysis. At admission, the best performing solution (k-Nearest Neighbors regression, KNN) resulted in a median validation error of 0.59 points [IQR 0.14] and a classification accuracy of dichotomized GOS-E of 88.6%. Coherently, at 3 months, the best model resulted in a median validation error of 0.49 points [IQR 0.11] and a classification accuracy of 92.6%. Interpreting the admission KNN showed how the negative effect of older age is strengthened when patients' communication levels are high and ameliorated when no communication is present. The model trained at 3 months showed appropriate adaptation of the admission prediction according to the severity of the developed medical complexity in the first 3 months. In this work, we developed and cross-validated an interpretable decision support tool capable of distinguishing patients which will reach sufficient independence levels at 6 months (GOS-E > 4). Furthermore, we provide an updated prediction at 3 months, keeping in consideration the rehabilitative path and the risen medical complexity.

Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis

OBJECTIVE

To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19.

DESIGN

Two stage individual participant data meta-analysis.

SETTING

Secondary and tertiary care.

PARTICIPANTS

46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021.

DATA SOURCES

Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge.

MODEL SELECTION AND ELIGIBILITY CRITERIA

Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor.

METHODS

Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters.

MAIN OUTCOME MEASURES

30 day mortality or in-hospital mortality.

RESULTS

Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al's model (0.96, 0.59 to 1.55, 0.21 to 4.28).

CONCLUSION

The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Challenges in realising the potential of wastewater-based epidemiology to quantitatively monitor and predict the spread of disease

Researchers around the world have demonstrated correlations between measurements of SARS-CoV-2 RNA in wastewater (WW) and case rates of COVID-19 derived from direct testing of individuals. This has raised concerns that wastewater-based epidemiology (WBE) methods might be used to quantify the spread of this and other diseases, perhaps faster than direct testing, and with less expense and intrusion. We illustrate, using data from Scotland and the USA, the issues regarding the construction of effective predictive models for disease case rates. We discuss the effects of variation in, and the problem of aligning, public health (PH) reporting and WW measurements. We investigate time-varying effects in PH-reported case rates and their relationship to WW measurements. We show the lack of proportionality of WW measurements to case rates with associated spatial heterogeneity. We illustrate how the precision of predictions is affected by the level of aggregation chosen. We determine whether PH or WW measurements are the leading indicators of disease and how they may be used in conjunction to produce predictive models. The prospects of using WW-based predictive models with or without ongoing PH data are discussed.

Integrating landmark modeling framework and machine learning algorithms for dynamic prediction of tuberculosis treatment outcomes

OBJECTIVE

This study aims to establish an informative dynamic prediction model of treatment outcomes using follow-up records of tuberculosis (TB) patients, which can timely detect cases when the current treatment plan may not be effective.

MATERIALS AND METHODS

We used 122 267 follow-up records from 17 958 new cases of pulmonary TB in the Republic of Moldova. A dynamic prediction framework integrating landmark modeling and machine learning algorithms was designed to predict patient outcomes during the course of treatment. Sensitivity and positive predictive value (PPV) were calculated to evaluate performance of the model at critical time points. New measures were defined to determine when follow-up laboratory tests should be conducted to obtain most informative results.

RESULTS

The random-forest algorithm performed better than support vector machine and penalized multinomial logistic regression models for predicting TB treatment outcomes. For all 3 outcome classes (ie, cured, not cured, and died after 24 months following treatment initiation), sensitivity and PPV of prediction models improved as more follow-up information was collected. Specifically, sensitivity and PPV increased from 0.55 to 0.84 and from 0.32 to 0.88, respectively, for the not cured class.

CONCLUSION

The dynamic prediction framework utilizes longitudinal laboratory test results to predict patient outcomes at various landmarks. Sputum culture and smear results are among the important variables for prediction; however, the most recent sputum result is not always the most informative one. This framework can potentially facilitate a more effective treatment monitoring program and provide insights for policymakers toward improved guidelines on follow-up tests.

Evaluation of Prognostic and Predictive Models in the Oncology Clinic

Predictive and prognostic models hold great potential to support clinical decision making in oncology and could ultimately facilitate a paradigm shift to a more personalised form of treatment. While a large number of models relevant to the field of oncology have been developed, few have been translated into clinical use and assessment of clinical utility is not currently considered a routine part of model development. In this narrative review of the clinical evaluation of prediction models in oncology, we propose a high-level process diagram for the life cycle of a clinical model, encompassing model commissioning, clinical implementation and ongoing quality assurance, which aims to bridge the gap between model development and clinical implementation.

Validation of the Trauma and Injury Severity Score for Prediction of Mortality in a Greek Trauma Population

BACKGROUND

Although the Trauma and Injury Severity Score (TRISS) has been extensively used for mortality risk adjustment in trauma, its applicability in contemporary trauma populations is increasingly questioned.

OBJECTIVE

The study aimed to evaluate the predictive performance of the TRISS in its original and revised version and compare these with a recalibrated version, including current data from a Greek trauma population.

METHODS

This is a retrospective cohort study of admitted trauma patients conducted in two tertiary Greek hospitals from January 2016 to December 2018. The model algorithm was calculated based on the Major Trauma Outcome Study coefficients (TRISSMTOS), the National Trauma Data Bank coefficients (TRISSNTDB), and reweighted coefficients of logistic regression obtained from a Greek trauma dataset (TRISSGrTD). The primary endpoint was inhospital mortality. Models' prediction was performed using discrimination and calibration statistics.

RESULTS

A total of 8,988 trauma patients were included, of whom 854 died (9.5%). The TRISSMTOS displayed excellent discrimination with an area under the curve (AUC) of 0.912 (95% CI 0.902-0.923) and comparable with TRISSNTDB (AUC = 0.908, 95% CI 0.897-0.919, p = .1195). Calibration of both models was poor (Hosmer-Lemeshow test p < .001), tending to underestimate the probability of mortality across almost all risk groups. The TRISSGrTD resulted in statistically significant improvement in discrimination (AUC = 0.927, 95% CI 0.918-0.936, p < .0001) and acceptable calibration (Hosmer-Lemeshow test p = .113).

CONCLUSION

In this cohort of Greek trauma patients, the performance of the original TRISS was suboptimal, and there was no evidence that it has benefited from its latest revision. By contrast, a strong case exists for supporting a locally recalibrated version to render the TRISS applicable for mortality prediction and performance benchmarking.

Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review

While the opportunities of ML and AI in healthcare are promising, the growth of complex data-driven prediction models requires careful quality and applicability assessment before they are applied and disseminated in daily practice. This scoping review aimed to identify actionable guidance for those closely involved in AI-based prediction model (AIPM) development, evaluation and implementation including software engineers, data scientists, and healthcare professionals and to identify potential gaps in this guidance. We performed a scoping review of the relevant literature providing guidance or quality criteria regarding the development, evaluation, and implementation of AIPMs using a comprehensive multi-stage screening strategy. PubMed, Web of Science, and the ACM Digital Library were searched, and AI experts were consulted. Topics were extracted from the identified literature and summarized across the six phases at the core of this review: (1) data preparation, (2) AIPM development, (3) AIPM validation, (4) software development, (5) AIPM impact assessment, and (6) AIPM implementation into daily healthcare practice. From 2683 unique hits, 72 relevant guidance documents were identified. Substantial guidance was found for data preparation, AIPM development and AIPM validation (phases 1-3), while later phases clearly have received less attention (software development, impact assessment and implementation) in the scientific literature. The six phases of the AIPM development, evaluation and implementation cycle provide a framework for responsible introduction of AI-based prediction models in healthcare. Additional domain and technology specific research may be necessary and more practical experience with implementing AIPMs is needed to support further guidance.

Rethinking Autism Intervention Science: A Dynamic Perspective

Recent advances in longitudinal methodologies for observational studies have contributed to a better understanding of Autism as a neurodevelopmental condition characterized by within-person and between-person variability over time across behavioral domains. However, this finer-grained approach to the study of developmental variability has yet to be applied to Autism intervention science. The widely adopted experimental designs in the field-randomized control trials and quasi-experimental designs-hold value for inferring treatment effects; at the same time, they are limited in elucidating what works for whom, why, and when, given the idiosyncrasies of neurodevelopmental disorders where predictors and outcomes are often dynamic in nature. This perspective paper aims to serve as a primer for Autism intervention scientists to rethink the way we approach predictors of treatment response and treatment-related change using a dynamic lens. We discuss several empirical gaps, and potential methodological challenges and opportunities pertaining to: (1) capturing finer-grained treatment effects in specific behavioral domains as indexed by micro-level within-person changes during and beyond intervention; and (2) examining and modeling dynamic prediction of treatment response. Addressing these issues can contribute to enhanced study designs and methodologies that generate evidence to inform the development of more personalized interventions and stepped care approaches for individuals on the heterogeneous spectrum of Autism with changing needs across development.

Advanced cardiovascular risk prediction in the emergency department: updating a clinical prediction model - a large database study protocol

BACKGROUND

Patients presenting with chest pain represent a large proportion of attendances to emergency departments. In these patients clinicians often consider the diagnosis of acute myocardial infarction (AMI), the timely recognition and treatment of which is clinically important. Clinical prediction models (CPMs) have been used to enhance early diagnosis of AMI. The Troponin-only Manchester Acute Coronary Syndromes (T-MACS) decision aid is currently in clinical use across Greater Manchester. CPMs have been shown to deteriorate over time through calibration drift. We aim to assess potential calibration drift with T-MACS and compare methods for updating the model.

METHODS

We will use routinely collected electronic data from patients who were treated using TMACS at two large NHS hospitals. This is estimated to include approximately 14,000 patient episodes spanning June 2016 to October 2020. The primary outcome of acute myocardial infarction will be sourced from NHS Digital's admitted patient care dataset. We will assess the calibration drift of the existing model and the benefit of updating the CPM by model recalibration, model extension and dynamic updating. These models will be validated by bootstrapping and one step ahead prequential testing. We will evaluate predictive performance using calibrations plots and c-statistics. We will also examine the reclassification of predicted probability with the updated TMACS model.

DISCUSSION

CPMs are widely used in modern medicine, but are vulnerable to deteriorating calibration over time. Ongoing refinement using routinely collected electronic data will inevitably be more efficient than deriving and validating new models. In this analysis we will seek to exemplify methods for updating CPMs to protect the initial investment of time and effort. If successful, the updating methods could be used to continually refine the algorithm used within TMACS, maintaining or even improving predictive performance over time.

TRIAL REGISTRATION

ISRCTN number: ISRCTN41008456.

Dynamic prediction models improved the risk classification of type 2 diabetes compared with classical static models

OBJECTIVE

Dynamic prediction models use the repeated measurements of predictors to estimate coefficients that link the longitudinal predictors to a static model (i.e. Cox regression). This study aims to develop and validate a dynamic prediction for incident type 2 diabetes (T2DM) as the outcome.

STUDY DESIGN AND SETTING

Data from the Tehran lipid and glucose study was used to develop (n = 5291 individuals; phases 1 to 3) and validate (n = 3147 individuals; phases 3 to 6) the dynamic prediction model among individuals aged ≥ 20 years. We used repeated measurements of fasting plasma glucose (FPG) or waist circumference (WC) in the framework of the joint modeling (JM) of longitudinal and time-to-event analysis.

RESULTS

Compared with the Cox which used just baseline data, JM showed the same discrimination, better calibration, and higher clinical usefulness (i.e. with a net benefit considering both true and false positive decisions); all were shown with repeated measurements of FPG/WC. Additionally, in our study, the dynamic models improve the risk reclassification (net reclassification index 33% for FPG and 24% for WC model).

CONCLUSION

Dynamic prediction models, compared with the static one could yield significant improvements in the prediction of T2DM. The complexity of the dynamic models could be addressed by using decision support systems.

Validation and comparison of 28 risk prediction models for coronary artery disease

AIMS

Risk prediction models (RPMs) for coronary artery disease (CAD), using variables to calculate CAD risk, are potentially valuable tools in prevention strategies. However, their use in the clinical practice is limited by a lack of poor model description, external validation, and head-to-head comparisons.

METHODS AND RESULTS

CAD RPMs were identified through Tufts PACE CPM Registry and a systematic PubMed search. Every RPM was externally validated in the three cohorts (the UK Biobank, LifeLines, and PREVEND studies) for the primary endpoint myocardial infarction (MI) and secondary endpoint CAD, consisting of MI, percutaneous coronary intervention, and coronary artery bypass grafting. Model discrimination (C-index), calibration (intercept and regression slope), and accuracy (Brier score) were assessed and compared head-to-head between RPMs. Linear regression analysis was performed to evaluate predictive factors to estimate calibration ability of an RPM. Eleven articles containing 28 CAD RPMs were included. No single best-performing RPM could be identified across all cohorts and outcomes. Most RPMs yielded fair discrimination ability: mean C-index of RPMs was 0.706 ± 0.049, 0.778 ± 0.097, and 0.729 ± 0.074 (P < 0.01) for prediction of MI in UK Biobank, LifeLines, and PREVEND, respectively. Endpoint incidence in the original development cohorts was identified as a significant predictor for external validation performance.

CONCLUSION

Performance of CAD RPMs was comparable upon validation in three large cohorts, based on which no specific RPM can be recommended for predicting CAD risk.

Big data and predictive modelling for the opioid crisis: existing research and future potential

A need exists to accurately estimate overdose risk and improve understanding of how to deliver treatments and interventions in people with opioid use disorder in a way that reduces such risk. We consider opportunities for predictive analytics and routinely collected administrative data to evaluate how overdose could be reduced among people with opioid use disorder. Specifically, we summarise global trends in opioid use and overdoses; describe the use of big data in research into opioid overdose; consider the potential for predictive modelling, including machine learning, for prevention and monitoring of opioid overdoses; and outline the challenges and risks relating to the use of big data and machine learning in reducing harms that are related to opioid use. Future research for improving the coverage and provision of existing interventions, treatments, and resources for opioid use disorder requires collaboration of multiple agencies. Predictive modelling could transport the concept of stratified medicine to public health through novel methods, such as predictive modelling and emulated trials for evaluating diagnoses and prognoses of opioid use disorder, predicting treatment response, and providing targeted treatment recommendations.