There is no such thing as a validated prediction model

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Development and validation of a lifetime prediction model for incident type 2 diabetes in patients with established cardiovascular disease: the CVD2DM model

AIMS

Identifying patients with established cardiovascular disease (CVD) who are at high risk of type 2 diabetes (T2D) may allow for early interventions, reducing the development of T2D and associated morbidity. The aim of this study was to develop and externally validate the CVD2DM model to estimate the 10-year and lifetime risks of T2D in patients with established CVD.

METHODS AND RESULTS

Sex-specific, competing risk-adjusted Cox proportional hazard models were derived in 19 281 participants with established CVD and without diabetes at baseline from the UK Biobank. The core model's pre-specified predictors were age, current smoking, family history of diabetes mellitus, body mass index, systolic blood pressure, fasting plasma glucose, and HDL cholesterol. The extended model also included HbA1c. The model was externally validated in 3481 patients from the UCC-SMART study. During a median follow-up of 12.2 years (interquartile interval 11.3-13.1), 1628 participants with established CVD were diagnosed with T2D in the UK Biobank. External validation c-statistics were 0.79 [95% confidence interval (CI) 0.76-0.82] for the core model and 0.81 (95% CI 0.78-0.84) for the extended model. Calibration plots showed agreement between predicted and observed 10-year risk of T2D.

CONCLUSION

The 10-year and lifetime risks of T2D can be estimated with the CVD2DM model in patients with established CVD, using readily available clinical predictors. The model would benefit from further validation across diverse ethnic groups to enhance its applicability. Informing patients about their T2D risk could motivate them further to adhere to a healthy lifestyle.

External validation and calibration of risk equations for prediction of diabetic kidney diseases among patients with type 2 diabetes in Taiwan

BACKGROUND

Most existing risk equations for predicting/stratifying individual diabetic kidney disease (DKD) risks were developed using relatively dated data from selective and homogeneous trial populations comprising predominately Caucasian type 2 diabetes (T2D) patients. We seek to adapt risk equations for prediction of DKD progression (microalbuminuria, macroalbuminuria, and renal failure) using empiric data from a real-world population with T2D in Taiwan.

METHODS

Risk equations from three well-known simulation models: UKPDS-OM2, RECODe, and CHIME models, were adapted. Discrimination and calibration were determined using the area under the receiver operating characteristic curve (AUROC), a calibration plot (slope and intercept), and the Greenwood-Nam-D'Agostino (GND) test. Recalibration was performed for unsatisfactory calibration (p-value of GND test < 0.05) by adjusting the baseline hazards of risk equations to address risk variations among patients.

RESULTS

The RECODe equations for microalbuminuria and macroalbuminuria showed moderate discrimination (AUROC: 0.62 and 0.76) but underestimated the event risks (calibration slope > 1). The CHIME equation had the best discrimination for renal failure (AUROCs from CHIME, UKPDS-OM2 and RECODe: 0.77, 0.60 and 0.64, respectively). All three equations overestimated renal failure risk (calibration slope < 1). After rigorous updating, the calibration slope/intercept of the recalibrated RECODe for predicting microalbuminuria (0.87/0.0459) and macroalbuminuria (1.10/0.0004) risks as well as the recalibrated CHIME equation for predicting renal failure risk (0.95/-0.0014) were improved.

CONCLUSIONS

Risk equations for prediction of DKD progression in real-world Taiwanese T2D patients were established, which can be incorporated into a multi-state simulation model to project and differentiate individual DKD risks for supporting timely interventions and health economic research.

Identifying individuals at high risk for dementia in primary care: Development and validation of the DemRisk risk prediction model using routinely collected patient data

INTRODUCTION

Health policy in the UK and globally regarding dementia, emphasises prevention and risk reduction. These goals could be facilitated by automated assessment of dementia risk in primary care using routinely collected patient data. However, existing applicable tools are weak at identifying patients at high risk for dementia. We set out to develop improved risk prediction models deployable in primary care.

METHODS

Electronic health records (EHRs) for patients aged 60-89 from 393 English general practices were extracted from the Clinical Practice Research Datalink (CPRD) GOLD database. 235 and 158 practices respectively were randomly assigned to development and validation cohorts. Separate dementia risk models were developed for patients aged 60-79 (development cohort n = 616,366; validation cohort n = 419,126) and 80-89 (n = 175,131 and n = 118,717). The outcome was incident dementia within 5 years and more than 60 evidence-based risk factors were evaluated. Risk models were developed and validated using multivariable Cox regression.

RESULTS

The age 60-79 development cohort included 10,841 incident cases of dementia (6.3 per 1,000 person-years) and the age 80-89 development cohort included 15,994 (40.2 per 1,000 person-years). Discrimination and calibration for the resulting age 60-79 model were good (Harrell's C 0.78 (95% CI: 0.78 to 0.79); Royston's D 1.74 (1.70 to 1.78); calibration slope 0.98 (0.96 to 1.01)), with 37% of patients in the top 1% of risk scores receiving a dementia diagnosis within 5 years. Fit statistics were lower for the age 80-89 model but dementia incidence was higher and 79% of those in the top 1% of risk scores subsequently developed dementia.

CONCLUSION

Our models can identify individuals at higher risk of dementia using routinely collected information from their primary care record, and outperform an existing EHR-based tool. Discriminative ability was greatest for those aged 60-79, but the model for those aged 80-89 may also be clinical useful.

Predicting blood transfusion following traumatic injury using machine learning models: A systematic review and narrative synthesis

BACKGROUND

Hemorrhage is a leading cause of preventable death in trauma. Accurately predicting a patient's blood transfusion requirement is essential but can be difficult. Machine learning (ML) is a field of artificial intelligence that is emerging within medicine for accurate prediction modeling. This systematic review aimed to identify and evaluate all ML models that predict blood transfusion in trauma.

METHODS

This systematic review was registered on the International Prospective register of Systematic Reviews (CRD4202237110). MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials were systematically searched. Publications reporting an ML model that predicted blood transfusion in injured adult patients were included. Data extraction and risk of bias assessment were performed using validated frameworks. Data were synthesized narratively because of significant heterogeneity.

RESULTS

Twenty-five ML models for blood transfusion prediction in trauma were identified. Models incorporated diverse predictors and varied ML methodologies. Predictive performance was variable, but eight models achieved excellent discrimination (area under the receiver operating characteristic curve, >0.9) and nine models achieved good discrimination (area under the receiver operating characteristic curve, >0.8) in internal validation. Only two models reported measures of calibration. Four models have been externally validated in prospective cohorts: the Bleeding Risk Index, Compensatory Reserve Index, the Marsden model, and the Mina model. All studies were considered at high risk of bias often because of retrospective data sets, small sample size, and lack of external validation.

DISCUSSION

This review identified 25 ML models developed to predict blood transfusion requirement after injury. Seventeen ML models demonstrated good to excellent performance in silico, but only four models were externally validated. To date, ML models demonstrate the potential for early and individualized blood transfusion prediction, but further research is critically required to narrow the gap between ML model development and clinical application.

LEVEL OF EVIDENCE

Systematic Review Without Meta-analysis; Level IV.

A tool to predict the risk of lower extremity amputation in patients starting dialysis

BACKGROUND

Non-traumatic lower extremity amputation (LEA) is a severe complication during dialysis. To inform decision-making for physicians, we developed a multivariable prediction model for LEA after starting dialysis.

METHODS

Data from the Swedish Renal Registry (SNR) between 2010 and 2020 were geographically split into a development and validation cohort. Data from Netherlands Cooperative Study on the Adequacy of Dialysis (NECOSAD) between 1997 and 2009 were used for validation targeted at Dutch patients. Inclusion criteria were no previous LEA and kidney transplant and age ≥40 years at baseline. A Fine-Gray model was developed with LEA within 3 years after starting dialysis as the outcome of interest. Death and kidney transplant were treated as competing events. One coefficient, ordered by expected relevance, per 20 events was estimated. Performance was assessed with calibration and discrimination.

RESULTS

SNR was split into an urban development cohort with 4771 individuals experiencing 201 (4.8%) events and a rural validation cohort with 4.876 individuals experiencing 155 (3.2%) events. NECOSAD contained 1658 individuals experiencing 61 (3.7%) events. Ten predictors were included: female sex, age, diabetes mellitus, peripheral artery disease, cardiovascular disease, congestive heart failure, obesity, albumin, haemoglobin and diabetic retinopathy. In SNR, calibration intercept and slope were -0.003 and 0.912, respectively. The C-index was estimated as 0.813 (0.783-0.843). In NECOSAD, calibration intercept and slope were 0.001 and 1.142 respectively. The C-index was estimated as 0.760 (0.697-0.824). Calibration plots showed good calibration.

CONCLUSION

A newly developed model to predict LEA after starting dialysis showed good discriminatory performance and calibration. By identifying high-risk individuals this model could help select patients for preventive measures.

Generalizability assessment of AI models across hospitals in a low-middle and high income country

The integration of artificial intelligence (AI) into healthcare systems within low-middle income countries (LMICs) has emerged as a central focus for various initiatives aiming to improve healthcare access and delivery quality. In contrast to high-income countries (HICs), which often possess the resources and infrastructure to adopt innovative healthcare technologies, LMICs confront resource limitations such as insufficient funding, outdated infrastructure, limited digital data, and a shortage of technical expertise. Consequently, many algorithms initially trained on data from non-LMIC settings are now being employed in LMIC contexts. However, the effectiveness of these systems in LMICs can be compromised when the unique local contexts and requirements are not adequately considered. In this study, we evaluate the feasibility of utilizing models developed in the United Kingdom (a HIC) within hospitals in Vietnam (a LMIC). Consequently, we present and discuss practical methodologies aimed at improving model performance, emphasizing the critical importance of tailoring solutions to the distinct healthcare systems found in LMICs. Our findings emphasize the necessity for collaborative initiatives and solutions that are sensitive to the local context in order to effectively tackle the healthcare challenges that are unique to these regions.

Machine learning for predicting duration of surgery and length of stay: A literature review on joint arthroplasty

INTRODUCTION

In recent years, different factors such as population aging have caused escalating demand for hip and knee arthroplasty straining already limited hospitals' resources. To address this challenge, focus is put on medical and operational efficiency improvements. This includes an increased use of machine learning (ML) to predict duration of surgery (DOS) and length of stay (LOS) for total knee and total hip arthroplasty, which can be utilized for optimizing resource allocation to satisfy medical and operational limitations. This paper explores the development and performance of ML models in predicting DOS and LOS.

METHODS

A systematic search of publications between 2010-2023 was conducted following PRISMA guidelines. Considering the inclusion and exclusion criteria, 28 out of 722 gathered papers from PubMed, Web of Science, and manual search were included in the study. Descriptive statistics was used to analyze the extracted data regarding data preprocessing, model development, and model performance assessment.

RESULTS

Most of the papers work on LOS as a binary variable. Patient's age was identified as the most frequently used and reported as important variable for predicting DOS and LOS. Investigations also illustrated that within the resulting 28 papers, more than 71% of models reached good to perfect performance based on the area under the receiver operating characteristic curve (AUC), where artificial neural networks and ensemble learning models had the biggest share among the best-performing models.

CONCLUSION

The utilization of ML models is increasing in the literature. The current performance level indicates that ML can potentially turn to powerful tools in predicting DOS and LOS for different purposes. Meanwhile, the literature is not matured yet in reporting real-life application. Future studies can focus on model specification and validation by considering empirical application.

Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review

BACKGROUND

External validation studies create evidence about a clinical prediction rule's (CPR's) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential.

METHODS

A forward citation search of the Framingham Wilson CHD risk rule's derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule.

RESULTS

We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data.

DISCUSSION

Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule's generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.

Prognostic models in multiple sclerosis: progress and challenges in clinical integration

As a chronic inflammatory disease of the central nervous system, multiple sclerosis (MS) is of great individual health and socio-economic significance. To date, there is no prognostic model that is used in routine clinical care to predict the very heterogeneous course of the disease. Despite several research groups working on different prognostic models using traditional statistics, machine learning and/or artificial intelligence approaches, the use of published models in clinical decision making is limited because of poor model performance, lack of transferability and/or lack of validated models. To provide a systematic overview, we conducted a "Cochrane review" that assessed 75 published prediction models using relevant checklists (CHARMS, PROBAST, TRIPOD). We have summarized the relevant points from this analysis here so that the use of prognostic models for therapy decisions in clinical routine can be successful in the future.

Arnicolide D: a multi-targeted anticancer sesquiterpene lactone-preclinical efficacy and mechanistic insights

Arnicolide D, a potent sesquiterpene lactone from Centipeda minima, has emerged as a promising anticancer candidate, demonstrating significant efficacy in inhibiting cancer cell proliferation, inducing apoptosis, and suppressing metastasis across various cancer models. This comprehensive study delves into the molecular underpinnings of Arnicolide D's anticancer actions, emphasizing its impact on key signaling pathways such as PI3K/AKT/mTOR and STAT3, and its role in modulating cell cycle and survival mechanisms. Quantitative data from preclinical studies reveal Arnicolide D's dose-dependent cytotoxicity against cancer cell lines, including nasopharyngeal carcinoma, triple-negative breast cancer, and human colon carcinoma, showcasing its broad-spectrum anticancer potential. Given its multifaceted mechanisms and preclinical efficacy, Arnicolide D warrants further investigation in clinical settings to validate its therapeutic utility against cancer. The evidence presented underscores the need for rigorous pharmacokinetic and toxicological studies to establish safe dosing parameters for future clinical trials.

Planned but ever published? A retrospective analysis of clinical prediction model studies registered on clinicaltrials.gov since 2000

OBJECTIVES

To describe the characteristics and publication outcomes of clinical prediction model studies registered on clinicaltrials.gov since 2000.

STUDY DESIGN AND SETTING

Observational studies registered on clinicaltrials.gov between January 1, 2000, and March 2, 2022, describing the development of a new clinical prediction model or the validation of an existing model for predicting individual-level prognostic or diagnostic risk were analyzed. Eligible clinicaltrials.gov records were classified by modeling study type (development, validation) and the model outcome being predicted (prognostic, diagnostic). Recorded characteristics included study status, sample size information, Medical Subject Headings, and plans to share individual participant data. Publication outcomes were analyzed by linking National Clinical Trial numbers for eligible records with PubMed abstracts.

RESULTS

Nine hundred twenty-eight records were analyzed from a possible 89,896 observational study records. Publications searches found 170 matching peer-reviewed publications for 137 clinicaltrials.gov records. The estimated proportion of records with 1 or more matching publications after accounting for time since study start was 2.8% at 2 years (95% CI: 1.7%, 3.9%), 12.3% at 5 years (9.8% to 14.9%) and 27% at 10 years (23% to 33%). Stratifying records by study start year indicated that publication proportions improved over time. Records tended to prioritize the development of new prediction models over the validation of existing models (76%; 704/928 vs. 24%; 182/928). At the time of download, 27% of records were marked as complete, 35% were still recruiting, and 14.7% had unknown status. Only 7.4% of records stated plans to share individual participant data.

CONCLUSION

Published clinical prediction model studies are only a fraction of overall research efforts, with many studies planned but not completed or published. Improving the uptake of study preregistration and follow-up will increase the visibility of planned research. Introducing additional registry features and guidance may improve the identification of clinical prediction model studies posted to clinical registries.

The PROgnostic ModEl for chronic lung disease (PRO-MEL): development and temporal validation

BACKGROUND

Patients with chronic lung diseases (CLDs), defined as progressive and life-limiting respiratory conditions, experience a heavy symptom burden as the conditions become more advanced, but palliative referral rates are low and late. Prognostic tools can help clinicians identify CLD patients at high risk of deterioration for needs assessments and referral to palliative care. As current prognostic tools may not generalize well across all CLD conditions, we aim to develop and validate a general model to predict one-year mortality in patients presenting with any CLD.

METHODS

A retrospective cohort study of patients with a CLD diagnosis at a public hospital from July 2016 to October 2017 was conducted. The outcome of interest was all-cause mortality within one-year of diagnosis. Potential prognostic factors were identified from reviews of prognostic studies in CLD, and data was extracted from electronic medical records. Missing data was imputed using multiple imputation by chained equations. Logistic regression models were developed using variable selection methods and validated in patients seen from January 2018 to December 2019. Discriminative ability, calibration and clinical usefulness of the model was assessed. Model coefficients and performance were pooled across all imputed datasets and reported.

RESULTS

Of the 1000 patients, 122 (12.2%) died within one year. Patients had chronic obstructive pulmonary disease or emphysema (55%), bronchiectasis (38%), interstitial lung diseases (12%), or multiple diagnoses (6%). The model selected through forward stepwise variable selection had the highest AUC (0.77 (0.72-0.82)) and consisted of ten prognostic factors. The model AUC for the validation cohort was 0.75 (0.70, 0.81), and the calibration intercept and slope were - 0.14 (-0.54, 0.26) and 0.74 (0.53, 0.95) respectively. Classifying patients with a predicted risk of death exceeding 0.30 as high risk, the model would correctly identify 3 out 10 decedents and 9 of 10 survivors.

CONCLUSIONS

We developed and validated a prognostic model for one-year mortality in patients with CLD using routinely available administrative data. The model will support clinicians in identifying patients across various CLD etiologies who are at risk of deterioration for a basic palliative care assessment to identify unmet needs and trigger an early referral to palliative medicine.

TRIAL REGISTRATION

Not applicable (retrospective study).

Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage

BACKGROUND

Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients.

METHODS

We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell's c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I2-statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a "membership model" to obtain a gross estimate of transportability.

RESULTS

Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70-0.78, range 0.52-0.84, I2 = 0.92), the mean intercept was -0.06 (95%CI -0.37-0.24, range -1.40-0.75, I2 = 0.97), and the mean slope was 0.96 (95%CI 0.78-1.13, range 0.53-1.31, I2 = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model.

CONCLUSIONS

We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls.

Contesting the conventional wisdom of periodontal risk assessment

Over the years, several reviews of periodontal risk assessment tools have been published. However, major misunderstandings still prevail in repeated attempts to use these tools for prognostic risk prediction. Here we review the principles of risk prediction and discuss the value and the challenges of using prediction models in periodontology. Most periodontal risk prediction models have not been properly developed according to guidance given for the risk prediction model development. This shortcoming has led to several problems, including the creation of arbitrary risk scores. These scores are often labelled as 'high risk' without explicit boundaries or thresholds for the underlying continuous risk estimates of patient-important outcomes. Moreover, it is apparent that prediction models are often misinterpreted as causal models by clinicians and researchers although they cannot be used as such. Additional challenges like the critical assessment of transportability and applicability of these prediction models, as well as their impact on clinical practice and patient outcomes, are not considered in the literature. Nevertheless, these instruments are promoted with claims regarding their ability to deliver more individualized and precise periodontitis treatment and prevention, purportedly resulting in improved patient outcomes. However, people with or without periodontitis deserve proper information about their risk of developing patient-important outcomes such as tooth loss or pain. The primary objective of disseminating such information should not be to emphasize assumed treatment efficacy, hype individualization of care, or promote business interests. Instead, the focus should be on providing individuals with locally validated and regularly updated predictions of specific risks based on readily accessible and valid key predictors (e.g. age and smoking).

Don't be misled: 3 misconceptions about external validation of clinical prediction models

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Prognosticating the outcome of intensive care in older patients-a narrative review

Prognosis determines major decisions regarding treatment for critically ill patients. Statistical models have been developed to predict the probability of survival and other outcomes of intensive care. Although they were trained on the characteristics of large patient cohorts, they often do not represent very old patients (age ≥ 80 years) appropriately. Moreover, the heterogeneity within this particular group impairs the utility of statistical predictions for informing decision-making in very old individuals. In addition to these methodological problems, the diversity of cultural attitudes, available resources as well as variations of legal and professional norms limit the generalisability of prediction models, especially in patients with complex multi-morbidity and pre-existing functional impairments. Thus, current approaches to prognosticating outcomes in very old patients are imperfect and can generate substantial uncertainty about optimal trajectories of critical care in the individual. This article presents the state of the art and new approaches to predicting outcomes of intensive care for these patients. Special emphasis has been given to the integration of predictions into the decision-making for individual patients. This requires quantification of prognostic uncertainty and a careful alignment of decisions with the preferences of patients, who might prioritise functional outcomes over survival. Since the performance of outcome predictions for the individual patient may improve over time, time-limited trials in intensive care may be an appropriate way to increase the confidence in decisions about life-sustaining treatment.

Predictive models of Alzheimer's disease dementia risk in older adults with mild cognitive impairment: a systematic review and critical appraisal

BACKGROUND

Mild cognitive impairment has received widespread attention as a high-risk population for Alzheimer's disease, and many studies have developed or validated predictive models to assess it. However, the performance of the model development remains unknown.

OBJECTIVE

The objective of this review was to provide an overview of prediction models for the risk of Alzheimer's disease dementia in older adults with mild cognitive impairment.

METHOD

PubMed, EMBASE, Web of Science, and MEDLINE were systematically searched up to October 19, 2023. We included cohort studies in which risk prediction models for Alzheimer's disease dementia in older adults with mild cognitive impairment were developed or validated. The Predictive Model Risk of Bias Assessment Tool (PROBAST) was employed to assess model bias and applicability. Random-effects models combined model AUCs and calculated (approximate) 95% prediction intervals for estimations. Heterogeneity across studies was evaluated using the I2 statistic, and subgroup analyses were conducted to investigate sources of heterogeneity. Additionally, funnel plot analysis was utilized to identify publication bias.

RESULTS

The analysis included 16 studies involving 9290 participants. Frequency analysis of predictors showed that 14 appeared at least twice and more, with age, functional activities questionnaire, and Mini-mental State Examination scores of cognitive functioning being the most common predictors. From the studies, only two models were externally validated. Eleven studies ultimately used machine learning, and four used traditional modelling methods. However, we found that in many of the studies, there were problems with insufficient sample sizes, missing important methodological information, lack of model presentation, and all of the models were rated as having a high or unclear risk of bias. The average AUC of the 15 best-developed predictive models was 0.87 (95% CI: 0.83, 0.90).

DISCUSSION

Most published predictive modelling studies are deficient in rigour, resulting in a high risk of bias. Upcoming research should concentrate on enhancing methodological rigour and conducting external validation of models predicting Alzheimer's disease dementia. We also emphasize the importance of following the scientific method and transparent reporting to improve the accuracy, generalizability and reproducibility of study results.

REGISTRATION

This systematic review was registered in PROSPERO (Registration ID: CRD42023468780).

Artificial intelligence to unlock real-world evidence in clinical oncology: A primer on recent advances

PURPOSE

Real world evidence is crucial to understanding the diffusion of new oncologic therapies, monitoring cancer outcomes, and detecting unexpected toxicities. In practice, real world evidence is challenging to collect rapidly and comprehensively, often requiring expensive and time-consuming manual case-finding and annotation of clinical text. In this Review, we summarise recent developments in the use of artificial intelligence to collect and analyze real world evidence in oncology.

METHODS

We performed a narrative review of the major current trends and recent literature in artificial intelligence applications in oncology.

RESULTS

Artificial intelligence (AI) approaches are increasingly used to efficiently phenotype patients and tumors at large scale. These tools also may provide novel biological insights and improve risk prediction through multimodal integration of radiographic, pathological, and genomic datasets. Custom language processing pipelines and large language models hold great promise for clinical prediction and phenotyping.

CONCLUSIONS

Despite rapid advances, continued progress in computation, generalizability, interpretability, and reliability as well as prospective validation are needed to integrate AI approaches into routine clinical care and real-time monitoring of novel therapies.

Validation of the PediBIRN-7 clinical prediction rule for pediatric abusive head trauma

BACKGROUND

The PediBIRN-7 clinical prediction rule incorporates the (positive or negative) predictive contributions of completed abuse evaluations to estimate abusive head trauma (AHT) probability after abuse evaluation. Applying definitional criteria as proxies for AHT and non-AHT ground truth, it performed with sensitivity 0.73 (95 % CI: 0.66-0.79), specificity 0.87 (95 % CI: 0.82-0.90), and ROC-AUC 0.88 (95 % CI: 0.85-0.92) in its derivation study.

OBJECTIVE

To validate the PediBIRN-7's AHT prediction performance in a novel, equivalent, patient population.

PARTICIPANTS AND SETTINGS

Consecutive, acutely head-injured children <3 years hospitalized for intensive care across eight sites between 2017 and 2020 with completed skeletal surveys and retinal exams (N = 342).

METHODS

Secondary analysis of an existing, cross-sectional, prospective dataset, including assignment of patient-specific estimates of AHT probability, calculation of AHT prediction performance measures (ROC-AUC, sensitivity, specificity, predictive values), and completion of sensitivity analyses to estimate best- and worst-case prediction performances.

RESULTS

Applying the same definitional criteria, the PediBIRN-7 performed with sensitivity 0.74 (95 % CI: 0.66-0.81), specificity 0.77 (95 % CI: 0.70-0.83), and ROC-AUC 0.83 (95 % CI: 0.78-0.88). The reduction in ROC-AUC was statistically insignificant (p = .07). Applying physicians' final consensus diagnoses as proxies for AHT and non-AHT ground truth, the PediBIRN-7 performed with sensitivity 0.73 (95 % CI: 0.66-0.79), specificity 0.87 (95 % CI: 0.82-0.90), and ROC-AUC 0.90 (95 % CI: 0.87-0.94). Sensitivity analyses demonstrated minimal changes in rule performance.

CONCLUSION

The PediBIRN-7's overall AHT prediction performance has been validated in a novel, equivalent, patient population. Its patient-specific estimates of AHT probability can inform physicians' AHT-related diagnostic reasoning after abuse evaluation.

Harnessing multi-source data for individualized care in Hodgkin Lymphoma

Hodgkin lymphoma is a rare, but highly curative form of cancer, primarily afflicting adolescents and young adults. Despite multiple seminal trials over the past twenty years, there is no single consensus-based treatment approach beyond use of multi-agency chemotherapy with curative intent. The use of radiation continues to be debated in early-stage disease, as part of combined modality treatment, as well as in salvage, as an important form of consolidation. While short-term disease outcomes have varied little across these different approaches across both early and advanced stage disease, the potential risk of severe, longer-term risk has varied considerably. Over the past decade novel therapeutics have been employed in the retrieval setting in preparation to and as consolidation after autologous stem cell transplant. More recently, these novel therapeutics have moved to the frontline setting, initially compared to standard-of-care treatment and later in a direct head-to-head comparison combined with multi-agent chemotherapy. In 2018, we established the HoLISTIC Consortium, bringing together disease and methods experts to develop clinical decision models based on individual patient data to guide providers, patients, and caregivers in decision-making. In this review, we detail the steps we followed to create the master database of individual patient data from patients treated over the past 20 years, using principles of data science. We then describe different methodological approaches we are taking to clinical decision making, beginning with clinical prediction tools at the time of diagnosis, to multi-state models, incorporating treatments and their response. Finally, we describe how simulation modeling can be used to estimate risks of late effects, based on cumulative exposure from frontline and salvage treatment. The resultant database and tools employed are dynamic with the expectation that they will be updated as better and more complete information becomes available.

Tutorial: dos and don'ts in clinical prediction research for venous thromboembolism

Clinical prediction modeling has become an increasingly popular domain of venous thromboembolism research in recent years. Prediction models can help healthcare providers make decisions regarding starting or withholding therapeutic interventions, or referrals for further diagnostic workup, and can form a basis for risk stratification in clinical trials. The aim of the current guide is to assist in the practical application of complicated methodological requirements for well-performed prediction research by presenting key dos and don'ts while expanding the understanding of predictive research in general for (clinical) researchers who are not specifically trained in the topic; throughout we will use prognostic venous thromboembolism scores as an exemplar.

Prediction of Morbidity and Mortality After Esophagectomy: A Systematic Review

BACKGROUND

Esophagectomy for esophageal cancer has a complication rate of up to 60%. Prediction models could be helpful to preoperatively estimate which patients are at increased risk of morbidity and mortality. The objective of this study was to determine the best prediction models for morbidity and mortality after esophagectomy and to identify commonalities among the models.

PATIENTS AND METHODS

A systematic review was performed in accordance to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and was prospectively registered in PROSPERO ( https://www.crd.york.ac.uk/prospero/ , study ID CRD42022350846). Pubmed, Embase, and Clarivate Analytics/Web of Science Core Collection were searched for studies published between 2010 and August 2022. The Prediction model Risk of Bias Assessment Tool was used to assess the risk of bias. Extracted data were tabulated and a narrative synthesis was performed.

RESULTS

Of the 15,011 articles identified, 22 studies were included using data from tens of thousands of patients. This systematic review included 33 different models, of which 18 models were newly developed. Many studies showed a high risk of bias. The prognostic accuracy of models differed between 0.51 and 0.85. For most models, variables are readily available. Two models for mortality and one model for pulmonary complications have the potential to be developed further.

CONCLUSIONS

The availability of rigorous prediction models is limited. Several models are promising but need to be further developed. Some models provide information about risk factors for the development of complications. Performance status is a potential modifiable risk factor. None are ready for clinical implementation.

Sustainable deployment of clinical prediction tools-a 360° approach to model maintenance

BACKGROUND

As the enthusiasm for integrating artificial intelligence (AI) into clinical care grows, so has our understanding of the challenges associated with deploying impactful and sustainable clinical AI models. Complex dataset shifts resulting from evolving clinical environments strain the longevity of AI models as predictive accuracy and associated utility deteriorate over time.

OBJECTIVE

Responsible practice thus necessitates the lifecycle of AI models be extended to include ongoing monitoring and maintenance strategies within health system algorithmovigilance programs. We describe a framework encompassing a 360° continuum of preventive, preemptive, responsive, and reactive approaches to address model monitoring and maintenance from critically different angles.

DISCUSSION

We describe the complementary advantages and limitations of these four approaches and highlight the importance of such a coordinated strategy to help ensure the promise of clinical AI is not short-lived.

Comparison of the ADNEX and ROMA risk prediction models for the diagnosis of ovarian cancer: a multicentre external validation in patients who underwent surgery

BACKGROUND

Several diagnostic prediction models to help clinicians discriminate between benign and malignant adnexal masses are available. This study is a head-to-head comparison of the performance of the Assessment of Different NEoplasias in the adneXa (ADNEX) model with that of the Risk of Ovarian Malignancy Algorithm (ROMA).

METHODS

This is a retrospective study based on prospectively included consecutive women with an adnexal tumour scheduled for surgery at five oncology centres and one non-oncology centre in four countries between 2015 and 2019. The reference standard was histology. Model performance for ADNEX and ROMA was evaluated regarding discrimination, calibration, and clinical utility.

RESULTS

The primary analysis included 894 patients, of whom 434 (49%) had a malignant tumour. The area under the receiver operating characteristic curve (AUC) was 0.92 (95% CI 0.88-0.95) for ADNEX with CA125, 0.90 (0.84-0.94) for ADNEX without CA125, and 0.85 (0.80-0.89) for ROMA. ROMA, and to a lesser extent ADNEX, underestimated the risk of malignancy. Clinical utility was highest for ADNEX. ROMA had no clinical utility at decision thresholds <27%.

CONCLUSIONS

ADNEX had better ability to discriminate between benign and malignant adnexal tumours and higher clinical utility than ROMA.

CLINICAL TRIAL REGISTRATION

clinicaltrials.gov NCT01698632 and NCT02847832.

External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan

BACKGROUND

The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings.

METHODS

We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM.

RESULTS

Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed.

CONCLUSIONS

In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.

Modest performance of text mining to extract health outcomes may be almost sufficient for high-quality prognostic model development

BACKGROUND

Across medicine, prognostic models are used to estimate patient risk of certain future health outcomes (e.g., cardiovascular or mortality risk). To develop (or train) prognostic models, historic patient-level training data is needed containing both the predictive factors (i.e., features) and the relevant health outcomes (i.e., labels). Sometimes, when the health outcomes are not recorded in structured data, these are first extracted from textual notes using text mining techniques. Because there exist many studies utilizing text mining to obtain outcome data for prognostic model development, our aim is to study the impact of the text mining quality on downstream prognostic model performance.

METHODS

We conducted a simulation study charting the relationship between text mining quality and prognostic model performance using an illustrative case study about in-hospital mortality prediction in intensive care unit patients. We repeatedly developed and evaluated a prognostic model for in-hospital mortality, using outcome data extracted by multiple text mining models of varying quality.

RESULTS

Interestingly, we found in our case study that a relatively low-quality text mining model (F1 score ≈ 0.50) could already be used to train a prognostic model with quite good discrimination (area under the receiver operating characteristic curve of around 0.80). The calibration of the risks estimated by the prognostic model seemed unreliable across the majority of settings, even when text mining models were of relatively high quality (F1 ≈ 0.80).

DISCUSSION

Developing prognostic models on text-extracted outcomes using imperfect text mining models seems promising. However, it is likely that prognostic models developed using this approach may not produce well-calibrated risk estimates, and require recalibration in (possibly a smaller amount of) manually extracted outcome data.

Computed Tomography Radiomics for Preoperative Prediction of Spread Through Air Spaces in the Early Stage of Surgically Resected Lung Adenocarcinomas

PURPOSE

To assess the added value of radiomics models from preoperative chest CT in predicting the presence of spread through air spaces (STAS) in the early stage of surgically resected lung adenocarcinomas using multiple validation datasets.

MATERIALS AND METHODS

This retrospective study included 550 early-stage surgically resected lung adenocarcinomas in 521 patients, classified into training, test, internal validation, and temporal validation sets (n=211, 90, 91, and 158, respectively). Radiomics features were extracted from the segmented tumors on preoperative chest CT, and a radiomics score (Rad-score) was calculated to predict the presence of STAS. Diagnostic performance of the conventional model and the combined model, based on a combination of conventional and radiomics features, for the diagnosis of the presence of STAS were compared using the area under the curve (AUC) of the receiver operating characteristic curve.

RESULTS

Rad-score was significantly higher in the STAS-positive group compared to the STAS-negative group in the training, test, internal, and temporal validation sets. The performance of the combined model was significantly higher than that of the conventional model in the training set {AUC: 0.784 [95% confidence interval (CI): 0.722-0.846] vs. AUC: 0.815 (95% CI: 0.759-0.872), p=0.042}. In the temporal validation set, the combined model showed a significantly higher AUC than that of the conventional model (p=0.001). The combined model showed a higher AUC than the conventional model in the test and internal validation sets, albeit with no statistical significance.

CONCLUSION

A quantitative CT radiomics model can assist in the non-invasive prediction of the presence of STAS in the early stage of lung adenocarcinomas.

Why do probabilistic clinical models fail to transport between sites

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models.

Updating M6 pregnancy of unknown location risk-prediction model including evaluation of clinical factors

OBJECTIVES

Ectopic pregnancy (EP) is a major high-risk outcome following a pregnancy of unknown location (PUL) classification. Biochemical markers are used to triage PUL as high vs low risk to guide appropriate follow-up. The M6 model is currently the best risk-prediction model. We aimed to update the M6 model and evaluate whether performance can be improved by including clinical factors.

METHODS

This prospective cohort study recruited consecutive PUL between January 2015 and January 2017 at eight units (Phase 1), with two centers continuing recruitment between January 2017 and March 2021 (Phase 2). Serum samples were collected routinely and sent for β-human chorionic gonadotropin (β-hCG) and progesterone measurement. Clinical factors recorded were maternal age, pain score, bleeding score and history of EP. Based on transvaginal ultrasonography and/or biochemical confirmation during follow-up, PUL were classified subsequently as failed PUL (FPUL), intrauterine pregnancy (IUP) or EP (including persistent PUL (PPUL)). The M6 models with (M6P ) and without (M6NP ) progesterone were refitted and extended with clinical factors. Model validation was performed using internal-external cross-validation (IECV) (Phase 1) and temporal external validation (EV) (Phase 2). Missing values were handled using multiple imputation.

RESULTS

Overall, 5473 PUL were recruited over both phases. A total of 709 PUL were excluded because maternal age was < 16 years or initial β-hCG was ≤ 25 IU/L, leaving 4764 (87%) PUL for analysis (2894 in Phase 1 and 1870 in Phase 2). For the refitted M6P model, the area under the receiver-operating-characteristics curve (AUC) for EP/PPUL vs IUP/FPUL was 0.89 for IECV and 0.84-0.88 for EV, with respective sensitivities of 94% and 92-93%. For the refitted M6NP model, the AUCs were 0.85 for IECV and 0.82-0.86 for EV, with respective sensitivities of 92% and 93-94%. Calibration performance was good overall, but with heterogeneity between centers. Net Benefit confirmed clinical utility. The change in AUC when M6P was extended to include maternal age, bleeding score and history of EP was between -0.02 and 0.01, depending on center and phase. The corresponding change in AUC when M6NP was extended was between -0.01 and 0.03. At the 5% threshold to define high risk of EP/PPUL, extending M6P altered sensitivity by -0.02 to -0.01, specificity by 0.03 to 0.04 and Net Benefit by -0.005 to 0.006. Extending M6NP altered sensitivity by -0.03 to -0.01, specificity by 0.05 to 0.07 and Net Benefit by -0.005 to 0.006.

CONCLUSIONS

The updated M6 model offers accurate diagnostic performance, with excellent sensitivity for EP. Adding clinical factors to the model improved performance in some centers, especially when progesterone levels were not suitable or unavailable. © 2023 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Leveraging artificial intelligence for perioperative cancer risk assessment of oral potentially malignant disorders

Oral potentially malignant disorders (OPMDs) are mucosal conditions with an inherent disposition to develop oral squamous cell carcinoma. Surgical management is the most preferred strategy to prevent malignant transformation in OPMDs, and surgical approaches to treatment include conventional scalpel excision, laser surgery, cryotherapy, and photodynamic therapy. However, in reality, since all patients with OPMDs will not develop oral squamous cell carcinoma in their lifetime, there is a need to stratify patients according to their risk of malignant transformation to streamline surgical intervention for patients with the highest risks. Artificial intelligence (AI) has the potential to integrate disparate factors influencing malignant transformation for robust, precise, and personalized cancer risk stratification of OPMD patients than current methods to determine the need for surgical resection, excision, or re-excision. Therefore, this article overviews existing AI models and tools, presents a clinical implementation pathway, and discusses necessary refinements to aid the clinical application of AI-based platforms for cancer risk stratification of OPMDs in surgical practice.

Clinical data mining: challenges, opportunities, and recommendations for translational applications

Clinical data mining of predictive models offers significant advantages for re-evaluating and leveraging large amounts of complex clinical real-world data and experimental comparison data for tasks such as risk stratification, diagnosis, classification, and survival prediction. However, its translational application is still limited. One challenge is that the proposed clinical requirements and data mining are not synchronized. Additionally, the exotic predictions of data mining are difficult to apply directly in local medical institutions. Hence, it is necessary to incisively review the translational application of clinical data mining, providing an analytical workflow for developing and validating prediction models to ensure the scientific validity of analytic workflows in response to clinical questions. This review systematically revisits the purpose, process, and principles of clinical data mining and discusses the key causes contributing to the detachment from practice and the misuse of model verification in developing predictive models for research. Based on this, we propose a niche-targeting framework of four principles: Clinical Contextual, Subgroup-Oriented, Confounder- and False Positive-Controlled (CSCF), to provide guidance for clinical data mining prior to the model's development in clinical settings. Eventually, it is hoped that this review can help guide future research and develop personalized predictive models to achieve the goal of discovering subgroups with varied remedial benefits or risks and ensuring that precision medicine can deliver its full potential.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies

Objectives

To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance.

Design

Systematic review and meta-analysis of external validation studies.

Data sources

Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023.

Eligibility criteria for selecting studies

All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed.

Results

47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125).

Conclusions

The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed.

Systematic review registration

PROSPERO CRD42022373182.

Personalized decision-making for aneurysm treatment of aneurysmal subarachnoid hemorrhage: development and validation of a clinical prediction tool

BACKGROUND

In patients with aneurysmal subarachnoid hemorrhage suitable for endovascular coiling and neurosurgical clip-reconstruction, the aneurysm treatment decision-making process could be improved by considering heterogeneity of treatment effect and durability of treatment. We aimed to develop and validate a tool to predict individualized treatment benefit of endovascular coiling compared to neurosurgical clip-reconstruction.

METHODS

We used randomized data (International Subarachnoid Aneurysm Trial, n = 2143) to develop models to predict 2-month functional outcome and to predict time-to-rebleed-or-retreatment. We modeled for heterogeneity of treatment effect by adding interaction terms of treatment with prespecified predictors and with baseline risk of the outcome. We predicted outcome with both treatments and calculated absolute treatment benefit. We described the patient characteristics of patients with ≥ 5% point difference in the predicted probability of favorable functional outcome (modified Rankin Score 0-2) and of no rebleed or retreatment within 10 years. Model performance was expressed with the c-statistic and calibration plots. We performed bootstrapping and leave-one-cluster-out cross-validation and pooled cluster-specific c-statistics with random effects meta-analysis.

RESULTS

The pooled c-statistics were 0.72 (95% CI: 0.69-0.75) for the prediction of 2-month favorable functional outcome and 0.67 (95% CI: 0.63-0.71) for prediction of no rebleed or retreatment within 10 years. We found no significant interaction between predictors and treatment. The average predicted benefit in favorable functional outcome was 6% (95% CI: 3-10%) in favor of coiling, but 11% (95% CI: 9-13%) for no rebleed or retreatment in favor of clip-reconstruction. 134 patients (6%), young and in favorable clinical condition, had negligible functional outcome benefit of coiling but had a ≥ 5% point benefit of clip-reconstruction in terms of durability of treatment.

CONCLUSIONS

We show that young patients in favorable clinical condition and without extensive vasospasm have a negligible benefit in functional outcome of endovascular coiling - compared to neurosurgical clip-reconstruction - while at the same time having a substantially lower probability of retreatment or rebleeding from neurosurgical clip-reconstruction - compared to endovascular coiling. The SHARP prediction tool ( https://sharpmodels.shinyapps.io/sharpmodels/ ) could support and incentivize a multidisciplinary discussion about aneurysm treatment decision-making by providing individualized treatment benefit estimates.

Radiomics and artificial intelligence for risk stratification of pulmonary nodules: Ready for primetime?

Pulmonary nodules are ubiquitously found on computed tomography (CT) imaging either incidentally or via lung cancer screening and require careful diagnostic evaluation and management to both diagnose malignancy when present and avoid unnecessary biopsy of benign lesions. To engage in this complex decision-making, clinicians must first risk stratify pulmonary nodules to determine what the best course of action should be. Recent developments in imaging technology, computer processing power, and artificial intelligence algorithms have yielded radiomics-based computer-aided diagnosis tools that use CT imaging data including features invisible to the naked human eye to predict pulmonary nodule malignancy risk and are designed to be used as a supplement to routine clinical risk assessment. These tools vary widely in their algorithm construction, internal and external validation populations, intended-use populations, and commercial availability. While several clinical validation studies have been published, robust clinical utility and clinical effectiveness data are not yet currently available. However, there is reason for optimism as ongoing and future studies aim to target this knowledge gap, in the hopes of improving the diagnostic process for patients with pulmonary nodules.

Development and internal validation of a clinical prediction model for serious complications after emergency laparotomy

PURPOSE

Emergency laparotomy (EL) is a common operation with high risk for postoperative complications, thereby requiring accurate risk stratification to manage vulnerable patients optimally. We developed and internally validated a predictive model of serious complications after EL.

METHODS

Data for eleven carefully selected candidate predictors of 30-day postoperative complications (Clavien-Dindo grade >  = 3) were extracted from the HELAS cohort of EL patients in 11 centres in Greece and Cyprus. Logistic regression with Least Absolute Shrinkage and Selection Operator (LASSO) was applied for model development. Discrimination and calibration measures were estimated and clinical utility was explored with decision curve analysis (DCA). Reproducibility and heterogeneity were examined with Bootstrap-based internal validation and Internal-External Cross-Validation. The American College of Surgeons National Surgical Quality Improvement Program's (ACS-NSQIP) model was applied to the same cohort to establish a benchmark for the new model.

RESULTS

From data on 633 eligible patients (175 complication events), the SErious complications After Laparotomy (SEAL) model was developed with 6 predictors (preoperative albumin, blood urea nitrogen, American Society of Anaesthesiology score, sepsis or septic shock, dependent functional status, and ascites). SEAL had good discriminative ability (optimism-corrected c-statistic: 0.80, 95% confidence interval [CI] 0.79-0.81), calibration (optimism-corrected calibration slope: 1.01, 95% CI 0.99-1.03) and overall fit (scaled Brier score: 25.1%, 95% CI 24.1-26.1%). SEAL compared favourably with ACS-NSQIP in all metrics, including DCA across multiple risk thresholds.

CONCLUSION

SEAL is a simple and promising model for individualized risk predictions of serious complications after EL. Future external validations should appraise SEAL's transportability across diverse settings.

Artificial intelligence-based image analysis in clinical testing: lessons from cervical cancer screening

Novel screening and diagnostic tests based on artificial intelligence (AI) image recognition algorithms are proliferating. Some initial reports claim outstanding accuracy followed by disappointing lack of confirmation, including our own early work on cervical screening. This is a presentation of lessons learned, organized as a conceptual step-by-step approach to bridge the gap between the creation of an AI algorithm and clinical efficacy. The first fundamental principle is specifying rigorously what the algorithm is designed to identify and what the test is intended to measure (eg, screening, diagnostic, or prognostic). Second, designing the AI algorithm to minimize the most clinically important errors. For example, many equivocal cervical images cannot yet be labeled because the borderline between cases and controls is blurred. To avoid a misclassified case-control dichotomy, we have isolated the equivocal cases and formally included an intermediate, indeterminate class (severity order of classes: case>indeterminate>control). The third principle is evaluating AI algorithms like any other test, using clinical epidemiologic criteria. Repeatability of the algorithm at the borderline, for indeterminate images, has proven extremely informative. Distinguishing between internal and external validation is also essential. Linking the AI algorithm results to clinical risk estimation is the fourth principle. Absolute risk (not relative) is the critical metric for translating a test result into clinical use. Finally, generating risk-based guidelines for clinical use that match local resources and priorities is the last principle in our approach. We are particularly interested in applications to lower-resource settings to address health disparities. We note that similar principles apply to other domains of AI-based image analysis for medical diagnostic testing.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

Predicting adverse outcomes in adults with a community-acquired lower respiratory tract infection: a protocol for the development and validation of two prediction models for (i) all-cause hospitalisation and mortality and (ii) cardiovascular outcomes

BACKGROUND

Community-acquired lower respiratory tract infections (LRTI) are common in primary care and patients at particular risk of adverse outcomes, e.g., hospitalisation and mortality, are challenging to identify. LRTIs are also linked to an increased incidence of cardiovascular diseases (CVD) following the initial infection, whereas concurrent CVD might negatively impact overall prognosis in LRTI patients. Accurate risk prediction of adverse outcomes in LRTI patients, while considering the interplay with CVD, can aid general practitioners (GP) in the clinical decision-making process, and may allow for early detection of deterioration. This paper therefore presents the design of the development and external validation of two models for predicting individual risk of all-cause hospitalisation or mortality (model 1) and short-term incidence of CVD (model 2) in adults presenting to primary care with LRTI.

METHODS

Both models will be developed using linked routine electronic health records (EHR) data from Dutch primary and secondary care, and the mortality registry. Adults aged ≥ 40 years with a GP-diagnosis of LRTI between 2016 and 2019 are eligible for inclusion. Relevant patient demographics, medical history, medication use, presenting signs and symptoms, and vital and laboratory measurements will be considered as candidate predictors. Outcomes of interest include 30-day all-cause hospitalisation or mortality (model 1) and 90-day CVD (model 2). Multivariable elastic net regression techniques will be used for model development. During the modelling process, the incremental predictive value of CVD for hospitalisation or all-cause mortality (model 1) will also be assessed. The models will be validated through internal-external cross-validation and external validation in an equivalent cohort of primary care LRTI patients.

DISCUSSION

Implementation of currently available prediction models for primary care LRTI patients is hampered by limited assessment of model performance. While considering the role of CVD in LRTI prognosis, we aim to develop and externally validate two models that predict clinically relevant outcomes to aid GPs in clinical decision-making. Challenges that we anticipate include the possibility of low event rates and common problems related to the use of EHR data, such as candidate predictor measurement and missingness, how best to retrieve information from free text fields, and potential misclassification of outcome events.

Machine Learning Models for Blood Glucose Level Prediction in Patients With Diabetes Mellitus: Systematic Review and Network Meta-Analysis

BACKGROUND

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important.

OBJECTIVE

In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity.

METHODS

PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events.

RESULTS

In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia.

CONCLUSIONS

Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced.

TRIAL REGISTRATION

PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Detecting changes in the performance of a clinical machine learning tool over time

BACKGROUND

Excessive use of blood cultures (BCs) in Emergency Departments (EDs) results in low yields and high contamination rates, associated with increased antibiotic use and unnecessary diagnostics. Our team previously developed and validated a machine learning model to predict BC outcomes and enhance diagnostic stewardship. While the model showed promising initial results, concerns over performance drift due to evolving patient demographics, clinical practices, and outcome rates warrant continual monitoring and evaluation of such models.

METHODS

A real-time evaluation of the model's performance was conducted between October 2021 and September 2022. The model was integrated into Amsterdam UMC's Electronic Health Record system, predicting BC outcomes for all adult patients with BC draws in real time. The model's performance was assessed monthly using metrics including the Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores. Statistical Process Control (SPC) charts were used to monitor variation over time.

FINDINGS

Across 3.035 unique adult patient visits, the model achieved an average AUC of 0.78, AUPRC of 0.41, and a Brier score of 0.10 for predicting the outcome of BCs drawn in the ED. While specific population characteristics changed over time, no statistical points outside the statistical control range were detected in the AUC, AUPRC, and Brier scores, indicating stable model performance. The average BC positivity rate during the study period was 13.4%.

INTERPRETATION

Despite significant changes in clinical practice, our BC stewardship tool exhibited stable performance, suggesting its robustness to changing environments. Using SPC charts for various metrics enables simple and effective monitoring of potential performance drift. The assessment of the variation of outcome rates and population changes may guide the specific interventions, such as intercept correction or recalibration, that may be needed to maintain a stable model performance over time. This study suggested no need to recalibrate or correct our BC stewardship tool.

FUNDING

No funding to disclose.