There is no such thing as a validated prediction model

Development and validation of a nomogram to predict the risk of in-hospital MACE for emergence NSTE-ACS: A retrospective multicenter study based on the Chinese population

PURPOSE

Our study aims to develop and validate an effective in-hospital major adverse cardiovascular events(MACE) prediction model for patients with emergency Non-ST elevation acute coronary syndrome(NSTE-ACS).

METHODS

We retrospectively collected NSTE-ACS patients in three tertiary hospitals in Chongqing. In-hospital MACE was the predicted outcome. Patients from one hospital were divided into training set and internal validation set according to the ratio of 7:3. Besides, 662 patients from two other tertiary hospitals were for external validation. Patient information including demographics, laboratory tests results and disease course records were for comprehensive analysis. Finally, LASSO were used to identify the predictors and develop the model. This model was subsequently visualized as a nomogram, followed by both internal and external validations.The receiver operating characteristic curve, calibration curve and clinical decision curve analysis were used to assess the model's discrimination, calibration and clinical applicability, respectively.

RESULTS

A total of 3,308 patients were included, 375 of whom developed in-hospital MACE. The LR model demonstrated that length of stay, neutrophils, myoglobin, NYHA, CCI, NT-proBNP, LVEF and respiratory failure were risk factors for in-hospital MACE in emergence NSTE-ACS patients. In the training set, the AUC was 0.860 (95%CI:0.831-0.889). In external validation,the AUC was 0.855(95%CI:0.808-0.902), and both the calibration curve and DCA in validation set also revealed stable predictive accuracy and clinical validity.Additionally,it is available to calculate the MACE risk online via the web page (https://cocozhou99.shinyapps.io/DynNomapp/).

CONCLUSION

The prediction model we constructed has good predictive performance and can help healthcare professionals accurately assess the risk of in-hospital MACE in emergence NSTE-ACS patients.

Personalized surveillance in colorectal cancer: Integrating circulating tumor DNA and artificial intelligence into post-treatment follow-up

Given the growing burden of colorectal cancer (CRC) as a global health challenge, it becomes imperative to focus on strategies that can mitigate its impact. Post-treatment surveillance has emerged as essential for early detection of recurrence, significantly improving patient outcomes. However, intensive surveillance strategies have shown mixed results compared to less intensive methods, emphasizing the necessity for personalized, risk-adapted approaches. The observed suboptimal adherence to existing surveillance protocols underscores the urgent need for more tailored and efficient strategies. In this context, circulating tumor DNA (ctDNA) emerges as a promising biomarker with significant potential to revolutionize post-treatment surveillance, demonstrating high specificity [0.95, 95% confidence interval (CI): 0.91-0.97] and robust diagnostic odds (37.6, 95%CI: 20.8-68.0) for recurrence detection. Furthermore, artificial intelligence and machine learning models integrating patient-specific and tumor features can enhance risk stratification and optimize surveillance strategies. The reported area under the receiver operating characteristic curve, measuring artificial intelligence model performance in predicting CRC recurrence, ranged from 0.581 and 0.593 at the lowest to 0.979 and 0.978 at the highest in training and validation cohorts, respectively. Despite this promise, addressing cost, accessibility, and extensive validation remains crucial for equitable integration into clinical practice.

External validation of SCORE2-Diabetes in The Netherlands across various socioeconomic levels in native-Dutch and non-Dutch populations

AIMS

Adults with type 2 diabetes have an increased risk of cardiovascular events (CVEs), the world's leading cause of mortality. The SCORE2-Diabetes model is a tool designed to estimate the 10-year risk of CVE specifically in individuals with type 2 diabetes. However, the performance of such models may vary across different demographic and socioeconomic groups, necessitating validation and assessment in diverse populations. This study aims to externally validate SCORE2-Diabetes and assess its performance across various socioeconomic and migration origins in The Netherlands.

METHODS AND RESULTS

We selected adults with type 2 diabetes, aged 40-79 years and without previous CVE from the Extramural LUMC Academic Network (ELAN) primary care data cohort from 2007 to 2023. ELAN data were linked with Statistics Netherlands registry data to obtain information about the country of origin and socioeconomic status (SES). Cardiovascular event was defined as myocardial infarction, stroke, or CV mortality. Non-CV mortality was considered a competing event. Analyses were stratified by sex, Dutch vs. other non-Dutch countries of origin, and quintiles of SES. Of the 26 544 included adults with type 2 diabetes, 2518 developed CVE. SCORE2-Diabetes showed strong predictive accuracy for CVE in the Dutch population [observed-to-expected ratio (OE) = 1.000, 95% CI = 0.990-1.008 for men, and OE = 1.050, 95% CI = 1.042-1.057 for women]. For non-Dutch individuals, the model underestimated CVE risk (OE = 1.121, 95% CI = 1.108-1.131 for men, and OE = 1.100, 95% CI = 1.092-1.111 for women). The model also underestimated the CVE risk (OE > 1) in low SES groups and overestimated the risk (OE < 1) in high SES groups. Discrimination was moderate across subgroups with c-indices between 0.6 and 0.7.

CONCLUSION

SCORE2-Diabetes accurately predicted the risk of CVE in the Dutch population. However, it underpredicted the risk of CVE in the low SES groups and non-Dutch origins, while overpredicting the risk in high SES men and women. Additional clinical judgment must be considered when using SCORE2-Diabetes for different SES and countries of origin.

LAY SUMMARY

A new study validates the SCORE2-Diabetes model for predicting a 10-year risk of cardiovascular events in type 2 diabetes. Strong accuracy for the Dutch population, but underestimation of the risk for low SES and non-Dutch groups. SCORE2-Diabetes should be used with extra caution across diverse subgroups.

The comparative performance of models predicting patient and graft survival after kidney transplantation: A systematic review

BACKGROUND

Cox proportional hazard models have long been the model of choice for survival prediction after kidney transplantation. In recent years, a variety of novel model types have been proposed. We investigate the prediction performance across different model types, including machine learning models and traditional model types.

METHODS

A systematic review was conducted following PROBAST and CHARMS, also considering extensions to TRIPOD+AI and PROBAST+AI, for data collection and risk of bias assessment. The review only included publications that reported on prediction performance for models of different types. A comparative analysis tested performance differences between the model types.

RESULTS

The review included 37 publications which presented 134 comparative studies. The designs of many studies left room for improvement and most studies had high risk of bias. The collected data admitted testing of performance differences for 22 pairs of model types, ten of which yielded significant differences. Support Vector Machines and Logistic Regression were never found to outperform other model types. Other comparisons, however, provide inconclusive comparative performance results and none of the model types performed consistently and significantly better than alternatives.

CONCLUSIONS

Rigorous review of current evidence and comparative performance evidence finds no significant kidney transplant survival prediction performance differences that Cox Proportional Hazard models are being outperformed. The design of many of the studies implies high risk of bias and more and better designed studies which reutilize best performing models are needed. This enables to resolve model biases, reporting issues, and to increase the power of comparative performance analysis.

Clinical prediction models to diagnose neonatal sepsis in low-income and middle-income countries: a scoping review

INTRODUCTION

Neonatal sepsis causes significant morbidity and mortality worldwide but is difficult to diagnose clinically. Clinical prediction models (CPMs) could improve diagnostic accuracy, facilitating earlier treatment for cases and avoiding antibiotic overuse. Neonates in low-income and middle-income countries (LMICs) are disproportionately affected by sepsis, yet no review has comprehensively synthesised evidence for CPMs validated in this setting.

METHODS

We performed a scoping review of CPMs to diagnose neonatal sepsis using Ovid MEDLINE, Ovid Embase, Scopus, Web of Science, Global Index Medicus and the Cochrane Library. The most recent searches were performed on 16 June 2024. We included studies published in English or Spanish that validated a new or existing CPM for neonatal sepsis in any healthcare setting in an LMIC. Studies were excluded if they validated a prognostic model or where data for neonates could not be separated from a larger paediatric population. Studies were selected by two independent reviewers and summarised by narrative synthesis.

RESULTS

From 4598 unique records, we included 82 studies validating 44 distinct models in 24 252 neonates. Most studies were set in neonatal intensive or special care units (n=64, 78%) in middle-income countries (n=81, 99%) and included neonates already suspected of sepsis (n=58, 71%). Only four studies (5%) were set in the WHO African region, and only one study included data from a low-income country. Two-thirds of CPMs (n=30) required laboratory parameters, and three-quarters (n=34) were only validated in one study.

CONCLUSION

Our review highlights several literature gaps, particularly a paucity of studies validating models in the lowest-income countries where neonatal sepsis is most prevalent, and models for the undifferentiated neonatal population that do not rely on laboratory tests. Furthermore, heterogeneity in study populations, definitions of sepsis and reporting of models inhibits meaningful comparison between studies and may hinder progress towards useful diagnostic tools.

Predicting prolonged work absence due to musculoskeletal disorders: development, validation, and clinical usefulness of prognostic prediction models

PURPOSE

Given the lack of robust prognostic models for early identification of individuals at risk of work disability, this study aimed to develop and externally validate three models for prolonged work absence among individuals on sick leave due to musculoskeletal disorders.

METHODS

We developed three multivariable logistic regression models using data from 934 individuals on sick leave for 4-12 weeks due to musculoskeletal disorders, recruited through the Norwegian Labour and Welfare Administration. The models predicted three outcomes: (1) > 90 consecutive sick days, (2) > 180 consecutive sick days, and (3) any new or increased work assessment allowance or disability pension within 12 months. Each model was externally validated in a separate cohort of participants (8-12 weeks of sick leave) from a different geographical region in Norway. We evaluated model performance using discrimination (c-statistic), calibration, and assessed clinical usefulness using decision curve analysis (net benefit). Bootstrapping was used to adjust for overoptimism.

RESULTS

All three models showed good predictive performance in the external validation sample, with c-statistics exceeding 0.76. The model predicting > 180 days performed best, demonstrating good calibration and discrimination (c-statistic 0.79 (95% CI 0.73-0.85), and providing net benefit across a range of decision thresholds from 0.10 to 0.80.

CONCLUSIONS

These models, particularly the one predicting > 180 days, may facilitate secondary prevention strategies and guide future clinical trials. Further validation and refinement are necessary to optimise the models and to test their performance in larger samples.

A methodological systematic review of validation and performance of sepsis real-time prediction models

Sepsis real-time prediction models (SRPMs) provide timely alerts and may improve patient outcomes but face limited clinical adoption due to inconsistent validation methods and potential biases. Comprehensive evaluation, including external full-window validation with model- and outcome-level metrics, is crucial for real-world effectiveness, yet performance evidence remains scarce. This study systematically reviewed SRPM performance across validation methods, analyzing 91 studies from multiple databases. Only 54.9% applied full-window validation with both metric types. Performance decreased under external and full-window validation, with median AUROCs of 0.886 and 0.861 at 6- and 12-hours pre-onset, dropping to 0.783 in full-window external validation. Median Utility Scores declined from 0.381 in internal to -0.164 in external validation. Combining AUROC and Utility Score identified top-performing SRPMs in 18.7% of studies. Hand-crafted features significantly improved performance. Future research should focus on multi-center datasets, hand-crafted features, multi-metric full-window validation, and prospective trials to support clinical implementation.

Clinical prognostic models for sarcomas: a systematic review and critical appraisal of development and validation studies

BACKGROUND

Current clinical guidelines recommend the use of clinical prognostic models (CPMs) for therapeutic decision-making in sarcoma patients. However, the number and quality of developed and externally validated CPMs is unknown. Therefore, we aimed to describe and critically assess CPMs for sarcomas.

METHODS

We performed a systematic review including all studies describing the development and/or external validation of a CPM for sarcomas. We searched the databases MEDLINE, EMBASE, Cochrane Central, and Scopus from inception until June 7th, 2022. The risk of bias was assessed using the prediction model risk of bias assessment tool (PROBAST).

RESULTS

Seven thousand six hundred fifty-six records were screened, of which 145 studies were eventually included, developing 182 and externally validating 59 CPMs. The most frequently modeled type of sarcoma was osteosarcoma (43/182; 23.6%), and the most frequently predicted outcome was overall survival (81/182; 44.5%). The most used predictors were the patient's age (133/182; 73.1%) and tumor size (116/182; 63.7%). Univariable screening was used in 137 (75.3%) CPMs, and only 7 (3.9%) CPMs were developed using pre-specified predictors based on clinical knowledge or literature. The median c-statistic on the development dataset was 0.74 (interquartile range [IQR] 0.71, 0.78). Calibration was reported for 142 CPMs (142/182; 78.0%). The median c-statistic of external validations was 0.72 (IQR 0.68-0.75). Calibration was reported for 46 out of 59 (78.0%) externally validated CPMs. We found 169 out of 241 (70.1%) CPMs to be at high risk of bias, mostly due to the high risk of bias in the analysis domain.

DISCUSSION

While various CPMs for sarcomas have been developed, the clinical utility of most of them is hindered by a high risk of bias and limited external validation. Future research should prioritise validating and updating existing well-developed CPMs over developing new ones to ensure reliable prognostic tools.

TRIAL REGISTRATION

PROSPERO CRD42022335222.

Challenges in making an evidence-based prognosis

Prognosis is one of three traditional roles of clinicians, along with diagnosis and therapy. Prognostication-predicting and communicating about what to expect-plays a major, if overlooked, role in the day-to-day practice of both obstetricians and neonatologists. This article describes several challenges in formulating an evidence-based prognosis that practicing clinicians may find helpful to consider in their practice.

Deep learning for electrocardiogram interpretation: Bench to bedside

BACKGROUND

Recent advancements in deep learning (DL), a subset of artificial intelligence, have shown the potential to automate and improve disease recognition, phenotyping and prediction of disease onset and outcomes by analysing various sources of medical data. The electrocardiogram (ECG) is a valuable tool for diagnosing and monitoring cardiovascular conditions.

METHODS

The implementation of DL in ECG analysis has been used to detect and predict rhythm abnormalities and conduction abnormalities, ischemic and structural heart diseases, with performance comparable to physicians. However, despite promising development of DL algorithms for automatic ECG analysis, the integration of DL-based ECG analysis and deployment of medical devices incorporating these algorithms into routine clinical practice remains limited.

RESULTS

This narrative review highlights the applications of DL in 12-lead ECG analysis. Furthermore, we review randomized controlled trials that assess the clinical effectiveness of these DL tools. Finally, it addresses different key barriers to widespread implementation in clinical practice, including regulatory hurdles, algorithm transparency and data privacy concerns.

CONCLUSIONS

By outlining both the progress and the obstacles in this field, this review aims to provide insights into how DL could shape the future of ECG analysis and enhance cardiovascular care in daily clinical practice.

Clinical prediction models for in vitro fertilization outcomes: a systematic review, meta-analysis, and external validation

STUDY QUESTION

What is the best-performing model currently predicting live birth outcomes for IVF or ICSI?

SUMMARY ANSWER

Among the identified prognostic models, McLernon's post-treatment model outperforms other models in both the meta-analysis and external validation of a Chinese cohort.

WHAT IS KNOWN ALREADY

With numerous similar models available across different time periods and using various predictors in IVF prognostic models, there is a need to summarize and evaluate them, due to a lack of validated evidence distinguishing high-quality from low-quality prediction tools. However, there is a notable dearth of research in the form of meta-analysis or external validation assessing the performance of models in predicting live births in this field.

STUDY DESIGN, SIZE, DURATION

The researchers conducted a comprehensive literature review in PubMed, EMBASE, and Web of Science, using keywords related to prognostic models and IVF/ICSI live birth outcomes. The search included studies published up to 3 April 2024, and was limited to English language studies.

PARTICIPANTS/MATERIALS, SETTING, METHODS

The review included studies that developed or validated prognostic models for IVF live birth outcomes while providing clear reports on model characteristics. Researchers extracted and analysed the data in accordance with the guidelines outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses and other model-related guidelines. For model effects in meta-analysis, the choice would be based on the heterogeneity assessed using the I2 statistic and the Cochrane Q test. Model performance was evaluated by assessing their area under the receiver operating characteristic curves (AUCs) and calibration plots in the studies.

MAIN RESULTS AND THE ROLE OF CHANCE

This review provides a comprehensive summary of data derived from 72 studies with an overall ROB of high or unclear. These studies contained a total of 132 predictors and 86 prognostic models, and then meta-analyses were performed for each of the five selected models. The total random effects of Templeton's, Nelson's, McLernon's pre-treatment and post-treatment model demonstrated AUCs of 0.65 (95% CI: 0.61-0.69), 0.63 (95% CI: 0.63-0.64), 0.67 (95% CI: 0.62-0.71), and 0.73 (95% CI: 0.71-0.75), respectively. The total fixed effects of the intelligent data analysis score (iDAScore) model estimated an AUC of 0.66 (95% CI: 0.63-0.68). The external validation of the initial four models in our cohort produced AUCs ranging from 0.53 to 0.58, and the calibration was confirmed through calibration plots.

LIMITATIONS, REASONS FOR CAUTION

While the focus on English-language studies and live birth outcomes may constrain the generalizability of the findings to diverse populations, this approach equips clinicians, who view live births as the ultimate objective, with more precise and actionable reference guidelines.

WIDER IMPLICATIONS OF THE FINDINGS

This study represents the first meta-analysis in the field of IVF prediction models, definitively confirming the superior performance of McLernon's post-treatment model. The conclusion is reinforced by independent validation from another perspective. Nevertheless, further investigation is warranted to develop new models and to externally validate existing high-performing models for prognostic accuracy in IVF outcomes.

STUDY FUNDING/COMPETING INTEREST(S)

This study was supported by the National Natural Science Foundation of China (Grant No. 82174517). The authors report no conflict of interest.

REGISTRATION NUMBER

2022 CRD42022312018.

Prediction model of the T cell-mediated rejection after liver transplantation in children and adults: A case-controlled study

OBJECTIVE

T cell-mediated rejection (TCMR) is a major concern following liver transplantation (LT), and identifying its predictors could help improve post-transplant prognosis. This study aimed to develop a model to predict the risk of TCMR in children and adults after LT.

METHOD

Pre-transplant demographic characteristics, intraoperative parameters, and especially early post-transplant laboratory data for 1221 LT recipients (1096 adults and 125 children) were obtained from the Hospital, University, between 1 January 2015, and 1 January 2022. These data were analyzed to develop the prediction model.

RESULT

The incidence of TCMR was higher in pediatric LT recipients than in adults (17.6% vs. 6.4%, P < 0.001). In adult recipients, seven predictors were identified: donor sex, recipient age, recipient height, and post-transplant levels of serum direct bilirubin, urea, platelets, and neutrophil-to-lymphocyte ratio. In pediatric recipients, four predictors were identified: post-transplant levels of serum monocyte percentage, direct bilirubin, albumin, and gamma-glutamyl transferase. The area under the model's curve incorporating these variables for predicting TCMR after LT was 0.713 (95% confidence interval, CI: 0.655-0.770) in adults and 0.786 (95% CI: 0.675-0.896) in children. Decision curve analyses demonstrated the clinical significance of the model.

CONCLUSION

This study developed a prediction model that may be useful in identifying high-TCMR-risk populations in both adult and pediatric LT recipients.

External validation of a model to predict recurrence-free and melanoma-specific survival for patients with melanoma after sentinel node biopsy

BACKGROUND

Recently, a model to predict 5-year recurrence-free survival (RFS) and melanoma-specific survival (MSS) after sentinel lymph node biopsy (SLNB) was published. The aim of this study was to validate that model in a large independent international cohort.

METHODS

The database of the Sentinel Lymph Node Working Group (SLNWG) was analysed for patients with malignant melanoma who underwent SLNB. Patients with clinical stage III melanoma, a history of other malignancies, or receiving concomitant systemic therapies during follow-up were excluded. The model's predictive performance was evaluated using discrimination and calibration metrics in the eligible cohort. Decision curve analysis was performed to assess the clinical value of the model.

RESULTS

The external validation cohort consisted of 6174 patients of the SLNWG from the USA, Europe, and Israel. A positive sentinel node was found in 788 patients (12.8%). The area under the time-dependent receiver operating characteristic (ROC) curve of the external validation was 0.76 (95% c.i. 0.74 to 0.77) for RFS and 0.79 (95% c.i. 0.76 to 0.81) for MSS. The model was well calibrated, as the observed 5-year survival rates aligned closely with the predicted survival rates (calibration slope of 0.98 for RFS and calibration slope of 0.99 for MSS). The model provided a net benefit versus the 'treat all' and 'treat none' strategies at the predetermined probability threshold for recurrence of 45%.

CONCLUSION

The model demonstrated good performance in a large heterogeneous independent cohort, emphasizing its robustness. Decision curve analysis revealed a clear net benefit of the model over a treat all strategy, highlighting its potential for clinical use.

Dynamic Updating of Psychosis Prediction Models in Individuals at Ultra-High Risk of Psychosis

BACKGROUND

The performance of psychiatric risk calculators can deteriorate over time due to changes in patient population, referral pathways, and medical advances. Such temporal biases in existing models may lead to suboptimal decisions when translated into clinical practice. Methods are available to correct this bias, but no research has been conducted to investigate their utility in psychiatry.

METHODS

We aimed to analyze the performance of model updating methods for predicting psychosis onset by 1 year in 780 individuals at ultra-high risk (UHR) of psychosis from the UHR 1000+ cohort, a longitudinal cohort of UHR individuals recruited to research studies at Orygen, Melbourne, Australia, between 1995 and 2020. Model updating was performed using a yearly adjusted model (recalibration), a continuously updated model (refitting), and a continuous Bayesian updating model (dynamic updating) and compared with a static logistic regression prediction model (original) regarding calibration, discrimination, and clinical net benefit.

RESULTS

The original model was poorly calibrated over the entire validation period. All 3 updating methods improved the predictive performance compared with the original model (recalibration: p = .009; refitting: p = .020; dynamic updating: p = .001). The dynamic updating method demonstrated the best predictive performance (Harrell's C-index = 0.71; 95% CI, 0.60 to 0.82), calibration slope (slope = 1.12; 95% CI, 0.46 to 1.87), and clinical net benefit over the entire validation period.

CONCLUSIONS

Dynamic updating of psychosis prediction models may help to mitigate decreases in performance over time. Therefore, existing psychosis prediction models need to be monitored for temporal biases to mitigate potentially harmful decisions.

Diabetes Screening in the Emergency Department: Development of a Predictive Model for Elevated Hemoglobin A1c

Aims: We developed a prediction model for elevated hemoglobin A1c (HbA1c) among patients presenting to the emergency department (ED) at risk for diabetes to identify important factors that may influence follow-up patient care. Methods: Retrospective electronic health records data among patients screened for diabetes at the ED in May 2021 was used. The primary outcome was elevated HbA1c (≥ 5.7%). The data was divided into a derivation set (80%) and a test set (20%) stratified by elevated HbA1c. In the derivation set, we estimated the optimal significance level for backward elimination using a 10-fold cross-validation method. A final model was derived using the entire derivation set and validated on the test set. Performance statistics included C-statistic, sensitivity, specificity, predictive values, Hosmer-Lemeshow test, and Brier score. Results: There were 590 ED patients screened for diabetes in May 2021. The final model included nine variables: age, race/ethnicity, insurance, chief complaints of back pain and fever/chills, and a past medical history of obesity, hyperlipidemia, chronic obstructive pulmonary disease, and substance misuse. Adequate model discrimination (C-statistic = 0.75; sensitivity, specificity, and predictive values > 0.70), no evidence of model ill fit (Hosmer-Lemeshow test = 0.29), and moderate Brier score (0.21) suggest acceptable model performance. Conclusion: In addition to age, obesity, and hyperlipidemia, a history of substance misuse was identified as an important predictor of elevated HbA1c levels among patients screened for diabetes in the ED. Our findings suggest that substance misuse may be an important factor to consider when facilitating follow-up care for patients identified with prediabetes or diabetes in the ED and warrants further investigation. Future research efforts should also include external validation in larger samples of ED patients.

Declining Clinical Utility of Tools for Predicting Sentinel Lymph Node Biopsy Status: A Single Institution Experience from 2000 to 2021

INTRODUCTION

Clinicopathologic data-based sentinel lymph node (SLN) prediction models are used to select patients with melanoma for sentinel lymph node biopsy (SLNB). However, the temporal performance of these models is unknown. Therefore, we investigated whether the performance and clinical utility of the Melanoma Institute of Australia, Memorial Sloan Kettering Cancer Center, and Friedman et al. models changed over time.

PATIENTS AND METHODS

Primary cutaneous melanoma cases that underwent SLNB at a single tertiary-care cancer center from 2000 to 2021 were identified from a prospectively maintained database. Calibration plots were generated. Values for estimated risks of SLN positivity and area under the receiver operator curve (AUC) were calculated. Clinical utility was assessed at thresholds between 5 and 10% using decision curve analysis.

RESULTS

In total, 2977 SLNB cases were included. The estimated risk of SLN positivity and AUCs were similar across periods for all models. However, calibration decreased over time for all models, with progressive underprediction of SLN positivity. Clinical utility also declined over time; in the most recent period investigated (2018-2021), no model offered clinical utility at risk thresholds ≤ 8%, and only the Friedman model provided clinical utility at risk thresholds of 9-10%.

CONCLUSIONS

The calibration and clinical utility of three predominant models for SLN prediction declined over time. There is a need to periodically reassess the performance of SLN prognostic tools as they are applied to contemporary cohorts. Future studies are needed to determine whether findings are generalizable outside of this study cohort.

Development and validation of a prediction model for serious infections in rheumatoid arthritis patients treated with tocilizumab in Japan

OBJECTIVES

To develop a prediction model for serious infections (SIs) in rheumatoid arthritis (RA) patients treated with tocilizumab in Japan and to evaluate the model's performance compared to previously developed models, i.e., 'DANBIO' and 'postmarketing surveillance' (PMS).

METHOD

This non-interventional retrospective cohort study utilized the Medical Data Vision database in Japan. The study population was derived from patients ≥ 18 years with RA who initiated tocilizumab between April 2008 and July 2021. SIs were assessed during the 1-year follow-up from tocilizumab initiation. The candidate predictors were identified based on previous studies, known risk factors, potentially relevant factors, and data availability. The prediction model was developed using logistic regression. The model's performance was compared with previously developed models using cross-entropy and area under the receiver operating characteristic curve (AUC).

RESULTS

Of the 6501 RA patients, 4.57% experienced SIs during the 1-year follow-up. The model included 17 predictors for SI (e.g., age (odds ratio 1.013 (95% confidence interval 1.002-1.024)), history of SIs (2.569 (1.636-3.745)), diverticulitis (2.183 (1.000-3.989))). The model showed a lower cross-entropy and a higher AUC (0.1488; 0.712) compared to DANBIO (0.1932; 0.591) and PMS (0.1561; 0.565) models, and the sensitivity, specificity, positive predictive value, and negative predictive value using 5% threshold were 72%, 64%, 7%, and 98%, respectively.

CONCLUSIONS

The model developed in this study seems to have the potential to inform the risk of SIs in RA patients treated with tocilizumab and may help the early identification of patients at risk of SIs to reduce morbidity and mortality.

A multimodal deep learning model for cervical pre-cancers and cancers prediction: Development and internal validation study

BACKGROUND

The current cervical cancer screening and diagnosis have limitations due to their subjectivity and lack of reproducibility. We describe the development of a deep learning (DL)-based diagnostic risk prediction model and evaluate its potential for clinical impact.

METHOD

We developed and internally validated a DL model which accommodates both clinical data and colposcopy images in predicting the patients CIN2+ status using a retrospective cohort of 6356 cases of LEEP-conization/cone-biopsy (gold-standard diagnosis) following an abnormal screening result. The overall performance, discrimination, and calibration of the model were compared to expert clinician's colposcopic impression. The potential for clinical impact was assessed with rate of unnecessary conizations that could be avoided by using our model.

RESULTS

The model combining clinical history and colposcopy images demonstrated superior performance prediction of CIN2+(AUC-ROC = 95.3 %, accuracy = 90.8 %, PPV = 94.1 %, NPV = 87.9 %) and better calibration compared to models that used image or clinical history data alone and outperformed clinician's colposcopic impressions. Moreover, if a decision threshold of 10 % is applied to the predicted probability from this model to recommend conization, up to 35 % of conizations could be avoided without missing any true CIN2+ cases.

CONCLUSION

We present a novel DL model to predict cervical neoplasia with potential for reducing unnecessary conization. External validation studies are warranted for assessing generalizability.

Instability of the AUROC of Clinical Prediction Models

BACKGROUND

External validations are essential to assess the performance of a clinical prediction model (CPM) before deployment. Apart from model misspecification, also differences in patient population, the standard of care, predictor definitions, and other factors influence a model's discriminative ability, as commonly quantified by the AUC (or c-statistic). We aimed to quantify the variation in AUCs across sets of external validation studies and propose ways to adjust expectations of a model's performance in a new setting.

METHODS

The Tufts-PACE CPM Registry holds a collection of CPMs for prognosis in cardiovascular disease. We analyzed the AUC estimates of 469 CPMs with at least one external validation. Combined, these CPMs had a total of 1603 external validations reported in the literature. For each CPM and its associated set of validation studies, we performed a random-effects meta-analysis to estimate the between-study standard deviationτ $$ \tau $$ among the AUCs. Since the majority of these meta-analyses have only a handful of validations, this leads to very poor estimates ofτ $$ \tau $$ . So, instead of focusing on a single CPM, we estimated a log-normal distribution ofτ $$ \tau $$ across all 469 CPMs. We then used this distribution as an empirical prior. We used cross-validation to compare this empirical Bayesian approach with frequentist fixed and random-effects meta-analyses.

RESULTS

The 469 CPMs included in our study had a median of 2 external validations with an IQR of [1-3]. The estimated distribution ofτ $$ \tau $$ had a mean of 0.055 and a standard deviation of 0.015. Ifτ $$ \tau $$ = 0.05, then the 95% prediction interval for the AUC in a new setting has a width of at least+ / - $$ +/- $$ 0.1, no matter how many validations have been done. When there are fewer than 5 validations, which is typically the case, the usual frequentist methods grossly underestimate the uncertainty about the AUC in a new setting. Accounting forτ $$ \tau $$ in a Bayesian approach achieved near nominal coverage.

CONCLUSION

Due to large heterogeneity among the validated AUC values of a CPM, there is great irreducible uncertainty in predicting the AUC in a new setting. This uncertainty is underestimated by existing methods. The proposed empirical Bayes approach addresses this problem which merits wide application in judging the validity of prediction models.

The urgency of centering safety-net organizations in AI governance

Although robust AI governance requires the engagement of diverse stakeholders across the artificial intelligence (AI) ecosystem, the US safety net has largely been excluded from this kind of collaboration. Without a reorientation of the AI governance agenda, marginalized patients will disproportionately bear the risks of AI in the US healthcare system. To prevent this replication of digital inequity and an organizational digital divide, we suggest specific next steps for diverse stakeholders to progress toward more equitable policy and practice.

Deep Learning in Glaucoma Detection and Progression Prediction: A Systematic Review and Meta-Analysis

Purpose: To evaluate the performance of deep learning (DL) in diagnosing glaucoma and predicting its progression using fundus photography and retinal optical coherence tomography (OCT) images. Materials and Methods: Relevant studies published up to 30 October 2024 were retrieved from PubMed, Medline, EMBASE, Cochrane Library, Web of Science, and ClinicalKey. A bivariate random-effects model was employed to calculate pooled sensitivity, specificity, positive and negative likelihood ratios, and area under the receiver operating characteristic curve (AUROC). Results: A total of 48 studies were included in the meta-analysis. DL algorithms demonstrated high diagnostic performance in glaucoma detection using fundus photography and OCT images. For fundus photography, the pooled sensitivity and specificity were 0.92 (95% CI: 0.89-0.94) and 0.93 (95% CI: 0.90-0.95), respectively, with an AUROC of 0.90 (95% CI: 0.88-0.92). For the OCT imaging, the pooled sensitivity and specificity were 0.90 (95% CI: 0.84-0.94) and 0.87 (95% CI: 0.81-0.91), respectively, with an AUROC of 0.86 (95% CI: 0.83-0.90). In predicting glaucoma progression, DL models generally showed less robust performance, with pooled sensitivities and specificities ranging lower than in diagnostic tasks. Internal validation datasets showed higher accuracy than external validation datasets. Conclusions: DL algorithms achieve excellent performance in diagnosing glaucoma using fundus photography and OCT imaging. To enhance the prediction of glaucoma progression, future DL models should integrate multimodal data, including functional assessments, such as visual field measurements, and undergo extensive validation in real-world clinical settings.

The diagnostic and prognostic capability of artificial intelligence in spinal cord injury: A systematic review

Background

Artificial intelligence (AI) models have shown potential for diagnosing and prognosticating traumatic spinal cord injury (tSCI), but their clinical utility remains uncertain.

Method

ology: The primary aim was to evaluate the performance of AI algorithms in diagnosing and prognosticating tSCI. Subsequent systematic searching of seven databases identified studies evaluating AI models. PROBAST and TRIPOD tools were used to assess the quality and reporting of included studies (PROSPERO: CRD42023464722). Fourteen studies, comprising 20 models and 280,817 pooled imaging datasets, were included. Analysis was conducted in line with the SWiM guidelines.

Results

For prognostication, 11 studies predicted outcomes including AIS improvement (30%), mortality and ambulatory ability (20% each), and discharge or length of stay (10%). The mean AUC was 0.770 (range: 0.682-0.902), indicating moderate predictive performance. Diagnostic models utilising DTI, CT, and T2-weighted MRI with CNN-based segmentation achieved a weighted mean accuracy of 0.898 (range: 0.813-0.938), outperforming prognostic models.

Conclusion

AI demonstrates strong diagnostic accuracy (mean accuracy: 0.898) and moderate prognostic capability (mean AUC: 0.770) for tSCI. However, the lack of standardised frameworks and external validation limits clinical applicability. Future models should integrate multimodal data, including imaging, patient characteristics, and clinician judgment, to improve utility and alignment with clinical practice.

Prognostic models for prediction of perioperative allogeneic red blood cell transfusion in adult cardiac surgery: A systematic review and meta-analysis

OBJECTIVES

Identifying cardiac surgical patients at risk of requiring red blood cell (RBC) transfusion is crucial for optimizing their outcome. We critically appraised prognostic models preoperatively predicting perioperative exposure to RBC transfusion in adult cardiac surgery and summarized model performance.

METHODS

Design: Systematic review and meta-analysis.

STUDY ELIGIBILITY CRITERIA

Studies developing and/or externally validating models preoperatively predicting perioperative RBC transfusion in adult cardiac surgery. Information sources MEDLINE, CENTRAL & CDSR, Embase, Transfusion Evidence Library, Web of Science, Scopus, ClinicalTrials.gov, and WHO ICTRP. Risk of bias and applicability: Quality of reporting was assessed with the Transparent Reporting of studies on prediction models for Individual Prognosis or Diagnosis adherence form, and risk of bias and applicability with the Prediction model Risk of Bias ASsessment Tool.

SYNTHESIS METHODS

Random-effects meta-analyses of concordance-statistics and total observed:expected ratios for models externally validated ≥5 times.

RESULTS

Nine model development, and 27 external validation studies were included. The average TRIPOD adherence score was 66.4% (range 44.1%-85.2%). All studies but 1 were rated high risk of bias. For TRUST and TRACK, the only models externally validated ≥5 times, summary c-statistics were 0.74 (95% CI: 0.65-0.84; 6 contributing studies) and 0.72 (95% CI: 0.68-0.75; 5 contributing studies) respectively, and summary total observed:expected ratios were 0.86 (95% CI: 0.71-1.05; 5 contributing studies) and 0.94 (95% CI: 0.74-1.19; 5 contributing studies), respectively. Considerable heterogeneity was observed in all meta-analyses.

DISCUSSION

Future high quality external validation and model updating studies which strictly adhere to reporting guidelines, are warranted.

Development and validation of artificial intelligence models for early detection of postoperative infections (PERISCOPE): a multicentre study using electronic health record data

Background

Postoperative infections significantly impact patient outcomes and costs, exacerbated by late diagnoses, yet early reliable predictors are scarce. Existing artificial intelligence (AI) models for postoperative infection prediction often lack external validation or perform poorly in local settings when validated. We aimed to develop locally valid models as part of the PERISCOPE AI system to enable early detection, safer discharge, and more timely treatment of patients.

Methods

We developed and validated XGBoost models to predict postoperative infections within 7 and 30 days of surgery. Using retrospective pre-operative and intra-operative electronic health record data from 2014 to 2023 across various surgical specialities, the models were developed at Hospital A and validated and updated at Hospitals B and C in the Netherlands and Belgium. Model performance was evaluated before and after updating using the two most recent years of data as temporal validation datasets. Main outcome measures were model discrimination (area under the receiver operating characteristic curve (AUROC)), calibration (slope, intercept, and plots), and clinical utility (decision curve analysis with net benefit).

Findings

The study included 253,010 surgical procedures with 23,903 infections within 30-days. Discriminative performance, calibration properties, and clinical utility significantly improved after updating. Final AUROCs after updating for Hospitals A, B, and C were 0.82 (95% confidence interval (CI) 0.81-0.83), 0.82 (95% CI 0.81-0.83), and 0.91 (95% CI 0.90-0.91) respectively for 30-day predictions on the temporal validation datasets (2022-2023). Calibration plots demonstrated adequate correspondence between observed outcomes and predicted risk. All local models were deemed clinically useful as the net benefit was higher than default strategies (treat all and treat none) over a wide range of clinically relevant decision thresholds.

Interpretation

PERISCOPE can accurately predict overall postoperative infections within 7- and 30-days post-surgery. The robust performance implies potential for improving clinical care in diverse clinical target populations. This study supports the need for approaches to local updating of AI models to account for domain shifts in patient populations and data distributions across different clinical settings.

Funding

This study was funded by a REACT EU grant from European Regional Development Fund (ERDF) and Kansen voor West.

Predicting patent challenges for small-molecule drugs: A cross-sectional study

BACKGROUND

The high cost of prescription drugs in the United States is maintained by brand-name manufacturers' competition-free period made possible in part through patent protection, which generic competitors must challenge to enter the market early. Understanding the predictors of these challenges can inform policy development to encourage timely generic competition. Identifying categories of drugs systematically overlooked by challengers, such as those with low market size, highlights gaps where unchecked patent quality and high prices persist, and can help design policy interventions to help promote timely patient access to generic drugs including enhanced patent scrutiny or incentives for challenges. Our objective was to characterize and assess the extent to which market size and other drug characteristics can predict patent challenges for brand-name drugs.

METHODS AND FINDINGS

This cross-sectional study included new patented small-molecule drugs approved by the FDA from 2007 to 2018. Market size, patent, and patent challenge data came from IQVIA MIDAS pharmaceutical quarterly sales data, the FDA's Orange Book database, and the FDA's Paragraph IV list. Predictive models were constructed using random forest and elastic net classification. The primary outcome was the occurrence of a patent challenge within the first year of eligibility. Of the 210 new small-molecule drugs included in the sample, 55% experienced initiation of patent challenge within the first year of eligibility. Market value was the most important predictor variable, with larger markets being more likely to be associated with patent challenges. Drugs in the anti-infective therapeutic class or those with fast-track approval were less likely to be challenged. The limitations of this work arise from the exclusion of variables that were not readily available publicly, will be the target of future research, or were deemed beyond the scope of this project.

CONCLUSIONS

Generic competition does not occur with the same timeliness across all drug markets, which can leave granted patents of questionable merit in place and sustain high brand-name drug prices. Predictive models may help direct limited resources for post-grant patent validity review and adjust policy when generic competition is lacking.

Balancing accuracy and Interpretability: An R package assessing complex relationships beyond the Cox model and applications to clinical prediction

BACKGROUND

Accurate and interpretable models are essential for clinical decision-making, where predictions can directly impact patient care. Machine learning (ML) survival methods can handle complex multidimensional data and achieve high accuracy but require post-hoc explanations. Traditional models such as the Cox Proportional Hazards Model (Cox-PH) are less flexible, but fast, stable, and intrinsically transparent. Moreover, ML does not always outperform Cox-PH in clinical settings, warranting a diligent model validation. We aimed to develop a set of R functions to help explore the limits of Cox-PH compared to the tree-based and deep learning survival models for clinical prediction modelling, employing ensemble learning and nested cross-validation.

METHODS

We developed a set of R functions, publicly available as the package "survcompare". It supports Cox-PH and Cox-Lasso, and Survival Random Forest (SRF) and DeepHit are the ML alternatives, along with the ensemble methods integrating Cox-PH with SRF or DeepHit designed to isolate the marginal value of ML. The package performs a repeated nested cross-validation and tests for statistical significance of the ML's superiority using the survival-specific performance metrics, the concordance index, time-dependent AUC-ROC and calibration slope. To get practical insights, we applied this methodology to clinical and simulated datasets with varying complexities and sizes.

RESULTS

In simulated data with non-linearities or interactions, ML models outperformed Cox-PH at sample sizes ≥ 500. ML superiority was also observed in imaging and high-dimensional clinical data. However, for tabular clinical data, the performance gains of ML were minimal; in some cases, regularised Cox-Lasso recovered much of the ML's performance advantage with significantly faster computations. Ensemble methods combining Cox-PH and ML predictions were instrumental in quantifying Cox-PH's limits and improving ML calibration. Traditional models like Cox-PH or Cox-Lasso should not be overlooked while developing clinical predictive models from tabular data or data of limited size.

CONCLUSION

Our package offers researchers a framework and practical tool for evaluating the accuracy-interpretability trade-off, helping make informed decisions about model selection.

Updating methods for artificial intelligence-based clinical prediction models: a scoping review

OBJECTIVES

To give an overview of methods for updating artificial intelligence (AI)-based clinical prediction models based on new data.

STUDY DESIGN AND SETTING

We comprehensively searched Scopus and Embase up to August 2022 for articles that addressed developments, descriptions, or evaluations of prediction model updating methods. We specifically focused on articles in the medical domain involving AI-based prediction models that were updated based on new data, excluding regression-based updating methods as these have been extensively discussed elsewhere. We categorized and described the identified methods used to update the AI-based prediction model as well as the use cases in which they were used.

RESULTS

We included 78 articles. The majority of the included articles discussed updating for neural network methods (93.6%) with medical images as input data (65.4%). In many articles (51.3%) existing, pretrained models for broad tasks were updated to perform specialized clinical tasks. Other common reasons for model updating were to address changes in the data over time and cross-center differences; however, more unique use cases were also identified, such as updating a model from a broad population to a specific individual. We categorized the identified model updating methods into four categories: neural network-specific methods (described in 92.3% of the articles), ensemble-specific methods (2.5%), model-agnostic methods (9.0%), and other (1.3%). Variations of neural network-specific methods are further categorized based on the following: (1) the part of the original neural network that is kept, (2) whether and how the original neural network is extended with new parameters, and (3) to what extent the original neural network parameters are adjusted to the new data. The most frequently occurring method (n = 30) involved selecting the first layer(s) of an existing neural network, appending new, randomly initialized layers, and then optimizing the entire neural network.

CONCLUSION

We identified many ways to adjust or update AI-based prediction models based on new data, within a large variety of use cases. Updating methods for AI-based prediction models other than neural networks (eg, random forest) appear to be underexplored in clinical prediction research.

PLAIN LANGUAGE SUMMARY

AI-based prediction models are increasingly used in health care, helping clinicians with diagnosing diseases, guiding treatment decisions, and informing patients. However, these prediction models do not always work well when applied to hospitals, patient populations, or times different from those used to develop the models. Developing new models for every situation is neither practical nor desired, as it wastes resources, time, and existing knowledge. A more efficient approach is to adjust existing models to new contexts ('updating'), but there is limited guidance on how to do this for AI-based clinical prediction models. To address this, we reviewed 78 studies in detail to understand how researchers are currently updating AI-based clinical prediction models, and the types of situations in which these updating methods are used. Our findings provide a comprehensive overview of the available methods to update existing models. This is intended to serve as guidance and inspiration for researchers. Ultimately, this can lead to better reuse of existing models and improve the quality and efficiency of AI-based prediction models in health care.

External validation of twelve existing survival prediction models for patients with spinal metastases

BACKGROUND CONTEXT

Survival prediction models for patients with spinal metastases may inform patients and clinicians in shared decision-making.

PURPOSE

To externally validate all existing survival prediction models for patients with spinal metastases.

DESIGN

Prospective cohort study using retrospective data.

PATIENT SAMPLE

953 patients.

OUTCOME MEASURES

Survival in months, area under the curve (AUC), and calibration intercept and slope.

METHOD

This study included patients with spinal metastases referred to a single tertiary referral center between 2016 and 2021. Twelve models for predicting 3, 6, and 12-month survival were externally validated Bollen, Mizumoto, Modified Bauer, New England Spinal Metastasis Score, Original Bauer, Oswestry Spinal Risk Index (OSRI), PathFx, Revised Katagiri, Revised Tokuhashi, Skeletal Oncology Research Group Machine Learning Algorithm (SORG-MLA), Tomita, and Van der Linden. Discrimination was assessed using (AUC) and calibration using the intercept and slope. Calibration was considered appropriate if calibration measures were close to their ideal values with narrow confidence intervals.

RESULTS

In total, 953 patients were included. Survival was 76.4% at 3 months (728/953), 62.2% at 6 months (593/953), and 50.3% at 12 months (479/953). Revised Katagiri yielded AUCs of 0.79 (95% CI, 0.76-0.82) to 0.81 (95% CI, 0.79-0.84), Bollen yielded AUCs of 0.76 (95% CI, 0.73-0.80) to 0.77 (95% CI, 0.75-0.80), and OSRI yielded AUCs of 0.75 (95% CI, 0.72-0.78) to 0.77 (95% CI, 0.74-0.79). The other 9 prediction models yielded AUCs ranging from 0.59 (95% CI, 0.55-0.63) to 0.76 (95% CI, 0.74-0.79). None of the twelve models yielded appropriate calibration.

CONCLUSIONS

Twelve survival prediction models for patients with spinal metastases yielded poor to fair discrimination and poor calibration. Survival prediction models may inform decision-making in patients with spinal metastases, provided that recalibration using recent patient data is performed.

Developing a clinical prediction model to modify empirical antibiotics for non-typhoidal Salmonella bloodstream infection in children under-five in the Democratic Republic of Congo

BACKGROUND

Non-typhoidal Salmonella (NTS) frequently cause bloodstream infection in children under-five in sub-Saharan Africa, particularly in malaria-endemic areas. Due to increasing drug resistance, NTS are often not covered by standard-of-care empirical antibiotics for severe febrile illness. We developed a clinical prediction model to orient the choice of empirical antibiotics (standard-of-care versus alternative antibiotics) for children admitted to hospital in settings with high proportions of drug-resistant NTS.

METHODS

Data were collected during a prospective cohort study in children (> 28 days-< 5 years) admitted with severe febrile illness to Kisantu district hospital, DR Congo. The outcome variable was blood culture confirmed NTS bloodstream infection; the comparison group were children without NTS bloodstream infection. Predictors were selected a priori based on systematic literature review. The prediction model was developed with multivariable logistic regression; a simplified scoring system was derived. Internal validation to estimate optimism-corrected performance was performed using bootstrapping and net benefits were calculated to evaluate clinical usefulness.

RESULTS

NTS bloodstream infection was diagnosed in 12.7% (295/2327) of enrolled children. The area under the curve was 0.79 (95%CI: 0.76-0.82) for the prediction model, and 0.78 (0.85-0.80) for the scoring system. The estimated calibration slopes were 0.95 (model) and 0.91 (scoring system). At a decision threshold of 20% NTS risk, the prediction model and scoring system had 57% and 53% sensitivity, and 85% specificity. The net benefit for decisions thresholds < 30% ranged from 2.4 to 3.9 per 100 children.

CONCLUSION

The model predicts NTS bloodstream infection and can support the choice of empiric antibiotics to include coverage of drug-resistant NTS, in particular for decision thresholds < 30%. External validation studies are needed to investigate generalizability.

TRIAL REGISTRATION

DeNTS study, clinicaltrials.gov: NCT04473768 (registration 16/07/2020) and TreNTS study, clinicaltrials.gov: NCT04850677 (registration 20/04/2021).

How Outcome Prediction Could Aid Clinical Practice

Predictive algorithms have myriad potential clinical decision-making implications from prognostic counselling to improving clinical trial efficiency. Large observational (or "real world") cohorts are a common data source for the development and evaluation of such tools. There is significant optimism regarding the benefits and use cases for risk-based care, but there is a notable disparity between the volume of clinical prediction models published and implementation into healthcare systems that drive and realise patient benefit. Considering the perspective of a clinician or clinical researcher that may encounter clinical predictive algorithms in the near future as a user or developer, this editorial: (1) discusses the ways in which prediction models built using observational data could inform better clinical decisions; (2) summarises the main steps in producing a model with special focus on key appraisal factors; and (3) highlights recent work driving evolution in the ways that we should conceptualise, build and evaluate these tools.

Risk prediction tools for pressure injury occurrence: an umbrella review of systematic reviews reporting model development and validation methods

BACKGROUND

Pressure injuries (PIs) place a substantial burden on healthcare systems worldwide. Risk stratification of those who are at risk of developing PIs allows preventive interventions to be focused on patients who are at the highest risk. The considerable number of risk assessment scales and prediction models available underscores the need for a thorough evaluation of their development, validation, and clinical utility. Our objectives were to identify and describe available risk prediction tools for PI occurrence, their content and the development and validation methods used.

METHODS

The umbrella review was conducted according to Cochrane guidance. MEDLINE, Embase, CINAHL, EPISTEMONIKOS, Google Scholar, and reference lists were searched to identify relevant systematic reviews. The risk of bias was assessed using adapted AMSTAR-2 criteria. Results were described narratively. All included reviews contributed to building a comprehensive list of risk prediction tools.

RESULTS

We identified 32 eligible systematic reviews only seven of which described the development and validation of risk prediction tools for PI. Nineteen reviews assessed the prognostic accuracy of the tools and 11 assessed clinical effectiveness. Of the seven reviews reporting model development and validation, six included only machine learning models. Two reviews included external validations of models, although only one review reported any details on external validation methods or results. This was also the only review to report measures of both discrimination and calibration. Five reviews presented measures of discrimination, such as the area under the curve (AUC), sensitivities, specificities, F1 scores, and G-means. For the four reviews that assessed the risk of bias assessment using the PROBAST tool, all models but one were found to be at high or unclear risk of bias.

CONCLUSIONS

Available tools do not meet current standards for the development or reporting of risk prediction models. The majority of tools have not been externally validated. Standardised and rigorous approaches to risk prediction model development and validation are needed.

TRIAL REGISTRATION

The protocol was registered on the Open Science Framework ( https://osf.io/tepyk ).

Predicting type 2 diabetes and testosterone effects in high-risk Australian men: development and external validation of a 2-year risk model

OBJECTIVE

We have shown that men aged 50 years+ at high risk of type 2 diabetes treated with testosterone together with a lifestyle program reduced the risk of type 2 diabetes at 2 years by 40% compared to a lifestyle program alone. To develop a personalized approach to treatment, we aimed to explore a prognostic model for incident type 2 diabetes at 2 years and investigate biomarkers predictive of the testosterone effect.

DESIGN

Model development in 783 men with impaired glucose tolerance but not type 2 diabetes from Testosterone for Prevention of Type 2 Diabetes; a multicenter, 2-year trial of Testosterone vs placebo. External validation performed in 236 men from the Examining Outcomes in Chronic Disease in the 45 and Up Study (EXTEND-45, n = 267 357).

METHODS

Type 2 diabetes at 2 years defined as 2-h fasting glucose by oral glucose tolerance test (OGTT) ≥11.1 mmol/L. Risk factors, including predictive biomarkers of testosterone treatment, were assessed using penalized logistic regression.

RESULTS

Baseline HbA1c and 2-h OGTT glucose were dominant predictors, together with testosterone, age, and an interaction between testosterone and HbA1c (P = .035, greater benefit with HbA1c ≥ 5.6%, 38 mmol/mol). The final model identified men who developed type 2 diabetes, with C-statistics 0.827 in development and 0.798 in validation. After recalibration, the model accurately predicted a participant's absolute risk of type 2 diabetes.

CONCLUSIONS

Baseline HbA1c and 2-h OGTT glucose predict incident type 2 diabetes at 2 years in high-risk men, with risk modified independently by testosterone treatment. Men with HbA1c ≥ 5.6% (38 mmol/mol) benefit most from testosterone treatment, beyond a lifestyle program.

External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis

BACKGROUND

Machine learning (ML) is increasingly used to predict clinical deterioration in intensive care unit (ICU) patients through scoring systems. Although promising, such algorithms often overfit their training cohort and perform worse at new hospitals. Thus, external validation is a critical - but frequently overlooked - step to establish the reliability of predicted risk scores to translate them into clinical practice. We systematically reviewed how regularly external validation of ML-based risk scores is performed and how their performance changed in external data.

METHODS

We searched MEDLINE, Web of Science, and arXiv for studies using ML to predict deterioration of ICU patients from routine data. We included primary research published in English before December 2023. We summarised how many studies were externally validated, assessing differences over time, by outcome, and by data source. For validated studies, we evaluated the change in area under the receiver operating characteristic (AUROC) attributable to external validation using linear mixed-effects models.

RESULTS

We included 572 studies, of which 84 (14.7%) were externally validated, increasing to 23.9% by 2023. Validated studies made disproportionate use of open-source data, with two well-known US datasets (MIMIC and eICU) accounting for 83.3% of studies. On average, AUROC was reduced by -0.037 (95% CI -0.052 to -0.027) in external data, with more than 0.05 reduction in 49.5% of studies.

DISCUSSION

External validation, although increasing, remains uncommon. Performance was generally lower in external data, questioning the reliability of some recently proposed ML-based scores. Interpretation of the results was challenged by an overreliance on the same few datasets, implicit differences in case mix, and exclusive use of AUROC.

External Validation of the Walter Index for Posthospitalization Mortality Prediction in Older Adults

Importance

The Walter Index is a widely used prognostic tool for assessing 12-month mortality risk among hospitalized older adults. Developed in the US in 2001, its accuracy in contemporary non-US contexts is unclear.

Objective

To evaluate the external validity of the Walter Index in predicting posthospitalization mortality risk in Brazilian older adult inpatients.

Design, Setting, and Participants

This prognostic study used data from a cohort of adults aged 70 years or older admitted to the geriatric unit of a university hospital in Brazil from January 1, 2009, to February 28, 2020. Participants underwent comprehensive geriatric assessments at admission, were reevaluated at discharge, and were subsequently followed up for 48 months. Data were analyzed from March to July 2024.

Main Outcomes and Measures

The Walter Index, a score based on 6 risk factors (male sex, dependent activities of daily living at discharge, heart failure, cancer, high creatinine level, and low albumin level), was calculated to assess its predictive accuracy for 12-month mortality as well as 6-, 24-, and 48-month mortality. The study investigated whether incorporating delirium, frailty, or C-reactive protein level enhanced accuracy. Performance was assessed using discrimination, calibration, and clinical utility measures.

Results

In total, 2780 participants (mean [SD] age, 81 [7] years; 1795 [65%] female) were included, with 89 (3%) lost to follow-up. The 12-month posthospitalization mortality rate was 23% (646 participants). Mortality was 7% (47 of 634) in the lowest-risk group (0-1 point), 17% (111 of 668) for 2 to 3 points, 25% (198 of 803) for 4 to 6 points, and 43% (290 of 675) in the highest-risk group (≥7 points). The index demonstrated an area under the receiver operating characteristic curve (AUC) of 0.714 (95% CI, 0.691-0.736) for predicting 12-month posthospitalization mortality (AUCs were 0.75 and 0.80 in the original derivation and validation cohorts, respectively). Comparable results were observed for mortality at 6 months (AUC, 0.726; 95% CI, 0.700-0.752), 24 months (AUC, 0.711; 95% CI, 0.691-0.730), and 48 months (AUC, 0.719; 95% CI, 0.700-0.738). Adding delirium modestly increased the index's discrimination (AUC, 0.723; 95% CI, 0.702-0.749); additionally including frailty and C-reactive protein level did not improve discrimination further (AUC, 0.723; 95% CI, 0.701-0.744).

Conclusions and Relevance

In this prognostic study of hospitalized older adults in Brazil, the Walter Index showed similar discrimination in predicting postdischarge mortality as it did 2 decades ago in the US. These findings highlight the need for continuous validation and potential modification of established prognostic tools to improve their applicability across settings.

Laboratory Data as a Potential Source of Bias in Healthcare Artificial Intelligence and Machine Learning Models

Artificial intelligence (AI) and machine learning (ML) are anticipated to transform the practice of medicine. As one of the largest sources of digital data in healthcare, laboratory results can strongly influence AI and ML algorithms that require large sets of healthcare data for training. Embedded bias introduced into AI and ML models not only has disastrous consequences for quality of care but also may perpetuate and exacerbate health disparities. The lack of test harmonization, which is defined as the ability to produce comparable results and the same interpretation irrespective of the method or instrument platform used to produce the result, may introduce aggregation bias into algorithms with potential adverse outcomes for patients. Limited interoperability of laboratory results at the technical, syntactic, semantic, and organizational levels is a source of embedded bias that limits the accuracy and generalizability of algorithmic models. Population-specific issues, such as inadequate representation in clinical trials and inaccurate race attribution, not only affect the interpretation of laboratory results but also may perpetuate erroneous conclusions based on AI and ML models in the healthcare literature.

Longitudinal Model Shifts of Machine Learning-Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use Cases Across Different Hospitals

BACKGROUND

In recent years, machine learning (ML)-based models have been widely used in clinical domains to predict clinical risk events. However, in production, the performances of such models heavily rely on changes in the system and data. The dynamic nature of the system environment, characterized by continuous changes, has significant implications for prediction models, leading to performance degradation and reduced clinical efficacy. Thus, monitoring model shifts and evaluating their impact on prediction models are of utmost importance.

OBJECTIVE

This study aimed to assess the impact of a model shift on ML-based prediction models by evaluating 3 different use cases-delirium, sepsis, and acute kidney injury (AKI)-from 2 hospitals (M and H) with different patient populations and investigate potential model deterioration during the COVID-19 pandemic period.

METHODS

We trained prediction models using retrospective data from earlier years and examined the presence of a model shift using data from more recent years. We used the area under the receiver operating characteristic curve (AUROC) to evaluate model performance and analyzed the calibration curves over time. We also assessed the influence on clinical decisions by evaluating the alert rate, the rates of over- and underdiagnosis, and the decision curve.

RESULTS

The 2 data sets used in this study contained 189,775 and 180,976 medical cases for hospitals M and H, respectively. Statistical analyses (Z test) revealed no significant difference (P>.05) between the AUROCs from the different years for all use cases and hospitals. For example, in hospital M, AKI did not show a significant difference between 2020 (AUROC=0.898) and 2021 (AUROC=0.907, Z=-1.171, P=.242). Similar results were observed in both hospitals and for all use cases (sepsis and delirium) when comparing all the different years. However, when evaluating the calibration curves at the 2 hospitals, model shifts were observed for the delirium and sepsis use cases but not for AKI. Additionally, to investigate the clinical utility of our models, we performed decision curve analysis (DCA) and compared the results across the different years. A pairwise nonparametric statistical comparison showed no differences in the net benefit at the probability thresholds of interest (P>.05). The comprehensive evaluations performed in this study ensured robust model performance of all the investigated models across the years. Moreover, neither performance deteriorations nor alert surges were observed during the COVID-19 pandemic period.

CONCLUSIONS

Clinical risk prediction models were affected by the dynamic and continuous evolution of clinical practices and workflows. The performance of the models evaluated in this study appeared stable when assessed using AUROCs, showing no significant variations over the years. Additional model shift investigations suggested that a calibration shift was present for certain use cases (delirium and sepsis). However, these changes did not have any impact on the clinical utility of the models based on DCA. Consequently, it is crucial to closely monitor data changes and detect possible model shifts, along with their potential influence on clinical decision-making.

Development and internal validation of time-to-event risk prediction models for major medical complications within 30 days after elective colectomy

BACKGROUND

Patients undergoing colectomy are at risk of numerous major complications. However, existing binary risk stratification models do not predict when a patient may be at highest risks of each complication. Accurate prediction of the timing of complications facilitates targeted, resource-efficient monitoring. We sought to develop and internally validate Cox proportional hazards models to predict time-to-complication of major complications within 30 days after elective colectomy.

METHODS

We studied a retrospective cohort from the multicentered American College of Surgeons National Surgical Quality Improvement Program procedure-targeted colectomy dataset. Patients aged 18 years or above, who underwent elective colectomy between January 1, 2014 and December 31, 2019 were included. A priori candidate predictors were selected based on variable availability, literature review, and multidisciplinary team consensus. Outcomes were mortality, hospital readmission, myocardial infarction, cerebral vascular events, pneumonia, venous thromboembolism, acute renal failure, and sepsis or septic shock within 30 days after surgery.

RESULTS

The cohort consisted of 132145 patients (mean ± SD age, 61 ± 15 years; 52% females). Complication rates ranged between 0.3% (n = 383) for cardiac arrest and acute renal failure to 5.3% (n = 6986) for bleeding requiring transfusion, with readmission rate of 8.6% (n = 11415). We observed distinct temporal patterns for each complication: the median [quartiles] postoperative day of complication diagnosis ranged from 1 [0, 2] days for bleeding requiring transfusion to 12 [6, 18] days for venous thromboembolism. Models for mortality, myocardial infarction, pneumonia, and renal failure showed good discrimination with a concordance > 0.8, while models for readmission, venous thromboembolism, and sepsis performed poorly with a concordance of 0.6 to 0.7. Models exhibited good calibration but ranges were limited to low probability areas.

CONCLUSIONS

We developed and internally validated time-to-event prediction models for complications after elective colectomy. Once further validated, the models can facilitate tailored monitoring of high risk patients during high risk periods.

TRIAL REGISTRATION

Clinicaltrials.gov (NCT05150548; Principal Investigator: Janny Xue Chen Ke, M.D., M.Sc., F.R.C.P.C.; initial posting: November 25, 2021).

International multi-institutional external validation of preoperative risk scores for 30-day in-hospital mortality in paediatric patients

BACKGROUND

Risk prediction scores are used to guide clinical decision-making. Our primary objective was to externally validate two patient-specific risk scores for 30-day in-hospital mortality using the Multicenter Perioperative Outcomes Group (MPOG) registry: the Pediatric Risk Assessment (PRAm) score and the intrinsic surgical risk score. The secondary objective was to recalibrate these scores.

METHODS

Data from 56 US and Dutch hospitals with paediatric caseloads were included. The primary outcome was 30-day mortality. To assess model discrimination, the area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUC-PR) were calculated. Model calibration was assessed by plotting the observed and predicted probabilities. Decision analytic curves were fit.

RESULTS

The 30-day mortality was 0.14% (822/606 488). The AUROC for the PRAm upon external validation was 0.856 (95% confidence interval 0.844-0.869), and the AUC-PR was 0.008. Upon recalibration, the AUROC was 0.873 (0.861-0.886), and the AUC-PR was 0.031. The AUROC for the external validation of the intrinsic surgical risk score was 0.925 (0.914-0.936) and AUC-PR was 0.085. Upon recalibration, the AUROC was 0.925 (0.915-0.936), and the AUC-PR was 0.094. Calibration metrics for both scores were favourable because of the large cluster of cases with low probabilities of mortality. Decision curve analyses showed limited benefit to using either score.

CONCLUSIONS

The intrinsic surgical risk score performed better than the PRAm, but both resulted in large numbers of false positives. Both scores exhibited decreased performance compared with the original studies. ASA physical status scores in sicker patients drove the superior performance of the intrinsic surgical risk score, suggesting the use of a risk score does not improve prediction.

Machine learning for predicting duration of surgery and length of stay: A literature review on joint arthroplasty

INTRODUCTION

In recent years, different factors such as population aging have caused escalating demand for hip and knee arthroplasty straining already limited hospitals' resources. To address this challenge, focus is put on medical and operational efficiency improvements. This includes an increased use of machine learning (ML) to predict duration of surgery (DOS) and length of stay (LOS) for total knee and total hip arthroplasty, which can be utilized for optimizing resource allocation to satisfy medical and operational limitations. This paper explores the development and performance of ML models in predicting DOS and LOS.

METHODS

A systematic search of publications between 2010-2023 was conducted following PRISMA guidelines. Considering the inclusion and exclusion criteria, 28 out of 722 gathered papers from PubMed, Web of Science, and manual search were included in the study. Descriptive statistics was used to analyze the extracted data regarding data preprocessing, model development, and model performance assessment.

RESULTS

Most of the papers work on LOS as a binary variable. Patient's age was identified as the most frequently used and reported as important variable for predicting DOS and LOS. Investigations also illustrated that within the resulting 28 papers, more than 71% of models reached good to perfect performance based on the area under the receiver operating characteristic curve (AUC), where artificial neural networks and ensemble learning models had the biggest share among the best-performing models.

CONCLUSION

The utilization of ML models is increasing in the literature. The current performance level indicates that ML can potentially turn to powerful tools in predicting DOS and LOS for different purposes. Meanwhile, the literature is not matured yet in reporting real-life application. Future studies can focus on model specification and validation by considering empirical application.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Identifying persistent high-cost patients in the hospital for care management: development and validation of prediction models

BACKGROUND

Healthcare use by High-Need High-Cost (HNHC) patients is believed to be modifiable through better coordination of care. To identify patients for care management, a hybrid approach is recommended that combines clinical assessment of need with model-based prediction of cost. Models that predict high healthcare costs persisting over time are relevant but scarce. We aimed to develop and validate two models predicting Persistent High-Cost (PHC) status upon hospital outpatient visit and hospital admission, respectively.

METHODS

We performed a retrospective cohort study using claims data from a national health insurer in the Netherlands-a regulated competitive health care system with universal coverage. We created two populations of adults based on their index event in 2016: a first hospital outpatient visit (i.e., outpatient population) or hospital admission (i.e., hospital admission population). Both were divided in a development (January-June) and validation (July-December) cohort. Our outcome of interest, PHC status, was defined as belonging to the top 10% of total annual healthcare costs for three consecutive years after the index event. Predictors were predefined based on an earlier systematic review and collected in the year prior to the index event. Predictor effects were quantified through logistic multivariable regression analysis. To increase usability, we also developed smaller models containing the lowest number of predictors while maintaining comparable performance. This was based on relative predictor importance (Wald χ2). Model performance was evaluated by means of discrimination (C-statistic) and calibration (plots).

RESULTS

In the outpatient development cohort (n = 135,558), 2.2% of patients (n = 3,016) was PHC. In the hospital admission development cohort (n = 24,805), this was 5.8% (n = 1,451). Both full models included 27 predictors, while their smaller counterparts had 10 (outpatient model) and 11 predictors (hospital admission model). In the outpatient validation cohort (n = 84,009) and hospital admission validation cohort (n = 20,768), discrimination was good for full models (C-statistics 0.75; 0.74) and smaller models (C-statistics 0.70; 0.73), while calibration plots indicated that models were well-calibrated.

CONCLUSIONS

We developed and validated two models predicting PHC status that demonstrate good discrimination and calibration. Both models are suitable for integration into electronic health records to aid a hybrid case-finding strategy for HNHC care management.

Soluble Urokinase-Type Plasminogen Activator Receptor and Inflammatory Biomarker Response with Prognostic Significance after Acute Neuronal Injury - a Prospective Cohort Study

Aneurysmal subarachnoid hemorrhage (aSAH), ischemic stroke (IS), and traumatic brain injury (TBI) are severe conditions impacting individuals and society. Identifying reliable prognostic biomarkers for predicting survival or recovery remains a challenge. Soluble urokinase type plasminogen activator receptor (suPAR) has gained attention as a potential prognostic biomarker in acute sepsis. This study evaluates suPAR and related neuroinflammatory biomarkers in serum for brain injury prognosis. This prospective study included 31 aSAH, 30 IS, 13 TBI, and three healthy controls (n = 77). Serum samples were collected on average 5.9 days post-injury, analyzing suPAR, IL-1β, cyclophilin A, and TNFα levels using ELISA. Outcomes were assessed 90 days post-injury with the modified Rankin Scale (mRS), categorized as favorable (mRS 0-2) or unfavorable (mRS 3-6). Statistical analyses included 2-tailed t-tests, Pearson's correlations, and machine learning linear discriminant analysis (LDA) for biomarker combinations. Elevated suPAR levels were found in brain injury patients compared to controls (p = 0.017). Increased suPAR correlated with unfavorable outcomes (p = 0.0018) and showed prognostic value (AUC = 0.66, p = 0.03). IL-1β levels were higher in the unfavorable group (p = 0.0015). LDA combinatory analysis resulted a fair prognostic accuracy with canonical equation = 0.775[suPAR] + 0.667[IL1-β] (AUC = 0.77, OR 0.296, sensitivity 93.1%, specificity 53.1%, p = 0.0007). No correlation was found between suPAR and CRP or infection status. Elevated suPAR levels in acute brain injury patients were associated with poorer outcomes, highlighting suPAR's potential as a prognostic biomarker across different brain injury types. Combining IL-1β with suPAR improved prognostic accuracy, supporting a multimodal biomarker approach for predicting outcomes.

Comprehensive Symptom Prediction in Inpatients With Acute Psychiatric Disorders Using Wearable-Based Deep Learning Models: Development and Validation Study

BACKGROUND

Assessing the complex and multifaceted symptoms of patients with acute psychiatric disorders proves to be significantly challenging for clinicians. Moreover, the staff in acute psychiatric wards face high work intensity and risk of burnout, yet research on the introduction of digital technologies in this field remains limited. The combination of continuous and objective wearable sensor data acquired from patients with deep learning techniques holds the potential to overcome the limitations of traditional psychiatric assessments and support clinical decision-making.

OBJECTIVE

This study aimed to develop and validate wearable-based deep learning models to comprehensively predict patient symptoms across various acute psychiatric wards in South Korea.

METHODS

Participants diagnosed with schizophrenia and mood disorders were recruited from 4 wards across 3 hospitals and prospectively observed using wrist-worn wearable devices during their admission period. Trained raters conducted periodic clinical assessments using the Brief Psychiatric Rating Scale, Hamilton Anxiety Rating Scale, Montgomery-Asberg Depression Rating Scale, and Young Mania Rating Scale. Wearable devices collected patients' heart rate, accelerometer, and location data. Deep learning models were developed to predict psychiatric symptoms using 2 distinct approaches: single symptoms individually (Single) and multiple symptoms simultaneously via multitask learning (Multi). These models further addressed 2 problems: within-subject relative changes (Deterioration) and between-subject absolute severity (Score). Four configurations were consequently developed for each scale: Single-Deterioration, Single-Score, Multi-Deterioration, and Multi-Score. Data of participants recruited before May 1, 2024, underwent cross-validation, and the resulting fine-tuned models were then externally validated using data from the remaining participants.

RESULTS

Of the 244 enrolled participants, 191 (78.3%; 3954 person-days) were included in the final analysis after applying the exclusion criteria. The demographic and clinical characteristics of participants, as well as the distribution of sensor data, showed considerable variations across wards and hospitals. Data of 139 participants were used for cross-validation, while data of 52 participants were used for external validation. The Single-Deterioration and Multi-Deterioration models achieved similar overall accuracy values of 0.75 in cross-validation and 0.73 in external validation. The Single-Score and Multi-Score models attained overall R² values of 0.78 and 0.83 in cross-validation and 0.66 and 0.74 in external validation, respectively, with the Multi-Score model demonstrating superior performance.

CONCLUSIONS

Deep learning models based on wearable sensor data effectively classified symptom deterioration and predicted symptom severity in participants in acute psychiatric wards. Despite lower computational costs, Multi models demonstrated equivalent or superior performance than Single models, suggesting that multitask learning is a promising approach for comprehensive symptom prediction. However, significant variations were observed across wards, which presents a key challenge for developing clinical decision support systems in acute psychiatric wards. Future studies may benefit from recurring local validation or federated learning to address generalizability issues.

Predicting periprosthetic joint infection: external validation of preoperative prediction models

Introduction: Prediction models for periprosthetic joint infections (PJIs) are gaining interest due to their potential to improve clinical decision-making. However, their external validity across various settings remains uncertain. This study aimed to externally validate promising preoperative PJI prediction models in a recent multinational European cohort. Methods: Three preoperative PJI prediction models - by Tan et al. (2018), Del Toro et al. (2019), and Bülow et al. (2022) - that have previously demonstrated high levels of accuracy were selected for validation. A retrospective observational analysis of patients undergoing total hip arthroplasty (THA) and total knee arthroplasty (TKA) at centers in the Netherlands, Portugal, and Spain between January 2020 and December 2021 was conducted. Patient characteristics were compared between our cohort and those used to develop the models. Performance was assessed through discrimination and calibration. Results: The study included 2684 patients, 60 of whom developed a PJI (2.2 %). Our cohort differed from the models' original cohorts with respect to demographic variables, procedural variables, and comorbidity prevalence. The overall accuracies of the models, measured with the c  statistic, were 0.72, 0.69, and 0.72 for the Tan, Del Toro, and Bülow models, respectively. Calibration was reasonable, but the PJI risk estimates were most accurate for predicted infection risks below 3 %-4 %. The Tan model overestimated PJI risk above 4 %, whereas the Del Toro model underestimated PJI risk above 3 %. Conclusions: The Tan, Del Toro, and Bülow PJI prediction models were externally validated in this multinational cohort, demonstrating potential for clinical application in identifying high-risk patients and enhancing preoperative counseling and prevention strategies.

Validity of the Johns Hopkins Adjusted Clinical Groups system on the utilisation of healthcare services in Norway: a retrospective cross-sectional study

BACKGROUND

The Adjusted Clinical Groups (ACG) System is a validated electronic risk stratification system. However, there is a lack of studies on the association between different ACG risk scores and the utilisation of different healthcare services using different sources of input data. The aim of this study was therefore to assess the validity of the association between five different ACG risk scores and the utilisation of a range of different healthcare services using input data from either general practitioners (GPs) or hospitals.

METHODS

Registry-based study of all adult inhabitants in four Norwegian municipalities that received somatic healthcare in one year (N = 168 285). The ACG risk scores resource utilisation band, unscaled ACG concurrent risk, unscaled concurrent risk, frailty flag and chronic condition count were calculated using age, sex and diagnosis codes from GPs and a hospital, respectively. Healthcare utilisation covered GP, municipal and hospital services. Areas under the receiver operating curve (AUC) were calculated and compared to the AUC of a model using only age and sex.

RESULTS

Utilisation of all healthcare services increased with increasing scores in the "resource utilisation band" (RUB) and all other investigated ACG risk scores. The risk scores overall distinguished well between levels of utilisation of GP visits (AUC up to 0.84), hospitalisation (AUC up to 0.8) and specialist outpatient visits (AUC up to 0.72), but not out-of-hours GP visits (AUC up to 0.62). The score "unscaled ACG concurrent risk" overall performed best. Risk scores based on data from either GPs or hospitals performed better for the classification of healthcare services in their respective domains. The model based on age and sex performed better for distinguishing between levels of utilisation of municipal services (AUC 0.83-0.90 compared to 0.46-0.79).

CONCLUSIONS

Risk scores from the ACG system is valid for classifying GP visits, hospitalisation and specialist outpatient visits. It does not outperform simpler models in the classification of utilisation of municipal services such as nursing homes and home services and outpatient emergency care in primary healthcare. The ACG system can be applied in Norway using administrative data from either GPs or hospitals.