Transparent reporting of multivariable prediction models for individual prognosis or diagnosis: checklist for systematic reviews and meta-analyses (TRIPOD-SRMA)

A machine learning-based predictive model for the in-hospital mortality of critically ill patients with atrial fibrillation

BACKGROUND

Atrial fibrillation (AF) is common among intensive care unit (ICU) patients and significantly raises the in-hospital mortality rate. Existing scoring systems or models have limited predictive capabilities for AF patients in ICU. Our study developed and validated machine learning models to predict the risk of in-hospital mortality in ICU patients with AF.

METHODS AND RESULTS

Medical Information Mart for Intensive Care (MIMIC)-IV dataset and eICU Collaborative Research Database (eICU-CRD) were analyzed. Among ten classifiers compared, adaptive boosting (AdaBoost) showed better performance in predicting all-cause mortality in AF patients. A compact model with 15 features was developed and validated. Both the all variable and compact models exhibited excellent performance with area under the receiver operating characteristic curves (AUCs) of 1(95%confidence interval [CI]: 1.0-1.0) in the training set. In the MIMIC-IV testing set, the AUCs of the all variable and compact models were 0.978 (95% CI: 0.973-0.982) and 0.977 (95% CI: 0.972-0.982), respectively. In the external validation set, the AUCs of all variable and compact models were 0.825 (95% CI: 0.815-0.834) and 0.807 (95% CI: 0.796-0.817), respectively.

CONCLUSION

An AdaBoost-based predictive model was subjected to internal and external validation, highlighting its strong predictive capacity for assessing the risk of in-hospital mortality in ICU patients with AF.

Prediction Models for Perioperative Blood Transfusion in Patients Undergoing Gynecologic Surgery: A Systematic Review

This systematic review aimed to evaluate prediction models for perioperative blood transfusion in patients undergoing gynecologic surgery. Given the inherent risks associated with blood transfusion and the critical need for accurate prediction, this study identified and assessed models based on their development, validation, and predictive performance. The review included five studies encompassing various surgical procedures and approaches. Predicting factors commonly used across these models included preoperative hematocrit, race, surgical route, and uterine fibroid characteristics. However, the review highlighted significant variability in the definition of perioperative periods, a lack of standardization in transfusion criteria, and a high risk of bias in most models due to methodological issues, such as a low number of events per variable, inappropriate handling of continuous and categorical predictors, inappropriate handling of missing data, improper methods of predictor selection, inappropriate measurement methods for model performance, and inadequate evaluations of model overfitting and optimism in model performance. Despite some models demonstrating good discrimination and calibration, the overall quality and external validation of these models were limited. Consequently, there is a clear need for more robust and externally validated models to improve clinical decision-making and patient outcomes in gynecologic surgery. Future research should focus on refining these models, incorporating rigorous validation, and adhering to standardized reporting practices.

Predicting Recovery After Concussion in Pediatric Patients: A Meta-Analysis

CONTEXT

Prognostic prediction models (PPMs) can help clinicians predict outcomes.

OBJECTIVE

To critically examine peer-reviewed PPMs predicting delayed recovery among pediatric patients with concussion.

DATA SOURCES

Ovid Medline, Embase, Ovid PsycInfo, Web of Science Core Collection, Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, Google Scholar.

STUDY SELECTION

The study had to report a PPM for pediatric patients to be used within 28 days of injury to estimate risk of delayed recovery at 28 days to 1 year postinjury. Studies had to have at least 30 participants.

DATA EXTRACTION

The Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies checklist was completed.

RESULTS

Six studies of 13 PPMs were included. These studies primarily reflected male patients in late childhood or early adolescence presenting to an emergency department meeting the Concussion in Sport Group concussion criteria. No study authors used the same outcome definition nor evaluated the clinical utility of a model. All studies demonstrated high risk of bias. Quality of evidence was best for the Predicting and Preventing Postconcussive Problems in Pediatrics (5P) clinical risk score.

LIMITATIONS

No formal PPM Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) process exists.

CONCLUSIONS

The 5P clinical risk score may be considered for clinical use. Rigorous external validations, particularly in other settings, are needed. The remaining PPMs require external validation. Lack of consensus regarding delayed recovery criteria limits these PPMs.

Diagnostic prediction models for bacterial meningitis in children with a suspected central nervous system infection: a systematic review and prospective validation study

OBJECTIVES

Diagnostic prediction models exist to assess the probability of bacterial meningitis (BM) in paediatric patients with suspected meningitis. To evaluate the diagnostic accuracy of these models in a broad population of children suspected of a central nervous system (CNS) infection, we performed external validation.

METHODS

We performed a systematic literature review in Medline to identify articles on the development, refinement or validation of a prediction model for BM, and validated these models in a prospective cohort of children aged 0-18 years old suspected of a CNS infection.

PRIMARY AND SECONDARY OUTCOME MEASURES

We calculated sensitivity, specificity, predictive values, the area under the receiver operating characteristic curve (AUC) and evaluated calibration of the models for diagnosis of BM.

RESULTS

In total, 23 prediction models were validated in a cohort of 450 patients suspected of a CNS infection included between 2012 and 2015. In 75 patients (17%), the final diagnosis was a CNS infection including 30 with BM (7%). AUCs ranged from 0.69 to 0.94 (median 0.83, interquartile range [IQR] 0.79-0.87) overall, from 0.74 to 0.96 (median 0.89, IQR 0.82-0.92) in children aged ≥28 days and from 0.58 to 0.91 (median 0.79, IQR 0.75-0.82) in neonates.

CONCLUSIONS

Prediction models show good to excellent test characteristics for excluding BM in children and can be of help in the diagnostic workup of paediatric patients with a suspected CNS infection, but cannot replace a thorough history, physical examination and ancillary testing.

Gluten-Free Diet Adherence Tools for Individuals with Celiac Disease: A Systematic Review and Meta-Analysis of Tools Compared to Laboratory Tests

This systematic review aimed to find the tool that best predicts celiac individuals' adherence to a gluten-free diet (GFD). The Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis (TRIPOD-SRMA) guideline was used for the construction and collection of data from eight scientific databases (PubMed, EMBASE, LILACS, Web of Science, LIVIVO, SCOPUS, Google Scholar, and Proquest) on 16 November 2023. The inclusion criteria were studies involving individuals with celiac disease (CD) who were over 18 years old and on a GFD for at least six months, using a questionnaire to predict adherence to a GFD, and comparing it with laboratory tests (serological tests, gluten immunogenic peptide-GIP, or biopsy). Review articles, book chapters, and studies without sufficient data were excluded. The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) was used for data collection from the selected primary studies, and their risk of bias and quality was assessed using the Prediction Risk of Bias Assessment Tool (PROBAST). The association between the GFD adherence determined by the tool and laboratory test was assessed using the phi contingency coefficient. The studies included in this review used four different tools to evaluate GFD adherence: BIAGI score, Coeliac Dietary Adherence Test (CDAT), self-report questions, and interviews. The comparison method most often used was biopsy (n = 19; 59.3%), followed by serology (n = 14; 43.7%) and gluten immunogenic peptides (GIPs) (n = 4; 12.5%). There were no significant differences between the interview, self-report, and BIAGI tools used to evaluate GFD adherence. These tools were better associated with GFD adherence than the CDAT. Considering their cost, application time, and prediction capacity, the self-report and BIAGI were the preferred tools for evaluating GFD adherence.

Performance of risk prediction models for diabetic foot ulcer: a meta-analysis

Background

The number of prediction models for diabetic foot ulcer (DFU) risk is increasing, but their methodological quality and clinical applicability are uncertain. We conducted a systematic review to assess their performance.

Methods

We searched PubMed, Cochrane Library, and Embase databases up to 10 February 2024 and extracted relevant information from selected prediction models. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) checklist was used to assess bias risk and applicability. All statistical analyses were conducted in Stata 14.0.

Results

Initially, 13,562 studies were retrieved, leading to the inclusion of five development and five validation models from eight studies. DFU incidence ranged from 6% to 16.8%, with age and hemoglobin A1C (HbA1c) commonly used as predictive factors. All included studies had a high risk of bias, mainly due to disparities in population characteristics and methodology. In the meta-analysis, we observed area under the curve (AUC) values of 0.78 (95% CI [0.69-0.89]) for development models and 0.84 (95% CI [0.79-0.90]) for validation models.

Conclusion

DFU risk prediction models show good overall accuracy, but there is a risk of bias. Adherence to the PROBAST checklist is crucial for improving their clinical applicability.

Predictive models of Alzheimer's disease dementia risk in older adults with mild cognitive impairment: a systematic review and critical appraisal

BACKGROUND

Mild cognitive impairment has received widespread attention as a high-risk population for Alzheimer's disease, and many studies have developed or validated predictive models to assess it. However, the performance of the model development remains unknown.

OBJECTIVE

The objective of this review was to provide an overview of prediction models for the risk of Alzheimer's disease dementia in older adults with mild cognitive impairment.

METHOD

PubMed, EMBASE, Web of Science, and MEDLINE were systematically searched up to October 19, 2023. We included cohort studies in which risk prediction models for Alzheimer's disease dementia in older adults with mild cognitive impairment were developed or validated. The Predictive Model Risk of Bias Assessment Tool (PROBAST) was employed to assess model bias and applicability. Random-effects models combined model AUCs and calculated (approximate) 95% prediction intervals for estimations. Heterogeneity across studies was evaluated using the I2 statistic, and subgroup analyses were conducted to investigate sources of heterogeneity. Additionally, funnel plot analysis was utilized to identify publication bias.

RESULTS

The analysis included 16 studies involving 9290 participants. Frequency analysis of predictors showed that 14 appeared at least twice and more, with age, functional activities questionnaire, and Mini-mental State Examination scores of cognitive functioning being the most common predictors. From the studies, only two models were externally validated. Eleven studies ultimately used machine learning, and four used traditional modelling methods. However, we found that in many of the studies, there were problems with insufficient sample sizes, missing important methodological information, lack of model presentation, and all of the models were rated as having a high or unclear risk of bias. The average AUC of the 15 best-developed predictive models was 0.87 (95% CI: 0.83, 0.90).

DISCUSSION

Most published predictive modelling studies are deficient in rigour, resulting in a high risk of bias. Upcoming research should concentrate on enhancing methodological rigour and conducting external validation of models predicting Alzheimer's disease dementia. We also emphasize the importance of following the scientific method and transparent reporting to improve the accuracy, generalizability and reproducibility of study results.

REGISTRATION

This systematic review was registered in PROSPERO (Registration ID: CRD42023468780).

Palliative prognostic scores for survival prediction of cancer patients: a systematic review and meta-analysis

BACKGROUND

The palliative prognostic score is the most widely validated prognostic tool for cancer survival prediction, with modified versions available. A systematic evaluation of palliative prognostic score tools is lacking. This systematic review and meta-analysis aimed to evaluate the performance and prognostic utility of palliative prognostic score, delirium-palliative prognostic score, and palliative prognostic score without clinician prediction in predicting 30-day survival of cancer patients and to compare their performance.

METHODS

Six databases were searched for peer-reviewed studies and grey literature published from inception to June 2, 2023. English studies must assess palliative prognostic score, delirium-palliative prognostic score, or palliative prognostic score without clinician-predicted survival for 30-day survival in adults aged 18 years and older with any stage or type of cancer. Outcomes were pooled using the random effects model or summarized narratively when meta-analysis was not possible.

RESULTS

A total of 39 studies (n = 10 617 patients) were included. Palliative prognostic score is an accurate prognostic tool (pooled area under the curve [AUC] = 0.82, 95% confidence interval [CI] = 0.79 to 0.84) and outperforms palliative prognostic score without clinician-predicted survival (pooled AUC = 0.74, 95% CI = 0.71 to 0.78), suggesting that the original palliative prognostic score should be preferred. The meta-analysis found palliative prognostic score and delirium-palliative prognostic score performance to be comparable. Most studies reported survival probabilities corresponding to the palliative prognostic score risk groups, and higher risk groups were statistically significantly associated with shorter survival.

CONCLUSIONS

Palliative prognostic score is a validated prognostic tool for cancer patients that can enhance clinicians' confidence and accuracy in predicting survival. Future studies should investigate if accuracy differs depending on clinician characteristics. Reporting of validation studies must be improved, as most studies were at high risk of bias, primarily because calibration was not assessed.

Clinical validation of the "Straight-Leg-Evaluation-Trauma-Test" (SILENT) as a rapid assessment tool for injuries of the lower extremity in trauma bay patients

PURPOSE

Clinical assessment of the major trauma patient follows international validated guidelines without standardized trauma-specific assessment of the lower extremities for injuries. This study aimed to validate a novel clinical test for lower extremity evaluation during trauma resuscitation phase.

METHODS

This diagnostic, prognostic observational cohort study was performed on trauma patient treated at one level I trauma center between Mar 2022 and Mar 2023. The Straight-Leg-Evaluation-Trauma (SILENT) test follows three steps during the primary survey: inspection for obvious fractures (e.g., open fracture), active elevation of the leg, and cautious elevation of the lower extremity from the heel. SILENT was considered positive when obvious fracture was present and painful or pathological mobility was observed. The SILENT test was compared with standardized radiographs (CT scan or X-ray) as the reference test for fractures. Statistical analysis included sensitivity, specificity, and receiver operating characteristic testing.

RESULTS

403 trauma bay patients were included, mean age 51.6 (SD 21.2) years with 83 fractures of the lower extremity and 27 pelvic/acetabular fractures. Overall sensitivity was 75% (95%CI 64 to 84%), and overall specificity was 99% (95%CI 97 to 100%). Highest sensitivity was for detection of tibia fractures (93%, 95%CI 77 to 99%). Sensitivity of SILENT was higher in the unconscious patient (96%, 95%CI 78 to 100%) with a near 100% specificity. AUC was highest for tibia fractures (0.96, 95%CI 0.92 to 1.0) followed by femur fractures (0.92, 95%CI 0.84 to 0.99).

CONCLUSION

The SILENT test is a clinical applicable and feasible rule-out test for relevant injuries of the lower extremity. A negative SILENT test of the femur or the tibia might reduce the requirement of additional radiological imaging. Further large-scale prospective studies might be required to corroborate the beneficial effects of the SILENT test.

Construction of a nomogram based on clinicopathologic features to predict the likelihood of No. 253 lymph node metastasis in rectal cancer patients

PURPOSE

To explore the high-risk factors for rectal cancer No.253 lymph node metastasis (LNM) and to construct a risk nomogram for the individualized prediction of No.253 LNM.

METHODS

This was a retrospective analysis of 425 patients with rectal cancer who underwent laparoscopic-assisted radical surgery. Independent risk factors for rectal cancer No.253 LNM was identified using multivariate logistic regression analysis, and a risk prediction nomogram was constructed based on the independent risk factors. In addition, the performance of the model was evaluated by discrimination, calibration, and clinical benefit.

RESULTS

Multivariate logistic regression analysis showed that No.253 lymphadenectasis on CT (OR 10.697, P < 0.001), preoperative T4-stage (OR 4.431, P = 0.001), undifferentiation (OR 3.753, P = 0.004), and preoperative Ca199 level > 27 U/ml (OR 2.628, P = 0.037) were independent risk factors for No.253 LNM. A nomogram was constructed based on the above four factors. The calibration curve of the nomogram was closer to the ideal diagonal, indicating that the nomogram had a better fitting ability. The area under the ROC curve (AUC) was 0.865, which indicated that the nomogram had high discriminative ability. In addition, decision curve analysis (DCA) showed that the model could show better clinical benefit when the threshold probability was between 1% and 50%.

CONCLUSION

Preoperative No.253 lymphadenectasis on CT, preoperative T4-stage, undifferentiation, and elevated preoperative Ca199 level were found to be independent risk factors for the No.253 LNM. A predictive model based on these risk factors can help surgeons make rational clinical decisions.

Predicting short- to medium-term care home admission risk in older adults: a systematic review of externally validated models

INTRODUCTION

Predicting risk of care home admission could identify older adults for early intervention to support independent living but require external validation in a different dataset before clinical use. We systematically reviewed external validations of care home admission risk prediction models in older adults.

METHODS

We searched Medline, Embase and Cochrane Library until 14 August 2023 for external validations of prediction models for care home admission risk in adults aged ≥65 years with up to 3 years of follow-up. We extracted and narratively synthesised data on study design, model characteristics, and model discrimination and calibration (accuracy of predictions). We assessed risk of bias and applicability using Prediction model Risk Of Bias Assessment Tool.

RESULTS

Five studies reporting validations of nine unique models were included. Model applicability was fair but risk of bias was mostly high due to not reporting model calibration. Morbidities were used as predictors in four models, most commonly neurological or psychiatric diseases. Physical function was also included in four models. For 1-year prediction, three of the six models had acceptable discrimination (area under the receiver operating characteristic curve (AUC)/c statistic 0.70-0.79) and the remaining three had poor discrimination (AUC < 0.70). No model accounted for competing mortality risk. The only study examining model calibration (but ignoring competing mortality) concluded that it was excellent.

CONCLUSIONS

The reporting of models was incomplete. Model discrimination was at best acceptable, and calibration was rarely examined (and ignored competing mortality risk when examined). There is a need to derive better models that account for competing mortality risk and report calibration as well as discrimination.

Critical appraisal of machine learning prognostic models for acute pancreatitis: protocol for a systematic review

Acute pancreatitis (AP) is an acute inflammatory disorder that is common, costly, and is increasing in incidence worldwide with over 300,000 hospitalizations occurring yearly in the United States alone. As its course and outcomes vary widely, a critical knowledge gap in the field has been a lack of accurate prognostic tools to forecast AP patients' outcomes. Despite several published studies in the last three decades, the predictive performance of published prognostic models has been found to be suboptimal. Recently, non-regression machine learning models (ML) have garnered intense interest in medicine for their potential for better predictive performance. Each year, an increasing number of AP models are being published. However, their methodologic quality relating to transparent reporting and risk of bias in study design has never been systematically appraised. Therefore, through collaboration between a group of clinicians and data scientists with appropriate content expertise, we will perform a systematic review of papers published between January 2021 and December 2023 containing artificial intelligence prognostic models in AP. To systematically assess these studies, the authors will leverage the CHARMS checklist, PROBAST tool for risk of bias assessment, and the most current version of the TRIPOD-AI. (Research Registry ( http://www.reviewregistry1727 .).

Clinical Decision Support System Used in Spinal Disorders: Scoping Review

BACKGROUND

Spinal disorders are highly prevalent worldwide with high socioeconomic costs. This cost is associated with the demand for treatment and productivity loss, prompting the exploration of technologies to improve patient outcomes. Clinical decision support systems (CDSSs) are computerized systems that are increasingly used to facilitate safe and efficient health care. Their applications range in depth and can be found across health care specialties.

OBJECTIVE

This scoping review aims to explore the use of CDSSs in patients with spinal disorders.

METHODS

We used the Joanna Briggs Institute methodological guidance for this scoping review and reported according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) statement. Databases, including PubMed, Embase, Cochrane, CINAHL, Web of Science, Scopus, ProQuest, and PsycINFO, were searched from inception until October 11, 2022. The included studies examined the use of digitalized CDSSs in patients with spinal disorders.

RESULTS

A total of 4 major CDSS functions were identified from 31 studies: preventing unnecessary imaging (n=8, 26%), aiding diagnosis (n=6, 19%), aiding prognosis (n=11, 35%), and recommending treatment options (n=6, 20%). Most studies used the knowledge-based system. Logistic regression was the most commonly used method, followed by decision tree algorithms. The use of CDSSs to aid in the management of spinal disorders was generally accepted over the threat to physicians' clinical decision-making autonomy.

CONCLUSIONS

Although the effectiveness was frequently evaluated by examining the agreement between the decisions made by the CDSSs and the health care providers, comparing the CDSS recommendations with actual clinical outcomes would be preferable. In addition, future studies on CDSS development should focus on system integration, considering end user's needs and preferences, and external validation and impact studies to assess effectiveness and generalizability.

TRIAL REGISTRATION

OSF Registries osf.io/dyz3f; https://osf.io/dyz3f.

Blood levels of glial fibrillary acidic protein for predicting clinical progression to Alzheimer's disease in adults without dementia: a systematic review and meta-analysis protocol

BACKGROUND

There is urgent clinical need to identify reliable prognostic biomarkers that predict the progression of dementia symptoms in individuals with early-phase Alzheimer's disease (AD) especially given the research on and predicted applications of amyloid-beta (Aβ)-directed immunotherapies to remove Aβ from the brain. Cross-sectional studies have reported higher levels of cerebrospinal fluid and blood glial fibrillary acidic protein (GFAP) in individuals with AD-associated dementia than in cognitively unimpaired individuals. Further, recent longitudinal studies have assessed the prognostic potential of baseline blood GFAP levels as a predictor of future cognitive decline in cognitively unimpaired individuals and in those with mild cognitive impairment (MCI) due to AD. In this systematic review and meta-analysis, we propose analyzing longitudinal studies on blood GFAP levels to predict future cognitive decline.

METHODS

This study will include prospective and retrospective cohort studies that assessed blood GFAP levels as a prognostic factor and any prediction models that incorporated blood GFAP levels in cognitively unimpaired individuals or those with MCI. The primary outcome will be conversion to MCI or AD in cognitively unimpaired individuals or conversion to AD in individuals with MCI. Articles from PubMed and Embase will be extracted up to December 31, 2023, without language restrictions. An independent dual screening of abstracts and potentially eligible full-text reports will be conducted. Data will be dual-extracted using the CHeck list for critical appraisal, data extraction for systematic Reviews of prediction Modeling Studies (CHARMS)-prognostic factor, and CHARMS checklists, and we will dual-rate the risk of bias and applicability using the Quality In Prognosis Studies and Prediction Study Risk-of-Bias Assessment tools. We will qualitatively synthesize the study data, participants, index biomarkers, predictive model characteristics, and clinical outcomes. If appropriate, random-effects meta-analyses will be performed to obtain summary estimates. Finally, we will assess the body of evidence using the Grading of Recommendation, Assessment, Development, and Evaluation Approach.

DISCUSSION

This systematic review and meta-analysis will comprehensively evaluate and synthesize existing evidence on blood GFAP levels for prognosticating presymptomatic individuals and those with MCI to help advance risk-stratified treatment strategies for early-phase AD.

TRIAL REGISTRATION

PROSPERO CRD42023481200.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies

Objectives

To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance.

Design

Systematic review and meta-analysis of external validation studies.

Data sources

Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023.

Eligibility criteria for selecting studies

All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed.

Results

47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125).

Conclusions

The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed.

Systematic review registration

PROSPERO CRD42022373182.

Risk prediction models for lung cancer in people who have never smoked: a protocol of a systematic review

BACKGROUND

Lung cancer is one of the most commonly diagnosed cancers and the leading cause of cancer-related death worldwide. Although smoking is the primary cause of the cancer, lung cancer is also commonly diagnosed in people who have never smoked. Currently, the proportion of people who have never smoked diagnosed with lung cancer is increasing. Despite this alarming trend, this population is ineligible for lung screening. With the increasing proportion of people who have never smoked among lung cancer cases, there is a pressing need to develop prediction models to identify high-risk people who have never smoked and include them in lung cancer screening programs. Thus, our systematic review is intended to provide a comprehensive summary of the evidence on existing risk prediction models for lung cancer in people who have never smoked.

METHODS

Electronic searches will be conducted in MEDLINE (Ovid), Embase (Ovid), Web of Science Core Collection (Clarivate Analytics), Scopus, and Europe PMC and Open-Access Theses and Dissertations databases. Two reviewers will independently perform title and abstract screening, full-text review, and data extraction using the Covidence review platform. Data extraction will be performed based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS). The risk of bias will be evaluated independently by two reviewers using the Prediction model Risk-of-Bias Assessment Tool (PROBAST) tool. If a sufficient number of studies are identified to have externally validated the same prediction model, we will combine model performance measures to evaluate the model's average predictive accuracy (e.g., calibration, discrimination) across diverse settings and populations and explore sources of heterogeneity.

DISCUSSION

The results of the review will identify risk prediction models for lung cancer in people who have never smoked. These will be useful for researchers planning to develop novel prediction models, and for clinical practitioners and policy makers seeking guidance for clinical decision-making and the formulation of future lung cancer screening strategies for people who have never smoked.

SYSTEMATIC REVIEW REGISTRATION

This protocol has been registered in PROSPERO under the registration number CRD42023483824.

Diagnostic prediction models to identify patients at risk for healthcare-facility-onset Clostridioides difficile: A systematic review of methodology and reporting

OBJECTIVE

To systematically review the methodology, performance, and generalizability of diagnostic models for predicting the risk of healthcare-facility-onset (HO) Clostridioides difficile infection (CDI) in adult hospital inpatients (aged ≥18 years).

BACKGROUND

CDI is the most common cause of healthcare-associated diarrhea. Prediction models that identify inpatients at risk of HO-CDI have been published; however, the quality and utility of these models remain uncertain.

METHODS

Two independent reviewers evaluated articles describing the development and/or validation of multivariable HO-CDI diagnostic models in an inpatient setting. All publication dates, languages, and study designs were considered. Model details (eg, sample size and source, outcome, and performance) were extracted from the selected studies based on the CHARMS checklist. The risk of bias was further assessed using PROBAST.

RESULTS

Of the 3,030 records evaluated, 11 were eligible for final analysis, which described 12 diagnostic models. Most studies clearly identified the predictors and outcomes but did not report how missing data were handled. The most frequent predictors across all models were advanced age, receipt of high-risk antibiotics, history of hospitalization, and history of CDI. All studies reported the area under the receiver operating characteristic curve (AUROC) as a measure of discriminatory ability. However, only 3 studies reported the model calibration results, and only 2 studies were externally validated. All of the studies had a high risk of bias.

CONCLUSION

The studies varied in their ability to predict the risk of HO-CDI. Future models will benefit from the validation on a prospective external cohort to maximize external validity.

Nomograms for predicting clinically significant prostate cancer in men with PI-RADS-3 biparametric magnetic resonance imaging

This study aimed to construct nomograms for predicting the likelihood of clinically significant prostate cancer (csPCa) in patients with lesions rated as Prostate Imaging Reporting and Data System (PI-RADS) 3 on biparametric magnetic resonance imaging (bpMRI). We retrospectively analyzed a cohort of 457 patients from the Peking Union Medical College Hospital (January 2017-July 2021) to develop the model and externally validated it with a cohort of 238 patients from the Second Hospital of Tianjin Medical University (September 2017-September 2021). Univariate and multivariate logistic regression analyses identified significant predictors of csPCa, defined by tumor volumes ≥ 0.5 cm3, Gleason score ≥ 7, or presence of extracapsular extension. Diagnostic performance for the peripheral zone (PZ) and transitional zone (TZ) was compared using the receiver operating characteristic (ROC) curve and decision curve analysis (DCA). Through univariate and multivariate logistic regression analyses, we identified age, prostate-specific antigen (PSA), and prostate volume (PV) as predictors of csPCa for the PZ, and age, serum-free to total PSA ratio (f/t PSA), and PSA density (PSAD) for the TZ. The nomograms demonstrated robust discriminative ability, with an area under the ROC curve (AUC) of 0.819 for PZ and 0.804 for TZ. The external validation corroborated the model's high predictive accuracy (AUC of 0.831 for PZ and 0.773 for TZ). Calibration curves indicated excellent agreement between predicted and observed outcomes, and DCA underscored the nomogram's clinical utility for both PZ and TZ. Overall, the nomograms offer high predictive accuracy for csPCa at initial biopsy, potentially reducing unnecessary biopsies in clinical settings.

Critical evaluation of established risk prediction models for acute respiratory distress syndrome in adult patients: A systematic review and meta-analysis

AIM

To assess the performance of validated prediction models for acute respiratory distress syndrome (ARDS) by systematic review and meta-analysis.

METHODS

Eight databases (Medline, CINAHL, Embase, The Cochrane Library, CNKI, WanFang Data, Sinomed, and VIP) were searched up to March 26, 2023. Studies developed and validated a prediction model for ARDS in adult patients were included. Items on study design, incidence, derivation methods, predictors, discrimination, and calibration were collected. The risk of bias was assessed by the Prediction model Risk of Bias Assessment Tool. Models with a reported area under the curve of the receiver operating characteristic (AUC) metric were analyzed.

RESULTS

A total of 25 studies were retrieved, including 48 unique prediction models. Discrimination was reported in all studies, with AUC ranging from 0.701 to 0.95. Emerged AUC value of the logistic regression model was 0.837 (95% CI: 0.814 to 0.859). Besides, the value in the ICU group was 0.856 (95% CI: 0.812 to 0.899), the acute pancreatitis group was 0.863 (95% CI: 0.844 to 0.882), and the postoperation group was 0.835 (95% CI: 0.808 to 0.861). In total, 24 of the included studies had a high risk of bias, which was mostly due to the improper methods in predictor screening (13/24), model calibration assessment (9/24), and dichotomization of continuous predictors (6/24).

CONCLUSIONS

This study shows that most prediction models for ARDS are at high risk of bias, and the discrimination ability of the model is excellent. Adherence to standardized guidelines for model development is necessary to derive a prediction model of value to clinicians.

Predicting pediatric emergence delirium using data-driven machine learning applied to electronic health record dataset at a quaternary care pediatric hospital

Objectives

Pediatric emergence delirium is an undesirable outcome that is understudied. Development of a predictive model is an initial step toward reducing its occurrence. This study aimed to apply machine learning (ML) methods to a large clinical dataset to develop a predictive model for pediatric emergence delirium.

Materials and Methods

We performed a single-center retrospective cohort study using electronic health record data from February 2015 to December 2019. We built and evaluated 4 commonly used ML models for predicting emergence delirium: least absolute shrinkage and selection operator, ridge regression, random forest, and extreme gradient boosting. The primary outcome was the occurrence of emergence delirium, defined as a Watcha score of 3 or 4 recorded at any time during recovery.

Results

The dataset included 54 776 encounters across 43 830 patients. The 4 ML models performed similarly with performance assessed by the area under the receiver operating characteristic curves ranging from 0.74 to 0.75. Notable variables associated with increased risk included adenoidectomy with or without tonsillectomy, decreasing age, midazolam premedication, and ondansetron administration, while intravenous induction and ketorolac were associated with reduced risk of emergence delirium.

Conclusions

Four different ML models demonstrated similar performance in predicting postoperative emergence delirium using a large pediatric dataset. The prediction performance of the models draws attention to our incomplete understanding of this phenomenon based on the studied variables. The results from our modeling could serve as a first step in designing a predictive clinical decision support system, but further optimization and validation are needed.

Clinical trial number and registry URL

Not applicable.

Prognostic models predicting transition to psychotic disorder using blood-based biomarkers: a systematic review and critical appraisal

Accumulating evidence suggests individuals with psychotic disorder show abnormalities in metabolic and inflammatory processes. Recently, several studies have employed blood-based predictors in models predicting transition to psychotic disorder in risk-enriched populations. A systematic review of the performance and methodology of prognostic models using blood-based biomarkers in the prediction of psychotic disorder from risk-enriched populations is warranted. Databases (PubMed, EMBASE and PsycINFO) were searched for eligible texts from 1998 to 15/05/2023, which detailed model development or validation studies. The checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was used to guide data extraction from eligible texts and the Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias and applicability of the studies. A narrative synthesis of the included studies was performed. Seventeen eligible studies were identified: 16 eligible model development studies and one eligible model validation study. A wide range of biomarkers were assessed, including nucleic acids, proteins, metabolites, and lipids. The range of C-index (area under the curve) estimates reported for the models was 0.67-1.00. No studies assessed model calibration. According to PROBAST criteria, all studies were at high risk of bias in the analysis domain. While a wide range of potentially predictive biomarkers were identified in the included studies, most studies did not account for overfitting in model performance estimates, no studies assessed calibration, and all models were at high risk of bias according to PROBAST criteria. External validation of the models is needed to provide more accurate estimates of their performance. Future studies which follow the latest available methodological and reporting guidelines and adopt strategies to accommodate required sample sizes for model development or validation will clarify the value of including blood-based biomarkers in models predicting psychosis.

Prognostic prediction models for clinical outcomes in patients diagnosed with visceral leishmaniasis: protocol for a systematic review

INTRODUCTION

Visceral leishmaniasis (VL) is a neglected tropical disease responsible for many thousands of preventable deaths each year. Symptomatic patients often struggle to access effective treatment, without which death is the norm. Risk prediction tools support clinical teams and policymakers in identifying high-risk patients who could benefit from more intensive management pathways. Investigators interested in using their clinical data for prognostic research should first identify currently available models that are candidates for validation and possible updating. Addressing these needs, we aim to identify, summarise and appraise the available models predicting clinical outcomes in VL patients.

METHODS AND ANALYSIS

We will include studies that have developed, validated or updated prognostic models predicting future clinical outcomes in patients diagnosed with VL. Systematic reviews and meta-analyses that include eligible studies are also considered for review. Conference abstracts and educational theses are excluded. Data extraction, appraisal and reporting will follow current methodological guidelines. Ovid Embase; Ovid MEDLINE; the Web of Science Core Collection, SciELO and LILACS are searched from database inception to 1 March 2023 using terms developed for the identification of prediction models, and with no language restriction. Screening, data extraction and risk of bias assessment will be performed in duplicate with discordance resolved by a third independent reviewer. Risk of bias will be assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Tables and figures will compare and contrast key model information, including source data, participants, model development and performance measures, and risk of bias. We will consider the strengths, limitations and clinical applicability of the identified models.

ETHICS AND DISSEMINATION

Ethics approval is not required for this review. The systematic review and all accompanying data will be submitted to an open-access journal. Findings will also be disseminated through the research group's website (www.iddo.org/research-themes/visceral-leishmaniasis) and social media channels.

PROSPERO REGISTRATION NUMBER

CRD42023417226.

Predicting Extended-Spectrum Beta-Lactamase and Carbapenem Resistance in Enterobacteriaceae Bacteremia: A Diagnostic Model Systematic Review and Meta-Analysis

Enterobacteriaceae bacteremia, particularly when associated with antimicrobial resistance, can result in increased mortality, emphasizing the need for timely effective therapy. Clinical risk prediction models are promising tools, stratifying patients based on their risk of resistance due to ESBL and carbapenemase-producing Enterobacteriaceae in bloodstream infections (BSIs) and, thereby, improving therapeutic decisions. This systematic review and meta-analysis synthesized the literature on the performance of these models. Searches of PubMed and EMBASE led to the identification of 10 relevant studies with 6106 unique patient encounters. Nine studies concerned ESBL prediction, and one focused on the prediction of carbapenemases. For the two ESBL model derivation studies, the discrimination performance showed sensitivities of 53-85% and specificities of 93-95%. Among the four ESBL model derivation and validation studies, the sensitivities were 43-88%, and the specificities were 77-99%. The sensitivity and specificity for the subsequent external validation studies were 7-37% and 88-96%, respectively. For the three external validation studies, only two models were evaluated across multiple studies, with a pooled AUROC of 65-71%, with one study omitting the sensitivity/specificity. Only two studies measured clinical utility through hypothetical therapy assessments. Given the limited evidence on their interventional application, it would be beneficial to further assess these or future models, to better understand their clinical utility and ensure their safe and impactful implementation.

Clinical-radiomics nomogram using contrast-enhanced CT to predict histological grade and survival in pancreatic ductal adenocarcinoma

Objectives

Tumor grading is important for prognosis of pancreatic ductal adenocarcinoma (PDAC). In this study, we developed preoperative clinical-radiomics nomograms using features from contrast-enhanced CT (CECT), to discriminate high-grade and low-grade PDAC and predict overall survival (OS).

Methods

In this single-center, retrospective study conducted from February 2014 to April 2021, consecutive PDAC patients who underwent CECT and had pathologically identified grading were randomized to training (n=200) and test (n=84) cohorts for development of model to predict histological grade based on radiomics scores from CECT (HGrad). Another 42 patients were used as external validation cohort of HGrad. A nomogram (HGnom) was constructed using radiomics score, CA12-5 and smoking to predict histological grade. A second nomogram (Pnom) was constructed using radiomics score, CA12-5, TNM, adjuvant treatment, resection margin and microvascular invasion to predict OS in radical resection patients (217 of 284).

Results

Among 326 patients, 122 were high-grade (120 poorly differentiated and 2 undifferentiated). The HGrad yielded AUCs of 0.75 (95% CI: 0.64, 0.85) and 0.76 (95% CI: 0.60, 0.91) in test and validation cohorts. The HGnom achieved AUCs of 0.77 (95% CI: 0.66, 0.87), and the predicted grades calibrated well with actual grades (P=.13). OS was different between the grades predicted by radiomics scores (P=.01). The integrated AUC of the Pnom for predicting OS was 0.80 (95% CI: 0.75, 0.88).

Conclusion

Compared with the HGrad using features from CECT, the HGnom demonstrated higher performance for predicting histological grade. The Pnom helped identify patients with high survival outcome in pancreatic ductal adenocarcinoma.