Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Prediction models for COVID-19 disease outcomes

SARS-CoV-2 has caused over 6.9 million deaths and continues to produce lasting health consequences. COVID-19 manifests broadly from no symptoms to death. In a retrospective cross-sectional study, we developed personalized risk assessment models that predict clinical outcomes for individuals with COVID-19 and inform targeted interventions. We sequenced viruses from SARS-CoV-2-positive nasopharyngeal swab samples between July 2020 and July 2022 from 4450 individuals in Missouri and retrieved associated disease courses, clinical history, and urban-rural classification. We integrated this data to develop machine learning-based predictive models to predict hospitalization, ICU admission, and long COVID.The mean age was 38.3 years (standard deviation = 21.4) with 55.2% (N = 2453) females and 44.8% (N = 1994) males (not reported, N = 4). Our analyses revealed a comprehensive set of predictors for each outcome, encompassing human, environment, and virus genome-wide genetic markers. Immunosuppression, cardiovascular disease, older age, cardiac, gastrointestinal, and constitutional symptoms, rural residence, and specific amino acid substitutions were associated with hospitalization. ICU admission was associated with acute respiratory distress syndrome, ventilation, bacterial co-infection, rural residence, and non-wild type SARS-CoV-2 variants. Finally, long COVID was associated with hospital admission, ventilation, and female sex.Overall, we developed risk assessment models that offer the capability to identify patients with COVID-19 necessitating enhanced monitoring or early interventions. Of importance, we demonstrate the value of including key elements of virus, host, and environmental factors to predict patient outcomes, serving as a valuable platform in the field of personalized medicine with the potential for adaptation to other infectious diseases.

Risk Assessment Tool in Predicting the Therapeutic Outcomes of Antiseizure Medication in Adults with Epilepsy

Aim

Identifying a patient's risk for poor outcomes after starting antiseizure medication (ASM) therapy is crucial in managing epilepsy pharmacologically. To date, there is a lack of designated tools to assess such risks.

Purpose

To develop and validate a risk assessment tool for the therapeutic outcomes of ASM therapy.

Patients and Methods

A cross-sectional study was carried out in a hospital-based specialist clinic from September 2022 to August 2023. Data was analyzed from patients' medical records and face-to-face assessments. The seizure control domain was determined from the patients' medical records while seizure severity (SS) and adverse effects (AE) of ASM were assessed using the Seizure Severity Questionnaire and the Liverpool Adverse Event Profile respectively. The developed tool was devised from prediction models using logistic and linear regressions. Concurrent validity and interrater reliability methods were employed for validity assessments.

Results

A total of 397 patients were included in the analysis. For seizure control, the identified predictors include ≥10 years' epilepsy duration (OR:1.87,95% CI:1.10-3.17), generalized onset (OR:7.42,95% CI:2.95-18.66), focal onset seizure (OR:8.24,95% CI:2.98-22.77), non-adherence (OR:3.55,95% CI:1.52-8.27) and having ≥3 ASM (OR:3.29,95% CI:1.32-8.24). Younger age at epilepsy onset (≤40) (OR:3.29,95% CI:1.32-8.24) and neurological deficit (OR:3.55,95% CI:1.52-8.27) were significant predictors for SS. For AE, the positive predictors were age >35 (OR:0.12,95% CI:0.03-0.20), <13 years epilepsy duration (OR:2.89,95% CI:0.50-5.29) and changes in ASM regimen (OR:2.93,95% CI: 0.24-5.62). The seizure control domain showed a good discriminatory ability with a c-index of 0.711. From the Bonferroni (ANOVA) analysis, only SS predicted scores generated a linear plot against the mean of the actual scores. The AE domain was omitted from the final tool because it did not meet the requirements for validity assessment.

Conclusion

This newly developed tool (RAS-TO) is a promising tool that could help healthcare providers in determining optimal treatment strategies for adults with epilepsy.

External validation of Finnish diabetes risk score (FINDRISC) and Latin American FINDRISC for screening of undiagnosed dysglycemia: Analysis in a Peruvian hospital health care workers sample

AIMS

To evaluate the external validity of Finnish diabetes risk score (FINDRISC) and Latin American FINDRISC (LAFINDRISC) for undiagnosed dysglycemia in hospital health care workers.

METHODS

We carried out a cross-sectional study on health workers without a prior history of diabetes mellitus (DM). Undiagnosed dysglycemia (prediabetes or diabetes mellitus) was defined using fasting glucose and two-hour oral glucose tolerance test. LAFINDRISC is an adapted version of FINDRISC with different waist circumference cut-off points. We calculated the area under the receptor operational characteristic curve (AUROC) and explored the best cut-off point.

RESULTS

We included 549 participants in the analysis. The frequency of undiagnosed dysglycemia was 17.8%. The AUROC of LAFINDRISC and FINDRISC were 71.5% and 69.2%; p = 0.007, respectively. The optimal cut-off for undiagnosed dysglycemiaaccording to Index Youden was ≥ 11 in LAFINDRISC (Sensitivity: 78.6%; Specificity: 51.7%) and ≥12 in FINDRISC (Sensitivity: 70.4%; Specificity: 53.9%).

CONCLUSION

The discriminative capacity of both questionnaires is good for the diagnosis of dysglycemia in the healthcare personnel of the María Auxiliadora hospital. The LAFINDRISC presented a small statistical difference, nontheless clinically similar, since there was no difference by age or sex. Further studies in the general population are required to validate these results.

Point-of-Care Noninvasive Prediction of Liver-Related Events in Patients With Nonalcoholic Fatty Liver Disease

BACKGROUND & AIMS

Individual risk prediction of liver-related events (LRE) is needed for clinical assessment of nonalcoholic fatty liver disease (NAFLD)/nonalcoholic steatohepatitis (NASH) patients. We aimed to provide point-of-care validated liver stiffness measurement (LSM)-based risk prediction models for the development of LRE in patients with NAFLD, focusing on selecting patients for clinical trials at risk of clinical events.

METHODS

Two large multicenter cohorts were evaluated, 2638 NAFLD patients covering all LSM values as the derivation cohort and 679 more advanced patients as the validation cohort. We used Cox regression to develop and validate risk prediction models based on LSM alone, and the ANTICIPATE and ANTICIPATE-NASH models for clinically significant portal hypertension. The main outcome of the study was the rate of LRE in the first 3 years after initial assessment.

RESULTS

The 3 predictive models had similar performance in the derivation cohort with a very high discriminative value (c-statistic, 0.87-0.91). In the validation cohort, the LSM-LRE alone model had a significant inferior discrimination (c-statistic, 0.75) compared with the other 2 models, whereas the ANTICIPATE-NASH-LRE model (0.81) was significantly better than the ANTICIPATE-LRE model (0.79). In addition, the ANTICIPATE-NASH-LRE model presented very good calibration in the validation cohort (integrated calibration index, 0.016), and was better than the ANTICIPATE-LRE model.

CONCLUSIONS

The ANTICIPATE-LRE models, and especially the ANTICIPATE-NASH-LRE model, could be valuable validated clinical tools to individually assess the risk of LRE at 3 years in patients with NAFLD/NASH.

Injury severity bias in missing prehospital vital signs: Prevalence and implications for trauma registries

BACKGROUND

Vital signs are important factors in assessing injury severity and guiding trauma resuscitation, especially among severely injured patients. Despite this, physiological data are frequently missing from trauma registries. This study aimed to evaluate the extent of missing prehospital data in a hospital-based trauma registry and to assess the associations between prehospital physiological data completeness and indicators of injury severity.

METHODS

A retrospective review was conducted on all adult trauma patients brought directly to a level 1 trauma center in Toronto, Ontario by paramedics from January 1, 2015, to December 31, 2019. The proportion of missing data was evaluated for each variable and patterns of missingness were assessed. To investigate the associations between prehospital data completeness and injury severity factors, descriptive and unadjusted logistic regression analyses were performed.

RESULTS

A total of 3,528 patients were included. We considered prehospital data missing if any of heart rate, systolic blood pressure, respiratory rate or oxygen saturation were incomplete. Each individual variable was missing from the registry in approximately 20 % of patients, with oxygen saturation missing most frequently (n = 831; 23.6 %). Over 25 % (n = 909) of patients were missing at least one prehospital vital sign, of which 69.1 % (n = 628) were missing all four of these variables. Patients with incomplete data were more severely injured, had higher mortality, and more frequently received lifesaving interventions such as blood transfusion and intubation. Patients were most likely to have missing prehospital physiological data if they died in the trauma bay (unadjusted OR: 9.79; 95 % CI: 6.35-15.10), did not survive to discharge (unadjusted OR: 3.55; 95 % CI: 2.76-4.55), or had a prehospital GCS less than 9 (OR: 3.24; 95 % CI: 2.59-4.06).

CONCLUSION

In this single center trauma registry, key prehospital variables were frequently missing, particularly among more severely injured patients. Patients with missing data had higher mortality, more severe injury characteristics and received more life-saving interventions in the trauma bay, suggesting an injury severity bias in prehospital vital sign missingness. To ensure the validity of research based on trauma registry data, patterns of missingness must be carefully considered to ensure missing data is appropriately addressed.

Protein Biomarkers in Lung Cancer Screening: Technical Considerations and Feasibility Assessment

Lung cancer remains the leading cause of cancer-related deaths worldwide, mainly due to late diagnosis and the presence of metastases. Several countries around the world have adopted nation-wide LDCT-based lung cancer screening that will benefit patients, shifting the stage at diagnosis to earlier stages with more therapeutic options. Biomarkers can help to optimize the screening process, as well as refine the TNM stratification of lung cancer patients, providing information regarding prognostics and recommending management strategies. Moreover, novel adjuvant strategies will clearly benefit from previous knowledge of the potential aggressiveness and biological traits of a given early-stage surgically resected tumor. This review focuses on proteins as promising biomarkers in the context of lung cancer screening. Despite great efforts, there are still no successful examples of biomarkers in lung cancer that have reached the clinics to be used in early detection and early management. Thus, the field of biomarkers in early lung cancer remains an evident unmet need. A more specific objective of this review is to present an up-to-date technical assessment of the potential use of protein biomarkers in early lung cancer detection and management. We provide an overview regarding the benefits, challenges, pitfalls and constraints in the development process of protein-based biomarkers. Additionally, we examine how a number of emerging protein analytical technologies may contribute to the optimization of novel robust biomarkers for screening and effective management of lung cancer.

Grasping the big picture: impact analysis of screening tools for timely referral for device-aided therapies

Several screening tools are available to assist general neurologists in the timely identification of patients with advanced Parkinson's disease (PD) who may be eligible for referral for a device-aided therapy (DAT). However, it should be noted that not all of these clinical decision rules have been developed and validated in a thorough and consistent manner. Furthermore, only a limited number of head-to-head comparisons have been performed. Available studies suggest that D-DATS has a higher positive predictive value and higher specificity than the 5-2-1 criteria, while the sensitivity of both screening tools is similar. However, unanswered questions remain regarding the validity of the decision rules, such as whether the diagnostic performance measures from validation studies are generalizable to other populations. Ultimately, the question is whether a screening tool will effectively and efficiently improve the quality of life of patients with PD. To address this key question, an impact analysis should be performed. The authors intend to set up a multinational cluster randomised controlled trial to compare the D-DATS and 5-2-1 criteria on the downstream consequences of implementing these screening tools, with a particular focus on the impact on disability and quality of life.

Evaluating classification tools for the prediction of in-vitro microbial pyruvate yield from organic carbon sources

The laboratory-scale (in-vitro) microbial fermentation based on screening of process parameters (factors) and statistical validation of parameters (responses) using regression analysis. The recent trends have shifted from full factorial design towards more complex response surface methodology designs such as Box-Behnken design, Central Composite design. Apart from the optimisation methodologies, the listed designs are not flexible enough in deducing properties of parameters in terms of class variables. Machine learning algorithms have unique visualisations for the dataset presented with appropriate learning algorithms. The classification algorithms cannot be applied on all datasets and selection of classifier is essential in this regard. To resolve this issue, factor-response relationship needs to be evaluated as dataset and subsequent preprocessing could lead to appropriate results. The aim of the current study was to investigate the data-mining accuracy on the dataset developed using in-vitro pyruvate production using organic sources for the first time. The attributes were subjected to comparative classification on various classifiers and based on accuracy, multilayer perceptron (neural network algorithm) was selected as classifier. As per the results, the model showed significant results for prediction of classes and a good fit. The learning curve developed also showed the datasets converging and were linearly separable.

Prediction models for diagnosis and prognosis of the colonization or infection of multidrug-resistant organisms in adults: a systematic review, critical appraisal, and meta-analysis

BACKGROUND

Prediction models help to target patients at risk of multidrug-resistant organism (MDRO) colonization or infection and could serve as tools informing clinical practices to prevent MDRO transmission and inappropriate empiric antibiotic therapy. However, there is limited evidence to identify which among the available models are of low risk of bias and suitable for clinical application.

OBJECTIVES

To identify, describe, appraise, and summarise the performance of all prognostic and diagnostic models developed or validated for predicting MDRO colonization or infection.

DATA SOURCES

Six electronic literature databases and clinical registration databases were searched until April 2022.

STUDY ELIGIBILITY CRITERIA

Development and validation studies of any multivariable prognostic and diagnostic models to predict MDRO colonization or infection in adults.

PARTICIPANTS

Adults (≥ 18 years old) without MDRO colonization or infection (in prognostic models) or with unknown or suspected MDRO colonization or infection (in diagnostic models).

ASSESSMENT OF RISK OF BIAS

The Prediction Model Risk of Bias Assessment Tool was used to assess the risk of bias. Evidence certainty was assessed using the Grading of Recommendations Assessment, Development, and Evaluation approach.

METHODS OF DATA SYNTHESIS

Meta-analyses were conducted to summarize the discrimination and calibration of the models' external validations conducted in at least two non-overlapping datasets.

RESULTS

We included 162 models (108 studies) developed for diagnosing (n = 135) and predicting (n = 27) MDRO colonization or infection. Models exhibited a high-risk of bias, especially in statistical analysis. High-frequency predictors were age, recent invasive procedures, antibiotic usage, and prior hospitalization. Less than 25% of the models underwent external validations, with only seven by independent teams. Meta-analyses for one diagnostic and two prognostic models only produced very low to low certainty of evidence.

CONCLUSIONS

The review comprehensively described the models for identifying patients at risk of MDRO colonization or infection. We cannot recommend which models are ready for application because of the high-risk of bias, limited validations, and low certainty of evidence from meta-analyses, indicating a clear need to improve the conducting and reporting of model development and external validation studies to facilitate clinical application.

Identifying Predictor Variables for a Composite Risk Prediction Tool for Gestational Diabetes and Hypertensive Disorders of Pregnancy: A Modified Delphi Study

A composite cardiometabolic risk prediction tool will support the systematic identification of women at increased cardiometabolic risk during pregnancy to enable early screening and intervention. This study aims to identify and select predictor variables for a composite risk prediction tool for cardiometabolic risk (gestational diabetes mellitus and/or hypertensive disorders of pregnancy) for use in the first trimester. A two-round modified online Delphi study was undertaken. A prior systematic literature review generated fifteen potential predictor variables for inclusion in the tool. Multidisciplinary experts (n = 31) rated the clinical importance of variables in an online survey and nominated additional variables for consideration (Round One). An online meeting (n = 14) was held to deliberate the importance, feasibility and acceptability of collecting variables in early pregnancy. Consensus was reached in a second online survey (Round Two). Overall, 24 variables were considered; 9 were eliminated, and 15 were selected for inclusion in the tool. The final 15 predictor variables related to maternal demographics (age, ethnicity/race), pre-pregnancy history (body mass index, height, history of chronic kidney disease/polycystic ovarian syndrome, family history of diabetes, pre-existing diabetes/hypertension), obstetric history (parity, history of macrosomia/pre-eclampsia/gestational diabetes mellitus), biochemical measures (blood glucose levels), hemodynamic measures (systolic blood pressure). Variables will inform the development of a cardiometabolic risk prediction tool in subsequent research. Evidence-based, clinically relevant and routinely collected variables were selected for a composite cardiometabolic risk prediction tool for early pregnancy.

Clinical prediction models in children that use repeated measurements with time-varying covariates: a scoping review

BACKGROUND

Emerging evidence suggests that clinical prediction models that use repeated (time-varying) measurements within each patient may have higher predictive accuracy than models that use patient information from a single measurement.

OBJECTIVE

To determine the breadth of the published literature reporting the development of clinical prediction models in children that use time-varying predictors.

DATA SOURCES

MEDLINE, EMBASE and Cochrane databases.

ELIGIBILITY CRITERIA

We included studies reporting the development of a multivariable clinical prediction model in children, with or without validation, to predict a repeatedly measured binary or time-to-event outcome and utilizing at least one repeatedly measured predictor.

SYNTHESIS METHODS

We categorized included studies by the method used to model time-varying predictors.

RESULTS

Of 99 clinical prediction model studies that had a repeated measurements data structure, only 27 (27%) used methods that incorporated the repeated measurements as time-varying predictors in a single model. Among these 27 time-varying prediction model studies, we grouped model types into nine categories: time-dependent Cox regression, generalized estimating equations, random effects model, landmark model, joint model, neural network, K-nearest neighbor, support vector machine and tree-based algorithms. Where there was comparison of time-varying models to single measurement models, using time-varying predictors improved predictive accuracy.

CONCLUSIONS

Various methods have been used to develop time-varying prediction models in children, but there is a paucity of pediatric time-varying models in the literature. Incorporating time-varying covariates in pediatric prediction models may improve predictive accuracy. Future research in pediatric prediction model development should further investigate whether incorporation of time-varying covariates improves predictive accuracy.

Receiver Operating Characteristic (ROC) Curves: The Basics and Beyond

Diagnostic tests and clinical prediction rules are frequently used to help estimate the probability of a disease or outcome. How well a test or rule distinguishes between disease or no disease (discrimination) can be measured by plotting a receiver operating characteristic (ROC) curve and calculating the area under it (AUROC). In this paper, we review the features of ROC curves and interpretation of ROC curves and AUROC values. We highlight 5 underappreciated features of ROC curves: (1) the slope of the ROC curve over a test result interval is the likelihood ratio for that interval; (2) the optimal cutoff for calling a test positive depends not only on the shape of the ROC curve, but also on the pretest probability of disease and relative harms of false-positive and false-negative results; (3) the AUROC measures discrimination only, not the accuracy of the predicted probabilities; (4) the AUROC is not a good measure of discrimination if the slope of the ROC curve is not consistently decreasing; and (5) the AUROC can be increased by including a large number of people correctly identified as being at very low risk for the outcome of interest. We illustrate this last concept using 3 published studies.

Comparing penalization methods for linear models on large observational health data

OBJECTIVE

This study evaluates regularization variants in logistic regression (L1, L2, ElasticNet, Adaptive L1, Adaptive ElasticNet, Broken adaptive ridge [BAR], and Iterative hard thresholding [IHT]) for discrimination and calibration performance, focusing on both internal and external validation.

MATERIALS AND METHODS

We use data from 5 US claims and electronic health record databases and develop models for various outcomes in a major depressive disorder patient population. We externally validate all models in the other databases. We use a train-test split of 75%/25% and evaluate performance with discrimination and calibration. Statistical analysis for difference in performance uses Friedman's test and critical difference diagrams.

RESULTS

Of the 840 models we develop, L1 and ElasticNet emerge as superior in both internal and external discrimination, with a notable AUC difference. BAR and IHT show the best internal calibration, without a clear external calibration leader. ElasticNet typically has larger model sizes than L1. Methods like IHT and BAR, while slightly less discriminative, significantly reduce model complexity.

CONCLUSION

L1 and ElasticNet offer the best discriminative performance in logistic regression for healthcare predictions, maintaining robustness across validations. For simpler, more interpretable models, L0-based methods (IHT and BAR) are advantageous, providing greater parsimony and calibration with fewer features. This study aids in selecting suitable regularization techniques for healthcare prediction models, balancing performance, complexity, and interpretability.

Medical calculators derived synthetic cohorts: a novel method for generating synthetic patient data

This study shows that we can use synthetic cohorts created from medical risk calculators to gain insights into how risk estimations, clinical reasoning, data-driven subgrouping, and the confidence in risk calculator scores are connected. When prediction variables aren't evenly distributed in these synthetic cohorts, they can be used to group similar cases together, revealing new insights about how cohorts behave. We also found that the confidence in predictions made by these calculators can vary depending on patient characteristics. This suggests that it might be beneficial to include a "normalized confidence" score in future versions of these calculators for healthcare professionals. We plan to explore this idea further in our upcoming research.

Predictors contributing to the estimation of pulmonary tuberculosis among adults in a resource-limited setting: A systematic review of diagnostic predictions

Background

Although tuberculosis is highly prevalent in low- and middle-income countries, millions of cases remain undetected using current diagnostic methods. To address this problem, researchers have proposed prediction rules.

Objective

We analyzed existing prediction rules for the diagnosis of pulmonary tuberculosis and identified factors with a moderate to high strength of association with the disease.

Methods

We conducted a comprehensive search of relevant databases (MEDLINE/PubMed, Cochrane Library, Science Direct, Global Health for Reports, and Google Scholar) up to 14 November 2022. Studies that developed diagnostic algorithms for pulmonary tuberculosis in adults from low and middle-income countries were included. Two reviewers performed study screening, data extraction, and quality assessment. The study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2. We performed a narrative synthesis.

Results

Of the 26 articles selected, only half included human immune deficiency virus-positive patients. In symptomatic human immune deficiency virus patients, radiographic findings and body mass index were strong predictors of pulmonary tuberculosis, with an odds ratio of >4. However, in human immune deficiency virus-negative individuals, the biomarkers showed a moderate association with the disease. In symptomatic human immune deficiency virus patients, a C-reactive protein level ⩾10 mg/L had a sensitivity and specificity of 93% and 40%, respectively, whereas a trial of antibiotics had a specificity of 86% and a sensitivity of 43%. In smear-negative patients, anti-tuberculosis treatment showed a sensitivity of 52% and a specificity of 63%.

Conclusions

The performance of predictors and diagnostic algorithms differs among patient subgroups, such as in human immune deficiency virus-positive patients, radiographic findings, and body mass index were strong predictors of pulmonary tuberculosis. However, in human immune deficiency virus-negative individuals, the biomarkers showed a moderate association with the disease. A few models have reached the World Health Organization's recommendation. Therefore, more work should be done to strengthen the predictive models for tuberculosis screening in the future, and they should be developed rigorously, considering the heterogeneity of the population in clinical work.

Machine Learning-Based Predictive Models for Patients with Venous Thromboembolism: A Systematic Review

BACKGROUND

 Venous thromboembolism (VTE) is a chronic disorder with a significant health and economic burden. Several VTE-specific clinical prediction models (CPMs) have been used to assist physicians in decision-making but have several limitations. This systematic review explores if machine learning (ML) can enhance CPMs by analyzing extensive patient data derived from electronic health records. We aimed to explore ML-CPMs' applications in VTE for risk stratification, outcome prediction, diagnosis, and treatment.

METHODS

 Three databases were searched: PubMed, Google Scholar, and IEEE electronic library. Inclusion criteria focused on studies using structured data, excluding non-English publications, studies on non-humans, and certain data types such as natural language processing and image processing. Studies involving pregnant women, cancer patients, and children were also excluded. After excluding irrelevant studies, a total of 77 studies were included.

RESULTS

 Most studies report that ML-CPMs outperformed traditional CPMs in terms of receiver operating area under the curve in the four clinical domains that were explored. However, the majority of the studies were retrospective, monocentric, and lacked detailed model architecture description and external validation, which are essential for quality audit. This review identified research gaps and highlighted challenges related to standardized reporting, reproducibility, and model comparison.

CONCLUSION

 ML-CPMs show promise in improving risk assessment and individualized treatment recommendations in VTE. Apparently, there is an urgent need for standardized reporting and methodology for ML models, external validation, prospective and real-world data studies, as well as interventional studies to evaluate the impact of artificial intelligence in VTE.

Prediction Models for Successful External Cephalic Version: An Updated Systematic Review

OBJECTIVE

 To review the decision aids currently available or being developed to predict a patient's odds that their external cephalic version (ECV) will be successful.

STUDY DESIGN

 We searched PubMed/MEDLINE, Cochrane Central, and ClinicalTrials.gov from 2015 to 2022. Articles from a pre-2015 systematic review were also included. We selected English-language articles describing or evaluating models (prediction rules) designed to predict an outcome of ECV for an individual patient. Acceptable model outcomes included cephalic presentation after the ECV attempt and whether the ECV ultimately resulted in a vaginal delivery. Two authors independently performed article selection following PRISMA 2020 guidelines. Since 2015, 380 unique records underwent title and abstract screening, and 49 reports underwent full-text review. Ultimately, 17 new articles and 8 from the prior review were included. Of the 25 articles, 22 proposed one to two models each for a total of 25 models, while the remaining 3 articles validated prior models without proposing new ones.

RESULTS

 Of the 17 new articles, 10 were low, 6 moderate, and 1 high risk of bias. Almost all articles were from Europe (11/25) or Asia (10/25); only one study in the last 20 years was from the United States. The models found had diverse presentations including score charts, decision trees (flowcharts), and equations. The majority (13/25) had no form of validation and only 5/25 reached external validation. Only the Newman-Peacock model (United States, 1993) was repeatedly externally validated (Pakistan, 2012 and Portugal, 2018). Most models (14/25) were published in the last 5 years. In general, newer models were designed more robustly, used larger sample sizes, and were more mathematically rigorous. Thus, although they await further validation, there is great potential for these models to be more predictive than the Newman-Peacock model.

CONCLUSION

 Only the Newman-Peacock model is ready for regular clinical use. Many newer models are promising but require further validation.

KEY POINTS

· 25 ECV prediction models have been published; 14 were in the last 5 years.. · The Newman-Peacock model is currently the only one with sufficient validation for clinical use.. · Many newer models appear to perform better but await further validation..

A Systematic Review of Predictor Composition, Outcomes, Risk of Bias, and Validation of COVID-19 Prognostic Scores

BACKGROUND

Numerous prognostic scores have been published to support risk stratification for patients with coronavirus disease 2019 (COVID-19).

METHODS

We performed a systematic review to identify the scores for confirmed or clinically assumed COVID-19 cases. An in-depth assessment and risk of bias (ROB) analysis (Prediction model Risk Of Bias ASsessment Tool [PROBAST]) was conducted for scores fulfilling predefined criteria ([I] area under the curve [AUC)] ≥ 0.75; [II] a separate validation cohort present; [III] training data from a multicenter setting [≥2 centers]; [IV] point-scale scoring system).

RESULTS

Out of 1522 studies extracted from MEDLINE/Web of Science (20/02/2023), we identified 242 scores for COVID-19 outcome prognosis (mortality 109, severity 116, hospitalization 14, long-term sequelae 3). Most scores were developed using retrospective (75.2%) or single-center (57.1%) cohorts. Predictor analysis revealed the primary use of laboratory data and sociodemographic information in mortality and severity scores. Forty-nine scores were included in the in-depth analysis. The results indicated heterogeneous quality and predictor selection, with only five scores featuring low ROB. Among those, based on the number and heterogeneity of validation studies, only the 4C Mortality Score can be recommended for clinical application so far.

CONCLUSIONS

The application and translation of most existing COVID scores appear unreliable. Guided development and predictor selection would have improved the generalizability of the scores and may enhance pandemic preparedness in the future.

Enhanced Risk Stratification for Children and Young Adults with B-Cell Acute Lymphoblastic Leukemia: A Children's Oncology Group Report

Current strategies to treat pediatric acute lymphoblastic leukemia rely on risk stratification algorithms using categorical data. We investigated whether using continuous variables assigned different weights would improve risk stratification. We developed and validated a multivariable Cox model for relapse-free survival (RFS) using information from 21199 patients. We constructed risk groups by identifying cutoffs of the COG Prognostic Index (PICOG) that maximized discrimination of the predictive model. Patients with higher PICOG have higher predicted relapse risk. The PICOG reliably discriminates patients with low vs. high relapse risk. For those with moderate relapse risk using current COG risk classification, the PICOG identifies subgroups with varying 5-year RFS. Among current COG standard-risk average patients, PICOG identifies low and intermediate risk groups with 96% and 90% RFS, respectively. Similarly, amongst current COG high-risk patients, PICOG identifies four groups ranging from 96% to 66% RFS, providing additional discrimination for future treatment stratification. When coupled with traditional algorithms, the novel PICOG can more accurately risk stratify patients, identifying groups with better outcomes who may benefit from less intensive therapy, and those who have high relapse risk needing innovative approaches for cure.

Prospective validity of a clinical prediction rule for response to non-surgical multidisciplinary management of knee osteoarthritis in tertiary care: a multisite prospective longitudinal study

OBJECTIVES

We tested a previously developed clinical prediction tool-a nomogram consisting of four patient measures (lower patient-expected benefit, lower patient-reported knee function, greater knee varus angle and severe medial knee radiological degeneration) that were related to poor response to non-surgical management of knee osteoarthritis. This study sought to prospectively evaluate the predictive validity of this nomogram to identify patients most likely to respond poorly to non-surgical management of knee osteoarthritis.

DESIGN

Multisite prospective longitudinal study.

SETTING

Advanced practice physiotherapist-led multidisciplinary service across six tertiary hospitals.

PARTICIPANTS

Participants with knee osteoarthritis deemed appropriate for trial of non-surgical management following an initial assessment from an advanced practice physiotherapist were eligible for inclusion.

INTERVENTIONS

Baseline clinical nomogram scores were collected before a trial of individualised non-surgical management commenced.

PRIMARY OUTCOME MEASURE

Clinical outcome (Global Rating of Change) was collected 6 months following commencement of non-surgical management and dichotomised to responder (a little better to a very great deal better) or poor responder (almost the same to a very great deal worse). Clinical nomogram accuracy was evaluated from receiver operating characteristics curve analysis and area under the curve, and sensitivity/specificity and positive/negative likelihood ratios were calculated.

RESULTS

A total of 242 participants enrolled. Follow-up scores were obtained from 210 participants (87% response rate). The clinical nomogram demonstrated an area under the curve of 0.70 (p<0.001), with greatest combined sensitivity 0.65 and specificity 0.64. The positive likelihood ratio was 1.81 (95% CI 1.32 to 2.36) and negative likelihood ratio 0.55 (95% CI 0.41 to 0.75).

CONCLUSIONS

The knee osteoarthritis clinical nomogram prediction tool may have capacity to identify patients at risk of poor response to non-surgical management. Further work is required to determine the implications for service delivery, feasibility and impact of implementing the nomogram in clinical practice.

Modifiable prognostic factors of high societal costs among people on sick leave due to musculoskeletal disorders: findings from an occupational cohort study

OBJECTIVES

The objective was to identify modifiable prognostic factors of high societal costs among people on sick leave due to musculoskeletal disorders, and to identify modifiable prognostic factors of high costs related to separately healthcare utilisation and productivity loss.

DESIGN

A prospective cohort study with a 1-year follow-up.

PARTICIPANTS AND SETTING

A total of 549 participants (aged 18-67 years) on sick leave (≥ 4 weeks) due to musculoskeletal disorders in Norway were included.

OUTCOME MEASURES AND METHOD

The primary outcome was societal costs aggregated for 1 year of follow-up and dichotomised as high or low, defined by the top 25th percentile. Secondary outcomes were high costs related to separately healthcare utilisation and productivity loss aggregated for 1 year of follow-up. Healthcare utilisation was collected from public records and included primary, secondary and tertiary healthcare use. Productivity loss was collected from public records and included absenteeism, work assessment allowance and disability pension. Nine modifiable prognostic factors were selected based on previous literature. Univariable and multivariable binary logistic regression analyses were performed to identify associations (crude and adjusted for selected covariates) between each modifiable prognostic factor and having high costs.

RESULTS

Adjusted for selected covariates, six modifiable prognostic factors associated with high societal costs were identified: pain severity, disability, self-perceived health, sleep quality, return to work expectation and long-lasting disorder expectation. Depressive symptoms, work satisfaction and health literacy showed no prognostic value. More or less similar results were observed when high costs were related to separately healthcare utilisation and productivity loss.

CONCLUSION

Factors identified in this study are potential target areas for interventions which could reduce high societal costs among people on sick leave due to musculoskeletal disorders. However, future research aimed at replicating these findings is warranted.

TRIAL REGISTRATION NUMBER

NCT04196634, 12 December 2019.

Prediction of 30-day, 90-day, and 1-year mortality after colorectal cancer surgery using a data-driven approach

PURPOSE

To develop prediction models for short-term mortality risk assessment following colorectal cancer surgery.

METHODS

Data was harmonized from four Danish observational health databases into the Observational Medical Outcomes Partnership Common Data Model. With a data-driven approach using the Least Absolute Shrinkage and Selection Operator logistic regression on preoperative data, we developed 30-day, 90-day, and 1-year mortality prediction models. We assessed discriminative performance using the area under the receiver operating characteristic and precision-recall curve and calibration using calibration slope, intercept, and calibration-in-the-large. We additionally assessed model performance in subgroups of curative, palliative, elective, and emergency surgery.

RESULTS

A total of 57,521 patients were included in the study population, 51.1% male and with a median age of 72 years. The model showed good discrimination with an area under the receiver operating characteristic curve of 0.88, 0.878, and 0.861 for 30-day, 90-day, and 1-year mortality, respectively, and a calibration-in-the-large of 1.01, 0.99, and 0.99. The overall incidence of mortality were 4.48% for 30-day mortality, 6.64% for 90-day mortality, and 12.8% for 1-year mortality, respectively. Subgroup analysis showed no improvement of discrimination or calibration when separating the cohort into cohorts of elective surgery, emergency surgery, curative surgery, and palliative surgery.

CONCLUSION

We were able to train prediction models for the risk of short-term mortality on a data set of four combined national health databases with good discrimination and calibration. We found that one cohort including all operated patients resulted in better performing models than cohorts based on several subgroups.

Diagnostic performance of clinical prediction rules to detect group A beta-haemolytic streptococci in people with acute pharyngitis: a systematic review

OBJECTIVE

To assess and compare the diagnostic performance of Clinical Prediction Rules (CPRs) developed to detect group A Beta-haemolytic streptococci in people with acute pharyngitis (or sore throat).

STUDY DESIGN

A systematic review.

METHODS

We searched PubMed, Embase and Web of Science (inception-September 2022) for studies deriving and/or validating CPRs comprised of ≥2 predictors from an individual's history or physical examination. Two authors independently screened articles, extracted data and assessed risk of bias in included studies. A meta-analysis was not possible due to heterogeneity. Instead we compared the performance of CPRs when they were validated in the same study population (head-to-head comparisons). We used a modified grading of recommendations, assessment, development, and evaluations (GRADE) approach to assess certainty of the evidence.

RESULTS

We included 63 studies, all judged at high risk of bias. Of 24 derived CPRs, 7 were externally validated (in 46 external validations). Five validation studies provided data for head-to-head comparison of four pairs of CPRs. Very low certainty evidence favoured the Centor CPR over the McIsaac (2 studies) and FeverPain CPRs (1 study) and found the Centor CPR was equivalent to the Walsh CPR (1 study). The AbuReesh and Steinhoff 2005 CPRs had a similar poor discriminative ability (1 study). Within and between study comparisons suggested the performance of the Centor CPR may be better in adults (>18 years).

CONCLUSION

Very low certainty evidence suggests a better performance of the Centor CPR. When deciding about antibiotic prescribing for pharyngitis patients, involving patients in a shared decision making discussion about the likely benefits and harms, including antibiotic resistance, is recommended. Further research of higher rigour, which compares CPRs across multiple settings, is needed.

Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750)

BACKGROUND

Surgical sentinel lymph node biopsy (SLNB) is routinely used to reliably stage axillary lymph nodes in early breast cancer (BC). However, SLNB may be associated with postoperative arm morbidities. For most patients with BC undergoing SLNB, the findings are benign, and the procedure is currently questioned. A decision-support tool for the prediction of benign sentinel lymph nodes based on preoperatively available data has been developed using artificial neural network modelling.

METHODS

This was a retrospective geographical and temporal validation study of the noninvasive lymph node staging (NILS) model, based on preoperatively available data from 586 women consecutively diagnosed with primary BC at two sites. Ten preoperative clinicopathological characteristics from each patient were entered into the web-based calculator, and the probability of benign lymph nodes was predicted. The performance of the NILS model was assessed in terms of discrimination with the area under the receiver operating characteristic curve (AUC) and calibration, that is, comparison of the observed and predicted event rates of benign axillary nodal status (N0) using calibration slope and intercept. The primary endpoint was axillary nodal status (discrimination, benign [N0] vs. metastatic axillary nodal status [N+]) determined by the NILS model compared to nodal status by definitive pathology.

RESULTS

The mean age of the women in the cohort was 65 years, and most of them (93%) had luminal cancers. Approximately three-fourths of the patients had no metastases in SLNB (N0 74% and 73%, respectively). The AUC for the predicted probabilities for the whole cohort was 0.6741 (95% confidence interval: 0.6255-0.7227). More than one in four patients (n = 151, 26%) were identified as candidates for SLNB omission when applying the predefined cut-off for lymph node-negative status from the development cohort. The NILS model showed the best calibration in patients with a predicted high probability of healthy axilla.

CONCLUSION

The performance of the NILS model was satisfactory. In approximately every fourth patient, SLNB could potentially be omitted. Considering the shift from postoperatively to preoperatively available predictors in this validation study, we have demonstrated the robustness of the NILS model. The clinical usability of the web interface will be evaluated before its clinical implementation.

TRIAL REGISTRATION

Registered in the ISRCTN registry with study ID ISRCTN14341750. Date of registration 23/11/2018.

Diagnostic scoring systems for tuberculous pleural effusion in patients with lymphocyte-predominant exudative pleural profile: A development study

Background

Diagnosing tuberculous pleural effusion (TPE) in patients presenting with Lymphocyte-Predominant Exudative pleural effusion (LPE) is challenging, due to the poor clinical utility of TB culture. Adenosine deaminase (ADA) has been recommended for diagnosis, but its high cost and limited availability hinder its clinical utility. We aim to develop diagnostic prediction tools for Thai patients with LPE in scenarios where pleural fluid ADA is available but yields negative results and in situations where pleural fluid ADA is not available.

Methods

Two diagnostic prediction tools were developed using retrospective data from patients with LPE at Surin Hospital. Model 1 is for ADA-negative results, and Model 2 is for situations where pleural fluid ADA testing is unavailable. The models were derived using multivariable logistic regression and presented as two clinical scoring systems: round-up and count scoring. The score cut-point that achieves a positive predictive value (PPV) comparable to the post-test probability of a pleural fluid ADA at a cut-point of 40 U/L was used as a threshold for initiating anti-TB treatment.

Results

A total of 359 patients were eligible for analysis, with 166 diagnosed with TPE and 193 diagnosed with non-TPE. Age <40 years, fever, pleural fluid protein ≥5 g/dL, male gender, pleural fluid color, and pleural fluid ADA ≥20 U/L were identified as final predictors. Both models demonstrated excellent discriminative ability (AuROC: 0.85 to 0.89). The round-up scoring demonstrated PPV above 90% at cut-off points of 4 and 4.5, while the count scoring achieved cut-off points of 3 and 4 for Model 1 (Lex-2P2A) and Model 2 (Lex-2P-MAC), respectively.

Conclusion

These diagnostic tools offer valuable assistance in differentiating between TPE and non-TPE in LPE patients with negative pleural fluid ADA (Lex-2P2A) and in settings where pleural fluid ADA testing is not available (Lex-2P-MAC). Implementing these diagnostic scores may have the potential to improve TPE diagnosis and facilitate prompt initiation of treatment.

Intelligent digital tools for screening of brain connectivity and dementia risk estimation in people affected by mild cognitive impairment: the AI-Mind clinical study protocol

More than 10 million Europeans show signs of mild cognitive impairment (MCI), a transitional stage between normal brain aging and dementia stage memory disorder. The path MCI takes can be divergent; while some maintain stability or even revert to cognitive norms, alarmingly, up to half of the cases progress to dementia within 5 years. Current diagnostic practice lacks the necessary screening tools to identify those at risk of progression. The European patient experience often involves a long journey from the initial signs of MCI to the eventual diagnosis of dementia. The trajectory is far from ideal. Here, we introduce the AI-Mind project, a pioneering initiative with an innovative approach to early risk assessment through the implementation of advanced artificial intelligence (AI) on multimodal data. The cutting-edge AI-based tools developed in the project aim not only to accelerate the diagnostic process but also to deliver highly accurate predictions regarding an individual's risk of developing dementia when prevention and intervention may still be possible. AI-Mind is a European Research and Innovation Action (RIA H2020-SC1-BHC-06-2020, No. 964220) financed between 2021 and 2026. First, the AI-Mind Connector identifies dysfunctional brain networks based on high-density magneto- and electroencephalography (M/EEG) recordings. Second, the AI-Mind Predictor predicts dementia risk using data from the Connector, enriched with computerized cognitive tests, genetic and protein biomarkers, as well as sociodemographic and clinical variables. AI-Mind is integrated within a network of major European initiatives, including The Virtual Brain, The Virtual Epileptic Patient, and EBRAINS AISBL service for sensitive data, HealthDataCloud, where big patient data are generated for advancing digital and virtual twin technology development. AI-Mind's innovation lies not only in its early prediction of dementia risk, but it also enables a virtual laboratory scenario for hypothesis-driven personalized intervention research. This article introduces the background of the AI-Mind project and its clinical study protocol, setting the stage for future scientific contributions.

Development and validation of a prognostic nomogram for patients with laryngeal cancer with synchronous or metachronous lung cancer

BACKGROUND

The objective of this study was to examine the independent prognostic factors of laryngeal cancer with synchronous or metachronous lung cancer (LCSMLC), and to generate and verify a clinical prediction model.

METHODS

In this study, laryngeal cancer alone and LCSMLC were defined using the Surveillance, Epidemiology, and End Results (SEER) database. Risk factors of patients with LCSMLC were analyzed through univariate and multivariate logistic regression analysis. Independent prognostic factors were selected by Cox regression analyses, on the basis of which a nomogram was constructed using R code. Kaplan-Meier survival analyses were applied to test the application of a risk stratification system. Finally, we conducted a comparison of the American Joint Committee on Cancer (AJCC) staging system of laryngeal cancer with the new model of nomogram and risk stratification. For further validation of the nomogram, data from patients at two Chinese independent institutions were also analyzed.

RESULTS

According to the eligibility criteria, 32 429 patients with laryngeal cancer alone and 641 patients with LCSMLC from the SEER database (the training cohort) and additional 61 patients from two Chinese independent institutions (the external validation cohort) were included for final analyses. Compared with patients with laryngeal cancer who did not have synchronous or metachronous lung cancer, age, sex, race, primary site of laryngeal cancer, grade, and stage were risk factors for LCSMLC, while marriage, surgery, radiation therapy, and chemotherapy are not their risk factors. Age, two cancers' interval, pathological type, stage, surgery, radiation, primary lung site, and primary throat site were independent prognostic predictors of LCSMLC. The risk stratification system of high-, medium-, and low-risk groups significantly distinguished the prognosis in different patients with LCSMLC, regardless of the training cohort or the validation cohort. Compared with the 6th AJCC TNM stage of laryngeal cancer, the new model of nomogram and risk stratification showed an improved net benefit.

CONCLUSIONS

Age, sex, race, primary site of laryngeal cancer, grade, and stage were risk factors for LCSMLC. An individualized clinical prognostic predictive model by nomogram was generated and validated, which showed superior prediction ability for LCSMLC.

Development and validation of a clinical prediction rule for acute appendicitis in children in primary care

BACKGROUND

Recognising acute appendicitis in children presenting with acute abdominal pain in primary care is challenging. General practitioners (GPs) may benefit from a clinical prediction rule.

OBJECTIVES

To develop and validate a clinical prediction rule for acute appendicitis in children presenting with acute abdominal pain in primary care.

METHODS

In a historical cohort study data was retrieved from GP electronic health records included in the Integrated Primary Care Information database. We assigned children aged 4-18 years presenting with acute abdominal pain (≤ 7 days) to development (2010-2012) and validation (2013-2016) cohorts, using acute appendicitis within six weeks as the outcome. Multiple logistic regression was used to develop a prediction model based on predictors with > 50% data availability derived from existing rules for secondary care. We performed internal and external temporal validation and derived a point score to stratify risk of appendicitis into three groups, i.e. low-risk, medium-risk and high-risk.

RESULTS

The development and validation cohorts included 2,041 and 3,650 children, of whom 95 (4.6%) and 195 (5.3%) had acute appendicitis. The model included male sex, pain duration (<24, 24-48, > 48 h), nausea/vomiting, elevated temperature (≥ 37.3 °C), abnormal bowel sounds, right lower quadrant tenderness, and peritoneal irritation. Internal and temporal validation showed good discrimination (C-statistics: 0.93 and 0.90, respectively) and excellent calibration. In the three groups, the risks of acute appendicitis were 0.5%, 7.5%, and 41%.

CONCLUSION

Combined with further testing in the medium-risk group, the prediction rule could improve clinical decision making and outcomes.

External validation and updating of prognostic prediction models for nonrecovery among older adults seeking primary care for back pain

ABSTRACT

Prognostic prediction models for 3 different definitions of nonrecovery were developed in the Back Complaints in the Elders study in the Netherlands. The models' performance was good (optimism-adjusted area under receiver operating characteristics [AUC] curve ≥0.77, R2 ≥0.3). This study aimed to assess the external validity of the 3 prognostic prediction models in the Norwegian Back Complaints in the Elders study. We conducted a prospective cohort study, including 452 patients aged ≥55 years, seeking primary care for a new episode of back pain. Nonrecovery was defined for 2 outcomes, combining 6- and 12-month follow-up data: Persistent back pain (≥3/10 on numeric rating scale) and persistent disability (≥4/24 on Roland-Morris Disability Questionnaire). We could not assess the third model (self-reported nonrecovery) because of substantial missing data (>50%). The models consisted of biopsychosocial prognostic factors. First, we assessed Nagelkerke R2 , discrimination (AUC) and calibration (calibration-in-the-large [CITL], slope, and calibration plot). Step 2 was to recalibrate the models based on CITL and slope. Step 3 was to reestimate the model coefficients and assess if this improved performance. The back pain model demonstrated acceptable discrimination (AUC 0.74, 95% confidence interval: 0.69-0.79), and R2 was 0.23. The disability model demonstrated excellent discrimination (AUC 0.81, 95% confidence interval: 0.76-0.85), and R2 was 0.35. Both models had poor calibration (CITL <0, slope <1). Recalibration yielded acceptable calibration for both models, according to the calibration plots. Step 3 did not improve performance substantially. The recalibrated models may need further external validation, and the models' clinical impact should be assessed.

Prediction models in prehospital and emergency medicine research: How to derive and internally validate a clinical prediction model

Clinical prediction models are created to help clinicians with medical decision making, aid in risk stratification, and improve diagnosis and/or prognosis. With growing availability of both prehospital and in-hospital observational registries and electronic health records, there is an opportunity to develop, validate, and incorporate prediction models into clinical practice. However, many prediction models have high risk of bias due to poor methodology. Given that there are no methodological standards aimed at developing prediction models specifically in the prehospital setting, the objective of this paper is to describe the appropriate methodology for the derivation and validation of clinical prediction models in this setting. What follows can also be applied to the emergency medicine (EM) setting. There are eight steps that should be followed when developing and internally validating a prediction model: (1) problem definition, (2) coding of predictors, (3) addressing missing data, (4) ensuring adequate sample size, (5) variable selection, (6) evaluating model performance, (7) internal validation, and (8) model presentation. Subsequent steps include external validation, assessment of impact, and cost-effectiveness. By following these steps, researchers can develop a prediction model with the methodological rigor and quality required for prehospital and EM research.

Development of a Multivariable Seizure Likelihood Assessment Based on Clinical Information and Short Autonomic Activity Recordings for Children With Epilepsy

BACKGROUND

Predicting seizure likelihood for the following day would enable clinicians to extend or potentially schedule video-electroencephalography (EEG) monitoring when seizure risk is high. Combining standardized clinical data with short-term recordings of wearables to predict seizure likelihood could have high practical relevance as wearable data is easy and fast to collect. As a first step toward seizure forecasting, we classified patients based on whether they had seizures or not during the following recording.

METHODS

Pediatric patients admitted to the epilepsy monitoring unit wore a wearable that recorded the heart rate (HR), heart rate variability (HRV), electrodermal activity (EDA), and peripheral body temperature. We utilized short recordings from 9:00 to 9:15 pm and compared mean values between patients with and without impending seizures. In addition, we collected clinical data: age, sex, age at first seizure, generalized slowing, focal slowing, and spikes on EEG, magnetic resonance imaging findings, and antiseizure medication reduction. We used conventional machine learning techniques with cross-validation to classify patients with and without impending seizures.

RESULTS

We included 139 patients: 78 had no seizures and 61 had at least one seizure after 9 pm during the concurrent video-EEG and E4 recordings. HR (P < 0.01) and EDA (P < 0.01) were lower and HRV (P = 0.02) was higher for patients with than for patients without impending seizures. The average accuracy of group classification was 66%, and the mean area under the receiver operating characteristics was 0.72.

CONCLUSIONS

Short-term wearable recordings in combination with clinical data have great potential as an easy-to-use seizure likelihood assessment tool.

What is important to the GP in recognizing acute appendicitis in children: a delphi study

BACKGROUND

For diagnostic research on appendicitis in registration data, insight is needed in the way GPs generate medical records. We aimed to reach a consensus on the features that GPs consider important in the consultation and medical records when evaluating a child with suspected appendicitis.

METHODS

We performed a three-round Delphi study among Dutch GPs selected by purposive sampling. An initial feature list was created based on a literature search and features in the relevant Dutch guideline. Finally, using a vignette describing a child who needed later reassessment, we asked participants to complete an online questionnaire about which consultation features should be addressed and recorded.

RESULTS

A literature review and Dutch guideline yielded 95 consultation features. All three rounds were completed by 22 GPs, with the final consensus list containing 26 symptoms, 29 physical assessments and signs, 2 additional tests, and 8 further actions (including safety-netting, i.e., informing the patient about when to contact the GP again). Of these, participants reached consensus that 37 should be actively addressed and that 20 need to be recorded if findings are negative.

CONCLUSIONS

GPs agreed that negative findings do not need to be recorded for most features and that records should include the prognostic and safety-netting advice given. The results have implications in three main domains: for research, that negative findings are likely to be missing; for medicolegal purposes, that documentation cannot be expected to be complete; and for clinical practice, that safety-netting advice should be given and documented.

Crossroads of Antimicrobial and Diagnostic Stewardship: Assessing Risks to Develop Clinical Decision Support to Combat Multidrug-Resistant Pseudomonas

Background

Early detection of multidrug-resistant Pseudomonas aeruginosa (MDRP) remains challenging. Existing risk prediction tools are difficult to translate to bedside application. The goal of this study was to develop a simple electronic medical record (EMR)-integrated tool for prediction of MDRP infection.

Methods

This was a mixed-methods study. We conducted a split-sample cohort study of adult critical care patients with P aeruginosa infections. Two previously published tools were validated using c-statistic. A subset of variables based on strength of association and ease of EMR extraction was selected for further evaluation. A simplified tool was developed using multivariable logistic regression. Both c-statistic and theoretical trade-off of over- versus underprescribing of broad-spectrum MDRP therapy were assessed in the validation cohort. A qualitative survey of frontline clinicians assessed understanding of risks for MDRP and potential usability of an EMR-integrated tool to predict MDRP.

Results

The 2 previous risk prediction tools demonstrated similar accuracy in the derivation cohort (c-statistic of 0.76 [95% confidence interval {CI}, .69-.83] and 0.73 [95% CI, .66-.8]). A simplified tool based on 4 variables demonstrated reasonable accuracy (c-statistic of 0.71 [95% CI, .57-.85]) without significant overprescribing in the validation cohort. The risk factors were prior MDRP infection, ≥4 antibiotics prior to culture, infection >3 days after admission, and dialysis. Fourteen clinicians completed the survey. An alert providing context regarding individual patient risk factors for MDRP was preferred.

Conclusions

These results can be used to develop a local EMR-integrated tool to improve timeliness of effective therapy in those at risk of MDRP infections.

Risk-prediction models for intravenous immunoglobulin resistance in Kawasaki disease: Risk-of-Bias Assessment using PROBAST

BACKGROUND

The prediction model of intravenous immunoglobulin (IVIG) resistance in Kawasaki disease can calculate the probability of IVIG resistance and provide a basis for clinical decision-making. We aim to assess the quality of these models developed in the children with Kawasaki disease.

METHODS

Studies of prediction models for IVIG-resistant Kawasaki disease were identified through searches in the PubMed, Web of Science, and Embase databases. Two investigators independently performed literature screening, data extraction, quality evaluation, and discrepancies were settled by a statistician. The checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS) was used for data extraction, and the prediction models were evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

RESULTS

Seventeen studies meeting the selection criteria were included in the qualitative analysis. The top three predictors were neutrophil measurements (peripheral neutrophil count and neutrophil %), serum albumin level, and C-reactive protein (CRP) level. The reported area under the curve (AUC) values for the developed models ranged from 0.672 (95% confidence interval [CI]: 0.631-0.712) to 0.891 (95% CI: 0.837-0.945); The studies showed a high risk of bias (ROB) for modeling techniques, yielding a high overall ROB.

CONCLUSION

IVIG resistance models for Kawasaki disease showed high ROB. An emphasis on improving their quality can provide high-quality evidence for clinical practice.

IMPACT STATEMENT

This study systematically evaluated the risk of bias (ROB) of existing prediction models for intravenous immunoglobulin (IVIG) resistance in Kawasaki disease to provide guidance for future model development meeting clinical expectations. This is the first study to systematically evaluate the ROB of IVIG resistance in Kawasaki disease by using PROBAST. ROB may reduce model performance in different populations. Future prediction models should account for this problem, and PROBAST can help improve the methodological quality and applicability of prediction model development.

Prediction models for post-thrombectomy brain edema in patients with acute ischemic stroke: a systematic review and meta-analysis

Objective

The objective of this study is to systematically evaluate prediction models for post-thrombectomy brain edema in acute ischemic stroke (AIS) patients. This analysis aims to equip clinicians with evidence-based guidance for the selection of appropriate prediction models, thereby facilitating the early identification of patients at risk of developing brain edema post-surgery.

Methods

A comprehensive literature search was conducted across multiple databases, including PubMed, Web of Science, Embase, The Cochrane Library, CNKI, Wanfang, and Vip, aiming to identify studies on prediction models for post-thrombectomy brain edema in AIS patients up to January 2023. Reference lists of relevant articles were also inspected. Two reviewers independently screened the literature and extracted data. The Prediction Model Risk of Bias Assessment Tool (PROBAST) and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines were employed to assess study bias and literature quality, respectively. We then used random-effects bivariate meta-analysis models to summarize the studies.

Results

The review included five articles, yielding 10 models. These models exhibited a relatively high risk of bias. Random effects model demonstrated that the AUC was 0.858 (95% CI 0.817-0.899).

Conclusion

Despite the promising discriminative ability shown by studies on prediction models for post-thrombectomy brain edema in AIS patients, concerns related to a high risk of bias and limited external validation remain. Future research should prioritize the external validation and optimization of these models. There is an urgent need for large-scale, multicenter studies to develop robust, user-friendly models for real-world clinical application.

Systematic review registration

https://www.crd.york.ac.uk, unique Identifier: CRD42022382790.

Multivariable prediction models for the recovery of and claim closure related to post-collision neck pain and associated disorders

OBJECTIVE

Few clinical prediction models are available to clinicians to predict the recovery of patients with post-collision neck pain and associated disorders. We aimed to develop evidence-based clinical prediction models to predict (1) self-reported recovery and (2) insurance claim closure from neck pain and associated disorders (NAD) caused or aggravated by a traffic collision.

METHODS

The selection of potential predictors was informed by a systematic review of the literature. We used Cox regression to build models in an incident cohort of Saskatchewan adults (n = 4923). The models were internally validated using bootstrapping and replicated in participants from a randomized controlled trial conducted in Ontario (n = 340). We used C-statistics to describe predictive ability.

RESULTS

Participants from both cohorts (Saskatchewan and Ontario) were similar at baseline. Our prediction model for self-reported recovery included prior traffic-related neck injury claim, expectation of recovery, age, percentage of body in pain, disability, neck pain intensity and headache intensity (C = 0.643; 95% CI 0.634-0.653). The prediction model for claim closure included prior traffic-related neck injury claim, expectation of recovery, age, percentage of body in pain, disability, neck pain intensity, headache intensity and depressive symptoms (C = 0.637; 95% CI 0.629-0.648).

CONCLUSIONS

We developed prediction models for the recovery and claim closure of NAD caused or aggravated by a traffic collision. Future research needs to focus on improving the predictive ability of the models.

Prediction of long-term mortality following hip fracture surgery: evaluation of three risk models

INTRODUCTION

Several prognostic models have been developed for mortality in hip fracture patients, but their accuracy for long-term prediction is unclear. This study evaluates the performance of three models assessing 30-day, 1-year and 8-year mortality after hip fracture surgery: the Nottingham Hip Fracture Score (NHFS), the model developed by Holt et al. and the Hip fracture Estimator of Mortality Amsterdam (HEMA).

MATERIALS AND METHODS

Patients admitted with a fractured hip between January 2012 and June 2013 were included in this retrospective cohort study. Relevant variables used by the three models were collected, as were mortality data. Predictive performance was assessed in terms of discrimination with the area under the receiver operating characteristic curve and calibration with the Hosmer-Lemeshow goodness-of-fit test. Clinical usefulness was evaluated by determining risk groups for each model, comparing differences in mortality using Kaplan-Meier curves, and by assessing positive and negative predictive values.

RESULTS

A total of 344 patients were included for analysis. Observed mortality rates were 6.1% after 30 days, 19.1% after 1 year and 68.6% after 8 years. The NHFS and the model by Holt et al. demonstrated good to excellent discrimination and adequate calibration for both short- and long-term mortality prediction, with similar clinical usefulness measures. The HEMA demonstrated inferior prediction of 30-day and 8-year mortality, with worse discriminative abilities and a significant lack of fit.

CONCLUSIONS

The NHFS and the model by Holt et al. allowed for accurate identification of low- and high-risk patients for both short- and long-term mortality after a fracture of the hip. The HEMA performed poorly. When considering predictive performance and ease of use, the NHFS seems most suitable for implementation in daily clinical practice.

Epigenetic biomarkers for smoking cessation

Cigarette smoking has been associated with epigenetic alterations that may be reversible upon cessation. As the most-studied epigenetic modification, DNA methylation is strongly associated with smoking exposure, providing a potential mechanism that links smoking to adverse health outcomes. Here, we reviewed the reversibility of DNA methylation in accessible peripheral tissues, mainly blood, in relation to cigarette smoking cessation and the utility of DNA methylation as a biomarker signature to differentiate current, former, and never smokers and to quantify time since cessation. We summarized thousands of differentially methylated Cytosine-Guanine (CpG) dinucleotides and regions associated with smoking cessation from candidate gene and epigenome-wide association studies, as well as the prediction accuracy of the multi-CpG predictors for smoking status. Overall, there is robust evidence for DNA methylation signature of cigarette smoking cessation. However, there are still gaps to fill, including (1) cell-type heterogeneity in measuring blood DNA methylation; (2) underrepresentation of non-European ancestry populations; (3) limited longitudinal data to quantitatively measure DNA methylation after smoking cessation over time; and (4) limited data to study the impact of smoking cessation on other epigenetic features, noncoding RNAs, and histone modifications. Epigenetic machinery provides promising biomarkers that can improve success in smoking cessation in the clinical setting. To achieve this goal, larger and more-diverse samples with longitudinal measures of a broader spectrum of epigenetic marks will be essential to developing a robust DNA methylation biomarker assay, followed by meeting validation requirements for the assay before being implemented as a clinically useful tool.

Machine learning applied in maternal and fetal health: a narrative review focused on pregnancy diseases and complications

Introduction

Machine learning (ML) corresponds to a wide variety of methods that use mathematics, statistics and computational science to learn from multiple variables simultaneously. By means of pattern recognition, ML methods are able to find hidden correlations and accomplish accurate predictions regarding different conditions. ML has been successfully used to solve varied problems in different areas of science, such as psychology, economics, biology and chemistry. Therefore, we wondered how far it has penetrated into the field of obstetrics and gynecology.

Aim

To describe the state of art regarding the use of ML in the context of pregnancy diseases and complications.

Methodology

Publications were searched in PubMed, Web of Science and Google Scholar. Seven subjects of interest were considered: gestational diabetes mellitus, preeclampsia, perinatal death, spontaneous abortion, preterm birth, cesarean section, and fetal malformations.

Current state

ML has been widely applied in all the included subjects. Its uses are varied, the most common being the prediction of perinatal disorders. Other ML applications include (but are not restricted to) biomarker discovery, risk estimation, correlation assessment, pharmacological treatment prediction, drug screening, data acquisition and data extraction. Most of the reviewed articles were published in the last five years. The most employed ML methods in the field are non-linear. Except for logistic regression, linear methods are rarely used.

Future challenges

To improve data recording, storage and update in medical and research settings from different realities. To develop more accurate and understandable ML models using data from cutting-edge instruments. To carry out validation and impact analysis studies of currently existing high-accuracy ML models.

Conclusion

The use of ML in pregnancy diseases and complications is quite recent, and has increased over the last few years. The applications are varied and point not only to the diagnosis, but also to the management, treatment, and pathophysiological understanding of perinatal alterations. Facing the challenges that come with working with different types of data, the handling of increasingly large amounts of information, the development of emerging technologies, and the need of translational studies, it is expected that the use of ML continue growing in the field of obstetrics and gynecology.

What proportion of clinical prediction models make it to clinical practice? Protocol for a two-track follow-up study of prediction model development publications

INTRODUCTION

It is known that only a limited proportion of developed clinical prediction models (CPMs) are implemented and/or used in clinical practice. This may result in a large amount of research waste, even when considering that some CPMs may demonstrate poor performance. Cross-sectional estimates of the numbers of CPMs that have been developed, validated, evaluated for impact or utilized in practice, have been made in specific medical fields, but studies across multiple fields and studies following up the fate of CPMs are lacking.

METHODS AND ANALYSIS

We have conducted a systematic search for prediction model studies published between January 1995 and December 2020 using the Pubmed and Embase databases, applying a validated search strategy. Taking random samples for every calendar year, abstracts and articles were screened until a target of 100 CPM development studies were identified. Next, we will perform a forward citation search of the resulting CPM development article cohort to identify articles on external validation, impact assessment or implementation of those CPMs. We will also invite the authors of the development studies to complete an online survey to track implementation and clinical utilization of the CPMs.We will conduct a descriptive synthesis of the included studies, using data from the forward citation search and online survey to quantify the proportion of developed models that are validated, assessed for their impact, implemented and/or used in patient care. We will conduct time-to-event analysis using Kaplan-Meier plots.

ETHICS AND DISSEMINATION

No patient data are involved in the research. Most information will be extracted from published articles. We request written informed consent from the survey respondents. Results will be disseminated through publication in a peer-reviewed journal and presented at international conferences. OSF REGISTRATION: (https://osf.io/nj8s9).

Stretching and muscle-performance exercises for chronic nonspecific neck pain: who may benefit most?

BACKGROUND

Although exercise is the mainstay of treatment for neck pain (NP), uncertainty remains over optimal decision-making concerning who may benefit most from such, particularly in the long term.

OBJECTIVE

To identify the subgroup of patients with nonspecific NP most likely to benefit from stretching and muscle-performance exercises.

METHODS

This was a secondary analysis of treatment outcomes of 70 patients (10 of whom dropped out) with a primary complaint of nonspecific NP in one treatment arm of a prospective, randomized, controlled trial. All patients performed the exercises, twice weekly for 6 weeks, and a home exercise program. Blinded outcome measurements were collected at baseline, after the 6-week program, and at a 6-month follow-up. Patients rated their perceived recovery on a 15-point global rating of change scale; a rating of "quite a bit better" (+5) or higher was defined as a successful outcome. Clinical predictor variables were developed via logistic regression analysis to classify patients with NP that may benefit from exercise-based treatment.

RESULTS

NP duration since onset≤6 months, no cervicogenic headache, and shoulder protraction were independent predictor variables. The pretest probability of success was 47% after the 6-week intervention and 40% at the 6-month follow-up. The corresponding posttest probabilities of success for participants with all three variables were 86% and 71%, respectively; such participants were likely to recover.

CONCLUSION

The clinical predictor variables developed in this study may identify patients with nonspecific NP likely to benefit most from stretching and muscle-performance exercises in the short and long terms.

Performance of a prediabetes risk prediction model: A systematic review

Backgrounds

The prediabetes population is large and easily overlooked because of the lack of obvious symptoms, which can progress to diabetes. Early screening and targeted interventions can substantially reduce the rate of conversion of prediabetes to diabetes. Therefore, this study systematically reviewed prediabetes risk prediction models, performed a summary and quality evaluation, and aimed to recommend the optimal model.

Methods

We systematically searched five databases (Cochrane, PubMed, Embase, Web Of Science, and CNKI) for published literature related to prediabetes risk prediction models and excluded preprints, duplicate publications, reviews, editorials, and other studies, with a search time frame of March 01, 2023. Data were categorized and summarized using a standardized data extraction form that extracted data including author; publication date; study design; country; demographic characteristics; assessment tool name; sample size; study type; and model-related indicators. The PROBAST tool was used to assess the risk of bias profile of included studies.

Findings

14 studies with a total of 15 models were eventually included in the systematic review. We found that the most common predictors of models were age, family history of diabetes, gender, history of hypertension, and BMI. Most of the studies (83.3%) had a high risk of bias, mainly related to under-reporting of outcome information and poor methodological design during the development and validation of models. Due to the low quality of included studies, the evidence for predictive validity of the available models is unclear.

Interpretation

We should pay attention to the early screening of prediabetes patients and give timely pharmacological and lifestyle interventions. The predictive performance of the existing model is not satisfactory, and the model building process can be standardized and external validation can be added to improve the accuracy of the model in the future.

Predicting mortality in patients diagnosed with advanced dementia presenting at an acute care hospital: the PROgnostic Model for Advanced DEmentia (PRO-MADE)

BACKGROUND

Challenges in prognosticating patients diagnosed with advanced dementia (AD) hinders timely referrals to palliative care. We aim to develop and validate a prognostic model to predict one-year all-cause mortality (ACM) in patients with AD presenting at an acute care hospital.

METHODS

This retrospective cohort study utilised administrative and clinical data from Tan Tock Seng Hospital (TTSH). Patients admitted to TTSH between 1st July 2016 and 31st October 2017 and identified to have AD were included. The primary outcome was ACM within one-year of AD diagnosis. Multivariable logistic regression was used. The PROgnostic Model for Advanced Dementia (PRO-MADE) was internally validated using a bootstrap resampling of 1000 replications and externally validated on a more recent cohort of AD patients. The model was evaluated for overall predictive accuracy (Nagelkerke's R² and Brier score), discriminative [area-under-the-curve (AUC)], and calibration [calibration slope and calibration-in-the-large (CITL)] properties.

RESULTS

A total of 1,077 patients with a mean age of 85 (SD: 7.7) years old were included, and 318 (29.5%) patients died within one-year of AD diagnosis. Predictors of one-year ACM were age > 85 years (OR:1.87; 95%CI:1.36 to 2.56), male gender (OR:1.62; 95%CI:1.18 to 2.22), presence of pneumonia (OR:1.75; 95%CI:1.25 to 2.45), pressure ulcers (OR:2.60; 95%CI:1.57 to 4.31), dysphagia (OR:1.53; 95%CI:1.11 to 2.11), Charlson Comorbidity Index ≥ 8 (OR:1.39; 95%CI:1.01 to 1.90), functional dependency in ≥ 4 activities of daily living (OR: 1.82; 95%CI:1.32 to 2.53), abnormal urea (OR:2.16; 95%CI:1.58 to 2.95) and abnormal albumin (OR:3.68; 95%CI:2.07 to 6.54) values. Internal validation results for optimism-adjusted Nagelkerke's R², Brier score, AUC, calibration slope and CITL were 0.25 (95%CI:0.25 to 0.26), 0.17 (95%CI:0.17 to 0.17), 0.76 (95%CI:0.76 to 0.76), 0.95 (95% CI:0.95 to 0.96) and 0 (95%CI:-0.0001 to 0.001) respectively. When externally validated, the model demonstrated an AUC of 0.70 (95%CI:0.69 to 0.71), calibration slope of 0.64 (95%CI:0.63 to 0.66) and CITL of -0.27 (95%CI:-0.28 to -0.26).

CONCLUSION

The PRO-MADE attained good discrimination and calibration properties. Used synergistically with a clinician's judgement, this model can identify AD patients who are at high-risk of one-year ACM to facilitate timely referrals to palliative care.

A deep neural network framework to derive interpretable decision rules for accurate traumatic brain injury identification of infants

OBJECTIVE

We aimed to develop a robust framework to model the complex association between clinical features and traumatic brain injury (TBI) risk in children under age two, and identify significant features to derive clinical decision rules for triage decisions.

METHODS

In this retrospective study, four frequently used machine learning models, i.e., support vector machine (SVM), random forest (RF), deep neural network (DNN), and XGBoost (XGB), were compared to identify significant clinical features from 24 input features associated with the TBI risk in children under age two under the permutation feature importance test (PermFIT) framework by using the publicly available data set from the Pediatric Emergency Care Applied Research Network (PECARN) study. The prediction accuracy was determined by comparing the predicted TBI status with the computed tomography (CT) scan results since CT scan is the gold standard for diagnosing TBI.

RESULTS

At a significance level of [Formula: see text], DNN, RF, XGB, and SVM identified 9, 1, 2,  and 4 significant features, respectively. In a comparison of accuracy (Accuracy), the area under the curve (AUC), and the precision-recall area under the curve (PR-AUC), the permutation feature importance test for DNN model was the most powerful framework for identifying significant features and outperformed other methods, i.e., RF, XGB, and SVM, with Accuracy, AUC, and PR-AUC as 0.915, 0.794, and 0.974, respectively.

CONCLUSION

These results indicate that the PermFIT-DNN framework robustly identifies significant clinical features associated with TBI status and improves prediction performance. The findings could be used to inform the development of clinical decision tools designed to inform triage decisions.

Confronting ethical and social issues related to the genetics of musicality

New interdisciplinary research into genetic influences on musicality raises a number of ethical and social issues for future avenues of research and public engagement. The historical intersection of music cognition and eugenics heightens the need to vigilantly weigh the potential risks and benefits of these studies and the use of their outcomes. Here, we bring together diverse disciplinary expertise (complex trait genetics, music cognition, musicology, bioethics, developmental psychology, and neuroscience) to interpret and guide the ethical use of findings from recent and future studies. We discuss a framework for incorporating principles of ethically and socially responsible conduct of musicality genetics research into each stage of the research lifecycle: study design, study implementation, potential applications, and communication.