Calibration: the Achilles heel of predictive analytics

Precision medicine for traumatic cervical spinal cord injuries: accessible and interpretable machine learning models to predict individualized in-hospital outcomes

BACKGROUND CONTEXT

A traumatic spinal cord injury (SCI) can cause temporary or permanent motor and sensory impairment, leading to serious short and long-term consequences that can result in significant morbidity and mortality. The cervical spine is the most commonly affected area, accounting for about 60% of all traumatic SCI cases.

PURPOSE

This study aims to employ machine learning (ML) algorithms to predict various outcomes, such as in-hospital mortality, nonhome discharges, extended length of stay (LOS), extended length of intensive care unit stay (ICU-LOS), and major complications in patients diagnosed with cervical SCI (cSCI).

STUDY DESIGN

Our study was a retrospective machine learning classification study aiming to predict the outcomes of interest, which were binary categorical variables, in patients diagnosed with cSCI.

PATIENT SAMPLE

The data for this study were obtained from the American College of Surgeons (ACS) Trauma Quality Program (TQP) database, which was queried to identify patients who suffered from cSCI between 2019 and 2021.

OUTCOME MEASURES

The outcomes of interest of our study were in-hospital mortality, nonhome discharges, prolonged LOS, prolonged ICU-LOS, and major complications. The study evaluated the models' performance using both graphical and numerical methods. The receiver operating characteristic (ROC) and precision-recall curves (PRC) were used to assess model performance graphically. Numerical evaluation metrics included AUROC, balanced accuracy, weighted area under PRC (AUPRC), weighted precision, and weighted recall.

METHODS

The study employed data from the American College of Surgeons (ACS) Trauma Quality Program (TQP) database to identify patients with cSCI. Four ML algorithms, namely XGBoost, LightGBM, CatBoost, and Random Forest, were utilized to develop predictive models. The most effective models were then incorporated into a publicly available web application designed to forecast the outcomes of interest.

RESULTS

There were 71,661 patients included in the analysis for the outcome mortality, 67,331 for the outcome nonhome discharges, 76,782 for the outcome prolonged LOS, 26,615 for the outcome prolonged ICU-LOS, and 72,132 for the outcome major complications. The algorithms exhibited an AUROC value range of 0.78 to 0.839 for in-hospital mortality, 0.806 to 0.815 for nonhome discharges, 0.679 to 0.742 for prolonged LOS, 0.666 to 0.682 for prolonged ICU-LOS, and 0.637 to 0.704 for major complications. An open access web application was developed as part of the study, which can generate predictions for individual patients based on their characteristics.

CONCLUSIONS

Our study suggests that ML models can be valuable in assessing risk for patients with cervical cSCI and may have considerable potential for predicting outcomes during hospitalization. ML models demonstrated good predictive ability for in-hospital mortality and nonhome discharges, fair predictive ability for prolonged LOS, but poor predictive ability for prolonged ICU-LOS and major complications. Along with these promising results, the development of a user-friendly web application that facilitates the integration of these models into clinical practice is a significant contribution of this study. The product of this study may have significant implications in clinical settings to personalize care, anticipate outcomes, facilitate shared decision making and informed consent processes for cSCI patients.

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

Validity of the models predicting 10-year risk of cardiovascular diseases in Asia: A systematic review and prediction model meta-analysis

We aimed to review the validity of existing prediction models for cardiovascular diseases (CVDs) in Asia. In this systematic review and meta-analysis, we included studies that validated prediction models for CVD risk in the general population in Asia. Various databases, including PubMed, Web of Science conference proceedings citation index, Scopus, Global Index Medicus of the World Health Organization (WHO), and Open Access Thesis and Dissertations (OATD), were searched up to November 2022. Additional studies were identified through reference lists and related reviews. The risk of bias was assessed using the PROBAST prediction model risk of bias assessment tool. Meta-analyses were performed using the random effects model, focusing on the C-statistic as a discrimination index and the observed-to-expected ratio (OE) as a calibration index. Out of 1315 initial records, 16 studies were included, with 21 external validations of six models in Asia. The validated models consisted of Framingham models, pooled cohort equations (PCEs), SCORE, Globorisk, and WHO models, combined with the results of the first four models. The pooled C-statistic for men ranged from 0.72 (95% CI 0.70 to 0.75; PCEs) to 0.76 (95% CI 0.74 to 0.78; Framingham general CVD). In women, it varied from 0.74 (95% CI 0.22 to 0.97; SCORE) to 0.79 (95% CI 0.74 to 0.83; Framingham general CVD). The pooled OE ratio for men ranged from 0.21 (95% CI 0.018 to 2.49; Framingham CHD) to 1.11 (95%CI 0.65 to 1.89; PCEs). In women, it varied from 0.28 (95%CI 0.33 to 2.33; Framingham CHD) to 1.81 (95% CI 0.90 to 3.64; PCEs). The Framingham, PCEs, and SCORE models exhibited acceptable discrimination but poor calibration in predicting the 10-year risk of CVDs in Asia. Recalibration and updates are necessary before implementing these models in the region.

Predicting Beta-Lactam Target Non-Attainment in ICU Patients at Treatment Initiation: Development and External Validation of Three Novel (Machine Learning) Models

In the intensive care unit (ICU), infection-related mortality is high. Although adequate antibiotic treatment is essential in infections, beta-lactam target non-attainment occurs in up to 45% of ICU patients, which is associated with a lower likelihood of clinical success. To optimize antibiotic treatment, we aimed to develop beta-lactam target non-attainment prediction models in ICU patients. Patients from two multicenter studies were included, with intravenous intermittent beta-lactam antibiotics administered and blood samples drawn within 12-36 h after antibiotic initiation. Beta-lactam target non-attainment models were developed and validated using random forest (RF), logistic regression (LR), and naïve Bayes (NB) models from 376 patients. External validation was performed on 150 ICU patients. We assessed performance by measuring discrimination, calibration, and net benefit at the default threshold probability of 0.20. Age, sex, serum creatinine, and type of beta-lactam antibiotic were found to be predictive of beta-lactam target non-attainment. In the external validation, the RF, LR, and NB models confirmed good discrimination with an area under the curve of 0.79 [95% CI 0.72-0.86], 0.80 [95% CI 0.73-0.87], and 0.75 [95% CI 0.67-0.82], respectively, and net benefit in the RF and LR models. We developed prediction models for beta-lactam target non-attainment within 12-36 h after antibiotic initiation in ICU patients. These online-accessible models use readily available patient variables and help optimize antibiotic treatment. The RF and LR models showed the best performance among the three models tested.

Natural language processing system for rapid detection and intervention of mental health crisis chat messages

Patients experiencing mental health crises often seek help through messaging-based platforms, but may face long wait times due to limited message triage capacity. Here we build and deploy a machine-learning-enabled system to improve response times to crisis messages in a large, national telehealth provider network. We train a two-stage natural language processing (NLP) system with key word filtering followed by logistic regression on 721 electronic medical record chat messages, of which 32% are potential crises (suicidal/homicidal ideation, domestic violence, or non-suicidal self-injury). Model performance is evaluated on a retrospective test set (4/1/21-4/1/22, N = 481) and a prospective test set (10/1/22-10/31/22, N = 102,471). In the retrospective test set, the model has an AUC of 0.82 (95% CI: 0.78-0.86), sensitivity of 0.99 (95% CI: 0.96-1.00), and PPV of 0.35 (95% CI: 0.309-0.4). In the prospective test set, the model has an AUC of 0.98 (95% CI: 0.966-0.984), sensitivity of 0.98 (95% CI: 0.96-0.99), and PPV of 0.66 (95% CI: 0.626-0.692). The daily median time from message receipt to crisis specialist triage ranges from 8 to 13 min, compared to 9 h before the deployment of the system. We demonstrate that a NLP-based machine learning model can reliably identify potential crisis chat messages in a telehealth setting. Our system integrates into existing clinical workflows, suggesting that with appropriate training, humans can successfully leverage ML systems to facilitate triage of crisis messages.

Assessment of the ABC2-SPH risk score to predict invasive mechanical ventilation in COVID-19 patients and comparison to other scores

Background

Predicting the need for invasive mechanical ventilation (IMV) is important for the allocation of human and technological resources, improvement of surveillance, and use of effective therapeutic measures. This study aimed (i) to assess whether the ABC2-SPH score is able to predict the receipt of IMV in COVID-19 patients; (ii) to compare its performance with other existing scores; (iii) to perform score recalibration, and to assess whether recalibration improved prediction.

Methods

Retrospective observational cohort, which included adult laboratory-confirmed COVID-19 patients admitted in 32 hospitals, from 14 Brazilian cities. This study was conducted in two stages: (i) for the assessment of the ABC2-SPH score and comparison with other available scores, patients hospitalized from July 31, 2020, to March 31, 2022, were included; (ii) for ABC2-SPH score recalibration and also comparison with other existing scores, patients admitted from January 1, 2021, to March 31, 2022, were enrolled. For both steps, the area under the receiving operator characteristic score (AUROC) was calculated for all scores, while a calibration plot was assessed only for the ABC2-SPH score. Comparisons between ABC2-SPH and the other scores followed the Delong Test recommendations. Logistic recalibration methods were used to improve results and adapt to the studied sample.

Results

Overall, 9,350 patients were included in the study, the median age was 58.5 (IQR 47.0-69.0) years old, and 45.4% were women. Of those, 33.5% were admitted to the ICU, 25.2% received IMV, and 17.8% died. The ABC2-SPH score showed a significantly greater discriminatory capacity, than the CURB-65, STSS, and SUM scores, with potentialized results when we consider only patients younger than 80 years old (AUROC 0.714 [95% CI 0.698-0.731]). Thus, after the ABC2-SPH score recalibration, we observed improvements in calibration (slope = 1.135, intercept = 0.242) and overall performance (Brier score = 0.127).

Conclusion

The ABC2-SPHr risk score demonstrated a good performance to predict the need for mechanical ventilation in COVID-19 hospitalized patients under 80 years of age.

Developing and externally validating a machine learning risk prediction model for 30-day mortality after stroke using national stroke registers in the UK and Sweden

OBJECTIVES

We aimed to develop and externally validate a generalisable risk prediction model for 30-day stroke mortality suitable for supporting quality improvement analytics in stroke care using large nationwide stroke registers in the UK and Sweden.

DESIGN

Registry-based cohort study.

SETTING

Stroke registries including the Sentinel Stroke National Audit Programme (SSNAP) in England, Wales and Northern Ireland (2013-2019) and the national Swedish stroke register (Riksstroke 2015-2020).

PARTICIPANTS AND METHODS

Data from SSNAP were used for developing and temporally validating the model, and data from Riksstroke were used for external validation. Models were developed with the variables available in both registries using logistic regression (LR), LR with elastic net and interaction terms and eXtreme Gradient Boosting (XGBoost). Performances were evaluated with discrimination, calibration and decision curves.

OUTCOME MEASURES

The primary outcome was all-cause 30-day in-hospital mortality after stroke.

RESULTS

In total, 488 497 patients who had a stroke with 12.4% 30-day in-hospital mortality were used for developing and temporally validating the model in the UK. A total of 128 360 patients who had a stroke with 10.8% 30-day in-hospital mortality and 13.1% all mortality were used for external validation in Sweden. In the SSNAP temporal validation set, the final XGBoost model achieved the highest area under the receiver operating characteristic curve (AUC) (0.852 (95% CI 0.848 to 0.855)) and was well calibrated. The performances on the external validation in Riksstroke were as good and achieved AUC at 0.861 (95% CI 0.858 to 0.865) for in-hospital mortality. For Riksstroke, the models slightly overestimated the risk for in-hospital mortality, while they were better calibrated at the risk for all mortality.

CONCLUSION

The risk prediction model was accurate and externally validated using high quality registry data. This is potentially suitable to be deployed as part of quality improvement analytics in stroke care to enable the fair comparison of stroke mortality outcomes across hospitals and health systems across countries.

External validation of a cardiovascular risk model for Omani patients with type 2 diabetes mellitus: a retrospective cohort study

OBJECTIVES

To externally validate a recently developed cardiovascular disease (CVD) risk model for Omanis with type 2 diabetes mellitus (T2DM).

DESIGN

Retrospective cohort study.

SETTING

Nine primary care centres in Muscat Governorate, Oman.

PARTICIPANTS

A total of 809 male and female adult Omani patients with T2DM free of CVD at baseline were selected using a systematic random sampling strategy.

OUTCOME MEASURES

Data regarding CVD risk factors and outcomes were collected from the patients' electronic medical records between 29 August 2020 and 2 May 2021. The ability of the model to discriminate CVD risk was assessed by calculating the area under the curve (AUC) of the receiver-operating characteristic curve. Calibration of the model was evaluated using a Hosmer-Lemeshow χ2 test and the Brier score.

RESULTS

The incidence of CVD events over the 5-year follow-up period was 4.6%, with myocardial infarction being most frequent (48.6%), followed by peripheral arterial disease (27%) and non-fatal stroke (21.6%). A cut-off risk value of 11.8% demonstrated good sensitivity (67.6%) and specificity (66.5%). The area under the curve (AUC) was 0.7 (95% CI 0.60 to 0.78) and the Brier score was 0.01. However, the overall mean predicted risk was greater than the overall observed risk (11.8% vs 4.6%) and the calibration graph showed a relatively significant difference between predicted and observed risk levels in different subgroups.

CONCLUSIONS

Although the model slightly overestimated the CVD risk, it demonstrated good discrimination. Recalibration of the model is required, after which it has the potential to be applied to patients presenting to diabetic care centres elsewhere in Oman.

Severe aortic stenosis detection by deep learning applied to echocardiography

BACKGROUND AND AIMS

Early diagnosis of aortic stenosis (AS) is critical to prevent morbidity and mortality but requires skilled examination with Doppler imaging. This study reports the development and validation of a novel deep learning model that relies on two-dimensional (2D) parasternal long axis videos from transthoracic echocardiography without Doppler imaging to identify severe AS, suitable for point-of-care ultrasonography.

METHODS AND RESULTS

In a training set of 5257 studies (17 570 videos) from 2016 to 2020 [Yale-New Haven Hospital (YNHH), Connecticut], an ensemble of three-dimensional convolutional neural networks was developed to detect severe AS, leveraging self-supervised contrastive pretraining for label-efficient model development. This deep learning model was validated in a temporally distinct set of 2040 consecutive studies from 2021 from YNHH as well as two geographically distinct cohorts of 4226 and 3072 studies, from California and other hospitals in New England, respectively. The deep learning model achieved an area under the receiver operating characteristic curve (AUROC) of 0.978 (95% CI: 0.966, 0.988) for detecting severe AS in the temporally distinct test set, maintaining its diagnostic performance in geographically distinct cohorts [0.952 AUROC (95% CI: 0.941, 0.963) in California and 0.942 AUROC (95% CI: 0.909, 0.966) in New England]. The model was interpretable with saliency maps identifying the aortic valve, mitral annulus, and left atrium as the predictive regions. Among non-severe AS cases, predicted probabilities were associated with worse quantitative metrics of AS suggesting an association with various stages of AS severity.

CONCLUSION

This study developed and externally validated an automated approach for severe AS detection using single-view 2D echocardiography, with potential utility for point-of-care screening.

Maternal near-miss prediction model development among pregnant women in Bahir Dar City administration, northwest Ethiopia: a study protocol

INTRODUCTION

Maternal near-miss is a condition when a woman nearly died but survived from complications that happened during pregnancy, childbirth or within 42 days after delivery. Maternal near-miss is more prevalent among women in developing nations. Previous studies have identified the impact of different predictor variables on maternal near-miss but shared prognostic predictors are not adequately explored in Ethiopia. It is therefore necessary to build a clinical prediction model for maternal near-misses in Ethiopia. Hence, the aim of this study is to develop and validate a prognostic prediction model, and generate a risk score for maternal near-miss among pregnant women in Bahir Dar City Administration.

METHODS AND ANALYSIS

A prospective follow-up study design will be employed among 2110 selected pregnant women in the Bahir Dar City administration from 1 May 2023 to 1 April 2024. At the initial antenatal visit, pregnant women will be systematically selected. Then, they will be followed until 42 days following birth. Data will be collected using structured questionnaires and data extraction sheet. The model will be created using Cox proportional hazard regression analysis. The performance of the model will be assessed based on its capacity for discrimination using c-index and calibration using calibration plot, intercept and slope. The model's internal validity will be evaluated through the bootstrapping method. Ultimately, the model will be illustrated through a nomogram and decision tree, which will be made available to prospective users.

ETHICS AND DISSEMINATION

Ethical approval has been obtained from the Institutional Review Board of the College of Medicine and Health Sciences, Bahir Dar University (protocol number 704/2023). Findings will be published in peer-reviewed journals and local and international seminars, conferences, symposiums and workshops. Manuscripts will be prepared and published in scientifically reputable journals. In addition, policy briefs will be prepared.

External Validation of Models for Predicting Disability in Community-Dwelling Older People in the Netherlands: A Comparative Study

Background

Advanced statistical modeling techniques may help predict health outcomes. However, it is not the case that these modeling techniques always outperform traditional techniques such as regression techniques. In this study, external validation was carried out for five modeling strategies for the prediction of the disability of community-dwelling older people in the Netherlands.

Methods

We analyzed data from five studies consisting of community-dwelling older people in the Netherlands. For the prediction of the total disability score as measured with the Groningen Activity Restriction Scale (GARS), we used fourteen predictors as measured with the Tilburg Frailty Indicator (TFI). Both the TFI and the GARS are self-report questionnaires. For the modeling, five statistical modeling techniques were evaluated: general linear model (GLM), support vector machine (SVM), neural net (NN), recursive partitioning (RP), and random forest (RF). Each model was developed on one of the five data sets and then applied to each of the four remaining data sets. We assessed the performance of the models with calibration characteristics, the correlation coefficient, and the root of the mean squared error.

Results

The models GLM, SVM, RP, and RF showed satisfactory performance characteristics when validated on the validation data sets. All models showed poor performance characteristics for the deviating data set both for development and validation due to the deviating baseline characteristics compared to those of the other data sets.

Conclusion

The performance of four models (GLM, SVM, RP, RF) on the development data sets was satisfactory. This was also the case for the validation data sets, except when these models were developed on the deviating data set. The NN models showed a much worse performance on the validation data sets than on the development data sets.

Prediction models in prehospital and emergency medicine research: How to derive and internally validate a clinical prediction model

Clinical prediction models are created to help clinicians with medical decision making, aid in risk stratification, and improve diagnosis and/or prognosis. With growing availability of both prehospital and in-hospital observational registries and electronic health records, there is an opportunity to develop, validate, and incorporate prediction models into clinical practice. However, many prediction models have high risk of bias due to poor methodology. Given that there are no methodological standards aimed at developing prediction models specifically in the prehospital setting, the objective of this paper is to describe the appropriate methodology for the derivation and validation of clinical prediction models in this setting. What follows can also be applied to the emergency medicine (EM) setting. There are eight steps that should be followed when developing and internally validating a prediction model: (1) problem definition, (2) coding of predictors, (3) addressing missing data, (4) ensuring adequate sample size, (5) variable selection, (6) evaluating model performance, (7) internal validation, and (8) model presentation. Subsequent steps include external validation, assessment of impact, and cost-effectiveness. By following these steps, researchers can develop a prediction model with the methodological rigor and quality required for prehospital and EM research.

Clinical Characteristics and Outcomes of Drug-Induced Acute Kidney Injury Cases

Introduction

Drug-induced acute kidney injury (DI-AKI) is a frequent adverse event. The identification of DI-AKI is challenged by competing etiologies, clinical heterogeneity among patients, and a lack of accurate diagnostic tools. Our research aims to describe the clinical characteristics and predictive variables of DI-AKI.

Methods

We analyzed data from the Drug-Induced Renal Injury Consortium (DIRECT) study (NCT02159209), an international, multicenter, observational cohort study of enriched clinically adjudicated DI-AKI cases. Cases met the primary inclusion criteria if the patient was exposed to at least 1 nephrotoxic drug for a minimum of 24 hours prior to AKI onset. Cases were clinically adjudicated, and inter-rater reliability (IRR) was measured using Krippendorff's alpha. Variables associated with DI-AKI were identified using L1 regularized multivariable logistic regression. Model performance was assessed using the area under the receiver operating characteristic curve (ROC AUC).

Results

A total of 314 AKI cases met the eligibility criteria for this analysis, and 271 (86%) cases were adjudicated as DI-AKI. The majority of the AKI cases were recruited from the United States (68%). The most frequent causal nephrotoxic drugs were vancomycin (48.7%), nonsteroidal antiinflammatory drugs (18.2%), and piperacillin/tazobactam (17.8%). The IRR for DI-AKI adjudication was 0.309. The multivariable model identified age, vascular capacity, hyperglycemia, infections, pyuria, serum creatinine (SCr) trends, and contrast media as significant predictors of DI-AKI with good performance (ROC AUC 0.86).

Conclusion

The identification of DI-AKI is challenging even with comprehensive adjudication by experienced nephrologists. Our analysis identified key clinical characteristics and outcomes of DI-AKI compared to other AKI etiologies.

Development of personalized machine learning-based prediction models for short-term postoperative outcomes in patients undergoing cervical laminoplasty

PURPOSE

By predicting short-term postoperative outcomes before surgery, patients undergoing cervical laminoplasty (CLP) surgery could benefit from more accurate patient care strategies that could reduce the likelihood of adverse outcomes. With this study, we developed a series of machine learning (ML) models for predicting short-term postoperative outcomes and integrated them into an open-source online application.

METHODS

National surgical quality improvement program database was utilized to identify individuals who have undergone CLP surgery. The investigated outcomes were prolonged length of stay (LOS), non-home discharges, 30-day readmissions, unplanned reoperations, and major complications. ML models were developed and implemented on a website to predict these three outcomes.

RESULTS

A total of 1740 patients that underwent CLP were included in the analysis. Performance evaluation indicated that the top-performing models for each outcome were the models built with TabPFN and LightGBM algorithms. The TabPFN models yielded AUROCs of 0.830, 0.847, and 0.858 in predicting non-home discharges, unplanned reoperations, and major complications, respectively. The LightGBM models yielded AUROCs of 0.812 and 0.817 in predicting prolonged LOS, and 30-day readmissions, respectively.

CONCLUSION

The potential of ML approaches to predict postoperative outcomes following spine surgery is significant. As the volume of data in spine surgery continues to increase, the development of predictive models as clinically relevant decision-making tools could significantly improve risk assessment and prognosis. Here, we present an accessible predictive model for predicting short-term postoperative outcomes following CLP intended to achieve the stated objectives.

Comparing supervised and semi-supervised machine learning approaches in NTCP modeling to predict complications in head and neck cancer patients

Background and purpose

Head and neck cancer (HNC) patients treated with radiotherapy often suffer from radiation-induced toxicities. Normal Tissue Complication Probability (NTCP) modeling can be used to determine the probability to develop these toxicities based on patient, tumor, treatment and dose characteristics. Since the currently used NTCP models are developed using supervised methods that discard unlabeled patient data, we assessed whether the addition of unlabeled patient data by using semi-supervised modeling would gain predictive performance.

Materials and methods

The semi-supervised method of self-training was compared to supervised regression methods with and without prior multiple imputation by chained equation (MICE). The models were developed for the most common toxicity outcomes in HNC patients, xerostomia (dry mouth) and dysphagia (difficulty swallowing), measured at six months after treatment, in a development cohort of 750 HNC patients. The models were externally validated in a validation cohort of 395 HNC patients. Model performance was assessed by discrimination and calibration.

Results

MICE and self-training did not improve performance in terms of discrimination or calibration at external validation compared to current regression models. In addition, the relative performance of the different models did not change upon a decrease in the amount of (labeled) data available for model development. Models using ridge regression outperformed the logistic models for the dysphagia outcome.

Conclusion

Since there was no apparent gain in the addition of unlabeled patient data by using the semi-supervised method of self-training or MICE, the supervised regression models would still be preferred in current NTCP modeling for HNC patients.

Development of a Prediction Score for Evaluation of Extubation Readiness in Neurosurgical Patients with Mechanical Ventilation

BACKGROUND

There is no widely accepted consensus on the weaning and extubating protocols for neurosurgical patients, leading to heterogeneity in clinical practices and high rates of delayed extubation and extubation failure-related health complications.

METHODS

In this single-center prospective observational diagnostic study, mechanically ventilated neurosurgical patients with extubation attempts were consecutively enrolled for 1 yr. Responsive physicians were surveyed for the reasons for delayed extubation and developed the Swallowing, Tongue protrusion, Airway protection reflected by spontaneous and suctioning cough, and Glasgow Coma Scale Evaluation (STAGE) score to predict the extubation success for neurosurgical patients already meeting other general extubation criteria.

RESULTS

A total of 3,171 patients were screened consecutively, and 226 patients were enrolled in this study. The rates of delayed extubation and extubation failure were 25% (57 of 226) and 19% (43 of 226), respectively. The most common reasons for the extubation delay were weak airway-protecting function and poor consciousness. The area under the receiver operating characteristics curve of the total STAGE score associated with extubation success was 0.72 (95% CI, 0.64 to 0.79). Guided by the highest Youden index, the cutoff point for the STAGE score was set at 6 with 59% (95% CI, 51 to 66%) sensitivity, 74% (95% CI, 59 to 86%) specificity, 90% (95% CI, 84 to 95%) positive predictive value, and 30% (95% CI, 21 to 39%) negative predictive value. At STAGE scores of 9 or higher, the model exhibited a 100% (95% CI, 90 to 100%) specificity and 100% (95% CI, 72 to 100%) positive predictive value for predicting extubation success.

CONCLUSIONS

After a survey of the reasons for delayed extubation, the STAGE scoring system was developed to better predict the extubation success rate. This scoring system has promising potential in predicting extubation readiness and may help clinicians avoid delayed extubation and failed extubation-related health complications in neurosurgical patients.

EDITOR’S PERSPECTIVE

Development of a prediction model for the acquisition of extended spectrum beta-lactam-resistant organisms in U.S. international travellers

BACKGROUND

Extended spectrum beta-lactamase producing Enterobacterales (ESBL-PE) present a risk to public health by limiting the efficacy of multiple classes of beta-lactam antibiotics against infection. International travellers may acquire these organisms and identifying individuals at high risk of acquisition could help inform clinical treatment or prevention strategies.

METHODS

We used data collected from a cohort of 528 international travellers enrolled in a multicentre US-based study to derive a clinical prediction rule (CPR) to identify travellers who developed ESBL-PE colonization, defined as those with new ESBL positivity in stool upon return to the United States. To select candidate features, we used data collected from pre-travel and post-travel questionnaires, alongside destination-specific data from external sources. We utilized LASSO regression for feature selection, followed by random forest or logistic regression modelling, to derive a CPR for ESBL acquisition.

RESULTS

A CPR using machine learning and logistic regression on 10 features has an internally cross-validated area under the receiver operating characteristic curve (cvAUC) of 0.70 (95% confidence interval 0.69-0.71). We also demonstrate that a four-feature model performs similarly to the 10-feature model, with a cvAUC of 0.68 (95% confidence interval 0.67-0.69). This model uses traveller's diarrhoea, and antibiotics as treatment, destination country waste management rankings and destination regional probabilities as predictors.

CONCLUSIONS

We demonstrate that by integrating traveller characteristics with destination-specific data, we could derive a CPR to identify those at highest risk of acquiring ESBL-PE during international travel.

Prognostic models predicting transition to psychotic disorder using blood-based biomarkers: a systematic review and critical appraisal

Accumulating evidence suggests individuals with psychotic disorder show abnormalities in metabolic and inflammatory processes. Recently, several studies have employed blood-based predictors in models predicting transition to psychotic disorder in risk-enriched populations. A systematic review of the performance and methodology of prognostic models using blood-based biomarkers in the prediction of psychotic disorder from risk-enriched populations is warranted. Databases (PubMed, EMBASE and PsycINFO) were searched for eligible texts from 1998 to 15/05/2023, which detailed model development or validation studies. The checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was used to guide data extraction from eligible texts and the Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias and applicability of the studies. A narrative synthesis of the included studies was performed. Seventeen eligible studies were identified: 16 eligible model development studies and one eligible model validation study. A wide range of biomarkers were assessed, including nucleic acids, proteins, metabolites, and lipids. The range of C-index (area under the curve) estimates reported for the models was 0.67-1.00. No studies assessed model calibration. According to PROBAST criteria, all studies were at high risk of bias in the analysis domain. While a wide range of potentially predictive biomarkers were identified in the included studies, most studies did not account for overfitting in model performance estimates, no studies assessed calibration, and all models were at high risk of bias according to PROBAST criteria. External validation of the models is needed to provide more accurate estimates of their performance. Future studies which follow the latest available methodological and reporting guidelines and adopt strategies to accommodate required sample sizes for model development or validation will clarify the value of including blood-based biomarkers in models predicting psychosis.

Prognosis Individualized: Survival predictions for WHO grade II and III gliomas with a machine learning-based web application

WHO grade II and III gliomas demonstrate diverse biological behaviors resulting in variable survival outcomes. In the context of glioma prognosis, machine learning (ML) approaches could facilitate the navigation through the maze of factors influencing survival, aiding clinicians in generating more precise and personalized survival predictions. Here we report the utilization of ML models in predicting survival at 12, 24, 36, and 60 months following grade II and III glioma diagnosis. From the National Cancer Database, we analyze 10,001 WHO grade II and 11,456 grade III cranial gliomas. Using the area under the receiver operating characteristic (AUROC) values, we deploy the top-performing models in a web application for individualized predictions. SHapley Additive exPlanations (SHAP) enhance the interpretability of the models. Top-performing predictive models are the ones built with LightGBM and Random Forest algorithms. For grade II gliomas, the models yield AUROC values ranging from 0.813 to 0.896 for predicting mortality across different timeframes, and for grade III gliomas, the models yield AUROCs ranging from 0.855 to 0.878. ML models provide individualized survival forecasts for grade II and III glioma patients across multiple clinically relevant time points. The user-friendly web application represents a pioneering digital tool to potentially integrate predictive analytics into neuro-oncology clinical practice, to empower prognostication and personalize clinical decision-making.

Deep learning-derived spatial organization features on histology images predicts prognosis in colorectal liver metastasis patients after hepatectomy

Histopathological images of colorectal liver metastases (CRLM) contain rich morphometric information that may predict patients' outcomes. However, to our knowledge, no study has reported any practical deep learning framework based on the histology images of CRLM, and their direct association with prognosis remains largely unknown. In this study, we developed a deep learning-based framework for fully automated tissue classification and quantification of clinically relevant spatial organization features (SOFs) in H&E-stained images of CRLM. The SOFs based risk-scoring system demonstrated a strong and robust prognostic value that is independent of the current clinical risk score (CRS) system in independent clinical cohorts. Our framework enables fully automated tissue classification of H&E images of CRLM, which could significantly reduce assessment subjectivity and the workload of pathologists. The risk-scoring system provides a time- and cost-efficient tool to assist clinical decision-making for patients with CRLM, which could potentially be implemented in clinical practice.

Constructing training set using distance between learnt graphical models of time series data on patient physiology, to predict disease scores

Interventional endeavours in medicine include prediction of a score that parametrises a new subject's susceptibility to a given disease, at the pre-onset stage. Here, for the first time, we provide reliable learning of such a score in the context of the potentially-terminal disease VOD, that often arises after bone marrow transplants. Indeed, the probability of surviving VOD, is correlated with early intervention. In our work, the VOD-score of each patient in a retrospective cohort, is defined as the distance between the (posterior) probability of a random graph variable-given the inter-variable partial correlation matrix of the time series data on variables that represent different aspects of patient physiology-and that given such time series data of an arbitrarily-selected reference patient. Such time series data is recorded from a pre-transplant to a post-transplant time, for each patient in this cohort, though the data available for distinct patients bear differential temporal coverage, owing to differential patient longevities. Each graph is a Soft Random Geometric Graph drawn in a probabilistic metric space, and the computed inter-graph distance is oblivious to the length of the time series data. The VOD-score learnt in this way, and the corresponding pre-transplant parameter vector of each patient in this retrospective cohort, then results in the training data, using which we learn the function that takes VOD-score as its input, and outputs the vector of pre-transplant parameters. We model this function with a vector-variate Gaussian Process, the covariance structure of which is kernel parametrised. Such modelling is easier than if the score variable were the output. Then for any prospective patient, whose pre-transplant variables are known, we learn the VOD-score (and the hyperparameters of the covariance kernel), using Markov Chain Monte Carlo based inference.

Machine learning risk estimation and prediction of death in continuing care facilities using administrative data

In this study, we aimed to identify the factors that were associated with mortality among continuing care residents in Alberta, during the coronavirus disease 2019 (COVID-19) pandemic. We achieved this by leveraging and linking various administrative datasets together. Then, we examined pre-processing methods in terms of prediction performance. Finally, we developed several machine learning models and compared the results of these models in terms of performance. We conducted a retrospective cohort study of all continuing care residents in Alberta, Canada, from March 1, 2020, to March 31, 2021. We used a univariable and a multivariable logistic regression (LR) model to identify predictive factors of 60-day all-cause mortality by estimating odds ratios (ORs) with a 95% confidence interval. To determine the best sensitivity-specificity cut-off point, the Youden index was employed. We developed several machine learning models to determine the best model regarding performance. In this cohort study, increased age, male sex, symptoms, previous admissions, and some specific comorbidities were associated with increased mortality. Machine learning and pre-processing approaches offer a potentially valuable method for improving risk prediction for mortality, but more work is needed to show improvement beyond standard risk factors.

Applying machine learning algorithms to develop a survival prediction model for lung adenocarcinoma based on genes related to fatty acid metabolism

Background: The progression of lung adenocarcinoma (LUAD) may be related to abnormal fatty acid metabolism (FAM). The present study investigated the relationship between FAM-related genes and LUAD prognosis. Methods: LUAD samples from The Cancer Genome Atlas were collected. The scores of FAM-associated pathways from the Kyoto Encyclopedia of Genes and Genomes website were calculated using the single sample gene set enrichment analysis. ConsensusClusterPlus and cumulative distribution function were used to classify molecular subtypes for LUAD. Key genes were obtained using limma package, Cox regression analysis, and six machine learning algorithms (GBM, LASSO, XGBoost, SVM, random forest, and decision trees), and a RiskScore model was established. According to the RiskScore model and clinical features, a nomogram was developed and evaluated for its prediction performance using a calibration curve. Differences in immune abnormalities among patients with different subtypes and RiskScores were analyzed by the Estimation of STromal and Immune cells in MAlignant Tumours using Expression data, CIBERSORT, and single sample gene set enrichment analysis. Patients' drug sensitivity was predicted by the pRRophetic package in R language. Results: LUAD samples had lower scores of FAM-related pathways. Three molecular subtypes (C1, C2, and C3) were defined. Analysis on differential prognosis showed that the C1 subtype had the most favorable prognosis, followed by the C2 subtype, and the C3 subtype had the worst prognosis. The C3 subtype had lower immune infiltration. A total of 12 key genes (SLC2A1, PKP2, FAM83A, TCN1, MS4A1, CLIC6, UBE2S, RRM2, CDC45, IGF2BP1, ANGPTL4, and CD109) were screened and used to develop a RiskScore model. Survival chance of patients in the high-RiskScore group was significantly lower. The low-RiskScore group showed higher immune score and higher expression of most immune checkpoint genes. Patients with a high RiskScore were more likely to benefit from the six anticancer drugs we screened in this study. Conclusion: We developed a RiskScore model using FAM-related genes to help predict LUAD prognosis and develop new targeted drugs.

Prognostic models for surgical-site infection in gastrointestinal surgery: systematic review

BACKGROUND

Identification of patients at high risk of surgical-site infection may allow clinicians to target interventions and monitoring to minimize associated morbidity. The aim of this systematic review was to identify and evaluate prognostic tools for the prediction of surgical-site infection in gastrointestinal surgery.

METHODS

This systematic review sought to identify original studies describing the development and validation of prognostic models for 30-day SSI after gastrointestinal surgery (PROSPERO: CRD42022311019). MEDLINE, Embase, Global Health, and IEEE Xplore were searched from 1 January 2000 to 24 February 2022. Studies were excluded if prognostic models included postoperative parameters or were procedure specific. A narrative synthesis was performed, with sample-size sufficiency, discriminative ability (area under the receiver operating characteristic curve), and prognostic accuracy compared.

RESULTS

Of 2249 records reviewed, 23 eligible prognostic models were identified. A total of 13 (57 per cent) reported no internal validation and only 4 (17 per cent) had undergone external validation. Most identified operative contamination (57 per cent, 13 of 23) and duration (52 per cent, 12 of 23) as important predictors; however, there remained substantial heterogeneity in other predictors identified (range 2-28). All models demonstrated a high risk of bias due to the analytic approach, with overall low applicability to an undifferentiated gastrointestinal surgical population. Model discrimination was reported in most studies (83 per cent, 19 of 23); however, calibration (22 per cent, 5 of 23) and prognostic accuracy (17 per cent, 4 of 23) were infrequently assessed. Of externally validated models (of which there were four), none displayed 'good' discrimination (area under the receiver operating characteristic curve greater than or equal to 0.7).

CONCLUSION

The risk of surgical-site infection after gastrointestinal surgery is insufficiently described by existing risk-prediction tools, which are not suitable for routine use. Novel risk-stratification tools are required to target perioperative interventions and mitigate modifiable risk factors.

Machine learning clinical prediction models for acute kidney injury: the impact of baseline creatinine on prediction efficacy

BACKGROUND

There are many Machine Learning (ML) models which predict acute kidney injury (AKI) for hospitalised patients. While a primary goal of these models is to support clinical decision-making, the adoption of inconsistent methods of estimating baseline serum creatinine (sCr) may result in a poor understanding of these models' effectiveness in clinical practice. Until now, the performance of such models with different baselines has not been compared on a single dataset. Additionally, AKI prediction models are known to have a high rate of false positive (FP) events regardless of baseline methods. This warrants further exploration of FP events to provide insight into potential underlying reasons.

OBJECTIVE

The first aim of this study was to assess the variance in performance of ML models using three methods of baseline sCr on a retrospective dataset. The second aim was to conduct an error analysis to gain insight into the underlying factors contributing to FP events.

MATERIALS AND METHODS

The Intensive Care Unit (ICU) patients of the Medical Information Mart for Intensive Care (MIMIC)-IV dataset was used with the KDIGO (Kidney Disease Improving Global Outcome) definition to identify AKI episodes. Three different methods of estimating baseline sCr were defined as (1) the minimum sCr, (2) the Modification of Diet in Renal Disease (MDRD) equation and the minimum sCr and (3) the MDRD equation and the mean of preadmission sCr. For the first aim of this study, a suite of ML models was developed for each baseline and the performance of the models was assessed. An analysis of variance was performed to assess the significant difference between eXtreme Gradient Boosting (XGB) models across all baselines. To address the second aim, Explainable AI (XAI) methods were used to analyse the XGB errors with Baseline 3.

RESULTS

Regarding the first aim, we observed variances in discriminative metrics and calibration errors of ML models when different baseline methods were adopted. Using Baseline 1 resulted in a 14% reduction in the f1 score for both Baseline 2 and Baseline 3. There was no significant difference observed in the results between Baseline 2 and Baseline 3. For the second aim, the FP cohort was analysed using the XAI methods which led to relabelling data with the mean of sCr in 180 to 0 days pre-ICU as the preferred sCr baseline method. The XGB model using this relabelled data achieved an AUC of 0.85, recall of 0.63, precision of 0.54 and f1 score of 0.58. The cohort size was 31,586 admissions, of which 5,473 (17.32%) had AKI.

CONCLUSION

In the absence of a widely accepted method of baseline sCr, AKI prediction studies need to consider the impact of different baseline methods on the effectiveness of ML models and their potential implications in real-world implementations. The utilisation of XAI methods can be effective in providing insight into the occurrence of prediction errors. This can potentially augment the success rate of ML implementation in routine care.

Development of an Asthma Exacerbation Risk Prediction Model for Conversational Use by Adults in England

Background

Improving accurate risk assessment of asthma exacerbations, and reduction via relevant behaviour change among people with asthma could save lives and reduce health care costs. We developed a simple personalised risk prediction model for asthma exacerbations using factors collected in routine healthcare data for use in a risk modelling feature for automated conversational systems.

Methods

We used pseudonymised primary care electronic healthcare records from the Clinical Practice Research Datalink (CPRD) Aurum database in England. We combined variables for prediction of asthma exacerbations using logistic regression including age, gender, ethnicity, Index of Multiple Deprivation, geographical region and clinical variables related to asthma events.

Results

We included 1,203,741 patients divided into three cohorts to implement temporal validation: 898,763 (74.7%) in the training sample, 226,754 (18.8%) in the testing sample and 78,224 (6.5%) in the validation sample. The Area under the ROC curve (AUC) for the full model was 0.72 and for the restricted model was 0.71. Using a cut-off point of 0.1, approximately 27 asthma reviews by clinicians per 100 patients would be prevented compared with a strategy that all patients are regarded as high risk. Compared with patients without an exacerbation, patients who exacerbated were older, more likely to be female, prescribed more SABA and ICS in the preceding 12 months, have history of GORD, COPD, anxiety, depression, live in very deprived areas and have more severe disease.

Conclusion

Using information available from routinely collected electronic healthcare record data, we developed a model that has moderate ability to separate patients who had an asthma exacerbation within 3 months from their index date from patients who did not. When comparing this model with a simplified model with variables that can easily be self-reported through a WhatsApp chatbot, we have shown that the predictive performance of the model is not substantially different.

External validation of the Health Care Homes hospital admission risk stratification tool in the Aboriginal Australian population of the Northern Territory

Objective This study aimed to externally validate the Commonwealth's Health Care Homes (HCH) algorithm for Aboriginal Australians living in the Northern Territory (NT). Methods A retrospective cohort study design using linked primary health care (PHC) and hospital data was used to analyse the performance of the HCH algorithm in predicting the risk of hospitalisation for the NT study population. The study population consisted of Aboriginal Australians residing in the NT who have visited a PHC clinic at one of the 54 NT Government clinics at least once between 1 January 2013 and 31 December 2017. Predictors of hospitalisation included demographics, patient observations, medications, diagnoses, pathology results and previous hospitalisation. Results There were a total of 3256 (28.5%) emergency attendances or preventable hospitalisations during the study period. The HCH algorithm had an area under the receiver operating characteristic curve (AUC) of 0.58 for the NT remote Aboriginal population, compared with 0.66 in the Victorian cohort. A refitted model including 'previous hospitalisation' had an AUC of 0.72, demonstrating better discrimination than the HCH algorithm. Calibration was also improved in the refitted model, with an intercept of 0.00 and a slope of 1.00, compared with an intercept of 1.29 and a slope of 0.55 in the HCH algorithm. Conclusion The HCH algorithm performed poorly on the NT cohort compared with the Victorian cohort, due to differences in population demographics and burden of disease. A population-specific hospitalisation risk algorithm is required for the NT.

Preoperative Spirometry in Patients With Known or Suspected Chronic Obstructive Pulmonary Disease Undergoing Major Surgery: The Prospective Observational PREDICT Study

BACKGROUND

Pulmonary function tests (PFTs) such as spirometry and blood gas analysis have been claimed to improve preoperative pulmonary risk assessment, but the scientific literature is conflicting. The Preoperative Diagnostic Tests for Pulmonary Risk Assessment in Chronic Obstructive Pulmonary Disease (PREDICT) study aimed to determine whether preoperative PFTs improve the prediction of postoperative pulmonary complications (PPCs) in patients with known or suspected chronic obstructive pulmonary disease (COPD) undergoing major surgery. A secondary aim was to determine whether the Global Initiative for Chronic Obstructive Lung Diseases (GOLD) classification of airflow limitation severity (grades I-IV) is associated with PPC.

METHODS

In this prospective, single-center study, patients with GOLD key indicators for COPD scheduled for major surgery received PFTs. Patients with confirmed COPD (forced expiratory volume in 1 second [FEV1]/forced vital capacity [FVC] ≤0.7) were included in the COPD cohort and compared with a reference cohort without COPD. We developed 3 multivariable risk prediction models and compared their ability to predict PPC: the "standard model" (medical preconditions, and sociodemographic and surgical data), the "COPD assessment model" (additional GOLD key indicators, pack-years, and poor exercise capacity), and the "PFT model" (additional PFT parameters selected by adaptive least absolute shrinkage and selection operator [LASSO] regression). Multiple LASSO regressions were used for cross-validation.

RESULTS

A total of 31,714 patients were assessed for eligibility; 1271 individuals received PFTs. Three hundred twenty patients (240 with confirmed COPD: 78 GOLD I, 125 GOLD II, 28 GOLD III, 9 GOLD IV, and 80 without COPD) completed follow-up. The diagnostic performance was similar among the standard model (cross-validated area under the curve [cvAUC], 0.723; bias-corrected bootstrapped [bc-b] 95% confidence interval [CI], 0.663-0.775), COPD assessment model (cvAUC, 0.724; bc-b 95% CI, 0.662-0.777), and PFT model (cvAUC, 0.729; bc-b 95% CI, 0.668-0.782). Previously known COPD was an independent predictor in the standard and COPD assessment model. %FEV1 PRED was the only PFT parameter selected by LASSO regression and was an independent predictor in the PFT model (adjusted odds ratios [OR], 0.98; 95% CI, 0.967-.0.998; P = .030). The risk for PPC significantly increased with GOLD grades ( P < .001). COPD was newly diagnosed in 53.8% of the patients with confirmed COPD; however, these individuals were not at increased risk for PPC ( P = .338).

CONCLUSIONS

COPD is underdiagnosed in surgical patients. Patients with newly diagnosed COPD commonly presented with low GOLD severity grades and were not at higher risk for PPC. Neither a structured COPD-specific assessment nor preoperative PFTs added incremental diagnostic value to the standard clinical preassessment in patients with known or suspected COPD. Unnecessary postponement of surgery and undue health care costs can be avoided.

The role of CXCL8 and CCNB1 in predicting hepatocellular carcinoma in the context of cirrhosis: implications for early detection and immune-based therapies

BACKGROUND

Cirrhosis is a serious condition characterized by the replacement of healthy liver tissue with scar tissue, which can progress to liver failure if left untreated. Hepatocellular carcinoma (HCC) is a concerning complication of cirrhosis. It can be challenge to identify individuals with cirrhosis who are at high risk of developing HCC, particularly in the absence of known risk factors.

METHODS

In this study, statistical and bioinformatics methods were utilized to construct a protein-protein interaction network and identify disease-related hub genes. We analyzed two hub genes, CXCL8 and CCNB1, and developed a mathematical model to predict the likelihood of developing HCC in individuals with cirrhosis. We also investigated immune cell infiltration, functional analysis under ontology terms, pathway analysis, distinct clusters of cells, and protein-drug interactions.

RESULTS

The results indicated that CXCL8 and CCNB1 were associated with the development of cirrhosis-induced HCC. A prognostic model based on these two genes was able to predict the occurrence and survival time of HCC. In addition, the candidate drugs were also discovered based on our model.

CONCLUSION

The findings offer the potential for earlier detection of cirrhosis-induced HCC and provide a new instrument for clinical diagnosis, prognostication, and the development of immunological medications. This study also identified distinct clusters of cells in HCC patients using UMAP plot analysis and analyzed the expression of CXCL8 and CCNB1 within these cells, indicating potential therapeutic opportunities for targeted drug therapies to benefit HCC patients.

Predicting future sleep problems in young autistic children

LAY ABSTRACT

Sleep problems are common in autistic children and negatively impact daytime functioning. A method for predicting sleep problems could help with treatment and prevention of such problems. This study aimed to determine predictors of sleep problems among young autistic children. Study participants consisted of autistic children aged 2-5 years who did not have sleep problems at a first visit (Autism Treatment Network Registry) and had sleep data available at a subsequent visit (Registry Call-Back Assessment study). Sleep problems for five study cohorts of children were defined by different methods, including parent questionnaires and parent- or clinician-report of sleep problems. We found that self-injurious behavior, sensory issues, dental problems, and lower primary caregiver education level were significant risk factors of future sleep problems. These predictors may help clinicians provide prevention or earlier treatment for children who are at risk of developing sleep problems.

Development and External Validation of a Machine Learning Model for Prediction of Lymph Node Metastasis in Patients with Prostate Cancer

BACKGROUND

Pelvic lymph node dissection (PLND) is the gold standard for diagnosis of lymph node involvement (LNI) in patients with prostate cancer. The Roach formula, Memorial Sloan Kettering Cancer Center (MSKCC) calculator, and Briganti 2012 nomogram are elegant and simple traditional tools used to estimate the risk of LNI and select patients for PLND.

OBJECTIVE

To determine whether machine learning (ML) can improve patient selection and outperform currently available tools for predicting LNI using similar readily available clinicopathologic variables.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective data for patients treated with surgery and PLND between 1990 and 2020 in two academic institutions were used.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

We trained three models (two logistic regression models and one gradient-boosted trees-based model [XGBoost]) on data provided from one institution (n = 20267) with age, prostate-specific antigen (PSA) levels, clinical T stage, percentage positive cores, and Gleason scores as inputs. We externally validated these models using data from another institution (n = 1322) and compared their performance to that of the traditional models using the area under the receiver operating characteristic curve (AUC), calibration, and decision curve analysis (DCA).

RESULTS AND LIMITATIONS

LNI was present in 2563 patients (11.9%) overall, and in 119 patients (9%) in the validation data set. XGBoost had the best performance among all the models. On external validation, its AUC outperformed that of the Roach formula by 0.08 (95% confidence interval [CI] 0.042-0.12), the MSKCC nomogram by 0.05 (95% CI 0.016-0.070), and the Briganti nomogram by 0.03 (95% CI 0.0092-0.051; all p < 0.05). It also had better calibration and clinical utility in terms of net benefit on DCA across relevant clinical thresholds. The main limitation of the study is its retrospective design.

CONCLUSIONS

Taking all measures of performance together, ML using standard clinicopathologic variables outperforms traditional tools in predicting LNI.

PATIENT SUMMARY

Determining the risk of cancer spread to the lymph nodes in patients with prostate cancer allows surgeons to perform lymph node dissection only in patients who need it and avoid the side effects of the procedure in those who do not. In this study, we used machine learning to develop a new calculator to predict the risk of lymph node involvement that outperformed traditional tools currently used by oncologists.

Breast cancer risk stratification using genetic and non-genetic risk assessment tools for 246,142 women in the UK Biobank

PURPOSE

The benefit of using individual risk prediction tools to identify high-risk individuals for breast cancer (BC) screening is uncertain, despite the personalized approach of risk-based screening.

METHODS

We studied the overlap of predicted high-risk individuals among 246,142 women enrolled in the UK Biobank. Risk predictors assessed include the Gail model (Gail), BC family history (FH, binary), BC polygenic risk score (PRS), and presence of loss-of-function (LoF) variants in BC predisposition genes. Youden J-index was used to select optimal thresholds for defining high-risk.

RESULTS

In total, 147,399 were considered at high risk for developing BC within the next 2 years by at least 1 of the 4 risk prediction tools examined (Gail2-year > 0.5%: 47%, PRS2-yea r > 0.7%: 30%, FH: 6%, and LoF: 1%); 92,851 (38%) were flagged by only 1 risk predictor. The overlap between individuals flagged as high-risk because of genetic (PRS) and Gail model risk factors was 30%. The best-performing combinatorial model comprises a union of high-risk women identified by PRS, FH, and, LoF (AUC2-year [95% CI]: 62.2 [60.8 to 63.6]). Assigning individual weights to each risk prediction tool increased discriminatory ability.

CONCLUSION

Risk-based BC screening may require a multipronged approach that includes PRS, predisposition genes, FH, and other recognized risk factors.

Predicting the onset of diabetes-related complications after a diabetes diagnosis with machine learning algorithms

AIMS

Using machine learning algorithms and administrative data, we aimed to predict the risk of being diagnosed with several diabetes-related complications after one-, two- and three-year post-diabetes diagnosis.

METHODS

We used longitudinal data from administrative registers of 610,019 individuals in Catalonia with a diagnosis of diabetes and checked the presence of several complications after diabetes onset from 2013 to 2017: hypertension, renal failure, myocardial infarction, cardiovascular disease, retinopathy, congestive heart failure, cerebrovascular disease, peripheral vascular disease and stroke. Four different machine learning (ML) algorithms (logistic regression (LR), Decision tree (DT), Random Forest (RF), and Extreme Gradient Boosting (XGB)) will be used to assess their prediction performance and to evaluate the prediction accuracy of complications changes over the period considered.

RESULTS

610,019 people with diabetes were included. After three years since diabetes diagnosis, the area under the curve values ranged from 60% (retinopathy) to 69% (congestive heart failure), whereas accuracy rates varied between 60% (retinopathy) to 75% (hypertension). RF was the most relevant technique for hypertension, myocardial and retinopathy, and LR for the rest of the comorbidities. The Shapley additive explanations values showed that age was associated with an elevated risk for all diabetes-related complications except retinopathy. Gender, other comorbidities, co-payment levels and age were the most relevant factors for comorbidity diagnosis prediction.

CONCLUSIONS

Our ML models allow for the identification of individuals newly diagnosed with diabetes who are at increased risk of developing diabetes-related complications. The prediction performance varied across complications but within acceptable ranges as prediction tools.

Machine learning in precision diabetes care and cardiovascular risk prediction

Artificial intelligence and machine learning are driving a paradigm shift in medicine, promising data-driven, personalized solutions for managing diabetes and the excess cardiovascular risk it poses. In this comprehensive review of machine learning applications in the care of patients with diabetes at increased cardiovascular risk, we offer a broad overview of various data-driven methods and how they may be leveraged in developing predictive models for personalized care. We review existing as well as expected artificial intelligence solutions in the context of diagnosis, prognostication, phenotyping, and treatment of diabetes and its cardiovascular complications. In addition to discussing the key properties of such models that enable their successful application in complex risk prediction, we define challenges that arise from their misuse and the role of methodological standards in overcoming these limitations. We also identify key issues in equity and bias mitigation in healthcare and discuss how the current regulatory framework should ensure the efficacy and safety of medical artificial intelligence products in transforming cardiovascular care and outcomes in diabetes.

Harmonization of supervised machine learning practices for efficient source attribution of Listeria monocytogenes based on genomic data

BACKGROUND

Genomic data-based machine learning tools are promising for real-time surveillance activities performing source attribution of foodborne bacteria such as Listeria monocytogenes. Given the heterogeneity of machine learning practices, our aim was to identify those influencing the source prediction performance of the usual holdout method combined with the repeated k-fold cross-validation method.

METHODS

A large collection of 1 100 L. monocytogenes genomes with known sources was built according to several genomic metrics to ensure authenticity and completeness of genomic profiles. Based on these genomic profiles (i.e. 7-locus alleles, core alleles, accessory genes, core SNPs and pan kmers), we developed a versatile workflow assessing prediction performance of different combinations of training dataset splitting (i.e. 50, 60, 70, 80 and 90%), data preprocessing (i.e. with or without near-zero variance removal), and learning models (i.e. BLR, ERT, RF, SGB, SVM and XGB). The performance metrics included accuracy, Cohen's kappa, F1-score, area under the curves from receiver operating characteristic curve, precision recall curve or precision recall gain curve, and execution time.

RESULTS

The testing average accuracies from accessory genes and pan kmers were significantly higher than accuracies from core alleles or SNPs. While the accuracies from 70 and 80% of training dataset splitting were not significantly different, those from 80% were significantly higher than the other tested proportions. The near-zero variance removal did not allow to produce results for 7-locus alleles, did not impact significantly the accuracy for core alleles, accessory genes and pan kmers, and decreased significantly accuracy for core SNPs. The SVM and XGB models did not present significant differences in accuracy between each other and reached significantly higher accuracies than BLR, SGB, ERT and RF, in this order of magnitude. However, the SVM model required more computing power than the XGB model, especially for high amount of descriptors such like core SNPs and pan kmers.

CONCLUSIONS

In addition to recommendations about machine learning practices for L. monocytogenes source attribution based on genomic data, the present study also provides a freely available workflow to solve other balanced or unbalanced multiclass phenotypes from binary and categorical genomic profiles of other microorganisms without source code modifications.

Machine learning and EEG can classify passive viewing of discrete categories of visual stimuli but not the observation of pain

Previous studies have demonstrated the potential of machine learning (ML) in classifying physical pain from non-pain states using electroencephalographic (EEG) data. However, the application of ML to EEG data to categorise the observation of pain versus non-pain images of human facial expressions or scenes depicting pain being inflicted has not been explored. The present study aimed to address this by training Random Forest (RF) models on cortical event-related potentials (ERPs) recorded while participants passively viewed faces displaying either pain or neutral expressions, as well as action scenes depicting pain or matched non-pain (neutral) scenarios. Ninety-one participants were recruited across three samples, which included a model development group (n = 40) and a cross-subject validation group (n = 51). Additionally, 25 participants from the model development group completed a second experimental session, providing a within-subject temporal validation sample. The analysis of ERPs revealed an enhanced N170 component in response to faces compared to action scenes. Moreover, an increased late positive potential (LPP) was observed during the viewing of pain scenes compared to neutral scenes. Additionally, an enhanced P3 response was found when participants viewed faces displaying pain expressions compared to neutral expressions. Subsequently, three RF models were developed to classify images into faces and scenes, neutral and pain scenes, and neutral and pain expressions. The RF model achieved classification accuracies of 75%, 64%, and 69% for cross-validation, cross-subject, and within-subject classifications, respectively, along with reasonably calibrated predictions for the classification of face versus scene images. However, the RF model was unable to classify pain versus neutral stimuli above chance levels when presented with subsequent tasks involving images from either category. These results expand upon previous findings by externally validating the use of ML in classifying ERPs related to different categories of visual images, namely faces and scenes. The results also indicate the limitations of ML in distinguishing pain and non-pain connotations using ERP responses to the passive viewing of visually similar images.

A prognostic model for systemic lupus erythematosus-associated pulmonary arterial hypertension: CSTAR-PAH cohort study

BACKGROUND

Pulmonary arterial hypertension is a major cause of death in systemic lupus erythematosus, but there are no tools specialized for predicting survival in systemic lupus erythematosus-associated pulmonary arterial hypertension.

RESEARCH QUESTION

To develop a practical model for predicting long-term prognosis in patients with systemic lupus erythematosus-associated pulmonary arterial hypertension.

METHODS

A prognostic model was developed from a multicenter, longitudinal national cohort of consecutively evaluated patients with systemic lupus erythematosus-associated pulmonary arterial hypertension. The study was conducted between November 2006 and February 2020. All-cause death was defined as the endpoint. Cox regression and least absolute shrinkage and selection operators were used to fit the model. Internal validation of the model was assessed by discrimination and calibration using bootstrapping.

RESULTS

Of 310 patients included in the study, 81 (26.1%) died within a median follow-up of 5.94 years (interquartile range 4.67-7.46). The final prognostic model included eight variables: modified World Health Organization functional class, 6-min walking distance, pulmonary vascular resistance, estimated glomerular filtration rate, thrombocytopenia, mild interstitial lung disease, N-terminal pro-brain natriuretic peptide/brain natriuretic peptide level, and direct bilirubin level. A 5-year death probability predictive algorithm was established and validated using the C-index (0.77) and a satisfactory calibration curve. Risk stratification was performed based on the predicted probability to improve clinical decision-making.

CONCLUSIONS

This new risk stratification model for systemic lupus erythematosus-associated pulmonary arterial hypertension may provide individualized prognostic probability using readily obtained clinical risk factors. External validation is required to demonstrate the accuracy of this model's predictions in diverse patient populations.

Dynamics in cardiac surgery: trends in population characteristics and the performance of the EuroSCORE II over time

OBJECTIVES

The aim of this study was to investigate the performance of the EuroSCORE II over time and dynamics in values of predictors included in the model.

METHODS

A cohort study was performed using data from the Netherlands Heart Registration. All cardiothoracic surgical procedures performed between 1 January 2013 and 31 December 2019 were included for analysis. Performance of the EuroSCORE II was assessed across 3-month intervals in terms of calibration and discrimination. For subgroups of major surgical procedures, performance of the EuroSCORE II was assessed across 12-month time intervals. Changes in values of individual EuroSCORE II predictors over time were assessed graphically.

RESULTS

A total of 103 404 cardiothoracic surgical procedures were included. Observed mortality risk ranged between 1.9% [95% confidence interval (CI) 1.6-2.4] and 3.6% (95% CI 2.6-4.4) across 3-month intervals, while the mean predicted mortality risk ranged between 3.4% (95% CI 3.3-3.6) and 4.2% (95% CI 3.9-4.6). The corresponding observed:expected ratios ranged from 0.50 (95% CI 0.46-0.61) to 0.95 (95% CI 0.74-1.16). Discriminative performance in terms of the c-statistic ranged between 0.82 (95% CI 0.78-0.89) and 0.89 (95% CI 0.87-0.93). The EuroSCORE II consistently overestimated mortality compared to observed mortality. This finding was consistent across all major cardiothoracic surgical procedures. Distributions of values of individual predictors varied broadly across predictors over time. Most notable trends were a decrease in elective surgery from 75% to 54% and a rise in patients with no or New York Heart Association I class heart failure from 27% to 33%.

CONCLUSIONS

The EuroSCORE II shows good discriminative performance, but consistently overestimates mortality risks of all types of major cardiothoracic surgical procedures in the Netherlands.

A Retrospective Cohort Study: Predicting 90-Day Mortality for ICU Trauma Patients with a Machine Learning Algorithm Using XGBoost Using MIMIC-III Database

Objective

The aim of this study was to develop and validate a machine learning-based predictive model that predicts 90-day mortality in ICU trauma patients.

Methods

Data of patients with severe trauma were extracted from the Medical Information Mart for Intensive Care III (MIMIC-III) database. The performances of mortality prediction models generated using nine machine learning extreme gradient boosting (XGBoost), logistic regression, random forest, AdaBoost, multilayer perceptron (MLP) neural networks, support vector machine (SVM), light gradient boosting machine (GBM), k nearest neighbors (KNN) and gaussian naive bayes (GNB). The performance of the model was evaluated in terms of discrimination, calibration and clinical application.

Results

We found that the accuracy, sensitivity, specificity, PPV, NPV and F1 score of our proposed XGBoost model were 82.8%, 79.7%, 77.6%, 51.2%, 91.5% and 0.624, respectively. Among the nine models, the XGBoost model performed best. Compared with traditional logistic regression, the calibration curves of the XGBoost model and decision curve analysis (DCA) performed well.

Conclusion

Our study shows that the XGBoost model outperforms other machine learning models in predicting 90-day mortality in trauma patients. It can be used to assist clinicians in the early identification of mortality risk factors and early intervention to reduce mortality.

Evidence of questionable research practices in clinical prediction models

BACKGROUND

Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result.

METHODS

We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.

RESULTS

The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.

CONCLUSIONS

The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.

Genetic Studies Investigating Susceptibility to Psoriatic Arthritis: A Narrative Review

PURPOSE

Approximately 30% of patients with psoriasis will develop psoriatic arthritis (PsA), leading to a decreased quality of life for the patient caused by increasing disability and additional health complications. The identification of risk factors for the development of PsA would facilitate the development of risk prediction models in which patients with psoriasis at high risk of developing PsA could be targeted in a stratified medicine approach, enabling early intervention and treatment. PsA is known to have a genetic contribution to susceptibility, and the identification of genetic risk factors that differentiate PsA from cutaneous-only psoriasis is a key area of research. This narrative review summarizes the discovery of genetic risk factors and, with the aid of a primer on risk prediction models, discusses their potential role for the classification of PsA risk and diagnosis.

METHODS

All relevant research articles were identified through searches of the PubMed database for literature published up until December 2022. Search terms included psoriatic arthritis, genetic susceptibility, genetic association, genome-wide association study, GWAS, prediction, and polygenic risk score.

FINDINGS

The current literature reveals considerable overlap between the genetic susceptibility loci for PsA and psoriasis. Several PsA-specific genetic risk factors have been reported, and most notably these implicate the HLA-B and IL23R genes. Efforts to include genetic risk factors in prediction models for the development of PsA have reported good discrimination.

IMPLICATIONS

Key messages emerging from this narrative are as follows: the limited number of PsA-specific susceptibility loci reported to date suggest larger studies are required, facilitated by international collaboration, to achieve the power to detect further genetic factors; the early promising results for genetic-based risk prediction require further validation in independent datasets; and risk prediction models combining clinical and genetic risk factors have yet to be explored.

Predicting whether patients will achieve minimal clinically important differences following hip or knee arthroplasty

Aims

A substantial fraction of patients undergoing knee arthroplasty (KA) or hip arthroplasty (HA) do not achieve an improvement as high as the minimal clinically important difference (MCID), i.e. do not achieve a meaningful improvement. Using three patient-reported outcome measures (PROMs), our aim was: 1) to assess machine learning (ML), the simple pre-surgery PROM score, and logistic-regression (LR)-derived performance in their prediction of whether patients undergoing HA or KA achieve an improvement as high or higher than a calculated MCID; and 2) to test whether ML is able to outperform LR or pre-surgery PROM scores in predictive performance.

Methods

MCIDs were derived using the change difference method in a sample of 1,843 HA and 1,546 KA patients. An artificial neural network, a gradient boosting machine, least absolute shrinkage and selection operator (LASSO) regression, ridge regression, elastic net, random forest, LR, and pre-surgery PROM scores were applied to predict MCID for the following PROMs: EuroQol five-dimension, five-level questionnaire (EQ-5D-5L), EQ visual analogue scale (EQ-VAS), Hip disability and Osteoarthritis Outcome Score-Physical Function Short-form (HOOS-PS), and Knee injury and Osteoarthritis Outcome Score-Physical Function Short-form (KOOS-PS).

Results

Predictive performance of the best models per outcome ranged from 0.71 for HOOS-PS to 0.84 for EQ-VAS (HA sample). ML statistically significantly outperformed LR and pre-surgery PROM scores in two out of six cases.

Conclusion

MCIDs can be predicted with reasonable performance. ML was able to outperform traditional methods, although only in a minority of cases.

Evaluating prediction model performance

This article highlights important performance metrics to consider when evaluating models developed for supervised classification or regression tasks using clinical data. When evaluating model performance, we detail the basics of confusion matrices, receiver operating characteristic curves, F1 scores, precision-recall curves, mean squared error, and other considerations. In this era, defined by the rapid proliferation of advanced prediction models, familiarity with various performance metrics beyond the area under the receiver operating characteristic curves and the nuances of evaluating model value upon implementation is essential to ensure effective resource allocation and optimal patient care delivery.

Advancing Naturalistic Affective Science with Deep Learning

People express their own emotions and perceive others' emotions via a variety of channels, including facial movements, body gestures, vocal prosody, and language. Studying these channels of affective behavior offers insight into both the experience and perception of emotion. Prior research has predominantly focused on studying individual channels of affective behavior in isolation using tightly controlled, non-naturalistic experiments. This approach limits our understanding of emotion in more naturalistic contexts where different channels of information tend to interact. Traditional methods struggle to address this limitation: manually annotating behavior is time-consuming, making it infeasible to do at large scale; manually selecting and manipulating stimuli based on hypotheses may neglect unanticipated features, potentially generating biased conclusions; and common linear modeling approaches cannot fully capture the complex, nonlinear, and interactive nature of real-life affective processes. In this methodology review, we describe how deep learning can be applied to address these challenges to advance a more naturalistic affective science. First, we describe current practices in affective research and explain why existing methods face challenges in revealing a more naturalistic understanding of emotion. Second, we introduce deep learning approaches and explain how they can be applied to tackle three main challenges: quantifying naturalistic behaviors, selecting and manipulating naturalistic stimuli, and modeling naturalistic affective processes. Finally, we describe the limitations of these deep learning methods, and how these limitations might be avoided or mitigated. By detailing the promise and the peril of deep learning, this review aims to pave the way for a more naturalistic affective science.

Training and testing of a gradient boosted machine learning model to predict adverse outcome in patients presenting to emergency departments with suspected covid-19 infection in a middle-income setting

COVID-19 infection rates remain high in South Africa. Clinical prediction models may be helpful for rapid triage, and supporting clinical decision making, for patients with suspected COVID-19 infection. The Western Cape, South Africa, has integrated electronic health care data facilitating large-scale linked routine datasets. The aim of this study was to develop a machine learning model to predict adverse outcome in patients presenting with suspected COVID-19 suitable for use in a middle-income setting. A retrospective cohort study was conducted using linked, routine data, from patients presenting with suspected COVID-19 infection to public-sector emergency departments (EDs) in the Western Cape, South Africa between 27th August 2020 and 31st October 2021. The primary outcome was death or critical care admission at 30 days. An XGBoost machine learning model was trained and internally tested using split-sample validation. External validation was performed in 3 test cohorts: Western Cape patients presenting during the Omicron COVID-19 wave, a UK cohort during the ancestral COVID-19 wave, and a Sudanese cohort during ancestral and Eta waves. A total of 282,051 cases were included in a complete case training dataset. The prevalence of 30-day adverse outcome was 4.0%. The most important features for predicting adverse outcome were the requirement for supplemental oxygen, peripheral oxygen saturations, level of consciousness and age. Internal validation using split-sample test data revealed excellent discrimination (C-statistic 0.91, 95% CI 0.90 to 0.91) and calibration (CITL of 1.05). The model achieved C-statistics of 0.84 (95% CI 0.84 to 0.85), 0.72 (95% CI 0.71 to 0.73), and 0.62, (95% CI 0.59 to 0.65) in the Omicron, UK, and Sudanese test cohorts. Results were materially unchanged in sensitivity analyses examining missing data. An XGBoost machine learning model achieved good discrimination and calibration in prediction of adverse outcome in patients presenting with suspected COVID19 to Western Cape EDs. Performance was reduced in temporal and geographical external validation.

Risk-prediction models for intravenous immunoglobulin resistance in Kawasaki disease: Risk-of-Bias Assessment using PROBAST

BACKGROUND

The prediction model of intravenous immunoglobulin (IVIG) resistance in Kawasaki disease can calculate the probability of IVIG resistance and provide a basis for clinical decision-making. We aim to assess the quality of these models developed in the children with Kawasaki disease.

METHODS

Studies of prediction models for IVIG-resistant Kawasaki disease were identified through searches in the PubMed, Web of Science, and Embase databases. Two investigators independently performed literature screening, data extraction, quality evaluation, and discrepancies were settled by a statistician. The checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS) was used for data extraction, and the prediction models were evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

RESULTS

Seventeen studies meeting the selection criteria were included in the qualitative analysis. The top three predictors were neutrophil measurements (peripheral neutrophil count and neutrophil %), serum albumin level, and C-reactive protein (CRP) level. The reported area under the curve (AUC) values for the developed models ranged from 0.672 (95% confidence interval [CI]: 0.631-0.712) to 0.891 (95% CI: 0.837-0.945); The studies showed a high risk of bias (ROB) for modeling techniques, yielding a high overall ROB.

CONCLUSION

IVIG resistance models for Kawasaki disease showed high ROB. An emphasis on improving their quality can provide high-quality evidence for clinical practice.

IMPACT STATEMENT

This study systematically evaluated the risk of bias (ROB) of existing prediction models for intravenous immunoglobulin (IVIG) resistance in Kawasaki disease to provide guidance for future model development meeting clinical expectations. This is the first study to systematically evaluate the ROB of IVIG resistance in Kawasaki disease by using PROBAST. ROB may reduce model performance in different populations. Future prediction models should account for this problem, and PROBAST can help improve the methodological quality and applicability of prediction model development.

Developing an artificial intelligence-based diagnostic model of headaches from a dataset of clinic patients' records

OBJECTIVE

We developed an artificial intelligence (AI)-based headache diagnosis model using a large questionnaire database from a headache-specializing clinic.

BACKGROUND

Misdiagnosis of headache disorders is a serious issue and AI-based headache diagnosis models are scarce.

METHODS

We developed an AI-based headache diagnosis model and conducted internal validation based on a retrospective investigation of 6058 patients (4240 training dataset for model development and 1818 test dataset for internal validation) diagnosed by a headache specialist. The ground truth was the diagnosis by the headache specialist. The diagnostic performance of the AI model was evaluated.

RESULTS

The dataset included 4829/6058 (79.7%) patients with migraine, 834/6058 (13.8%) with tension-type headache, 78/6058 (1.3%) with trigeminal autonomic cephalalgias, 38/6058 (0.6%) with other primary headache disorders, and 279/6058 (4.6%) with other headaches. The mean (standard deviation) age was 34.7 (14.5) years, and 3986/6058 (65.8%) were female. The model's micro-average accuracy, sensitivity (recall), specificity, precision, and F-values for the test dataset were 93.7%, 84.2%, 84.2%, 96.1%, and 84.2%, respectively. The diagnostic performance for migraine was high, with a sensitivity of 88.8% and c-statistics of 0.89 (95% confidence interval 0.87-0.91).

CONCLUSIONS

Our AI model demonstrated high diagnostic performance for migraine. If secondary headaches can be ruled out, the model can be a powerful tool for diagnosing migraine; however, further data collection and external validation are required to strengthen the performance, ensure the generalizability in other outpatients, and demonstrate its utility in real-world settings.

A Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis analysis to evaluate the quality of reporting of postoperative pancreatic fistula prediction models after pancreatoduodenectomy: A systematic review

BACKGROUND

Postoperative pancreatic fistula is a frequent and potentially lethal complication after pancreatoduodenectomy. Several models have been developed to predict postoperative pancreatic fistula risk. This study was performed to evaluate the quality of reporting of postoperative pancreatic fistula prediction models after pancreatoduodenectomy using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist that provides guidelines on reporting prediction models to enhance transparency and to help in the decision-making regarding the implementation of the appropriate risk models into clinical practice.

METHODS

Studies that described prediction models to predict postoperative pancreatic fistula after pancreatoduodenectomy were searched according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The TRIPOD checklist was used to evaluate the adherence rate. The area under the curve and other performance measures were extracted if reported. A quadrant matrix chart is created to plot the area under the curve against TRIPOD adherence rate to find models with a combination of above-average TRIPOD adherence and area under the curve.

RESULTS

In total, 52 predictive models were included (23 development, 15 external validation, 4 incremental value, and 10 development and external validation). No risk model achieved 100% adherence to the TRIPOD. The mean adherence rate was 65%. Most authors failed to report on missing data and actions to blind assessment of predictors. Thirteen models had an above-average performance for TRIPOD checklist adherence and area under the curve.

CONCLUSION

Although the average TRIPOD adherence rate for postoperative pancreatic fistula models after pancreatoduodenectomy was 65%, higher compared to other published models, it does not meet TRIPOD standards for transparency. This study identified 13 models that performed above average in TRIPOD adherence and area under the curve, which could be the appropriate models to be used in clinical practice.

Evaluating equity in performance of an electronic health record-based 6-month mortality risk model to trigger palliative care consultation: a retrospective model validation analysis

OBJECTIVE

Evaluate predictive performance of an electronic health record (EHR)-based, inpatient 6-month mortality risk model developed to trigger palliative care consultation among patient groups stratified by age, race, ethnicity, insurance and socioeconomic status (SES), which may vary due to social forces (eg, racism) that shape health, healthcare and health data.

DESIGN

Retrospective evaluation of prediction model.

SETTING

Three urban hospitals within a single health system.

PARTICIPANTS

All patients ≥18 years admitted between 1 January and 31 December 2017, excluding observation, obstetric, rehabilitation and hospice (n=58 464 encounters, 41 327 patients).

MAIN OUTCOME MEASURES

General performance metrics (c-statistic, integrated calibration index (ICI), Brier Score) and additional measures relevant to health equity (accuracy, false positive rate (FPR), false negative rate (FNR)).

RESULTS

For black versus non-Hispanic white patients, the model's accuracy was higher (0.051, 95% CI 0.044 to 0.059), FPR lower (-0.060, 95% CI -0.067 to -0.052) and FNR higher (0.049, 95% CI 0.023 to 0.078). A similar pattern was observed among patients who were Hispanic, younger, with Medicaid/missing insurance, or living in low SES zip codes. No consistent differences emerged in c-statistic, ICI or Brier Score. Younger age had the second-largest effect size in the mortality prediction model, and there were large standardised group differences in age (eg, 0.32 for non-Hispanic white versus black patients), suggesting age may contribute to systematic differences in the predicted probabilities between groups.

CONCLUSIONS

An EHR-based mortality risk model was less likely to identify some marginalised patients as potentially benefiting from palliative care, with younger age pinpointed as a possible mechanism. Evaluating predictive performance is a critical preliminary step in addressing algorithmic inequities in healthcare, which must also include evaluating clinical impact, and governance and regulatory structures for oversight, monitoring and accountability.

Advancing personalized prognosis in atypical and anaplastic meningiomas through interpretable machine learning models

PURPOSE

The primary purpose of this study was to utilize machine learning (ML) models to create a web application that can predict survival outcomes for patients diagnosed with atypical and anaplastic meningiomas.

METHODS

In this retrospective cohort study, patients diagnosed with WHO grade II and III meningiomas were selected from the National Cancer Database (NCDB) to analyze survival outcomes at 12, 36, and 60 months. Five machine learning algorithms - TabPFN, TabNet, XGBoost, LightGBM, and Random Forest were employed and optimized using the Optuna library for hyperparameter tuning. The top-performing models were then deployed into our web-based application.

RESULTS

From the NCDB, 12,197 adult patients diagnosed with histologically confirmed WHO grade II and III meningiomas were retrieved. The mean age was 61 (± 20), and 6,847 (56.1%) of these were females. Performance evaluation indicated that the top-performing models for each outcome were the models built with the TabPFN algorithm. The TabPFN models yielded area under the receiver operating characteristic (AUROC) values of 0.805, 0.781, and 0.815 in predicting 12-, 36-, and 60-month mortality, respectively.

CONCLUSION

With the continuous growth of neuro-oncology data, ML algorithms act as key tools in predicting survival outcomes for WHO grade II and III meningioma patients. By incorporating these interpretable models into a web application, we can practically utilize them to improve risk evaluation and prognosis for meningioma patients.