Improving the performance of machine learning algorithms for health outcomes predictions in multicentric cohorts

Development of a machine learning model and a web application for predicting neurological outcome at hospital discharge in spinal cord injury patients

BACKGROUND

Spinal cord injury (SCI) is a devastating condition with profound physical, psychological, and socioeconomic consequences. Despite advances in SCI treatment, accurately predicting functional recovery remains a significant challenge. Conventional prognostic methods often fall short in capturing the complex interplay of factors influencing SCI outcomes. There is an urgent demand for more precise and comprehensive prognostic tools that can guide clinical decision-making and improve patient care in SCI.

PURPOSE

This study aims to develop and validate a machine learning (ML) model for predicting American Spinal Injury Association (ASIA) Impairment Scale (AIS) at discharge in SCI patients. We also aim to convert this model into an open-access web application.

STUDY DESIGN/SETTING

This was a retrospective cohort study enrolling traumatic SCI patients from 1991 to 2015, analyzed in 2023. Data were obtained from the Japan Rehabilitation Database (JARD), a comprehensive nationwide database that includes SCI patients from specialized SCI centers and rehabilitation hospitals across Japan.

PATIENTS SAMPLE

4,108 SCI cases from JARD were reviewed, excluding 405 cases, patients caused by nontraumatic injuries, patients who were graded as AIS E at admission, and patients without data of AIS at discharge, resulting in 3,703 cases being included in the study. Patient demographics and specific SCI injury characteristics at admission were utilized for model training and prediction.

OUTCOME MEASURES

Model performance was evaluated based on R2, accuracy, and the weighted Kappa coefficient. Shapley additive explanations (SHAP) values highlighted significant features influencing the model's output.

METHODS

The primary outcome was AIS at discharge, treated as a continuous variable (0-4) to capture the ordinal nature and clinical significance of potential misclassifications. Data preprocessing included multicollinearity removal, feature selection using the Boruta algorithm, and iterative imputation for missing data. The dataset was split using the hold-out method with a 7:3 ratio resulting in 2,592 cases for training and 1,111 cases for testing the regression model. A best performing model was defined as the highest R2 using PyCaret's automated model comparison. Final predictions of regression model were discretized to the original AIS categories for clinical interpretation.

RESULTS

The Gradient Boosting Regressor (GBR) was identified as the optimal model. The GBR model showed an R² of 0.869, accuracy of 0.814, and weighted Kappa of 0.940. Eleven key variables, including AIS at admission, the day from injury to admission, and the motor score of L3, were identified as significant based on SHAP values. This model was then adapted into a web application via Streamlit.

CONCLUSIONS

We developed a high-accuracy ML model for predicting the AIS at discharge, which effectively captures the ordinal nature of the AIS scale, using 11 key variables. This model demonstrated its performance to provide reliable prognostic information. The model has been integrated into a user-friendly, open-access web application (http://3.138.174.54:8502/). This tool has the potential to aid in resource allocation and enhance treatment for each patient.

Liquid biopsy to identify Barrett's oesophagus, dysplasia and oesophageal adenocarcinoma: the EMERALD multicentre study

BACKGROUND

There is no clinically relevant serological marker for the early detection of oesophageal adenocarcinoma (EAC) and its precursor lesion, Barrett's oesophagus (BE).

OBJECTIVE

To develop and test a blood-based assay for EAC and BE.

DESIGN

Oesophageal MicroRNAs of BaRRett, Adenocarcinoma and Dysplasia (EMERALD) was a large, international, multicentre biomarker cohort study involving 792 patient samples from 4 countries (NCT06381583) to develop and validate a circulating miRNA signature for the early detection of EAC and high-risk BE. Tissue-based miRNA sequencing and microarray datasets (n=134) were used to identify candidate miRNAs of diagnostic potential, followed by validation using 42 pairs of matched cancer and normal tissues. The usefulness of the candidate miRNAs was initially assessed using 108 sera (44 EAC, 34 EAC precursors and 30 non-disease controls). We finally trained a machine learning model (XGBoost+AdaBoost) on RT-qPCR results from circulating miRNAs from a training cohort (n=160) and independently tested it in an external cohort (n=295).

RESULTS

After a strict process of biomarker discovery and selection, we identified six miRNAs that were overexpressed in all sera of patients compared with non-disease controls from three independent cohorts of different nationalities (miR-106b, miR-146a, miR-15a, miR-18a, miR-21 and miR-93). We established a six-miRNA diagnostic signature using the training cohort (area under the receiver operating characteristic curve (AUROC): 97.6%) and tested it in an independent cohort (AUROC: 91.9%). This assay could also identify patients with BE among patients with gastro-oesophageal reflux disease (AUROC: 94.8%, sensitivity: 92.8%, specificity: 85.1%).

CONCLUSION

Using a comprehensive approach integrating unbiased genome-wide biomarker discovery and several independent experimental validations, we have developed and validated a novel blood test that might complement screening options for BE/EAC.

TRIAL REGISTRATION NUMBER

NCT06381583.

Multicenter comparative analysis of local and aggregated data training strategies in COVID-19 outcome prediction with Machine learning

Machine learning (ML) is a promising tool in assisting clinical decision-making for improving diagnosis and prognosis, especially in developing regions. It is often used with large samples, aggregating data from different regions and hospitals. However, it is unclear how this affects predictions in local centers. This study aims to compare data aggregation strategies of several hospitals in Brazil with a local training strategy in each hospital to predict two COVID-19 outcomes: Intensive Care Unit admission (ICU) and mechanical ventilation use (MV). The study included 6,046 patients from 14 hospitals, with local sample sizes ranging from 47 to 1500 patients. Machine learning models were trained using extreme gradient boosting, lightGBM, and catboost for structured data. Seven data aggregation strategies based on hospital geographic regions were compared with local training, and the best strategy was determined by analyzing the area under the ROC curve (AUROC). SHAP (Shapley Additive exPlanations) values were used to assess the contribution of variables to predictions. Additionally, a metafeatures analysis examined how hospital characteristics influence the selection of the best strategy. The study found that the local training strategy was the most effective approach, in the case of ICU outcomes, for 11 of the 14 hospitals (79%), and, in the case of MV, for 10 hospitals (71%). Metafeatures analysis suggested that hospitals with smaller sample sizes generally performed better using an aggregated data strategy compared to local training. Our study brings to light an important concern about the impact of grouping data from different hospitals in predictive machine learning models. These findings contribute to the ongoing debate about the trade-off between increasing sample size and bringing together heterogeneous scenarios.

Efficient clinical decision-making process via AI-based multimodal data fusion: A COVID-19 case study

COVID-19 is an infectious disease that caused a global pandemic in 2020. In the critical moments of this healthcare emergencies, the medical staff needs to take important decisions in a context of limited resources that must be carefully managed. To this end, the computer-aided diagnosis methods are extremely powerful and help them to better recognize the evidences of high-risk patients. This can be done with the support of relevant information extracted from electronic health records, lab tests and imaging studies. In this work, we present a novel fully-automatic efficient method to help the clinical decision-making process in the context of COVID-19 risk estimation, using multimodal data fusion of clinical features and deep features extracted from chest X-ray images. The risk estimation is studied in two of the most relevant and critical encountered scenarios: the risk of hospitalization and mortality. This study shows which are the most important features for each scenario, the ratio of clinical and imaging features present in the top ranking and the performance of the used machine learning models. The results demonstrate a great performance by the classifiers, estimating the risk of hospitalization with an AUC-ROC of 0.8452 ± 0.0133 and the risk of death with an AUC-ROC of 0.8285 ± 0.0210, only using a subset of the original features, and highlight the significant contribution of imaging features to hospitalization risk assessment, while clinical features become more crucial for mortality risk evaluation. Furthermore, multimodal data fusion can outperform the approaches that use one data source. Despite the model's complexity, it requires fewer features, an advantage in scenarios with limited computational resources. This streamlined, fully-automated method shows promising potential to improve the clinical decision-making process and better manage medical resources, not only in the context of COVID-19, but also in other clinical scenarios.

Determination of prognostic markers for COVID-19 disease severity using routine blood tests and machine learning

The need for the identification of risk factors associated to COVID-19 disease severity remains urgent. Patients' care and resource allocation can be potentially different and are defined based on the current classification of disease severity. This classification is based on the analysis of clinical parameters and routine blood tests, which are not standardized across the globe. Some laboratory test alterations have been associated to COVID-19 severity, although these data are conflicting partly due to the different methodologies used across different studies. This study aimed to construct and validate a disease severity prediction model using machine learning (ML). Seventy-two patients admitted to a Brazilian hospital and diagnosed with COVID-19 through RT-PCR and/or ELISA, and with varying degrees of disease severity, were included in the study. Their electronic medical records and the results from daily blood tests were used to develop a ML model to predict disease severity. Using the above data set, a combination of five laboratorial biomarkers was identified as accurate predictors of COVID-19 severe disease with a ROC-AUC of 0.80 ​±​ 0.13. Those biomarkers included prothrombin activity, ferritin, serum iron, ATTP and monocytes. The application of the devised ML model may help rationalize clinical decision and care.

ClotCatcher: a novel natural language model to accurately adjudicate venous thromboembolism from radiology reports

INTRODUCTION

Accurate identification of venous thromboembolism (VTE) is critical to develop replicable epidemiological studies and rigorous predictions models. Traditionally, VTE studies have relied on international classification of diseases (ICD) codes which are inaccurate - leading to misclassification bias. Here, we developed ClotCatcher, a novel deep learning model that uses natural language processing to detect VTE from radiology reports.

METHODS

Radiology reports to detect VTE were obtained from patients admitted to Emory University Hospital (EUH) and Grady Memorial Hospital (GMH). Data augmentation was performed using the Google PEGASUS paraphraser. This data was then used to fine-tune ClotCatcher, a novel deep learning model. ClotCatcher was validated on both the EUH dataset alone and GMH dataset alone.

RESULTS

The dataset contained 1358 studies from EUH and 915 studies from GMH (n = 2273). The dataset contained 1506 ultrasound studies with 528 (35.1%) studies positive for VTE, and 767 CT studies with 91 (11.9%) positive for VTE. When validated on the EUH dataset, ClotCatcher performed best (AUC = 0.980) when trained on both EUH and GMH dataset without paraphrasing. When validated on the GMH dataset, ClotCatcher performed best (AUC = 0.995) when trained on both EUH and GMH dataset with paraphrasing.

CONCLUSION

ClotCatcher, a novel deep learning model with data augmentation rapidly and accurately adjudicated the presence of VTE from radiology reports. Applying ClotCatcher to large databases would allow for rapid and accurate adjudication of incident VTE. This would reduce misclassification bias and form the foundation for future studies to estimate individual risk for patient to develop incident VTE.