Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial

Machine learning applied to the prediction of relapse, hospitalization, and suicide in bipolar disorder using neuroimaging and clinical data: A systematic review

BACKGROUND

Bipolar disorder (BD) is associated with increased morbidity/mortality. Adverse outcome prediction might help with the management of patients with BD.

METHODS

We systematically reviewed the performance of machine learning (ML) studies in predicting adverse outcomes (relapse or recurrence, hospital admission, and suicide-related events) in patients with BD. Demographic, clinical, and neuroimaging-related poor outcome predictors were also reviewed. Three databases (PubMed, Scopus, and Web of Science) were explored from inception to July 2023.

RESULTS

Eighteen studies, accounting for >30,000 patients, were included. Support vector machine, decision trees, random forest, and logistic regression were the most frequently used ML algorithms. ML models' area under the receiver operating characteristic (ROC) curve (AUC), sensitivity, and specificity ranged from 0.71 to 0.98, 72.7-92.8 %, and 59.0-95.2 % for relapse/recurrence prediction (4 studies (3 on relapses and 1 on recurrences). The corresponding values were 0.78-0.88, 21.4-100 %, and 77.0-99.7 % for hospital admissions (3 studies, 21,266 patients), and 0.71-0.99, 44.4-97.9 %, and 38.9-95.0 % for suicide-related events (10 studies, 5558 patients). Also, one study addressed a combination of the interest outcomes. Adverse outcome predictors included early onset BD, BD type I, comorbid psychiatric or substance use disorder, circadian rhythm disruption, hospitalization characteristics, and neuroimaging parameters, including increased dynamic amplitude of low-frequency fluctuation, decreased frontolimbic functional connectivity and aberrant dynamic functional connectivity in corticostriatal circuitry.

CONCLUSIONS

ML models can predict adverse outcomes of BD with relatively acceptable performance measures. Future studies with larger samples and nested cross-validation validation should be conducted to reach more reliable results.

Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Models (CREMLS)

The number of papers presenting machine learning (ML) models that are being submitted to and published in the Journal of Medical Internet Research and other JMIR Publications journals has steadily increased. Editors and peer reviewers involved in the review process for such manuscripts often go through multiple review cycles to enhance the quality and completeness of reporting. The use of reporting guidelines or checklists can help ensure consistency in the quality of submitted (and published) scientific manuscripts and, for example, avoid instances of missing information. In this Editorial, the editors of JMIR Publications journals discuss the general JMIR Publications policy regarding authors' application of reporting guidelines and specifically focus on the reporting of ML studies in JMIR Publications journals, using the Consolidated Reporting of Machine Learning Studies (CREMLS) guidelines, with an example of how authors and other journals could use the CREMLS checklist to ensure transparency and rigor in reporting.