1
|
Ning Y, Li S, Ng YY, Chia MYC, Gan HN, Tiah L, Mao DR, Ng WM, Leong BSH, Doctor N, Ong MEH, Liu N. Variable importance analysis with interpretable machine learning for fair risk prediction. PLOS DIGITAL HEALTH 2024; 3:e0000542. [PMID: 38995879 PMCID: PMC11244764 DOI: 10.1371/journal.pdig.0000542] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Accepted: 06/01/2024] [Indexed: 07/14/2024]
Abstract
Machine learning (ML) methods are increasingly used to assess variable importance, but such black box models lack stability when limited in sample sizes, and do not formally indicate non-important factors. The Shapley variable importance cloud (ShapleyVIC) addresses these limitations by assessing variable importance from an ensemble of regression models, which enhances robustness while maintaining interpretability, and estimates uncertainty of overall importance to formally test its significance. In a clinical study, ShapleyVIC reasonably identified important variables when the random forest and XGBoost failed to, and generally reproduced the findings from smaller subsamples (n = 2500 and 500) when statistical power of the logistic regression became attenuated. Moreover, ShapleyVIC reasonably estimated non-significant importance of race to justify its exclusion from the final prediction model, as opposed to the race-dependent model from the conventional stepwise model building. Hence, ShapleyVIC is robust and interpretable for variable importance assessment, with potential contribution to fairer clinical risk prediction.
Collapse
Affiliation(s)
- Yilin Ning
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
| | - Siqi Li
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
| | - Yih Yng Ng
- Digital and Smart Health Office, Ng Teng Fong Centre for Healthcare Innovation, Singapore, Singapore
- Department of Preventive and Population Medicine, Tan Tock Seng Hospital, Singapore, Singapore
| | | | - Han Nee Gan
- Accident & Emergency, Changi General Hospital, Singapore, Singapore
| | - Ling Tiah
- Accident & Emergency, Changi General Hospital, Singapore, Singapore
| | - Desmond Renhao Mao
- Department of Acute and Emergency Care, Khoo Teck Puat Hospital, Singapore, Singapore
| | - Wei Ming Ng
- Emergency Medicine Department, Ng Teng Fong General Hospital, Singapore, Singapore
| | | | - Nausheen Doctor
- Department of Emergency Medicine, Sengkang General Hospital, Singapore, Singapore
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore
- Health Services Research Centre, Singapore Health Services, Singapore, Singapore
| | - Nan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore
- Institute of Data Science, National University of Singapore, Singapore, Singapore
| |
Collapse
|
2
|
Meng L, Ho P. A systematic review of prediction models on arteriovenous fistula: Risk scores and machine learning approaches. J Vasc Access 2024:11297298241237830. [PMID: 38658814 DOI: 10.1177/11297298241237830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/26/2024] Open
Abstract
OBJECTIVE Failure-to-mature and early stenosis remains the Achille's heel of hemodialysis arteriovenous fistula (AVF) creation. The maturation and patency of an AVF can be influenced by a variety of demographic, comorbidity, and anatomical factors. This study aims to review the prediction models of AVF maturation and patency with various risk scores and machine learning models. DATA SOURCES AND REVIEW METHODS Literature search was performed on PubMed, Scopus, and Embase to identify eligible articles. The quality of the studies was assessed using the Prediction model Risk Of Bias ASsessment (PROBAST) Tool. The performance (discrimination and calibration) of the included studies were extracted. RESULTS Fourteen studies (seven studies used risk score approaches; seven studies used machine learning approaches) were included in the review. Among them, 12 studies were rated as high or unclear "risk of bias." Six studies were rated as high concern or unclear for "applicability." C-statistics (Model discrimination metric) was reported in five studies using risk score approach (0.70-0.886) and three utilized machine learning methods (0.80-0.85). Model calibration was reported in three studies. Failure-to-mature risk score developed by one of the studies has been externally validated in three different patient populations, however the model discrimination degraded significantly (C-statistics: 0.519-0.53). CONCLUSION The performance of existing predictive models for AVF maturation/patency is underreported. They showed satisfactory performance in their own study population. However, there was high risk of bias in methodology used to build some of the models. The reviewed models also lack external validation or had reduced performance in external cohort.
Collapse
Affiliation(s)
- Lingyan Meng
- Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, Singapore
| | - Pei Ho
- Department of Surgery, Yong Loo Lin School of Medicine, National University of Singapore, Singapore
- Department of Cardiac, Thoracic and Vascular Surgery, National University Health System, Singapore
| |
Collapse
|
3
|
Jawad BN, Shaker SM, Altintas I, Eugen-Olsen J, Nehlin JO, Andersen O, Kallemose T. Development and validation of prognostic machine learning models for short- and long-term mortality among acutely admitted patients based on blood tests. Sci Rep 2024; 14:5942. [PMID: 38467752 PMCID: PMC10928126 DOI: 10.1038/s41598-024-56638-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Accepted: 03/08/2024] [Indexed: 03/13/2024] Open
Abstract
Several scores predicting mortality at the emergency department have been developed. However, all with shortcomings either simple and applicable in a clinical setting, with poor performance, or advanced, with high performance, but clinically difficult to implement. This study aimed to explore if machine learning algorithms could predict all-cause short- and long-term mortality based on the routine blood test collected at admission. METHODS We analyzed data from a retrospective cohort study, including patients > 18 years admitted to the Emergency Department (ED) of Copenhagen University Hospital Hvidovre, Denmark between November 2013 and March 2017. The primary outcomes were 3-, 10-, 30-, and 365-day mortality after admission. PyCaret, an automated machine learning library, was used to evaluate the predictive performance of fifteen machine learning algorithms using the area under the receiver operating characteristic curve (AUC). RESULTS Data from 48,841 admissions were analyzed, of these 34,190 (70%) were randomly divided into training data, and 14,651 (30%) were in test data. Eight machine learning algorithms achieved very good to excellent results of AUC on test data in a of range 0.85-0.93. In prediction of short-term mortality, lactate dehydrogenase (LDH), leukocyte counts and differentials, Blood urea nitrogen (BUN) and mean corpuscular hemoglobin concentration (MCHC) were the best predictors, whereas prediction of long-term mortality was favored by age, LDH, soluble urokinase plasminogen activator receptor (suPAR), albumin, and blood urea nitrogen (BUN). CONCLUSION The findings suggest that measures of biomarkers taken from one blood sample during admission to the ED can identify patients at high risk of short-and long-term mortality following emergency admissions.
Collapse
Affiliation(s)
- Baker Nawfal Jawad
- Department of Clinical Research, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark.
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark.
| | | | - Izzet Altintas
- Department of Clinical Research, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
- Emergency Department, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Jesper Eugen-Olsen
- Department of Clinical Research, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
| | - Jan O Nehlin
- Department of Clinical Research, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
| | - Ove Andersen
- Department of Clinical Research, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
- Emergency Department, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Thomas Kallemose
- Department of Clinical Research, Copenhagen University Hospital Amager and Hvidovre, Hvidovre, Denmark
| |
Collapse
|
4
|
Ho CSH, Tan TWK, Khoe HCH, Chan YL, Tay GWN, Tang TB. Using an Interpretable Amino Acid-Based Machine Learning Method to Enhance the Diagnosis of Major Depressive Disorder. J Clin Med 2024; 13:1222. [PMID: 38592058 PMCID: PMC10931723 DOI: 10.3390/jcm13051222] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Revised: 02/13/2024] [Accepted: 02/19/2024] [Indexed: 04/10/2024] Open
Abstract
Background: Major depressive disorder (MDD) is a leading cause of disability worldwide. At present, however, there are no established biomarkers that have been validated for diagnosing and treating MDD. This study sought to assess the diagnostic and predictive potential of the differences in serum amino acid concentration levels between MDD patients and healthy controls (HCs), integrating them into interpretable machine learning models. Methods: In total, 70 MDD patients and 70 HCs matched in age, gender, and ethnicity were recruited for the study. Serum amino acid profiling was conducted by means of chromatography-mass spectrometry. A total of 21 metabolites were analysed, with 17 from a preset amino acid panel and the remaining 4 from a preset kynurenine panel. Logistic regression was applied to differentiate MDD patients from HCs. Results: The best-performing model utilised both feature selection and hyperparameter optimisation and yielded a moderate area under the receiver operating curve (AUC) classification value of 0.76 on the testing data. The top five metabolites identified as potential biomarkers for MDD were 3-hydroxy-kynurenine, valine, kynurenine, glutamic acid, and xanthurenic acid. Conclusions: Our study highlights the potential of using an interpretable machine learning analysis model based on amino acids to aid and increase the diagnostic accuracy of MDD in clinical practice.
Collapse
Affiliation(s)
- Cyrus Su Hui Ho
- Department of Psychological Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 117543, Singapore;
| | - Trevor Wei Kiat Tan
- Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 117543, Singapore;
- Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 117543, Singapore
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore 117583, Singapore
- N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore 117456, Singapore
- Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore 119077, Singapore
| | - Howard Cai Hao Khoe
- Singapore Psychiatry Residency, National Healthcare Group, Singapore 308433, Singapore;
| | - Yee Ling Chan
- Centre for Intelligent Signal and Imaging Research (CISIR), Universiti Teknologi PETRONAS (UTP), Seri Iskandar 32610, Perak, Malaysia; (Y.L.C.); (T.B.T.)
| | - Gabrielle Wann Nii Tay
- Department of Psychological Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 117543, Singapore;
| | - Tong Boon Tang
- Centre for Intelligent Signal and Imaging Research (CISIR), Universiti Teknologi PETRONAS (UTP), Seri Iskandar 32610, Perak, Malaysia; (Y.L.C.); (T.B.T.)
| |
Collapse
|
5
|
Xie F, Ning Y, Liu M, Li S, Saffari SE, Yuan H, Volovici V, Ting DSW, Goldstein BA, Ong MEH, Vaughan R, Chakraborty B, Liu N. A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes. STAR Protoc 2023; 4:102302. [PMID: 37178115 DOI: 10.1016/j.xpro.2023.102302] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 03/13/2023] [Accepted: 04/21/2023] [Indexed: 05/15/2023] Open
Abstract
The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),1 Xie et al. (2022)2, Saffari et al. (2022)3 and the online tutorial https://nliulab.github.io/AutoScore/.
Collapse
Affiliation(s)
- Feng Xie
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Yilin Ning
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Mingxuan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Siqi Li
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Seyed Ehsan Saffari
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Han Yuan
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Victor Volovici
- Department of Neurosurgery, Erasmus MC University Medical Center, 3015 GD Rotterdam, the Netherlands; Department of Public Health, Erasmus MC, 3015 GD Rotterdam, the Netherlands
| | - Daniel Shu Wei Ting
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 168751, Singapore; SingHealth AI Office, Singapore Health Services, Singapore 168582, Singapore
| | - Benjamin Alan Goldstein
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27710, USA
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; Health Services Research Centre, Singapore Health Services, Singapore 169856, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore 169608, Singapore
| | - Roger Vaughan
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Bibhas Chakraborty
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27710, USA; Department of Statistics and Data Science, National University of Singapore, Singapore 117546, Singapore
| | - Nan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; SingHealth AI Office, Singapore Health Services, Singapore 168582, Singapore; Institute of Data Science, National University of Singapore, Singapore 117602, Singapore.
| |
Collapse
|
6
|
Settouti N, Saidi M. Preliminary analysis of explainable machine learning methods for multiple myeloma chemotherapy treatment recognition. EVOLUTIONARY INTELLIGENCE 2023. [DOI: 10.1007/s12065-023-00833-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023]
|
7
|
Benchmarking emergency department prediction models with machine learning and public electronic health records. Sci Data 2022; 9:658. [PMID: 36302776 PMCID: PMC9610299 DOI: 10.1038/s41597-022-01782-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 10/14/2022] [Indexed: 11/26/2022] Open
Abstract
The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision support systems to address these challenges. To date, there is no widely accepted clinical prediction benchmark related to the ED based on large-scale public EHRs. An open-source benchmark data platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. Based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we created a benchmark dataset and proposed three clinical prediction benchmarks. This study provides future researchers with insights, suggestions, and protocols for managing data and developing predictive tools for emergency care.
Collapse
|