1
|
Thirunavukarasu AJ, Elangovan K, Gutierrez L, Hassan R, Li Y, Tan TF, Cheng H, Teo ZL, Lim G, Ting DSW. Clinical performance of automated machine learning: A systematic review. ANNALS OF THE ACADEMY OF MEDICINE, SINGAPORE 2024; 53:187-207. [PMID: 38920245 DOI: 10.47102/annals-acadmedsg.2023113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/27/2024]
Abstract
Introduction Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other. Method This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and if required, arbitration by a third researcher. Results There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35-1.00, F1-score 0.16-0.99, area under the precision-recall curve (AUPRC) 0.51-1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27. Conclusion A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.
Collapse
Affiliation(s)
- Arun James Thirunavukarasu
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
- University of Cambridge School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom
| | - Kabilan Elangovan
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
| | - Laura Gutierrez
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
| | - Refaat Hassan
- University of Cambridge School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom
| | - Yong Li
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
- Duke-NUS Medical School, National University of Singapore, Singapore
| | - Ting Fang Tan
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
| | - Haoran Cheng
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
- Duke-NUS Medical School, National University of Singapore, Singapore
- Rollins School of Public Health, Emory University, Atlanta, Georgia, USA
| | | | - Gilbert Lim
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
| | - Daniel Shu Wei Ting
- Artificial Intelligence and Digital Innovation Research Group, Singapore Eye Research Institute, Singapore
- Duke-NUS Medical School, National University of Singapore, Singapore
- Singapore National Eye Centre, Singapore
| |
Collapse
|
2
|
Benjamin MM, Rabbat MG. Artificial Intelligence in Transcatheter Aortic Valve Replacement: Its Current Role and Ongoing Challenges. Diagnostics (Basel) 2024; 14:261. [PMID: 38337777 PMCID: PMC10855497 DOI: 10.3390/diagnostics14030261] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2023] [Revised: 01/18/2024] [Accepted: 01/20/2024] [Indexed: 02/12/2024] Open
Abstract
Transcatheter aortic valve replacement (TAVR) has emerged as a viable alternative to surgical aortic valve replacement, as accumulating clinical evidence has demonstrated its safety and efficacy. TAVR indications have expanded beyond high-risk or inoperable patients to include intermediate and low-risk patients with severe aortic stenosis. Artificial intelligence (AI) is revolutionizing the field of cardiology, aiding in the interpretation of medical imaging and developing risk models for at-risk individuals and those with cardiac disease. This article explores the growing role of AI in TAVR procedures and assesses its potential impact, with particular focus on its ability to improve patient selection, procedural planning, post-implantation monitoring and contribute to optimized patient outcomes. In addition, current challenges and future directions in AI implementation are highlighted.
Collapse
Affiliation(s)
- Mina M. Benjamin
- Division of Cardiovascular Medicine, SSM—Saint Louis University Hospital, Saint Louis University, Saint Louis, MO 63104, USA
| | - Mark G. Rabbat
- Department of Cardiovascular Medicine, Loyola University Medical Center, Maywood, IL 60153, USA;
- Department of Cardiology, Edward Hines Jr. VA Hospital, Hines, IL 60141, USA
| |
Collapse
|
3
|
Krajnc D, Spielvogel CP, Grahovac M, Ecsedi B, Rasul S, Poetsch N, Traub-Weidinger T, Haug AR, Ritter Z, Alizadeh H, Hacker M, Beyer T, Papp L. Automated data preparation for in vivo tumor characterization with machine learning. Front Oncol 2022; 12:1017911. [PMID: 36303841 PMCID: PMC9595446 DOI: 10.3389/fonc.2022.1017911] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2022] [Accepted: 09/23/2022] [Indexed: 11/23/2022] Open
Abstract
Background This study proposes machine learning-driven data preparation (MLDP) for optimal data preparation (DP) prior to building prediction models for cancer cohorts. Methods A collection of well-established DP methods were incorporated for building the DP pipelines for various clinical cohorts prior to machine learning. Evolutionary algorithm principles combined with hyperparameter optimization were employed to iteratively select the best fitting subset of data preparation algorithms for the given dataset. The proposed method was validated for glioma and prostate single center cohorts by 100-fold Monte Carlo (MC) cross-validation scheme with 80-20% training-validation split ratio. In addition, a dual-center diffuse large B-cell lymphoma (DLBCL) cohort was utilized with Center 1 as training and Center 2 as independent validation datasets to predict cohort-specific clinical endpoints. Five machine learning (ML) classifiers were employed for building prediction models across all analyzed cohorts. Predictive performance was estimated by confusion matrix analytics over the validation sets of each cohort. The performance of each model with and without MLDP, as well as with manually-defined DP were compared in each of the four cohorts. Results Sixteen of twenty established predictive models demonstrated area under the receiver operator characteristics curve (AUC) performance increase utilizing the MLDP. The MLDP resulted in the highest performance increase for random forest (RF) (+0.16 AUC) and support vector machine (SVM) (+0.13 AUC) model schemes for predicting 36-months survival in the glioma cohort. Single center cohorts resulted in complex (6-7 DP steps) DP pipelines, with a high occurrence of outlier detection, feature selection and synthetic majority oversampling technique (SMOTE). In contrast, the optimal DP pipeline for the dual-center DLBCL cohort only included outlier detection and SMOTE DP steps. Conclusions This study demonstrates that data preparation prior to ML prediction model building in cancer cohorts shall be ML-driven itself, yielding optimal prediction models in both single and multi-centric settings.
Collapse
Affiliation(s)
- Denis Krajnc
- QIMP Team, Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| | - Clemens P. Spielvogel
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
- Christian Doppler Laboratory for Applied Metabolomics, Medical University of Vienna, Vienna, Austria
| | - Marko Grahovac
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Boglarka Ecsedi
- QIMP Team, Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| | - Sazan Rasul
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Nina Poetsch
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Tatjana Traub-Weidinger
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Alexander R. Haug
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
- Christian Doppler Laboratory for Applied Metabolomics, Medical University of Vienna, Vienna, Austria
| | - Zsombor Ritter
- Department of Medical Imaging, University of Pécs, Medical School, Pécs, Hungary
| | - Hussain Alizadeh
- 1st Department of Internal Medicine, University of Pécs, Medical School, Pécs, Hungary
| | - Marcus Hacker
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Thomas Beyer
- QIMP Team, Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
- *Correspondence: Thomas Beyer,
| | - Laszlo Papp
- QIMP Team, Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
- Applied Quantum Computing group, Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| |
Collapse
|