1
|
Yearley AG, Goedmakers CMW, Panahi A, Doucette J, Rana A, Ranganathan K, Smith TR. FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval. Artif Intell Med 2023; 143:102607. [PMID: 37673576 DOI: 10.1016/j.artmed.2023.102607] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 05/30/2023] [Accepted: 06/05/2023] [Indexed: 09/08/2023]
Abstract
Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.
Collapse
Affiliation(s)
- Alexander G Yearley
- Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA; Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA.
| | - Caroline M W Goedmakers
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; Department of Neurosurgery, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, Netherlands
| | - Armon Panahi
- The George Washington University School of Medicine and Health Sciences, 2300 I St NW, Washington, DC 20052, USA
| | - Joanne Doucette
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; School of Pharmacy, MCPHS University, 179 Longwood Ave, Boston, MA 02115, USA
| | - Aakanksha Rana
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA
| | - Kavitha Ranganathan
- Division of Plastic Surgery, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115, USA
| | - Timothy R Smith
- Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA; Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA
| |
Collapse
|
2
|
Park HJ, Lee JY, Yang JJ, Kim HJ, Kim YS, Kim JY, Choi YY. Prediction of Amyloid β-Positivity with both MRI Parameters and Cognitive Function Using Machine Learning. JOURNAL OF THE KOREAN SOCIETY OF RADIOLOGY 2023; 84:638-652. [PMID: 37325007 PMCID: PMC10265247 DOI: 10.3348/jksr.2022.0084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 09/05/2022] [Accepted: 10/02/2022] [Indexed: 06/17/2023]
Abstract
Purpose To investigate the MRI markers for the prediction of amyloid β (Aβ)-positivity in mild cognitive impairment (MCI) and Alzheimer's disease (AD), and to evaluate the differences in MRI markers between Aβ-positive (Aβ [+]) and -negative groups using the machine learning (ML) method. Materials and Methods This study included 139 patients with MCI and AD who underwent amyloid PET-CT and brain MRI. Patients were divided into Aβ (+) (n = 84) and Aβ-negative (n = 55) groups. Visual analysis was performed with the Fazekas scale of white matter hyperintensity (WMH) and cerebral microbleeds (CMB) scores. The WMH volume and regional brain volume were quantitatively measured. The multivariable logistic regression and ML using support vector machine, and logistic regression were used to identify the best MRI predictors of Aβ-positivity. Results The Fazekas scale of WMH (p = 0.02) and CMB scores (p = 0.04) were higher in Aβ (+). The volumes of hippocampus, entorhinal cortex, and precuneus were smaller in Aβ (+) (p < 0.05). The third ventricle volume was larger in Aβ (+) (p = 0.002). The logistic regression of ML showed a good accuracy (81.1%) with mini-mental state examination (MMSE) and regional brain volumes. Conclusion The application of ML using the MMSE, third ventricle, and hippocampal volume is helpful in predicting Aβ-positivity with a good accuracy.
Collapse
|
3
|
Berson ER, Aboian MS, Malhotra A, Payabvash S. Artificial Intelligence for Neuroimaging and Musculoskeletal Radiology: Overview of Current Commercial Algorithms. Semin Roentgenol 2023; 58:178-183. [PMID: 37087138 PMCID: PMC10122717 DOI: 10.1053/j.ro.2023.03.002] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2022] [Revised: 03/05/2023] [Accepted: 03/08/2023] [Indexed: 04/03/2023]
Abstract
There is a rapidly increasing number of artificial intelligence (AI) products cleared by the Food and Drug Administration (FDA) for quantification, identification, and even diagnosis in clinical radiology. This review article aims to summarize the landscape of current commercial software products in neuroimaging and musculoskeletal radiology. We will discuss key applications, provide an overview of current FDA cleared products, and summarize relevant peer reviewed publications of these products when available.
Collapse
Affiliation(s)
- Elisa R Berson
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT
| | - Mariam S Aboian
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT
| | - Ajay Malhotra
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT
| | - Seyedmehdi Payabvash
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT.
| |
Collapse
|
4
|
Lorking N, Murray AD, O'Brien JT. The use of positron emission tomography/magnetic resonance imaging in dementia: A literature review. Int J Geriatr Psychiatry 2021; 36:1501-1513. [PMID: 34490651 DOI: 10.1002/gps.5586] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/15/2021] [Revised: 04/22/2021] [Accepted: 05/17/2021] [Indexed: 11/08/2022]
Abstract
OBJECTIVES Positron emission tomography-magnetic resonance imaging (PET/MRI) is an emerging hybrid imaging system in clinical nuclear medicine. Research demonstrates a comparative utility to current unimodal and hybrid methods, including PET-computed tomography (PET/CT), in several medical subspecialities such as neuroimaging. The aim of this review is to critically evaluate the literature from 2016 to 2021 using PET/MRI for the investigation of patients with mild cognitive impairment or dementia, and discuss the evidence base for widening its application into clinical practice. METHODS A comprehensive literature search using the PubMed database was conducted to retrieve studies using PET/MRI in relation to the topics of mild cognitive impairment, dementia, or Alzheimer's disease between January 2016 and January 2021. This search strategy enabled studies on all dementia types to be included in the analysis. Studies were required to have a minimum of 10 human subjects and incorporate simultaneous PET/MRI. RESULTS A total of 116 papers were retrieved, with 39 papers included in the final selection. These were broadly categorised into reviews (12), technical/methodological papers (11) and new data studies (16). For the current review, discussion focused on findings from the new data studies. CONCLUSIONS PET/MRI offers additional insight into the underlying anatomical, metabolic and functional changes associated with dementia when compared with unimodal methods and PET/CT, particularly relating to brain regions including the hippocampus and default mode network. Furthermore, the improved diagnostic utility of PET/MRI, as reported by radiologists, offers improved classification of dementia patients, with important implications for clinical management.
Collapse
Affiliation(s)
- Nicole Lorking
- School of Medicine, University of Aberdeen, Scotland, UK
| | | | - John T O'Brien
- Department of Psychiatry, University of Cambridge, Cambridge, UK
| |
Collapse
|
5
|
Pemberton HG, Zaki LAM, Goodkin O, Das RK, Steketee RME, Barkhof F, Vernooij MW. Technical and clinical validation of commercial automated volumetric MRI tools for dementia diagnosis-a systematic review. Neuroradiology 2021; 63:1773-1789. [PMID: 34476511 PMCID: PMC8528755 DOI: 10.1007/s00234-021-02746-3] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2021] [Accepted: 06/02/2021] [Indexed: 12/22/2022]
Abstract
Developments in neuroradiological MRI analysis offer promise in enhancing objectivity and consistency in dementia diagnosis through the use of quantitative volumetric reporting tools (QReports). Translation into clinical settings should follow a structured framework of development, including technical and clinical validation steps. However, published technical and clinical validation of the available commercial/proprietary tools is not always easy to find and pathways for successful integration into the clinical workflow are varied. The quantitative neuroradiology initiative (QNI) framework highlights six necessary steps for the development, validation and integration of quantitative tools in the clinic. In this paper, we reviewed the published evidence regarding regulatory-approved QReports for use in the memory clinic and to what extent this evidence fulfils the steps of the QNI framework. We summarize unbiased technical details of available products in order to increase the transparency of evidence and present the range of reporting tools on the market. Our intention is to assist neuroradiologists in making informed decisions regarding the adoption of these methods in the clinic. For the 17 products identified, 11 companies have published some form of technical validation on their methods, but only 4 have published clinical validation of their QReports in a dementia population. Upon systematically reviewing the published evidence for regulatory-approved QReports in dementia, we concluded that there is a significant evidence gap in the literature regarding clinical validation, workflow integration and in-use evaluation of these tools in dementia MRI diagnosis.
Collapse
Affiliation(s)
- Hugh G Pemberton
- Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, London, UK.
- UCL Queen Square Institute of Neurology, University College London, London, UK.
- Dementia Research Centre, UCL Queen Square Institute of Neurology, University College London, London, UK.
| | - Lara A M Zaki
- Department of Radiology and Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands
| | - Olivia Goodkin
- Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, London, UK
- UCL Queen Square Institute of Neurology, University College London, London, UK
| | - Ravi K Das
- Clinical, Educational and Health Psychology, University College London, London, UK
| | - Rebecca M E Steketee
- Department of Radiology and Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands
| | - Frederik Barkhof
- Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, London, UK
- UCL Queen Square Institute of Neurology, University College London, London, UK
- Radiology & Nuclear Medicine, VU University Medical Center, Amsterdam, The Netherlands
| | - Meike W Vernooij
- Department of Radiology and Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands
- Department of Epidemiology, Erasmus MC University Medical Center, Rotterdam, The Netherlands
| |
Collapse
|