1
|
Yearley AG, Goedmakers CMW, Panahi A, Doucette J, Rana A, Ranganathan K, Smith TR. FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval. Artif Intell Med 2023; 143:102607. [PMID: 37673576 DOI: 10.1016/j.artmed.2023.102607] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 05/30/2023] [Accepted: 06/05/2023] [Indexed: 09/08/2023]
Abstract
Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.
Collapse
Affiliation(s)
- Alexander G Yearley
- Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA; Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA.
| | - Caroline M W Goedmakers
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; Department of Neurosurgery, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, Netherlands
| | - Armon Panahi
- The George Washington University School of Medicine and Health Sciences, 2300 I St NW, Washington, DC 20052, USA
| | - Joanne Doucette
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; School of Pharmacy, MCPHS University, 179 Longwood Ave, Boston, MA 02115, USA
| | - Aakanksha Rana
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA
| | - Kavitha Ranganathan
- Division of Plastic Surgery, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115, USA
| | - Timothy R Smith
- Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA; Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA
| |
Collapse
|
2
|
Fadda G, Cardenas de la Parra A, O'Mahony J, Waters P, Yeh EA, Bar-Or A, Marrie RA, Narayanan S, Arnold DL, Collins DL, Banwell B. Deviation From Normative Whole Brain and Deep Gray Matter Growth in Children With MOGAD, MS, and Monophasic Seronegative Demyelination. Neurology 2023; 101:e425-e437. [PMID: 37258297 PMCID: PMC10435061 DOI: 10.1212/wnl.0000000000207429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2022] [Accepted: 04/04/2023] [Indexed: 06/02/2023] Open
Abstract
BACKGROUND AND OBJECTIVES Pediatric-acquired demyelination of the CNS associated with antibodies directed against myelin oligodendrocyte glycoprotein (MOG; MOG antibody-associated disease [MOGAD]) occurs as a monophasic or relapsing disease and with variable but often extensive T2 lesions in the brain. The impact of MOGAD on brain growth during maturation is unknown. We quantified the effect of pediatric MOGAD on brain growth trajectories and compared this with the growth trajectories of age-matched and sex-matched healthy children and children with multiple sclerosis (MS, a chronic relapsing disease known to lead to failure of normal brain growth and to loss of brain volume) and monophasic seronegative demyelination. METHODS We included children enrolled at incident attack in the prospective longitudinal Canadian Pediatric Demyelinating Disease Study who were recruited at the 3 largest enrollment sites, underwent research brain MRI scans, and were tested for serum MOG-IgG. Children seropositive for MOG-IgG were diagnosed with MOGAD. MS was diagnosed per the 2017 McDonald criteria. Monophasic seronegative demyelination was confirmed in children with no clinical or MRI evidence of recurrent demyelination and negative results for MOG-IgG and aquaporin-4-IgG. Whole and regional brain volumes were computed through symmetric nonlinear registration to templates. We computed age-normalized and sex-normalized z scores for brain volume using a normative dataset of 813 brain MRI scans obtained from typically developing children and used mixed-effect models to assess potential deviation from brain growth trajectories. RESULTS We assessed brain volumes of 46 children with MOGAD, 26 with MS, and 51 with monophasic seronegative demyelinating syndrome. Children with MOGAD exhibited delayed (p < 0.001) age-expected and sex-expected growth of thalamus, caudate, and globus pallidus, normalized for the whole brain volume. Divergence from expected growth was particularly pronounced in the first year postonset and was detected even in children with monophasic MOGAD. Thalamic volume abnormalities were less pronounced in children with MOGAD compared with those in children with MS. DISCUSSION The onset of MOGAD during childhood adversely affects the expected trajectory of growth of deep gray matter structures, with accelerated changes in the months after an acute attack. Further studies are required to better determine the relative impact of monophasic vs relapsing MOGAD and whether relapsing MOGAD with attacks isolated to the optic nerves or spinal cord affects brain volume over time.
Collapse
Affiliation(s)
- Giulia Fadda
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Alonso Cardenas de la Parra
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Julia O'Mahony
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Patrick Waters
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - E Ann Yeh
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Amit Bar-Or
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Ruth Ann Marrie
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Sridar Narayanan
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Douglas L Arnold
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - D Louis Collins
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania
| | - Brenda Banwell
- From the Department of Medicine (G.F), University of Ottawa, Ottawa Hospital Research Institute; Montreal Neurological Institute (A.C.P., S.N., D.L.A., D.L.C.), McGill University, Quebec; Department of Community Health Sciences (J.O.M., R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; Nuffield Department of Clinical Neurosciences (P.W.), John Radcliffe Hospital, University of Oxford, United Kingdom; Department of Pediatrics (E.A.Y.), University of Toronto, Ontario, Canada; Center for Neuroinflammation and Neurotherapeutics (A.B.-O.), and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia; Department of Internal Medicine (R.A.M.), Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Canada; and Division of Child Neurology (B.B.), Department of Neurology, The Children's Hospital of Philadelphia, Perelman School of Medicine, University of Pennsylvania.
| |
Collapse
|
3
|
Ross DE, Seabaugh J, Seabaugh JM, Barcelona J, Seabaugh D, Wright K, Norwind L, King Z, Graham TJ, Baker J, Lewis T. Updated Review of the Evidence Supporting the Medical and Legal Use of NeuroQuant ® and NeuroGage ® in Patients With Traumatic Brain Injury. Front Hum Neurosci 2022; 16:715807. [PMID: 35463926 PMCID: PMC9027332 DOI: 10.3389/fnhum.2022.715807] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2021] [Accepted: 03/03/2022] [Indexed: 02/05/2023] Open
Abstract
Over 40 years of research have shown that traumatic brain injury affects brain volume. However, technical and practical limitations made it difficult to detect brain volume abnormalities in patients suffering from chronic effects of mild or moderate traumatic brain injury. This situation improved in 2006 with the FDA clearance of NeuroQuant®, a commercially available, computer-automated software program for measuring MRI brain volume in human subjects. More recent strides were made with the introduction of NeuroGage®, commercially available software that is based on NeuroQuant® and extends its utility in several ways. Studies using these and similar methods have found that most patients with chronic mild or moderate traumatic brain injury have brain volume abnormalities, and several of these studies found-surprisingly-more abnormal enlargement than atrophy. More generally, 102 peer-reviewed studies have supported the reliability and validity of NeuroQuant® and NeuroGage®. Furthermore, this updated version of a previous review addresses whether NeuroQuant® and NeuroGage® meet the Daubert standard for admissibility in court. It concludes that NeuroQuant® and NeuroGage® meet the Daubert standard based on their reliability, validity, and objectivity. Due to the improvements in technology over the years, these brain volumetric techniques are practical and readily available for clinical or forensic use, and thus they are important tools for detecting signs of brain injury.
Collapse
Affiliation(s)
- David E. Ross
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
- Department of Psychiatry, Virginia Commonwealth University, Richmond, VA, United States
| | - John Seabaugh
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
- Department of Radiology, St. Mary’s Hospital School of Medical Imaging, Richmond, VA, United States
| | - Jan M. Seabaugh
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
| | - Justis Barcelona
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
| | - Daniel Seabaugh
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
| | - Katherine Wright
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
- Department of Psychiatry, Virginia Commonwealth University, Richmond, VA, United States
| | - Lee Norwind
- Karp, Wigodsky, Norwind, Kudel & Gold, P.A., Rockville, MD, United States
| | - Zachary King
- Karp, Wigodsky, Norwind, Kudel & Gold, P.A., Rockville, MD, United States
| | | | - Joseph Baker
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
- Department of Neuroscience, Christopher Newport University, Newport News, VA, United States
| | - Tanner Lewis
- Virginia Institute of Neuropsychiatry, Midlothian, VA, United States
- NeuroGage LLC, Midlothian, VA, United States
- Department of Undergraduate Studies, University of Virginia, Charlottesville, VA, United States
| |
Collapse
|