1
|
Yearley AG, Goedmakers CMW, Panahi A, Doucette J, Rana A, Ranganathan K, Smith TR. FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval. Artif Intell Med 2023; 143:102607. [PMID: 37673576 DOI: 10.1016/j.artmed.2023.102607] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 05/30/2023] [Accepted: 06/05/2023] [Indexed: 09/08/2023]
Abstract
Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.
Collapse
Affiliation(s)
- Alexander G Yearley
- Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA; Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA.
| | - Caroline M W Goedmakers
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; Department of Neurosurgery, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, Netherlands
| | - Armon Panahi
- The George Washington University School of Medicine and Health Sciences, 2300 I St NW, Washington, DC 20052, USA
| | - Joanne Doucette
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; School of Pharmacy, MCPHS University, 179 Longwood Ave, Boston, MA 02115, USA
| | - Aakanksha Rana
- Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA 02139, USA
| | - Kavitha Ranganathan
- Division of Plastic Surgery, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115, USA
| | - Timothy R Smith
- Harvard Medical School, 25 Shattuck St, Boston, MA 02115, USA; Computational Neuroscience Outcomes Center (CNOC), Department of Neurosurgery, Brigham and Women's Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA
| |
Collapse
|
2
|
Liu J, Wang J, Wu J, Gu S, Yao Y, Li J, Li Y, Ren H, Luo T. Comparison of two computed tomography perfusion post-processing software to assess infarct volume in patients with acute ischemic stroke. Front Neurosci 2023; 17:1151823. [PMID: 37179549 PMCID: PMC10166848 DOI: 10.3389/fnins.2023.1151823] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Accepted: 04/07/2023] [Indexed: 05/15/2023] Open
Abstract
Objectives We used two automated software commonly employed in clinical practice-Olea Sphere (Olea) and Shukun-PerfusionGo (PerfusionGo)-to compare the diagnostic utility and volumetric agreement of computed tomography perfusion (CTP)-predicted final infarct volume (FIV) with true FIV in patients with anterior-circulation acute ischemic stroke (AIS). Methods In all, 122 patients with anterior-circulation AIS who met the inclusion and exclusion criteria were retrospectively enrolled and divided into two groups: intervention group (n = 52) and conservative group (n = 70), according to recanalization of blood vessels and clinical outcome (NIHSS) after different treatments. Patients in both groups underwent one-stop 4D-CT angiography (CTA)/CTP, and the raw CTP data were processed on a workstation using Olea and PerfusionGo post-processing software, to calculate and obtain the ischemic core (IC) and hypoperfusion (IC plus penumbra) volumes, hypoperfusion in the conservative group and IC in the intervention group were used to define the predicted FIV. The ITK-SNAP software was used to manually outline and measure true FIV on the follow-up non-enhanced CT or MRI-DWI images. Intraclass correlation coefficients (ICC), Bland-Altman, and Kappa analysis were used to compare the differences in IC and penumbra volumes calculated by the Olea and PerfusionGo software to investigate the relationship between their predicted FIV and true FIV. Results The IC and penumbra difference between Olea and PerfusionGo within the same group (p < 0.001) was statistically significant. Olea obtained larger IC and smaller penumbra than PerfusionGo. Both software partially overestimated the infarct volume, but Olea significantly overestimated it by a larger percentage. ICC analysis showed that Olea performed better than PerfusionGo (intervention-Olea: ICC 0.633, 95%CI 0.439-0.771; intervention-PerfusionGo: ICC 0.526, 95%CI 0.299-0.696; conservative-Olea: ICC 0.623, 95%CI 0.457-0.747; conservative-PerfusionGo: ICC 0.507, 95%CI 0.312-0.662). Olea and PerfusionGo had the same capacity in accurately diagnosing and classifying patients with infarct volume <70 ml. Conclusion Both software had differences in the evaluation of the IC and penumbra. Olea's predicted FIV was more closely correlated with the true FIV than PerfusionGo's prediction. Accurate assessment of infarction on CTP post-processing software remains challenging. Our results may have important practice implications for the clinical use of perfusion post-processing software.
Collapse
Affiliation(s)
- Jiayang Liu
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Jingjie Wang
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Jiajing Wu
- Department of Radiology, Hospital of PLA Army, Chongqing, China
| | - Sirun Gu
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Yunzhuo Yao
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Jing Li
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Yongmei Li
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Huanhuan Ren
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
- Department of Radiology, Chongqing General Hospital, Chongqing, China
| | - Tianyou Luo
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| |
Collapse
|
3
|
Predictive Value of CT Perfusion in Hemorrhagic Transformation after Acute Ischemic Stroke: A Systematic Review and Meta-Analysis. Brain Sci 2023; 13:brainsci13010156. [PMID: 36672136 PMCID: PMC9856940 DOI: 10.3390/brainsci13010156] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Revised: 01/02/2023] [Accepted: 01/12/2023] [Indexed: 01/18/2023] Open
Abstract
Background: Existing studies indicate that some computed tomography perfusion (CTP) parameters may predict hemorrhagic transformation (HT) after acute ischemic stroke (AIS), but there is an inconsistency in the conclusions alongside a lack of comprehensive comparison. Objective: To comprehensively evaluate the predictive value of CTP parameters in HT after AIS. Data sources: A systematical literature review of existing studies was conducted up to 1st October 2022 in six mainstream databases that included original data on the CTP parameters of HT and non-HT groups or on the diagnostic performance of relative cerebral blood flow (rCBF), relative permeability-surface area product (rPS), or relative cerebral blood volume (rCBV) in patients with AIS that completed CTP within 24 h of onset. Data Synthesis: Eighteen observational studies were included. HT and non-HT groups had statistically significant differences in CBF, CBV, PS, rCBF, rCBV, and rPS (p < 0.05 for all). The hierarchical summary receiver operating characteristic (HSROC) revealed that rCBF (area under the curve (AUC) = 0.9), rPS (AUC = 0.89), and rCBV (AUC = 0.85) had moderate diagnostic performances in predicting HT. The pooled sensitivity and specificity of rCBF were 0.85 (95% CI, 0.75−0.91) and 0.83 (95% CI, 0.63−0.94), respectively. Conclusions: rCBF, rPS, and rCBV had moderate diagnostic performances in predicting HT, and rCBF had the best pooled sensitivity and specificity.
Collapse
|
4
|
Lu Q, Fu J, Lv K, Han Y, Pan Y, Xu Y, Zhang J, Geng D. Agreement of three CT perfusion software packages in patients with acute ischemic stroke: A comparison with RAPID. Eur J Radiol 2022; 156:110500. [PMID: 36099834 DOI: 10.1016/j.ejrad.2022.110500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2022] [Revised: 08/07/2022] [Accepted: 08/22/2022] [Indexed: 11/26/2022]
Abstract
PURPOSE To compare ischemic core volume (ICV) and penumbra volume (PV) measured by MIStar, F-STROKE, and Syngo.via with that measured by RAPID in acute ischemic stroke (AIS), and their concordance in selecting patients for endovascular thrombectomy (EVT). METHODS Computed tomography perfusion (CTP) data were processed with four software packages. Bland-Altman analysis and intraclass correlation coefficient (ICC) were performed to evaluate their agreement in quantifying ICV and PV. Kappa test was conducted to assess consistency in the selection of EVT candidates. The correlation between predicted ICV and segmented final infarct volume (FIV) on follow-up images was investigated. RESULTS A total of 91 patients were retrospectively included. F-STROKE had the best consistency with RAPID (ICV: ICC = 0.97; PV: ICC = 0.84) and Syngo.via had the worst consistency (ICV: ICC = 0.77; PV: ICC = 0.66). F-STROKE had the narrowest limits of agreements both in ICV (-27.02, 24.40 mL) and PV (-85.59, 101.80 mL). When selecting EVT candidates, MIStar (kappa = 0.71-0.88) and F-STROKE (kappa = 0.84-0.90) had good to excellent consistency with RAPID, while Syngo.via had poor consistency (kappa = 0.20-0.41). ICV predicted by MIStar was correlated strongest with FIV (r = 0.77). CONCLUSIONS F-STROKE is most consistent with RAPID in quantitative ICV and PV. F-STROKE and MIStar exhibit similar EVT candidate selection to RAPID. Syngo.via, for its part, seems to have overestimated ICV and underestimated PV, leading to an overly restrictive selection of EVT candidates.
Collapse
Affiliation(s)
- Qingqing Lu
- Department of Radiology, Huashan Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200040, China; Department of Radiology, Ningbo First Hospital, Ningbo 315000, China
| | - Junyan Fu
- Department of Radiology, Huashan Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200040, China
| | - Kun Lv
- Department of Radiology, Huashan Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200040, China
| | - Yan Han
- Department of Radiology, Huashan Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200040, China
| | - Yuning Pan
- Department of Radiology, Ningbo First Hospital, Ningbo 315000, China
| | - Yiren Xu
- Department of Radiology, Ningbo First Hospital, Ningbo 315000, China
| | - Jun Zhang
- Department of Radiology, Huashan Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200040, China; National Center for Neurological Disorders, Shanghai 200040, China.
| | - Daoying Geng
- Department of Radiology, Huashan Hospital, State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200040, China; Center for Shanghai Intelligent Imaging for Critical Brain Diseases Engineering and Technology Reasearch, Huashan Hospital, Fudan Universtiy, Shanghai 200040, China.
| |
Collapse
|
5
|
Zhang Y, Song S, Li Z, Huang B, Geng Y, Zhang L. The Application of Software "Rapid Processing of Perfusion and Diffusion" in Acute Ischemic Stroke. Brain Sci 2022; 12:1451. [PMID: 36358379 PMCID: PMC9688896 DOI: 10.3390/brainsci12111451] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2022] [Revised: 10/20/2022] [Accepted: 10/20/2022] [Indexed: 08/30/2023] Open
Abstract
In the event of an acute ischemic stroke, saving the penumbra is the most important aspect of early treatment. The rapid and accurate identification of ischemic penumbra plays a key role in its comprehensive treatment. At present, the identification method and evaluation standard of ischemic penumbra have not been unified. Numerous pieces of software identifying ischemic penumbra have been developed, such as rapid processing of perfusion and diffusion (RAPID), Sphere, Vitrea, and computed tomography perfusion+ (CTP+). The RAPID software, analyzing and integrating multi-mode image data (mainly based on perfusion weighted imaging (PWI) or computed tomography perfusion (CTP) images, shows good performance in identifying ischemic penumbra and has been utilized for the assessment of ischemic penumbra in many ischemic stroke clinical studies, achieving good outcomes and promoting the transition from "time window" to "tissue window" in the treatment of early stage AIS. To obtain a comprehensive understanding of the RAPID software and its accuracy in evaluating ischemic penumbra, this paper reviews the background and development of the RAPID software, summarizes the published acute cerebral infarction trials using the RAPID software, generalizes the threshold parameters in different time windows, and further discusses its application and limitations.
Collapse
Affiliation(s)
- Yudi Zhang
- Department of Neurology, The Second Hospital of Hebei Medical University, Shijiazhuang 050000, China
| | - Shuang Song
- Department of Neurology, The Second Hospital of Hebei Medical University, Shijiazhuang 050000, China
| | - Zhenzhong Li
- Department of Neurology, The Second Hospital of Hebei Medical University, Shijiazhuang 050000, China
| | - Boyuan Huang
- Department of Radiology, The Second Hospital of Hebei Medical University, Shijiazhuang 050000, China
| | - Yanlu Geng
- Department of Neurology, The Second Hospital of Hebei Medical University, Shijiazhuang 050000, China
- Department of Neurology, Central Hospital of Qinghe County, Xingtai 054800, China
| | - Lihong Zhang
- Department of Neurology, The Second Hospital of Hebei Medical University, Shijiazhuang 050000, China
| |
Collapse
|
6
|
Zeng D, Zeng C, Zeng Z, Li S, Deng Z, Chen S, Bian Z, Ma J. Basis and current state of computed tomography perfusion imaging: a review. Phys Med Biol 2022; 67. [PMID: 35926503 DOI: 10.1088/1361-6560/ac8717] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2021] [Accepted: 08/04/2022] [Indexed: 12/30/2022]
Abstract
Computed tomography perfusion (CTP) is a functional imaging that allows for providing capillary-level hemodynamics information of the desired tissue in clinics. In this paper, we aim to offer insight into CTP imaging which covers the basics and current state of CTP imaging, then summarize the technical applications in the CTP imaging as well as the future technological potential. At first, we focus on the fundamentals of CTP imaging including systematically summarized CTP image acquisition and hemodynamic parameter map estimation techniques. A short assessment is presented to outline the clinical applications with CTP imaging, and then a review of radiation dose effect of the CTP imaging on the different applications is presented. We present a categorized methodology review on known and potential solvable challenges of radiation dose reduction in CTP imaging. To evaluate the quality of CTP images, we list various standardized performance metrics. Moreover, we present a review on the determination of infarct and penumbra. Finally, we reveal the popularity and future trend of CTP imaging.
Collapse
Affiliation(s)
- Dong Zeng
- School of Biomedical Engineering, Southern Medical University, Guangdong 510515, China; and Guangzhou Key Laboratory of Medical Radiation Imaging and Detection Technology, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Cuidie Zeng
- School of Biomedical Engineering, Southern Medical University, Guangdong 510515, China; and Guangzhou Key Laboratory of Medical Radiation Imaging and Detection Technology, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Zhixiong Zeng
- School of Biomedical Engineering, Southern Medical University, Guangdong 510515, China; and Guangzhou Key Laboratory of Medical Radiation Imaging and Detection Technology, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Sui Li
- School of Biomedical Engineering, Southern Medical University, Guangdong 510515, China; and Guangzhou Key Laboratory of Medical Radiation Imaging and Detection Technology, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Zhen Deng
- Department of Neurology, Nanfang Hospital, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Sijin Chen
- Department of Medical Imaging Center, Nanfang Hospital, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Zhaoying Bian
- School of Biomedical Engineering, Southern Medical University, Guangdong 510515, China; and Guangzhou Key Laboratory of Medical Radiation Imaging and Detection Technology, Southern Medical University, Guangdong 510515, People's Republic of China
| | - Jianhua Ma
- School of Biomedical Engineering, Southern Medical University, Guangdong 510515, China; and Guangzhou Key Laboratory of Medical Radiation Imaging and Detection Technology, Southern Medical University, Guangdong 510515, People's Republic of China
| |
Collapse
|
7
|
Al-Kawaz M, Primiani C, Urrutia V, Hui F. Impact of RapidAI mobile application on treatment times in patients with large vessel occlusion. J Neurointerv Surg 2021; 14:233-236. [PMID: 33795483 DOI: 10.1136/neurintsurg-2021-017365] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Revised: 03/10/2021] [Accepted: 03/14/2021] [Indexed: 02/04/2023]
Abstract
BACKGROUND Current efforts to reduce door to groin puncture time (DGPT) aim to optimize clinical outcomes in stroke patients with large vessel occlusions (LVOs). The RapidAI mobile application (Rapid Mobile App) provides quick access to perfusion and vessel imaging in patients with LVOs. We hypothesize that utilization of RapidAI mobile application can significantly reduce treatment times in stroke care by accelerating the process of mobilizing stroke clinicians and interventionalists. METHODS We analyzed patients presenting with LVOs between June 2019 and October 2020. Thirty-one patients were treated between June 2019 and March 2020 (pre-app group). Thirty-three patients presented between March 2020 and October 2020 (post-app group). Mann-Whitney U test and Kruskal-Wallis tests were used to examine variables that are not normally distributed. In a secondary analysis we analyzed interhospital time metrics between primary stroke centers and our comprehensive stroke center. RESULTS Baseline demographic and vascular risk factors were similar in both groups. Use of Rapid Mobile App resulted in 33 min reduction in DGPT (P=0.02), 35 min reduction in door to first pass time (P=0.02), and 37 min reduction in door to recanalization time (P=0.02) in univariate analyses when compared with patients treated pre-app. In a multiple linear regression model, utilization of Rapid Mobile App significantly predicted shorter DGPT (P=0.002). In an adjusted model, National Institutes of Health Stroke Scale (NIHSS) 24 hours after procedure and at discharge were significantly lower in the post-app group (P=0.03). Time of transfer between primary and comprehensive stroke center was comparable in both groups (P=0.26). CONCLUSION In patients with LVOs, the implementation of the RapidAI mobile application was independently associated with reductions in intrahospital treatment times.
Collapse
Affiliation(s)
- Mais Al-Kawaz
- Neurosciences Critical Care Division, Department of Neurology, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Christopher Primiani
- Department of Neurology, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Victor Urrutia
- Department of Neurology, Johns Hopkins University School of Medicine, Baltimore, Maryland, USA
| | - Ferdinand Hui
- Department of Radiology and Radiological Sciences, Johns Hopkins Hospital, Baltimore, Maryland, USA
| |
Collapse
|