6
|
Rudie JD, Duda J, Duong MT, Chen PH, Xie L, Kurtz R, Ware JB, Choi J, Mattay RR, Botzolakis EJ, Gee JC, Bryan RN, Cook TS, Mohan S, Nasrallah IM, Rauschecker AM. Brain MRI Deep Learning and Bayesian Inference System Augments Radiology Resident Performance. J Digit Imaging 2021; 34:1049-1058. [PMID: 34131794 PMCID: PMC8455800 DOI: 10.1007/s10278-021-00470-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2020] [Revised: 04/28/2021] [Accepted: 05/25/2021] [Indexed: 12/15/2022] Open
Abstract
Automated quantitative and probabilistic medical image analysis has the potential to improve the accuracy and efficiency of the radiology workflow. We sought to determine whether AI systems for brain MRI diagnosis could be used as a clinical decision support tool to augment radiologist performance. We utilized previously developed AI systems that combine convolutional neural networks and expert-derived Bayesian networks to distinguish among 50 diagnostic entities on multimodal brain MRIs. We tested whether these systems could augment radiologist performance through an interactive clinical decision support tool known as Adaptive Radiology Interpretation and Education System (ARIES) in 194 test cases. Four radiology residents and three academic neuroradiologists viewed half of the cases unassisted and half with the results of the AI system displayed on ARIES. Diagnostic accuracy of radiologists for top diagnosis (TDx) and top three differential diagnosis (T3DDx) was compared with and without ARIES. Radiology resident performance was significantly better with ARIES for both TDx (55% vs 30%; P < .001) and T3DDx (79% vs 52%; P = 0.002), with the largest improvement for rare diseases (39% increase for T3DDx; P < 0.001). There was no significant difference between attending performance with and without ARIES for TDx (72% vs 69%; P = 0.48) or T3DDx (86% vs 89%; P = 0.39). These findings suggest that a hybrid deep learning and Bayesian inference clinical decision support system has the potential to augment diagnostic accuracy of non-specialists to approach the level of subspecialists for a large array of diseases on brain MRI.
Collapse
Affiliation(s)
- Jeffrey D Rudie
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA.
- Department of Radiology & Biomedical Imaging, University of California, San Francisco, San Francisco, CA, USA.
| | - Jeffrey Duda
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Michael Tran Duong
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Po-Hao Chen
- Department of Radiology, Cleveland Clinic Imaging Institute, Cleveland, OH, USA
| | - Long Xie
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Robert Kurtz
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Jeffrey B Ware
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Joshua Choi
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Raghav R Mattay
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | | | - James C Gee
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - R Nick Bryan
- Department of Diagnostic Medicine, Dell Medical School, University of Texas, Austin, TX, USA
| | - Tessa S Cook
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Suyash Mohan
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Ilya M Nasrallah
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
| | - Andreas M Rauschecker
- Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, PA, Philadelphia, USA
- Department of Radiology & Biomedical Imaging, University of California, San Francisco, San Francisco, CA, USA
| |
Collapse
|
7
|
Masur JH, Schmitt JE, Lalevic D, Cook TS, Bagley LJ, Mohan S, Nayate AP. Am I Ready to Be an Independent Neuroradiologist? Objective Trends in Neuroradiology Fellows' Performance during the Fellowship Year. AJNR Am J Neuroradiol 2021; 42:815-823. [PMID: 33664112 DOI: 10.3174/ajnr.a7030] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2020] [Accepted: 11/19/2020] [Indexed: 11/07/2022]
Abstract
BACKGROUND AND PURPOSE Aside from basic Accreditation Council for Graduate Medical Education guidelines, few metrics are in place to monitor fellows' progress. The purpose of this study was to determine objective trends in neuroradiology fellowship training on-call performance during an academic year. MATERIALS AND METHODS We retrospectively reviewed the number of cross-sectional neuroimaging studies dictated with complete reports by neuroradiology fellows during independent call. Monthly trends in total call cases, report turnaround times, relationships between volume and report turnaround times, and words addended to preliminary reports by attending neuroradiologists were evaluated with regression models. Monthly variation in frequencies of call-discrepancy macros were assessed via χ2 tests. Changes in frequencies of specific macro use between fellowship semesters were assessed via serial 2-sample tests of proportions. RESULTS From 2012 to 2017, for 29 fellows, monthly median report turnaround times significantly decreased during the academic year: July (first month) = 79 minutes (95% CI, 71-86 minutes) and June (12th month) = 55 minutes (95% CI, 52-60 minutes; P value = .023). Monthly report turnaround times were inversely correlated with total volumes for CT (r = -0.70, F = 9.639, P value = .011) but not MR imaging. Words addended to preliminary reports, a surrogate measurement of report clarity, slightly improved and discrepancy rates decreased during the last 6 months of fellowship. A nadir for report turnaround times, discrepancy errors, and words addended to reports was seen in December and January. CONCLUSIONS Progress through fellowship correlates with a decline in report turnaround times and discrepancy rates for cross-sectional neuroimaging call studies and slight improvement in indirect quantitative measurement of report clarity. These metrics can be tracked throughout the academic year, and the midyear would be a logical time point for programs to assess objective progress of fellows and address any deficiencies.
Collapse
Affiliation(s)
- J H Masur
- From the Department of Radiology (J.H.M., J.E.S., D.L., T.S.C., L.J.B., S.M.), Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - J E Schmitt
- From the Department of Radiology (J.H.M., J.E.S., D.L., T.S.C., L.J.B., S.M.), Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - D Lalevic
- From the Department of Radiology (J.H.M., J.E.S., D.L., T.S.C., L.J.B., S.M.), Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - T S Cook
- From the Department of Radiology (J.H.M., J.E.S., D.L., T.S.C., L.J.B., S.M.), Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - L J Bagley
- From the Department of Radiology (J.H.M., J.E.S., D.L., T.S.C., L.J.B., S.M.), Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - S Mohan
- From the Department of Radiology (J.H.M., J.E.S., D.L., T.S.C., L.J.B., S.M.), Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania
| | - A P Nayate
- Department of Radiology (A.P.N.), University Hospitals Cleveland Medical Center, Cleveland, Ohio
| |
Collapse
|
8
|
Rauschecker AM, Rudie JD, Xie L, Wang J, Duong MT, Botzolakis EJ, Kovalovich AM, Egan J, Cook TC, Bryan RN, Nasrallah IM, Mohan S, Gee JC. Artificial Intelligence System Approaching Neuroradiologist-level Differential Diagnosis Accuracy at Brain MRI. Radiology 2020; 295:626-637. [PMID: 32255417 DOI: 10.1148/radiol.2020190283] [Citation(s) in RCA: 58] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
Abstract
Background Although artificial intelligence (AI) shows promise across many aspects of radiology, the use of AI to create differential diagnoses for rare and common diseases at brain MRI has not been demonstrated. Purpose To evaluate an AI system for generation of differential diagnoses at brain MRI compared with radiologists. Materials and Methods This retrospective study tested performance of an AI system for probabilistic diagnosis in patients with 19 common and rare diagnoses at brain MRI acquired between January 2008 and January 2018. The AI system combines data-driven and domain-expertise methodologies, including deep learning and Bayesian networks. First, lesions were detected by using deep learning. Then, 18 quantitative imaging features were extracted by using atlas-based coregistration and segmentation. Third, these image features were combined with five clinical features by using Bayesian inference to develop probability-ranked differential diagnoses. Quantitative feature extraction algorithms and conditional probabilities were fine-tuned on a training set of 86 patients (mean age, 49 years ± 16 [standard deviation]; 53 women). Accuracy was compared with radiology residents, general radiologists, neuroradiology fellows, and academic neuroradiologists by using accuracy of top one, top two, and top three differential diagnoses in 92 independent test set patients (mean age, 47 years ± 18; 52 women). Results For accuracy of top three differential diagnoses, the AI system (91% correct) performed similarly to academic neuroradiologists (86% correct; P = .20), and better than radiology residents (56%; P < .001), general radiologists (57%; P < .001), and neuroradiology fellows (77%; P = .003). The performance of the AI system was not affected by disease prevalence (93% accuracy for common vs 85% for rare diseases; P = .26). Radiologists were more accurate at diagnosing common versus rare diagnoses (78% vs 47% across all radiologists; P < .001). Conclusion An artificial intelligence system for brain MRI approached overall top one, top two, and top three differential diagnoses accuracy of neuroradiologists and exceeded that of less-specialized radiologists. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Zaharchuk in this issue.
Collapse
Affiliation(s)
- Andreas M Rauschecker
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Jeffrey D Rudie
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Long Xie
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Jiancong Wang
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Michael Tran Duong
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Emmanuel J Botzolakis
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Asha M Kovalovich
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - John Egan
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Tessa C Cook
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - R Nick Bryan
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Ilya M Nasrallah
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - Suyash Mohan
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| | - James C Gee
- From the Department of Radiology & Biomedical Imaging, University of California, San Francisco, 513 Parnassus Ave, Room S-261, Box 0628, San Francisco, CA 94143-0628 (A.M.R.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA 19104 (A.M.R., J.D.R., L.X., J.W., M.T.D., A.M.K., J.E., T.C.C., I.M.N., S.M., J.C.G.); Mecklenburg Radiology Associates, Charlotte, NC (E.J.B.); and Department of Radiology, University of Texas at Austin, Austin, TX (R.N.B.)
| |
Collapse
|
9
|
Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol 2019; 92:20190389. [PMID: 31322909 DOI: 10.1259/bjr.20190389] [Citation(s) in RCA: 80] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023] Open
Abstract
In the era of personalized medicine, the emphasis of health care is shifting from populations to individuals. Artificial intelligence (AI) is capable of learning without explicit instruction and has emerging applications in medicine, particularly radiology. Whereas much attention has focused on teaching radiology trainees about AI, here our goal is to instead focus on how AI might be developed to better teach radiology trainees. While the idea of using AI to improve education is not new, the application of AI to medical and radiological education remains very limited. Based on the current educational foundation, we highlight an AI-integrated framework to augment radiology education and provide use case examples informed by our own institution's practice. The coming age of "AI-augmented radiology" may enable not only "precision medicine" but also what we describe as "precision medical education," where instruction is tailored to individual trainees based on their learning styles and needs.
Collapse
Affiliation(s)
- Michael Tran Duong
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Andreas M Rauschecker
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology & Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA
| | - Jeffrey D Rudie
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology & Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA
| | - Po-Hao Chen
- Imaging Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Tessa S Cook
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - R Nick Bryan
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Diagnostic Medicine, Dell Medical School, University of Texas at Austin, Austin, TX, USA
| | - Suyash Mohan
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Neurosurgery, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|