1
|
Del Gaizo AJ, Osborne TF, Shahoumian T, Sherrier R. Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time. Radiol Artif Intell 2024; 6:e240067. [PMID: 39017032 DOI: 10.1148/ryai.240067] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/18/2024]
Abstract
The diagnostic performance of an artificial intelligence (AI) clinical decision support solution for acute intracranial hemorrhage (ICH) detection was assessed in a large teleradiology practice. The impact on radiologist read times and system efficiency was also quantified. A total of 61 704 consecutive noncontrast head CT examinations were retrospectively evaluated. System performance was calculated along with mean and median read times for CT studies obtained before (baseline, pre-AI period; August 2021 to May 2022) and after (post-AI period; January 2023 to February 2024) AI implementation. The AI solution had a sensitivity of 75.6%, specificity of 92.1%, accuracy of 91.7%, prevalence of 2.70%, and positive predictive value of 21.1%. Of the 56 745 post-AI CT scans with no bleed identified by a radiologist, examinations falsely flagged as suspected ICH by the AI solution (n = 4464) took an average of 9 minutes 40 seconds (median, 8 minutes 7 seconds) to interpret as compared with 8 minutes 25 seconds (median, 6 minutes 48 seconds) for unremarkable CT scans before AI (n = 49 007) (P < .001) and 8 minutes 38 seconds (median, 6 minutes 53 seconds) after AI when ICH was not suspected by the AI solution (n = 52 281) (P < .001). CT scans with no bleed identified by the AI but reported as positive for ICH by the radiologist (n = 384) took an average of 14 minutes 23 seconds (median, 13 minutes 35 seconds) to interpret as compared with 13 minutes 34 seconds (median, 12 minutes 30 seconds) for CT scans correctly reported as a bleed by the AI (n = 1192) (P = .04). With lengthened read times for falsely flagged examinations, system inefficiencies may outweigh the potential benefits of using the tool in a high volume, low prevalence environment. Keywords: Artificial Intelligence, Intracranial Hemorrhage, Read Time, Report Turnaround Time, System Efficiency Supplemental material is available for this article. © RSNA, 2024.
Collapse
Affiliation(s)
- Andrew James Del Gaizo
- From the VA National Teleradiology Program, 795 Willow Rd, Bldg 3342, Menlo Park, CA 94025 (A.J.D.G., R.S.); VA Palo Alto Health Care System, Palo Alto, Calif (T.F.O.); Department of Radiology, Stanford University School of Medicine, Stanford, Calif (T.F.O.); and VA Health Solutions, Patient Care Services, Washington, DC (T.S.)
| | - Thomas F Osborne
- From the VA National Teleradiology Program, 795 Willow Rd, Bldg 3342, Menlo Park, CA 94025 (A.J.D.G., R.S.); VA Palo Alto Health Care System, Palo Alto, Calif (T.F.O.); Department of Radiology, Stanford University School of Medicine, Stanford, Calif (T.F.O.); and VA Health Solutions, Patient Care Services, Washington, DC (T.S.)
| | - Troy Shahoumian
- From the VA National Teleradiology Program, 795 Willow Rd, Bldg 3342, Menlo Park, CA 94025 (A.J.D.G., R.S.); VA Palo Alto Health Care System, Palo Alto, Calif (T.F.O.); Department of Radiology, Stanford University School of Medicine, Stanford, Calif (T.F.O.); and VA Health Solutions, Patient Care Services, Washington, DC (T.S.)
| | - Robert Sherrier
- From the VA National Teleradiology Program, 795 Willow Rd, Bldg 3342, Menlo Park, CA 94025 (A.J.D.G., R.S.); VA Palo Alto Health Care System, Palo Alto, Calif (T.F.O.); Department of Radiology, Stanford University School of Medicine, Stanford, Calif (T.F.O.); and VA Health Solutions, Patient Care Services, Washington, DC (T.S.)
| |
Collapse
|
2
|
Guenette JP, Young GS, Gupta R, Romero JM, Huang RY. A Public Health Perspective on Radiologists' Interpretation Volumes. AJNR Am J Neuroradiol 2024; 45:E10-E11. [PMID: 38697789 PMCID: PMC11288582 DOI: 10.3174/ajnr.a8240] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/05/2024]
Affiliation(s)
- Jeffrey P Guenette
- Division of NeuroradiologyBrigham and Women's HospitalBoston, Massachusetts
- Center for Evidence-Based ImagingBrigham and Women's HospitalBoston, Massachusetts
| | - Geoffrey S Young
- Division of NeuroradiologyBrigham and Women's HospitalBoston, Massachusetts
| | - Rajiv Gupta
- Division of NeuroradiologyBrigham and Women's HospitalBoston, Massachusetts
| | - Javier M Romero
- Division of NeuroradiologyBrigham and Women's HospitalBoston, Massachusetts
| | - Raymond Y Huang
- Division of NeuroradiologyBrigham and Women's HospitalBoston, Massachusetts
| |
Collapse
|
3
|
Rao K, Perry S, Hagedorn J, Carter K, Balkenende B, Policeni B. Impact of a Reading Room Coordinator on Efficiency of On-Call Radiology Residents. J Am Coll Radiol 2024; 21:642-650. [PMID: 37777077 DOI: 10.1016/j.jacr.2023.05.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Revised: 05/07/2023] [Accepted: 05/09/2023] [Indexed: 10/02/2023]
Abstract
OBJECTIVES Few level I trauma, tertiary care, academic centers have a paid, permanent reading room coordinator (RRC) to facilitate image management services during off-hour calls, to minimize interruptions to reading workflow. The purpose of this study is to investigate the effect of an RRC on the efficiency of radiology residents signing preliminary reports for emergency department (ED) and inpatient studies. METHODS A pre- and postintervention retrospective review was performed, using carestream PACS to retrieve imaging studies read on call during two time periods-July 1 to December 1, 2019 (pre-RRC), and July 1 to December 1, 2021 (post-RRC). Efficiency of residents signing preliminary reports was measured by turnaround time (TAT), defined as the time from when a study was marked complete by a technologist to when a preliminary report was signed by a resident, in PACS. RESULTS In the above time periods, residents interpreted a total of 64,406 studies on call. For ED studies, the mean TAT was 7.0 min shorter post-RRC, compared with pre-RRC (95% confidence interval [CI]: -7.8 to -6.1, (t = 15.50, degrees of freedom (df) = 31,866, P < .0001). The percentage of ED studies signed within 30 min increased from 57.7% to 65.8%, an increase of 8.1% (95% CI: 7.0% to 9.1%) after employing an RRC (χ2 = 228.11, df = 1, P < .0001). For inpatient studies, the mean TAT was 10.2 min shorter post-RRC (95% CI: -12.3 to -8.0, t = 9.22, df = 25,193, P < .0001). CONCLUSIONS An RRC increased radiology resident on-call workflow efficiency, facilitating care for patients in both the ED and inpatient setting.
Collapse
Affiliation(s)
- Karan Rao
- Department of Radiology, University of Iowa, Iowa City, Iowa
| | - Sarah Perry
- Department of Biostatistics, University of Iowa, Iowa City, Iowa
| | - Joshua Hagedorn
- Carver College of Medicine, University of Iowa, Iowa City, Iowa
| | - Knute Carter
- Clinical Associate Professor, Department of Biostatistics; Deputy Director, Center for Public Health Statistics, College of Public Health, University of Iowa, Iowa City, Iowa
| | | | - Bruno Policeni
- Clinical Professor, Director of Neuroradiology Fellowship; Vice-Chair for Operations and Education, Department of Radiology, University of Iowa, Iowa City, Iowa.
| |
Collapse
|
4
|
Sarkar N, Kumagai M, Meyr S, Pothapragada S, Unberath M, Li G, Ahmed SR, Smith EB, Davis MA, Khatri GD, Agrawal A, Delproposto ZS, Chen H, Caballero CG, Dreizin D. An ASER AI/ML expert panel formative user research study for an interpretable interactive splenic AAST grading graphical user interface prototype. Emerg Radiol 2024; 31:167-178. [PMID: 38302827 PMCID: PMC11257379 DOI: 10.1007/s10140-024-02202-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 01/08/2024] [Indexed: 02/03/2024]
Abstract
PURPOSE The AAST Organ Injury Scale is widely adopted for splenic injury severity but suffers from only moderate inter-rater agreement. This work assesses SpleenPro, a prototype interactive explainable artificial intelligence/machine learning (AI/ML) diagnostic aid to support AAST grading, for effects on radiologist dwell time, agreement, clinical utility, and user acceptance. METHODS Two trauma radiology ad hoc expert panelists independently performed timed AAST grading on 76 admission CT studies with blunt splenic injury, first without AI/ML assistance, and after a 2-month washout period and randomization, with AI/ML assistance. To evaluate user acceptance, three versions of the SpleenPro user interface with increasing explainability were presented to four independent expert panelists with four example cases each. A structured interview consisting of Likert scales and free responses was conducted, with specific questions regarding dimensions of diagnostic utility (DU); mental support (MS); effort, workload, and frustration (EWF); trust and reliability (TR); and likelihood of future use (LFU). RESULTS SpleenPro significantly decreased interpretation times for both raters. Weighted Cohen's kappa increased from 0.53 to 0.70 with AI/ML assistance. During user acceptance interviews, increasing explainability was associated with improvement in Likert scores for MS, EWF, TR, and LFU. Expert panelists indicated the need for a combined early notification and grading functionality, PACS integration, and report autopopulation to improve DU. CONCLUSIONS SpleenPro was useful for improving objectivity of AAST grading and increasing mental support. Formative user research identified generalizable concepts including the need for a combined detection and grading pipeline and integration with the clinical workflow.
Collapse
Affiliation(s)
- Nathan Sarkar
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
| | - Mitsuo Kumagai
- University of Maryland College Park, 4603 Calvert Rd, College Park, MD, 20740, USA
| | - Samantha Meyr
- University of Maryland College Park, 4603 Calvert Rd, College Park, MD, 20740, USA
| | - Sriya Pothapragada
- University of Maryland College Park, 4603 Calvert Rd, College Park, MD, 20740, USA
| | - Mathias Unberath
- Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA
| | - Guang Li
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
| | - Sagheer Rauf Ahmed
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
- R Adams Cowley Shock Trauma Center, 22 S Greene St, Baltimore, MD, 21201, USA
| | - Elana Beth Smith
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA
- R Adams Cowley Shock Trauma Center, 22 S Greene St, Baltimore, MD, 21201, USA
| | | | | | - Anjali Agrawal
- Teleradiology Solutions, 22 Lianfair Road Unit 6, Ardmore, PA, 19003, USA
| | | | - Haomin Chen
- Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA
| | | | - David Dreizin
- University of Maryland School of Medicine, 655 W. Baltimore Street, Baltimore, MD, 21201, USA.
- R Adams Cowley Shock Trauma Center, 22 S Greene St, Baltimore, MD, 21201, USA.
| |
Collapse
|
5
|
Gillingham N, Gupta D, Kamath A, Kagen A. Implementation of Medical Students as Radiology Reading Room Coordinators. Curr Probl Diagn Radiol 2024; 53:150-153. [PMID: 37925236 DOI: 10.1067/j.cpradiol.2023.10.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 10/18/2023] [Indexed: 11/06/2023]
Abstract
OBJECTIVE Effort has been made to minimize the burden of non-interpretive tasks (NITs), in particular by hiring and training non-radiologist support staff as reading room coordinators (RRCs). Our medical center recruited and trained senior medical students from our affiliated school of medicine to work alongside on-call radiology residents as RRCs. METHODS A 12-month Malpractice Carrier monetary grant was acquired to fund medical students at with the aim to reduce malpractice risk. After the first year, residents were surveyed regarding the impact of the RRCs on perceived on-call efficiency and morale. Furthermore, report turnaround times (TAT) on call shifts that were and were not accompanied by a RRC were compared. RESULTS 89 % of residents strongly agreed that the RRC improved workflow efficiency, decreased distractions, and felt less stressed during the call shift when the RRC was on duty. 78 % strongly agreed to be more likely to contact a referring clinician when the RRC was able to help coordinate. The mean TAT in the presence of a RRC was 36.8 min, and the mean TAT in the absence of a RRC was 36.9 min DISCUSSION: After hiring medical students to assist on-call radiology residents with noninterpretive tasks, residents reported subjective indicators of program success, but average report turnaround time was unaffected. Nevertheless, we predict that this type of program will continue to grow among academic radiology departments, though additional research is required to evaluate national trends and impacts on radiologist productivity and well-being.
Collapse
Affiliation(s)
- Nicolas Gillingham
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai West. 1000 10th Ave, Radiology Department, 4B 25, New York, NY 10019, USA.
| | - Divya Gupta
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai West. 1000 10th Ave, Radiology Department, 4B 25, New York, NY 10019, USA
| | - Amita Kamath
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai Hospital and Mount Sinai West. 1000 10th Ave, Radiology Department, 4B 25, New York, NY 10019, USA
| | - Alexander Kagen
- Site Chair, Department of Diagnostic, Molecular, and Interventional Radiology, Mount Sinai West and Mount Sinai St. Luke's Hospitals, Icahn School of Medicine at Mount Sinai. 1000 10th Ave, Radiology Department, 4B 25, New York, NY 10019, USA
| |
Collapse
|
6
|
Sreekrishnan A, Giurgiutiu DV, Kitamura F, Martinelli C, Abdala N, Haerian H, Dehkharghani S, Kwok K, Yedavalli V, Heit JJ. Decreasing false-positive detection of intracranial hemorrhage (ICH) using RAPID ICH 3. J Stroke Cerebrovasc Dis 2023; 32:107396. [PMID: 37883825 PMCID: PMC10877378 DOI: 10.1016/j.jstrokecerebrovasdis.2023.107396] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2023] [Revised: 09/17/2023] [Accepted: 09/26/2023] [Indexed: 10/28/2023] Open
Abstract
INTRODUCTION The prompt detection of intracranial hemorrhage (ICH) on a non-contrast head CT (NCCT) is critical for the appropriate triage of patients, particularly in high volume/high acuity settings. Several automated ICH detection tools have been introduced; however, at present, most suffer from suboptimal specificity leading to false-positive notifications. METHODS NCCT scans from 4 large databases were evaluated for the presence of an ICH (IPH, IVH, SAH or SDH) of >0.4 ml using fully-automated RAPID ICH 3.0 as compared to consensus detection from at least two neuroradiology experts. Scans were excluded for (1) severe CT artifacts, (2) prior neurosurgical procedures, or (3) recent intravenous contrast. ICH detection accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios by were determined. RESULTS A total of 881 studies were included. The automated software correctly identified 453/463 ICH-positive cases and 416/418 ICH-negative cases, resulting in a sensitivity of 97.84% and specificity 99.52%, positive predictive value 99.56%, and negative predictive value 97.65% for ICH detection. The positive and negative likelihood ratios for ICH detection were similarly favorable at 204.49 and 0.02 respectively. Mean processing time was <40 seconds. CONCLUSIONS In this large data set of nearly 900 patients, the automated software demonstrated high sensitivity and specificity for ICH detection, with rare false-positives.
Collapse
Affiliation(s)
| | | | - Felipe Kitamura
- DasaInova, Dasa, Universidade Federal de São Paulo, São Paulo, Brazil
| | | | - Nitamar Abdala
- Department of Diagnostic Imaging, Universidade Federal de São Paulo, São Paulo, Brazil
| | - Hafez Haerian
- Department of Neuroradiology, Northwest Hospital, Randallstown, MD, USA
| | - Seena Dehkharghani
- Departments of Radiology and Neurology, New York University Langone Health, New York, NY, USA
| | - Keith Kwok
- Department of Radiology, Central Valley Imaging Medical Associates/Regional Medical Center of San Jose, San Jose, CA, USA
| | - Vivek Yedavalli
- Department of Neuroradiology, Johns Hopkins Hospital, Baltimore, MD, USA
| | - Jeremy J Heit
- Department of Neurosurgery, Stanford Hospital, Palo Alto, CA, USA
| |
Collapse
|
7
|
Batra K, Xi Y, Bhagwat S, Espino A, Peshock RM. Radiologist Worklist Reprioritization Using Artificial Intelligence: Impact on Report Turnaround Times for CTPA Examinations Positive for Acute Pulmonary Embolism. AJR Am J Roentgenol 2023; 221:324-333. [PMID: 37095668 DOI: 10.2214/ajr.22.28949] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/08/2023]
Abstract
BACKGROUND. In patients with acute pulmonary embolism (PE), timely intervention (e.g., initiation of anticoagulation) is critical for optimizing clinical outcomes. OBJECTIVE. The purpose of this study was to evaluate the effect of artificial intelligence (AI)-based radiologist worklist reprioritization on report turnaround times for pulmonary CTA (CTPA) examinations positive for acute PE. METHODS. This retrospective single-center study included patients who underwent CTPA before (October 1, 2018-March 31, 2019 [pre-AI period]) and after (October 1, 2019-March 31, 2020 [post-AI period]) implementation of an AI tool that reprioritized CTPA examinations to the top of radiologists' reading worklists if acute PE was detected. EMR and dictation system timestamps were used to determine the wait time (time from examination completion to report initiation), read time (time from report initiation to report availability), and report turnaround time (sum of wait and read times) for the examinations. Times for reports positive for PE, with final radiology reports as reference, were compared between periods. RESULTS. The study included 2501 examinations of 2197 patients (1307 women, 890 men; mean age, 57.4 ± 17.0 [SD] years), including 1335 examinations from the pre-AI period and 1166 from the post-AI period. The frequency of acute PE, based on radiology reports, was 15.1% (201/1335) during the pre-AI period and 12.3% (144/1166) during the post-AI period. During the post-AI period, the AI tool reprioritized 12.7% (148/1166) of examinations. For PE-positive examinations, the post-AI period, compared with the pre-AI period, had significantly shorter mean report turnaround time (47.6 vs 59.9 minutes; mean difference, 12.3 minutes [95% CI, 0.6-26.0 minutes]) and mean wait time (21.4 vs 33.4 minutes; mean difference, 12.0 minutes [95% CI, 0.9-25.3 minutes]) but no significant difference in mean read time (26.3 vs 26.5 minutes; mean difference, 0.2 minutes [95% CI, -2.8 to 3.2 minutes]). During regular operational hours, wait time was significantly shorter in the post-AI than in the pre-AI period for routine-priority examinations (15.3 vs 43.7 minutes; mean difference, 28.4 minutes [95% CI, 2.2-64.7 minutes]) but not for stat- or urgent-priority examinations. CONCLUSION. AI-driven worklist reprioritization yielded reductions in report turnaround time and wait time for PE-positive CTPA examinations. CLINICAL IMPACT. By assisting radiologists in providing rapid diagnoses, the AI tool has potential for enabling earlier interventions for acute PE.
Collapse
Affiliation(s)
- Kiran Batra
- Department of Radiology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390
| | - Yin Xi
- Department of Radiology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390
| | - Siddharth Bhagwat
- Department of Radiology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390
| | - Adriana Espino
- Department of Radiology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390
| | - Ronald M Peshock
- Department of Radiology, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX 75390
- Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|
8
|
Agrawal A, Khatri GD, Khurana B, Sodickson AD, Liang Y, Dreizin D. A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations. Emerg Radiol 2023; 30:267-277. [PMID: 36913061 PMCID: PMC10362990 DOI: 10.1007/s10140-023-02121-0] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Accepted: 02/28/2023] [Indexed: 03/14/2023]
Abstract
PURPOSE There is a growing body of diagnostic performance studies for emergency radiology-related artificial intelligence/machine learning (AI/ML) tools; however, little is known about user preferences, concerns, experiences, expectations, and the degree of penetration of AI tools in emergency radiology. Our aim is to conduct a survey of the current trends, perceptions, and expectations regarding AI among American Society of Emergency Radiology (ASER) members. METHODS An anonymous and voluntary online survey questionnaire was e-mailed to all ASER members, followed by two reminder e-mails. A descriptive analysis of the data was conducted, and results summarized. RESULTS A total of 113 members responded (response rate 12%). The majority were attending radiologists (90%) with greater than 10 years' experience (80%) and from an academic practice (65%). Most (55%) reported use of commercial AI CAD tools in their practice. Workflow prioritization based on pathology detection, injury or disease severity grading and classification, quantitative visualization, and auto-population of structured reports were identified as high-value tasks. Respondents overwhelmingly indicated a need for explainable and verifiable tools (87%) and the need for transparency in the development process (80%). Most respondents did not feel that AI would reduce the need for emergency radiologists in the next two decades (72%) or diminish interest in fellowship programs (58%). Negative perceptions pertained to potential for automation bias (23%), over-diagnosis (16%), poor generalizability (15%), negative impact on training (11%), and impediments to workflow (10%). CONCLUSION ASER member respondents are in general optimistic about the impact of AI in the practice of emergency radiology and its impact on the popularity of emergency radiology as a subspecialty. The majority expect to see transparent and explainable AI models with the radiologist as the decision-maker.
Collapse
Affiliation(s)
- Anjali Agrawal
- New Delhi operations, Teleradiology Solutions, Delhi, India
| | - Garvit D Khatri
- Nuclear Medicine, Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA
| | - Bharti Khurana
- Emergency Radiology, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Aaron D Sodickson
- Emergency Radiology, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Yuanyuan Liang
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - David Dreizin
- Trauma and Emergency Radiology, Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
9
|
Dreizin D, Staziaki PV, Khatri GD, Beckmann NM, Feng Z, Liang Y, Delproposto ZS, Klug M, Spann JS, Sarkar N, Fu Y. Artificial intelligence CAD tools in trauma imaging: a scoping review from the American Society of Emergency Radiology (ASER) AI/ML Expert Panel. Emerg Radiol 2023; 30:251-265. [PMID: 36917287 PMCID: PMC10640925 DOI: 10.1007/s10140-023-02120-1] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Accepted: 02/27/2023] [Indexed: 03/16/2023]
Abstract
BACKGROUND AI/ML CAD tools can potentially improve outcomes in the high-stakes, high-volume model of trauma radiology. No prior scoping review has been undertaken to comprehensively assess tools in this subspecialty. PURPOSE To map the evolution and current state of trauma radiology CAD tools along key dimensions of technology readiness. METHODS Following a search of databases, abstract screening, and full-text document review, CAD tool maturity was charted using elements of data curation, performance validation, outcomes research, explainability, user acceptance, and funding patterns. Descriptive statistics were used to illustrate key trends. RESULTS A total of 4052 records were screened, and 233 full-text articles were selected for content analysis. Twenty-one papers described FDA-approved commercial tools, and 212 reported algorithm prototypes. Works ranged from foundational research to multi-reader multi-case trials with heterogeneous external data. Scalable convolutional neural network-based implementations increased steeply after 2016 and were used in all commercial products; however, options for explainability were narrow. Of FDA-approved tools, 9/10 performed detection tasks. Dataset sizes ranged from < 100 to > 500,000 patients, and commercialization coincided with public dataset availability. Cross-sectional torso datasets were uniformly small. Data curation methods with ground truth labeling by independent readers were uncommon. No papers assessed user acceptance, and no method included human-computer interaction. The USA and China had the highest research output and frequency of research funding. CONCLUSIONS Trauma imaging CAD tools are likely to improve patient care but are currently in an early stage of maturity, with few FDA-approved products for a limited number of uses. The scarcity of high-quality annotated data remains a major barrier.
Collapse
Affiliation(s)
- David Dreizin
- Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| | - Pedro V Staziaki
- Cardiothoracic Imaging, Department of Radiology, Larner College of Medicine, University of Vermont, Burlington, VT, USA
| | - Garvit D Khatri
- Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA
| | - Nicholas M Beckmann
- Memorial Hermann Orthopedic & Spine Hospital, McGovern Medical School at UTHealth, Houston, TX, USA
| | - Zhaoyong Feng
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Yuanyuan Liang
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Zachary S Delproposto
- Division of Emergency Radiology, Department of Radiology, University of Michigan, Ann Arbor, MI, USA
| | | | - J Stephen Spann
- Department of Radiology, University of Alabama at Birmingham Heersink School of Medicine, Birmingham, AL, USA
| | - Nathan Sarkar
- University of Maryland School of Medicine, Baltimore, MD, USA
| | - Yunting Fu
- Health Sciences and Human Services Library, University of Maryland, Baltimore, Baltimore, MD, USA
| |
Collapse
|
10
|
Chen H, Unberath M, Dreizin D. Toward automated interpretable AAST grading for blunt splenic injury. Emerg Radiol 2023; 30:41-50. [PMID: 36371579 DOI: 10.1007/s10140-022-02099-1] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2022] [Accepted: 11/04/2022] [Indexed: 11/13/2022]
Abstract
BACKGROUND The American Association for the Surgery of Trauma (AAST) splenic organ injury scale (OIS) is the most frequently used CT-based grading system for blunt splenic trauma. However, reported inter-rater agreement is modest, and an algorithm that objectively automates grading based on transparent and verifiable criteria could serve as a high-trust diagnostic aid. PURPOSE To pilot the development of an automated interpretable multi-stage deep learning-based system to predict AAST grade from admission trauma CT. METHODS Our pipeline includes 4 parts: (1) automated splenic localization, (2) Faster R-CNN-based detection of pseudoaneurysms (PSA) and active bleeds (AB), (3) nnU-Net segmentation and quantification of splenic parenchymal disruption (SPD), and (4) a directed graph that infers AAST grades from detection and segmentation results. Training and validation is performed on a dataset of adult patients (age ≥ 18) with voxelwise labeling, consensus AAST grading, and hemorrhage-related outcome data (n = 174). RESULTS AAST classification agreement (weighted κ) between automated and consensus AAST grades was substantial (0.79). High-grade (IV and V) injuries were predicted with accuracy, positive predictive value, and negative predictive value of 92%, 95%, and 89%. The area under the curve for predicting hemorrhage control intervention was comparable between expert consensus and automated AAST grading (0.83 vs 0.88). The mean combined inference time for the pipeline was 96.9 s. CONCLUSIONS The results of our method were rapid and verifiable, with high agreement between automated and expert consensus grades. Diagnosis of high-grade lesions and prediction of hemorrhage control intervention produced accurate results in adult patients.
Collapse
Affiliation(s)
- Haomin Chen
- Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
| | - Mathias Unberath
- Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
| | - David Dreizin
- Emergency and Trauma Imaging, Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
11
|
Subhas N, Johnson S, Caruso C, Kollai E, Obuchowski NA, Mody R, Parker HJ, Borkowski GP. Imaging Service Navigators: An Approach Toward More Efficient and Effective Communications. J Am Coll Radiol 2023; 20:79-86. [PMID: 36494062 DOI: 10.1016/j.jacr.2022.10.012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2022] [Revised: 10/13/2022] [Accepted: 10/20/2022] [Indexed: 12/12/2022]
Abstract
PURPOSE Many practices have implemented support services to assist radiologists with noninterpretive tasks; however, little research has been performed to assess the overall effect of these services. The purpose of this study was to evaluate the effect of a team of imaging service navigators (ISNs) incorporated into a practice on (1) number of communications, (2) time saved by radiologists, and (3) radiologist satisfaction with the service. METHODS The numbers and types of reports dictated by radiologists were captured for 6-month periods before and after ISN implementation. Communication rates before and after implementation were then calculated. The amount of perceived time savings using the ISN team and satisfaction with the service were assessed through pre- and postimplementation surveys of participating radiologists. Mean and median time savings and satisfaction rates were calculated. RESULTS The overall communication rate increased from 2.196% before ISNs to 3.278% after ISNs (49% increase; 95% confidence interval, 47%-52%). Communication rates increased among all communication subtypes (critical, urgent, routine, and actionable), with the highest increases in urgent (94%) and actionable (75%) findings. Before implementation, radiologists reported spending 39 min on average per day on communications tasks, with only 33% of radiologists indicating that the communication process was efficient. After implementation, radiologists reported mean time savings of 28 min (95% confidence interval, 19.9-35.1), and 82% of radiologists indicated a positive or highly positive view of the ISN service. CONCLUSIONS After ISN implementation, communication rates increased and radiologists reported spending less time performing communications. Most radiologists were satisfied with the service.
Collapse
Affiliation(s)
- Naveen Subhas
- Institute Vice Chair, Imaging Institute, Cleveland Clinic, Cleveland, Ohio.
| | | | | | | | - Nancy A Obuchowski
- Department Vice Chair, Quantitative Health Sciences, Cleveland Clinic, Cleveland, Ohio
| | - Rekha Mody
- Institute Quality Officer, Imaging Institute, Cleveland Clinic, Cleveland, Ohio
| | - H Joseph Parker
- Institute Administrator, Imaging Institute, Cleveland Clinic, Cleveland, Ohio
| | | |
Collapse
|
12
|
Dreizin D, Champ K, Dattwyler M, Bodanapally U, Smith EB, Li G, Singh R, Wang Z, Liang Y. Blunt splenic injury in adults: Association between volumetric quantitative CT parameters and intervention. J Trauma Acute Care Surg 2023; 94:125-132. [PMID: 35546417 PMCID: PMC9652480 DOI: 10.1097/ta.0000000000003684] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
BACKGROUND. Several ordinal grading systems are employed in deciding whether to perform angioembolization or splenectomy following blunt splenic injury. The 2018 AAST Organ Injury Scale (OIS) incorporates vascular lesions but not hemoperitoneum, which is considered in the Thompson classifier. Granular and verifiable quantitative measurements of these features may have a future role in facilitating objective decision-making. PURPOSE. To compare performance of CT volumetry-based quantitative modeling to the 1994 and 2018 AAST OIS and Thompson classifier for the following endpoints: decision to perform splenectomy (SPY), and the composite of SPY or angioembolization (AE) MATERIALS AND METHODS. Adult BSI patients (age ≥ 18 years) scanned with dual-phase CT prior to intervention at a single level I trauma center from 2017-2019 were included in this retrospective study (n=174). Scoring using 2018 AAST, 1994 AAST, and Thompson systems was performed retrospectively by two radiologists and arbitrated by a third. Endpoints included 1. SPY and 2. The composite of SPY or AE. Logistic regression models were developed from segmented active bleed, contained vascular lesion, splenic parenchymal disruption, and hemoperitoneum volumes. AUCs for ordinal systems and volumetric models were compared. RESULTS. Forty-seven BSI patients (27%) underwent SPY, and 87 patients (50%) underwent SPY or AE. Quantitative model AUCs (0.85- SPY, 0.82-composite) were not significantly different from 2018 AAST AUCs (0.81, 0.88, p=0.66, 0.14) for both endpoints, and were significantly improved over Thompson scoring (0.76, p=0.02; 0.77, p=0.04). CONCLUSION: Quantitative CT volumetry can be used to model intervention for BSI with accuracy comparable to 2018 AAST scoring and significantly higher than Thompson scoring. Study Type: Prognostic Level of Evidence: IV CT volumetry of blunt splenic injury-related features predicts splenectomy and angioembolization in adults and identifies clinically important target features for computer vision and automation research.
Collapse
Affiliation(s)
- David Dreizin
- From the Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine (D.D., M.D., U.B., E.B.S., G.L., Z.W., K.C., R.S.); and Department of Epidemiology and Public Health (Y.L.), University of Maryland School of Medicine, Baltimore, Maryland
| | | | | | | | | | | | | | | | | |
Collapse
|
13
|
Scheinfeld A, Schwartz C, Cunqueiro A. Resident pre-dictation of ED case delay signoff of those reports, but overall, residents improve report turnaround time. Emerg Radiol 2022; 30:153-159. [PMID: 36547740 PMCID: PMC9773659 DOI: 10.1007/s10140-022-02110-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2022] [Accepted: 12/07/2022] [Indexed: 12/24/2022]
Abstract
PURPOSE In academic radiology departments, attendings and resident are increasingly working together overnight for Emergency Department (ED) radiology coverage. The purpose of this study was to quantify reporting turnaround time for overnight cases read by residents and for overnight shifts with residents on duty. METHODS A retrospective study was performed at a hospital system where one overnight attending covers two hospitals with a 2nd/3rd year overnight resident, and a second overnight attending covers two other hospitals 80% of the time independently and 20% of the time with a fourth-year resident. In the first analysis, the median difference in turnaround time, from the time when the case was completed by the technologist to the time the attending finalized it, between cases read independently by attendings and cases pre-dictated by residents was calculated. In the second analysis, the median difference in turnaround time for all cases performed at the second two hospitals was compared on nights when an attending was on duty alone versus nights when a fourth-year resident was also on duty, regardless of if the resident had pre-dictated the case. RESULTS For computed tomography (CT), radiographs (XR), and ultrasound (US), there was a significant delay in turnaround time for cases pre-dictated by residents compared to cases read independently by attendings, ranging between 11 and 49 min depending on resident seniority and modality (p ≤ 0.001). For all cases on nights with a 4th year resident working, overall median report turnaround time decreased by 7 min (p < 0.001). CONCLUSION Resident pre-dictation causes delay in the finalization of individual CT, US, and XR reports; however, overall, working with residents results in a significant decrease in report turnaround time supporting the belief that overnight resident education does not delay patient care.
Collapse
Affiliation(s)
- Adina Scheinfeld
- grid.251993.50000000121791997Montefiore Medical Center, Albert Einstein College of Medicine, Department of Radiology, Division of Emergency Radiology, 111 East 210 Street, Bronx, NY 10467 USA
| | - Carly Schwartz
- grid.251993.50000000121791997Montefiore Medical Center, Albert Einstein College of Medicine, Department of Radiology, Division of Emergency Radiology, 111 East 210 Street, Bronx, NY 10467 USA
| | - Alain Cunqueiro
- grid.251993.50000000121791997Montefiore Medical Center, Albert Einstein College of Medicine, Department of Radiology, Division of Emergency Radiology, 111 East 210 Street, Bronx, NY 10467 USA
| |
Collapse
|
14
|
Virarkar M, Coleman JA, Siddiqui ZA, Leal VB, McClinton AE, Steele JR, Yang W, Bhosale PR, Klekers AR. Engaging nurse practitioners and physician assistants to improve patient care and drive productivity in a radiology consult practice at a comprehensive cancer center. J Am Assoc Nurse Pract 2022; 34:941-947. [PMID: 35796110 DOI: 10.1097/jxx.0000000000000721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2021] [Accepted: 03/09/2022] [Indexed: 11/26/2022]
Abstract
BACKGROUND The demand for health care in the United States is increasing because of an aging population and an increase in the number of individuals insured. This has led to requests to revamp the primary care infrastructure fundamentally. LOCAL PROBLEM The optimal use of nurse practitioners (NPs) and physician assistants (PAs) is still a subject of debate, but recently, it was reported that for many medical conditions, NP and PA-managed care outcomes are consistent with physician-managed care outcomes. METHODS Radiologists' productivity was measured according to relative value units (RVUs)/shift and professional billing changes. Patient care metrics measured were prescribed protocol to patient appointment lead time and number of same-day prescribed imaging protocol changes. INTERVENTIONS The focus was on radiologists' productivity and patient care for three months before and three months after integrating NP and PA into our abdominal radiology consult service. RESULTS We observed significant increases in the mean RVUs/shift (15.2 ± 0.9 vs. 6.2 ± 1.8; p = .02), studies read per shift (10.1 ± 0.5 vs. 4.4 ± 1.5; p = .003), revenue per shift hour ($756.20 ± 55.40 vs. $335.40 ± 32.60; p = .007), and protocol prescription to patient appointment lead time (39.3 ± 6.7 days vs. 16.3 ± 2.9 days; p = .005) and saw significant decreases in the mean prescribed CT (19.3 ± 0.6 vs. 3.3 ± 0.6; p = .001) and MRI (11.7 ± 0.6 vs. 8.30 ± 0.12; p = .011) same day protocol changes in NP and PA integrated workflow. CONCLUSIONS These findings suggest that NP and PA can be effectively integrated into the abdominal radiology consult service, increasing radiologists' productivity and enhancing clinical care.
Collapse
Affiliation(s)
- Mayur Virarkar
- Division of Abdominal and General Body Imaging, University of Florida College of Medicine, Jacksonville, Florida
| | - Joseph A Coleman
- College of Natural Sciences, Biology Major, University of Houston, Houston, Texas
| | - Zeeshan A Siddiqui
- Department of Ambulatory Operation & Access, The University of Texas MD Anderson Cancer Center, Houston, Texas
| | - Viola B Leal
- Department of Health Services Research, The University of Texas MD Anderson Cancer Center, Houston, Texas
| | - Angel E McClinton
- Department of Surgical Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas
| | - Joseph R Steele
- Department of Interventional Radiology, The University of Texas MD Anderson Cancer Center, Houston, Texas
| | - Wei Yang
- Department of Breast Imaging, The University of Texas MD Anderson Cancer Center, Houston, Texas
| | - Priya R Bhosale
- Department of Abdominal Radiology, The University of Texas MD Anderson Cancer Center, Houston, Texas
| | - Albert R Klekers
- Department of Abdominal Radiology, The University of Texas MD Anderson Cancer Center, Houston, Texas
| |
Collapse
|
15
|
Virarkar M, Coleman JA, Siddiqui ZA, Leal VB, McClinton AE, Steele JR, Yang W, Bhosale P, Klekers AR. PAs and NPs improve patient care and productivity in a radiology consult practice. JAAPA 2022; 35:46-51. [PMID: 35762956 DOI: 10.1097/01.jaa.0000832596.64788.f1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
OBJECTIVE To evaluate the effects on efficiency and patient care of the addition of physician assistants (PAs) and NPs to the abdominal radiology consult service. METHODS We obtained radiologist productivity and patient care metrics for 3 months before and 3 months after the integration of PAs and NPs into our consult service. RESULTS Integrating PAs and NPs into the workflow led to a significant increase in mean RVUs/shift (15.2 ± 0.9 versus 6.2 ± 1.8; P = .02), number of studies read per shift (10.1 ± 0.5 versus 4.4 ± 1.5; P = .003), revenue per shift hour ($756.20 ± $55.40 versus $335.40 ± $132.60; P = .007), protocol prescription to patient appointment lead time (39.3 ± 6.7 versus 16.3 ± 2.9 days; P = .005), and significant decreases in mean CT (19.3% ± 0.6 versus 3.3% ± 0.6; P = .001) and MRI (11.7% ± 0.6 versus 8.3% ± 0.12; P = .011) same-day protocol changes as patient appointments. CONCLUSIONS PAs and NPs can be effectively integrated into abdominal radiology consult service, increasing the productivity of radiologists, and enhancing clinical care.
Collapse
Affiliation(s)
- Mayur Virarkar
- Mayur Virarkar is an assistant professor in the Division of Abdominal and General Body Imaging at the University of Florida College of Medicine in Jacksonville, Fla. Joseph A. Coleman is a student at the University of Houston in Houston, Tex. At the University of Texas' MD Anderson Cancer Center in Houston, Zeeshan A. Siddiqui is a project director in the Department of Ambulatory Operation and Access, Viola B. Leal is a program manager, Angel E. McClinton practices in surgical oncology, Joseph R. Steele is an interventional radiologist, Wei Yang is a professor and chair of the Department of Diagnostic Radiology, Priya Bhosale is a professor in the departments of diagnostic radiology and abdominal imaging, and Albert R. Klekers is an assistant professor in the Department of Abdominal Radiology. The authors have disclosed no potential conflicts of interest, financial or otherwise
| | | | | | | | | | | | | | | | | |
Collapse
|
16
|
Performance of Automated RAPID Intracranial Hemorrhage Detection in Real-World Practice: A Single-Institution Experience. J Comput Assist Tomogr 2022; 46:770-774. [PMID: 35617649 DOI: 10.1097/rct.0000000000001335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
BACKGROUND AND PURPOSE Intracranial hemorrhage (ICH) is a common finding in patients presenting to the emergency department with acute neurological symptoms. Noncontrast head computed tomography (NCCT) is the primary modality for assessment and detection of ICH in the acute setting. RAPID ICH software aims to automatically detect ICH on NCCT and was previously shown to have high accuracy when applied to a curated test data set. Here, we measured the test performance characteristics of RAPID ICH software in detecting ICH on NCCT performed in patients undergoing emergency stroke evaluation at a tertiary academic comprehensive stroke center. MATERIALS AND METHODS This retrospective study assessed consecutive patients over a 6-month period who presented with acute neurological symptoms suspicious for stroke and underwent NCCT with RAPID ICH postprocessing. RAPID ICH detection was compared with the interpretation of a reference standard comprising a board-certified or board-eligible neuroradiologist, or in cases of discrepancy, adjudicated by a consensus panel of 3 neuroradiologists. Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of RAPID ICH for ICH detection were determined. RESULTS Three hundred seven NCCT scans were included in the study. RAPID ICH correctly identified 34 of 37 cases with ICH and 228 of 270 without ICH. RAPID ICH had a sensitivity of 91.9% (78.1%-98.3%), specificity of 84.4% (79.6%-88.6%), NPV of 98.7% (96.3%-99.6%), PPV of 44.7% (37.6%-52.1%), and overall accuracy of 85.3% (80.9%-89.1%). CONCLUSIONS In a real-world scenario, RAPID ICH software demonstrated high NPV but low PPV for the presence of ICH when evaluating possible stroke patients.
Collapse
|
17
|
Alis D, Alis C, Yergin M, Topel C, Asmakutlu O, Bagcilar O, Senli YD, Ustundag A, Salt V, Dogan SN, Velioglu M, Selcuk HH, Kara B, Ozer C, Oksuz I, Kizilkilic O, Karaarslan E. A joint convolutional-recurrent neural network with an attention mechanism for detecting intracranial hemorrhage on noncontrast head CT. Sci Rep 2022; 12:2084. [PMID: 35136123 PMCID: PMC8826390 DOI: 10.1038/s41598-022-05872-x] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 12/30/2021] [Indexed: 12/27/2022] Open
Abstract
To investigate the performance of a joint convolutional neural networks-recurrent neural networks (CNN-RNN) using an attention mechanism in identifying and classifying intracranial hemorrhage (ICH) on a large multi-center dataset; to test its performance in a prospective independent sample consisting of consecutive real-world patients. All consecutive patients who underwent emergency non-contrast-enhanced head CT in five different centers were retrospectively gathered. Five neuroradiologists created the ground-truth labels. The development dataset was divided into the training and validation set. After the development phase, we integrated the deep learning model into an independent center’s PACS environment for over six months for assessing the performance in a real clinical setting. Three radiologists created the ground-truth labels of the testing set with a majority voting. A total of 55,179 head CT scans of 48,070 patients, 28,253 men (58.77%), with a mean age of 53.84 ± 17.64 years (range 18–89) were enrolled in the study. The validation sample comprised 5211 head CT scans, with 991 being annotated as ICH-positive. The model's binary accuracy, sensitivity, and specificity on the validation set were 99.41%, 99.70%, and 98.91, respectively. During the prospective implementation, the model yielded an accuracy of 96.02% on 452 head CT scans with an average prediction time of 45 ± 8 s. The joint CNN-RNN model with an attention mechanism yielded excellent diagnostic accuracy in assessing ICH and its subtypes on a large-scale sample. The model was seamlessly integrated into the radiology workflow. Though slightly decreased performance, it provided decisions on the sample of consecutive real-world patients within a minute.
Collapse
Affiliation(s)
- Deniz Alis
- Radiology Department, Acibadem Mehmet Ali Aydinlar University School of Medicine, Istanbul, Turkey
| | - Ceren Alis
- Neurology Department, Istanbul Istinye State Hospital, Istanbul, Turkey.
| | - Mert Yergin
- Department of Software Engineering and Applied Sciences, Bahcesehir University, Istanbul, Turkey
| | - Cagdas Topel
- Department of Radiology, Istanbul Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital, Halkali, Istanbul, Turkey
| | - Ozan Asmakutlu
- Department of Radiology, Istanbul Mehmet Akif Ersoy Thoracic and Cardiovascular Surgery Training and Research Hospital, Halkali, Istanbul, Turkey
| | - Omer Bagcilar
- Radiology Department, Istanbul Silivri State Hospital, Istanbul, Turkey
| | - Yeseren Deniz Senli
- Radiology Department, Cerrahpaşa Medical Faculty, Istanbul University-Cerrahpasa, Istanbul, Turkey
| | - Ahmet Ustundag
- Radiology Department, Cerrahpaşa Medical Faculty, Istanbul University-Cerrahpasa, Istanbul, Turkey
| | - Vefa Salt
- Radiology Department, Cerrahpaşa Medical Faculty, Istanbul University-Cerrahpasa, Istanbul, Turkey
| | | | - Murat Velioglu
- Radiology Department, Istanbul Fatih Sultan Mehmet Training and Research Hospital, Istanbul, Turkey
| | - Hakan Hatem Selcuk
- Radiology Department, Istanbul Bakırköy Sadi Konuk Training and Research Hospital, Istanbul, Turkey
| | - Batuhan Kara
- Radiology Department, Istanbul Bakırköy Sadi Konuk Training and Research Hospital, Istanbul, Turkey
| | - Caner Ozer
- Computer Engineering Department, Istanbul Technical University, Istanbul, Turkey
| | - Ilkay Oksuz
- Computer Engineering Department, Istanbul Technical University, Istanbul, Turkey
| | - Osman Kizilkilic
- Radiology Department, Cerrahpaşa Medical Faculty, Istanbul University-Cerrahpasa, Istanbul, Turkey
| | - Ercan Karaarslan
- Radiology Department, Acibadem Mehmet Ali Aydinlar University School of Medicine, Istanbul, Turkey
| |
Collapse
|
18
|
Pourvaziri A, Narayan AK, Tso D, Baliyan V, Glover M, Bizzo BC, Kako B, Succi MD, Lev MH, Flores EJ. Imaging Information Overload: Quantifying the burden of interpretive and non-interpretive tasks for CT angiography for aortic pathologies in emergency radiology. Curr Probl Diagn Radiol 2022; 51:546-551. [DOI: 10.1067/j.cpradiol.2022.01.008] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 12/18/2021] [Accepted: 01/05/2022] [Indexed: 12/20/2022]
|
19
|
Daugaard Jørgensen M, Antulov R, Hess S, Lysdahlgaard S. Convolutional neural network performance compared to radiologists in detecting intracranial hemorrhage from brain computed tomography: A systematic review and meta-analysis. Eur J Radiol 2021; 146:110073. [PMID: 34847397 DOI: 10.1016/j.ejrad.2021.110073] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2021] [Revised: 10/01/2021] [Accepted: 11/22/2021] [Indexed: 12/23/2022]
Abstract
PURPOSE To compare the diagnostic accuracy of convolutional neural networks (CNN) with radiologists as the reference standard in the diagnosis of intracranial hemorrhages (ICH) with non contrast computed tomography of the cerebrum (NCTC). METHODS PubMed, Embase, Scopus, and Web of Science were searched for the period from 1 January 2012 to 20 July 2020; eligible studies included patients with and without ICH as the target condition undergoing NCTC, studies had deep learning algorithms based on CNNs and radiologists reports as the minimum reference standard. Pooled sensitivities, specificities and a summary receiver operating characteristics curve (SROC) were employed for meta-analysis. RESULTS 5,119 records were identified through database searching. Title-screening left 47 studies for full-text assessment and 6 studies for meta-analysis. Comparing the CNN performance to reference standards in the retrospective studies found a pooled sensitivity of 96.00% (95% CI: 93.00% to 97.00%), pooled specificity of 97.00% (95% CI: 90.00% to 99.00%) and SROC of 98.00% (95% CI: 97.00% to 99.00%), and combining retrospective and studies with external datasets found a pooled sensitivity of 95.00% (95% CI: 91.00% to 97.00%), pooled specificity of 96.00% (95% CI: 91.00% to 98.00%) and a pooled SROC of 98.00% (95% CI: 97.00% to 99.00%). CONCLUSION This review found the diagnostic performance of CNNs to be equivalent to that of radiologists for retrospective studies. Out-of-sample external validation studies pooled with retrospective studies found CNN performance to be slightly worse. There is a critical need for studies with a robust reference standard and external data-set validation.
Collapse
Affiliation(s)
| | - Ronald Antulov
- Department of Radiology and Nuclear Medicine, Hospital of South West Jutland, University Hospital of Southern Denmark, Esbjerg, Denmark; Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark
| | - Søren Hess
- Department of Radiology and Nuclear Medicine, Hospital of South West Jutland, University Hospital of Southern Denmark, Esbjerg, Denmark; Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark
| | - Simon Lysdahlgaard
- Department of Radiology and Nuclear Medicine, Hospital of South West Jutland, University Hospital of Southern Denmark, Esbjerg, Denmark; Department of Regional Health Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark.
| |
Collapse
|
20
|
Salehinejad H, Kitamura J, Ditkofsky N, Lin A, Bharatha A, Suthiphosuwan S, Lin HM, Wilson JR, Mamdani M, Colak E. A real-world demonstration of machine learning generalizability in the detection of intracranial hemorrhage on head computerized tomography. Sci Rep 2021; 11:17051. [PMID: 34426587 PMCID: PMC8382750 DOI: 10.1038/s41598-021-95533-2] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2020] [Accepted: 07/22/2021] [Indexed: 11/13/2022] Open
Abstract
Machine learning (ML) holds great promise in transforming healthcare. While published studies have shown the utility of ML models in interpreting medical imaging examinations, these are often evaluated under laboratory settings. The importance of real world evaluation is best illustrated by case studies that have documented successes and failures in the translation of these models into clinical environments. A key prerequisite for the clinical adoption of these technologies is demonstrating generalizable ML model performance under real world circumstances. The purpose of this study was to demonstrate that ML model generalizability is achievable in medical imaging with the detection of intracranial hemorrhage (ICH) on non-contrast computed tomography (CT) scans serving as the use case. An ML model was trained using 21,784 scans from the RSNA Intracranial Hemorrhage CT dataset while generalizability was evaluated using an external validation dataset obtained from our busy trauma and neurosurgical center. This real world external validation dataset consisted of every unenhanced head CT scan (n = 5965) performed in our emergency department in 2019 without exclusion. The model demonstrated an AUC of 98.4%, sensitivity of 98.8%, and specificity of 98.0%, on the test dataset. On external validation, the model demonstrated an AUC of 95.4%, sensitivity of 91.3%, and specificity of 94.1%. Evaluating the ML model using a real world external validation dataset that is temporally and geographically distinct from the training dataset indicates that ML generalizability is achievable in medical imaging applications.
Collapse
Affiliation(s)
- Hojjat Salehinejad
- Li Ka Shing Centre for Healthcare Analytics Research and Training, St. Michael's Hospital, Toronto, Canada.,Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada
| | | | - Noah Ditkofsky
- Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada.,Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Amy Lin
- Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada.,Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Aditya Bharatha
- Li Ka Shing Centre for Healthcare Analytics Research and Training, St. Michael's Hospital, Toronto, Canada.,Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada.,Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Suradech Suthiphosuwan
- Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada.,Faculty of Medicine, University of Toronto, Toronto, Canada
| | - Hui-Ming Lin
- Li Ka Shing Centre for Healthcare Analytics Research and Training, St. Michael's Hospital, Toronto, Canada
| | - Jefferson R Wilson
- Li Ka Shing Centre for Healthcare Analytics Research and Training, St. Michael's Hospital, Toronto, Canada.,Faculty of Medicine, University of Toronto, Toronto, Canada.,Division of Neurosurgery, Department of Surgery, University of Toronto, Toronto, Canada
| | - Muhammad Mamdani
- Li Ka Shing Centre for Healthcare Analytics Research and Training, St. Michael's Hospital, Toronto, Canada.,Faculty of Medicine, University of Toronto, Toronto, Canada.,Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Canada.,Dalla Lana Faculty of Public Health, University of Toronto, Toronto, Canada
| | - Errol Colak
- Li Ka Shing Centre for Healthcare Analytics Research and Training, St. Michael's Hospital, Toronto, Canada. .,Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, 30 Bond Street, Toronto, ON, M5B 1W8, Canada. .,Faculty of Medicine, University of Toronto, Toronto, Canada.
| |
Collapse
|
21
|
Ginat D. Implementation of Machine Learning Software on the Radiology Worklist Decreases Scan View Delay for the Detection of Intracranial Hemorrhage on CT. Brain Sci 2021; 11:brainsci11070832. [PMID: 34201775 PMCID: PMC8301803 DOI: 10.3390/brainsci11070832] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2021] [Revised: 06/01/2021] [Accepted: 06/18/2021] [Indexed: 11/16/2022] Open
Abstract
BACKGROUND AND PURPOSE Prompt identification of acute intracranial hemorrhage on CT is important. The goal of this study was to assess the impact of artificial intelligence software for prioritizing positive cases. MATERIALS AND METHODS Cases analyzed by Aidoc (Tel Aviv, Israel) software for triaging acute intracranial hemorrhage cases on non-contrast head CT were retrospectively reviewed. The scan view delay time was calculated as the difference between the time the study was completed on PACS and the time the study was first opened by a radiologist. The scan view delay was stratified by scan location, including emergency, inpatient, and outpatient. The scan view delay times for cases flagged as positive by the software were compared to those that were not flagged. RESULTS A total of 8723 scans were assessed by the software, including 6894 cases that were not flagged and 1829 cases that were flagged as positive. Although there was no statistically significant difference in the scan view time for emergency cases, there was a significantly lower scan view time for positive outpatient and inpatient cases flagged by the software versus negative cases, with a reduction of 604 min on average, 90% in the scan view delay (p-value < 0.0001) for outpatients, and a reduction of 38 min on average, and 10% in the scan view delay (p-value <= 0.01) for inpatients. CONCLUSION The use of artificial intelligence triage software for acute intracranial hemorrhage on head CT scans is associated with a significantly shorter scan view delay for cases flagged as positive than cases not flagged among outpatients and inpatients at an academic medical center.
Collapse
Affiliation(s)
- Daniel Ginat
- Department of Radiology, University of Chicago, Chicago, IL 60615, USA
| |
Collapse
|
22
|
Prabhu SP, Breen MA. Challenges within our current education model and where to direct innovation: the three-strand triquetra approach. Pediatr Radiol 2021; 51:1097-1100. [PMID: 33245374 PMCID: PMC7692422 DOI: 10.1007/s00247-020-04911-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/01/2020] [Revised: 09/01/2020] [Accepted: 11/09/2020] [Indexed: 11/20/2022]
Affiliation(s)
- Sanjay P. Prabhu
- grid.2515.30000 0004 0378 8438Department of Radiology, Boston Children’s Hospital and Harvard Medical School, 300 Longwood Ave., Boston, MA 02115 USA
| | - Micheál A. Breen
- grid.2515.30000 0004 0378 8438Department of Radiology, Boston Children’s Hospital and Harvard Medical School, 300 Longwood Ave., Boston, MA 02115 USA
| |
Collapse
|
23
|
Heit JJ, Coelho H, Lima FO, Granja M, Aghaebrahim A, Hanel R, Kwok K, Haerian H, Cereda CW, Venkatasubramanian C, Dehkharghani S, Carbonera LA, Wiener J, Copeland K, Mont'Alverne F. Automated Cerebral Hemorrhage Detection Using RAPID. AJNR Am J Neuroradiol 2020; 42:273-278. [PMID: 33361378 DOI: 10.3174/ajnr.a6926] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2020] [Accepted: 09/13/2020] [Indexed: 12/28/2022]
Abstract
BACKGROUND AND PURPOSE Intracranial hemorrhage (ICH) is an important event that is diagnosed on head NCCT. Increased NCCT utilization in busy hospitals may limit timely identification of ICH. RAPID ICH is an automated hybrid 2D-3D convolutional neural network application designed to detect ICH that may allow for expedited ICH diagnosis. We determined the accuracy of RAPID ICH for ICH detection and ICH volumetric quantification on NCCT. MATERIALS AND METHODS NCCT scans were evaluated for ICH by RAPID ICH. Consensus detection of ICH by 3 neuroradiology experts was used as the criterion standard for RAPID ICH comparison. ICH volume was also automatically determined by RAPID ICH in patients with intraparenchymal or intraventricular hemorrhage and compared with manually segmented ICH volumes by a single neuroradiology expert. ICH detection accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios by RAPID ICH were determined. RESULTS We included 308 studies. RAPID ICH correctly identified 151/158 ICH cases and 143/150 ICH-negative cases, which resulted in high sensitivity (0.956, CI: 0.911-0.978), specificity (0.953, CI: 0.907-0.977), positive predictive value (0.956, CI: 0.911-0.978), and negative predictive value (0.953, CI: 0.907-0.977) for ICH detection. The positive likelihood ratio (20.479, CI 9.928-42.245) and negative likelihood ratio (0.046, CI 0.023-0.096) for ICH detection were similarly favorable. RAPID ICH volumetric quantification for intraparenchymal and intraventricular hemorrhages strongly correlated with expert manual segmentation (correlation coefficient r = 0.983); the median absolute error was 3 mL. CONCLUSIONS RAPID ICH is highly accurate in the detection of ICH and in the volumetric quantification of intraparenchymal and intraventricular hemorrhages.
Collapse
Affiliation(s)
- J J Heit
- From the Department of Radiology, Neuroimaging, and Neurointervention Division (J.J.H.), Stanford University School of Medicine, Stanford, California
| | - H Coelho
- Interventional Radiology Service (H.C., F.M.)
| | - F O Lima
- Department of Neurology (F.O.L.), Hospital Geral de Fortaleza, R. Ávila Goulart, Fortaleza, Brazil
| | - M Granja
- Baptist Neurological Institute (M.G., A.A., R.H.), Lyerly Neurosurgery/Baptist Health, Jacksonville, Florida.,Diagnostic Imaging Department (M.G., A.A., R.H.), Fundación Santa Fe de Bogota University Hospital, Bogotá, Colombia
| | - A Aghaebrahim
- Baptist Neurological Institute (M.G., A.A., R.H.), Lyerly Neurosurgery/Baptist Health, Jacksonville, Florida.,Diagnostic Imaging Department (M.G., A.A., R.H.), Fundación Santa Fe de Bogota University Hospital, Bogotá, Colombia
| | - R Hanel
- Baptist Neurological Institute (M.G., A.A., R.H.), Lyerly Neurosurgery/Baptist Health, Jacksonville, Florida.,Diagnostic Imaging Department (M.G., A.A., R.H.), Fundación Santa Fe de Bogota University Hospital, Bogotá, Colombia
| | - K Kwok
- Department of Radiology (K.K.), Central Valley Imaging Medical Associates, Manteca, California
| | - H Haerian
- Department of Radiology (H.H.), LifeBridge Health, Baltimore, Maryland
| | - C W Cereda
- Department of Neurology (C.W.C.), EOC Ospedale Regionale di Lugano, Lugano, Switzerland
| | - C Venkatasubramanian
- Neurocritical Care and Stroke, Department of Neurology (C.V.), Stanford University, Palo Alto, California
| | - S Dehkharghani
- Department of Radiology (S.D.), NY University Langone Health, New York, New York
| | - L A Carbonera
- Hospital das Clínicas de Porto Alegre (L.A.C.), Bairro Santa Cecilia, Brazil
| | - J Wiener
- Department of Radiology (J.W.), Boca Raton Regional Hospital, Boca Raton, Florida
| | - K Copeland
- Boulder Statistics (K.C.), Steamboat Springs, Colorado
| | | |
Collapse
|
24
|
Bhayana R, Wang CD, Menezes RJ, Bartlett ES, Choi J. Optimising after-hours workflow of computed tomography orders in the emergency department. BMJ Open Qual 2020; 9:bmjoq-2020-000969. [PMID: 32665302 PMCID: PMC7365424 DOI: 10.1136/bmjoq-2020-000969] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2020] [Revised: 06/08/2020] [Accepted: 06/17/2020] [Indexed: 12/30/2022] Open
Abstract
Ordering and protocolling CT scans after-hours from the emergency department (ED) at our institution previously required discussion between the ED physician and radiology resident, which led to workflow inefficiency. Our intervention consisted of creating an electronic list of CT requests that radiology residents would monitor. Radiology protocolled straightforward requests and contacted the ordering physician for more details when required. We aimed to improve workflow efficiency, increase provider satisfaction and reduce CT turnaround time without significantly affecting CT utilisation. Plan-do-study-act cycles were used to plan and evaluate the intervention. The intervention was initiated on weekday evenings and then expanded to weekend hours after an interim analysis. Qualitative outcomes were measured via electronic survey, and quantitative outcomes were collected from administrative data and analysed via control charts and other statistical methods. Survey response was high from ED physicians (76%, n=82/108) and radiology residents (79%, n=30/38). After the intervention, the majority of ED staff and radiology residents perceived improved workflow efficiency (96.3%, 73.3%), radiology residents noted a subjective decrease in disruptions (83.3%) and most ED staff felt that scans were performed more quickly (84.1%). Radiology residents received fewer pages per shift, adjusted for scan volume. There was a reduction in time from order entry to protocol on weekday shifts only, with no statistically significant effect on time from order entry to scan. Segmented regression analysis demonstrated a background increase in utilisation over time (0.7–2.0 CT/100 ED visits/year, p<0.0005), but the intervention itself did not contribute to an overall increase in CT utilisation. In conclusion, our intervention led to improved perceived workflow efficiency and reduced pages. Scans were protocoled more quickly on weekdays, but turnaround times were otherwise not significantly affected by the intervention. Background CT utilisation increased over time, but this increase was not attributable to our intervention.
Collapse
Affiliation(s)
- Rajesh Bhayana
- Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada
| | - Chenhan D Wang
- Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada
| | - Ravi J Menezes
- Department of Medical Imaging, University Health Network, Toronto, Ontario, Canada
| | - Eric S Bartlett
- Department of Medical Imaging, University Health Network, Toronto, Ontario, Canada
| | - Joseph Choi
- Emergency Medicine, University Health Network, Toronto, Ontario, Canada
| |
Collapse
|
25
|
Makary MS, Hartwell C, Egbert NK, Prevedello LM. Streamlining Communications and Enabling Point-of-care Education in Radiology Through a Mobile Application Solution. Curr Probl Diagn Radiol 2020; 49:150-153. [DOI: 10.1067/j.cpradiol.2019.04.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2019] [Revised: 03/18/2019] [Accepted: 04/02/2019] [Indexed: 11/22/2022]
|
26
|
Levy JL, Freeman CW, Cho JK, Iyalomhe O, Scanlon MH. Evaluating the Impact of a Call Triage Assistant on Resident Efficiency, Errors, and Stress. J Am Coll Radiol 2020; 17:414-420. [DOI: 10.1016/j.jacr.2019.11.007] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2019] [Revised: 11/02/2019] [Accepted: 11/12/2019] [Indexed: 10/25/2022]
|
27
|
Sabat S, Kalapos P, Slonimsky E. Quantifying disruption of workflow by phone calls to the neuroradiology reading room. BMJ Open Qual 2019; 8:e000442. [PMID: 31637315 PMCID: PMC6768390 DOI: 10.1136/bmjoq-2018-000442] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2018] [Revised: 08/12/2019] [Accepted: 08/18/2019] [Indexed: 11/17/2022] Open
Abstract
Introduction The purpose of this study was to understand the source and the reason for the phone calls to our neuroradiology suit and to quantify the size of the problem in terms of duration of individual and aggregated calls. Materials and methods Observation of the neuroradiology reading room for the entire duration of the working hours over three non-consecutive days was performed, and included telephone calls start time, end time and calls duration for incoming telephone calls. After each phone call the recipients were queried on the details of the phone call; the origin of the call, the reason for the call and the response. Results The average total number of minutes (min) spent on the phone each day was 64 min per working day with a total of 39 phone calls per day and 4.4 per hour on average. The trainees answered 71% of the phone calls with additional intervention by attending in 13% of phone calls. The most common source of phone calls was from either the MRI/CT technicians (48%), followed by providers (20%) and returning pages (18%). Conclusion Cumulative time spent on the phone by neuroradiologists in the reading room ended up in more than an hour per working day, while trainees were taking the majority of phone calls. Most phone calls originated from technicians, hence, requiring specific solutions to mitigate this kind of interruption.
Collapse
Affiliation(s)
- Shyam Sabat
- Penn State Health Milton S Hershey Medical Center, Hershey, Pennsylvania, USA
| | - Paul Kalapos
- Penn State Health Milton S Hershey Medical Center, Hershey, Pennsylvania, USA
| | - Einat Slonimsky
- Penn State Health Milton S Hershey Medical Center, Hershey, Pennsylvania, USA
| |
Collapse
|
28
|
Banaste N, Caurier B, Bratan F, Bergerot JF, Thomson V, Millet I. Whole-Body CT in Patients with Multiple Traumas: Factors Leading to Missed Injury. Radiology 2018; 289:374-383. [PMID: 30084754 DOI: 10.1148/radiol.2018180492] [Citation(s) in RCA: 42] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Purpose To determine radiologic and clinical markers predictive of missed injuries at early whole-body CT image interpretation. Materials and Methods For this retrospective study, 2354 consecutive whole-body CT examinations were performed in patients with multiple traumas from 26 hospitals interpreted at a teleradiology center study during on-call period from February 2011 to September 2016. All whole-body CT images were interpreted by the on-call radiologist and reviewed within 12-48 hours by another radiologist to detect missed injury as the standard of reference. The first and review reports of all examinations were retrospectively reviewed. Univariable and multivariable logistic regression with a stepwise selection method were performed to identify clinical and radiologic predictors of missed injury. Results This study included 639 women (27.1%) and 1715 men (72.8%). The median age of men, women, and the entire population was 34 years (age range, 1-96 years). On a per-scan basis, there were 304 (12.9%) missed injuries and 59 (2.5%) were clinically significant. On a per-injury basis, the missed injury rate was 530 of 5979 (8.8%). More than two injured body parts (odds ratio, 1.4 [95% confidence interval: 1.1, 1.8]; P = .01), patient age older than 30 years (odds ratio, 2.8 [95% confidence interval: 2.1, 3.8]; P < .001), and an initial clinical severity class of 1 (odds ratio, 1.9 [95% confidence interval: 1.3, 2.8]; P < .001) were independent predictive factors of missed injury. Conclusion Multiple traumas with more than two injured body parts, age older than 30 years, or an initial clinical severity class of 1 were associated with missed injury at whole-body CT. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Novelline in this issue.
Collapse
Affiliation(s)
- Nathan Banaste
- From the Department of Radiology, CHU Centre Léon Bérard, Lyon, France (N.B.); Lyon 1 University, Villeurbanne, France (N.B., B.C.); Department of Emergency Medicine, CH Saint-Joseph Saint-Luc, Lyon, France (B.C.); Imadis Teleradiology, Lyon, France (N.B., F.B., J.F.B., V.T.); and Department of Radiology, CHU Montpellier, Université de Montpellier, 191 Ave du Doyen Gaston Giraud, 34295 Montpellier, France (I.M.)
| | - Bérénice Caurier
- From the Department of Radiology, CHU Centre Léon Bérard, Lyon, France (N.B.); Lyon 1 University, Villeurbanne, France (N.B., B.C.); Department of Emergency Medicine, CH Saint-Joseph Saint-Luc, Lyon, France (B.C.); Imadis Teleradiology, Lyon, France (N.B., F.B., J.F.B., V.T.); and Department of Radiology, CHU Montpellier, Université de Montpellier, 191 Ave du Doyen Gaston Giraud, 34295 Montpellier, France (I.M.)
| | - Flavie Bratan
- From the Department of Radiology, CHU Centre Léon Bérard, Lyon, France (N.B.); Lyon 1 University, Villeurbanne, France (N.B., B.C.); Department of Emergency Medicine, CH Saint-Joseph Saint-Luc, Lyon, France (B.C.); Imadis Teleradiology, Lyon, France (N.B., F.B., J.F.B., V.T.); and Department of Radiology, CHU Montpellier, Université de Montpellier, 191 Ave du Doyen Gaston Giraud, 34295 Montpellier, France (I.M.)
| | - Jean-François Bergerot
- From the Department of Radiology, CHU Centre Léon Bérard, Lyon, France (N.B.); Lyon 1 University, Villeurbanne, France (N.B., B.C.); Department of Emergency Medicine, CH Saint-Joseph Saint-Luc, Lyon, France (B.C.); Imadis Teleradiology, Lyon, France (N.B., F.B., J.F.B., V.T.); and Department of Radiology, CHU Montpellier, Université de Montpellier, 191 Ave du Doyen Gaston Giraud, 34295 Montpellier, France (I.M.)
| | - Vivien Thomson
- From the Department of Radiology, CHU Centre Léon Bérard, Lyon, France (N.B.); Lyon 1 University, Villeurbanne, France (N.B., B.C.); Department of Emergency Medicine, CH Saint-Joseph Saint-Luc, Lyon, France (B.C.); Imadis Teleradiology, Lyon, France (N.B., F.B., J.F.B., V.T.); and Department of Radiology, CHU Montpellier, Université de Montpellier, 191 Ave du Doyen Gaston Giraud, 34295 Montpellier, France (I.M.)
| | - Ingrid Millet
- From the Department of Radiology, CHU Centre Léon Bérard, Lyon, France (N.B.); Lyon 1 University, Villeurbanne, France (N.B., B.C.); Department of Emergency Medicine, CH Saint-Joseph Saint-Luc, Lyon, France (B.C.); Imadis Teleradiology, Lyon, France (N.B., F.B., J.F.B., V.T.); and Department of Radiology, CHU Montpellier, Université de Montpellier, 191 Ave du Doyen Gaston Giraud, 34295 Montpellier, France (I.M.)
| |
Collapse
|
29
|
Chang PD, Kuoy E, Grinband J, Weinberg BD, Thompson M, Homo R, Chen J, Abcede H, Shafie M, Sugrue L, Filippi CG, Su MY, Yu W, Hess C, Chow D. Hybrid 3D/2D Convolutional Neural Network for Hemorrhage Evaluation on Head CT. AJNR Am J Neuroradiol 2018; 39:1609-1616. [PMID: 30049723 DOI: 10.3174/ajnr.a5742] [Citation(s) in RCA: 129] [Impact Index Per Article: 21.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2018] [Accepted: 06/06/2018] [Indexed: 01/28/2023]
Abstract
BACKGROUND AND PURPOSE Convolutional neural networks are a powerful technology for image recognition. This study evaluates a convolutional neural network optimized for the detection and quantification of intraparenchymal, epidural/subdural, and subarachnoid hemorrhages on noncontrast CT. MATERIALS AND METHODS This study was performed in 2 phases. First, a training cohort of all NCCTs acquired at a single institution between January 1, 2017, and July 31, 2017, was used to develop and cross-validate a custom hybrid 3D/2D mask ROI-based convolutional neural network architecture for hemorrhage evaluation. Second, the trained network was applied prospectively to all NCCTs ordered from the emergency department between February 1, 2018, and February 28, 2018, in an automated inference pipeline. Hemorrhage-detection accuracy, area under the curve, sensitivity, specificity, positive predictive value, and negative predictive value were assessed for full and balanced datasets and were further stratified by hemorrhage type and size. Quantification was assessed by the Dice score coefficient and the Pearson correlation. RESULTS A 10,159-examination training cohort (512,598 images; 901/8.1% hemorrhages) and an 862-examination test cohort (23,668 images; 82/12% hemorrhages) were used in this study. Accuracy, area under the curve, sensitivity, specificity, positive predictive value, and negative-predictive value for hemorrhage detection were 0.975, 0.983, 0.971, 0.975, 0.793, and 0.997 on training cohort cross-validation and 0.970, 0.981, 0.951, 0.973, 0.829, and 0.993 for the prospective test set. Dice scores for intraparenchymal hemorrhage, epidural/subdural hemorrhage, and SAH were 0.931, 0.863, and 0.772, respectively. CONCLUSIONS A customized deep learning tool is accurate in the detection and quantification of hemorrhage on NCCT. Demonstrated high performance on prospective NCCTs ordered from the emergency department suggests the clinical viability of the proposed deep learning tool.
Collapse
Affiliation(s)
- P D Chang
- From the Departments of Radiology (P.D.C., E.K., M.T., R.H., M.-Y.S., D.C.).,Departments of Radiology (P.D.C., L.S., C.H.), University of California, San Francisco, California
| | - E Kuoy
- From the Departments of Radiology (P.D.C., E.K., M.T., R.H., M.-Y.S., D.C.)
| | - J Grinband
- Department of Radiology (J.G.), Columbia University, New York, New York
| | - B D Weinberg
- Department of Radiology (B.D.W.), Emory University School of Medicine, Atlanta, Georgia
| | - M Thompson
- From the Departments of Radiology (P.D.C., E.K., M.T., R.H., M.-Y.S., D.C.)
| | - R Homo
- From the Departments of Radiology (P.D.C., E.K., M.T., R.H., M.-Y.S., D.C.)
| | | | - H Abcede
- Neurology (H.A., M.S., W.Y.), University of California Irvine
| | - M Shafie
- Neurology (H.A., M.S., W.Y.), University of California Irvine
| | - L Sugrue
- Departments of Radiology (P.D.C., L.S., C.H.), University of California, San Francisco, California
| | - C G Filippi
- Department of Radiology (C.G.F.), North Shore University Hospital, Long Island, New York
| | - M-Y Su
- From the Departments of Radiology (P.D.C., E.K., M.T., R.H., M.-Y.S., D.C.)
| | - W Yu
- Neurology (H.A., M.S., W.Y.), University of California Irvine
| | - C Hess
- Departments of Radiology (P.D.C., L.S., C.H.), University of California, San Francisco, California
| | - D Chow
- From the Departments of Radiology (P.D.C., E.K., M.T., R.H., M.-Y.S., D.C.)
| |
Collapse
|