1
|
Wang D, Jin R, Shieh CC, Ng AY, Pham H, Dugal T, Barnett M, Winoto L, Wang C, Barnett Y. Real world validation of an AI-based CT hemorrhage detection tool. Front Neurol 2023; 14:1177723. [PMID: 37602253 PMCID: PMC10435741 DOI: 10.3389/fneur.2023.1177723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 07/12/2023] [Indexed: 08/22/2023] Open
Abstract
Introduction Intracranial hemorrhage (ICH) is a potentially life-threatening medical event that requires expedited diagnosis with computed tomography (CT). Automated medical imaging triaging tools can rapidly bring scans containing critical abnormalities, such as ICH, to the attention of radiologists and clinicians. Here, we retrospectively investigated the real-world performance of VeriScout™, an artificial intelligence-based CT hemorrhage detection and triage tool. Methods Ground truth for the presence or absence of ICH was iteratively determined by expert consensus in an unselected dataset of 527 consecutively acquired non-contrast head CT scans, which were sub-grouped according to the presence of artefact, post-operative features and referral source. The performance of VeriScout™ was compared with the ground truths for all groups. Results VeriScout™ detected hemorrhage with a sensitivity of 0.92 (CI 0.84-0.96) and a specificity of 0.96 (CI 0.94-0.98) in the global dataset, exceeding the sensitivity of general radiologists (0.88) with only a minor relative decrement in specificity (0.98). Crucially, the AI tool detected 13/14 cases of subarachnoid hemorrhage, a potentially fatal condition that is often missed in emergency department settings. There was no decrement in the performance of VeriScout™ in scans containing artefact or postoperative change. Using an integrated informatics platform, VeriScout™ was deployed into the existing radiology workflow. Detected hemorrhage cases were flagged in the hospital radiology information system (RIS) and relevant, annotated, preview images made available in the picture archiving and communications system (PACS) within 10 min. Conclusion AI-based radiology worklist prioritization for critical abnormalities, such as ICH, may enhance patient care without adding to radiologist or clinician burden.
Collapse
Affiliation(s)
- Dongang Wang
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
- Brain and Mind Centre, University of Sydney, Sydney, NSW, Australia
| | - Ruilin Jin
- Department of Medical Imaging, St. Vincent’s Hospital, Sydney, NSW, Australia
| | | | - Adrian Y. Ng
- Emergency Department, St. Vincent’s Hospital, Sydney, NSW, Australia
| | - Hiep Pham
- Department of Medical Imaging, St. Vincent’s Hospital, Sydney, NSW, Australia
| | - Tej Dugal
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Michael Barnett
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
- Brain and Mind Centre, University of Sydney, Sydney, NSW, Australia
| | - Luis Winoto
- Emergency Department, St. Vincent’s Hospital, Sydney, NSW, Australia
| | - Chenyu Wang
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
- Brain and Mind Centre, University of Sydney, Sydney, NSW, Australia
| | - Yael Barnett
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
- Department of Medical Imaging, St. Vincent’s Hospital, Sydney, NSW, Australia
| |
Collapse
|
2
|
Kundisch A, Hönning A, Mutze S, Kreissl L, Spohn F, Lemcke J, Sitz M, Sparenberg P, Goelz L. Deep learning algorithm in detecting intracranial hemorrhages on emergency computed tomographies. PLoS One 2021; 16:e0260560. [PMID: 34843559 PMCID: PMC8629230 DOI: 10.1371/journal.pone.0260560] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2021] [Accepted: 10/26/2021] [Indexed: 02/05/2023] Open
Abstract
BACKGROUND Highly accurate detection of intracranial hemorrhages (ICH) on head computed tomography (HCT) scans can prove challenging at high-volume centers. This study aimed to determine the number of additional ICHs detected by an artificial intelligence (AI) algorithm and to evaluate reasons for erroneous results at a level I trauma center with teleradiology services. METHODS In a retrospective multi-center cohort study, consecutive emergency non-contrast HCT scans were analyzed by a commercially available ICH detection software (AIDOC, Tel Aviv, Israel). Discrepancies between AI analysis and initial radiology report (RR) were reviewed by a blinded neuroradiologist to determine the number of additional ICHs detected and evaluate reasons leading to errors. RESULTS 4946 HCT (05/2020-09/2020) from 18 hospitals were included in the analysis. 205 reports (4.1%) were classified as hemorrhages by both radiology report and AI. Out of a total of 162 (3.3%) discrepant reports, 62 were confirmed as hemorrhages by the reference neuroradiologist. 33 ICHs were identified exclusively via RRs. The AI algorithm detected an additional 29 instances of ICH, missed 12.4% of ICH and overcalled 1.9%; RRs missed 10.9% of ICHs and overcalled 0.2%. Many of the ICHs missed by the AI algorithm were located in the subarachnoid space (42.4%) and under the calvaria (48.5%). 85% of ICHs missed by RRs occurred outside of regular working-hours. Calcifications (39.3%), beam-hardening artifacts (18%), tumors (15.7%), and blood vessels (7.9%) were the most common reasons for AI overcalls. ICH size, image quality, and primary examiner experience were not found to be significantly associated with likelihood of incorrect AI results. CONCLUSION Complementing human expertise with AI resulted in a 12.2% increase in ICH detection. The AI algorithm overcalled 1.9% HCT. TRIAL REGISTRATION German Clinical Trials Register (DRKS-ID: DRKS00023593).
Collapse
Affiliation(s)
- Almut Kundisch
- Center for Emergency Training, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Alexander Hönning
- Center for Clinical Research, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Sven Mutze
- Department of Radiology and Neuroradiology, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany.,Institute for Diagnostic Radiology and Neuroradiology, University Medicine Greifswald, Greifswald, Germany
| | - Lutz Kreissl
- Department of Radiology and Neuroradiology, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Frederik Spohn
- Department of Radiology and Neuroradiology, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Johannes Lemcke
- Department of Neurosurgery, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Maximilian Sitz
- Department of Neurosurgery, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Paul Sparenberg
- Department of Neurology, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany
| | - Leonie Goelz
- Department of Radiology and Neuroradiology, BG Klinikum Unfallkrankenhaus Berlin, Berlin, Germany.,Institute for Diagnostic Radiology and Neuroradiology, University Medicine Greifswald, Greifswald, Germany
| |
Collapse
|
3
|
On-call radiology 2020: Where trainees look for help in a high stakes and time sensitive environment. Clin Imaging 2021; 77:219-223. [PMID: 33992883 DOI: 10.1016/j.clinimag.2021.05.003] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2020] [Revised: 04/21/2021] [Accepted: 05/03/2021] [Indexed: 11/24/2022]
Abstract
BACKGROUND The radiology trainee on-call experience has undergone many changes in the past decade. The development of numerous online information sources has changed the landscape of opportunities for trainees seeking information while on-call. In this study, we sought to understand the current on-call information seeking behaviors of radiology trainees. METHODS We surveyed radiology fellows and residents at three major metropolitan area academic institutions. Survey topics included demographic information, on-call volumes, on-call resource seeking behaviors, preferred first and second line on-call resources and rationale for particular resource usage. RESULTS A total of 78 responses from trainees were recorded, 30.5% of the entire surveyed population. 70.5% of trainees preferred Radiopaedia as their first line resource. 26.9% of trainees preferred StatDx as their second line resource. 75.6% of respondents preferred their first line resource because it was easiest and fastest to access. 70.3% of respondents assigned a rating of 4 out of 5 when asked how often information they look for is found while on-call. There was a statistically significant difference according to gender (p = 0.002) with a higher percentage of males listing Radiopaedia as their first line resource compared to females. DISCUSSION The radiology trainee on-call experience is influenced by various factors. Over the past decade, online resources, particularly the open access resource Radiopaedia and the paid service StatDx, have overwhelmingly become the preferred first and second line options, as demonstrated by our study results.
Collapse
|
4
|
Scheinfeld MH, Dym RJ. Update on establishing and managing an overnight emergency radiology division. Emerg Radiol 2021; 28:993-1001. [PMID: 33881670 PMCID: PMC8059109 DOI: 10.1007/s10140-021-01935-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2021] [Accepted: 04/13/2021] [Indexed: 11/27/2022]
Abstract
Emergency department (ED) radiology divisions that serve to provide overnight attending coverage have become an increasingly common feature of radiology departments. The purpose of this article is to review the common ED radiology coverage models, describe desirable traits of emergency radiologists, and discuss workflow in the ED radiology setting. ED radiologists may be trained as ED radiologists or may develop the necessary skills and adopt the subspecialty. Choosing radiologists with the correct traits such as being a “night owl” and remaining calm under pressure and implementing an acceptable work schedule such as shift length of 9–10 h and a “one week on, two weeks off” schedule contribute to sustainability of the position. Strategies to address the unique stressors and workflow challenges of overnight emergency radiology coverage are also presented. Workflow facilitators including trainees, PAs, radiology assistants, and clerks all have roles to play in managing high case volumes and in making sure that the service is well staffed. Usage of artificial intelligence software is the latest technique to streamline workflow by identifying cases which should be prioritized on a busy worklist. Implementing such strategies will maintain quality of care for patients regardless of time of day as well as sustainability and quality of life for overnight emergency radiologists.
Collapse
Affiliation(s)
- Meir H. Scheinfeld
- Department of Radiology, Division of Emergency Radiology, Montefiore Medical Center, Albert Einstein College of Medicine, 111 East 210 Street, Bronx, NY 10467 USA
| | - R. Joshua Dym
- Department of Radiology, Division of Emergency Radiology, University Hospital, Rutgers New Jersey Medical School, 185 South Orange Avenue, MSB F508B, Newark, NJ 07103 USA
| |
Collapse
|