Quantifying the Impact of Noninterpretive Tasks on Radiology Report Turn-Around Times

Deep Learning to Detect Intracranial Hemorrhage in a National Teleradiology Program and the Impact on Interpretation Time

The diagnostic performance of an artificial intelligence (AI) clinical decision support solution for acute intracranial hemorrhage (ICH) detection was assessed in a large teleradiology practice. The impact on radiologist read times and system efficiency was also quantified. A total of 61 704 consecutive noncontrast head CT examinations were retrospectively evaluated. System performance was calculated along with mean and median read times for CT studies obtained before (baseline, pre-AI period; August 2021 to May 2022) and after (post-AI period; January 2023 to February 2024) AI implementation. The AI solution had a sensitivity of 75.6%, specificity of 92.1%, accuracy of 91.7%, prevalence of 2.70%, and positive predictive value of 21.1%. Of the 56 745 post-AI CT scans with no bleed identified by a radiologist, examinations falsely flagged as suspected ICH by the AI solution (n = 4464) took an average of 9 minutes 40 seconds (median, 8 minutes 7 seconds) to interpret as compared with 8 minutes 25 seconds (median, 6 minutes 48 seconds) for unremarkable CT scans before AI (n = 49 007) (P < .001) and 8 minutes 38 seconds (median, 6 minutes 53 seconds) after AI when ICH was not suspected by the AI solution (n = 52 281) (P < .001). CT scans with no bleed identified by the AI but reported as positive for ICH by the radiologist (n = 384) took an average of 14 minutes 23 seconds (median, 13 minutes 35 seconds) to interpret as compared with 13 minutes 34 seconds (median, 12 minutes 30 seconds) for CT scans correctly reported as a bleed by the AI (n = 1192) (P = .04). With lengthened read times for falsely flagged examinations, system inefficiencies may outweigh the potential benefits of using the tool in a high volume, low prevalence environment. Keywords: Artificial Intelligence, Intracranial Hemorrhage, Read Time, Report Turnaround Time, System Efficiency Supplemental material is available for this article. © RSNA, 2024.

Impact of a Reading Room Coordinator on Efficiency of On-Call Radiology Residents

OBJECTIVES

Few level I trauma, tertiary care, academic centers have a paid, permanent reading room coordinator (RRC) to facilitate image management services during off-hour calls, to minimize interruptions to reading workflow. The purpose of this study is to investigate the effect of an RRC on the efficiency of radiology residents signing preliminary reports for emergency department (ED) and inpatient studies.

METHODS

A pre- and postintervention retrospective review was performed, using carestream PACS to retrieve imaging studies read on call during two time periods-July 1 to December 1, 2019 (pre-RRC), and July 1 to December 1, 2021 (post-RRC). Efficiency of residents signing preliminary reports was measured by turnaround time (TAT), defined as the time from when a study was marked complete by a technologist to when a preliminary report was signed by a resident, in PACS.

RESULTS

In the above time periods, residents interpreted a total of 64,406 studies on call. For ED studies, the mean TAT was 7.0 min shorter post-RRC, compared with pre-RRC (95% confidence interval [CI]: -7.8 to -6.1, (t = 15.50, degrees of freedom (df) = 31,866, P < .0001). The percentage of ED studies signed within 30 min increased from 57.7% to 65.8%, an increase of 8.1% (95% CI: 7.0% to 9.1%) after employing an RRC (χ2 = 228.11, df = 1, P < .0001). For inpatient studies, the mean TAT was 10.2 min shorter post-RRC (95% CI: -12.3 to -8.0, t = 9.22, df = 25,193, P < .0001).

CONCLUSIONS

An RRC increased radiology resident on-call workflow efficiency, facilitating care for patients in both the ED and inpatient setting.

An ASER AI/ML expert panel formative user research study for an interpretable interactive splenic AAST grading graphical user interface prototype

PURPOSE

The AAST Organ Injury Scale is widely adopted for splenic injury severity but suffers from only moderate inter-rater agreement. This work assesses SpleenPro, a prototype interactive explainable artificial intelligence/machine learning (AI/ML) diagnostic aid to support AAST grading, for effects on radiologist dwell time, agreement, clinical utility, and user acceptance.

METHODS

Two trauma radiology ad hoc expert panelists independently performed timed AAST grading on 76 admission CT studies with blunt splenic injury, first without AI/ML assistance, and after a 2-month washout period and randomization, with AI/ML assistance. To evaluate user acceptance, three versions of the SpleenPro user interface with increasing explainability were presented to four independent expert panelists with four example cases each. A structured interview consisting of Likert scales and free responses was conducted, with specific questions regarding dimensions of diagnostic utility (DU); mental support (MS); effort, workload, and frustration (EWF); trust and reliability (TR); and likelihood of future use (LFU).

RESULTS

SpleenPro significantly decreased interpretation times for both raters. Weighted Cohen's kappa increased from 0.53 to 0.70 with AI/ML assistance. During user acceptance interviews, increasing explainability was associated with improvement in Likert scores for MS, EWF, TR, and LFU. Expert panelists indicated the need for a combined early notification and grading functionality, PACS integration, and report autopopulation to improve DU.

CONCLUSIONS

SpleenPro was useful for improving objectivity of AAST grading and increasing mental support. Formative user research identified generalizable concepts including the need for a combined detection and grading pipeline and integration with the clinical workflow.

Implementation of Medical Students as Radiology Reading Room Coordinators

OBJECTIVE

Effort has been made to minimize the burden of non-interpretive tasks (NITs), in particular by hiring and training non-radiologist support staff as reading room coordinators (RRCs). Our medical center recruited and trained senior medical students from our affiliated school of medicine to work alongside on-call radiology residents as RRCs.

METHODS

A 12-month Malpractice Carrier monetary grant was acquired to fund medical students at with the aim to reduce malpractice risk. After the first year, residents were surveyed regarding the impact of the RRCs on perceived on-call efficiency and morale. Furthermore, report turnaround times (TAT) on call shifts that were and were not accompanied by a RRC were compared.

RESULTS

89 % of residents strongly agreed that the RRC improved workflow efficiency, decreased distractions, and felt less stressed during the call shift when the RRC was on duty. 78 % strongly agreed to be more likely to contact a referring clinician when the RRC was able to help coordinate. The mean TAT in the presence of a RRC was 36.8 min, and the mean TAT in the absence of a RRC was 36.9 min DISCUSSION: After hiring medical students to assist on-call radiology residents with noninterpretive tasks, residents reported subjective indicators of program success, but average report turnaround time was unaffected. Nevertheless, we predict that this type of program will continue to grow among academic radiology departments, though additional research is required to evaluate national trends and impacts on radiologist productivity and well-being.

Decreasing false-positive detection of intracranial hemorrhage (ICH) using RAPID ICH 3

INTRODUCTION

The prompt detection of intracranial hemorrhage (ICH) on a non-contrast head CT (NCCT) is critical for the appropriate triage of patients, particularly in high volume/high acuity settings. Several automated ICH detection tools have been introduced; however, at present, most suffer from suboptimal specificity leading to false-positive notifications.

METHODS

NCCT scans from 4 large databases were evaluated for the presence of an ICH (IPH, IVH, SAH or SDH) of >0.4 ml using fully-automated RAPID ICH 3.0 as compared to consensus detection from at least two neuroradiology experts. Scans were excluded for (1) severe CT artifacts, (2) prior neurosurgical procedures, or (3) recent intravenous contrast. ICH detection accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios by were determined.

RESULTS

A total of 881 studies were included. The automated software correctly identified 453/463 ICH-positive cases and 416/418 ICH-negative cases, resulting in a sensitivity of 97.84% and specificity 99.52%, positive predictive value 99.56%, and negative predictive value 97.65% for ICH detection. The positive and negative likelihood ratios for ICH detection were similarly favorable at 204.49 and 0.02 respectively. Mean processing time was <40 seconds.

CONCLUSIONS

In this large data set of nearly 900 patients, the automated software demonstrated high sensitivity and specificity for ICH detection, with rare false-positives.

Radiologist Worklist Reprioritization Using Artificial Intelligence: Impact on Report Turnaround Times for CTPA Examinations Positive for Acute Pulmonary Embolism

BACKGROUND. In patients with acute pulmonary embolism (PE), timely intervention (e.g., initiation of anticoagulation) is critical for optimizing clinical outcomes. OBJECTIVE. The purpose of this study was to evaluate the effect of artificial intelligence (AI)-based radiologist worklist reprioritization on report turnaround times for pulmonary CTA (CTPA) examinations positive for acute PE. METHODS. This retrospective single-center study included patients who underwent CTPA before (October 1, 2018-March 31, 2019 [pre-AI period]) and after (October 1, 2019-March 31, 2020 [post-AI period]) implementation of an AI tool that reprioritized CTPA examinations to the top of radiologists' reading worklists if acute PE was detected. EMR and dictation system timestamps were used to determine the wait time (time from examination completion to report initiation), read time (time from report initiation to report availability), and report turnaround time (sum of wait and read times) for the examinations. Times for reports positive for PE, with final radiology reports as reference, were compared between periods. RESULTS. The study included 2501 examinations of 2197 patients (1307 women, 890 men; mean age, 57.4 ± 17.0 [SD] years), including 1335 examinations from the pre-AI period and 1166 from the post-AI period. The frequency of acute PE, based on radiology reports, was 15.1% (201/1335) during the pre-AI period and 12.3% (144/1166) during the post-AI period. During the post-AI period, the AI tool reprioritized 12.7% (148/1166) of examinations. For PE-positive examinations, the post-AI period, compared with the pre-AI period, had significantly shorter mean report turnaround time (47.6 vs 59.9 minutes; mean difference, 12.3 minutes [95% CI, 0.6-26.0 minutes]) and mean wait time (21.4 vs 33.4 minutes; mean difference, 12.0 minutes [95% CI, 0.9-25.3 minutes]) but no significant difference in mean read time (26.3 vs 26.5 minutes; mean difference, 0.2 minutes [95% CI, -2.8 to 3.2 minutes]). During regular operational hours, wait time was significantly shorter in the post-AI than in the pre-AI period for routine-priority examinations (15.3 vs 43.7 minutes; mean difference, 28.4 minutes [95% CI, 2.2-64.7 minutes]) but not for stat- or urgent-priority examinations. CONCLUSION. AI-driven worklist reprioritization yielded reductions in report turnaround time and wait time for PE-positive CTPA examinations. CLINICAL IMPACT. By assisting radiologists in providing rapid diagnoses, the AI tool has potential for enabling earlier interventions for acute PE.

A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations

PURPOSE

There is a growing body of diagnostic performance studies for emergency radiology-related artificial intelligence/machine learning (AI/ML) tools; however, little is known about user preferences, concerns, experiences, expectations, and the degree of penetration of AI tools in emergency radiology. Our aim is to conduct a survey of the current trends, perceptions, and expectations regarding AI among American Society of Emergency Radiology (ASER) members.

METHODS

An anonymous and voluntary online survey questionnaire was e-mailed to all ASER members, followed by two reminder e-mails. A descriptive analysis of the data was conducted, and results summarized.

RESULTS

A total of 113 members responded (response rate 12%). The majority were attending radiologists (90%) with greater than 10 years' experience (80%) and from an academic practice (65%). Most (55%) reported use of commercial AI CAD tools in their practice. Workflow prioritization based on pathology detection, injury or disease severity grading and classification, quantitative visualization, and auto-population of structured reports were identified as high-value tasks. Respondents overwhelmingly indicated a need for explainable and verifiable tools (87%) and the need for transparency in the development process (80%). Most respondents did not feel that AI would reduce the need for emergency radiologists in the next two decades (72%) or diminish interest in fellowship programs (58%). Negative perceptions pertained to potential for automation bias (23%), over-diagnosis (16%), poor generalizability (15%), negative impact on training (11%), and impediments to workflow (10%).

CONCLUSION

ASER member respondents are in general optimistic about the impact of AI in the practice of emergency radiology and its impact on the popularity of emergency radiology as a subspecialty. The majority expect to see transparent and explainable AI models with the radiologist as the decision-maker.

Artificial intelligence CAD tools in trauma imaging: a scoping review from the American Society of Emergency Radiology (ASER) AI/ML Expert Panel

BACKGROUND

AI/ML CAD tools can potentially improve outcomes in the high-stakes, high-volume model of trauma radiology. No prior scoping review has been undertaken to comprehensively assess tools in this subspecialty.

PURPOSE

To map the evolution and current state of trauma radiology CAD tools along key dimensions of technology readiness.

METHODS

Following a search of databases, abstract screening, and full-text document review, CAD tool maturity was charted using elements of data curation, performance validation, outcomes research, explainability, user acceptance, and funding patterns. Descriptive statistics were used to illustrate key trends.

RESULTS

A total of 4052 records were screened, and 233 full-text articles were selected for content analysis. Twenty-one papers described FDA-approved commercial tools, and 212 reported algorithm prototypes. Works ranged from foundational research to multi-reader multi-case trials with heterogeneous external data. Scalable convolutional neural network-based implementations increased steeply after 2016 and were used in all commercial products; however, options for explainability were narrow. Of FDA-approved tools, 9/10 performed detection tasks. Dataset sizes ranged from < 100 to > 500,000 patients, and commercialization coincided with public dataset availability. Cross-sectional torso datasets were uniformly small. Data curation methods with ground truth labeling by independent readers were uncommon. No papers assessed user acceptance, and no method included human-computer interaction. The USA and China had the highest research output and frequency of research funding.

CONCLUSIONS

Trauma imaging CAD tools are likely to improve patient care but are currently in an early stage of maturity, with few FDA-approved products for a limited number of uses. The scarcity of high-quality annotated data remains a major barrier.

Toward automated interpretable AAST grading for blunt splenic injury

BACKGROUND

The American Association for the Surgery of Trauma (AAST) splenic organ injury scale (OIS) is the most frequently used CT-based grading system for blunt splenic trauma. However, reported inter-rater agreement is modest, and an algorithm that objectively automates grading based on transparent and verifiable criteria could serve as a high-trust diagnostic aid.

PURPOSE

To pilot the development of an automated interpretable multi-stage deep learning-based system to predict AAST grade from admission trauma CT.

METHODS

Our pipeline includes 4 parts: (1) automated splenic localization, (2) Faster R-CNN-based detection of pseudoaneurysms (PSA) and active bleeds (AB), (3) nnU-Net segmentation and quantification of splenic parenchymal disruption (SPD), and (4) a directed graph that infers AAST grades from detection and segmentation results. Training and validation is performed on a dataset of adult patients (age ≥ 18) with voxelwise labeling, consensus AAST grading, and hemorrhage-related outcome data (n = 174).

RESULTS

AAST classification agreement (weighted κ) between automated and consensus AAST grades was substantial (0.79). High-grade (IV and V) injuries were predicted with accuracy, positive predictive value, and negative predictive value of 92%, 95%, and 89%. The area under the curve for predicting hemorrhage control intervention was comparable between expert consensus and automated AAST grading (0.83 vs 0.88). The mean combined inference time for the pipeline was 96.9 s.

CONCLUSIONS

The results of our method were rapid and verifiable, with high agreement between automated and expert consensus grades. Diagnosis of high-grade lesions and prediction of hemorrhage control intervention produced accurate results in adult patients.

Imaging Service Navigators: An Approach Toward More Efficient and Effective Communications

PURPOSE

Many practices have implemented support services to assist radiologists with noninterpretive tasks; however, little research has been performed to assess the overall effect of these services. The purpose of this study was to evaluate the effect of a team of imaging service navigators (ISNs) incorporated into a practice on (1) number of communications, (2) time saved by radiologists, and (3) radiologist satisfaction with the service.

METHODS

The numbers and types of reports dictated by radiologists were captured for 6-month periods before and after ISN implementation. Communication rates before and after implementation were then calculated. The amount of perceived time savings using the ISN team and satisfaction with the service were assessed through pre- and postimplementation surveys of participating radiologists. Mean and median time savings and satisfaction rates were calculated.

RESULTS

The overall communication rate increased from 2.196% before ISNs to 3.278% after ISNs (49% increase; 95% confidence interval, 47%-52%). Communication rates increased among all communication subtypes (critical, urgent, routine, and actionable), with the highest increases in urgent (94%) and actionable (75%) findings. Before implementation, radiologists reported spending 39 min on average per day on communications tasks, with only 33% of radiologists indicating that the communication process was efficient. After implementation, radiologists reported mean time savings of 28 min (95% confidence interval, 19.9-35.1), and 82% of radiologists indicated a positive or highly positive view of the ISN service.

CONCLUSIONS

After ISN implementation, communication rates increased and radiologists reported spending less time performing communications. Most radiologists were satisfied with the service.

Blunt splenic injury in adults: Association between volumetric quantitative CT parameters and intervention

BACKGROUND.

Several ordinal grading systems are employed in deciding whether to perform angioembolization or splenectomy following blunt splenic injury. The 2018 AAST Organ Injury Scale (OIS) incorporates vascular lesions but not hemoperitoneum, which is considered in the Thompson classifier. Granular and verifiable quantitative measurements of these features may have a future role in facilitating objective decision-making.

PURPOSE.

To compare performance of CT volumetry-based quantitative modeling to the 1994 and 2018 AAST OIS and Thompson classifier for the following endpoints: decision to perform splenectomy (SPY), and the composite of SPY or angioembolization (AE)

MATERIALS AND METHODS.

Adult BSI patients (age ≥ 18 years) scanned with dual-phase CT prior to intervention at a single level I trauma center from 2017-2019 were included in this retrospective study (n=174). Scoring using 2018 AAST, 1994 AAST, and Thompson systems was performed retrospectively by two radiologists and arbitrated by a third. Endpoints included 1. SPY and 2. The composite of SPY or AE. Logistic regression models were developed from segmented active bleed, contained vascular lesion, splenic parenchymal disruption, and hemoperitoneum volumes. AUCs for ordinal systems and volumetric models were compared.

RESULTS.

Forty-seven BSI patients (27%) underwent SPY, and 87 patients (50%) underwent SPY or AE. Quantitative model AUCs (0.85- SPY, 0.82-composite) were not significantly different from 2018 AAST AUCs (0.81, 0.88, p=0.66, 0.14) for both endpoints, and were significantly improved over Thompson scoring (0.76, p=0.02; 0.77, p=0.04).

CONCLUSION:

Quantitative CT volumetry can be used to model intervention for BSI with accuracy comparable to 2018 AAST scoring and significantly higher than Thompson scoring.

Study Type:

Prognostic

Level of Evidence:

IV

CT volumetry of blunt splenic injury-related features predicts splenectomy and angioembolization in adults and identifies clinically important target features for computer vision and automation research.

Resident pre-dictation of ED case delay signoff of those reports, but overall, residents improve report turnaround time

PURPOSE

In academic radiology departments, attendings and resident are increasingly working together overnight for Emergency Department (ED) radiology coverage. The purpose of this study was to quantify reporting turnaround time for overnight cases read by residents and for overnight shifts with residents on duty.

METHODS

A retrospective study was performed at a hospital system where one overnight attending covers two hospitals with a 2nd/3rd year overnight resident, and a second overnight attending covers two other hospitals 80% of the time independently and 20% of the time with a fourth-year resident. In the first analysis, the median difference in turnaround time, from the time when the case was completed by the technologist to the time the attending finalized it, between cases read independently by attendings and cases pre-dictated by residents was calculated. In the second analysis, the median difference in turnaround time for all cases performed at the second two hospitals was compared on nights when an attending was on duty alone versus nights when a fourth-year resident was also on duty, regardless of if the resident had pre-dictated the case.

RESULTS

For computed tomography (CT), radiographs (XR), and ultrasound (US), there was a significant delay in turnaround time for cases pre-dictated by residents compared to cases read independently by attendings, ranging between 11 and 49 min depending on resident seniority and modality (p ≤ 0.001). For all cases on nights with a 4th year resident working, overall median report turnaround time decreased by 7 min (p < 0.001).

CONCLUSION

Resident pre-dictation causes delay in the finalization of individual CT, US, and XR reports; however, overall, working with residents results in a significant decrease in report turnaround time supporting the belief that overnight resident education does not delay patient care.

Engaging nurse practitioners and physician assistants to improve patient care and drive productivity in a radiology consult practice at a comprehensive cancer center

BACKGROUND

The demand for health care in the United States is increasing because of an aging population and an increase in the number of individuals insured. This has led to requests to revamp the primary care infrastructure fundamentally.

LOCAL PROBLEM

The optimal use of nurse practitioners (NPs) and physician assistants (PAs) is still a subject of debate, but recently, it was reported that for many medical conditions, NP and PA-managed care outcomes are consistent with physician-managed care outcomes.

METHODS

Radiologists' productivity was measured according to relative value units (RVUs)/shift and professional billing changes. Patient care metrics measured were prescribed protocol to patient appointment lead time and number of same-day prescribed imaging protocol changes.

INTERVENTIONS

The focus was on radiologists' productivity and patient care for three months before and three months after integrating NP and PA into our abdominal radiology consult service.

RESULTS

We observed significant increases in the mean RVUs/shift (15.2 ± 0.9 vs. 6.2 ± 1.8; p = .02), studies read per shift (10.1 ± 0.5 vs. 4.4 ± 1.5; p = .003), revenue per shift hour ($756.20 ± 55.40 vs. $335.40 ± 32.60; p = .007), and protocol prescription to patient appointment lead time (39.3 ± 6.7 days vs. 16.3 ± 2.9 days; p = .005) and saw significant decreases in the mean prescribed CT (19.3 ± 0.6 vs. 3.3 ± 0.6; p = .001) and MRI (11.7 ± 0.6 vs. 8.30 ± 0.12; p = .011) same day protocol changes in NP and PA integrated workflow.

CONCLUSIONS

These findings suggest that NP and PA can be effectively integrated into the abdominal radiology consult service, increasing radiologists' productivity and enhancing clinical care.

PAs and NPs improve patient care and productivity in a radiology consult practice

OBJECTIVE

To evaluate the effects on efficiency and patient care of the addition of physician assistants (PAs) and NPs to the abdominal radiology consult service.

METHODS

We obtained radiologist productivity and patient care metrics for 3 months before and 3 months after the integration of PAs and NPs into our consult service.

RESULTS

Integrating PAs and NPs into the workflow led to a significant increase in mean RVUs/shift (15.2 ± 0.9 versus 6.2 ± 1.8; P = .02), number of studies read per shift (10.1 ± 0.5 versus 4.4 ± 1.5; P = .003), revenue per shift hour ($756.20 ± $55.40 versus $335.40 ± $132.60; P = .007), protocol prescription to patient appointment lead time (39.3 ± 6.7 versus 16.3 ± 2.9 days; P = .005), and significant decreases in mean CT (19.3% ± 0.6 versus 3.3% ± 0.6; P = .001) and MRI (11.7% ± 0.6 versus 8.3% ± 0.12; P = .011) same-day protocol changes as patient appointments.

CONCLUSIONS

PAs and NPs can be effectively integrated into abdominal radiology consult service, increasing the productivity of radiologists, and enhancing clinical care.

Performance of Automated RAPID Intracranial Hemorrhage Detection in Real-World Practice: A Single-Institution Experience

BACKGROUND AND PURPOSE

Intracranial hemorrhage (ICH) is a common finding in patients presenting to the emergency department with acute neurological symptoms. Noncontrast head computed tomography (NCCT) is the primary modality for assessment and detection of ICH in the acute setting. RAPID ICH software aims to automatically detect ICH on NCCT and was previously shown to have high accuracy when applied to a curated test data set. Here, we measured the test performance characteristics of RAPID ICH software in detecting ICH on NCCT performed in patients undergoing emergency stroke evaluation at a tertiary academic comprehensive stroke center.

MATERIALS AND METHODS

This retrospective study assessed consecutive patients over a 6-month period who presented with acute neurological symptoms suspicious for stroke and underwent NCCT with RAPID ICH postprocessing. RAPID ICH detection was compared with the interpretation of a reference standard comprising a board-certified or board-eligible neuroradiologist, or in cases of discrepancy, adjudicated by a consensus panel of 3 neuroradiologists. Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of RAPID ICH for ICH detection were determined.

RESULTS

Three hundred seven NCCT scans were included in the study. RAPID ICH correctly identified 34 of 37 cases with ICH and 228 of 270 without ICH. RAPID ICH had a sensitivity of 91.9% (78.1%-98.3%), specificity of 84.4% (79.6%-88.6%), NPV of 98.7% (96.3%-99.6%), PPV of 44.7% (37.6%-52.1%), and overall accuracy of 85.3% (80.9%-89.1%).

CONCLUSIONS

In a real-world scenario, RAPID ICH software demonstrated high NPV but low PPV for the presence of ICH when evaluating possible stroke patients.

A joint convolutional-recurrent neural network with an attention mechanism for detecting intracranial hemorrhage on noncontrast head CT

To investigate the performance of a joint convolutional neural networks-recurrent neural networks (CNN-RNN) using an attention mechanism in identifying and classifying intracranial hemorrhage (ICH) on a large multi-center dataset; to test its performance in a prospective independent sample consisting of consecutive real-world patients. All consecutive patients who underwent emergency non-contrast-enhanced head CT in five different centers were retrospectively gathered. Five neuroradiologists created the ground-truth labels. The development dataset was divided into the training and validation set. After the development phase, we integrated the deep learning model into an independent center’s PACS environment for over six months for assessing the performance in a real clinical setting. Three radiologists created the ground-truth labels of the testing set with a majority voting. A total of 55,179 head CT scans of 48,070 patients, 28,253 men (58.77%), with a mean age of 53.84 ± 17.64 years (range 18–89) were enrolled in the study. The validation sample comprised 5211 head CT scans, with 991 being annotated as ICH-positive. The model's binary accuracy, sensitivity, and specificity on the validation set were 99.41%, 99.70%, and 98.91, respectively. During the prospective implementation, the model yielded an accuracy of 96.02% on 452 head CT scans with an average prediction time of 45 ± 8 s. The joint CNN-RNN model with an attention mechanism yielded excellent diagnostic accuracy in assessing ICH and its subtypes on a large-scale sample. The model was seamlessly integrated into the radiology workflow. Though slightly decreased performance, it provided decisions on the sample of consecutive real-world patients within a minute.

Convolutional neural network performance compared to radiologists in detecting intracranial hemorrhage from brain computed tomography: A systematic review and meta-analysis

PURPOSE

To compare the diagnostic accuracy of convolutional neural networks (CNN) with radiologists as the reference standard in the diagnosis of intracranial hemorrhages (ICH) with non contrast computed tomography of the cerebrum (NCTC).

METHODS

PubMed, Embase, Scopus, and Web of Science were searched for the period from 1 January 2012 to 20 July 2020; eligible studies included patients with and without ICH as the target condition undergoing NCTC, studies had deep learning algorithms based on CNNs and radiologists reports as the minimum reference standard. Pooled sensitivities, specificities and a summary receiver operating characteristics curve (SROC) were employed for meta-analysis.

RESULTS

5,119 records were identified through database searching. Title-screening left 47 studies for full-text assessment and 6 studies for meta-analysis. Comparing the CNN performance to reference standards in the retrospective studies found a pooled sensitivity of 96.00% (95% CI: 93.00% to 97.00%), pooled specificity of 97.00% (95% CI: 90.00% to 99.00%) and SROC of 98.00% (95% CI: 97.00% to 99.00%), and combining retrospective and studies with external datasets found a pooled sensitivity of 95.00% (95% CI: 91.00% to 97.00%), pooled specificity of 96.00% (95% CI: 91.00% to 98.00%) and a pooled SROC of 98.00% (95% CI: 97.00% to 99.00%).

CONCLUSION

This review found the diagnostic performance of CNNs to be equivalent to that of radiologists for retrospective studies. Out-of-sample external validation studies pooled with retrospective studies found CNN performance to be slightly worse. There is a critical need for studies with a robust reference standard and external data-set validation.

A real-world demonstration of machine learning generalizability in the detection of intracranial hemorrhage on head computerized tomography

Machine learning (ML) holds great promise in transforming healthcare. While published studies have shown the utility of ML models in interpreting medical imaging examinations, these are often evaluated under laboratory settings. The importance of real world evaluation is best illustrated by case studies that have documented successes and failures in the translation of these models into clinical environments. A key prerequisite for the clinical adoption of these technologies is demonstrating generalizable ML model performance under real world circumstances. The purpose of this study was to demonstrate that ML model generalizability is achievable in medical imaging with the detection of intracranial hemorrhage (ICH) on non-contrast computed tomography (CT) scans serving as the use case. An ML model was trained using 21,784 scans from the RSNA Intracranial Hemorrhage CT dataset while generalizability was evaluated using an external validation dataset obtained from our busy trauma and neurosurgical center. This real world external validation dataset consisted of every unenhanced head CT scan (n = 5965) performed in our emergency department in 2019 without exclusion. The model demonstrated an AUC of 98.4%, sensitivity of 98.8%, and specificity of 98.0%, on the test dataset. On external validation, the model demonstrated an AUC of 95.4%, sensitivity of 91.3%, and specificity of 94.1%. Evaluating the ML model using a real world external validation dataset that is temporally and geographically distinct from the training dataset indicates that ML generalizability is achievable in medical imaging applications.

Implementation of Machine Learning Software on the Radiology Worklist Decreases Scan View Delay for the Detection of Intracranial Hemorrhage on CT

BACKGROUND AND PURPOSE

Prompt identification of acute intracranial hemorrhage on CT is important. The goal of this study was to assess the impact of artificial intelligence software for prioritizing positive cases.

MATERIALS AND METHODS

Cases analyzed by Aidoc (Tel Aviv, Israel) software for triaging acute intracranial hemorrhage cases on non-contrast head CT were retrospectively reviewed. The scan view delay time was calculated as the difference between the time the study was completed on PACS and the time the study was first opened by a radiologist. The scan view delay was stratified by scan location, including emergency, inpatient, and outpatient. The scan view delay times for cases flagged as positive by the software were compared to those that were not flagged.

RESULTS

A total of 8723 scans were assessed by the software, including 6894 cases that were not flagged and 1829 cases that were flagged as positive. Although there was no statistically significant difference in the scan view time for emergency cases, there was a significantly lower scan view time for positive outpatient and inpatient cases flagged by the software versus negative cases, with a reduction of 604 min on average, 90% in the scan view delay (p-value < 0.0001) for outpatients, and a reduction of 38 min on average, and 10% in the scan view delay (p-value <= 0.01) for inpatients.

CONCLUSION

The use of artificial intelligence triage software for acute intracranial hemorrhage on head CT scans is associated with a significantly shorter scan view delay for cases flagged as positive than cases not flagged among outpatients and inpatients at an academic medical center.

Automated Cerebral Hemorrhage Detection Using RAPID

BACKGROUND AND PURPOSE

Intracranial hemorrhage (ICH) is an important event that is diagnosed on head NCCT. Increased NCCT utilization in busy hospitals may limit timely identification of ICH. RAPID ICH is an automated hybrid 2D-3D convolutional neural network application designed to detect ICH that may allow for expedited ICH diagnosis. We determined the accuracy of RAPID ICH for ICH detection and ICH volumetric quantification on NCCT.

MATERIALS AND METHODS

NCCT scans were evaluated for ICH by RAPID ICH. Consensus detection of ICH by 3 neuroradiology experts was used as the criterion standard for RAPID ICH comparison. ICH volume was also automatically determined by RAPID ICH in patients with intraparenchymal or intraventricular hemorrhage and compared with manually segmented ICH volumes by a single neuroradiology expert. ICH detection accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and positive and negative likelihood ratios by RAPID ICH were determined.

RESULTS

We included 308 studies. RAPID ICH correctly identified 151/158 ICH cases and 143/150 ICH-negative cases, which resulted in high sensitivity (0.956, CI: 0.911-0.978), specificity (0.953, CI: 0.907-0.977), positive predictive value (0.956, CI: 0.911-0.978), and negative predictive value (0.953, CI: 0.907-0.977) for ICH detection. The positive likelihood ratio (20.479, CI 9.928-42.245) and negative likelihood ratio (0.046, CI 0.023-0.096) for ICH detection were similarly favorable. RAPID ICH volumetric quantification for intraparenchymal and intraventricular hemorrhages strongly correlated with expert manual segmentation (correlation coefficient r = 0.983); the median absolute error was 3 mL.

CONCLUSIONS

RAPID ICH is highly accurate in the detection of ICH and in the volumetric quantification of intraparenchymal and intraventricular hemorrhages.

Optimising after-hours workflow of computed tomography orders in the emergency department

Ordering and protocolling CT scans after-hours from the emergency department (ED) at our institution previously required discussion between the ED physician and radiology resident, which led to workflow inefficiency. Our intervention consisted of creating an electronic list of CT requests that radiology residents would monitor. Radiology protocolled straightforward requests and contacted the ordering physician for more details when required. We aimed to improve workflow efficiency, increase provider satisfaction and reduce CT turnaround time without significantly affecting CT utilisation. Plan-do-study-act cycles were used to plan and evaluate the intervention. The intervention was initiated on weekday evenings and then expanded to weekend hours after an interim analysis. Qualitative outcomes were measured via electronic survey, and quantitative outcomes were collected from administrative data and analysed via control charts and other statistical methods. Survey response was high from ED physicians (76%, n=82/108) and radiology residents (79%, n=30/38). After the intervention, the majority of ED staff and radiology residents perceived improved workflow efficiency (96.3%, 73.3%), radiology residents noted a subjective decrease in disruptions (83.3%) and most ED staff felt that scans were performed more quickly (84.1%). Radiology residents received fewer pages per shift, adjusted for scan volume. There was a reduction in time from order entry to protocol on weekday shifts only, with no statistically significant effect on time from order entry to scan. Segmented regression analysis demonstrated a background increase in utilisation over time (0.7–2.0 CT/100 ED visits/year, p<0.0005), but the intervention itself did not contribute to an overall increase in CT utilisation. In conclusion, our intervention led to improved perceived workflow efficiency and reduced pages. Scans were protocoled more quickly on weekdays, but turnaround times were otherwise not significantly affected by the intervention. Background CT utilisation increased over time, but this increase was not attributable to our intervention.

Quantifying disruption of workflow by phone calls to the neuroradiology reading room

Introduction

The purpose of this study was to understand the source and the reason for the phone calls to our neuroradiology suit and to quantify the size of the problem in terms of duration of individual and aggregated calls.

Materials and methods

Observation of the neuroradiology reading room for the entire duration of the working hours over three non-consecutive days was performed, and included telephone calls start time, end time and calls duration for incoming telephone calls. After each phone call the recipients were queried on the details of the phone call; the origin of the call, the reason for the call and the response.

Results

The average total number of minutes (min) spent on the phone each day was 64 min per working day with a total of 39 phone calls per day and 4.4 per hour on average. The trainees answered 71% of the phone calls with additional intervention by attending in 13% of phone calls. The most common source of phone calls was from either the MRI/CT technicians (48%), followed by providers (20%) and returning pages (18%).

Conclusion

Cumulative time spent on the phone by neuroradiologists in the reading room ended up in more than an hour per working day, while trainees were taking the majority of phone calls. Most phone calls originated from technicians, hence, requiring specific solutions to mitigate this kind of interruption.

Whole-Body CT in Patients with Multiple Traumas: Factors Leading to Missed Injury

Purpose To determine radiologic and clinical markers predictive of missed injuries at early whole-body CT image interpretation. Materials and Methods For this retrospective study, 2354 consecutive whole-body CT examinations were performed in patients with multiple traumas from 26 hospitals interpreted at a teleradiology center study during on-call period from February 2011 to September 2016. All whole-body CT images were interpreted by the on-call radiologist and reviewed within 12-48 hours by another radiologist to detect missed injury as the standard of reference. The first and review reports of all examinations were retrospectively reviewed. Univariable and multivariable logistic regression with a stepwise selection method were performed to identify clinical and radiologic predictors of missed injury. Results This study included 639 women (27.1%) and 1715 men (72.8%). The median age of men, women, and the entire population was 34 years (age range, 1-96 years). On a per-scan basis, there were 304 (12.9%) missed injuries and 59 (2.5%) were clinically significant. On a per-injury basis, the missed injury rate was 530 of 5979 (8.8%). More than two injured body parts (odds ratio, 1.4 [95% confidence interval: 1.1, 1.8]; P = .01), patient age older than 30 years (odds ratio, 2.8 [95% confidence interval: 2.1, 3.8]; P < .001), and an initial clinical severity class of 1 (odds ratio, 1.9 [95% confidence interval: 1.3, 2.8]; P < .001) were independent predictive factors of missed injury. Conclusion Multiple traumas with more than two injured body parts, age older than 30 years, or an initial clinical severity class of 1 were associated with missed injury at whole-body CT. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Novelline in this issue.

Hybrid 3D/2D Convolutional Neural Network for Hemorrhage Evaluation on Head CT

BACKGROUND AND PURPOSE

Convolutional neural networks are a powerful technology for image recognition. This study evaluates a convolutional neural network optimized for the detection and quantification of intraparenchymal, epidural/subdural, and subarachnoid hemorrhages on noncontrast CT.

MATERIALS AND METHODS

This study was performed in 2 phases. First, a training cohort of all NCCTs acquired at a single institution between January 1, 2017, and July 31, 2017, was used to develop and cross-validate a custom hybrid 3D/2D mask ROI-based convolutional neural network architecture for hemorrhage evaluation. Second, the trained network was applied prospectively to all NCCTs ordered from the emergency department between February 1, 2018, and February 28, 2018, in an automated inference pipeline. Hemorrhage-detection accuracy, area under the curve, sensitivity, specificity, positive predictive value, and negative predictive value were assessed for full and balanced datasets and were further stratified by hemorrhage type and size. Quantification was assessed by the Dice score coefficient and the Pearson correlation.

RESULTS

A 10,159-examination training cohort (512,598 images; 901/8.1% hemorrhages) and an 862-examination test cohort (23,668 images; 82/12% hemorrhages) were used in this study. Accuracy, area under the curve, sensitivity, specificity, positive predictive value, and negative-predictive value for hemorrhage detection were 0.975, 0.983, 0.971, 0.975, 0.793, and 0.997 on training cohort cross-validation and 0.970, 0.981, 0.951, 0.973, 0.829, and 0.993 for the prospective test set. Dice scores for intraparenchymal hemorrhage, epidural/subdural hemorrhage, and SAH were 0.931, 0.863, and 0.772, respectively.

CONCLUSIONS

A customized deep learning tool is accurate in the detection and quantification of hemorrhage on NCCT. Demonstrated high performance on prospective NCCTs ordered from the emergency department suggests the clinical viability of the proposed deep learning tool.