Workflow Dynamics and the Imaging Value Chain: Quantifying the Effect of Designating a Nonimage-Interpretive Task Workflow

Shift Volume Directly Impacts Neuroradiology Error Rate at a Large Academic Medical Center: The Case for Volume Limits

BACKGROUND AND PURPOSE

Unlike in Europe and Japan, guidelines or recommendations from specialized radiological societies on workflow management and adaptive intervention to reduce error rates are currently lacking in the United States. This study of neuroradiologic reads at a large US academic medical center, which may hopefully contribute to this discussion, found a direct relationship between error rate and shift volume.

MATERIALS AND METHODS

CT and MR imaging reports from our institution's Neuroradiology Quality Assurance database (years 2014-2020) were searched for attending physician errors. Data were collected on shift volume specific error rates per 1000 interpreted studies and RADPEER scores. Optimal cutoff points for 2, 3 and 4 groups of shift volumes were computed along with subgroups' error rates.

RESULTS

A total of 643 errors were found, 91.7% of which were clinically significant (RADPEER 2b, 3b). The overall error rate (errors/1000 examinations) was 2.36. The best single shift volume cutoff point generated 2 groups: ≤ 26 studies (error rate 1.59) and > 26 studies (2.58; OR: 1.63, P < .001). The best 2 shift volume cutoff points generated 3 shift volume groups: ≤ 19 (1.34), 20-28 (1.88; OR: 1.4, P = .1) and ≥ 29 (2.6; OR: 1.94, P < .001). The best 3 shift volume cutoff points generated 4 groups: ≤ 24 (1.59), 25-66 (2.44; OR: 1.54, P < .001), 67-90 (3.03; OR: 1.91, P < .001), and ≥ 91 (2.07; OR: 1.30, P = .25). The group with shift volume ≥ 91 had a limited sample size.

CONCLUSIONS

Lower shift volumes yielded significantly lower error rates. The lowest error rates were observed with shift volumes that were limited to 19-26 studies. Error rates at shift volumes between 67-90 studies were 226% higher, compared with the error rate at shift volumes of ≤ 19 studies.

Reading room assistants to reduce workload and interruptions of radiology residents during on-call hours: Initial evaluation

PURPOSE

To determine how much timesaving and reduction of interruptions reading room assistants can provide by taking over non-image interpretation tasks (NITs) from radiology residents during on-call hours.

METHODS

Reading room assistants are medical students who were trained to take over NITs from radiology residents (e.g. answering telephone calls, administrative tasks and logistics) to reduce residents' workload during on-call hours. Reading room assistants' and residents' activities were tracked during 6 weekend dayshifts in a tertiary care academic center (with more than 2.5 million inhabitants in its catchment area) between 10 a.m. and 5p.m. (7-hour shift, 420 min), and time spent on each activity was recorded.

RESULTS

Reading room assistants spent the most time on the following timesaving activities for residents: answering incoming (41 min, 19%) and outgoing telephone calls (35 min, 16%), ultrasound machine related activities (19 min, 9%) and paramedical assistance such as supporting residents during ultrasound guided procedures and with patients (17 min, 8%). Reading room assistants saved 132 min of residents' time by taking over NITs while also spending circa 31 min consulting the resident, resulting in a net timesaving of 101 min (24%) during a 7-hour shift. The reading room assistants also prevented residents from being interrupted, at a mean of 18 times during the 7-hour shift.

CONCLUSION

This study shows that the implementation of reading room assistants to radiology on-call hours could provide a timesaving for residents and also reduce the number of times residents are being interrupted during their work.

Reading Room Interruptions are Less Disruptive When Using Asynchronous Communication Methods

Radiologist interruptions, though often necessary, can be disruptive. Prior literature has shown interruptions to be frequent, occurring during cases, and predominantly through synchronous communication methods such as phone or in person causing significant disengagement from the study being read. Asynchronous communication methods are now more widely available in hospital systems such as ours. Considering the increasing use of asynchronous communication methods, we conducted an observational study to understand the evolving nature of radiology interruptions. We hypothesize that compared to interruptions occurring through synchronous methods, interruptions via asynchronous methods reduce the disruptive nature of interruptions by occurring between cases, being shorter, and less severe. During standard weekday hours, 30 radiologists (14 attendings, 12 residents, and 4 fellows) were directly observed for approximately 90-min sessions across three different reading rooms (body, neuroradiology, general). The frequency of interruptions was documented including characteristics such as timing, severity, method, and length. Two hundred twenty-five interruptions (43 Teams, 47 phone, 89 in-person, 46 other) occurred, averaging 2 min and 5 s with 5.2 interruptions per hour. Microsoft Teams interruptions averaged 1 min 12 s with only 60.5% during cases. In-person interruptions averaged 2 min 12 s with 82% during cases. Phone interruptions averaged 2 min and 48 s with 97.9% during cases. A substantial portion of reading room interruptions occur via predominantly asynchronous communication tools, a new development compared to prior literature. Interruptions via predominantly asynchronous communications tools are shorter and less likely to occur during cases. In our practice, we are developing tools and mechanisms to promote asynchronous communication to harness these benefits.

"Are They Listening, and Do They Find It Useful?"-Evaluation of Mid-Rotation Formative Subjective and Objective Feedback to Radiology Trainees

BACKGROUND

Residents commonly receive only end-of-rotation evaluations and thus are often unaware of their progress during a rotation. In 2021, our neuroradiology section instituted mid-rotation feedback in which rotating residents received formative subjective and objective feedback. The purpose of this study was to describe our feedback method and to evaluate if residents found it helpful.

METHODS

Radiology residents rotate 3-4 times on the neuroradiology service for 1-month blocks. At the midpoint of the rotation (2 weeks), 7-10 neuroradiology attendings discussed the rotating residents' subjective performance. One attending was tasked with facilitating this discussion and taking notes. Objective metrics were obtained from our dictation software. Compiled feedback was relayed to residents via email. A 16-question anonymous survey was sent to 39 radiology residents (R1-R4) to evaluate the perceived value of mid-rotation feedback. Odds ratios and 95% confidence intervals were computed using logistic regression.

RESULTS

Sixty-nine percent (27/39) of residents responded to the survey; 92.6% (25/27) of residents reported receiving mid-rotation feedback in ≥50% of neuroradiology rotations; 92.3% (24/26) of residents found the subjective feedback helpful; 88.4% (23/26) of residents reported modifying their performance as suggested (100% R1-R2 vs 70% R3-R4; OR: 15.4 CI:1.26, >30.0);59.1% (13/22) of residents found the objective metrics helpful (75% R1-R2 vs 40% R3-R4; OR: 3.92 CI:0.74, 24.39) and 68.2% (15/22) stated they modified their performance based on these metrics (83.3% R1-R2 vs 50.0% R3-R4; OR:4.2 CI:0.73, 30.55); and 84.6% (22/26) of residents stated that mid-rotation subjective feedback and 45.5% (10/22) stated that mid-rotation objective feedback should be implemented in other sections.

CONCLUSIONS

Majority of residents found mid-rotation feedback to be helpful in informing them about their progress and areas for improvement in the neuroradiology rotation, more so for subjective feedback than objective feedback. The majority of residents stated all rotations should provide mid-rotation subjective feedback.

Radiologist Worklist Reprioritization Using Artificial Intelligence: Impact on Report Turnaround Times for CTPA Examinations Positive for Acute Pulmonary Embolism

BACKGROUND. In patients with acute pulmonary embolism (PE), timely intervention (e.g., initiation of anticoagulation) is critical for optimizing clinical outcomes. OBJECTIVE. The purpose of this study was to evaluate the effect of artificial intelligence (AI)-based radiologist worklist reprioritization on report turnaround times for pulmonary CTA (CTPA) examinations positive for acute PE. METHODS. This retrospective single-center study included patients who underwent CTPA before (October 1, 2018-March 31, 2019 [pre-AI period]) and after (October 1, 2019-March 31, 2020 [post-AI period]) implementation of an AI tool that reprioritized CTPA examinations to the top of radiologists' reading worklists if acute PE was detected. EMR and dictation system timestamps were used to determine the wait time (time from examination completion to report initiation), read time (time from report initiation to report availability), and report turnaround time (sum of wait and read times) for the examinations. Times for reports positive for PE, with final radiology reports as reference, were compared between periods. RESULTS. The study included 2501 examinations of 2197 patients (1307 women, 890 men; mean age, 57.4 ± 17.0 [SD] years), including 1335 examinations from the pre-AI period and 1166 from the post-AI period. The frequency of acute PE, based on radiology reports, was 15.1% (201/1335) during the pre-AI period and 12.3% (144/1166) during the post-AI period. During the post-AI period, the AI tool reprioritized 12.7% (148/1166) of examinations. For PE-positive examinations, the post-AI period, compared with the pre-AI period, had significantly shorter mean report turnaround time (47.6 vs 59.9 minutes; mean difference, 12.3 minutes [95% CI, 0.6-26.0 minutes]) and mean wait time (21.4 vs 33.4 minutes; mean difference, 12.0 minutes [95% CI, 0.9-25.3 minutes]) but no significant difference in mean read time (26.3 vs 26.5 minutes; mean difference, 0.2 minutes [95% CI, -2.8 to 3.2 minutes]). During regular operational hours, wait time was significantly shorter in the post-AI than in the pre-AI period for routine-priority examinations (15.3 vs 43.7 minutes; mean difference, 28.4 minutes [95% CI, 2.2-64.7 minutes]) but not for stat- or urgent-priority examinations. CONCLUSION. AI-driven worklist reprioritization yielded reductions in report turnaround time and wait time for PE-positive CTPA examinations. CLINICAL IMPACT. By assisting radiologists in providing rapid diagnoses, the AI tool has potential for enabling earlier interventions for acute PE.

Factors Associated With Neuroradiologic Diagnostic Errors at a Large Tertiary-Care Academic Medical Center: A Case-Control Study

BACKGROUND. Numerous studies have explored factors associated with diagnostic errors in neuroradiology; however, large-scale multivariable analyses are lacking. OBJECTIVE. The purpose of this study was to evaluate associations of interpretation time, shift volume, care setting, day of week, and trainee participation with diagnostic errors by neuroradiologists at a large academic medical center. METHODS. This retrospective case-control study using a large tertiary-care academic medical center's neuroradiology quality assurance database evaluated CT and MRI examinations for which neuroradiologists had assigned RADPEER scores. The database was searched from January 2014 through March 2020 for examinations without (RADPEER score of 1) or with (RADPEER scores of 2a, 2b, 3a, 3b, or 4) diagnostic error. For each examination with error, two examinations without error were randomly selected (unless only one examination could be identified) and matched by interpreting radiologist and examination type to form case and control groups. Marginal mixed-effects logistic regression models were used to assess associations of diagnostic error with interpretation time (number of minutes since the immediately preceding report's completion), shift volume (number of examinations interpreted during the shift), emergency/inpatient setting, weekend interpretation, and trainee participation in interpretation. RESULTS. The case group included 564 examinations in 564 patients (mean age, 50.0 ± 25.0 [SD] years; 309 men, 255 women); the control group included 1019 examinations in 1019 patients (mean age, 52.5 ± 23.2 years; 540 men, 479 women). In the case versus control group, mean interpretation time was 16.3 ± 17.2 [SD] minutes versus 14.8 ± 16.7 minutes; mean shift volume was 50.0 ± 22.1 [SD] examinations versus 45.4 ± 22.9 examinations. In univariable models, diagnostic error was associated with shift volume (OR = 1.22, p < .001) and weekend interpretation (OR = 1.60, p < .001) but not interpretation time, emergency/inpatient setting, or trainee participation (p > .05). However, in multivariable models, diagnostic error was independently associated with interpretation time (OR = 1.18, p = .003), shift volume (OR = 1.27, p < .001), and weekend interpretation (OR = 1.69, p = .02). In subanalysis, diagnostic error showed independent associations on weekdays with interpretation time (OR = 1.18, p = .003) and shift volume (OR = 1.27, p < .001); such associations were not observed on weekends (interpretation time: p = .62; shift volume: p = .58). CONCLUSION. Diagnostic errors in neuroradiology were associated with longer interpretation times, higher shift volumes, and weekend interpretation. CLINICAL IMPACT. These findings should be considered when designing work-flow-related interventions seeking to reduce neuroradiology interpretation errors.

Why, How Often, and What Happens When We Fail: A Retrospective Analysis of Failed Fluoroscopically Guided Lumbar Punctures

BACKGROUND AND PURPOSE

Important information regarding fluoroscopically guided lumbar puncture (FGLP) performance and referrals is lacking. The purpose of our study was to elucidate the success rate for initial FGLP attempts and re-attempts, reasons for unsuccessful FGLPs, and the relationship between clinical indications and whether patients will undergo a fluoroscopically guided re-attempt, among others.

MATERIALS AND METHODS

This retrospective study analyzed failed FGLP attempts in hospitalized adult patients at an academic hospital between June 2016 and March 2022. Unsuccessful FGLPs were labeled as insufficient CSF egress. FGLP reports and patients' clinical charts were analyzed for pertinent information such as clinical indication, reason for failure, whether patients received IV fluid before fluoroscopically guided spinal puncture attempt, and which patients returned for another FGLP attempt. Patients' ages and sex were analyzed using descriptive statistics. The OR was used to investigate the relationship between the clinical indications to perform FGLP and whether patients returned for a re-attempt.

RESULTS

Sixty-three of 1389 (4.5%) patients (median age, 62 years) had failed the initial FGLPs administered by 39 trainees. Twenty-eight of 63 (44.4%) patients (median age, 64 years) underwent a re-attempt within a median of 2 days after the first attempt, and 27/28 (96.4%) re-attempts were successful. A dry tap, no egress of CSF was the top reason (58.7%) for failed FGLP, and 12/13 of patients had a successful FGLP after IV hydration. Twenty-seven of 63 (43%) patients did not undergo a repeat attempt, and 100% were subsequently discharged from the hospital. There was no difference (P > .05) in the likelihood of patients returning for a repeat FGLP based on the clinical indications.

CONCLUSIONS

Initial and repeat FGLPs have very high success rates. No difference exists in the likelihood of patients returning for a re-attempt based on clinical indication.

Natural Language Processing for Imaging Protocol Assignment: Machine Learning for Multiclass Classification of Abdominal CT Protocols Using Indication Text Data

A correct protocol assignment is critical to high-quality imaging examinations, and its automation can be amenable to natural language processing (NLP). Assigning protocols for abdominal imaging CT scans is particularly challenging given the multiple organ specific indications and parameters. We compared conventional machine learning, deep learning, and automated machine learning builder workflows for this multiclass text classification task. A total of 94,501 CT studies performed over 4 years and their assigned protocols were obtained. Text data associated with each study including the ordering provider generated free text study indication and ICD codes were used for NLP analysis and protocol class prediction. The data was classified into one of 11 abdominal CT protocol classes before and after augmentations used to account for imbalances in the class sample sizes. Four machine learning (ML) algorithms, one deep learning algorithm, and an automated machine learning (AutoML) builder were used for the multilabel classification task: Random Forest (RF), Tree Ensemble (TE), Gradient Boosted Tree (GBT), multi-layer perceptron (MLP), Universal Language Model Fine-tuning (ULMFiT), and Google’s AutoML builder (Alphabet, Inc., Mountain View, CA), respectively. On the unbalanced dataset, the manually coded algorithms all performed similarly with F1 scores of 0.811 for RF, 0.813 for TE, 0.813 for GBT, 0.828 for MLP, and 0.847 for ULMFiT. The AutoML builder performed better with a F1 score of 0.854. On the balanced dataset, the tree ensemble machine learning algorithm performed the best with an F1 score of 0.803 and a Cohen’s kappa of 0.612. AutoML methods took a longer time for completion of NLP model training and evaluation, 4 h and 45 min compared to an average of 51 min for manual methods. Machine learning and natural language processing can be used for the complex multiclass classification task of abdominal imaging CT scan protocol assignment.

A Mixed-Methods Systematic Review of the Effectiveness and Experiences of Quality Improvement Interventions in Radiology

OBJECTIVE

This study aimed to compile and synthesize evidence regarding the effectiveness of quality improvement interventions in radiology and the experiences and perspectives of staff and patients.

METHODS

Databases searched for both published and unpublished studies were as follows: EMBASE, MEDLINE, CINAHL, Joanna Briggs Institute, Cochrane Central Register of Controlled Trials, PsycINFO, Scopus, Web of Science, Mednar, Trove, Google Gray, OCLC WorldCat, and Dissertations and Theses. This review included both qualitative and quantitative studies of patients undergoing radiological examinations and/or medical imaging health care professionals; a broad range of quality improvement interventions including introduction of health information technology, effects of training and education, improved reporting, safety programs, and medical devices; the experiences and perspectives of staff and patients; context of radiological setting; a broad range of outcomes including patient safety; and a result-based convergent synthesis design.

RESULTS

Eighteen studies were selected from 4846 identified by a systematic literature search. Five groups of interventions were identified: health information technology (n = 6), training and education (n = 6), immediate and critical reporting (n = 3), safety programs (n = 2), and the introduction of mobile radiography (n = 1), with demonstrated improvements in outcomes, such as improved operational and workflow efficiency, report turnaround time, and teamwork and communication.

CONCLUSIONS

The findings were constrained by the limited range of interventions and outcome measures. Further research should be conducted with study designs that might produce findings that are more generalizable, examine the other dimensions of quality, and address the issues of cost and risk versus benefit.

Diagnostic Errors in Neuroradiology: A Message to Emergency Radiologists and Trainees

Diagnostic errors in neuroradiology are inevitable, yet potentially avoidable. Through extensive literature search, we present an up-to-date review of the psychology of human decision making and how such complex process can lead to radiologic errors. Our focus is on neuroradiology, so we augmented our review with multiple explanatory figures to show how different errors can reflect on real-life clinical practice. We propose a new thematic categorization of perceptual and cognitive biases in this article to simplify message delivery to our target audience: emergency/general radiologists and trainees. Additionally, we highlight individual and organizational remedy strategies to decrease error rate and potential harm.

Bias in Radiology: The How and Why of Misses and Misinterpretations

Medical errors are a leading cause of morbidity and mortality in the medical field and are substantial contributors to medical costs. Radiologists play an integral role in the diagnosis and care of patients and, given that those in this field interpret millions of examinations annually, may therefore contribute to diagnostic errors. Errors can be categorized as a "miss" when a primary or critical finding is not observed or as a "misinterpretation" when errors in interpretation lead to an incorrect diagnosis. In this article, the authors describe the cognitive causes of such errors in diagnostic medicine, specifically in radiology. Recognizing the cognitive processes that radiologists use while interpreting images should improve one's awareness of the inherent biases that can impact decision making. The authors review the common biases that impact clinical decisions, as well as strategies to counteract or minimize the potential for misdiagnosis. System-level processes that can be implemented to minimize cognitive errors are reviewed, as well as ways to implement personal changes to minimize cognitive errors in daily practice. ^©RSNA, 2017.