Risk Factors for Perceptual-versus-Interpretative Errors in Diagnostic Neuroradiology

A deep learning-based automated diagnosis system for SPECT myocardial perfusion imaging

Images obtained from single-photon emission computed tomography for myocardial perfusion imaging (MPI SPECT) contain noises and artifacts, making cardiovascular disease diagnosis difficult. We developed a deep learning-based diagnosis support system using MPI SPECT images. Single-center datasets of MPI SPECT images (n = 5443) were obtained and labeled as healthy or coronary artery disease based on diagnosis reports. Three axes of four-dimensional datasets, resting, and stress conditions of three-dimensional reconstruction data, were reconstructed, and an AI model was trained to classify them. The trained convolutional neural network showed high performance [area under the curve (AUC) of the ROC curve: approximately 0.91; area under the recall precision curve: 0.87]. Additionally, using unsupervised learning and the Grad-CAM method, diseased lesions were successfully visualized. The AI-based automated diagnosis system had the highest performance (88%), followed by cardiologists with AI-guided diagnosis (80%) and cardiologists alone (65%). Furthermore, diagnosis time was shorter for AI-guided diagnosis (12 min) than for cardiologists alone (31 min). Our high-quality deep learning-based diagnosis support system may benefit cardiologists by improving diagnostic accuracy and reducing working hours.

Comparing the Diagnostic Performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and Radiologists in Challenging Neuroradiology Cases

PURPOSE

To compare the diagnostic performance among Generative Pre-trained Transformer (GPT)-4-based ChatGPT, GPT‑4 with vision (GPT-4V) based ChatGPT, and radiologists in challenging neuroradiology cases.

METHODS

We collected 32 consecutive "Freiburg Neuropathology Case Conference" cases from the journal Clinical Neuroradiology between March 2016 and December 2023. We input the medical history and imaging findings into GPT-4-based ChatGPT and the medical history and images into GPT-4V-based ChatGPT, then both generated a diagnosis for each case. Six radiologists (three radiology residents and three board-certified radiologists) independently reviewed all cases and provided diagnoses. ChatGPT and radiologists' diagnostic accuracy rates were evaluated based on the published ground truth. Chi-square tests were performed to compare the diagnostic accuracy of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists.

RESULTS

GPT‑4 and GPT-4V-based ChatGPTs achieved accuracy rates of 22% (7/32) and 16% (5/32), respectively. Radiologists achieved the following accuracy rates: three radiology residents 28% (9/32), 31% (10/32), and 28% (9/32); and three board-certified radiologists 38% (12/32), 47% (15/32), and 44% (14/32). GPT-4-based ChatGPT's diagnostic accuracy was lower than each radiologist, although not significantly (all p > 0.07). GPT-4V-based ChatGPT's diagnostic accuracy was also lower than each radiologist and significantly lower than two board-certified radiologists (p = 0.02 and 0.03) (not significant for radiology residents and one board-certified radiologist [all p > 0.09]).

CONCLUSION

While GPT-4-based ChatGPT demonstrated relatively higher diagnostic performance than GPT-4V-based ChatGPT, the diagnostic performance of GPT‑4 and GPT-4V-based ChatGPTs did not reach the performance level of either radiology residents or board-certified radiologists in challenging neuroradiology cases.

Accuracy of ChatGPT generated diagnosis from patient's medical history and imaging findings in neuroradiology cases

PURPOSE

The noteworthy performance of Chat Generative Pre-trained Transformer (ChatGPT), an artificial intelligence text generation model based on the GPT-4 architecture, has been demonstrated in various fields; however, its potential applications in neuroradiology remain unexplored. This study aimed to evaluate the diagnostic performance of GPT-4 based ChatGPT in neuroradiology.

METHODS

We collected 100 consecutive "Case of the Week" cases from the American Journal of Neuroradiology between October 2021 and September 2023. ChatGPT generated a diagnosis from patient's medical history and imaging findings for each case. Then the diagnostic accuracy rate was determined using the published ground truth. Each case was categorized by anatomical location (brain, spine, and head & neck), and brain cases were further divided into central nervous system (CNS) tumor and non-CNS tumor groups. Fisher's exact test was conducted to compare the accuracy rates among the three anatomical locations, as well as between the CNS tumor and non-CNS tumor groups.

RESULTS

ChatGPT achieved a diagnostic accuracy rate of 50% (50/100 cases). There were no significant differences between the accuracy rates of the three anatomical locations (p = 0.89). The accuracy rate was significantly lower for the CNS tumor group compared to the non-CNS tumor group in the brain cases (16% [3/19] vs. 62% [36/58], p < 0.001).

CONCLUSION

This study demonstrated the diagnostic performance of ChatGPT in neuroradiology. ChatGPT's diagnostic accuracy varied depending on disease etiologies, and its diagnostic accuracy was significantly lower in CNS tumors compared to non-CNS tumors.

Analysis of perceptual errors in skull-base pathology

PURPOSE

Medical errors result in significant mortality and morbidity. The purpose of this study is to analyze skull-base errors at a single tertiary institution, identify common anatomic sites of errors, and offer strategies to reduce errors in this region.

METHODS

A Neuroradiology Quality Assurance Database of radiologic errors was searched for attending physician computer tomography and magnetic resonance imaging errors in skull-base pathology from 2014 to 2020. Data were limited to CT and MRI reports. Errors were separated into four subcategories (tumor, trauma, vascular, and congenital) and further divided by relevant anatomic site.

RESULTS

A total of 90 skull-based errors were identified. Most errors were perceptual (87%), with common study types including MRI Brain (39%) and CT Head (24%). Most common errors were tumors (55%), followed by trauma (24%), vascular (10%), and congenital (7%). Six anatomic sites were identified and encompassed over half of errors (58%): sella, occipital bone, cerebellopontine angle/internal auditory canal (CPA/IAC), foramen magnum and clivus, cavernous sinus, and dural venous sinus.

SUMMARY

Most of the skull-base errors were perceptual. Placing a strong emphasis on both the pathology and closely examining its critical anatomic site (sella, occipital bone, CPA/IAC, foramen magnum and clivus, cavernous sinus, and dural venous sinus) could potentially reduce up to 60% of errors in these regions.

Factors Associated With Neuroradiologic Diagnostic Errors at a Large Tertiary-Care Academic Medical Center: A Case-Control Study

BACKGROUND. Numerous studies have explored factors associated with diagnostic errors in neuroradiology; however, large-scale multivariable analyses are lacking. OBJECTIVE. The purpose of this study was to evaluate associations of interpretation time, shift volume, care setting, day of week, and trainee participation with diagnostic errors by neuroradiologists at a large academic medical center. METHODS. This retrospective case-control study using a large tertiary-care academic medical center's neuroradiology quality assurance database evaluated CT and MRI examinations for which neuroradiologists had assigned RADPEER scores. The database was searched from January 2014 through March 2020 for examinations without (RADPEER score of 1) or with (RADPEER scores of 2a, 2b, 3a, 3b, or 4) diagnostic error. For each examination with error, two examinations without error were randomly selected (unless only one examination could be identified) and matched by interpreting radiologist and examination type to form case and control groups. Marginal mixed-effects logistic regression models were used to assess associations of diagnostic error with interpretation time (number of minutes since the immediately preceding report's completion), shift volume (number of examinations interpreted during the shift), emergency/inpatient setting, weekend interpretation, and trainee participation in interpretation. RESULTS. The case group included 564 examinations in 564 patients (mean age, 50.0 ± 25.0 [SD] years; 309 men, 255 women); the control group included 1019 examinations in 1019 patients (mean age, 52.5 ± 23.2 years; 540 men, 479 women). In the case versus control group, mean interpretation time was 16.3 ± 17.2 [SD] minutes versus 14.8 ± 16.7 minutes; mean shift volume was 50.0 ± 22.1 [SD] examinations versus 45.4 ± 22.9 examinations. In univariable models, diagnostic error was associated with shift volume (OR = 1.22, p < .001) and weekend interpretation (OR = 1.60, p < .001) but not interpretation time, emergency/inpatient setting, or trainee participation (p > .05). However, in multivariable models, diagnostic error was independently associated with interpretation time (OR = 1.18, p = .003), shift volume (OR = 1.27, p < .001), and weekend interpretation (OR = 1.69, p = .02). In subanalysis, diagnostic error showed independent associations on weekdays with interpretation time (OR = 1.18, p = .003) and shift volume (OR = 1.27, p < .001); such associations were not observed on weekends (interpretation time: p = .62; shift volume: p = .58). CONCLUSION. Diagnostic errors in neuroradiology were associated with longer interpretation times, higher shift volumes, and weekend interpretation. CLINICAL IMPACT. These findings should be considered when designing work-flow-related interventions seeking to reduce neuroradiology interpretation errors.

Impact of Shift Volume on Neuroradiology Diagnostic Errors at a Large Tertiary Academic Center

BACKGROUND AND PURPOSE

Medical errors can result in significant morbidity and mortality. The goal of our study is to evaluate correlation between shift volume and errors made by attending neuroradiologists at an academic medical center, using a large data set.

MATERIALS AND METHODS

CT and MRI reports from our Neuroradiology Quality Assurance database (years 2014 - 2020) were searched for attending physician errors. Data were collected on shift volume, category of missed findings, error type, interpretation setting, exam type, clinical significance.

RESULTS

654 reports contained diagnostic error. There was a significant difference between mean volume of interpreted studies on shifts when an error was made compared with shifts in which no error was documented (46.58 (SD=22.37) vs 34.09 (SD=18.60), p<0.00001); and between shifts when perceptual error was made compared with shifts when interpretive errors were made (49.50 (SD=21.9) vs 43.26 (SD=21.75), p=0.0094). 59.6% of errors occurred in the emergency/inpatient setting, 84% were perceptual and 91.1% clinically significant. Categorical distribution of errors was: vascular 25.8%, brain 23.4%, skull base 13.8%, spine 12.4%, head/neck 11.3%, fractures 10.2%, other 3.1%. Errors were detected most often on brain MRI (25.4%), head CT (18.7%), head/neck CTA (13.8%), spine MRI (13.7%).

CONCLUSION

Errors were associated with higher volume shifts, were primarily perceptual and clinically significant. We need National guidelines establishing a range of what is a safe number of interpreted cross-sectional studies per day.

Patient safety in radiology: Our experience

Background

Present-day radiology departments have very high footfall of patients and are prone to patient safety errors. This study analyses such errors in our hospital.

Methods

Observational cross-sectional analysis of errors over the last 30 months was performed. These were classified using the Eindhoven classification model into technical, organizational, and human errors. Technical errors focused on equipment safety. Organizational errors related to policies. Human errors were subclassified as per the skill rule knowledge model. Root cause analysis was performed wherever necessary, and possible mitigation strategies for ensuring safety were suggested. Errors peculiar to the Armed Forces environment were specifically addressed.

Results

Seventy-seven errors were analyzed. Two were equipment based including faulty pressure injector syringes and radiation leakage from the computed tomography gantry. Of 44 skill-based errors, 09 involved dispatch of wrong reports to dependents owing to identifying patients with serving personnel's name. Four were due to scanning wrong sites. Eleven involved reporting abnormality on the wrong side. Six involved underreporting due to not viewing specific images. The rest were due to failure to omit conflicting elements in the report. Rule-based errors included wrong protocol selection (9 errors), omitting a particular sequence due to individual preference (6 errors), and so on. Knowledge-based errors were due to misinterpretation of findings (4 errors), reporting an abnormality as normal (3 errors), and selection of wrong modality (3 errors).

Conclusion

The findings of this study highlights the importance of voluntary reporting, diligent recording, and in-depth analysis of errors for understanding the causes and formulating possible mitigation strategies.

Management of Patients with Spine Tumors Strengthened by a Dedicated Multidisciplinary Spine Tumor Board: A 15-Year Single-Institutional Experience

BACKGROUND

Patients with spine tumors frequently require timely, multistep, and multidisciplinary care. A Spine Tumor Board (STB) provides a consistent forum wherein diverse specialists can interact, facilitating complex coordinated care for these patients. This study aims to present a single, large academic center's STB experience specifically reviewing case diversity, recommendations, and quantifying growth over time.

METHODS

All patient cases discussed at STB from May 2006 (STB inception) to May 2021 were evaluated. Collected data submitted by presenting physicians and formal documentation completed during the STB are summarized.

RESULTS

A total of 4549 cases were reviewed by STB over the study period, representing 2618 unique patients. Over the course of the study, a 266% increase in number of cases presented per week was observed (4.1 to 15.0). Cases were presented by surgeons (74%), radiation oncologists (18%), neurologists (2%), and other specialists (6%). The most common pathologic diagnoses discussed were spinal metastases (n = 1832; 40%), intradural extramedullary tumors (n = 798; 18%), and primary glial tumors (n = 567; 12%). Treatment recommendations included surgery, radiation therapy, or systemic therapy for 1743 cases (38%), continued routine follow-up/expectant management for 1592 cases (35%), supplementary imaging to better clarify the diagnosis for 549 cases (12%), and variable tailored recommendations for the remainder of cases (18%).

CONCLUSIONS

Care of patients with spine tumors is complex. We believe that the formation of a stand-alone STB is instrumental to accessing multidisciplinary input, enhancing confidence in management decisions for both patients and providers, assisting with care orchestration, and improving quality of care for patients with spine tumors.

Understanding Reader Variability: A 25-Radiologist Study on Liver Metastasis Detection at CT

Background Substantial interreader variability exists for common tasks in CT imaging, such as detection of hepatic metastases. This variability can undermine patient care by leading to misdiagnosis. Purpose To determine the impact of interreader variability associated with (a) reader experience, (b) image navigation patterns (eg, eye movements, workstation interactions), and (c) eye gaze time at missed liver metastases on contrast-enhanced abdominal CT images. Materials and Methods In a single-center prospective observational trial at an academic institution between December 2020 and February 2021, readers were recruited to examine 40 contrast-enhanced abdominal CT studies (eight normal, 32 containing 91 liver metastases). Readers circumscribed hepatic metastases and reported confidence. The workstation tracked image navigation and eye movements. Performance was quantified by using the area under the jackknife alternative free-response receiver operator characteristic (JAFROC-1) curve and per-metastasis sensitivity and was associated with reader experience and image navigation variables. Differences in area under JAFROC curve were assessed with the Kruskal-Wallis test followed by the Dunn test, and effects of image navigation were assessed by using the Wilcoxon signed-rank test. Results Twenty-five readers (median age, 38 years; IQR, 31-45 years; 19 men) were recruited and included nine subspecialized abdominal radiologists, five nonabdominal staff radiologists, and 11 senior residents or fellows. Reader experience explained differences in area under the JAFROC curve, with abdominal radiologists demonstrating greater area under the JAFROC curve (mean, 0.77; 95% CI: 0.75, 0.79) than trainees (mean, 0.71; 95% CI: 0.69, 0.73) (P = .02) or nonabdominal subspecialists (mean, 0.69; 95% CI: 0.60, 0.78) (P = .03). Sensitivity was similar within the reader experience groups (P = .96). Image navigation variables that were associated with higher sensitivity included longer interpretation time (P = .003) and greater use of coronal images (P < .001). The eye gaze time was at least 0.5 and 2.0 seconds for 71% (266 of 377) and 40% (149 of 377) of missed metastases, respectively. Conclusion Abdominal radiologists demonstrated better discrimination for the detection of liver metastases on abdominal contrast-enhanced CT images. Missed metastases frequently received at least a brief eye gaze. Higher sensitivity was associated with longer interpretation time and greater use of liver display windows and coronal images. © RSNA, 2022 Online supplemental material is available for this article.

Analysis of "visible in retrospect" to monitor false-negative findings in radiological reports

PURPOSE

False-negative findings in radiological reports can lead to serious adverse patient outcomes. We determined the frequency and tendency of false-negative findings in radiological reports by searching for words related to "visible in retrospect".

METHODS

In the period of 34 months, we extracted radiological reports containing words related to "visible in retrospect". Of these reports, we extracted false-negative findings that were not described in past reports and were first detected retrospectively. Misinterpretations were excluded. The occurrences of the terms that we identified were analyzed by all examinations, modality, month, and anatomical and lesion classifications were analyzed.

RESULTS

Of the 135,251 examinations, 941 reports (0.71%) with 962 findings were detected, with an average of 1.4 findings per business day. By modality, 713 of 81,899 (0.87%) CT examinations, 208 of 36,174 (0.57%) MR, 34 of 9,585 (0.35%) FDG-PET-CT, 2 of 2,258 (0.09%) digital radiography, and 5 of 5,335 (0.09%) other nuclear medicine examinations were found. By anatomical classification, there were 383 (40%) in chest, 353 (37%) in abdomen, 162 (17%) in head, 42 (4.4%) in face and neck, 9 (0.93%) in extremity, and 13 (1.4%) in others. By lesion classification, we identified 665 (69%) for localized lesion, 170 (18%) for vascular lesion, 83 (8.6%) for inflammatory lesion, 14 (1.5%) for traumatic lesion, 12 (1.2%) for organ dysfunction, 11 (1.1%) for degenerative lesion, and 7 (0.7%) for the others. Notable high-frequency specific site diseases by modality were 210 (22%) of localized lesions in lung on CT.

CONCLUSION

Our results demonstrated that missed lung localized lesions on CT, which account for about a fifth of false-negative findings, were the most common false-negative finding.

Diagnostic Errors in Cerebrovascular Pathology: Retrospective Analysis of a Neuroradiology Database at a Large Tertiary Academic Medical Center

BACKGROUND AND PURPOSE

Diagnostic errors affect 2%-8% of neuroradiology studies, resulting in significant potential morbidity and mortality. This retrospective analysis of a large database at a single tertiary academic institution focuses on diagnostic misses in cerebrovascular pathology and suggests error-reduction strategies.

MATERIALS AND METHODS

CT and MR imaging reports from a consecutive database spanning 2015-2020 were searched for errors of attending physicians in cerebrovascular pathology. Data were collected on missed findings, study types, and interpretation settings. Errors were categorized as ischemic, arterial, venous, hemorrhagic, and "other."

RESULTS

A total of 245,762 CT and MR imaging neuroradiology examinations were interpreted during the study period. Vascular diagnostic errors were present in 165 reports, with a mean of 49.6 (SD, 23.3) studies on the shifts when an error was made, compared with 34.9 (SD, 19.2) on shifts without detected errors (P < .0001). Seventy percent of examinations occurred in the hospital setting; 93.3% of errors were perceptual; 6.7% were interpretive; and 93.9% (n = 155) were clinically significant (RADPEER 2B or 3B). The distribution of errors was arterial and ischemic each with 33.3%, hemorrhagic with 21.8%, and venous with 7.5%. Most errors involved brain MR imaging (30.3%) followed by head CTA (27.9%) and noncontrast head CT (26.1%). The most common misses were acute/subacute infarcts (25.1%), followed by aneurysms (13.7%) and subdural hematomas (9.7%).

CONCLUSIONS

Most cerebrovascular diagnostic errors were perceptual and clinically significant, occurred in the emergency/inpatient setting, and were associated with higher-volume shifts. Diagnostic errors could be minimized by adjusting search patterns to ensure vigilance on the sites of the frequently missed pathologies.

Improving Radiology Trainees' Perception Using Where's Waldo?

RATIONALE AND OBJECTIVES

Perception is an essential skill leading to expertise in diagnostic radiology. We determined if practicing "Where's Waldo?" images improves accuracy and speed with which first and second year radiology residents detect abnormalities on chest radiographs (CXRs).

MATERIALS AND METHODS

Residents at three institutions were pretested using 50 CXRs, identifying locations of potential abnormalities. They were then split into trained (examining 7 "Where's Waldo?" images over three weeks) and control groups (no "Where's Waldo?"). They were then re-tested on the 50 CXRs. At one site, visual search parameters were acquired. Data were analyzed with repeated measures ANOVAs.

RESULTS

There was no significant difference in performance for trained vs control (F = 0.622, p = 0.436), with both improving significantly on post-test (F = 4.72, p = 0.037). Session time decreased significantly for both groups from pre to post-test (F = 81.47, p < 0.0001) and the decrease was significantly more (F = 31.59, p < 0.0001) for the trained group than the control group as well as for PGY with PGY3 having a larger average decrease in session time than PGY2. Eye-tracking data also showed significant increases in per image search efficiency with training.

CONCLUSION

Practicing "Where's Waldo?" or similar nonradiology search tasks may facilitate the acquisition of radiology image search but not detection skills, impacting reading efficiency more than detection accuracy.

Analysis of misses in imaging of head and neck pathology by attending neuroradiologists at a single tertiary academic medical centre

AIM

To analyse errors in head and neck (H&N) pathology made by attending neuroradiologists at a single tertiary-care centre.

MATERIALS AND METHODS

A neuroradiology quality assurance (QA) database of radiological errors was searched for attending physician errors in H&N pathology from 2014-2020. Data were limited to computed tomography (CT) and magnetic resonance imaging (MRI) reports. Data were collected on missed pathologies and study types. Misses were grouped into three categories: central neck (thyroid gland, aerodigestive tract), lateral neck (salivary glands, lymph nodes, soft tissues), and face/orbits (orbits, sinuses, masticator space).

RESULTS

During the study period, a total of 283,248 CT and MRI neuroradiology examinations were interpreted (all indications). Seventy-four H&N misses were identified comprising 85.1% perceptual and 14.9% interpretive errors. The distribution of errors was face/orbits (37.8%), central neck (36.5%), and lateral neck (25.7%). Clinically significant errors were found most commonly in the aerodigestive tract (21%), orbits (17.7%), masticator space, and parotid glands (14.5% each). The majority (67.6%) of the misses were detected on examinations that were not performed for a primary H&N indication; MRI brain was the most common examination (27%). Clearly malignant or potentially malignant masses accounted for 48.6% of all misses.

CONCLUSION

The majority of H&N misses were perceptual and were detected on examinations not performed for a H&N indication. Clearly malignant or potentially malignant masses represented half of all misses.

Am I Ready to Be an Independent Neuroradiologist? Objective Trends in Neuroradiology Fellows' Performance during the Fellowship Year

BACKGROUND AND PURPOSE

Aside from basic Accreditation Council for Graduate Medical Education guidelines, few metrics are in place to monitor fellows' progress. The purpose of this study was to determine objective trends in neuroradiology fellowship training on-call performance during an academic year.

MATERIALS AND METHODS

We retrospectively reviewed the number of cross-sectional neuroimaging studies dictated with complete reports by neuroradiology fellows during independent call. Monthly trends in total call cases, report turnaround times, relationships between volume and report turnaround times, and words addended to preliminary reports by attending neuroradiologists were evaluated with regression models. Monthly variation in frequencies of call-discrepancy macros were assessed via χ² tests. Changes in frequencies of specific macro use between fellowship semesters were assessed via serial 2-sample tests of proportions.

RESULTS

From 2012 to 2017, for 29 fellows, monthly median report turnaround times significantly decreased during the academic year: July (first month) = 79 minutes (95% CI, 71-86 minutes) and June (12th month) = 55 minutes (95% CI, 52-60 minutes; P value = .023). Monthly report turnaround times were inversely correlated with total volumes for CT (r = -0.70, F = 9.639, P value = .011) but not MR imaging. Words addended to preliminary reports, a surrogate measurement of report clarity, slightly improved and discrepancy rates decreased during the last 6 months of fellowship. A nadir for report turnaround times, discrepancy errors, and words addended to reports was seen in December and January.

CONCLUSIONS

Progress through fellowship correlates with a decline in report turnaround times and discrepancy rates for cross-sectional neuroimaging call studies and slight improvement in indirect quantitative measurement of report clarity. These metrics can be tracked throughout the academic year, and the midyear would be a logical time point for programs to assess objective progress of fellows and address any deficiencies.