Revealing the most common reporting errors through data mining of the report proofreading process

Turning radiology reports into epidemiological data to track seasonal pulmonary infections and the COVID-19 pandemic

OBJECTIVES

To automatically label chest radiographs and chest CTs regarding the detection of pulmonary infection in the report text, to calculate the number needed to image (NNI) and to investigate if these labels correlate with regional epidemiological infection data.

MATERIALS AND METHODS

All chest imaging reports performed in the emergency room between 01/2012 and 06/2022 were included (64,046 radiographs; 27,705 CTs). Using a regular expression-based text search algorithm, reports were labeled positive/negative for pulmonary infection if described. Data for regional weekly influenza-like illness (ILI) consultations (10/2013-3/2022), COVID-19 cases, and hospitalization (2/2020-6/2022) were matched with report labels based on calendar date. Positive rate for pulmonary infection detection, NNI, and the correlation with influenza/COVID-19 data were calculated.

RESULTS

Between 1/2012 and 2/2020, a 10.8-16.8% per year positive rate for detecting pulmonary infections on chest radiographs was found (NNI 6.0-9.3). A clear and significant seasonal change in mean monthly detection counts (102.3 winter; 61.5 summer; p < .001) correlated moderately with regional ILI consultations (weekly data r = 0.45; p < .001). For 2020-2021, monthly pulmonary infection counts detected by chest CT increased to 64-234 (23.0-26.7% per year positive rate, NNI 3.7-4.3) compared with 14-94 (22.4-26.7% positive rate, NNI 3.7-4.4) for 2012-2019. Regional COVID-19 epidemic waves correlated moderately with the positive pulmonary infection CT curve for 2020-2022 (weekly new cases: r = 0.53; hospitalizations: r = 0.65; p < .001).

CONCLUSION

Text mining of radiology reports allows to automatically extract diagnoses. It provides a metric to calculate the number needed to image and to track the trend of diagnoses in real time, i.e., seasonality and epidemic course of pulmonary infections.

CLINICAL RELEVANCE

Digitally labeling radiology reports represent previously neglected data and may assist in automated disease tracking, in the assessment of physicians' clinical reasoning for ordering radiology examinations and serve as actionable data for hospital workflow optimization.

KEY POINTS

• Radiology reports, commonly not machine readable, can be automatically labeled with the contained diagnoses using a regular-expression based text search algorithm. • Chest radiograph reports positive for pulmonary infection moderately correlated with regional influenza-like illness consultations (weekly data; r = 0.45; p < .001) and chest CT reports with the course of the regional COVID-19 pandemic (new cases: r = 0.53; hospitalizations: r = 0.65; p < 0.001). • Rendering radiology reports into data labels provides a metric for automated disease tracking, the assessment of ordering physicians clinical reasoning and can serve as actionable data for workflow optimization.

Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy

Background Errors in radiology reports may occur because of resident-to-attending discrepancies, speech recognition inaccuracies, and large workload. Large language models, such as GPT-4 (ChatGPT; OpenAI), may assist in generating reports. Purpose To assess effectiveness of GPT-4 in identifying common errors in radiology reports, focusing on performance, time, and cost-efficiency. Materials and Methods In this retrospective study, 200 radiology reports (radiography and cross-sectional imaging [CT and MRI]) were compiled between June 2023 and December 2023 at one institution. There were 150 errors from five common error categories (omission, insertion, spelling, side confusion, and other) intentionally inserted into 100 of the reports and used as the reference standard. Six radiologists (two senior radiologists, two attending physicians, and two residents) and GPT-4 were tasked with detecting these errors. Overall error detection performance, error detection in the five error categories, and reading time were assessed using Wald χ2 tests and paired-sample t tests. Results GPT-4 (detection rate, 82.7%;124 of 150; 95% CI: 75.8, 87.9) matched the average detection performance of radiologists independent of their experience (senior radiologists, 89.3% [134 of 150; 95% CI: 83.4, 93.3]; attending physicians, 80.0% [120 of 150; 95% CI: 72.9, 85.6]; residents, 80.0% [120 of 150; 95% CI: 72.9, 85.6]; P value range, .522-.99). One senior radiologist outperformed GPT-4 (detection rate, 94.7%; 142 of 150; 95% CI: 89.8, 97.3; P = .006). GPT-4 required less processing time per radiology report than the fastest human reader in the study (mean reading time, 3.5 seconds ± 0.5 [SD] vs 25.1 seconds ± 20.1, respectively; P < .001; Cohen d = -1.08). The use of GPT-4 resulted in lower mean correction cost per report than the most cost-efficient radiologist ($0.03 ± 0.01 vs $0.42 ± 0.41; P < .001; Cohen d = -1.12). Conclusion The radiology report error detection rate of GPT-4 was comparable with that of radiologists, potentially reducing work hours and cost. © RSNA, 2024 See also the editorial by Forman in this issue.

ESR paper on structured reporting in radiology-update 2023

Structured reporting in radiology continues to hold substantial potential to improve the quality of service provided to patients and referring physicians. Despite many physicians' preference for structured reports and various efforts by radiological societies and some vendors, structured reporting has still not been widely adopted in clinical routine.While in many countries national radiological societies have launched initiatives to further promote structured reporting, cross-institutional applications of report templates and incentives for usage of structured reporting are lacking. Various legislative measures have been taken in the USA and the European Union to promote interoperable data formats such as Fast Healthcare Interoperability Resources (FHIR) in the context of the EU Health Data Space (EHDS) which will certainly be relevant for the future of structured reporting. Lastly, recent advances in artificial intelligence and large language models may provide innovative and efficient approaches to integrate structured reporting more seamlessly into the radiologists' workflow.The ESR will remain committed to advancing structured reporting as a key component towards more value-based radiology. Practical solutions for structured reporting need to be provided by vendors. Policy makers should incentivize the usage of structured radiological reporting, especially in cross-institutional setting.Critical relevance statement Over the past years, the benefits of structured reporting in radiology have been widely discussed and agreed upon; however, implementation in clinical routine is lacking due-policy makers should incentivize the usage of structured radiological reporting, especially in cross-institutional setting.Key points1. Various national societies have established initiatives for structured reporting in radiology.2. Almost no monetary or structural incentives exist that favor structured reporting.3. A consensus on technical standards for structured reporting is still missing.4. The application of large language models may help structuring radiological reports.5. Policy makers should incentivize the usage of structured radiological reporting.

Imaging of Overuse Injuries of the Ankle and Foot in Sport and Work

Overuse injuries of the ankle and foot are common injuries both in sport and in a work-related context. After clinical assessment, imaging is key for early diagnosis. In this overview article, we focus on imaging techniques, protocols, and imaging findings of overuse injuries of the ankle and foot; we emphasize the important role of structured reporting; and we discuss clinical symptoms, epidemiology, and risk factors in sports and in a work-related context.

State-of-the-art CT and MR imaging and assessment of atherosclerotic carotid artery disease: the reporting-a consensus document by the European Society of Cardiovascular Radiology (ESCR)

The European Society of Cardiovascular Radiology (ESCR) is the European specialist society of cardiac and vascular imaging. This society's highest priority is the continuous improvement, development, and standardization of education, training, and best medical practice, based on experience and evidence. The present intra-society consensus is based on the existing scientific evidence and on the individual experience of the members of the ESCR writing group on carotid diseases, the members of the ESCR guidelines committee, and the members of the executive committee of the ESCR. The recommendations published herein reflect the evidence-based society opinion of ESCR. The purpose of this second document is to discuss suggestions for standardized reporting based on the accompanying consensus document part I. KEY POINTS: • CT and MR imaging-based evaluation of carotid artery disease provides essential information for risk stratification and prediction of stroke. • The information in the report must cover vessel morphology, description of stenosis, and plaque imaging features. • A structured approach to reporting ensures that all essential information is delivered in a standardized and consistent way to the referring clinician.

Structured reporting of computed tomography in the polytrauma patient assessment: a Delphi consensus proposal

OBJECTIVES

To develop a structured reporting (SR) template for whole-body CT examinations of polytrauma patients, based on the consensus of a panel of emergency radiology experts from the Italian Society of Medical and Interventional Radiology.

METHODS

A multi-round Delphi method was used to quantify inter-panelist agreement for all SR sections. Internal consistency for each section and quality analysis in terms of average inter-item correlation were evaluated by means of the Cronbach's alpha (Cα) correlation coefficient.

RESULTS

The final SR form included 118 items (6 in the "Patient Clinical Data" section, 4 in the "Clinical Evaluation" section, 9 in the "Imaging Protocol" section, and 99 in the "Report" section). The experts' overall mean score and sum of scores were 4.77 (range 1-5) and 257.56 (range 206-270) in the first Delphi round, and 4.96 (range 4-5) and 208.44 (range 200-210) in the second round, respectively. In the second Delphi round, the experts' overall mean score was higher than in the first round, and standard deviation was lower (3.11 in the second round vs 19.71 in the first round), reflecting a higher expert agreement in the second round. Moreover, Cα was higher in the second round than in the first round (0.97 vs 0.87).

CONCLUSIONS

Our SR template for whole-body CT examinations of polytrauma patients is based on a strong agreement among panel experts in emergency radiology and could improve communication between radiologists and the trauma team.

Bigram frequency analysis for detection of radiology report errors

PURPOSE

This pilot study evaluates the utility of analyzing bigram frequencies for detecting radiology report errors.

METHODS

A corpus of 48,050 CT reports was used to enumerate the frequency of each bigram (F_AB), and the expected frequency of each bigram in the corpus based on the constituent unigram frequencies (P_AB). A test set consisted of a separate random sample of 200 radiology reports dictated by attendings for CT scans of the abdomen in 2019, as well as a random sample of 200 radiology reports for CT scans of the abdomen dictated in 2019 by 52 different residents or fellows prior to editing by the signing attendings. Bigrams in the test reports that occurred either rarely or not at all in the corpus were flagged for manual review by an abdominal radiologist.

FINDINGS

Of 682 n-grams flagged in attending reports, 11.6% were true errors, while of 1378 n-grams flagged in trainee reports, 7.9% were true errors. The largest group of flagged n-grams in both test sets involved bigrams that did not appear in the corpus, but whose constituent words did appear in the corpus. Subsets of 50 attending and 50 resident reports were manually reviewed, revealing that the flagging procedure had a sensitivity for errors of 58% (22/38) in the attending reports and 97% (31/32) in the resident reports.

CONCLUSION

Bigram frequency analysis may be of practical value in reviewing radiology reports for errors. Further methodological refinement to improve the positive predictive value of error detection is required.

Structured Reporting in Musculoskeletal Radiology

Musculoskeletal (MSK) radiologists are predominantly consultants in the service departments of health care. Unlike the manufacturing industry, quality controls are difficult to institute in a service industry and more variability is expected. Structured reporting is a unique way to institute quality standards, and by using the checklist approach with uniform terminology, it can lead to more homogeneity and consistency of reporting, concise lexicon use within and across practices, minimization of errors, enhancement of divisional and departmental branding, improvement of interdisciplinary communications, and future data mining. We share our experience from more than a decade of structured reporting in the domain of MSK radiology, our practice standards, and how reporting has evolved in our MSK practice. Further discussions include future directions aided by machine learning approaches with augmented reality and the possibility of virtual fellowship and training using consistent lexicons and structured reporting.

Tips for Reporting Musculoskeletal Imaging Studies: Lessons Learned

This paper is designed to be read by radiological trainees who are starting out with reporting musculoskeletal imaging studies. Based on the author's experience of over 25 years, it provides tips on how to report musculoskeletal imaging succinctly and effectively using a prose style report.

Delayed diagnosis of a piriformis abscess due to hyposkillia and communication error

A 69-year-old man with dementia presented to our hospital with fever and left back pain. He was diagnosed with acute pyelonephritis based on microscopy and CT findings indicating bacteriuria and elevated fat density around the right kidney, respectively. Three days post admission, the diagnosis of a urinary tract infection was queried and the patient was re-examined. Marked tenderness was observed in the left buttock; contrast CT scan was subsequently performed and a left piriformis abscess was diagnosed. This diagnostic delay occurred due to a communication error-the referral letter to the radiologist stated 'fever and back pain'-and the physician's hyposkillia; hence, differential diagnoses were not considered. Physicians can reduce the frequency of related errors by conducting proper physical examinations and not relying on imaging findings alone. Furthermore, to ensure accurate diagnoses, the comments to the radiologist should be clear and detailed when requesting diagnostic imaging.