Improvement in reporting skills of radiology residents with a structured reporting curriculum

The importance of educational tools and a new software solution for visualizing and quantifying report correction in radiology training

A novel software, DiffTool, was developed in-house to keep track of changes made by board-certified radiologists to preliminary reports created by residents and evaluate its impact on radiological hands-on training. Before (t0) and after (t2-4) the deployment of the software, 18 residents (median age: 29 years; 33% female) completed a standardized questionnaire on professional training. At t2-4 the participants were also requested to respond to three additional questions to evaluate the software. Responses were recorded via a six-point Likert scale ranging from 1 ("strongly agree") to 6 ("strongly disagree"). Prior to the release of the software, 39% (7/18) of the residents strongly agreed with the statement that they manually tracked changes made by board-certified radiologists to each of their radiological reports while 61% were less inclined to agree with that statement. At t2-4, 61% (11/18) stated that they used DiffTool to track differences. Furthermore, we observed an increase from 33% (6/18) to 44% (8/18) of residents who agreed to the statement "I profit from every corrected report". The DiffTool was well accepted among residents with a regular user base of 72% (13/18), while 78% (14/18) considered it a relevant improvement to their training. The results of this study demonstrate the importance of providing a time-efficient way to analyze changes made to preliminary reports as an additive for professional training.

Structured Reports and Radiology Residents: Friends or Foes?

Structured reports offer overall improvement in quality and safety, largely centered upon more effective communication. Structured reporting is helpful to trainees as a method to develop organized search patterns and include pertinent positive and negative findings. However, limitations of structured report use include lack of development of individualized search patterns and failure to recognize key elements of the report to be formulated in the impression. Instruction on the value of a structured reporting approach, its impact on patient care and clinical service, and compliance with billing requirements must be balanced with early and consistent feedback on appropriate use and reporting errors. It is incumbent upon radiology educators to integrate and optimize structured reporting in the learning environment. This manuscript addresses the impact of structured reporting on radiology education, reviewing quality and safety considerations, detailing benefits and drawbacks for trainees, and offering strategies for optimizing the use of structured reporting/templates in the training environment.

Structured reporting of brain MRI following mechanical thrombectomy in acute ischemic stroke patients

Background

To compare the quality of free-text reports (FTR) and structured reports (SR) of brain magnetic resonance imaging (MRI) examinations in patients following mechanical thrombectomy for acute stroke treatment.

Methods

A template for SR of brain MRI examinations based on decision trees was designed and developed in house and applied to twenty patients with acute ischemic stroke in addition to FTR. Two experienced stroke neurologists independently evaluated the quality of FTR and SR regarding clarity, content, presence of key features, information extraction, and overall report quality. The statistical analysis for the differences between FTR and SR was performed using the Mann–Whitney U-test or the Chi-squared test.

Results

Clarity (p < 0.001), comprehensibility (p < 0.001), inclusion of relevant findings (p = 0.016), structure (p = 0.005), and satisfaction with the content of the report for immediate patient management (p < 0.001) were evaluated significantly superior for the SR by both neurologist raters. One rater additionally found the explanation of the patient’s clinical symptoms (p = 0.003), completeness (p < 0.009) and length (p < 0.001) of SR to be significantly superior compared to FTR and stated that there remained no open questions, requiring further consultation of the radiologist (p < 0.001). Both neurologists preferred SR over FTR.

Conclusions

The use of SR for brain magnetic resonance imaging may increase the report quality and satisfaction of the referring physicians in acute ischemic stroke patients following mechanical thrombectomy.

Trial registration Retrospectively registered.

Creating high-quality radiology reports in foreign languages through multilingual structured reporting

OBJECTIVES

Globalization and migration are increasing the demand for reports in different languages. We aimed to examine if structured reports created by non-German-speaking radiologists with multilingual templates show significant differences in quality to structured reports and free-text reports by German native speakers.

METHODS

We used structured templates that allow radiologists to report in their mother tongue and then switch the report language to German or English automatically using proprietary software. German- and English-speaking radiology residents created structured reports in both German and English with these templates. Reports for three different exam types were created (intensive care chest x-ray, shoulder x-ray specifically for degenerative processes, and CT pulmonary angiogram for pulmonary embolism). The report quality of automatically translated German structured reports by English-speaking radiologists and German structured reports by German radiologists was then evaluated by German clinicians with a standardized questionnaire. The questionnaire was designed to assess attributes including content, comprehensibility, clinical consequences, and overall quality.

RESULTS

Structured reports by English-speaking radiologists that were automatically translated into German and German structured reports by German radiologists both received very high or high overall quality ratings in the majority of cases, showing no significant differences in quality. Likewise, no significant differences were observed between the two report types regarding comprehensibility and clinical consequences. Structured reports by German radiologists received significantly better ratings for overall quality and comprehensibility compared to free-text reports by German radiologists.

CONCLUSIONS

Multilingual structured reporting templates may serve as a feasible tool for creating high-quality radiology reports in foreign languages.

KEY POINTS

• Multilingualism in structured reporting templates can be a useful tool for creating high-quality radiology reports in foreign languages. • German reports created with multilingual structured reporting templates by English-speaking radiologists and German structured reports by German radiologists exhibit no significant differences in overall report quality. • Multilingual structured reporting templates can help radiologists overcome communication barriers and facilitate teleradiology.

Using a Natural Language Processing and Machine Learning Algorithm Program to Analyze Inter-Radiologist Report Style Variation and Compare Variation Between Radiologists When Using Highly Structured Versus More Free Text Reporting

PURPOSE

To use a natural language processing and machine learning algorithm to evaluate inter-radiologist report variation and compare variation between radiologists using highly structured versus more free text reporting.

MATERIALS AND METHODS

28,615 radiology reports were analyzed for 4 metrics: verbosity, observational terms only, unwarranted negative findings, and repeated language in different sections. Radiology reports for two imaging examinations were analyzed and compared - one which was more templated (ultrasound - appendicitis) and one which relied on more free text (chest radiograph - single view). For each metric, the mean and standard deviation for defined outlier results for all dictations (individual and group mean) was calculated. The mean number of outlier metrics per reader per study was calculated and compared between radiologists and between the two report types. Wilcoxon rank test and paired Wilcoxon signed rank test were applied. The radiologists were also ranked based on the number of outlier metrics identified per study.

RESULTS

There was great variability in radiologist dictation styles - outlier metrics per report varied greatly between radiologists with the maximum 10 times higher than the minimum score. Metric values were greater (P < 0.0001) on the standardized reports using free text than the more structured reports.

CONCLUSIONS

The algorithm successfully evaluated metrics showing variability in reporting profiles particularly when there is free text. This variability can be an obstacle to providing effective communication and reliability of care.

Structured Reporting of CT Angiography Runoff Examinations of the Lower Extremities

OBJECTIVES

The aim was to evaluate the effect of structured reporting of computed tomography angiography (CTA) runoff studies on clarity, completeness, clinical relevance, usefulness of the radiology reports, further testing, and therapy in patients with known or suspected peripheral arterial disease.

METHODS

Conventional reports (CRs) and structured reports (SRs) were generated for 52 patients who had been examined with a CTA runoff examination of the lower extremities. The sample size was based on power calculations with a power of 95% and a significance level of .007 (adjusted for multiple testing). CRs were dictated in a free text form; SRs contained a consistent ordering of observations with standardised subheadings. CRs were compared with SRs. Two vascular medicine specialists and two vascular surgeons rated the reports regarding their satisfaction with clarity, completeness, clinical relevance, and usefulness as well as overall satisfaction. Additionally, they made hypothetical decisions on further testing and therapy. Median ratings were compared using the Wilcoxon signed rank test and generalised linear mixed effects models.

RESULTS

SRs received higher ratings for satisfaction with clarity (median rating 9.0 vs. 7.0, p < .0001) and completeness (median rating 9.0 vs. 7.5, p < .0001) and were judged to be of greater clinical relevance (median rating 9.0 vs. 8.0, p < .0001) and usefulness (median rating 9.0 vs. 8.0, p < .0001). Overall satisfaction was also higher for SRs (median rating 9.0 vs. 7.0, p < .0001) than CRs. There were no significant differences in further testing or therapy.

CONCLUSION

Referring clinicians perceive SRs of CTA runoff examinations of the lower extremities as offering superior clarity, completeness, clinical relevance, and usefulness than CRs. Structured reporting does not appear to alter further testing or therapy in patients with known or suspected peripheral arterial disease.

Structured Reporting of Rectal Magnetic Resonance Imaging in Suspected Primary Rectal Cancer: Potential Benefits for Surgical Planning and Interdisciplinary Communication

OBJECTIVES

The aim of this study was to evaluate the effect of structured reports (SRs) in comparison to nonstructured, free-text (FT) rectal magnetic resonance imaging (MRI) reports in patients with histologically proven rectal cancer and potential effects of both types of reporting on referring surgeons' satisfaction, interdisciplinary communication, and further clinical decision making.

MATERIALS AND METHODS

The institutional review board approved this retrospective study with waiver of informed consent. Forty-nine patients with histologically proven rectal cancer were included in this study. All patients underwent rectal MRI for local rectal cancer staging before surgery. Free-text reports and SRs for local MR staging of rectal cancer were generated for all subjects by radiologists. Two experienced abdominal surgeons evaluated a questionnaire that included 9 questions regarding satisfaction with content, presence of reported key features, effort for information extraction, and report quality.

RESULTS

Structured reports achieved significantly higher satisfaction rates with report content and clarity, and included significantly more of the 13 predefined key features compared with FT reports (SRs: mean ± SD, 12.2 ± 4.6 [range, 9-13] versus FT reports: mean ± SD, 9.2 ± 10.8 [range, 5-13]) (P < 0.001). Definite further clinical decision making (surgery vs neoadjuvant radiochemotherapy) was possible in 96% of SRs and only in 60% of FT reports (P < 0.001). In case of surgery, the reported information was considered to be sufficient for surgical planning in 94% of SRs versus only 38% in FT reports (P < 0.001). Structured report received a significantly higher overall report quality rated on a Likert scale from 1 to 6 (1, insufficient; 6, excellent) with a mean of 5.8 ± 0.42 (range, 5-6) in comparison to FT reports with 3.6 ± 1.19 (range, 1-5) (P < 0.001).

CONCLUSIONS

Structured reporting of rectal MRI in patients with rectal cancer facilitates surgical planning and leads to a higher satisfaction level of referring surgeons in comparison to FT reports. Abdominal surgeons were more confident about report correctness and further clinical decision making on the basis of SRs.

Structured reporting of CT enterography for inflammatory bowel disease: effect on key feature reporting, accuracy across training levels, and subjective assessment of disease by referring physicians

PURPOSE

To compare the content and accuracy of structured reporting (SR) versus non-structured reporting (NSR) for computed tomographic enterography (CTE) of inflammatory bowel disease (IBD).

MATERIALS AND METHODS

This IRB-approved, HIPAA-compliant, retrospective study included 30 adult subjects (15 male, 15 female; mean age 41.9 years) with IBD imaged with CTE. Nine radiologists (3 faculty, 3 abdominal imaging fellows, and 3 senior radiology residents) independently interpreted all examinations using both NSR and SR, separated by four weeks. Reports were assessed for documentation of 15 key reporting features and a subset of 5 features was assessed for accuracy. Thirty faculty reports (15 NSR [5 per reader] and 15 SR [5 per reader]) were randomly selected for review by three referring physicians, who independently rated quality metrics for each report.

RESULTS

NSR documented the presence or absence of 8.2 ± 2.2 key features, while SR documented 14.6 ± 0.5 features (p < 0.001). SR resulted in increased documentation of 13 of 15 features including stricture (p < 0.001), fistula (p < 0.001), fluid collection (p = 0.003), and perianal disease (p < 0.001). Among a subset of five features, accuracy for diagnosing multifocal disease was minimally increased when using SR (76% NSR vs. 83% SR; p = 0.01), but accuracy for other features was not affected by report type. Referring physicians significantly preferred SR based on ease of information extraction (p < 0.01).

CONCLUSION

Structured reporting of CTE for IBD improved documentation of key reporting features for trainees and faculty, though there was minimal impact on accuracy. Referring physicians subjectively preferred the structured reports.

Chart-stimulated Recall as a Learning Tool for Improving Radiology Residents' Reports

RATIONALE AND OBJECTIVES

Workplace-based assessments gauge the highest tier of clinical competence. Chart-stimulated recall (CSR) is a workplace-based assessment method that complements chart audit with an interview based on the residents' notes. It allows evaluation of the residents' knowledge and heuristics while providing opportunities for feedback and self-reflection. We evaluated the utility of CSR for improving the radiology residents' reporting skills.

MATERIALS AND METHODS

Residents in each year of training were randomly assigned to an intervention group (n = 12) or a control group (n = 13). Five pre-intervention and five post-intervention reports of each resident were independently evaluated by three blinded reviewers using a modified Bristol Radiology Report Assessment Tool. The study intervention comprised a CSR interview tailored to each individual resident's learning needs based on the pre-intervention assessment. The CSR process focused on the clinical relevance of the radiology reports. Student's t test (P < .05) was used to compare pre- and post-intervention scores of each group.

RESULTS

A total of 125 pre-intervention and 125 post-intervention reports were evaluated (total 750 assessments). The Cronbach's alpha for the study tool was 0.865. A significant improvement was seen in the cumulative 19-item score (66% versus 73%, P < .001) and the global rating score (59% versus 72%, P < .001) of the intervention group after the CSR. The reports of the control group did not demonstrate any significant improvement.

CONCLUSION

CSR is a feasible workplace-based assessment method for improving reporting skills of the radiology residents.

A succinct rating scale for radiology report quality

Context:

Poorly written radiology reports are common among residents and are a significant challenge for radiology education. While training may improve report quality, a professionally developed reliable and valid scale to measure report quality does not exist.

Objectives:

To develop a measurement tool for report quality, the quality of report scale, with rigorous validation through empirical data.

Methods:

A research team of an experienced psychometrician and six senior radiologists conducted qualitative and quantitative studies. Five items were identified for the quality of report scale, each measuring a distinct aspect of report quality. Two dedicated training sessions were designed and implemented to help residents generate high-quality reports. In a blinded fashion, the quality of report scale was applied to 804 randomly selected reports issued before (n = 403) and after (n = 401) training. Full-scale psychometrical assessments were implemented onto the quality of report scale’s item- and scale-scores from the reports. The quality of report scale scores were correlated with report professionalism and attendings’ preference and were compared pre-/post-training.

Results:

The quality of report scale showed sound psychometrical properties, with high validity and reliability. Reports with higher quality of report scale score were more professional and preferable by attendings. Training improved the quality of report scale score, empirically validating the quality of report scale further.

Conclusion:

While succinct and practitioner friendly, the quality of report scale is a reliable and valid measure of radiology report quality and has the potential to be easily adapted to other fields such as pathology, where similar training would be beneficial.