Attending Radiologist Variability and Its Effect on Radiology Resident Discrepancy Rates

From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models

Expert feedback on trainees' preliminary reports is crucial for radiologic training, but real-time feedback can be challenging due to non-contemporaneous, remote reading and increasing imaging volumes. Trainee report revisions contain valuable educational feedback, but synthesizing data from raw revisions is challenging. Generative AI models can potentially analyze these revisions and provide structured, actionable feedback. This study used the OpenAI GPT-4 Turbo API to analyze paired synthesized and open-source analogs of preliminary and finalized reports, identify discrepancies, categorize their severity and type, and suggest review topics. Expert radiologists reviewed the output by grading discrepancies, evaluating the severity and category accuracy, and suggested review topic relevance. The reproducibility of discrepancy detection and maximal discrepancy severity was also examined. The model exhibited high sensitivity, detecting significantly more discrepancies than radiologists (W = 19.0, p < 0.001) with a strong positive correlation (r = 0.778, p < 0.001). Interrater reliability for severity and type were fair (Fleiss' kappa = 0.346 and 0.340, respectively; weighted kappa = 0.622 for severity). The LLM achieved a weighted F1 score of 0.66 for severity and 0.64 for type. Generated teaching points were considered relevant in ~ 85% of cases, and relevance correlated with the maximal discrepancy severity (Spearman ρ = 0.76, p < 0.001). The reproducibility was moderate to good (ICC (2,1) = 0.690) for the number of discrepancies and substantial for maximal discrepancy severity (Fleiss' kappa = 0.718; weighted kappa = 0.94). Generative AI models can effectively identify discrepancies in report revisions and generate relevant educational feedback, offering promise for enhancing radiology training.

Integrated cardio-thoracic imaging curriculum for residents: Design, implementation, and analysis of test and evaluation results

As the field of cardiac imaging has demonstrated exceptional growth over the past several decades, radiology departments and residency programs have struggled to integrate cardiac imaging instruction into training curricula.

PURPOSE

To create an integrated cardio-thoracic teaching and lecture curriculum and resident rotation in accordance with AGGME and Society of Thoracic Radiology (STR) guidelines.

MATERIALS AND METHODS

Consecutive PGY-2 to PGY-4 residents (n = 14) rotating through our Cardiothoracic Imaging (CTI) section from 1/1/2021 to 04/18/2022 were give pre- and post- rotation tests of knowledge and feedback evaluations. Attending feedback of the curriculum was obtained at 3-months and 9-months post curriculum implementation. A Wilcxon test was used to evaluate differences in improvement between pre- and post- rotation resident feedback scores, test scores for thoracic and cardiac test questions in addition to attending feedback scores at 3 and 9-months post curriculum implementation.

RESULTS

The overall post-rotation scores in addition to thoracic only and cardiac only scores improved, with the difference between improved versus stable or decreased scores being statistically significant overall (P = 0.039) and for cardiac scores (P = 0.003), but not for thoracic scores (P = 0.22). The overall (P = 0.002), thoracic (P = 0.027), and cardiac (P = 0.026) resident feedback scores were significantly improved post-rotation. Similarly, the overall attending feedback scores significantly improved over time (P = 0.021).

CONCLUSION

An integrated Cardio-thoracic Imaging teaching curriculum was well received by both residents and attendings with significant improvement in post rotation feedback scores by both groups. Moreover, residents demonstrated improved scores on knowledge tests post rotation.

Artificial intelligence-based education assists medical students' interpretation of hip fracture

Background

With recent transformations in medical education, the integration of technology to improve medical students’ abilities has become feasible. Artificial intelligence (AI) has impacted several aspects of healthcare. However, few studies have focused on medical education. We performed an AI-assisted education study and confirmed that AI can accelerate trainees’ medical image learning.

Materials

We developed an AI-based medical image learning system to highlight hip fracture on a plain pelvic film. Thirty medical students were divided into a conventional (CL) group and an AI-assisted learning (AIL) group. In the CL group, the participants received a prelearning test and a postlearning test. In the AIL group, the participants received another test with AI-assisted education before the postlearning test. Then, we analyzed changes in diagnostic accuracy.

Results

The prelearning performance was comparable in both groups. In the CL group, postlearning accuracy (78.66 ± 14.53) was higher than prelearning accuracy (75.86 ± 11.36) with no significant difference (p = .264). The AIL group showed remarkable improvement. The WithAI score (88.87 ± 5.51) was significantly higher than the prelearning score (75.73 ± 10.58, p < 0.01). Moreover, the postlearning score (84.93 ± 14.53) was better than the prelearning score (p < 0.01). The increase in accuracy was significantly higher in the AIL group than in the CL group.

Conclusion

The study demonstrated the viability of AI for augmenting medical education. Integrating AI into medical education requires dynamic collaboration from research, clinical, and educational perspectives.

Resident-faculty overnight discrepancy rates as a function of number of consecutive nights during a week of night float

Objectives

In 2018, the ACGME (Accreditation Council for Graduate Medical Education) made a change to the maximum permissible number of consecutive nights a resident trainee can be on “night float,” from six to seven nights. To our knowledge, although investigators have studied overall discrepancy rates and discrepancy rates as a function of shift length or perceived workload of a particular shift, no study has been performed to evaluate resident-faculty discrepancy rates as a quality/performance proxy, to see whether resident performance declines as a function of the number of consecutive nights. Our hypothesis is that we would observe a progressive increase in significant overnight resident – attending discrepancies over the 7 days’ time.

Methods

A total of 8,488 reports were extracted between 4/26/2019 to 8/22/2019 retrospectively. Data was obtained from the voice dictation system report server. Exported query was saved as a .csv file format and analyzed using Python packages. A “discrepancy checker” was created to search all finalized reports for the departmental standard heading of “Final Attending Report,” used to specify any significant changes from the preliminary interpretation.

Results

Model estimates varied on different days however there were no trends or patterns to indicate a deterioration in resident performance throughout the week. There were comparable probabilities throughout the week, with 2.17% on Monday, 2.35% on Thursday and 2.05% on Friday.

Conclusions

Our results reveal no convincing trend in terms of overnight report discrepancies between the preliminary report generated by the night float resident and the final report issued by a faculty the following morning. These results are in support of the ACGME’s recent change in the permissible number of consecutive nights on night float. We did not prove our hypothesis that resident performance on-call in the domain of report accuracy would diminish over seven consecutive nights while on the night float rotation. Our results found that performance remained fairly uniform over the course of the week.

Preliminary Radiology Report Discordances and Patient Outcomes

RATIONALE AND OBJECTIVES

At our institution, resident and fellow radiologists issue preliminary reports for off-hours imaging studies, which are overread by attending radiologists the next day using structured discrepancy templates. In this study, we examined the impact on patient management and outcome of studies with major discordance.

MATERIALS AND METHODS

For our retrospective observational study, preliminary reports between March and June 2017 that received major discordance were identified through report text search. Electronic medical records were reviewed for patient management change and patient outcome.

RESULTS

Of the 199 cases, 52 cases (26%) had management change and 119 cases (60%) did not have management change. In 25 cases (13%), the preliminary report was proven correct on subsequent management. Three cases (2%) were lost to follow-up. In only one case was adverse outcome directly related to the discordant finding. In cases with patient management change, there was higher proportion of perceptual error compared with those without management change (73% versus 59%). In 47 cases (24%), the discordant finding or diagnosis was known to the clinical team, and better history could have avoided the major change.

CONCLUSION

Adverse outcome from the discordant imaging finding was low (0.5%). Major change in preliminary report could be reduced with better clinical history. Patient management change was more frequently seen with perceptual errors, placing greater emphasis on strategies to reduce them.

Artificial intelligence for precision education in radiology

In the era of personalized medicine, the emphasis of health care is shifting from populations to individuals. Artificial intelligence (AI) is capable of learning without explicit instruction and has emerging applications in medicine, particularly radiology. Whereas much attention has focused on teaching radiology trainees about AI, here our goal is to instead focus on how AI might be developed to better teach radiology trainees. While the idea of using AI to improve education is not new, the application of AI to medical and radiological education remains very limited. Based on the current educational foundation, we highlight an AI-integrated framework to augment radiology education and provide use case examples informed by our own institution's practice. The coming age of "AI-augmented radiology" may enable not only "precision medicine" but also what we describe as "precision medical education," where instruction is tailored to individual trainees based on their learning styles and needs.

Resident Case Volume Correlates with Clinical Performance: Finding the Sweet Spot

RATIONALE AND OBJECTIVES

To determine whether the total number of studies interpreted during radiology residency correlates with clinical performance as measured by objective criteria.

MATERIALS AND METHODS

We performed a retrospective cohort study of three graduating classes of radiology residents from a single residency program between the years 2015-2017. The total number of studies interpreted by each resident during residency was tracked. Clinical performance was determined by tracking an individual resident's major discordance rate. A major discordance was recorded when there was a difference between the preliminary resident interpretation and final attending interpretation that could immediately impact patient care. Accreditation council for graduate medical education milestones at the completion of residency, Diagnostic radiology in-training scores in the third year, and score from the American board of radiology core exam were also tabulated. Pearson correlation coefficients and polynomial regression analysis were used to identify correlations between the total number of interpreted films and clinical, test, and milestone performance.

RESULTS

Thirty-seven residents interpreted a mean of 12,709 studies (range 8898-19,818; standard deviation [SD] 2351.9) in residency with a mean major discordance rate of 1.1% (range 0.34%-2.54%; stand dev 0.49%). There was a nonlinear correlation between total number of interpreted films and performance. As the number of interpreted films increased to approximately 16,000, clinical performance (p = 0.004) and test performance (p = 0.01) improved, but volumes over 16,000 correlated with worse performance.

CONCLUSION

The total number of studies interpreted during radiology training correlates with performance. Residencies should endeavor to find the "sweet spot": the amount of work that maximizes clinical exposure and knowledge without overburdening trainees.

Do Residency Selection Factors Predict Radiology Resident Performance?

RATIONALE AND OBJECTIVES

The purpose of our study is to determine what information in medical student residency applications predicts radiology residency success as defined by objective clinical performance data.

MATERIALS AND METHODS

We performed a retrospective cohort study of residents who entered our institution's residency program through the National Resident Matching Program as postgraduate year 2 residents and completed the program over the past 2 years. Medical school grades, selection to Alpha Omega Alpha (AOA) Honor Society, United States Medical Licensing Examination (USMLE) scores, publication in peer-reviewed journals, and whether the applicant was from a peer institution were the variables examined. Clinical performance was determined by calculating each resident's cumulative major discordance rate for on-call cases the resident read and gave a preliminary interpretation. A major discordance was defined as a difference between the preliminary resident and the final attending interpretations that could immediately impact the care of the patient. A multivariate logistic regression was performed to determine significant variables.

RESULTS

Twenty-seven residents provided preliminary reports on call for 67,145 studies. The mean major discordance rate was 1.08% (range 0.34%-2.54%). Higher USMLE Step 1 scores, publication before residency, and election to AOA Honor Society were all statistically significant predictors of lower major discordance rates (P values 0.01, 0.01,  and <0.001, respectively).

CONCLUSIONS

Overall resident performance was excellent. There are predictors that help select the better performing residents, namely higher USMLE Step 1 scores, one to two publications during medical school, and election to AOA in the junior year of medical school.