1
|
Lyo S, Mohan S, Hassankhani A, Noor A, Dako F, Cook T. From Revisions to Insights: Converting Radiology Report Revisions into Actionable Educational Feedback Using Generative AI Models. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01233-4. [PMID: 39160366 DOI: 10.1007/s10278-024-01233-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/13/2024] [Revised: 08/07/2024] [Accepted: 08/08/2024] [Indexed: 08/21/2024]
Abstract
Expert feedback on trainees' preliminary reports is crucial for radiologic training, but real-time feedback can be challenging due to non-contemporaneous, remote reading and increasing imaging volumes. Trainee report revisions contain valuable educational feedback, but synthesizing data from raw revisions is challenging. Generative AI models can potentially analyze these revisions and provide structured, actionable feedback. This study used the OpenAI GPT-4 Turbo API to analyze paired synthesized and open-source analogs of preliminary and finalized reports, identify discrepancies, categorize their severity and type, and suggest review topics. Expert radiologists reviewed the output by grading discrepancies, evaluating the severity and category accuracy, and suggested review topic relevance. The reproducibility of discrepancy detection and maximal discrepancy severity was also examined. The model exhibited high sensitivity, detecting significantly more discrepancies than radiologists (W = 19.0, p < 0.001) with a strong positive correlation (r = 0.778, p < 0.001). Interrater reliability for severity and type were fair (Fleiss' kappa = 0.346 and 0.340, respectively; weighted kappa = 0.622 for severity). The LLM achieved a weighted F1 score of 0.66 for severity and 0.64 for type. Generated teaching points were considered relevant in ~ 85% of cases, and relevance correlated with the maximal discrepancy severity (Spearman ρ = 0.76, p < 0.001). The reproducibility was moderate to good (ICC (2,1) = 0.690) for the number of discrepancies and substantial for maximal discrepancy severity (Fleiss' kappa = 0.718; weighted kappa = 0.94). Generative AI models can effectively identify discrepancies in report revisions and generate relevant educational feedback, offering promise for enhancing radiology training.
Collapse
Affiliation(s)
- Shawn Lyo
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA.
| | - Suyash Mohan
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA
| | - Alvand Hassankhani
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA
| | - Abass Noor
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA
| | - Farouk Dako
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA
| | - Tessa Cook
- Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
2
|
Barile MF, Kifjak D, El Kaddouri B, Bankier AA. Integrated cardio-thoracic imaging curriculum for residents: Design, implementation, and analysis of test and evaluation results. Eur J Radiol 2024; 177:111584. [PMID: 38909516 DOI: 10.1016/j.ejrad.2024.111584] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2024] [Revised: 05/20/2024] [Accepted: 06/19/2024] [Indexed: 06/25/2024]
Abstract
As the field of cardiac imaging has demonstrated exceptional growth over the past several decades, radiology departments and residency programs have struggled to integrate cardiac imaging instruction into training curricula. PURPOSE To create an integrated cardio-thoracic teaching and lecture curriculum and resident rotation in accordance with AGGME and Society of Thoracic Radiology (STR) guidelines. MATERIALS AND METHODS Consecutive PGY-2 to PGY-4 residents (n = 14) rotating through our Cardiothoracic Imaging (CTI) section from 1/1/2021 to 04/18/2022 were give pre- and post- rotation tests of knowledge and feedback evaluations. Attending feedback of the curriculum was obtained at 3-months and 9-months post curriculum implementation. A Wilcxon test was used to evaluate differences in improvement between pre- and post- rotation resident feedback scores, test scores for thoracic and cardiac test questions in addition to attending feedback scores at 3 and 9-months post curriculum implementation. RESULTS The overall post-rotation scores in addition to thoracic only and cardiac only scores improved, with the difference between improved versus stable or decreased scores being statistically significant overall (P = 0.039) and for cardiac scores (P = 0.003), but not for thoracic scores (P = 0.22). The overall (P = 0.002), thoracic (P = 0.027), and cardiac (P = 0.026) resident feedback scores were significantly improved post-rotation. Similarly, the overall attending feedback scores significantly improved over time (P = 0.021). CONCLUSION An integrated Cardio-thoracic Imaging teaching curriculum was well received by both residents and attendings with significant improvement in post rotation feedback scores by both groups. Moreover, residents demonstrated improved scores on knowledge tests post rotation.
Collapse
Affiliation(s)
- Maria F Barile
- Department of Radiology, University of Massachusetts Memorial Health and University of Massachusetts Chan Medical School, 55 N. lake Ave., Worcester, MA, United States.
| | - Daria Kifjak
- Department of Radiology, University of Massachusetts Memorial Health and University of Massachusetts Chan Medical School, 55 N. lake Ave., Worcester, MA, United States; Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria.
| | - Bilal El Kaddouri
- Department of Radiology, University of Massachusetts Memorial Health and University of Massachusetts Chan Medical School, 55 N. lake Ave., Worcester, MA, United States; Department of Radiology, Erasmus Hospital, Université Libre de Bruxelles, Brussels, Belgium.
| | - Alexander A Bankier
- Department of Radiology, University of Massachusetts Memorial Health and University of Massachusetts Chan Medical School, 55 N. lake Ave., Worcester, MA, United States.
| |
Collapse
|
3
|
Peterson C, Moore M, Gagnon E, Sarwani N, Mosher T, Bruno M. How are our residents doing on trauma tonight? The frequency of overnight resident-faculty report discrepancies in trauma patients. Emerg Radiol 2021; 28:1113-1117. [DOI: 10.1007/s10140-021-01963-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2021] [Accepted: 06/28/2021] [Indexed: 10/20/2022]
|
4
|
Cheng CT, Chen CC, Fu CY, Chaou CH, Wu YT, Hsu CP, Chang CC, Chung IF, Hsieh CH, Hsieh MJ, Liao CH. Artificial intelligence-based education assists medical students' interpretation of hip fracture. Insights Imaging 2020; 11:119. [PMID: 33226480 PMCID: PMC7683624 DOI: 10.1186/s13244-020-00932-0] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2020] [Accepted: 10/27/2020] [Indexed: 02/04/2023] Open
Abstract
Background With recent transformations in medical education, the integration of technology to improve medical students’ abilities has become feasible. Artificial intelligence (AI) has impacted several aspects of healthcare. However, few studies have focused on medical education. We performed an AI-assisted education study and confirmed that AI can accelerate trainees’ medical image learning. Materials We developed an AI-based medical image learning system to highlight hip fracture on a plain pelvic film. Thirty medical students were divided into a conventional (CL) group and an AI-assisted learning (AIL) group. In the CL group, the participants received a prelearning test and a postlearning test. In the AIL group, the participants received another test with AI-assisted education before the postlearning test. Then, we analyzed changes in diagnostic accuracy.
Results The prelearning performance was comparable in both groups. In the CL group, postlearning accuracy (78.66 ± 14.53) was higher than prelearning accuracy (75.86 ± 11.36) with no significant difference (p = .264). The AIL group showed remarkable improvement. The WithAI score (88.87 ± 5.51) was significantly higher than the prelearning score (75.73 ± 10.58, p < 0.01). Moreover, the postlearning score (84.93 ± 14.53) was better than the prelearning score (p < 0.01). The increase in accuracy was significantly higher in the AIL group than in the CL group. Conclusion The study demonstrated the viability of AI for augmenting medical education. Integrating AI into medical education requires dynamic collaboration from research, clinical, and educational perspectives.
Collapse
Affiliation(s)
- Chi-Tung Cheng
- Department of Traumatology and Emergency Surgery, Chang Gung Memorial Hospital, Chang Gung University, 5 Fu-Hsing Street, Kwei-Shan District, Taoyuan, Taiwan.,Center for Artificial Intelligence in Medicine, Chang Gung Memorial Hospital, Linkou, Taoyuan, Taiwan.,Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan
| | - Chih-Chi Chen
- Department of Rehabilitation and Physical Medicine, Chang Gung Memorial Hospital, Chang Gung University, Linkou, Taoyuan, Taiwan
| | - Chih-Yuan Fu
- Department of Traumatology and Emergency Surgery, Chang Gung Memorial Hospital, Chang Gung University, 5 Fu-Hsing Street, Kwei-Shan District, Taoyuan, Taiwan
| | - Chung-Hsien Chaou
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Chang Gung University, Taoyuan, Taiwan.,Medical Education Research Center, Chang Gung Memorial Hospital, Chang Gung University, Taoyuan, Taiwan
| | - Yu-Tung Wu
- Department of Traumatology and Emergency Surgery, Chang Gung Memorial Hospital, Chang Gung University, 5 Fu-Hsing Street, Kwei-Shan District, Taoyuan, Taiwan
| | - Chih-Po Hsu
- Department of Traumatology and Emergency Surgery, Chang Gung Memorial Hospital, Chang Gung University, 5 Fu-Hsing Street, Kwei-Shan District, Taoyuan, Taiwan
| | - Chih-Chen Chang
- Department of Medical Imaging and Intervention, Chang Gung Memorial Hospital, Chang Gung University, Taoyuan, Linkou, Taiwan
| | - I-Fang Chung
- Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan
| | - Chi-Hsun Hsieh
- Department of Traumatology and Emergency Surgery, Chang Gung Memorial Hospital, Chang Gung University, 5 Fu-Hsing Street, Kwei-Shan District, Taoyuan, Taiwan
| | - Ming-Ju Hsieh
- School of Medicine, Chang Gung University, Taoyuan, Taiwan
| | - Chien-Hung Liao
- Department of Traumatology and Emergency Surgery, Chang Gung Memorial Hospital, Chang Gung University, 5 Fu-Hsing Street, Kwei-Shan District, Taoyuan, Taiwan. .,Center for Artificial Intelligence in Medicine, Chang Gung Memorial Hospital, Linkou, Taoyuan, Taiwan.
| |
Collapse
|
5
|
Artificial Intelligence and the Trainee Experience in Radiology. J Am Coll Radiol 2020; 17:1388-1393. [DOI: 10.1016/j.jacr.2020.09.028] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Revised: 08/26/2020] [Accepted: 09/01/2020] [Indexed: 12/23/2022]
|
6
|
Peterson C, Moore M, Sarwani N, Gagnon E, Bruno MA, Kanekar S. Resident-faculty overnight discrepancy rates as a function of number of consecutive nights during a week of night float. Diagnosis (Berl) 2020; 8:368-372. [DOI: 10.1515/dx-2020-0092] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2020] [Accepted: 07/27/2020] [Indexed: 11/15/2022]
Abstract
Abstract
Objectives
In 2018, the ACGME (Accreditation Council for Graduate Medical Education) made a change to the maximum permissible number of consecutive nights a resident trainee can be on “night float,” from six to seven nights. To our knowledge, although investigators have studied overall discrepancy rates and discrepancy rates as a function of shift length or perceived workload of a particular shift, no study has been performed to evaluate resident-faculty discrepancy rates as a quality/performance proxy, to see whether resident performance declines as a function of the number of consecutive nights. Our hypothesis is that we would observe a progressive increase in significant overnight resident – attending discrepancies over the 7 days’ time.
Methods
A total of 8,488 reports were extracted between 4/26/2019 to 8/22/2019 retrospectively. Data was obtained from the voice dictation system report server. Exported query was saved as a .csv file format and analyzed using Python packages. A “discrepancy checker” was created to search all finalized reports for the departmental standard heading of “Final Attending Report,” used to specify any significant changes from the preliminary interpretation.
Results
Model estimates varied on different days however there were no trends or patterns to indicate a deterioration in resident performance throughout the week. There were comparable probabilities throughout the week, with 2.17% on Monday, 2.35% on Thursday and 2.05% on Friday.
Conclusions
Our results reveal no convincing trend in terms of overnight report discrepancies between the preliminary report generated by the night float resident and the final report issued by a faculty the following morning. These results are in support of the ACGME’s recent change in the permissible number of consecutive nights on night float. We did not prove our hypothesis that resident performance on-call in the domain of report accuracy would diminish over seven consecutive nights while on the night float rotation. Our results found that performance remained fairly uniform over the course of the week.
Collapse
Affiliation(s)
- Christine Peterson
- Penn State Health Milton S Hershey Medical Center , Radiology , Hershey , PA , USA
| | - Michael Moore
- Penn State Health Milton S Hershey Medical Center , Radiology , Hershey , PA , USA
| | - Nabeel Sarwani
- Penn State Health Milton S Hershey Medical Center , Radiology , Hershey , PA , USA
| | - Eric Gagnon
- Penn State Health Milton S Hershey Medical Center , Radiology , Hershey , PA , USA
| | - Michael A. Bruno
- Penn State Health Milton S Hershey Medical Center , Radiology , Hershey , PA , USA
| | - Sangam Kanekar
- Penn State Health Milton S Hershey Medical Center , Radiology , Hershey , PA , USA
| |
Collapse
|
7
|
Jo SY, Scanlon M, Cook T. Preliminary Radiology Report Discordances and Patient Outcomes. J Am Coll Radiol 2020; 17:1621-1625. [PMID: 32768423 DOI: 10.1016/j.jacr.2019.12.033] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2019] [Revised: 12/21/2019] [Accepted: 12/23/2019] [Indexed: 11/18/2022]
Abstract
RATIONALE AND OBJECTIVES At our institution, resident and fellow radiologists issue preliminary reports for off-hours imaging studies, which are overread by attending radiologists the next day using structured discrepancy templates. In this study, we examined the impact on patient management and outcome of studies with major discordance. MATERIALS AND METHODS For our retrospective observational study, preliminary reports between March and June 2017 that received major discordance were identified through report text search. Electronic medical records were reviewed for patient management change and patient outcome. RESULTS Of the 199 cases, 52 cases (26%) had management change and 119 cases (60%) did not have management change. In 25 cases (13%), the preliminary report was proven correct on subsequent management. Three cases (2%) were lost to follow-up. In only one case was adverse outcome directly related to the discordant finding. In cases with patient management change, there was higher proportion of perceptual error compared with those without management change (73% versus 59%). In 47 cases (24%), the discordant finding or diagnosis was known to the clinical team, and better history could have avoided the major change. CONCLUSION Adverse outcome from the discordant imaging finding was low (0.5%). Major change in preliminary report could be reduced with better clinical history. Patient management change was more frequently seen with perceptual errors, placing greater emphasis on strategies to reduce them.
Collapse
Affiliation(s)
| | - Mary Scanlon
- Vice Chair of Education, Chairperson, Radiation Safety Committee, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Tessa Cook
- Assistant Professor of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; Co-Director, Center for Practice Transformation; Fellowship Director, Imaging Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania
| |
Collapse
|
8
|
Levy JL, Freeman CW, Cho JK, Iyalomhe O, Scanlon MH. Evaluating the Impact of a Call Triage Assistant on Resident Efficiency, Errors, and Stress. J Am Coll Radiol 2020; 17:414-420. [DOI: 10.1016/j.jacr.2019.11.007] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2019] [Revised: 11/02/2019] [Accepted: 11/12/2019] [Indexed: 10/25/2022]
|
9
|
Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol 2019; 92:20190389. [PMID: 31322909 DOI: 10.1259/bjr.20190389] [Citation(s) in RCA: 76] [Impact Index Per Article: 15.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023] Open
Abstract
In the era of personalized medicine, the emphasis of health care is shifting from populations to individuals. Artificial intelligence (AI) is capable of learning without explicit instruction and has emerging applications in medicine, particularly radiology. Whereas much attention has focused on teaching radiology trainees about AI, here our goal is to instead focus on how AI might be developed to better teach radiology trainees. While the idea of using AI to improve education is not new, the application of AI to medical and radiological education remains very limited. Based on the current educational foundation, we highlight an AI-integrated framework to augment radiology education and provide use case examples informed by our own institution's practice. The coming age of "AI-augmented radiology" may enable not only "precision medicine" but also what we describe as "precision medical education," where instruction is tailored to individual trainees based on their learning styles and needs.
Collapse
Affiliation(s)
- Michael Tran Duong
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - Andreas M Rauschecker
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology & Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA
| | - Jeffrey D Rudie
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Radiology & Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA
| | - Po-Hao Chen
- Imaging Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Tessa S Cook
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA
| | - R Nick Bryan
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Diagnostic Medicine, Dell Medical School, University of Texas at Austin, Austin, TX, USA
| | - Suyash Mohan
- Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA.,Department of Neurosurgery, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
10
|
Agarwal V, Bump GM, Heller MT, Chen LW, Branstetter BF, Amesur NB, Hughes MA. Resident Case Volume Correlates with Clinical Performance: Finding the Sweet Spot. Acad Radiol 2019; 26:136-140. [PMID: 30087064 DOI: 10.1016/j.acra.2018.06.023] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2018] [Revised: 06/26/2018] [Accepted: 06/27/2018] [Indexed: 10/28/2022]
Abstract
RATIONALE AND OBJECTIVES To determine whether the total number of studies interpreted during radiology residency correlates with clinical performance as measured by objective criteria. MATERIALS AND METHODS We performed a retrospective cohort study of three graduating classes of radiology residents from a single residency program between the years 2015-2017. The total number of studies interpreted by each resident during residency was tracked. Clinical performance was determined by tracking an individual resident's major discordance rate. A major discordance was recorded when there was a difference between the preliminary resident interpretation and final attending interpretation that could immediately impact patient care. Accreditation council for graduate medical education milestones at the completion of residency, Diagnostic radiology in-training scores in the third year, and score from the American board of radiology core exam were also tabulated. Pearson correlation coefficients and polynomial regression analysis were used to identify correlations between the total number of interpreted films and clinical, test, and milestone performance. RESULTS Thirty-seven residents interpreted a mean of 12,709 studies (range 8898-19,818; standard deviation [SD] 2351.9) in residency with a mean major discordance rate of 1.1% (range 0.34%-2.54%; stand dev 0.49%). There was a nonlinear correlation between total number of interpreted films and performance. As the number of interpreted films increased to approximately 16,000, clinical performance (p = 0.004) and test performance (p = 0.01) improved, but volumes over 16,000 correlated with worse performance. CONCLUSION The total number of studies interpreted during radiology training correlates with performance. Residencies should endeavor to find the "sweet spot": the amount of work that maximizes clinical exposure and knowledge without overburdening trainees.
Collapse
|
11
|
Agarwal V, Bump GM, Heller MT, Chen LW, Branstetter BF, Amesur NB, Hughes MA. Do Residency Selection Factors Predict Radiology Resident Performance? Acad Radiol 2018; 25:397-402. [PMID: 29239834 DOI: 10.1016/j.acra.2017.09.020] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2017] [Revised: 09/12/2017] [Accepted: 09/21/2017] [Indexed: 10/18/2022]
Abstract
RATIONALE AND OBJECTIVES The purpose of our study is to determine what information in medical student residency applications predicts radiology residency success as defined by objective clinical performance data. MATERIALS AND METHODS We performed a retrospective cohort study of residents who entered our institution's residency program through the National Resident Matching Program as postgraduate year 2 residents and completed the program over the past 2 years. Medical school grades, selection to Alpha Omega Alpha (AOA) Honor Society, United States Medical Licensing Examination (USMLE) scores, publication in peer-reviewed journals, and whether the applicant was from a peer institution were the variables examined. Clinical performance was determined by calculating each resident's cumulative major discordance rate for on-call cases the resident read and gave a preliminary interpretation. A major discordance was defined as a difference between the preliminary resident and the final attending interpretations that could immediately impact the care of the patient. A multivariate logistic regression was performed to determine significant variables. RESULTS Twenty-seven residents provided preliminary reports on call for 67,145 studies. The mean major discordance rate was 1.08% (range 0.34%-2.54%). Higher USMLE Step 1 scores, publication before residency, and election to AOA Honor Society were all statistically significant predictors of lower major discordance rates (P values 0.01, 0.01, and <0.001, respectively). CONCLUSIONS Overall resident performance was excellent. There are predictors that help select the better performing residents, namely higher USMLE Step 1 scores, one to two publications during medical school, and election to AOA in the junior year of medical school.
Collapse
|