Are the Pilots Onboard? Equipping Radiologists for Clinical Implementation of AI

The incorporation of artificial intelligence into radiological clinical workflow is on the verge of being realized. To ensure that these tools are effective, measures must be taken to educate radiologists on tool performance and failure modes. Additionally, radiology systems should be designed to avoid automation bias and the potential decline in radiologist performance. Designed solutions should cater to every level of expertise so that patient care can be enhanced and risks reduced. Ultimately, the radiology community must provide education so that radiologists can learn about algorithms, their inputs and outputs, and potential ways they may fail. This manuscript will present suggestions on how to train radiologists to use these new digital systems, how to detect AI errors, and how to maintain underlying diagnostic competency when the algorithm fails.

Formation and Assessment of a Laryngology Pathology Video Atlas for Resident Education

OBJECTIVE

Conventional reference images of laryngeal pathologies may provide educational value for Otolaryngology-Head & Neck Surgery (OHNS) residents, but observing dynamic vocal fold function is critical for diagnosis. Our aim was to develop and validate a video atlas of laryngeal pathologies for resident education in OHNS.

DESIGN

A multi-institution, prospective case-control study.

SETTING/PARTICIPANTS

Ten videos showing 10 representative laryngeal pathologies were verified by two laryngologists. Six videos per category with kappa>0.8 were included in the video database. A collection of the videos was shown to a group of OHNS residents in a quiz fashion to determine if senior trainees would score higher than junior trainees. Another group of residents in OHNS was recruited and randomized to control or intervention. The control group was shown a quiz of 10 laryngeal videos at baseline and 24 weeks later. The intervention group was shown quizzes at baseline and every 6 weeks, ending at 24 weeks. Free-text diagnoses were scored for accuracy. Descriptive statistics, two-tailed tests, and analysis of covariance were performed.

RESULTS

Twenty-nine residents participated, with 14 (48.3%) randomized to control, and 15 (51.7%) to the intervention. The postgraduate year (PGY) level had a significant impact on diagnostic performance. PGY1 and 2 had a significantly lower score than PGY5 (P = 0.017 and P = 0.035, respectively). PGY3 and PGY4 scores were not statistically different from PGY5 scores. The mean score difference between groups decreases as the PGY level increases (mean difference between groups = 0.87, P = 0.153), but this was not significant.

CONCLUSIONS

The current study has created a validated collection of videos that are representative of common laryngeal pathologies and can be easily incorporated into resident video-based learning. Future directions include larger multi-site studies to further elucidate whether repeated viewing of this video atlas can improve OHNS resident laryngology knowledge.

Artificial Intelligence-Powered Clinical Decision Support and Simulation Platform for Radiology Trainee Education

Technological tools can redesign traditional approaches to radiology education, for example, with simulation cases and via computer-generated feedback. In this study, we investigated the use of an AI-powered, Bayesian inference-based clinical decision support (CDS) software to provide automated "real-time" feedback to trainees during interpretation of clinical and simulation brain MRI examinations. Radiology trainees participated in sessions in which they interpreted 3 brain MRIs: two cases from a routine clinical worklist (one without and one with CDS) and a teaching file-based simulation case with CDS. The CDS software required trainees to input imaging features and differential diagnoses, after which inferred diagnoses were displayed, and the case was reviewed with an attending neuroradiologist. An observer timed each case, including time spent on education, and trainees completed a survey rating their confidence in their findings and the educational value of the case. Ten trainees reviewed 75 brain MRI examinations during 25 reading sessions. Trainees had slightly lower confidence in their findings and diagnosis and rated the educational value slightly higher for simulation cases with CDS compared to clinical cases without CDS (p < 0.05). There were no significant differences in ratings of clinical cases with or without CDS. No differences in overall timing were found among the reading scenarios. Simulation cases with "CDS-provided feedback" may improve the educational value of interpreting imaging studies at a workstation without adding additional time. Further investigation will help drive innovation in trainee education, which may be particularly relevant in this era of increasing remote work and asynchronous attending review.

I.C.A.R.U.S. in Flight: A Radiology Simulator Teaches Imaging Appropriateness, Anatomy, and Image Interpretation Skills

RATIONALE AND OBJECTIVES

The Interactive Clinical Anatomy and Radiology Utilization Simulator (ICARUS) was created in 2012 as a novel simulation program with 25 peer reviewed electronic modules combining simulated patient encounters, American College of Radiology appropriateness criteria, radiologic anatomy, and basic imaging interpretation and patient safety topics. ICARUS integrates a gaming approach to improve student engagement.

MATERIALS AND METHODS

ICARUS modules have been used in the undergraduate medical curriculum at the lead author's institution since its pilot in 2012. Since completion of the full complement of 25 modules, modules have been used in the radiology clerkship (2012- 2015), in the integrated longitudinal curriculum (2015-present), and in various radiology electives (2015-present), including the inaugural virtual radiology electives at all authors' institutions (July 2020-present). Student evaluations were formally solicited in 2014 and again in 2020.

RESULTS

During the introductory radiology clerkship in 2014, 107 students were enrolled. Students were assigned 4 ICARUS modules. During the 2-week virtual elective from July-October 2020, 26 students were enrolled. Students were assigned all 25 modules. The majority of survey respondents "agreed" or "strongly agreed" that the modules were interactive, enjoyable, and self-directed, fostered critical thinking, provided practical value, met goals and objectives of the course, and seemed practical to their future practice.

CONCLUSION

ICARUS integrates key educational concepts in radiology with high fidelity simulation of clinical decision support software and PACS simulation image display to create a highly engaging learning environment that most accurately reflects future clinical experiences.

A narrative review of e-learning in professional education of healthcare professionals in medical imaging and radiation therapy

OBJECTIVES

This literature review attempts to explore the characteristics of e-learning tools used to develop the qualifications and skills of healthcare professionals in medical imaging and radiation therapy, and to promote the effectiveness and acceptance of e-learning through highlighting the outcomes of its implementation where applicable.

KEY FINDINGS

From the literature search in the PubMed and ResearchGate databases we concluded to 21 articles, which were included in the qualitative synthesis. Acceptance of e-learning tools was confirmed. Also, e-learning can be part of healthcare professionals' blended learning. The acquisition of new or improvement of existing knowledge, the improvement of clinical skills and the increase of the self-confidence of healthcare professionals in their daily practice were recorded, as outcomes of the e-learning implementation. The importance of human-computer interaction for the comprehension of theoretical concepts and practical aspects using multimedia was also captured. No significant findings emerged among the 21 articles against the adoption of the e-learning for the training of healthcare professionals. The Internet is the channel used for synchronous and asynchronous interaction of trainees with instructors.

CONCLUSIONS

We concluded that e-learning is an attractive training method, equally or occasionally more effective than the traditional educational methods for the lifelong training of healthcare professionals in the field of medical imaging and radiation therapy. Also, many collaborative web-based applications provide the necessary means to build an e-learning program, according to the training needs of each professional team.

IMPLICATIONS FOR PRACTICE

This new knowledge corroborates the perspective of e-learning beneficial contribution to remote interaction and collaboration of healthcare professionals in medical imaging and radiation therapy. Collaborative web-based tools are already available to decision makers and stakeholders, who want to develop an e-learning program.

Making Learning Fun: Gaming in Radiology Education

With continued technologic advances, it is not surprising that gaming techniques are increasingly being used in radiology residency programs. This comprehensive review on gaming in radiology education offers insight into the importance of gaming, types of games and principles utilized in gaming, as well as applications that are inherent in artificial intelligence and continued medical education. The advantages and disadvantages of gaming will be considered, as well as barriers to successful adoption of gaming.

Assessing Competence in Emergency Radiology Using an Online Simulator

RATIONALE AND OBJECTIVES

Traditional assessments in radiology residency focus on the Medical Expert CanMEDS role and typically rely upon a single or limited static images. We designed an Emergency Radiology Simulator that aimed to assess the breadth of competencies required across Medical and NonMedical Expert domains.

MATERIAL AND METHODS

An online simulator with typical emergency cases was administered in October 2015 to Post Graduate Year (PGY) 2-5 residents in Radiology. Residents provided preliminary reports, which were graded for style and content. The simulation also included prioritization, protocoling, counseling, and handover exercises geared to assess NonMedical Expert roles.

RESULTS

Fourty eight residents participated in the simulation. Level of resident was 11 PGY-2, 17 PGY-3, 13 PGY-4, and 7 PGY-5. There was a significant difference in resident performance between PGY-2 residents and those more senior in terms of the Medical Expert role (findings, diagnosis, recommendations, and clinical relevance of reports). Differences in performance between PGY levels were not seen in the NonMedical Expert roles (prioritization, protocoling, counseling, and handover).

CONCLUSION

Simulation provides an opportunity to assess radiology resident performance across multiple domains. PGY-2 residents performed worse on the Medical Expert domains, although performance did not significantly vary between the other years. This may suggest that competence in Emergency Radiology is achieved early in residency, possibly related to the importance placed on developing skills related to on-call performance during the PGY-2 year. The simulator should be extended to other areas of Radiology, in order to assess the ability to discriminate performance in other subspecialties.

Spaced radiology: encouraging durable memory using spaced testing in pediatric radiology

Applied memory research in the field of cognitive and educational psychology has generated a large body of data to support the use of spacing and testing to promote long-term or durable memory. Despite the consensus of this scientific community, most learners, including radiology residents, do not utilize these tools for learning new information. We present a discussion of these parallel and synergistic learning techniques and their incorporation into a software platform, called Spaced Radiology, which we created for teaching radiology residents. Specifically, this software uses these evidence-based strategies to teach pediatric radiology through a flashcard deck system.

Cognitive processing differences of experts and novices when correlating anatomy and cross-sectional imaging

The ability to correlate anatomical knowledge and medical imaging is crucial to radiology and as such, should be a critical component of medical education. However, we are hindered in our ability to teach this skill because we know very little about what expert practice looks like, and even less about novices' understanding. Using a unique simulation tool, this research conducted cognitive clinical interviews with experts and novices to explore differences in how they engage in this correlation and the underlying cognitive processes involved in doing so. This research supported what has been known in the literature, that experts are significantly faster at making decisions on medical imaging than novices. It also offers insight into the spatial ability and reasoning that is involved in the correlation of anatomy to medical imaging. There are differences in the cognitive processing of experts and novices with respect to meaningful patterns, organized content knowledge, and the flexibility of retrieval. Presented are some novice-expert similarities and differences in image processing. This study investigated extremes, opening an opportunity to investigate the sequential knowledge acquisition from student to resident to expert, and where educators can help intervene in this learning process.

Determining procedures for simulation-based training in radiology: a nationwide needs assessment

OBJECTIVES

New training modalities such as simulation are widely accepted in radiology; however, development of effective simulation-based training programs is challenging. They are often unstructured and based on convenience or coincidence. The study objective was to perform a nationwide needs assessment to identify and prioritize technical procedures that should be included in a simulation-based curriculum.

METHODS

A needs assessment using the Delphi method was completed among 91 key leaders in radiology. Round 1 identified technical procedures that radiologists should learn. Round 2 explored frequency of procedure, number of radiologists performing the procedure, risk and/or discomfort for patients, and feasibility for simulation. Round 3 was elimination and prioritization of procedures.

RESULTS

Response rates were 67 %, 70 % and 66 %, respectively. In Round 1, 22 technical procedures were included. Round 2 resulted in pre-prioritization of procedures. In round 3, 13 procedures were included in the final prioritized list. The three highly prioritized procedures were ultrasound-guided (US) histological biopsy and fine-needle aspiration, US-guided needle puncture and catheter drainage, and basic abdominal ultrasound.

CONCLUSION

A needs assessment identified and prioritized 13 technical procedures to include in a simulation-based curriculum. The list may be used as guide for development of training programs.

KEY POINTS

• Simulation-based training can supplement training on patients in radiology. • Development of simulation-based training should follow a structured approach. • The CAMES Needs Assessment Formula explores needs for simulation training. • A national Delphi study identified and prioritized procedures suitable for simulation training. • The prioritized list serves as guide for development of courses in radiology.

Simulation based teaching in interventional radiology training: is it effective?

AIM

To establish the educational effectiveness of simulation teaching in interventional radiology training.

MATERIALS AND METHODS

Electronic databases (MEDLINE, ERIC, Embase, OvidSP, and Cochrane Library) were searched (January 2000 to May 2015). Studies specifically with educational outcomes conducted on radiologists were eligible. All forms of simulation in interventional training were included. Data were extracted based on the population, intervention, comparison, and outcome (PICO) model. Kirkpatrick's hierarchy was used to establish educational intervention effectiveness. The quality of studies was assessed using the Cochrane risk of bias tool.

RESULTS

Search resulted in 377 articles, of which 15 met the inclusion criteria. Thirteen of the 15 studies achieved level 2 of Kirkpatrick's hierarchy with only one reaching level 4. Statistically significant improvements in performance metrics as objective measures, demonstrating trainee competence were seen in 12/15 studies. Subjective improvements in confidence were noted in 13/15. Only one study demonstrated skills transferability and improvements in patient outcomes.

CONCLUSION

Results demonstrate the relevance of simulated training to current education models in improving trainee competence; however, this is limited to the simulated environment as there is a lack of literature investigating its predictive validity and the effect on patient outcomes. The requirement for further research in this field is highlighted. Simulation is thus currently only deemed useful as an adjunct to current training models with the potential to play an influential role in the future of the interventional radiology training curriculum.

Why Isn't There More High-fidelity Simulation Training in Diagnostic Radiology? Results of a Survey of Academic Radiologists

RATIONALE AND OBJECTIVES

Despite its increasing use in training other medical specialties, high-fidelity simulation to prepare diagnostic radiology residents for call remains an underused educational resource. To attempt to characterize the barriers toward adoption of this technology, we conducted a survey of academic radiologists and radiology trainees.

MATERIALS AND METHODS

An Institutional Review Board-approved survey was distributed to the Association of University Radiologists members via e-mail. Survey results were collected electronically, tabulated, and analyzed.

RESULTS

A total of 68 survey responses representing 51 programs were received from program directors, department chairs, chief residents, and program administrators. The most common form of educational activity for resident call preparation was lectures. Faculty supervised "baby call" was also widely reported. Actual simulated call environments were quite rare with only three programs reporting this type of educational activity. Barriers to the use of simulation include lack of faculty time, lack of faculty expertise, and lack of perceived need.

CONCLUSIONS

High-fidelity simulation can be used to mimic the high-stress, high-stakes independent call environment that the typical radiology resident encounters during the second year of training, and can provide objective data for program directors to assess the Accreditation Council of Graduate Medical Education milestones. We predict that this technology will begin to supplement traditional diagnostic radiology teaching methods and to improve patient care and safety in the next decade.

PACS used while on-call: a national survey of radiology program directors and chief residents

The aim of this study was to determine the prevalence of different diagnostic image viewing platforms used by radiologists while on-call, and to assess the opinions and preferences of radiology program directors and chief residents regarding their use. An online survey was sent electronically to radiology residency program directors and chief residents via the Association of University Radiologists. Forty-two radiology program directors and 25 chief residents completed the survey, yielding response rates of 24.9 and 8.5 %, respectively. From the survey results, 10 different Picture Archiving Communications Systems (PACS) were identified; GE (25 %), Philips (17 %), and Agfa Impax (15 %) were the most prevalent. Interestingly, only 5 % of all respondents use a secondary "Digital Imaging and Communications in Medicine" viewer for on-call studies. Perceptions of PACS functionality were generally neutral to weakly positive. Most respondents strongly agreed that it is important to have a single integrated PACS for viewing on-call studies and agreed that the PACS should be integrated into the Electronic Medical Records (EMR). The overwhelming majority of respondents use their institution's PACS while on-call. The results show there is still a wide variety of PACS platforms used by different institutions; however, GE, Phillips, and Agfa were some of the most prevalent. Most radiologists surveyed have neutral to slightly positive perceptions about the functionality and ease of use of their PACS. Finally, while radiologists agree that PACS should be integrated with EMR, only 53 % of respondents currently have this arrangement.

Conventional Medical Education and the History of Simulation in Radiology

Simulation is a promising method for improving clinician performance, enhancing team training, increasing patient safety, and preventing errors. Training scenarios to enrich medical student and resident education, and apply toward competency assessment, recertification, and credentialing are important applications of simulation in radiology. This review will describe simulation training for procedural skills, interpretive and noninterpretive skills, team-based training and crisis management, professionalism and communication skills, as well as hybrid and in situ applications of simulation training. A brief overview of current simulation equipment and software and the barriers and strategies for implementation are described. Finally, methods of measuring competency and assessment are described, so that the interested reader can successfully implement simulation training into their practice.

Bedside ultrasound training using web-based e-learning and simulation early in the curriculum of residents

BACKGROUND

Focused bedside ultrasound is rapidly becoming a standard of care to decrease the risks of complications related to invasive procedures. The purpose of this study was to assess whether adding to the curriculum of junior residents an educational intervention combining web-based e-learning and hands-on training would improve the residents' proficiency in different clinical applications of bedside ultrasound as compared to using the traditional apprenticeship teaching method alone.

METHODS

Junior residents (n = 39) were provided with two educational interventions (vascular and pleural ultrasound). Each intervention consisted of a combination of web-based e-learning and bedside hands-on training. Senior residents (n = 15) were the traditionally trained group and were not provided with the educational interventions.

RESULTS

After the educational intervention, performance of the junior residents on the practical tests was superior to that of the senior residents. This was true for the vascular assessment (94% ± 5% vs. 68% ± 15%, unpaired student t test: p < 0.0001, mean difference: 26 (95% CI: 20 to 31)) and even more significant for the pleural assessment (92% ± 9% vs. 57% ± 25%, unpaired student t test: p < 0.0001, mean difference: 35 (95% CI: 23 to 44)). The junior residents also had a significantly higher success rate in performing ultrasound-guided needle insertion compared to the senior residents for both the transverse (95% vs. 60%, Fisher's exact test p = 0.0048) and longitudinal views (100% vs. 73%, Fisher's exact test p = 0.0055).

CONCLUSIONS

Our study demonstrated that a structured curriculum combining web-based education, hands-on training, and simulation integrated early in the training of the junior residents can lead to better proficiency in performing ultrasound-guided techniques compared to the traditional apprenticeship model.

Pulmonary embolism teaching file: a simple pilot study for rapidly increasing pulmonary embolism recognition among new residents using interactive cross-sectional imaging

RATIONALE AND OBJECTIVES

Chest radiographs can be demanding, making this an area of focus during most first-year resident chest rotations. This often comes at a cost of cross-sectional imaging, and new residents are often not initially comfortable with reading chest computed tomographic angiograms (CTAs) for pulmonary embolisms (PEs). We created a teaching file of CTAs to improve the detection of PEs.

MATERIALS AND METHODS

For initial testing, we used videos of 25 cases, which played for 90 seconds (to allow multiple passes) to residents with and without call experience. The presence and location of PEs and the readers' confidence scores were recorded. After initial testing, first-year residents without call experience were given 20 separate known positive CTA videos to scroll through on their own. The goal of this was to allow for individual review and development of individual search strategies. A second testing was done with all levels of residents with the same initial 25 cases, re-randomized to evaluate for improvement.

RESULTS

Initially, first-year residents without call experience identified an average of 14.7 of 18 examinations positive for PEs (versus 15.8 for more senior residents; P < .04). After reviewing the 20 known positive cases, the first-year residents improved, averaging 16.6 (versus 14.7 earlier; P < .01).

CONCLUSIONS

We created a fast, simple way to expose novice residents to CTA examinations and increase their accuracy in identifying PEs. After using a teaching file, the ability to recognize PEs improved significantly, and scores were no longer significantly different from those of residents with call experience.

E-learning and education in radiology

PURPOSE

To evaluate current applications of e-learning in radiology.

MATERIAL AND METHODS

A Medline search was performed using PubMed (National Library of Medicine, Bethesda, MD) for publications discussing the applications of e-learning in radiology. The search strategy employed a single combination of the following terms: (1) e-learning, and (2) education and (3) radiology. This review was limited to human studies and to English-language literature. We reviewed all the titles and subsequent the abstract of 29 articles that appeared pertinent. Additional articles were identified by reviewing the reference lists of relevant papers. Finally, the full text of 38 selected articles was reviewed.

RESULTS

Literature data shows that with the constant development of technology and global spread of computer networks, in particular of the Internet, the integration of multimedia and interactivity introduced into electronic publishing has allowed the creation of multimedia applications that provide valuable support for medical teaching and continuing medical education, specifically for radiology. Such technologies are valuable tools for collaboration, interactivity, simulation, and self-testing. However, not everything on the World Wide Web is useful, accurate, or beneficial: the quality and veracity of medical information on the World Wide Web is variable and much time can be wasted as many websites do not meet basic publication standards.

CONCLUSION

E-learning will become an important source of education in radiology.

The use of a simulation center to improve resident proficiency in performing ultrasound-guided procedures

RATIONALE AND OBJECTIVES

With advancements in technology and push for health care reform and reduced costs, minimally invasive procedures, such as those that are ultrasound-guided, have become an essential part of radiology, and are used in many divisions of radiology. By incorporating standardized training methodologies in a risk free environment through utilization of a simulation center with phantom training, we hope to improve proficiency and confidence in procedural performance.

MATERIALS AND METHODS

Twenty-nine radiology residents from four levels of training were enrolled in this prospective study. The residents were given written, video, and live interactive training on the basics of ultrasound-guided procedures in our simulation center on a phantom mannequin. All of the teaching materials were created by residents and staff radiologists at the institution.

RESULTS

Residents demonstrated statistically significant improvement (P < .05) between their pre- and posttest scores on both the written and practical examinations. They also showed a trend toward improved dexterity in the technical aspects of ultrasound-guided procedures (P = .07) after training. On the survey questionnaire, residents confirm improved knowledge level, technical ability, and confidence levels pertaining to ultrasound-guided procedures.

CONCLUSIONS

The use of controlled simulation based training can be an invaluable tool to improve the knowledge level, dexterity, and confidence of residents performing ultrasound-guided procedures. Additionally, a simulation model allows standardization of education.

Preparing first-year radiology residents and assessing their readiness for on-call responsibilities: results over 5 years

OBJECTIVE

The objective of our study was to evaluate the preparedness of postgraduate year (PGY)-2 residents for independent call responsibilities and the impact of the radiology residency training program on call preparedness using an objective DICOM-based simulation module over a 5-year period.

MATERIALS AND METHODS

A month-long emergency radiology lecture series, conducted over 5 consecutive years, was designed and given to radiology residents at all levels. A DICOM-based, interactive, computer-based testing module with actual emergency department cases was developed and administered at the end of the lecture series. Comparison was made between first-year and upper-level resident test scores using a Student's t test, generalized estimating equations, and individual fixed effects to determine PGY-2 residents' before-call preparedness and the effectiveness of the simulation module to assess call preparedness. Resident scoring on the simulation module was also plotted as a function of progression through their residency program to evaluate the impact of the training program on call preparedness.

RESULTS

Over 5 years, 45 PGY-2, 34 PGY-3, 32 PGY-4, and 35 PGY-5 residents attended the lecture series and completed the computer-based testing module. PGY-2 residents scored an average of 71% +/- 15% (SD), PGY-3 residents scored 79% +/- 11%, PGY-4 residents scored 84% +/- 10%, and PGY-5 residents scored 86% +/- 11% of the total points possible. A statistically significant (p < 0.05) difference in scoring on the simulation module was identified between the PGY-2 residents and each upper-level class over the 5-year period and during 4 of 5 examination years analyzed separately. A trend toward higher average scores for each cohort of residents as they progressed through residency training was identified.

CONCLUSION

Over a 5-year period, first-year radiology residents scored significantly lower than upper-level colleagues on an emergency radiology simulation module, suggesting a significant improvement in the ability of residents to interpret typical on-call imaging studies after the PGY-2 year.