Part I: preparing first-year radiology residents and assessing their readiness for on-call responsibilities

Interactive Web-based Emergency Neuroradiology Course with Self-Assessment for Radiology Residents: A Pilot Project

PURPOSE

To assess the effectiveness of an online, case-based interactive course in emergency neuroradiology to prepare radiology residents for night call in neuroradiology.

METHODS

A total of 15 residents participated in a pretest assessment of preparedness for neuroradiology call. After completing a 6-week interactive course incorporating case review, didactic lectures, quiz feedback, and references for further review, the same residents were quizzed on cognitive questions and feelings of readiness to enter the on-call pool for neuroradiology.

RESULTS

Knowledge and confidence both significantly increased due to the course. Knowledge-wise, the scores for precourse quiz to postcourse quiz went from 18.4%-72.2% (53.8% increase) in Brain Imaging and went from 22.3%-77.1% (54.8% increase) in Head, Neck, and Spine (P < 0.001 for both). Confidence-wise, residents demonstrated statistically significant increases in all 6 confidence measures. Prior to the course, 29% were not confident, 71% were fairly confident, and 0% were confident/very confident. After the course, 0% were not confident, 43% were fairly confident, and 57% were confident/very confident. Belief in the statement "I can provide high quality Neuroimaging services in the emergency care setting" increased from a confidence score of 1.29-2.57 after training (P = 0.004). Nearly all residents completing their first emergency call reported that they felt more confident reading neuroradiology studies during their call as a direct result of the course.

CONCLUSIONS

Completing the multi-pronged interactive, case-based online emergency neuroradiology course led to improved funds of knowledge and feelings of confidence and impacted imaging approach in residents taking neuroradiology call.

Preparing junior radiology residents for overnight call via peer-led, hands-on simulation

PURPOSE

The aim of this study was to assess the perceived value and impact of a hands-on mock call simulation program on resident confidence with interpretation of emergency department overnight call cases.

METHODS

A five-session course was implemented in June of 2018 for rising PGY-3/R2 residents to mimic the experience of overnight call with indirect supervision. Sessions were led by senior residents in the program and consisted of timed, independent interpretation of 15-20 high-yield cases per day which highlighted "do-not miss" critical findings and simulated workflow interruptions including phone calls, consultations, and questions from technologists. IRB-approved, and anonymous pre- and post-course surveys were administered to participants which assessed residents' degree of confidence in interpretation of on-call cases and comparison of the mock call experience with existing preparatory strategies. Survey responses were analyzed using McNemar's test and Mann-Whitney U test.

RESULTS

Our survey response rate was 91% (29/32). After completing the mock call simulation, there was a significant increase in the mean Likert score of resident confidence levels and feelings of preparedness from 4.59 to 7.38 (p < 0.01). The majority of respondents (72.4% [21/29]) felt that the mock call simulation was "extremely useful." One hundred percent of respondents indicated that the mock call simulation should be implemented for the following year.

CONCLUSION

Implementation of a hands-on mock call simulation significantly improves the confidence levels of radiology residents before assuming on-call responsibilities and may serve as an adjunct to existing preparatory strategies.

Effectiveness of a 12-Month Emergency Radiology Curriculum for Improving Self-Confidence and Competence of Postgraduate Year 1 Radiology Residents-A Canadian Study

PURPOSE

Limited radiology curriculum during postgraduate year 1 of radiology residency combined with increasing workloads during emergency radiology call have contributed to heightened anxiety and responsibilities for junior residents. This study aimed to evaluate the effectiveness of a 12-month emergency radiology curriculum on self-rated confidence and general competence of Canadian postgraduate year 1 radiology residents.

METHODS

A cohort of Canadian postgraduate year 1 Diagnostic Radiology residents voluntarily enrolled in a 12-month self-directed online emergency radiology curriculum (9 modules). Participants completed pretest and posttest surveys and examinations to gauge their self-rated competence on module material and knowledge acquisition, respectively. Average pretest and posttest scores were compared using Student 2-tailed unpaired t test, and Likert data from self-reported confidence were compared using a Mann Whitney U test. Statistical significance was defined as P < .05.

RESULTS

Sixty-six trainees completed at least 1 module, and 15 trainees completed all 9 modules. Both self-rated confidence and posttest scores were statistically higher after module completion (P < .001) for all 9 learning modules. The greatest improvement in test scores was seen in the female genitourinary module (28.12 ± 3.018; difference between pretest and posttest means ± SEM).

CONCLUSIONS

Our study demonstrates learning benefits for junior radiology trainees who participated in a self-directed online emergency radiology curriculum during postgraduate year 1. In the face of ever-increasing demands for imaging in on-call settings across Canada, inclusion of a self-directed online curriculum may become more important for upcoming competency-based medical education as it encourages a learner-driven and non-time-based method of education.

Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 1: Fixed Effects Identify Influences of Exam, Specialty, Fatigue, and Training on Resident Performance

RATIONALE AND OBJECTIVES

To describe our full-resolution simulation of critical care imaging coupled with posthoc grading of resident's interpretations and present results from the fixed effects terms in a comprehensive mixed regression model of the resulting scores.

MATERIALS AND METHODS

The system delivered full resolution DICOM studies via clinical-grade viewing software integrated with a custom built web-based workflow and reporting system. The interpretations submitted by participating residents from 47 different programs were graded (scores of 0-10) on a case by case basis by a cadre of faculty members from our department. The data from 5 yearly (2014-2018) cycles consisting of 992 separate 65 case, 8 hour simulation sessions were collated from the transaction records. We used a mixed (hierarchical) statistical model with nine fixed and four random independent variables. In this paper, we present the results from the nine fixed effects.

RESULTS

There were 19,916/63,839 (27.0%, CI 26.7%-27.4%) scores in the 0-2 range (i.e., clinically significant miss). Neurological cases were more difficult with adjusted scores 2.3 (CI 1.9-3.2) lower than body/musculoskeletal cases. There was a small (0.3, CI 0.20-0.38 points) but highly significant (p<0.0001) decrease in score for the final 13/65 cases (fifth quintile) as evidence of fatigue during the last hour of an 8 hour shift. By comparing adjusted scores from mid-R1 (quarter 3) to late-R3 (quarter 12) we estimate the training effect as an increase of 2.2 (CI 1.90-2.50) points.

CONCLUSION

Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. Analysis of the resulting scores yields multiple insights into the interpretative process.

Reshaping Radiology Precall Preparation: Integrating a Cloud-Based PACS Viewer Into a Flipped Classroom Model

Preparing residents for the on-call experience in Radiology is one of the most important aspects of education within a training program. Traditionally, this preparation has occurred via a combination of case conferences and didactic lectures by program faculty, daily teaching at the workstation, and precall assessments. Recently, a blended curricular model referred to as the flipped classroom has generated a lot of attention within the realm of graduate medical education. We applied this technique to resident precall education in the subspecialty of Neuroradiology, and surveyed the participants about their perceptions of the course. The structure, implementation, and web-based platform used to create the flipped classroom experience is described herein.

Integration of a Zero-footprint Cloud-based Picture Archiving and Communication System with Customizable Forms for Radiology Research and Education

RATIONALE AND OBJECTIVES

The purpose of this study was to integrate web-based forms with a zero-footprint cloud-based Picture Archiving and Communication Systems (PACS) to create a tool of potential benefit to radiology research and education.

MATERIALS AND METHODS

Web-based forms were created with a front-end and back-end architecture utilizing common programming languages including Vue.js, Node.js and MongoDB, and integrated into an existing zero-footprint cloud-based PACS.

RESULTS

The web-based forms application can be accessed in any modern internet browser on desktop or mobile devices and allows the creation of customizable forms consisting of a variety of questions types. Each form can be linked to an individual DICOM examination or a collection of DICOM examinations.

CONCLUSIONS

Several uses are demonstrated through a series of case studies, including implementation of a research platform for multi-reader multi-case (MRMC) studies and other imaging research, and creation of an online Objective Structure Clinical Examination (OSCE) and an educational case file.

Cognitive processing differences of experts and novices when correlating anatomy and cross-sectional imaging

The ability to correlate anatomical knowledge and medical imaging is crucial to radiology and as such, should be a critical component of medical education. However, we are hindered in our ability to teach this skill because we know very little about what expert practice looks like, and even less about novices' understanding. Using a unique simulation tool, this research conducted cognitive clinical interviews with experts and novices to explore differences in how they engage in this correlation and the underlying cognitive processes involved in doing so. This research supported what has been known in the literature, that experts are significantly faster at making decisions on medical imaging than novices. It also offers insight into the spatial ability and reasoning that is involved in the correlation of anatomy to medical imaging. There are differences in the cognitive processing of experts and novices with respect to meaningful patterns, organized content knowledge, and the flexibility of retrieval. Presented are some novice-expert similarities and differences in image processing. This study investigated extremes, opening an opportunity to investigate the sequential knowledge acquisition from student to resident to expert, and where educators can help intervene in this learning process.

Why Isn't There More High-fidelity Simulation Training in Diagnostic Radiology? Results of a Survey of Academic Radiologists

RATIONALE AND OBJECTIVES

Despite its increasing use in training other medical specialties, high-fidelity simulation to prepare diagnostic radiology residents for call remains an underused educational resource. To attempt to characterize the barriers toward adoption of this technology, we conducted a survey of academic radiologists and radiology trainees.

MATERIALS AND METHODS

An Institutional Review Board-approved survey was distributed to the Association of University Radiologists members via e-mail. Survey results were collected electronically, tabulated, and analyzed.

RESULTS

A total of 68 survey responses representing 51 programs were received from program directors, department chairs, chief residents, and program administrators. The most common form of educational activity for resident call preparation was lectures. Faculty supervised "baby call" was also widely reported. Actual simulated call environments were quite rare with only three programs reporting this type of educational activity. Barriers to the use of simulation include lack of faculty time, lack of faculty expertise, and lack of perceived need.

CONCLUSIONS

High-fidelity simulation can be used to mimic the high-stress, high-stakes independent call environment that the typical radiology resident encounters during the second year of training, and can provide objective data for program directors to assess the Accreditation Council of Graduate Medical Education milestones. We predict that this technology will begin to supplement traditional diagnostic radiology teaching methods and to improve patient care and safety in the next decade.

Conventional Medical Education and the History of Simulation in Radiology

Simulation is a promising method for improving clinician performance, enhancing team training, increasing patient safety, and preventing errors. Training scenarios to enrich medical student and resident education, and apply toward competency assessment, recertification, and credentialing are important applications of simulation in radiology. This review will describe simulation training for procedural skills, interpretive and noninterpretive skills, team-based training and crisis management, professionalism and communication skills, as well as hybrid and in situ applications of simulation training. A brief overview of current simulation equipment and software and the barriers and strategies for implementation are described. Finally, methods of measuring competency and assessment are described, so that the interested reader can successfully implement simulation training into their practice.

Description of a formal clinical pathology on-call simulation workshop in preparing pathology residents to address common after-hours phone calls

OBJECTIVES

Residents take clinical pathology call beginning in the second year of residency. However, this additional responsibility often causes anxiety in residents who may have had only limited contact with laboratory sections they will be covering on call. We set out to improve the practical knowledge and comfort of the residents before taking clinical pathology call.

METHODS

A scenario-based simulation-style workshop was developed to acquaint residents with the common issues that arise, and guide them through acquiring the necessary data and developing an action plan. A nine-question survey was given before and after the workshop to determine if the workshop improved their comfort level with taking call.

RESULTS

Of the nine questions, the six questions dealing with clinical pathology laboratory section-specific knowledge showed that residents were less anxious and more confident about taking call after the workshop.

CONCLUSIONS

A scenario-based workshop is an effective way to acquaint residents with the basics of taking call, and teaches practical approaches to common clinical pathology issues.

Assessing first year radiology resident competence pre-call: development and implementation of a computer-based exam before and after the 12 month training requirement

RATIONALE AND OBJECTIVES

Whether first-year radiology residents are ready to start call after 6 or 12 months has been a subject of much debate. The purpose of this study was to establish an assessment that would evaluate the call readiness of first-year radiology residents and identify any individual areas of weakness using a comprehensive computerized format. Secondarily, we evaluated for any significant differences in performance before and after the change in precall training requirement from 6 to 12 months.

MATERIALS AND METHODS

A list of >140 potential emergency radiology cases was given to first-year radiology residents at the beginning of the academic year. Over 4 years, three separate versions of a computerized examination were constructed using hyperlinked PowerPoint presentations and given to both first-year and second-year residents. No resident took the same version of the exam twice. Exam score and number of cases failed were assessed. Individual areas of weakness were identified and remediated with the residents. Statistical analysis was used to evaluate exam score and the number of cases failed, considering resident year and the three versions of the exam.

RESULTS

Over 4 years, 17 of 19 (89%) first-year radiology residents passed the exam on first attempt. The two who failed were remediated and passed a different version of the exam 6 weeks later. Using the oral board scoring system, first-year radiology residents scored an average of 70.7 with 13 cases failed, compared to 71.1 with eight cases failed for second-year residents who scored statistically significantly higher. No significant difference was found in first-year radiology resident scoring before and after the 12-month training requirement prior to call.

CONCLUSIONS

An emergency radiology examination was established to aid in the assessment of first-year radiology residents' competency prior to starting call, which has become a permanent part of the first-year curriculum. Over 4 years, all first-year residents were ultimately judged ready to start call. Of the variables assessed, only resident year showed a significant difference in scoring parameters. In particular, length of training prior to taking call showed no significant difference. Areas of weakness were identified for further study.

The key role of a transition course in preparing medical students for internship

Among the core transitions in medical education is the one from medical school to residency. Despite this challenging transition, the final year of medical school is known as lacking structure and clarity. The authors examine the preparation of medical students for the professional and personal challenges of internship in the context of transition courses. They first describe the development of a residency transition course, offered since 2001 at the University of California, San Francisco, School of Medicine (UCSF), exploring aspects of a needs assessment, course goals and content, core competencies, and course implementation. They then critically analyze the course, judging it successful based on high subjective satisfaction scores and increased perceived preparedness data. Next, the authors discuss the national context of transition courses, perspectives of various stakeholders, and lessons learned from the UCSF experience. Finally, they consider future directions, suggesting that internship transition courses be a standard part of the medical school curriculum.

Resident evaluation and remediation: a comprehensive approach

BACKGROUND

A comprehensive evaluation and remediation program is an essential component of any residency program. The evaluation system should identify problems accurately and early and allow residents with problems to be assigned to a remediation program that effectively deals with them. Elements of a proactive remediation program include a process for outlining deficiencies, providing resources for improvement, communicating clear goals for acceptable performance, and reevaluating performance against these goals.

INTERVENTION

In recognition of the importance of early detection and prompt remediation of the struggling resident, we sought to develop a multifaceted approach to resident evaluation with the aim of early identification and prompt remediation of difficulties. This article describes our comprehensive evaluation program and remediation program, which uses resources within our radiology department and institutional graduate medical education office.

DISCUSSION

An effective evaluation system should identify problems accurately and early, whereas a proactive remediation program should effectively deal with issues once they are identified.

Preparing first-year radiology residents and assessing their readiness for on-call responsibilities: results over 5 years

OBJECTIVE

The objective of our study was to evaluate the preparedness of postgraduate year (PGY)-2 residents for independent call responsibilities and the impact of the radiology residency training program on call preparedness using an objective DICOM-based simulation module over a 5-year period.

MATERIALS AND METHODS

A month-long emergency radiology lecture series, conducted over 5 consecutive years, was designed and given to radiology residents at all levels. A DICOM-based, interactive, computer-based testing module with actual emergency department cases was developed and administered at the end of the lecture series. Comparison was made between first-year and upper-level resident test scores using a Student's t test, generalized estimating equations, and individual fixed effects to determine PGY-2 residents' before-call preparedness and the effectiveness of the simulation module to assess call preparedness. Resident scoring on the simulation module was also plotted as a function of progression through their residency program to evaluate the impact of the training program on call preparedness.

RESULTS

Over 5 years, 45 PGY-2, 34 PGY-3, 32 PGY-4, and 35 PGY-5 residents attended the lecture series and completed the computer-based testing module. PGY-2 residents scored an average of 71% +/- 15% (SD), PGY-3 residents scored 79% +/- 11%, PGY-4 residents scored 84% +/- 10%, and PGY-5 residents scored 86% +/- 11% of the total points possible. A statistically significant (p < 0.05) difference in scoring on the simulation module was identified between the PGY-2 residents and each upper-level class over the 5-year period and during 4 of 5 examination years analyzed separately. A trend toward higher average scores for each cohort of residents as they progressed through residency training was identified.

CONCLUSION

Over a 5-year period, first-year radiology residents scored significantly lower than upper-level colleagues on an emergency radiology simulation module, suggesting a significant improvement in the ability of residents to interpret typical on-call imaging studies after the PGY-2 year.

A computer-based radiology simulator as a learning tool to help prepare first-year residents for being on call

RATIONALE AND OBJECTIVES

The start of call is a stressful time for radiology residents. Traditional teaching methods are not ideal for call preparation because they are radically different than the task performed on call. The purpose of this study is to determine if a computer-based radiology simulator would have an effect on resident confidence level or diagnostic abilities.

MATERIALS AND METHODS

A simulator was created to mimic the picture archive and communication system (PACS) at our hospital. Typical call-level cases were selected, anonymized, and entered into the database. The first-year residents were randomly split into a control group and a study group that used the simulator. Each resident took a survey 1 month before and after beginning call to measure his or her subjective feeling of preparedness and nervousness. Objective measures were also obtained through the use of discordance levels from on-call cases.

RESULTS

Seventy-one cases were entered into the simulator. Of the 12 residents in the first-year class, 7 were placed in the study group and 5 in the control group. The residents in both groups claimed they felt more prepared and less nervous 1 month after starting call. The differences at survey were not significant, but the residents in the study group trended toward feeling more prepared and less nervous. There was no statistical difference in the discordance rates for on-call cases between the two groups.

CONCLUSIONS

Although statistical significance was not reached between the users of the radiology simulator and the control group, there was a subjective feeling that the simulator was useful for call preparation and as an interactive learning tool. A larger sample study group size may show statistical significance.

Should 12 Months of Training Be Required Before Diagnostic Radiology Residents Take Independent Call?

To assess program director's reactions to a Residency Review Committee rule change requiring diagnostic radiology residents to complete 12 months of training prior to taking independent call, 303 Association of Program Directors members were surveyed. A total of 147 members responded (48.5% response rate). Overall, 66.1% of respondents were opposed to the change, 21.1% were in favor of it, and 12.8% were neutral. Although this rule change has already been accepted by the residency review committee for implementation July 1, 2008, there has been much debate over this modification and the impact it will have on varying programs.

Part II: preparing and assessing first-year radiology resident on-call readiness technical implementation

RATIONALE AND OBJECTIVES

The effectiveness of using a Digital Imaging and Communications in Medicine (DICOM)-based interactive examination system in evaluating the readiness of first year radiology residents before taking overnight call in the emergency department (ED) was reported in part I of this article. This report describes technical aspects for the design and implementation of this system.

MATERIALS AND METHODS

The examination system consists of two modules: Data Collection and Image Viewing. The Data Collection module was a personal computer (PC)-based DICOM storage server based on a free public domain software package, the Mallinckrodt Central Test Node. The Image Viewing module was a Java-based DICOM viewer created using another freeware package: zDicom ActiveX component.

RESULTS

The examination takes place once a year at the end of the first 6-month rotation. Cases selected for the examination were actual clinical cases according to the American Society of Emergency Radiology core curriculum. In the 3-hour timed examination, each resident was required to read the cases and provide clinical findings and recommendations. Upper-level residents also participated in the examination to serve as a control. Answers were scored by two staff radiologists.

CONCLUSION

We have been using this examination system successfully in our institution since 2003 to evaluate the readiness of the first-year residents before they take overnight call in the ED. This report describes a step-by-step procedure for implementing this system into a PC-based platform. This DICOM viewing software is available as freeware to other academic radiology institutions. The total cost for implementing this system is approximately 2000 US dollars.