Assessing first year radiology resident competence pre-call: development and implementation of a computer-based exam before and after the 12 month training requirement

Using Online Survey Software to Enhance Radiology Learning

RATIONALE AND OBJECTIVES

Online educational modules can augment radiology learning by creating opportunities to interact with images in more dynamic ways than with static presentation of images in lectures or journal articles. Building these modules on an online survey platform allows for quantitative assessment and learner feedback, without requiring programming knowledge or need for new website creation.

MATERIALS AND METHODS

Interactive online tutorials were built on a web-based survey platform (Qualtrics, Provo, Utah) accessible by computer or mobile device to teach radiology imaging findings of selected high-morbidity diagnoses. Topics included congenital-type internal hernias (module 1), acute appendicitis in the pregnant patient (module 2), and unintentionally retained surgical instruments (RSI; module 3). Modules consisted of pretest, educational module, and post-test components. For modules 1 and 2, graphics interchange formats were utilized to show CT and MRI image stacks for the diagnosis of congenital-type internal hernias and acute appendicitis in pregnant patients, respectively. For module 3, the "Heat Map" format was chosen to showcase intraoperative radiograph cases, which allowed participants to click on the potential RSI in the image. Pre- and post-test scores were evaluated. To determine statistical significance, an alpha level of 0.05 was utilized.

RESULTS

Module 1 (Internal Hernia): Twenty-one radiology trainees completed the module. The mean pretest score was 3.66 (±1.13) points out of a total 6 possible points (61%), compared to 4.52 (±1.03) points on the post-test (75%). This was a statistically significant increase on the post-test of 0.87 points (95% CI [confidence interval] 0.36, 1.38), t(20) = 3.53, p= 0.002. Module 2 (MR Appendicitis): Seventeen radiology trainees completed the module. The mean pretest score was 3.18 (±1.42) points out of a total 6 possible points (53%), compared to 5.12 (±0.86) points on the post-test (85%). This was a statistically significant increase on the post-test of 1.94 points (95% CI 1.12, 2.76), t(16) = 5.00, p< 0.001. Module 3 (RSI): One hundred seven participants completed the module. The mean pretest score was 3.60 (±1.53) points out of a total 6 possible points (60%), compared to 4.54 (±1.36) points on the post-test (76%). This was a statistically significant increase on the post-test of 0.94 points (95% CI 0.67, 1.21), t(106) = 6.84, p< 0.001.

CONCLUSION

An online survey platform can be used to build interactive education modules. Post-test scores significantly improved from pretest scores with these educational modules.

Preparing junior radiology residents for overnight call via peer-led, hands-on simulation

PURPOSE

The aim of this study was to assess the perceived value and impact of a hands-on mock call simulation program on resident confidence with interpretation of emergency department overnight call cases.

METHODS

A five-session course was implemented in June of 2018 for rising PGY-3/R2 residents to mimic the experience of overnight call with indirect supervision. Sessions were led by senior residents in the program and consisted of timed, independent interpretation of 15-20 high-yield cases per day which highlighted "do-not miss" critical findings and simulated workflow interruptions including phone calls, consultations, and questions from technologists. IRB-approved, and anonymous pre- and post-course surveys were administered to participants which assessed residents' degree of confidence in interpretation of on-call cases and comparison of the mock call experience with existing preparatory strategies. Survey responses were analyzed using McNemar's test and Mann-Whitney U test.

RESULTS

Our survey response rate was 91% (29/32). After completing the mock call simulation, there was a significant increase in the mean Likert score of resident confidence levels and feelings of preparedness from 4.59 to 7.38 (p < 0.01). The majority of respondents (72.4% [21/29]) felt that the mock call simulation was "extremely useful." One hundred percent of respondents indicated that the mock call simulation should be implemented for the following year.

CONCLUSION

Implementation of a hands-on mock call simulation significantly improves the confidence levels of radiology residents before assuming on-call responsibilities and may serve as an adjunct to existing preparatory strategies.

Radiology Education in the Time of COVID-19: A Novel Distance Learning Workstation Experience for Residents

RATIONALE AND OBJECTIVES

The coronavirus disease of 2019 (COVID-19) pandemic has challenged the educational missions of academic radiology departments nationwide. We describe a novel cloud-based HIPAA compliant and accessible education platform which simulates a live radiology workstation for continued education of first year radiology (R1) residents, with an emphasis on call preparation and peer to peer resident learning.

MATERIALS AND METHODS

Three tools were used in our education model: Pacsbin (Orion Medical Technologies, Baltimore, MD, pacsbin.com), Zoom (Zoom Video Communications, San Jose, CA, zoom.us), and Google Classroom (Google, Mountain View, CA, classroom.google.com). A senior radiology resident (R2-R4) (n = 7) driven workflow was established to provide scrollable Digital Imaging and Communications in Medicine (DICOM) based case collections to the R1 residents (n = 9) via Pacsbin. A centralized classroom was created using Google Classroom for assignments, reports, and discussion where attending radiologists could review content for accuracy. Daily case collections over an 8-week period from March to May were reviewed via Zoom video conference readout in small groups consisting of a R2-R4 teacher and R1 residents. Surveys were administered to R1 residents, R2-4 residents, and attending radiologist participants.

RESULTS

Hundred percent of R1 residents felt this model improved their confidence and knowledge to take independent call. Seventy-eight percent of the R1 residents (n = 7/9) demonstrated strong interest in continuing the project after pandemic related restrictions are lifted. Based on a Likert "helpfulness" scale of 1-5 with 5 being most helpful, the project earned an overall average rating of 4.9. Two R2-R4 teachers demonstrated increased interest in pursuing academic radiology.

CONCLUSION

In response to unique pandemic circumstances, our institution implemented a novel cloud-based distance learning solution to simulate the radiology workstation. This platform helped continue the program's educational mission, offered first year residents increased call preparation, and promoted peer to peer learning. This approach to case-based learning could be used at other institutions to educate residents.

Increasing clinical relevance in oral radiology: Benefits and challenges when implementing digital assessment

AIMS

The aims of the study were to investigate benefits and challenges in implementing a digital examination and study the clinical relevance of the digital examination in relation to clinical training and practice.

MATERIAL AND METHOD

The study was based on semi-structured focus-group interviews from two distinct student populations (2016 and 2017) in a bachelor programme in dental hygiene. In addition, conversational data from a plenary discussion from the whole second student population (2017) were collected and analysed. The data were approached on basis of content analysis.

RESULTS

A benefit experienced in the digital examination was the ease in typing and editing answers on the computer. This suggests an increased effectiveness in computer-based compared to analogue examinations. An additional advantage was the experienced relevance of the examination related to the clinic. This finding refers not only to the digital presentations of images, but also to the entire setting in the clinic and dental practice. The limitations reported by the students were non-optimal viewing conditions for presenting radiographic images and difficulties in obtaining an overview of the assignments compared to paper-based examinations due to the linear digital examination format. The last finding on lacking overview revealed an influence on student performances which should be taken seriously in designing digital examinations.

CONCLUSION

In conclusion, the digital layout increases efficiency and clinical relevance of examinations to a certain extent. Obstacles were found in limitations related to image presentation and lack of overview of the examination. The latter challenge raises questions related to developing suitable assessment software.

Reshaping Radiology Precall Preparation: Integrating a Cloud-Based PACS Viewer Into a Flipped Classroom Model

Preparing residents for the on-call experience in Radiology is one of the most important aspects of education within a training program. Traditionally, this preparation has occurred via a combination of case conferences and didactic lectures by program faculty, daily teaching at the workstation, and precall assessments. Recently, a blended curricular model referred to as the flipped classroom has generated a lot of attention within the realm of graduate medical education. We applied this technique to resident precall education in the subspecialty of Neuroradiology, and surveyed the participants about their perceptions of the course. The structure, implementation, and web-based platform used to create the flipped classroom experience is described herein.

Implementation of a Longitudinal Introduction to Radiology Course During Internship Year Improves Diagnostic Radiology Residents' Academic and Clinical Skills: A Canadian Experience

RATIONALE AND OBJECTIVES

In order to ease the transition from internship to diagnostic radiology residency, a year-long didactic introduction to radiology course was offered to post-graduate year one (PGY-1) diagnostic radiology residents during their internship, which consisted of 27 hours of lecture over 9 months. The purpose of this study was to determine the quantitative and qualitative educational value of this course and its effect with respect to on-call preparedness.

MATERIALS AND METHODS

Two consecutive cohorts of Diagnostic Radiology residents were included: the first cohort (PGY-1s in 2011-2012) did not participate in the new course (Old Curriculum Residents) and the second cohort (PGY-1s in 2012-2013) completed the new course (New Curriculum Residents). These two cohorts were compared both qualitatively and quantitatively. Scores were compared from the standardized Canadian National Pre-Call Observed Standardized Clinical Examination and American College of Radiology Diagnostic Radiology In-Training examination, which are taken in the PGY-2 year, at months 5 and 7, respectively. In addition, staff observation of on-call resident performance and resident self-reported preparedness were considered. Cohorts were compared using Mann-Whitney U test with significance defined as P value <0.05. P values from 0.05 to 0.10 were noted as possibly significant and further analyzed using a Cohen d test where the difference was determined to be small (0.2), medium (0.5), or large (0.8).

RESULTS

New Curriculum Residents reported that the content of the PGY1 curriculum was more appropriate than the old curriculum to prepare them for call in PGY2 (P = 0.013). New Curriculum Residents scored better than the Old Curriculum Residents on the Diagnostic Radiology In-Training examination (P = 0.039) and on the emergency cases of the Canadian National Pre-Call Observed Standardized Clinical Examination (P = 0.035). Staff radiologists, who were not blinded, reported that the New Curriculum Residents were better prepared for daytime (P = 0.006) and overnight (P = 0.008) independent call were better prepared to perform common ultrasound examinations alone (P = 0.049), and required less guidance while on call for nine competency areas. There was, however, no statistical difference between the residents' self-reported preparedness for independent call.

CONCLUSIONS

Participation in a lecture-based introductory radiology curriculum during the PGY-1 internship year improved both radiology residents' preparedness for call and their performance in PGY-2.

Resident experience increases diagnostic rate of thyroid fine-needle aspiration biopsies

RATIONALE AND OBJECTIVES

The aim of this study was to determine whether the diagnostic yield of thyroid fine-needle aspirations (FNAs) changes over the course of residency training.

MATERIALS AND METHODS

We identified 5418 ultrasound-guided thyroid nodule FNAs performed in our radiology department from 2004 through 2012. For each FNA, we recorded if the FNA was performed by a resident and if so the name of the resident and supervising attending radiologist. For each resident, we determined the level of training based on their graduation year from our residency program and the date of the FNA as well as prior surgical training and if they completed subsequent interventional radiology fellowship. Pathology reports were reviewed, and FNAs were classified as diagnostic or nondiagnostic (ND). Generalized mixed models were used to assess ND rate with postgraduate years, including residents with and without prior surgical training or if they subsequently completed an interventional radiology fellowship.

RESULTS

Of the 5418 thyroid FNAs, 3164 (58.4%) were performed by a radiology resident under the direct supervision of an attending physician. There was a significant decrease in ND rate as postgraduate years increased (P < .05). A significant decrease in ND rate was found as postgraduate years increased for residents without prior surgical training (P = .0007) or subsequent training in interventional radiology (P = .0014); however, no significant decrease was found for residents with surgical training (P = .37) or completing an interventional radiology fellowship (P = .08). In addition, no significant difference was found for ND rate between postgraduate year 4 (PGY4) and PGY5 (P > .05).

CONCLUSIONS

ND thyroid FNA rates progressively decrease with training level, suggesting that early and continued participation in procedures throughout residency improves outcomes. This is particularly true for residents without prior surgical training or subsequent interventional radiology fellowship.

Residents' ability to interpret radiology images: development and improvement of an assessment tool

RATIONALE AND OBJECTIVES

Despite increasing radiology coverage, nonradiology residents continue to preliminarily interpret basic radiologic studies independently, yet their ability to do so accurately is not routinely assessed.

MATERIALS AND METHODS

An online test of basic radiologic image interpretation was developed through an iterative process. Educational objectives were established, then questions and images were gathered to create an assessment. The test was administered online to first-year interns (postgraduate year [PGY] 1) from 14 different specialties, as well as a sample of third- and fourth-year radiology residents (PGY3/R2 and PGY4/R3).

RESULTS

Over a 2-year period, 368 residents were assessed, including PGY1 (n = 349), PGY3/R2 (n = 14), and PGY4/R3 (n = 5) residents. Overall, the test discriminated effectively between interns (average score = 66%) and advanced residents (R2 = 86%, R3 = 89%; P < .05). Item analysis indicated discrimination indices ranging from -0.72 to 48.3 (mean = 3.12, median 0.58) for individual questions, including four questions with negative discrimination indices. After removal of the negatively indexed questions, the overall predictive value of the instrument persisted and discrimination indices increased for all but one of the remaining questions (range 0.027-70.8, mean 5.76, median 0.94).

CONCLUSIONS

Validation of an initial iteration of an assessment of basic image-interpretation skills led to revisions that improved the test. The results offer a specific test of radiologic reading skills with validation evidence for residents. More generally, results demonstrate a principled approach to test development.

Residents' perception of a novel end-of-rotation evaluation method

RATIONALE AND OBJECTIVES

With the advent of the new core and certifying examinations, a need has arisen to restructure learning and assessment to better reflect the emphasis on continuous learning throughout residency. We developed a multiparametric end-of-rotation (EOR) evaluation tool to assess medical knowledge, oral presentation, and written communication skills administered to the residents at the end of each core subspecialty rotation. The benefit of continual assessment is obvious from a program perspective; the purpose of this article is to evaluate the residents' perception of the process.

MATERIALS AND METHODS

All residents (n = 31, 28 postgraduate years two through five and 3 postgraduate year one) participate in the mandatory EOR evaluation as a required component of the residency program. After receiving Institutional Review Board approval, informed consent was obtained from the residents wishing to participate in quarterly 16-question online surveys assessing their experience. Each survey consists of 15 questions with Likert scale responses (1 through 5 from strongly disagree to strongly agree) and one free text answer. Data are collected quarterly, starting in September 2011.

RESULTS

Overall, the residents' response has been positive. The new evaluation method is felt to be more meaningful than (average 3.9, standard deviation 0.9) and is favored by the residents over the traditional competency-based evaluation (average 4.0, standard deviation 1.0). However, residents retain neutral attitudes regarding preparation for boards or changes in study habits (average score 3.6, standard deviation 0.9 and 3.6, and standard deviation 1.1, respectively).

CONCLUSION

Residents rate the EOR evaluation experience positively, although they do not report changes in study habits or increased preparedness for the new certifying examination.