Evaluation of radiology residents' reporting skills using large language models: an observational study

PURPOSE

Large language models (LLMs) have the potential to objectively evaluate radiology resident reports; however, research on their use for feedback in radiology training and assessment of resident skill development remains limited. This study aimed to assess the effectiveness of LLMs in revising radiology reports by comparing them with reports verified by board-certified radiologists and to analyze the progression of resident's reporting skills over time.

MATERIALS AND METHODS

To identify the LLM that best aligned with human radiologists, 100 reports were randomly selected from 7376 reports authored by nine first-year radiology residents. The reports were evaluated based on six criteria: (1) addition of missing positive findings, (2) deletion of findings, (3) addition of negative findings, (4) correction of the expression of findings, (5) correction of the diagnosis, and (6) proposal of additional examinations or treatments. Reports were segmented into four time-based terms, and 900 reports (450 CT and 450 MRI) were randomly chosen from the initial and final terms of the residents' first year. The revised rates for each criterion were compared between the first and last terms using the Wilcoxon Signed-Rank test.

RESULTS

Among the three LLMs-ChatGPT-4 Omni (GPT-4o), Claude-3.5 Sonnet, and Claude-3 Opus-GPT-4o demonstrated the highest level of agreement with board-certified radiologists. Significant improvements were noted in Criteria 1-3 when comparing reports from the first and last terms (Criteria 1, 2, and 3; P < 0.001, P = 0.023, and P = 0.004, respectively) using GPT-4o. No significant changes were observed for Criteria 4-6. Despite this, all criteria except for Criteria 6 showed progressive enhancement over time.

CONCLUSION

LLMs can effectively provide feedback on commonly corrected areas in radiology reports, enabling residents to objectively identify and improve their weaknesses and monitor their progress. Additionally, LLMs may help reduce the workload of radiologists' mentors.

Quantifying Radiology Residents' Learning Curves in Report Writing Performance Through Report Comparison and Jaccard Similarity

Background Report writing skills are a core competency to be acquired during residency, yet objective tools for tracking performance are lacking. Purpose To investigate whether the Jaccard index, derived from report comparison, can objectively illustrate learning curves in report writing performance throughout radiology residency. Materials and Methods Retrospective data from 246 984 radiology reports written from September 2017 to November 2022 in a tertiary care radiology department were included. Reports were scored using the Jaccard similarity coefficient (ie, a quantitative expression of the amount of edits performed; range, 0-1) of residents' draft (unsupervised initial attempt at a complete report) or preliminary reports (following joint readout with attending physicians) and faculty-reviewed final reports. Weighted mean Jaccard similarity was compared between years of experience using Welch analysis of variance with post hoc testing overall, per imaging division, and per modality. Relationships with years and quarters of resident experience were assessed using Spearman correlation. Results This study included 53 residents (mean report count, 4660 ± 3546; 1-5 years of experience). Mean Jaccard similarity of preliminary reports increased by 6% from 1st-year to 5th-year residents (0.86 ± 0.22 to 0.92 ± 0.15; P < .001). Spearman correlation demonstrated a strong relationship between residents' experience and higher report similarity when aggregated for years (rs = 0.99 [95% CI: 0.85, 1.00]; P < .001) or quarters of experience (rs = 0.90 [95% CI: 0.73, 0.96]; P < .001). For residents' draft reports, Jaccard similarity increased by 14% over the course of the 5-year residency program (0.68 ± 0.27 to 0.82 ± 0.23; P < .001). Subgroup analysis confirmed similar trends for all imaging divisions and modalities (eg, in musculoskeletal imaging, from 0.77 ± 0.31 to 0.91 ± 0.16 [P < .001]; rs = 0.98 [95% CI: 0.72, 1.00] [P < .001]). Conclusion Residents' report writing performance increases with experience. Trends can be quantified with the Jaccard index, with a 6% improvement from 1st- to 5th-year residents, indicating its effectiveness as a tool for evaluating training progress and guiding education over the course of residency. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Bruno in this issue.

Radiographers' experiences of image interpretation training in a low-resource setting

INTRODUCTION

Radiographers extend their roles through formal and on-the-job training to keep up with clinical practice changes. One area of role extension that is now incorporated into undergraduate programmes is image interpretation, although the training provided may vary between institutions. This study explored the experiences of graduates from one higher education institution in a low-resource context with regard to their image interpretation training.

METHODS

A qualitative phenomenological research design was employed to investigate the experiences of ten radiography graduates who were purposively selected from one higher education institution. Individual semi-structured interviews were conducted with each participant after obtaining their informed consent. The interview recordings were transcribed and analysed using Atlas.ti Windows (Version 9.0) software, following Colaizzi's seven-steps of data analysis.

RESULTS

From the ten interviews conducted, teaching approach, clinical education, and assessment strategy emerged as areas of experience within the teaching and learning theme, while practitioner role modelling, skill utilisation, and industry impact were sub-themes under the paradoxical reality theme. The participants' experiences indicated a theory-practice gap in image interpretation among radiographers.

CONCLUSION

The participants' experiences reflected a misalignment in the educational process due to inadequacies in the teaching approach, clinical education, and assessment strategies. Participants encountered significant differences between their expectations and clinical realities during and after training. Image interpretation by radiographers was recognised as a relevant area for role extension in this low-resource setting.

IMPLICATIONS FOR PRACTICE

While these findings are specific to the experiences of the participants, conducting similar research in comparable contexts and implementing competency-based image interpretation assessments could help identify gaps and guide interventions to address shortcomings.

The augmented radiologist: artificial intelligence in the practice of radiology

In medicine, particularly in radiology, there are great expectations in artificial intelligence (AI), which can "see" more than human radiologists in regard to, for example, tumor size, shape, morphology, texture and kinetics - thus enabling better care by earlier detection or more precise reports. Another point is that AI can handle large data sets in high-dimensional spaces. But it should not be forgotten that AI is only as good as the training samples available, which should ideally be numerous enough to cover all variants. On the other hand, the main feature of human intelligence is content knowledge and the ability to find near-optimal solutions. The purpose of this paper is to review the current complexity of radiology working places, to describe their advantages and shortcomings. Further, we give an AI overview of the different types and features as used so far. We also touch on the differences between AI and human intelligence in problem-solving. We present a new AI type, labeled "explainable AI," which should enable a balance/cooperation between AI and human intelligence - thus bringing both worlds in compliance with legal requirements. For support of (pediatric) radiologists, we propose the creation of an AI assistant that augments radiologists and keeps their brain free for generic tasks.

Written-Based Progress Testing: A Scoping Review

PURPOSE

Progress testing is an increasingly popular form of assessment in which a comprehensive test is administered to learners repeatedly over time. To inform potential users, this scoping review aimed to document barriers, facilitators, and potential outcomes of the use of written progress tests in higher education.

METHOD

The authors followed Arksey and O'Malley's scoping review methodology to identify and summarize the literature on progress testing. They searched 6 databases (Academic Search Complete, CINAHL, ERIC, Education Source, MEDLINE, and PsycINFO) on 2 occasions (May 22, 2018, and April 21, 2020) and included articles written in English or French and pertaining to written progress tests in higher education. Two authors screened articles for the inclusion criteria (90% agreement), then data extraction was performed by pairs of authors. Using a snowball approach, the authors also screened additional articles identified from the included reference lists. They completed a thematic analysis through an iterative process.

RESULTS

A total of 104 articles were included. The majority of progress tests used a multiple-choice and/or true-or-false question format (95, 91.3%) and were administered 4 times a year (38, 36.5%). The most documented source of validity evidence was internal consistency (38, 36.5%). Four major themes were identified: (1) barriers and challenges to the implementation of progress testing (e.g., need for additional resources); (2) established collaboration as a facilitator of progress testing implementation; (3) factors that increase the acceptance of progress testing (e.g., formative use); and (4) outcomes and consequences of progress test use (e.g., progress testing contributes to an increase in knowledge).

CONCLUSIONS

Progress testing appears to have a positive impact on learning, and there is significant validity evidence to support its use. Although progress testing is resource- and time-intensive, strategies such as collaboration with other institutions may facilitate its use.

Expertise development in volumetric image interpretation of radiology residents: what do longitudinal scroll data reveal?

The current study used theories on expertise development (the holistic model of image perception and the information reduction hypothesis) as a starting point to identify and explore potentially relevant process measures to monitor and evaluate expertise development in radiology residency training. It is the first to examine expertise development in volumetric image interpretation (i.e., CT scans) within radiology residents using scroll data collected longitudinally over five years of residency training. Consistent with the holistic model of image perception, the percentage of time spent on full runs, i.e. scrolling through more than 50% of the CT-scan slices (global search), decreased within residents over residency training years. Furthermore, the percentage of time spent on question-relevant areas in the CT scans increased within residents over residency training years, consistent with the information reduction hypothesis. Second, we examined if scroll patterns can predict diagnostic accuracy. The percentage of time spent on full runs and the percentage of time spent on question-relevant areas did not predict diagnostic accuracy. Thus, although scroll patterns over training years are consistent with visual expertise theories, they could not be used as predictors of diagnostic accuracy in the current study. Therefore, the relation between scroll patterns and performance needs to be further examined, before process measures can be used to monitor and evaluate expertise development in radiology residency training.

Social Media in Radiology: Overview and Usefulness of Online Professional #SoMe Profiles

Social media is a ubiquitous part of society and used by many healthcare professionals, including radiologists. This article prepared by the 2019-2020 Radiology Research Alliance Social Media Task Force presents different uses and benefits of an online professional social media profile. Physician and healthcare-specific issues along with popular platforms such as LinkedIn, Facebook, Twitter, Instagram, and YouTube are used to explore topics including education, branding, research, mentorship, promotion, and leadership.

Medical knowledge and clinical productivity: independently correlated metrics during radiology residency

Objective

To determine the association between medical knowledge relevant to radiology practice (as measured by the Dutch radiology progress test [DRPT]) and clinical productivity during radiology residency.

Methods

This study analyzed the results of 6 DRPTs and time period–matched clinical production points of radiology residents affiliated to a tertiary care academic medical center between 2013 and 2016. The Spearman correlation analysis was performed to determine the association between DRPT percentile scores and average daily clinical production points. Linear regression analyses were performed to determine the association of DRPT percentile scores with average daily clinical production points, adjusted for age and gender of the radiology resident, and postgraduate year.

Results

Eighty-four DRPTs with time period–matched clinical production points were included. These 84 DRPTs were made by 29 radiology residents (18 males and 11 females) with a median age of 31 years (range: 26–38 years). The Spearman correlation coefficient between DRPT percentile scores and average daily clinical production points was 0.550 (95% confidence interval: 0.381–0.694) (p < 0.001), indicating a significant moderate positive association. On multivariate analysis, average daily clinical production points (β coefficient of 0.035, p = 0.003), female gender of the radiology resident (β coefficient of 12.690, p = 0.001), and postgraduate year (β coefficient of 10.179, p < 0.001) were significantly associated with DRPT percentile scores. These three independent variables achieved an adjusted R² of 0.527.

Conclusion

Clinical productivity is independently associated with medical knowledge relevant to radiology practice during radiology residency. These findings indicate that clinical productivity of a resident could be a potentially relevant metric in a radiology training program.

Key Points

• There is a significant moderate correlation between medical knowledge relevant to radiology practice and clinical productivity during radiology residency.

• Medical knowledge relevant to radiology practice remains independently associated with clinical productivity during radiology residency after adjustment for postgraduate year and gender.

• Clinical productivity of a resident may be regarded as a potentially relevant metric in a radiology training program.

Comparative assessment of midfoot osteoarthritis diagnostic sensitivity using weightbearing computed tomography vs weightbearing plain radiography

PURPOSE

Accuracy in diagnosing osteoarthritis in the midfoot using weightbearing plain radiography (WBPR) remains questionable due to the overlapping osseous architecture present, occluding visualization. Weightbearing computed tomography (WBCT), providing clearer bony landmark identification and joint space visualization, can also be used for evaluation. The aim of this project is to perform a standardized retrospective intra-patient analysis identifying the discrepancy of midfoot osteoarthritis diagnosis and osteoarthritis severity grading between WBPR and WBCT.

METHODS AND MATERIALS

A cohort of 302 patient feet was acquired from an internal, consecutive patient database using detailed inclusion criteria. The musculoskeletal radiologist interpretation of the WBCT and WBPR of each specimen was then assessed for any direct diagnosis or mention of osteoarthritic signs in specific articulations of 3 midfoot joint groups (Chopart, "central", and tarsometatarsal). WBPR sensitivity and specificity metrics were calculated with WBCT considered the gold standard for comparison.

RESULTS

From the WBPR radiologist interpretation, we found diagnostic sensitivity of 72.5 % and specificity of 87.9 % for Chopart joints; 61.5 % sensitivity, and 96.1 % specificity for central joints; and 68.4 % sensitivity, and 92.9 % specificity for tarsometatarsal joints. The severity of degenerative changes was also consistently underestimated when interpreted from WBPR relative to WBCT.

CONCLUSIONS

In this series, midfoot osteoarthritis was often undetected on WBPR. WBCT imaging facilitates an earlier, more reliable diagnosis and grading of midfoot osteoarthritis relative to WBPR.

A Pilot Study on Diagnostic Radiology Residency Case Volumes From a Canadian Perspective: A Marker of Resident Knowledge

Purpose:

New guidelines from the Accreditation Council for Graduate Medical Education (ACGME) have proposed minimum case volumes to be obtained during residency. While radiology residency programs in Canada are accredited by the Royal College of Physicians and Surgeons of Canada, there are currently no minimum case volumes standards for radiology residency training in Canada. New changes in residency training throughout Canada are coming in the form of competency-based medical education. Using data from a pilot study, this article examines radiology resident case volumes among recently graduated cohorts of residents and determines whether there is a correlation between case volumes and measures of resident success.

Materials and Methods:

Resident case volumes for 3 cohorts of graduated residents (2016-2018) were extracted from the institutional database. Achievement of minimum case volumes based on the ACGME guidelines was performed for each resident. Pearson correlation analysis (n = 9) was performed to examine the relationships between resident case volumes and markers of resident success including residents’ relative knowledge ranking and their American College of Radiology (ACR) in-training exam scores.

Results:

A statistically significant, positive correlation was observed between residents’ case volume and their relative knowledge ranking ( r = 0.682, P < .05). Residents’ relative knowledge ranking was also statistically significant and positively correlated with their ACR in-training percentile score ( r = 0.715, P < .05).

Conclusions:

This study suggests that residents who interpret more cases are more likely to demonstrate higher knowledge, thereby highlighting the utility of case volumes as a prognostic marker of resident success. As well, the results underscore the potential use of ACGME minimum case volumes as a prognostic marker. These findings can inform future curriculum planning and development in radiology residency training programs.

An Open Source Solution for "Hands-on" teaching of PET/CT to Medical Students under the COVID-19 Pandemic

AIMS

 Since 2017, medical students at the University of Bergen were taught PET/CT "hands-on" by viewing PET/CT cases in native format on diagnostic workstations in the hospital. Due to the COVID-19 pandemic, students were barred access. This prompted us to launch and evaluate a new freeware PET/CT viewing system hosted in the university network.

METHODS

 We asked our students to install the multiplatform Fiji viewer with Beth Israel PET/CT plugin (http://petctviewer.org) on their personal computers and connect to a central image database in the university network based on the public domain orthanc server (https://orthanc-server.com). At the end of course, we conducted an anonymous student survey.

RESULTS

 The new system was online within eight days, including regulatory approval. All 76 students (100 %) in the fifth year completed their course work, reading five anonymized PET/CT cases as planned. 41 (53 %) students answered the survey. Fiji was challenging to install with a mean score of 1.8 on a 5-point Likert scale (5 = easy, 1 = difficult). Fiji was more difficult to use (score 3.0) than the previously used diagnostic workstations in the hospital (score 4.1; p < 0.001, paired t-test). Despite the technical challenge, 47 % of students reported having learnt much (scores 4 and 5); only 11 % were negative (scores 1 and 2). 51 % found the PET/CT tasks engaging (scores 4 and 5) while 20 % and 5 % returned scores 2 and 1, respectively.

CONCLUSION

 Despite the initial technical challenge, "hands-on" learning of PET/CT based on the freeware Fiji/orthanc PET/CT-viewer was associated with a high degree of student satisfaction. We plan to continue running the system to give students permanent access to PET/CT cases in native format regardless of time or location.

Perceptual Learning of Appendicitis Diagnosis in Radiological Images

A sizeable body of work has demonstrated that participants have the capacity to show substantial increases in performance on perceptual tasks given appropriate practice. This has resulted in significant interest in the use of such perceptual learning techniques to positively impact performance in real-world domains where the extraction of perceptual information in the service of guiding decisions is at a premium. Radiological training is one clear example of such a domain. Here we examine a number of basic science questions related to the use of perceptual learning techniques in the context of a radiology-inspired task. On each trial of this task, participants were presented with a single axial slice from a CT image of the abdomen. They were then asked to indicate whether or not the image was consistent with appendicitis. We first demonstrate that, although the task differs in many ways from standard radiological practice, it nonetheless makes use of expert knowledge, as trained radiologists who underwent the task showed high (near ceiling) levels of performance. Then, in a series of four studies we show that (1) performance on this task does improve significantly over a reasonably short period of training (on the scale of a few hours); (2) the learning transfers to previously unseen images and to untrained image orientations; (3) purely correct/incorrect feedback produces weak learning compared to more informative feedback where the spatial position of the appendix is indicated in each image; and (4) there was little benefit seen from purposefully structuring the learning experience by starting with easier images and then moving on to more difficulty images (as compared to simply presenting all images in a random order). The implications for these various findings with respect to the use of perceptual learning techniques as part of radiological training are then discussed.

Diagnosis in a snap: a pilot study using Snapchat in radiologic didactics

Purpose

To evaluate Snapchat, an image-based social media platform, as a tool for emergency radiologic didactics comparing image interpretation on mobile devices with conventional analysis on a classroom screen.

Materials and methods

Seven radiology residents (4 juniors, 3 seniors;4 males, 3 females; 28.4 years old, ± 1.7 years) were shown 5 emergent radiologic cases using Snapchat and 5 cases of similar content and duration on a classroom projector over 4 weeks. All images depicted diagnoses requiring immediate communication to ordering physicians. Performance was scored 0–2 (0 = complete miss, 1 = major finding, but missed the diagnosis, 2 = correct diagnosis) by two attending radiologists in consensus.

Results

All residents performed better on Snapchat each week. In weeks 1–4, juniors scored 21/40 (52.5%), 23/40 (57.5%), 19/40 (47.5%), and 18/40 (45%) points using Snapchat compared with 13/40 (32.5%), 23/40 (57.5%), 14/40 (35%), and 13/40 (32.5%), respectively, each week by projector, while seniors scored 19/30 (63.3%), 21/30 (70%), 27/30 (90%), and 21/30 (70%) on Snapchat versus 16/30 (53.3%), 19/30 (63.3%), 20/30 (66.7%), and 20/30 (66.7%) on projector. Four-week totals showed juniors scoring 81/160 (50.6%) on Snapchat and 63/160 (39.4%) by projector compared with seniors scoring 88/120 (73.3%) and 75/120 (62.5%), respectively. Performance on Snapchat was statistically, significantly better than via projector during weeks 1 and 3 (p values 0.0019 and 0.0031).

Conclusion

Radiology residents interpreting emergency cases via Snapchat showed higher accuracy compared with using a traditional classroom screen. This pilot study suggests that Snapchat may have a role in the digital radiologic classroom’s evolution.

A Review of Perceptual Expertise in Radiology-How it develops, How we can test it, and Why humans still matter in the era of Artificial Intelligence

As the first step in image interpretation is detection, an error in perception can prematurely end the diagnostic process leading to missed diagnoses. Because perceptual errors of this sort-"failure to detect"-are the most common interpretive error (and cause of litigation) in radiology, understanding the nature of perceptual expertise is essential in decreasing radiology's long-standing error rates. In this article, we review what constitutes a perceptual error, the existing models of radiologic image perception, the development of perceptual expertise and how it can be tested, perceptual learning methods in training radiologists, and why understanding perceptual expertise is still relevant in the era of artificial intelligence. Adding targeted interventions, such as perceptual learning, to existing teaching practices, has the potential to enhance expertise and reduce medical error.

Evaluation of a new e-learning framework for teaching nuclear medicine and radiology to undergraduate medical students

Background

Radiology and nuclear medicine have traditionally been taught in a series of lectures and seminars concluded by an examination testing knowledge rather than skills.

Purpose

To emphasize skills in the students’ learning process, we developed and evaluated a new e-learning framework for teaching medical imaging.

Material and Methods

The framework consists of electronic lectures, a learning management system (LMS), and a diagnostic viewing system. Students were to review positron emission tomography/computed tomography (PET/CT) examinations of five cases of lung cancer. The framework was evaluated in an objective structured clinical examination (OSCE) taken by 139 undergraduate students at the end of their third year, and in a comparative survey of two groups of 85 and 84 students in the fifth and sixth year who were taught the same oncology course with and without mandatory PET/CT exercises, respectively.

Results

Of the 139 third-year students, 134 (96%) passed the OSCE PET/CT station according to the predefined criteria. While 85/85 (100%) of the fifth-year students had taken exercises when they were mandatory, only 2/84 (2%) of the sixth-year students had reviewed the cases on a voluntary basis (P < 0.001). Of the 25 survey responders in the fifth year, 24 (96%) agreed that the mandatory PET/CT exercises had promoted their learning while the sixth-year students, whose course concluded with a multiple-choice examination, emphasized the utility of online lectures.

Conclusion

The new e-learning framework teaches students basic competency in PET/CT navigation and interpretation and is associated with a high degree of student satisfaction.

Analysis of Perceptual Expertise in Radiology - Current Knowledge and a New Perspective

Radiologists rely principally on visual inspection to detect, describe, and classify findings in medical images. As most interpretive errors in radiology are perceptual in nature, understanding the path to radiologic expertise during image analysis is essential to educate future generations of radiologists. We review the perceptual tasks and challenges in radiologic diagnosis, discuss models of radiologic image perception, consider the application of perceptual learning methods in medical training, and suggest a new approach to understanding perceptional expertise. Specific principled enhancements to educational practices in radiology promise to deepen perceptual expertise among radiologists with the goal of improving training and reducing medical error.

Development of competence in volumetric image interpretation in radiology residents

BACKGROUND

During residency, radiology residents learn to interpret volumetric radiological images. The development of their competence for volumetric image interpretation, as opposed to 2D image reading, is not completely understood. The purpose of the present study was to investigate how competence for volumetric image interpretation develops in radiology residents and how this compares with competence development for 2D image interpretation, by studying resident scores on image-based items in digital radiology tests.

METHODS

We reviewed resident scores on volumetric and 2D image-based test items in 9 consecutive semi-annual digital radiology tests that were carried out from November 2013 to April 2018. We assessed percentage-correct sum scores for all test items about volumetric images and for all test items about 2D images in each test as well as for all residents across the 9 tests (i.e. 4.5 years of test materials). We used a paired t-test to analyze whether scores differed between volumetric and 2D image-based test items in individual residents in postgraduate year (PGY) 0-5, subdivided in 10 half-year phases (PGY 0-0.5, 0.5-1.0, 1.0-1.5 et cetera).

RESULTS

The percentage-correct scores on volumetric and 2D image-based items showed a comparable trend of development, increasing in the first half of residency and flattening off in the second half. Chance-corrected scores were generally lower in volumetric than in 2D items (on average 1-5% points). In PGY 1.5-4.5, this score difference was statistically significant (p-values ranging from 0.02 to < 0.001), with the largest difference found in PGY 2.5 (mean: 5% points; 95% CI: -7.3 - -3.4). At the end of training in PGY 5, there was no statistically significant score difference between both item types.

CONCLUSIONS

The development of competence in volumetric image interpretation fits a similar curvilinear growth curve during radiology residency as 2D image interpretation competence in digital radiology tests. Although residents performed significantly lower on volumetric than 2D items in PGY 1.5-4.5, we consider the magnitude of this difference as relatively small for our educational setting and we suggest that throughout radiology training there are no relevant differences in the development of both types of competences, as investigated by digital radiology tests.

Analysis of the French national evaluation of radiology residents

PURPOSE

In France, a national evaluation is given annually to radiology residents. The aim of this study was to perform both a docimological analysis of the quality of the questionnaire and a statistical analysis of the results.

MATERIALS AND METHODS

This retrospective study, which included French radiology residents from Year 1 to Year 5 of residency, was performed from 2015 to 2017 across 25 medical universities in France. Both qualitative and quantitative docimological analyses were performed as assessed by the Cronbach alpha coefficient, the difficulty of question (PDI), and the coefficient of discrimination (Rir). Results to the questionnaire were compared between years of residency.

RESULTS

The results of the analysis confirmed the quality of the questionnaire (Cronbach alpha coefficient=0.71, mean [PDI=0.40]) though the majority of questions could be answered by memory rather than cognitive ability. The mean Rir was 0.02, indicating that students could not be certified using only the questionnaire. The results measuring resident level of knowledge were moderate, with mean results ranging from 9.2/20 at the first year to 11.3/20 at the fifth year of residency (P<0.001). There were no significant differences in results obtained between the third, fourth, and fifth year of residency but results were significantly different among university hospitals.

CONCLUSION

Even if close interactions exist between learning and pedagogic environment, our results suggest that it may be useful to further develop an evaluation process in relation with pedagogic instructions in order to provide more optimal training.

Resident and Faculty Attitudes Toward the Dutch Radiology Progress Test as It Transitions from a Formative to a Summative Measure of Licensure Eligibility

BACKGROUND

Progress testing, a regularly administered comprehensive test of a complete knowledge domain, usually serves to provide learners feedback and has a formative nature.

OBJECTIVE

Our study aimed to investigate the acceptability of introducing a summative component in the postgraduate Dutch Radiology Progress Test (DRPT) among residents and program directors in a competency-based training program.

METHODS

A 15-item questionnaire with 3 items on acceptability of summative postgraduate knowledge testing, 7 on acceptability of the summative DRPT regulations, 4 on self-reported educational effects, and 1 open comment item was distributed nationally among 349 residents and 81 radiology program directors.

RESULTS

The questionnaire was filled out by 330 residents (95%) and 48 (59%) program directors. Summative postgraduate knowledge testing was regarded as acceptable by both groups, but more so by program directors than residents. The transition toward summative assessment in the DRPT was received neutrally to slightly positively by residents, while program directors regarded it as an improvement and estimated the summative criteria to be lighter and less stressful than did residents. The residents' self-reported educational effects of summative assessment in the DRPT were limited, whereas program directors expected a greater end-of-training knowledge improvement than residents.

CONCLUSIONS

Both residents and program directors support summative postgraduate knowledge testing, although it is more accepted by program directors. Residents receive summative radiological progress testing neutrally to slightly positively, while program directors generally value it more positively than residents. Directors should be aware of these different perspectives when introducing or developing summative progress testing in residency programs.

Determinants of Difficulty and Discriminating Power of Image-based Test Items in Postgraduate Radiological Examinations

RATIONALE AND OBJECTIVES

The psychometric characteristics of image-based test items in radiological written examinations are not well known. In this study, we explored difficulty and discriminating power of these test items in postgraduate radiological digital examinations.

MATERIALS AND METHODS

We reviewed test items of seven Dutch Radiology Progress Tests (DRPTs) that were taken from October 2013 to April 2017. The DRPT is a semiannual formative examination, required for all Dutch radiology residents. We assessed several stimulus and response characteristics of test items. The response format of test items included true or false, single right multiple choice with 2, 3, 4, or ≥5 answer options, pick-N multiple-choice, drag-and-drop, and long-list-menu formats. We calculated item P values and item-rest-correlation (Rir) values to assess difficulty and discriminating power. We performed linear regression analysis in image-based test items to investigate whether P and Rir values were significantly related to stimulus and response characteristics. Also, we compared psychometric indices between image-based test items and text-alone items.

RESULTS

P and Rir values of image-based items (n = 369) were significantly related to the type of response format (P < .001), and not to which of the seven DRPTs the item was obtained from, radiological subspecialty domain, nonvolumetric or volumetric character of images, or context-rich or context-free character of the stimulus. When accounted for type of response format, difficulty and discriminating power of image-based items did not differ significantly from text-alone items (n = 881). Test items with a relatively large number of answer options were generally more difficult, and discriminated better among high- and low-performing candidates.

CONCLUSION

In postgraduate radiological written examinations, difficulty and discriminating power of image-based test items are related to the type of response format and are comparable to those of text-alone items. We recommend a response format with a relatively large number of answer options to optimize psychometric indices of radiological image-based test items.

Time-Variable Training in Medicine: Theoretical Considerations

The introduction of competency-based medical education has shifted thinking from a fixed-time model to one stressing attained competencies, independent of the time needed to arrive at those competencies. In this article, the authors explore theoretical and conceptual issues related to time variability in medical training, starting with the Carroll model from the 1960s that put time in the equation of learning. They discuss mastery learning, deliberate practice, and learning curves.While such behaviorist theories apply well to structured courses and highly structured training settings, learning in the clinical workplace is not well captured in such theories or in the model that Carroll proposed. Important in clinical training are self-regulation and motivation; neurocognitive perspectives of time and learning; professional identity formation; and entrustment as an objective of training-all of which may be viewed from the perspective of the time needed to complete training. The authors conclude that, in approaching time variability, the Carroll equation is too simplistic in its application to the breadth of medical training. The equation may be expanded to include variables that determine effective workplace learning, but future work will need to examine the validity of these additional factors.