Volumetric CT-images improve testing of radiological image interpretation skills

The Role of Three-Dimensional Modeling to Improve Comprehension of Liver Anatomy and Tumor Characteristics for Medical Students and Surgical Residents

OBJECTIVE

Studying liver anatomy can be challenging for medical students and surgical residents due to its complexity. Three-dimensional visualization technology (3DVT) allows for a clearer and more precise view of liver anatomy. We sought to assess how 3DVT can assist students and surgical residents comprehend liver anatomy.

DESIGN

Data from 5 patients who underwent liver resection for malignancy at our institution between September 2020 and April 2022 were retrospectively reviewed and selected following consensus among the investigators. Participants were required to complete an online survey to investigate their understanding of tumor characteristics and vascular variations based on patients' computed tomography (CT) and 3DVT.

SETTING

The study was carried out at the General and Hepato-Biliary Surgery Department of the University of Verona.

PARTICIPANTS

Among 32 participants, 13 (40.6%) were medical students, and 19 (59.4%) were surgical residents.

RESULTS

Among 5 patients with intrahepatic lesions, 4 patients (80.0%) had at least 1 vascular variation. Participants identified number and location of lesions more correctly when evaluating the 3DVT (84.6% and 80.9%, respectively) compared with CT scans (61.1% and 64.8%, respectively) (both p ≤ 0.001). The identification of any vascular variations was more challenging using the CT scans, with only 50.6% of correct answers compared with 3DVT (72.2%) (p < 0.001). Compared with CT scans, 3DVT led to a 23.5%, 16.1%, and 21.6% increase in the correct definition of number and location of lesions, and vascular variations, respectively. 3DVT allowed for a decrease of 50.8 seconds (95% CI 23.6-78.0) in the time needed to answer the questions. All participants agreed on the usefulness of 3DVT in hepatobiliary surgery.

CONCLUSIONS

The 3DVT facilitated a more precise preoperative understanding of liver anatomy, tumor location and characteristics.

Expertise development in volumetric image interpretation of radiology residents: what do longitudinal scroll data reveal?

The current study used theories on expertise development (the holistic model of image perception and the information reduction hypothesis) as a starting point to identify and explore potentially relevant process measures to monitor and evaluate expertise development in radiology residency training. It is the first to examine expertise development in volumetric image interpretation (i.e., CT scans) within radiology residents using scroll data collected longitudinally over five years of residency training. Consistent with the holistic model of image perception, the percentage of time spent on full runs, i.e. scrolling through more than 50% of the CT-scan slices (global search), decreased within residents over residency training years. Furthermore, the percentage of time spent on question-relevant areas in the CT scans increased within residents over residency training years, consistent with the information reduction hypothesis. Second, we examined if scroll patterns can predict diagnostic accuracy. The percentage of time spent on full runs and the percentage of time spent on question-relevant areas did not predict diagnostic accuracy. Thus, although scroll patterns over training years are consistent with visual expertise theories, they could not be used as predictors of diagnostic accuracy in the current study. Therefore, the relation between scroll patterns and performance needs to be further examined, before process measures can be used to monitor and evaluate expertise development in radiology residency training.

An Open Source Solution for "Hands-on" teaching of PET/CT to Medical Students under the COVID-19 Pandemic

AIMS

 Since 2017, medical students at the University of Bergen were taught PET/CT "hands-on" by viewing PET/CT cases in native format on diagnostic workstations in the hospital. Due to the COVID-19 pandemic, students were barred access. This prompted us to launch and evaluate a new freeware PET/CT viewing system hosted in the university network.

METHODS

 We asked our students to install the multiplatform Fiji viewer with Beth Israel PET/CT plugin (http://petctviewer.org) on their personal computers and connect to a central image database in the university network based on the public domain orthanc server (https://orthanc-server.com). At the end of course, we conducted an anonymous student survey.

RESULTS

 The new system was online within eight days, including regulatory approval. All 76 students (100 %) in the fifth year completed their course work, reading five anonymized PET/CT cases as planned. 41 (53 %) students answered the survey. Fiji was challenging to install with a mean score of 1.8 on a 5-point Likert scale (5 = easy, 1 = difficult). Fiji was more difficult to use (score 3.0) than the previously used diagnostic workstations in the hospital (score 4.1; p < 0.001, paired t-test). Despite the technical challenge, 47 % of students reported having learnt much (scores 4 and 5); only 11 % were negative (scores 1 and 2). 51 % found the PET/CT tasks engaging (scores 4 and 5) while 20 % and 5 % returned scores 2 and 1, respectively.

CONCLUSION

 Despite the initial technical challenge, "hands-on" learning of PET/CT based on the freeware Fiji/orthanc PET/CT-viewer was associated with a high degree of student satisfaction. We plan to continue running the system to give students permanent access to PET/CT cases in native format regardless of time or location.

Assessing the neuroanatomy knowledge and spatial ability of radiotherapy technologist undergraduates using an interactive volumetric simulation tool-the RadioLOG project

OBJECTIVE

To assess the use of a volumetric image display simulation tool (VDST) for the evaluation of applied radiological neuroanatomy knowledge and spatial understanding of radiotherapy technologist (RTT) undergraduates.

METHODS

Ninety-two third-year RTT students from three French RTT schools took an examination using software that allows visualization of multiple volumetric image series. To serve as a reference, 77 first- and second-year undergraduates, as well as ten senior neuroradiologists, took the same examination. The test included 13 very-short-answer questions (VSAQ) and 21 exercises in which examinees positioned markers onto preloaded brain MR images from a healthy volunteer. The response time was limited. Each correct answer scored 100 points, with a maximum possible test score of 3,400 (VSAQ = 1,300; marker exercise = 2,100). Answers were marked automatically for the marker positioning exercise and semi-automatically for the VSAQs against prerecorded expected answers.

RESULTS

Overall, the mean test score was 1,787 (150-3,300) and the standard deviation was 781. Scores were highly significantly different between all evaluated groups (p < 0.001). The interoperator reproducibility was 0.90. All the evaluated groups could be discriminated by VSAQ, marker, and overall total scores independently (p ≤ 0.0001 to 0.001). The test was able to discriminate between the three schools either by VSAQ scores (p < 0.001 to 0.02) or by overall total score (p < 0.001 to 0.05).

CONCLUSION

This software is a high-quality evaluation tool for the assessment of radiological neuroanatomy knowledge and spatial understanding in RTT undergraduates.

KEY POINTS

• This VDST allows volumetric image analysis of MR studies. • A high reliability test could be created with this tool. • Test scores were strongly associated with the examinee expertise level.

Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents' Test Results After Transitioning from Formative Progress Testing

INTRODUCTION

Educational effects of transitioning from formative to summative progress testing are unclear. Our purpose was to investigate whether such transitioning in radiology residency is associated with a change in progress test results.

METHODS

We investigated a national cohort of radiology residents (N > 300) who were semi-annually assessed through a mandatory progress test. Until 2014, this test was purely formative for all residents, but in 2014/2015, it was transitioned (as part of a national radiology residency program revision) to include a summative pass requirement for new residents. In 7 posttransitioning tests in 2015-2019, including summatively and formatively tested residents who followed the revised and pre-transitioning residency program, respectively, we assessed residents' relative test scores and percentage of residents that reached pass standards.

RESULTS

Due to our educational setting, most posttransitioning tests had no residents in the summative condition in postgraduate year 4-5, nor residents in the formative condition in year 0.5-2. Across the 7 tests, relative test scores in postgraduate year 1-3 of the summative resident group and year 3.5-4.5 of the formative group differed significantly (p < 0.01 and p < 0.05, respectively, Kruskal-Wallis test). However, scores fluctuated without consistent time trends and without consistent differences between both resident groups. Percentage of residents reaching the pass standard did not differ significantly across tests or between groups.

DISCUSSION

Transitioning from formative to summative progress testing was associated with overall steady test results of the whole resident group in 4 post-transitioning years. We do not exclude that transitioning may have positive educational effects for resident subgroups.

Making Learning Fun: Gaming in Radiology Education

With continued technologic advances, it is not surprising that gaming techniques are increasingly being used in radiology residency programs. This comprehensive review on gaming in radiology education offers insight into the importance of gaming, types of games and principles utilized in gaming, as well as applications that are inherent in artificial intelligence and continued medical education. The advantages and disadvantages of gaming will be considered, as well as barriers to successful adoption of gaming.

The addition of 3D printed models to enhance the teaching and learning of bone spatial anatomy and fractures for undergraduate students: a randomized controlled study

Background

Whether or not the addition of 3D (three-dimension) printed models can enhance the teaching and learning environment for undergraduate students in regard to bone spatial anatomy is still unknown. In this study, we investigated the use of 3D printed models versus radiographic images as a technique for the education of medical students about bone spatial anatomy and fractures.

Methods

The computed tomography (CT) data from four patients, each with a different fracture type (one spinal fracture, one pelvic fracture, one upper limb fracture, and one lower limb fracture), were obtained, and 3D models of the fractures were printed. A total of 90 medical students were enrolled in the study and randomly divided into two groups as follows: a traditional radiographic image group (presented by PowerPoint) and a 3D printed model group (combined PowerPoint and 3D models). Each student answered 5 questions about one type of fracture and completed a visual analog scale of satisfaction (0-10 points).

Results

No significant differences were found in the upper limb or lower limb test scores between the 3D printed model group and the traditional radiographic image group; however, the scores on the pelvis and spine test for the traditional radiographic image group were significantly lower than the scores for the 3D printed model group (P=0.000). No significant differences were found in the test-taking times for the upper limb or lower limb (P=0.603 and P=0.746, respectively) between the two groups; however, the test-taking times for the pelvis and spine in the traditional radiographic image group were significantly longer than those of the 3D printed model group (P=0.000 and P=0.002, respectively).

Conclusions

The 3D printed model may improve medical students' understanding of bone spatial anatomy and fractures in some anatomically complex sites.

Volumetric image interpretation in radiology: scroll behavior and cognitive processes

The interpretation of medical images is a primary task for radiologists. Besides two-dimensional (2D) images, current imaging technologies allow for volumetric display of medical images. Whereas current radiology practice increasingly uses volumetric images, the majority of studies on medical image interpretation is conducted on 2D images. The current study aimed to gain deeper insight into the volumetric image interpretation process by examining this process in twenty radiology trainees who all completed four volumetric image cases. Two types of data were obtained concerning scroll behaviors and think-aloud data. Types of scroll behavior concerned oscillations, half runs, full runs, image manipulations, and interruptions. Think-aloud data were coded by a framework of knowledge and skills in radiology including three cognitive processes: perception, analysis, and synthesis. Relating scroll behavior to cognitive processes showed that oscillations and half runs coincided more often with analysis and synthesis than full runs, whereas full runs coincided more often with perception than oscillations and half runs. Interruptions were characterized by synthesis and image manipulations by perception. In addition, we investigated relations between cognitive processes and found an overall bottom-up way of reasoning with dynamic interactions between cognitive processes, especially between perception and analysis. In sum, our results highlight the dynamic interactions between these processes and the grounding of cognitive processes in scroll behavior. It suggests, that the types of scroll behavior are relevant to describe how radiologists interact with and manipulate volumetric images.

Determinants of Difficulty and Discriminating Power of Image-based Test Items in Postgraduate Radiological Examinations

RATIONALE AND OBJECTIVES

The psychometric characteristics of image-based test items in radiological written examinations are not well known. In this study, we explored difficulty and discriminating power of these test items in postgraduate radiological digital examinations.

MATERIALS AND METHODS

We reviewed test items of seven Dutch Radiology Progress Tests (DRPTs) that were taken from October 2013 to April 2017. The DRPT is a semiannual formative examination, required for all Dutch radiology residents. We assessed several stimulus and response characteristics of test items. The response format of test items included true or false, single right multiple choice with 2, 3, 4, or ≥5 answer options, pick-N multiple-choice, drag-and-drop, and long-list-menu formats. We calculated item P values and item-rest-correlation (Rir) values to assess difficulty and discriminating power. We performed linear regression analysis in image-based test items to investigate whether P and Rir values were significantly related to stimulus and response characteristics. Also, we compared psychometric indices between image-based test items and text-alone items.

RESULTS

P and Rir values of image-based items (n = 369) were significantly related to the type of response format (P < .001), and not to which of the seven DRPTs the item was obtained from, radiological subspecialty domain, nonvolumetric or volumetric character of images, or context-rich or context-free character of the stimulus. When accounted for type of response format, difficulty and discriminating power of image-based items did not differ significantly from text-alone items (n = 881). Test items with a relatively large number of answer options were generally more difficult, and discriminated better among high- and low-performing candidates.

CONCLUSION

In postgraduate radiological written examinations, difficulty and discriminating power of image-based test items are related to the type of response format and are comparable to those of text-alone items. We recommend a response format with a relatively large number of answer options to optimize psychometric indices of radiological image-based test items.

New approaches to the analysis of eye movement behaviour across expertise while viewing brain MRIs

Brain tumour detection and diagnosis requires clinicians to inspect and analyse brain magnetic resonance images. Eye-tracking is commonly used to examine observers' gaze behaviour during such medical image interpretation tasks, but analysis of eye movement sequences is limited. We therefore used ScanMatch, a novel technique that compares saccadic eye movement sequences, to examine the effect of expertise and diagnosis on the similarity of scanning patterns. Diagnostic accuracy was also recorded. Thirty-five participants were classified as Novices, Medics and Experts based on their level of expertise. Participants completed two brain tumour detection tasks. The first was a whole-brain task, which consisted of 60 consecutively presented slices from one patient; the second was an independent-slice detection task, which consisted of 32 independent slices from five different patients. Experts displayed the highest accuracy and sensitivity followed by Medics and then Novices in the independent-slice task. Experts showed the highest level of scanning pattern similarity, with medics engaging in the least similar scanning patterns, for both the whole-brain and independent-slice task. In the independent-slice task, scanning patterns were the least similar for false negatives across all expertise levels and most similar for experts when they responded correctly. These results demonstrate the value of using ScanMatch in the medical image perception literature. Future research adopting this tool could, for example, identify cases that yield low scanning similarity and so provide insight into why diagnostic errors occur and ultimately help in training radiologists.

Fourteen years of progress testing in radiology residency training: experiences from The Netherlands

Objectives

To describe the development of the Dutch Radiology Progress Test (DRPT) for knowledge testing in radiology residency training in The Netherlands from its start in 2003 up to 2016.

Methods

We reviewed all DRPTs conducted since 2003. We assessed key changes and events in the test throughout the years, as well as resident participation and dispensation for the DRPT, test reliability and discriminative power of test items.

Results

The DRPT has been conducted semi-annually since 2003, except for 2015 when one digital DRPT failed. Key changes in these years were improvements in test analysis and feedback, test digitalization (2013) and inclusion of test items on nuclear medicine (2016). From 2003 to 2016, resident dispensation rates increased (Pearson’s correlation coefficient 0.74, P-value <0.01) to maximally 16 %. Cronbach´s alpha for test reliability varied between 0.83 and 0.93. The percentage of DRPT test items with negative item-rest-correlations, indicating relatively poor discriminative power, varied between 4 % and 11 %.

Conclusions

Progress testing has proven feasible and sustainable in Dutch radiology residency training, keeping up with innovations in the radiological profession. Test reliability and discriminative power of test items have remained fair over the years, while resident dispensation rates have increased.

Key Points

• Progress testing allows for monitoring knowledge development from novice to senior trainee.

• In postgraduate medical training, progress testing is used infrequently.

• Progress testing is feasible and sustainable in radiology residency training.

Expertise Affects Inter-Observer Agreement at Peripheral Locations within a Brain Tumor

Magnetic resonance imaging (MRI) is a crucial tool for clinical brain tumor detection and delineation. Since the process of gross tumor volume delineation resides with clinicians, a better understanding of how they perform this task is required if improvements in life expectancy are to be made. Novice-expert comparison studies have been used to examine the effect of expertise on abnormality detection, but little research has investigated expertise-related differences in brain tumor delineation. In this study, undergraduate students (novices) and radiologists (experts) inspected a combination of T1 and T2 single and whole brain MRI scans, each containing a tumor. Using a tablet and stylus to provide an interactive environment, participants had an unlimited amount of time to scroll freely through the MRI slices and were instructed to delineate (i.e., draw a boundary) around any tumorous tissue. There was no reliable evidence for a difference in the gross tumor volume or total number of slices delineated between experts and novices. Agreement was low across both expertise groups and significantly lower at peripheral locations within a tumor than central locations. There was an interaction between expertise level and location within a tumor with experts displaying higher agreement at the peripheral slices than novices. An effect of brain image set on the order in which participants inspected the slices was also observed. The implications of these results for the training undertaken by early career radiologists and current practices in hospitals are discussed.

Identifying error types in visual diagnostic skill assessment

BACKGROUND

Misinterpretation of medical images is an important source of diagnostic error. Errors can occur in different phases of the diagnostic process. Insight in the error types made by learners is crucial for training and giving effective feedback. Most diagnostic skill tests however penalize diagnostic mistakes without an eye for the diagnostic process and the type of error. A radiology test with stepwise reasoning questions was used to distinguish error types in the visual diagnostic process. We evaluated the additional value of a stepwise question-format, in comparison with only diagnostic questions in radiology tests.

METHODS

Medical students in a radiology elective (n=109) took a radiology test including 11-13 cases in stepwise question-format: marking an abnormality, describing the abnormality and giving a diagnosis. Errors were coded by two independent researchers as perception, analysis, diagnosis, or undefined. Erroneous cases were further evaluated for the presence of latent errors or partial knowledge. Inter-rater reliabilities and percentages of cases with latent errors and partial knowledge were calculated.

RESULTS

The stepwise question-format procedure applied to 1351 cases completed by 109 medical students revealed 828 errors. Mean inter-rater reliability of error type coding was Cohen's κ=0.79. Six hundred and fifty errors (79%) could be coded as perception, analysis or diagnosis errors. The stepwise question-format revealed latent errors in 9% and partial knowledge in 18% of cases.

CONCLUSIONS

A stepwise question-format can reliably distinguish error types in the visual diagnostic process, and reveals latent errors and partial knowledge.

The Importance of Human-Computer Interaction in Radiology E-learning

With the development of cross-sectional imaging techniques and transformation to digital reading of radiological imaging, e-learning might be a promising tool in undergraduate radiology education. In this systematic review of the literature, we evaluate the emergence of image interaction possibilities in radiology e-learning programs and evidence for effects of radiology e-learning on learning outcomes and perspectives of medical students and teachers. A systematic search in PubMed, EMBASE, Cochrane, ERIC, and PsycInfo was performed. Articles were screened by two authors and included when they concerned the evaluation of radiological e-learning tools for undergraduate medical students. Nineteen articles were included. Seven studies evaluated e-learning programs with image interaction possibilities. Students perceived e-learning with image interaction possibilities to be a useful addition to learning with hard copy images and to be effective for learning 3D anatomy. Both e-learning programs with and without image interaction possibilities were found to improve radiological knowledge and skills. In general, students found e-learning programs easy to use, rated image quality high, and found the difficulty level of the courses appropriate. Furthermore, they felt that their knowledge and understanding of radiology improved by using e-learning. In conclusion, the addition of radiology e-learning in undergraduate medical education can improve radiological knowledge and image interpretation skills. Differences between the effect of e-learning with and without image interpretation possibilities on learning outcomes are unknown and should be subject to future research.

Radiology resident MR and CT image analysis skill assessment using an interactive volumetric simulation tool - the RadioLOG project

OBJECTIVE

Assess the use of a volumetric simulation tool for the evaluation of radiology resident MR and CT interpretation skills.

MATERIAL AND METHODS

Forty-three participants were evaluated with a software allowing the visualisation of multiple volumetric image series. There were 7 medical students, 28 residents and 8 senior radiologists among the participants. Residents were divided into two sub-groups (novice and advanced). The test was composed of 15 exercises on general radiology and lasted 45 min. Participants answered a questionnaire on their experience with the test using a 5-point Likert scale. This study was approved by the dean of the medical school and did not require ethics committee approval.

RESULTS

The reliability of the test was good with a Cronbach alpha value of 0.9. Test scores were significantly different in all sub-groups studies (p < 0.0225). The relation between test scores and the year of residency was logarithmic (R² = 0.974). Participants agreed that the test reflected their radiological practice (3.9 ± 0.9 on a 5-point scale) and was better than the conventional evaluation methods (4.6 ± 0.5 on a 5-point scale).

CONCLUSION

This software provides a high quality evaluation tool for the assessment of the interpretation skills in radiology residents.

KEY POINTS

• This tool allows volumetric image analysis of MR and CT studies. • A high reliability test could be created with this tool. • Test scores were strongly associated with the examinee expertise level. • Examinees positively evaluated the authenticity and usability of this tool.

Support for external validity of radiological anatomy tests using volumetric images

RATIONALE AND OBJECTIVES

Radiology practice has become increasingly based on volumetric images (VIs), but tests in medical education still mainly involve two-dimensional (2D) images. We created a novel, digital, VI test and hypothesized that scores on this test would better reflect radiological anatomy skills than scores on a traditional 2D image test. To evaluate external validity we correlated VI and 2D image test scores with anatomy cadaver-based test scores.

MATERIALS AND METHODS

In 2012, 246 medical students completed one of two comparable versions (A and B) of a digital radiology test, each containing 20 2D image and 20 VI questions. Thirty-three of these participants also took a human cadaver anatomy test. Mean scores and reliabilities of the 2D image and VI subtests were compared and correlated with human cadaver anatomy test scores. Participants received a questionnaire about perceived representativeness and difficulty of the radiology test.

RESULTS

Human cadaver test scores were not correlated with 2D image scores, but significantly correlated with VI scores (r = 0.44, P < .05). Cronbach's α reliability was 0.49 (A) and 0.65 (B) for the 2D image subtests and 0.65 (A) and 0.71 (B) for VI subtests. Mean VI scores (74.4%, standard deviation 2.9) were significantly lower than 2D image scores (83.8%, standard deviation 2.4) in version A (P < .001). VI questions were considered more representative of clinical practice and education than 2D image questions and less difficult (both P < .001).

CONCLUSIONS

VI tests show higher reliability, a significant correlation with human cadaver test scores, and are considered more representative for clinical practice than tests with 2D images.