A review of research into the development of radiologic expertise: implications for computer-based training

Contribution of Wireless Wi-Fi Intraoral Cameras to the Assessment of Swallowing Safety and Efficiency

BACKGROUND

Clinical evaluation of swallowing provides important clinical information but is limited in detecting penetration, aspiration, and pharyngeal residue in patients with suspected dysphagia. Although this is an old problem, there remains limited access to low-cost methods to evaluate swallowing safety and efficiency.

PURPOSE

The purpose of this technical report is to describe the experience of a single center that recently began using a wireless Wi-Fi intraoral camera for transoral endoscopic procedures as an adjunct to clinical swallowing evaluation. We describe the theoretical structure of this new clinical evaluation proposal. We present descriptive findings on its diagnostic performance in relation to videofluoroscopic swallowing study as the gold standard in a cohort of seven patients with dysphagia following head and neck cancer. We provide quantitative data on intra- and interrater reliability. Furthermore, this report discusses how this technology can be applied in the clinical practice of professionals who treat patients with dysphagia and provides directions for future research.

CONCLUSIONS

This preliminary retrospective study suggests that intraoral cameras can reveal the accumulated oropharyngeal secretions and postswallow pharyngolaryngeal residue in patients with suspected dysphagia. Future large-scale studies focusing on validating and exploring this contemporary low-cost technology as part of a clinical swallowing evaluation are warranted.

A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation

CONSTRUCT

For assessing the skill of visual diagnosis such as radiograph interpretation, competency standards are often developed in an ad hoc method, with a poorly delineated connection to the target clinical population.

BACKGROUND

Commonly used methods to assess for competency in radiograph interpretation are subjective and potentially biased due to a small sample size of cases, subjective evaluations, or include an expert-generated case-mix versus a representative sample from the clinical field. Further, while digital platforms are available to assess radiograph interpretation skill against an objective standard, they have not adopted a data-driven competency standard which informs educators and the public that a physician has achieved adequate mastery to enter practice where they will be making high-stakes clinical decisions.

APPROACH

Operating on a purposeful sample of radiographs drawn from the clinical domain, we adapted the Ebel Method, an established standard setting method, to ascertain a defensible, clinically relevant mastery learning competency standard for the skill of radiograph interpretation as a model for deriving competency thresholds in visual diagnosis. Using a previously established digital platform, emergency physicians interpreted pediatric musculoskeletal extremity radiographs. Using one-parameter item response theory, these data were used to categorize radiographs by interpretation difficulty terciles (i.e. easy, intermediate, hard). A panel of emergency physicians, orthopedic surgeons, and plastic surgeons rated each radiograph with respect to clinical significance (low, medium, high). These data were then used to create a three-by-three matrix where radiographic diagnoses were categorized by interpretation difficulty and significance. Subsequently, a multidisciplinary panel that included medical and parent stakeholders determined acceptable accuracy for each of the nine cells. An overall competency standard was derived from the weighted sum. Finally, to examine consequences of implementing this standard, we reported on the types of diagnostic errors that may occur by adhering to the derived competency standard.

FINDINGS

To determine radiograph interpretation difficulty scores, 244 emergency physicians interpreted 1,835 pediatric musculoskeletal extremity radiographs. Analyses of these data demonstrated that the median interpretation difficulty rating of the radiographs was -1.8 logits (IQR -4.1, 3.2), with a significant difference of difficulty across body regions (p < 0.0001). Physician review classified the radiographs as 1,055 (57.8%) as low, 424 (23.1%) medium or 356 (19.1%) high clinical significance. The multidisciplinary panel suggested a range of acceptable scores between cells in the three-by-three table of 76% to 95% and the sum of equal-weighted scores resulted in an overall performance-based competency score of 85.5% accuracy. Of the 14.5% diagnostic interpretation errors that may occur at the bedside if this competency standard were implemented, 9.8% would be in radiographs of low-clinical significance, while 2.5% and 2.3% would be in radiographs of medium or high clinical significance, respectively.

CONCLUSION(S)

This study's novel integration of radiograph selection and a standard setting method could be used to empirically drive evidence-based competency standard for radiograph interpretation and can serve as a model for deriving competency thresholds for clinical tasks emphasizing visual diagnosis.

What do experts look at and what do experts find when reading mammograms?

Purpose: Radiologists sometimes fail to report clearly visible, clinically significant findings. Eye tracking can provide insight into the causes of such errors. Approach: We tracked eye movements of 17 radiologists, searching for masses in 80 mammograms (60 with masses). Results: Errors were classified using the Kundel et al. (1978) taxonomy: search errors (target never fixated), recognition errors (fixated < 500    ms ), or decision errors (fixated > 500    ms ). Error proportions replicated Krupinski (1996): search 25%, recognition 25%, and decision 50%. Interestingly, we found few differences between experts and residents in accuracy or eye movement metrics. Error categorization depends on the definition of the useful field of view (UFOV) around fixation. We explored different UFOV definitions, based on targeting saccades and search saccades. Targeting saccades averaged slightly longer than search saccades. Of most interest, we found that the probability that the eyes would move to the target on the next saccade or even on one of the next three saccades was strikingly low ( ∼ 33 % , even when the eyes were < 2    deg from the target). This makes it clear that observers do not fully process everything within a UFOV. Using a probabilistic UFOV, we find, unsurprisingly, that observers cover more of the image when no target is present than when it is found. Interestingly, we do not find evidence that observers cover too little of the image on trials when they miss the target. Conclusions: These results indicate that many errors in mammography reflect failed deployment of attention; not failure to fixate clinically significant locations.

Training focal lung pathology detection using an eye movement modeling example

Purpose: Published reports suggest that nonoptimal visual search behavior is associated with false negatives in chest x-ray interpretation. Eye movement modeling example (EMME)-based training interventions, that is, interventions showing models of visual search to trainees, have been shown to improve visual search as well as task accuracy. Approach: We examined the detection of focal lung pathology on chest x-rays before and after two different EMME training interventions that have been shown to be efficient: (i) an EMME showing moving fixations on a blurred background (spotlight group) and (ii) an EMME showing moving fixations on a nonblurred background (circle group). These two experimental groups were compared to a control group that was only provided with the correct location of pathologies on the chest x-rays. Results: Performance outcomes showed improved detection sensitivity and specificity in all groups (also the control group). It appears that repetitive exposure to pathologies on chest x-rays with feedback resulted in enhanced pattern recognition. In addition, visual search strategies became more efficient during post-tests. Conclusion: Repetitive exposure to a focal lung pathology detection task with feedback improves overall performance. However, the specific EMME training interventions did not add any further advantages. Similar training interventions can be provided online to assess feasibility and reproducibility of such (or similar) training formats. Such training can, for example, reduce the number of false negative errors, especially for novices.

A think-aloud study to inform the design of radiograph interpretation practice

Models for diagnostic reasoning in radiology have been based on the observed behaviors of experienced radiologists but have not directly focused on the thought processes of novices as they improve their accuracy of image interpretation. By collecting think-aloud verbal reports, the current study was designed to investigate differences in specific thought processes between medical students (novices) as they learn and radiologists (experts), so that we can better design future instructional environments. Seven medical students and four physicians with radiology training were asked to interpret and diagnose pediatric elbow radiographs where fracture is suspected. After reporting their diagnosis of a case, they were given immediate feedback. Participants were asked to verbalize their thoughts while completing the diagnosis and while they reflected on the provided feedback. The protocol analysis of their verbalizations showed that participants used some combination of four processes to interpret the case: gestalt interpretation, purposeful search, rule application, and reasoning from a prior case. All types of processes except reasoning from a prior case were applied significantly more frequently by experts. Further, gestalt interpretation was used with higher frequency in abnormal cases while purposeful search was used more often for normal cases. Our assessment of processes could help guide the design of instructional environments with well-curated image banks and analytics to facilitate the novice's journey to expertise in image interpretation.

Sequence matters: patterns in task-based clinical reasoning

Background

The cognitive pathways that lead to an accurate diagnosis and efficient management plan can touch on various clinical reasoning tasks (1). These tasks can be employed at any point during the clinical reasoning process and though the four distinct categories of framing, diagnosis, management, and reflection provide some insight into how these tasks map onto clinical reasoning, much is still unknown about the task-based clinical reasoning process. For example, when and how are these tasks typically used? And more importantly, do these clinical reasoning task processes evolve when patient encounters become complex and/or challenging (i.e. with contextual factors)?

Methods

We examine these questions through the lens of situated cognition, context specificity, and cognitive load theory. Sixty think-aloud transcripts from 30 physicians who participated in two separate cases – one with a contextual factor and one without – were coded for 26 clinical reasoning tasks (1). These tasks were organized temporally, i.e. when they emerged in their think-aloud process. Frequencies of each of the 26 tasks were aggregated, categorized, and visualized in order to analyze task category sequences.

Results

We found that (a) as expected, clinical tasks follow a general sequence, (b) contextual factors can distort this emerging sequence, and (c) the presence of contextual factors prompts more experienced physicians to clinically reason similar to that of less experienced physicians.

Conclusions

These findings add to the existing literature on context specificity in clinical reasoning and can be used to strengthen teaching and assessment of clinical reasoning.

A protocol analysis of use of forward and backward reasoning during valuation problem solving

Purpose

Behavioural studies of valuers have suggested that valuers rely on a number of cognitive strategies involving reasoning and intuition when undertaking a valuation task. However, there are few studies of the actual reasoning mechanisms in valuation. In other fields, much attention has been paid to forward and backward reasoning, as this shows the choices and decisions that are made in undertaking a complex task. This paper studied this during a valuation task. The purpose of this paper is twofold: first, to develop a methodological approach for empirical research on valuers’ reasoning, and, second, to report expert-novice differences on valuers’ use of forward and backward reasoning during a valuation problem solving.

Design/methodology/approach

The study utilised a verbal protocol analysis (VPA) to elicit think-aloud data from a purposive sample of a group of valuers of different levels of expertise undertaking a commercial-valuation task. Through a content analysis interpretive strategy, the transcripts were analysed into different cognitive segments identifying the forward and backward reasoning strategies.

Findings

The findings showed that valuers accomplished the valuation task by dividing the overall problem into sub-problems. These sub-problems are thereafter solved by integrating available data with existing knowledge by relying more on forward reasoning than backward reasoning. However, there were effects associated with the level of expertise in the way the processes of forward and backward reasoning are used, with the expert and intermediate valuers being more thorough and comprehensive in their reasoning process than the novices.

Research limitations/implications

This study explores the possibility that forward and backward reasoning play an important role in commercial valuation problem solving using a limited sample of valuers. Given this, data cannot be generalised to all valuation practice settings but may motivate future research that examines the effectiveness of forward and backward reasoning in diverse valuation practice settings and develops a holistic model of valuation reasoning.

Practical implications

The findings of this study are applicable to valuation practice. Future training efforts need to evaluate the usefulness of teaching problem solving and explicitly recognise forward and backward reasoning, along with other problem-solving strategies uncovered in this study, as standard training strategies for influencing the quality of valuation decisions.

Originality/value

By adopting VPA, this study employs an insightful and rich dataset which allows an interpretation of thoughts of valuers into cognitive reasoning strategies that provide a deeper level of understanding of how valuers solve valuation problem; this has not been possible in previous related valuation studies.

Development, intra- and inter-rater reliability of the New Zealand Secretion Scale (NZSS)

Purpose: The New Zealand Secretion Scale (NZSS) has been developed for comprehensive assessment of accumulated secretions during endoscopy. The scale rates secretion severity under the subcategories location, amount and response. This study describes the scale's development and reliability when used by experts and untrained raters. Method: One expert scored 254 endoscopy videos using the NZSS and performed repeat measures on 100 randomly selected videos one month later. These 100 videos were scored by a second expert in a randomised order. In a second arm of the study, 28 raters scored 10 endoscopy videos, without training on the NZSS. Seventeen had experience in endoscopy (mean 1.8 years, SD = 1.0). Reliability was calculated across the entire cohort and as a function of experience interpreting endoscopy. Result: Strong internal consistency (Cronbach's α = 0.88), and high inter-item (>0.60) and corrected item-total correlations (>0.70) were established. Inter-rater (ICC = 0.99) and intra-rater reliability (ICC = 0.95) of the experts was excellent. Inter-rater reliability of the untrained raters ranged from ICC = 0.65-0.79, with no significant influence of experience on reliability. Conclusion: The NZSS is a reliable assessment of secretion severity during endoscopy and can be used without training. This comprehensive scale will support research evaluating the predictive value of accumulated secretions.

Analysis of Perceptual Expertise in Radiology - Current Knowledge and a New Perspective

Radiologists rely principally on visual inspection to detect, describe, and classify findings in medical images. As most interpretive errors in radiology are perceptual in nature, understanding the path to radiologic expertise during image analysis is essential to educate future generations of radiologists. We review the perceptual tasks and challenges in radiologic diagnosis, discuss models of radiologic image perception, consider the application of perceptual learning methods in medical training, and suggest a new approach to understanding perceptional expertise. Specific principled enhancements to educational practices in radiology promise to deepen perceptual expertise among radiologists with the goal of improving training and reducing medical error.

Analysis of the French national evaluation of radiology residents

PURPOSE

In France, a national evaluation is given annually to radiology residents. The aim of this study was to perform both a docimological analysis of the quality of the questionnaire and a statistical analysis of the results.

MATERIALS AND METHODS

This retrospective study, which included French radiology residents from Year 1 to Year 5 of residency, was performed from 2015 to 2017 across 25 medical universities in France. Both qualitative and quantitative docimological analyses were performed as assessed by the Cronbach alpha coefficient, the difficulty of question (PDI), and the coefficient of discrimination (Rir). Results to the questionnaire were compared between years of residency.

RESULTS

The results of the analysis confirmed the quality of the questionnaire (Cronbach alpha coefficient=0.71, mean [PDI=0.40]) though the majority of questions could be answered by memory rather than cognitive ability. The mean Rir was 0.02, indicating that students could not be certified using only the questionnaire. The results measuring resident level of knowledge were moderate, with mean results ranging from 9.2/20 at the first year to 11.3/20 at the fifth year of residency (P<0.001). There were no significant differences in results obtained between the third, fourth, and fifth year of residency but results were significantly different among university hospitals.

CONCLUSION

Even if close interactions exist between learning and pedagogic environment, our results suggest that it may be useful to further develop an evaluation process in relation with pedagogic instructions in order to provide more optimal training.

Computer-based self-training for CT colonography with and without CAD

OBJECTIVES

To determine whether (1) computer-based self-training for CT colonography (CTC) improves interpretation performance of novice readers; (2) computer-aided detection (CAD) use during training affects learning.

METHODS

Institutional review board approval and patients' informed consent were obtained for all cases included in this study. Twenty readers (17 radiology residents, 3 radiologists) with no experience in CTC interpretation were recruited in three centres. After an introductory course, readers performed a baseline assessment test (37 cases) using CAD as second reader. Then they were randomized (1:1) to perform either a computer-based self-training (150 cases verified at colonoscopy) with CAD as second reader or the same training without CAD. The same assessment test was repeated after completion of the training programs. Main outcome was per lesion sensitivity (≥ 6 mm). A generalized estimating equation model was applied to evaluate readers' performance and the impact of CAD use during training.

RESULTS

After training, there was a significant improvement in average per lesion sensitivity in the unassisted phase, from 74% (356/480) to 83% (396/480) (p < 0.001), and in the CAD-assisted phase, from 83% (399/480) to 87% (417/480) (p = 0.021), but not in average per patient sensitivity, from 93% (390/420) to 94% (395/420) (p = 0.41), and specificity, from 81% (260/320) to 86% (276/320) (p = 0.15). No significant effect of CAD use during training was observed on per patient sensitivity and specificity, nor on per lesion sensitivity.

CONCLUSIONS

A computer-based self-training program for CTC improves readers' per lesion sensitivity. CAD as second reader does not have a significant impact on learning if used during training.

KEY POINTS

• Computer-based self-training for CT colonography improves per lesion sensitivity of novice readers. • Self-training program does not increase per patient specificity of novice readers. • CAD used during training does not have significant impact on learning.

The Holistic Processing Account of Visual Expertise in Medical Image Perception: A Review

In the field of medical image perception, the holistic processing perspective contends that experts can rapidly extract global information about the image, which can be used to guide their subsequent search of the image (Swensson, 1980; Nodine and Kundel, 1987; Kundel et al., 2007). In this review, we discuss the empirical evidence supporting three different predictions that can be derived from the holistic processing perspective: Expertise in medical image perception is domain-specific, experts use parafoveal and/or peripheral vision to process large regions of the image in parallel, and experts benefit from a rapid initial glimpse of an image. In addition, we discuss a pivotal recent study (Litchfield and Donovan, 2016) that seems to contradict the assumption that experts benefit from a rapid initial glimpse of the image. To reconcile this finding with the existing literature, we suggest that global processing may serve multiple functions that extend beyond the initial glimpse of the image. Finally, we discuss future research directions, and we highlight the connections between the holistic processing account and similar theoretical perspectives and findings from other domains of visual expertise.

How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology

Eye tracking research has been conducted for decades to gain understanding of visual diagnosis such as in radiology. For educational purposes, it is important to identify visual search patterns that are related to high perceptual performance and to identify effective teaching strategies. This review of eye-tracking literature in the radiology domain aims to identify visual search patterns associated with high perceptual performance. Databases PubMed, EMBASE, ERIC, PsycINFO, Scopus and Web of Science were searched using 'visual perception' OR 'eye tracking' AND 'radiology' and synonyms. Two authors independently screened search results and included eye tracking studies concerning visual skills in radiology published between January 1, 1994 and July 31, 2015. Two authors independently assessed study quality with the Medical Education Research Study Quality Instrument, and extracted study data with respect to design, participant and task characteristics, and variables. A thematic analysis was conducted to extract and arrange study results, and a textual narrative synthesis was applied for data integration and interpretation. The search resulted in 22 relevant full-text articles. Thematic analysis resulted in six themes that informed the relation between visual search and level of expertise: (1) time on task, (2) eye movement characteristics of experts, (3) differences in visual attention, (4) visual search patterns, (5) search patterns in cross sectional stack imaging, and (6) teaching visual search strategies. Expert search was found to be characterized by a global-focal search pattern, which represents an initial global impression, followed by a detailed, focal search-to-find mode. Specific task-related search patterns, like drilling through CT scans and systematic search in chest X-rays, were found to be related to high expert levels. One study investigated teaching of visual search strategies, and did not find a significant effect on perceptual performance. Eye tracking literature in radiology indicates several search patterns are related to high levels of expertise, but teaching novices to search as an expert may not be effective. Experimental research is needed to find out which search strategies can improve image perception in learners.

Identifying error types in visual diagnostic skill assessment

BACKGROUND

Misinterpretation of medical images is an important source of diagnostic error. Errors can occur in different phases of the diagnostic process. Insight in the error types made by learners is crucial for training and giving effective feedback. Most diagnostic skill tests however penalize diagnostic mistakes without an eye for the diagnostic process and the type of error. A radiology test with stepwise reasoning questions was used to distinguish error types in the visual diagnostic process. We evaluated the additional value of a stepwise question-format, in comparison with only diagnostic questions in radiology tests.

METHODS

Medical students in a radiology elective (n=109) took a radiology test including 11-13 cases in stepwise question-format: marking an abnormality, describing the abnormality and giving a diagnosis. Errors were coded by two independent researchers as perception, analysis, diagnosis, or undefined. Erroneous cases were further evaluated for the presence of latent errors or partial knowledge. Inter-rater reliabilities and percentages of cases with latent errors and partial knowledge were calculated.

RESULTS

The stepwise question-format procedure applied to 1351 cases completed by 109 medical students revealed 828 errors. Mean inter-rater reliability of error type coding was Cohen's κ=0.79. Six hundred and fifty errors (79%) could be coded as perception, analysis or diagnosis errors. The stepwise question-format revealed latent errors in 9% and partial knowledge in 18% of cases.

CONCLUSIONS

A stepwise question-format can reliably distinguish error types in the visual diagnostic process, and reveals latent errors and partial knowledge.

A Big Data and Learning Analytics Approach to Process-Level Feedback in Cognitive Simulations

Collecting and analyzing large amounts of process data for the purposes of education can be considered a big data/learning analytics (BD/LA) approach to improving learning. However, in the education of health care professionals, the application of BD/LA is limited to date. The authors discuss the potential advantages of the BD/LA approach for the process of learning via cognitive simulations. Using the lens of a cognitive model of radiograph interpretation with four phases (orientation, searching/scanning, feature detection, and decision making), they reanalyzed process data from a cognitive simulation of pediatric ankle radiography where 46 practitioners from three expertise levels classified 234 cases online. To illustrate the big data component, they highlight the data available in a digital environment (time-stamped, click-level process data). Learning analytics were illustrated using algorithmic computer-enabled approaches to process-level feedback.For each phase, the authors were able to identify examples of potentially useful BD/LA measures. For orientation, the trackable behavior of re-reviewing the clinical history was associated with increased diagnostic accuracy. For searching/scanning, evidence of skipping views was associated with an increased false-negative rate. For feature detection, heat maps overlaid on the radiograph can provide a metacognitive visualization of common novice errors. For decision making, the measured influence of sequence effects can reflect susceptibility to bias, whereas computer-generated path maps can provide insights into learners' diagnostic strategies.In conclusion, the augmented collection and dynamic analysis of learning process data within a cognitive simulation can improve feedback and prompt more precise reflection on a novice clinician's skill development.

Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice

As a part of a special collection in this issue of Academic Medicine, which is focused on mastery learning in medical education, this Perspective describes how the expert-performance approach with deliberate practice is consistent with many characteristics of mastery learning. Importantly, this Perspective also explains how the expert-performance approach provides a very different perspective on the acquisition of skill. Whereas traditional education with mastery learning focuses on having students attain an adequate level of performance that is based on goals set by the existing curricula, the expert-performance approach takes an empirical approach and first identifies the final goal of training-namely, reproducibly superior objective performance (superior patient outcomes) for individuals in particular medical specialties. Analyzing this superior complex performance reveals three types of mental representations that permit expert performers to plan, execute, and monitor their own performance. By reviewing research on medical performance and education, the author describes evidence for these representations and their development within the expert-performance framework. He uses the research to generate suggestions for improved training of medical students and professionals. Two strategies-designing learning environments with libraries of cases and creating opportunities for individualized teacher-guided training-should enable motivated individuals to acquire a full set of refined mental representations. Providing the right resources to support the expert-performance approach will allow such individuals to become self-regulated learners-that is, members of the medical community who have the tools to improve their own and their team members' performances throughout their entire professional careers.

Development of a personalized training system using the Lung Image Database Consortium and Image Database resource Initiative Database

RATIONALE AND OBJECTIVES

The aim of this study was to develop a personalized training system using the Lung Image Database Consortium (LIDC) and Image Database resource Initiative (IDRI) Database, because collecting, annotating, and marking a large number of appropriate computed tomography (CT) scans, and providing the capability of dynamically selecting suitable training cases based on the performance levels of trainees and the characteristics of cases are critical for developing a efficient training system.

MATERIALS AND METHODS

A novel approach is proposed to develop a personalized radiology training system for the interpretation of lung nodules in CT scans using the Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) database, which provides a Content-Boosted Collaborative Filtering (CBCF) algorithm for predicting the difficulty level of each case of each trainee when selecting suitable cases to meet individual needs, and a diagnostic simulation tool to enable trainees to analyze and diagnose lung nodules with the help of an image processing tool and a nodule retrieval tool.

RESULTS

Preliminary evaluation of the system shows that developing a personalized training system for interpretation of lung nodules is needed and useful to enhance the professional skills of trainees.

CONCLUSIONS

The approach of developing personalized training systems using the LIDC/IDRL database is a feasible solution to the challenges of constructing specific training program in terms of cost and training efficiency.

A Performance Weighted Collaborative Filtering algorithm for personalized radiology education

Devising an accurate prediction algorithm that can predict the difficulty level of cases for individuals and then selects suitable cases for them is essential to the development of a personalized training system. In this paper, we propose a novel approach, called Performance Weighted Collaborative Filtering (PWCF), to predict the difficulty level of each case for individuals. The main idea of PWCF is to assign an optimal weight to each rating used for predicting the difficulty level of a target case for a trainee, rather than using an equal weight for all ratings as in traditional collaborative filtering methods. The assigned weight is a function of the performance level of the trainee at which the rating was made. The PWCF method and the traditional method are compared using two datasets. The experimental data are then evaluated by means of the MAE metric. Our experimental results show that PWCF outperforms the traditional methods by 8.12% and 17.05%, respectively, over the two datasets, in terms of prediction precision. This suggests that PWCF is a viable method for the development of personalized training systems in radiology education.

A content-boosted collaborative filtering algorithm for personalized training in interpretation of radiological imaging

Devising a method that can select cases based on the performance levels of trainees and the characteristics of cases is essential for developing a personalized training program in radiology education. In this paper, we propose a novel hybrid prediction algorithm called content-boosted collaborative filtering (CBCF) to predict the difficulty level of each case for each trainee. The CBCF utilizes a content-based filtering (CBF) method to enhance existing trainee-case ratings data and then provides final predictions through a collaborative filtering (CF) algorithm. The CBCF algorithm incorporates the advantages of both CBF and CF, while not inheriting the disadvantages of either. The CBCF method is compared with the pure CBF and pure CF approaches using three datasets. The experimental data are then evaluated in terms of the MAE metric. Our experimental results show that the CBCF outperforms the pure CBF and CF methods by 13.33 and 12.17 %, respectively, in terms of prediction precision. This also suggests that the CBCF can be used in the development of personalized training systems in radiology education.

A hinting strategy for online learning of radiograph interpretation by medical students

CONTEXT

We examined whether a 'hint' manoeuvre increases the time novice medical learners spend on reviewing a radiograph, thereby potentially increasing their interpretation accuracy.

METHODS

Senior year medical students were recruited into a randomised control, three-arm, multicentre trial. Students reviewed an online 50-case learning set that varied in degree of 'hint' intervention. The 'hint' was a dialogue box that appeared after a student submitted an answer, encouraging the student to re-evaluate their interpretation. The students in the control group received no hints. In the weak intervention group, students received 'hints' with 66% of their incorrect interpretations and 33% of those that were correct. In the strong intervention group, the incorrect interpretation hint frequency was 80%, whereas for correct responses it was 20%. All students completed a 20-case post-test immediately and 2 weeks after the 50 cases. The primary outcome was student performance on the immediate post-test, measured as the ability to discriminate between normal and abnormal films (dPrime). Secondary outcomes included the probability of considering the hint, time spent on learning cases and knowledge retention at 2 weeks.

RESULTS

We enrolled 117 medical students from three sites into the three study groups: control (36), weak intervention (40) and strong intervention (41) groups. The mean (standard deviation) dPrime in the control, weak and strong groups were 0.4 (1.1), 0.7 (1.1) and 0.4 (0.9), respectively (P = 0.4). In the weak and strong groups, participants reconsidered answers in 556 of 1944 (28.6%) hinting opportunities, and those who reconsidered their answers spent a mean (95% confidence interval) of 13.9 (11.9, 16.0) seconds longer on each case. There were no significant differences in knowledge retention at 2 weeks between the groups (P = 0.2).

CONCLUSIONS

Although the implemented hinting strategy did result in students spending more time considering a proportion of the cases, overall it was not effective in improving student performance.

Construct validation of progress testing to measure knowledge and visual skills in radiology

BACKGROUND

The Dutch Radiology Progress Test (DRPT) monitors the acquisition of knowledge and visual skills of radiology residents in the Netherlands.

AIM

We aimed to evaluate the quality of progress testing in postgraduate radiology training by studying the reliability of the DRPT and finding an indication for its construct validity. We expected that knowledge would increase rapidly in the first years of residency, leveling-off in later years, to allow for the development of visual skills. We hypothesized that scores on the DRPT reflect this pattern.

METHODS

Internal consistencies were estimated with Cronbach's alpha. Performance increase over program years were tested using one-way analysis of variance.

RESULTS

Data were available for 498 residents (2281 test results). Reliabilities were around Cronbach's alpha 0.90. There was a significant difference in the mean test results between the first three years of residency. After the fourth year no significant increase in test scores on knowledge could be measured on eight tests. The same pattern occurred for scores on visual skills. However, visual skills scores tend to increase more sharply than knowledge scores.

CONCLUSION

We found support for the reliability and construct validity of the DRPT. However, assessment on visual skill development needs further exploration.

Feasibility and satisfaction with a tailored web-based audit intervention for recalibrating radiologists' thresholds for conducting additional work-up

RATIONALE AND OBJECTIVES

To examine the feasibility of and satisfaction with a tailored web-based intervention designed to decrease radiologists' recommendation of inappropriate additional work-up after a screening mammogram.

MATERIALS AND METHODS

We developed a web-based educational intervention designed to reduce inappropriate recall. Radiologists were randomly assigned to participate in an early intervention group or a late (control) intervention group, the latter of which served as a control for a 9-month follow-up period, after which they were invited to participate in the intervention. Intervention content was derived from our prior research and included three modules: 1) an introduction to audit statistics for mammography performance; 2) a review of data showing radiologists' inflated perceptions of medical malpractice risks related to breast imaging, and 3) a review of data on breast cancer risk among women seen in their practices. Embedded within the intervention were individualized audit data for each participating radiologists obtained from the national Breast Cancer Surveillance Consortium.

RESULTS

Seventy-four radiologists (37.8%; 74/196) consented to the intervention, which was completed by 67.5% (27/40) of those randomized to the early intervention group and 41.2% (14/34) of those randomized to the late (control) group. Thus, a total of 41 (55%) completed the intervention. On average, three log-ins were used to complete the program (range 1-14), which took approximately 1 hour. Ninety-five percent found the program moderately to very helpful in understanding how to calculate basic performance measures. Ninety-three percent found viewing their own performance measures moderately to very helpful, and 83% reported it being moderately to very important to learn that the breast cancer risk in their screening population program was lower than perceived. The percentage of radiologists who reported that the risk of medical malpractice influences their recall rates dropped from 36.3% preintervention to 17.8% after intervention with a similar drop in perceived influence of malpractice risk on their recommendations for breast biopsy (36.4 to 17.3%). More than 75% of radiologists answered the postintervention knowledge questions correctly, and the percent of time spent in breast imaging did not appear to influence responses. The majority (>92%) of participants correctly responded that the target recall rate in the United States is 9%. The mean self-reported recall rates were 13.0 for radiologists spending <40% time in breast imaging and 14.9% for those spending >40% time spent in breast imaging, which was highly correlated with their actual recall rates (0.991; P < .001).

CONCLUSIONS

Radiologists who begin an internet-based tailored intervention designed to help reduce unnecessary recall will likely complete it, although only 55% who consented to the study actually undertook the intervention. Participants found the program useful in helping them understand why their recall rates may be elevated.

Evaluation of a standardized CT colonography training program for novice readers

PURPOSE

To determine how many computed tomographic (CT) colonography training studies have to be evaluated by novice readers to obtain an adequate level of competence in polyp detection.

MATERIALS AND METHODS

The study was approved by the Institutional Review Board. Informed consent was obtained from all participants. Six physicians (one radiologist, three radiology residents, two researchers) and three technicians completed a CT colonography training program. Two hundred CT colonographic examinations with colonoscopic verification were selected from a research database, with 100 CT colonographic examinations with at least one polyp 6 mm or larger. After a lecture session and short individual hands-on training, CT colonography training was done individually with immediate feedback of colonoscopy outcome. Per-polyp sensitivity was calculated for four sets of 50 CT colonographic examinations for lesions 6 mm or larger. By using logistic regression analyses, the number of CT colonographic examinations to reach 90% sensitivity for lesions 6 mm or larger was estimated. Reading times were registered.

RESULTS

The average per-polyp sensitivity for lesions 6 mm or larger was 76% (207 of 270) in the first set of 50 CT colonographic examinations, 77% (262 of 342) in the second (P = .96 vs first set), 80% (310 of 387) in the third (P = .67 vs first set), and 91% (261 of 288) in the fourth (P = .018). The estimated number of CT colonographic examinations for a sufficient sensitivity was 164. Six of nine readers reached this level of competence within 175 CT colonographic examinations. Reading times decreased significantly from the first to the second set of 50 CT colonographic examinations for six readers.

CONCLUSION

Novice CT colonography readers obtained sensitivity equal to that of experienced readers after practicing on average 164 CT colonographic studies.

When radiologists perform best: the learning curve in screening mammogram interpretation

PURPOSE

To examine changes in screening mammogram interpretation as radiologists with and radiologists without fellowship training in breast imaging gain clinical experience.

MATERIALS AND METHODS

In an institutional review board-approved HIPAA-compliant study, the performance of 231 radiologists who interpreted screen-film screening mammograms from 1996 to 2005 at 280 facilities that contribute data to the Breast Cancer Surveillance Consortium was examined. Radiologists' demographic data and clinical experience levels were collected by means of a mailed survey. Mammograms were grouped on the basis of how many years the interpreting radiologist had been practicing mammography, and the influence of increasing experience on performance was examined separately for radiologists with and those without fellowship training in breast imaging, taking into account case-mix and radiologist-level differences.

RESULTS

A total of 1 599 610 mammograms were interpreted during the study period. Performance for radiologists without fellowship training improved most during their 1st 3 years of clinical practice, when the odds of a false-positive reading dropped 11%-15% per year (P < .015) with no associated decrease in sensitivity (P > .89). The number of women recalled per breast cancer detected decreased from 33 for radiologists in their 1st year of practice to 24 for radiologists with 3 years of experience to 19 for radiologists with 20 years of experience. Radiologists with fellowship training in breast imaging experienced no learning curve and reached desirable goals during their 1st year of practice.

CONCLUSION

Radiologists' interpretations of screening mammograms improve during their first few years of practice and continue to improve throughout much of their careers. Additional residency training and targeted continuing medical education may help reduce the number of work-ups of benign lesions while maintaining high cancer detection rates.

Conspicuity of microcalcifications on digital screening mammograms using varying degrees of monitor zooming

RATIONALE AND OBJECTIVES

American College of Radiology guidelines suggest that digital screening mammographic images should be viewed at the full resolution at which they were acquired. This slows interpretation speed. The aim of this study was to examine the effect of various levels of zooming on the detection and conspicuity of microcalcifications.

MATERIALS AND METHODS

Six radiologists viewed 40 mammographic images five times in different random orders using five different levels of zooming: full resolution (100%) and 30%, 61%, 88%, and 126% of that size. Thirty-three images contained microcalcifications varying in subtlety, all associated with breast cancer. The clusters were circled. Seven images contained no malignant calcifications but also had randomly placed circles. The radiologists graded the presence or absence and visual conspicuity of any calcifications compared to calcifications in a reference image. They also counted the microcalcifications.

RESULTS

The radiologists saw the microcalcifications in 94% of the images at 30% size and in either 99% or 100% of the other tested levels of zooming. Conspicuity ratings were worst for the 30% size and fairly similar for the others. Using the 30% size, two radiologists failed to see the microcalcifications on either the craniocaudal or mediolateral oblique view taken from one patient. Interobserver agreement regarding the number of calcifications was lowest for the 30% images and second lowest for the 100% images.

CONCLUSIONS

Images at 30% size should not be relied on alone for systematic scanning for microcalcifications. The other four levels of magnification all performed well enough to warrant further testing.