The challenges of studying visual expertise in medical image diagnosis

Diagnostic errors during perceptual learning in dermatology: a prospective cohort study of Finnish undergraduate students

BACKGROUND

Perceptual learning modules (PLMs) have been shown to significantly improve learning outcomes in teaching dermatology.

OBJECTIVES

To investigate the quantity and quality of diagnostic errors made during undergraduate PLMs and their potential implications.

METHODS

The study data were acquired from 8 successive dermatology courses (2021-23) from 142 undergraduate medical students. Digital PLMs were held before, during and after the courses. We investigated the number and distribution of diagnostic errors, differences between specific skin conditions and classified the errors based on type.

RESULTS

Diagnostic errors were not randomly distributed. Some skin conditions were almost always correctly identified, whereas a significant number of errors were made for other diagnoses. Errors were classified into one of three groups: mostly systematic errors of relevant differential diagnoses ('similarity' errors); partly systematic errors ('mixed' errors); and 'random' errors. While a significant learning effect during the repeated measures was found in accuracy (P < 0.001, η²P = 0.64), confidence (P < 0.001, η²P = 0.60) and fluency (P < 0.001, η²P = 0.16), the three categories differed in all outcome measures (all P < 0.001, all η²P > 0.47). Visual learning was more difficult for diagnoses in the similarity category (all P < 0.001, all η²P > 0.12) than for those in the mixed and random categories.

CONCLUSIONS

Error analysis of PLMs provided relevant information about learning efficacy and progression, and systematic errors in tasks and more difficult-to-learn conditions. This information could be used in the development of adaptive, individual error-based PLMs to improve learning outcomes, both in dermatology and medical education in general.

Use of eye tracking in medical education

Eye tracking has become increasingly applied in medical education research for studying the cognitive processes that occur during the performance of a task, such as image interpretation and surgical skills development. However, analysis and interpretation of the large amount of data obtained by eye tracking can be confusing. In this article, our intention is to clarify the analysis and interpretation of the data obtained from eye tracking. Understanding the relationship between eye tracking metrics (such as gaze, pupil and blink rate) and cognitive processes (such as visual attention, perception, memory and cognitive workload) is essential. The importance of calibration and how the limitations of eye tracking can be overcome is also highlighted.

Factors related to the performance of laypersons diagnosing pigmented skin cancer: an explorative study

It is important but challenging for prospective health professionals to learn the visual distinction between potentially harmful and harmless skin lesions, such as malignant melanomas and benign nevi. Knowledge about factors related to diagnostic performance is sparse but a prerequisite for designing and evaluating evidence-based educational interventions. Hence, this study explored how the characteristics of 240 skin lesions, the number of classified lesions and the response times of 137 laypeople were related to performance in diagnosing pigmented skin cancer. Our results showed large differences between the lesions, as some were classified correctly by more than 90% and others by less than 10% of the participants. A t-test showed that for melanomas, the correct diagnosis was provided significantly more often than for nevi. Furthermore, we found a significant Pearson correlation between the number of solved tasks and performance in the first 50 diagnostic tasks. Finally, t-tests for investigating the response times revealed that compared to true decisions, participants spent longer on false-negative but not on false-positive decisions. These results provide novel knowledge about performance-related factors that can be useful when designing diagnostic tests and learning interventions for melanoma detection.

Webcam-based eye-tracking to measure visual expertise of medical students during online histology training

Objectives

Visual expertise is essential for image-based tasks that rely on visual cues, such as in radiology or histology. Studies suggest that eye movements are related to visual expertise and can be measured by near-infrared eye-tracking. With the popularity of device-embedded webcam eye-tracking technology, cost-effective use in educational contexts has recently become amenable. This study investigated the feasibility of such methodology in a curricular online-only histology course during the 2021 summer term.

Methods

At two timepoints (t1 and t2), third-semester medical students were asked to diagnose a series of histological slides while their eye movements were recorded. Students' eye metrics, performance and behavioral measures were analyzed using variance analyses and multiple regression models.

Results

First, webcam-eye tracking provided eye movement data with satisfactory quality (mean accuracy=115.7 px±31.1). Second, the eye movement metrics reflected the students' proficiency in finding relevant image sections (fixation count on relevant areas=6.96±1.56 vs. irrelevant areas=4.50±1.25). Third, students' eye movement metrics successfully predicted their performance (R2adj=0.39, p<0.001).

Conclusion

This study supports the use of webcam-eye-tracking expanding the range of educational tools available in the (digital) classroom. As the students' interest in using the webcam eye-tracking was high, possible areas of implementation will be discussed.

Perfusion-only imaging in pregnant women: A comparative reader study with implications for practice patterns

This study seeks to understand the value of ventilation imaging in pregnant patients imaged for suspected pulmonary embolism (PE). Ventilation-perfusion (VQ) scans in this high-risk population were compared to ventilation-only scans. We hypothesize that in this relatively healthy population, the exclusion of ventilation scans will not impact the rate of scans interpreted as positive. This retrospective blinded comparative reader study on collated VQ scans performed on pregnant patients in the course of routine clinical care in a > 5 year period (03/2012 to 07/2017). Each set of VQ and perfusion only (Q) studies were reviewed by 8 readers (4 nuclear radiology fellows and 4 nuclear medicine faculty) in random order; the Q scans simply omitted the ventilation images. Readers recorded each study as PE, no PE, or non-diagnostic (prospective investigative study of acute PE diagnosis classifications). Logistic mixed effects models were used to test the association between scan type (VQ vs Q). 203 pairs of studies in 197 patients were included (6 patients had 2 scans). Subjects ranged from 14 to 45 years of age, with a median 28 years. A significant association between scan type and positive/negative classification. Q-scans received more positive classifications than VQ-scans (median of 7.6% vs 6.7%). No association was seen between scan type and positive/indeterminate classification, nor between scan type and negative/indeterminate classification. The exclusion of ventilation images in VQ-scans was associated with a higher rate of positive studies, but this difference was small (<1%). Given the overwhelmingly normal percentage of Q-exams (>90% in our study), and the benefits of omitting ventilation imaging, perfusion-only imaging should be considered a reasonable option for imaging the pregnant patient to exclude PE.

Differences in Gaze Fixation Location and Duration Between Resident and Fellowship Sonographers Interpreting a Focused Assessment With Sonography in Trauma

OBJECTIVES

We quantified the gaze fixation duration of resident and fellowship sonographers interpreting a prerecorded focused assessment with sonography in trauma (FAST). We hypothesized that all sonographers would fixate on each relevant anatomic relationship but that the duration of fixation would differ.

METHODS

We conducted a cross-sectional study collecting and analyzing the gaze fixations of a convenience sample of current resident and fellowship sonographers. All sonographers viewed a standardized FAST video, and their gaze fixations were recorded using a Tobii X3-120 eye-tracking bar. Gaze fixations over nine anatomic regions of interest (ROIs) were identified. These were assessed for normality and analyzed using the Wilcoxon rank sum test at an alpha of 0.05 and Bonferroni correction p value of <0.0034. The chi-square test and Pearson's correlation were performed to assess statistical association.

RESULTS

The gaze fixation recordings of 24 resident and eight fellowship sonographers were suitable for analysis. Fourteen of the 24 resident sonographers viewed all ROIs in the FAST, whereas all eight fellowship sonographers viewed each of the nine relevant ROIs. Five ROIs were identified over which at least one resident sonographer did not have a gaze fixation. No statistically significant difference was identified between groups. Resident sonographers gaze fixated over the left upper quadrant (LUQ) splenorenal interface for a median (interquartile range) of 10.64 (9.73-11.60) seconds. The fellowship group viewed the same ROI for 8.43 (6.64-8.95) seconds (p < 0.003). All participants viewed this ROI. No other ROIs had a statistical difference.

CONCLUSION

Five ROIs were identified that were not visually interrogated by all resident sonographers. Only 14 of 24 resident sonographers visually interrogated every area in the FAST, whereas all fellowship sonographers interrogated every ROI. A statistically significant difference was found in gaze fixation duration between resident and fellowship sonographers in one ROI. Further study is required for gaze fixation assessment to become a tool for the interpretation component of point-of-care ultrasound.

How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study

The interpretation of medical images is an error-prone process that may yield severe consequences for patients. In dental medicine panoramic radiography (OPT) is a frequently used diagnostic procedure. OPTs typically contain multiple, diverse anomalies within one image making the diagnostic process very demanding, rendering students’ development of visual expertise a complex task. Radiograph interpretation is typically taught through massed practice; however, it is not known how effective this approach is nor how it changes students’ visual inspection of radiographs. Therefore, this study investigated how massed practice–an instructional method that entails massed learning of one type of material–affects processing of OPTs and the development of diagnostic performance. From 2017 to 2018, 47 dental students in their first clinical semester diagnosed 10 OPTs before and after their regular massed practice training, which is embedded in their curriculum. The OPTs contained between 3 to 26 to-be-identified anomalies. During massed practice they diagnosed 100 dental radiographs without receiving corrective feedback. The authors recorded students’ eye movements and assessed the number of correctly identified and falsely marked low- and high prevalence anomalies before and after massed practice. Massed practice had a positive effect on detecting anomalies especially with low prevalence (p < .001). After massed practice students covered a larger proportion of the OPTs (p < .001), which was positively related to the detection of low-prevalence anomalies (p = .04). Students also focused longer, more frequently, and earlier on low-prevalence anomalies after massed practice (p_s < .001). While massed practice improved visual expertise in dental students with limited prior knowledge, there is still substantial room for improvement. The results suggest integrating massed practice with more deliberate practice, where, for example, corrective feedback is provided, and support is adapted to students’ needs.

A think-aloud study to inform the design of radiograph interpretation practice

Models for diagnostic reasoning in radiology have been based on the observed behaviors of experienced radiologists but have not directly focused on the thought processes of novices as they improve their accuracy of image interpretation. By collecting think-aloud verbal reports, the current study was designed to investigate differences in specific thought processes between medical students (novices) as they learn and radiologists (experts), so that we can better design future instructional environments. Seven medical students and four physicians with radiology training were asked to interpret and diagnose pediatric elbow radiographs where fracture is suspected. After reporting their diagnosis of a case, they were given immediate feedback. Participants were asked to verbalize their thoughts while completing the diagnosis and while they reflected on the provided feedback. The protocol analysis of their verbalizations showed that participants used some combination of four processes to interpret the case: gestalt interpretation, purposeful search, rule application, and reasoning from a prior case. All types of processes except reasoning from a prior case were applied significantly more frequently by experts. Further, gestalt interpretation was used with higher frequency in abnormal cases while purposeful search was used more often for normal cases. Our assessment of processes could help guide the design of instructional environments with well-curated image banks and analytics to facilitate the novice's journey to expertise in image interpretation.

Undergraduate Radiology Education During the COVID-19 Pandemic: A Review of Teaching and Learning Strategies [Formula: see text]

The Coronavirus disease 2019 (COVID-19) pandemic has altered how medical education is delivered, worldwide. Didactic sessions have transitioned to electronic/online platforms and clinical teaching opportunities are limited. These changes will affect how radiology is taught to medical students at both the pre-clerkship (ie, year 1 and 2) and clinical (ie, year 3 and 4) levels. In the pre-clerkship learning environment, medical students are typically exposed to radiology through didactic lectures, integrated anatomy laboratories, case-based learning, and ultrasound clinical skills sessions. In the clinical learning environment, medical students primarily shadow radiologists and radiology residents and attend radiology resident teaching sessions. These formats of radiology education, which have been the tenets of the specialty, pose significant challenges during the pandemic. This article reviews how undergraduate radiology education is affected by COVID-19 and explores solutions for teaching and learning based on e-learning and blended learning theory.

Medical Students' Progress in Detecting and Interpreting Visual and Auditory Clinical Findings

Phenomenon: Detection of visual and auditory clinical findings is part of medical students' core clinical performance abilities that a medical education curriculum should teach, assess, and remediate. However, there is a limited understanding of how students develop these skills. While training physical exam technical skills has received significant attention and emphasis, teaching and assessing medical students' ability to detect and interpret visual and auditory clinical findings skills has been less systematic. Therefore, the purpose of this study is to investigate how medical students' visual and auditory clinical findings skills progress and develop over their four years of undergraduate medical education. This study will provide educators insights that can guide curriculum refinements that lead to improving students' abilities in this area. Approach: A computer-based progress exam was created to measure the longitudinal development of students' abilities to detect and interpret visual and auditory findings. After pilot testing, sixty test items were developed in collaboration with six clinical faculty members and two medical education researchers. The exam includes detection and description of ECG, x-ray, heart sounds, breath sounds, skin lesions, and movement findings. The exam was administered to students at the beginning of each training year since 2014. Additionally, the exam was administered to the Class of 2017 prior to their graduation. Measurement validity and reliability tests were conducted. Descriptive statistics and ANOVA were used to determine progress. Findings: More than 98% of students in four years of training completed the exam each year. The exam instrument had high reliabilities and demonstrated acceptable concurrent validity when compared with other academic performance data. Findings showed that students' visual and auditory clinical findings skills increased each training year until their fourth year. There was no performance improvement between incoming Year 4 students and graduating Year 4 students. While group means increased, class performance did not become more homogeneous across four years. Longitudinal data showed the same performance patterns as the cross-sectional data. Performance of the bottom quartile of graduating fourth-year students was not significantly higher than the performance of the top quartile of incoming first-year students who had not had formal medical training. Insights: A longitudinal study to follow learners' performance in detecting and interpreting visual and auditory clinical findings can provide meaningful insights regarding the effects of medical training programs on performance growth. The present study suggests that our medical curriculum is not effective in bringing all students to a higher level of performance in detecting and interpreting visual and auditory clinical findings. This study calls for further investigation how medical students can develop visual and auditory detection and interpretation skills in undergraduate medical education. There is a need for planned curriculum and assessment of medical students' skills in detecting and interpreting visual and auditory clinical findings.

Learning From Peers' Eye Movements in the Absence of Expert Guidance: A Proof of Concept Using Laboratory Stock Trading, Eye Tracking, and Machine Learning

Existing research shows that people can improve their decision skills by learning what experts paid attention to when faced with the same problem. However, in domains like financial education, effective instruction requires frequent, personalized feedback given at the point of decision, which makes it time-consuming for experts to provide and thus, prohibitively costly. We address this by demonstrating an automated feedback mechanism that allows amateur decision-makers to learn what information to attend to from one another, rather than from an expert. In the first experiment, eye movements of N = 100 subjects were recorded while they repeatedly performed a standard behavioral finance investment task. Consistent with previous studies, we found that a significant proportion of subjects were affected by decision bias. In the second experiment, a different group of N = 100 subjects faced the same task but, after each choice, they received individual, machine learning-generated feedback on whether their pre-decision eye movements resembled those made by Experiment 1 subjects prior to good decisions. As a result, Experiment 2 subjects learned to analyze information similarly to their successful peers, which in turn reduced their decision bias. Furthermore, subjects with low Cognitive Reflection Test scores gained more from the proposed form of process feedback than from standard behavioral feedback based on decision outcomes.

[Peer Teaching in Ultrasound Education - A Narrative Review]

Peer Teaching in Ultrasound Education - A Narrative Review Abstract. Learning about ultrasound is becoming an increasingly important component of the undergraduate medical curriculum. However, teaching about this is very time-consuming, which is why many universities use peer teaching in ultrasound training. Peer teaching has various advantages that go beyond reducing specialists' teaching time: students may learn more from their peers because their cognitive schemata are more congruent and they are more likely to be open about their learning deficits. In addition, the process of teaching leads to increased learning and motivation in the peer teachers themselves. Most studies that compare peer teaching with teaching from medically qualified tutors show comparable learning effects, with different advantages for the two settings.

Reversal of the hanging protocol of Contrast Enhanced Mammography leads to similar diagnostic performance yet decreased reading times

OBJECTIVES

Contrast-enhanced mammography (CEM) was found superior to Full-Field Digital Mammography (FFDM) for breast cancer detection. Current hanging protocols show low-energy (LE, similar to FFDM) images first, followed by recombined (RC) images. However, evidence regarding which hanging protocol leads to the most efficient reading process and highest diagnostic performance is lacking. This study investigates the effects of hanging-protocol ordering on the reading process and diagnostic performance of breast radiologists using eye-tracking methodology. Furthermore, it investigates differences in reading processes and diagnostic performance between LE, RC and FFDM images.

MATERIALS AND METHODS

Twenty-seven breast radiologists were randomized into three reading groups: LE-RC (commonly used hangings), RC-LE (reversed hangings) and FFDM. Thirty cases (nine malignant) were used. Fixation count, net dwell time and time-to-first fixation on malignancies as visual search measures were registered by the eye-tracker. Reading time per image was measured. Participants clicked on suspicious lesions to determine sensitivity and specificity. Area-under-the-ROC-curve (AUC) values were calculated.

RESULTS

RC-LE scored identical on visual search measures, t(16)= -1.45, p =  .17 or higher-p values, decreased reading time with 31%, t(16)= -2.20, p = .04, while scoring similar diagnostic performance compared to LE-RC, t(13.2) = -1.39, p - .20 or higher p-values. The reading process was more efficient on RC compared to LE. Diagnostic performance of CEM was superior to FFDM; F (2,26) = 16.1, p <  .001. Average reading time did not differ between the three groups, F (2,25) = 3.15, p =  .06.

CONCLUSION

The reversed CEM hanging protocol (RC-LE) scored similar on diagnostic performance compared to LE-RC, while reading time was a third faster. Abnormalities were interpreted quicker on RC images. A RC-LE hanging protocol is therefore recommended for clinical practice and training. Diagnostic performance of CEM was (again) superior to FFDM.

The effect of testing and feedback on the forgetting curves for radiograph interpretation skills

Objectives: Forgetting curves plot skill decay over time. After exposure to a simulation-based radiograph interpretation learning system, we determined the rate of learning decay and how this was impacted by testing (with and without feedback). Further, we examined the association of initial learning parameters on the forgetting curve. Methods: This was a multicenter, four-arm randomized control trial. Medical trainees completed 80 elbow radiographs and a 20-case post-test. Group 1 had no testing until 12 months; Groups 2-4 had testing every 2 months until 12 months. At 6 months, Group 3 testing was feedback-enhanced, while Group 4 had feedback-enhanced testing at 2, 6, and 10 months. Results: There were 106 participants (n = 42 Group 1; n = 22 Groups 2 and 3; n = 20 Group 4). Group 1 showed an -8.1% learning decay at 12-months relative to other groups. In Groups 2, 3, and 4, there was no significant learning decay (+0.8%), and there were no differences in skill decay between these groups. Initial score and learning curve slope were predictive of retained skill. Conclusions: Learning decay was mitigated by exposure to 20 test cases (with and without feedback) every two months. Initial learning parameters predicted learning retention and may inform refresher education scheduling.

Analysis of Perceptual Expertise in Radiology - Current Knowledge and a New Perspective

Radiologists rely principally on visual inspection to detect, describe, and classify findings in medical images. As most interpretive errors in radiology are perceptual in nature, understanding the path to radiologic expertise during image analysis is essential to educate future generations of radiologists. We review the perceptual tasks and challenges in radiologic diagnosis, discuss models of radiologic image perception, consider the application of perceptual learning methods in medical training, and suggest a new approach to understanding perceptional expertise. Specific principled enhancements to educational practices in radiology promise to deepen perceptual expertise among radiologists with the goal of improving training and reducing medical error.

Starting to Think Like an Expert: An Analysis of Resident Cognitive Processes During Simulation-Based Resuscitation Examinations

STUDY OBJECTIVE

Simulation is commonly used to teach crisis resource management skills and assess them in emergency medicine residents. However, our understanding of the cognitive processes underlying crisis resource management skills is limited because these processes are difficult to assess and describe. The objective of this study is to uncover and characterize the cognitive processes underlying crisis resource management skills and to describe how these processes vary between residents according to performance in a simulation-based examination.

METHODS

Twenty-two of 24 eligible emergency medicine trainees from 1 tertiary academic center completed 1 or 2 resuscitation-based examinations in the simulation laboratory. Resident performance was assessed by a blinded expert using an entrustment-based scoring tool. Participants wore eye-tracking glasses that generated first-person video that was used to augment subsequent interviews led by an emergency medicine faculty member. Interviews were audio recorded and then transcribed. An emergent thematic analysis was completed with a codebook that was developed by 4 research assistants, with subsequent analyses conducted by the lead research assistant with input from emergency medicine faculty. Themes from high- and low-performing residents were subsequently qualitatively compared.

RESULTS

Higher-performing residents were better able to anticipate, selectively attend to relevant information, and manage cognitive demands, and took a concurrent (as opposed to linear) approach to managing the simulated patient.

CONCLUSION

The results provide new insights into residents' cognitive processes while managing simulated patients in an examination environment and how these processes vary with performance. More work is needed to determine how best to apply these findings to improve crisis resource management education.

Simple eye movement metrics can predict future decision making performance: The case of financial choices

Decisions are often delegated to experts chosen based on their past performance record which may be subject to noise. For instance, a person with little skill could still make a lucky decision that proves correct ex-post, while a skilled expert could make the best possible use of available information to reach a decision that, with hindsight, turns out incorrect. We aimed to show that one could assess decision skills more accurately when analyzing not only the observed decisions, but also the decision-making process. Incorporating eye-tracking into an established behavioral finance experimental framework, we found that making an eye transition between pieces of information that previous research associated with bias makes one less likely to make good financial decisions in future trials. Thus, even the simplest, easy to obtain eye metrics could allow us to more accurately judge if a person’s performance is a reflection of skill, or down to luck and unlikely to be reproduced in the future.

Chest X-ray evaluation training: impact of normal and abnormal image ratio and instructional sequence

CONTEXT

Medical image perception training generally focuses on abnormalities, whereas normal images are more prevalent in medical practice. Furthermore, instructional sequences that let students practice prior to expert instruction (inductive) may lead to improved performance compared with methods that give students expert instruction before practice (deductive). This study investigates the effects of the proportion of normal images and practice-instruction order on learning to interpret medical images. It is hypothesised that manipulation of the proportion of normal images will lead to a sensitivity-specificity trade-off and that students in practice-first (inductive) conditons need more time per practice case but will correctly identify more test cases.

METHODS

Third-year medical students (n = 103) learned radiograph interpretation by practising cases with, respectively, 30% or 70% normal radiographs prior to expert instruction (practice-first order) or after expert instruction (instruction-first order). After training, students performed a test (60% normal) and sensitivity (% of correctly identified abnormal radiographs), specificity (% of correctly identified normal radiographs), diagnostic performance (% of correct diagnoses) and case duration were measured.

RESULTS

The conditions with 30% of normal images scored higher on sensitivity but the conditions with 70% of normal images scored higher on specificity, indicating a sensitivity and specificity trade-off. Those who participated in inductive conditions took less time per practice case but more per test case. They had similar test sensitivity, but scored lower on test specificity.

CONCLUSIONS

The proportion of normal images impacted the sensitivity-specificity trade-off. This trade-off should be an important consideration for the alignment of training with future practice. Furthermore, the deductive conditions unexpectedly scored higher on specificity when participants took less time per case. An inductive approach did not lead to higher diagnostic performance, possibly because participants might already have relevant prior knowledge. Deductive approaches are therefore advised for the training of advanced learners.

Getting the picture: visual interpretation in ophthalmology residency training

OBJECTIVES

Visual interpretation is essential in many fields of health care. Although diagnostic competency can be measured as an educational outcome, few accounts have addressed tacit aspects of visual interpretation in clinical training; these include the disciplining of the trainee's attentions and the trainee's acculturation into expected styles of communicating visual interpretations to others. This paper describes values and dispositions that are taught to ophthalmology trainees as they learn to reason through visual information, and explores how these qualities are evaluated during residency training.

METHODS

The project was based on 6 months of ethnographic participant observation and interviews in an ophthalmology residency programme. Observational notes and interview transcripts pertaining to visual interpretation were isolated for qualitative analysis in the tradition of sociocultural anthropology, guided by literature on communication in medical education and the socialisation of health professionals.

RESULTS

Residents and faculty members identified visual interpretation as one of the most challenging skills expected of ophthalmology trainees. They expressed a belief that 'systematic' approaches, where visual information is parsed in a stepwise fashion, reduce the chance of trainees overlooking or misinterpreting key diagnostic features. This sensory discipline was represented in narrative form when faculty members asked residents to interpret images aloud, as residents were expected to follow prescribed sequences for describing the content of images before commenting on possible diagnoses.

CONCLUSIONS

Sensory processing is ordinarily opaque to outside observers, but the ritual of describing images in highly regimented narratives allows residents to demonstrate how they gather and reason through visual information. The form of these narratives reflects values that residents are expected to embody during their training, such as being thorough and methodical; it may also serve a pedagogical function by entrenching those values. Further research is needed to characterise how the performance of speech genres shapes the interpretive skills of medical trainees.

A new way to look at simulation-based assessment: the relationship between gaze-tracking and exam performance

Objective

A key task of the team leader in a medical emergency is effective information gathering. Studying information gathering patterns is readily accomplished with the use of gaze-tracking glasses. This technology was used to generate hypotheses about the relationship between performance scores and expert-hypothesized visual areas of interest in residents across scenarios in simulated medical resuscitation examinations.

Methods

Emergency medicine residents wore gaze-tracking glasses during two simulation-based examinations (n=29 and 13 respectively). Blinded experts assessed video-recorded performances using a simulation performance assessment tool that has validity evidence in this context. The relationships between gaze patterns and performance scores were analyzed and potential hypotheses generated. Four scenarios were assessed in this study: diabetic ketoacidosis, bradycardia secondary to beta-blocker overdose, ruptured abdominal aortic aneurysm and metabolic acidosis caused by antifreeze ingestion.

Results

Specific gaze patterns were correlated with objective performance. High performers were more likely to fixate on task-relevant stimuli and appropriately ignore task-irrelevant stimuli compared with lower performers. For example, shorter latency to fixation on the vital signs in a case of diabetic ketoacidosis was positively correlated with performance (r=0.70, p<0.05). Conversely, total time spent fixating on lab values in a case of ruptured abdominal aortic aneurysm was negatively correlated with performance (r= −0.50, p<0.05).

Conclusions

There are differences between the visual patterns of high and low-performing residents. These findings may allow for better characterization of expertise development in resuscitation medicine and provide a framework for future study of visual behaviours in resuscitation cases.

Getting Inside the Expert's Head: An Analysis of Physician Cognitive Processes During Trauma Resuscitations

STUDY OBJECTIVE

Crisis resource management skills are integral to leading the resuscitation of a critically ill patient. Despite their importance, crisis resource management skills (and their associated cognitive processes) have traditionally been difficult to study in the real world. The objective of this study was to derive key cognitive processes underpinning expert performance in resuscitation medicine, using a new eye-tracking-based video capture method during clinical cases.

METHODS

During an 18-month period, a sample of 10 trauma resuscitations led by 4 expert trauma team leaders was analyzed. The physician team leaders were outfitted with mobile eye-tracking glasses for each case. After each resuscitation, participants were debriefed with a modified cognitive task analysis, based on a cued-recall protocol, augmented by viewing their own first-person perspective eye-tracking video from the clinical encounter.

RESULTS

Eye-tracking technology was successfully applied as a tool to aid in the qualitative analysis of expert performance in a clinical setting. All participants stated that using these methods helped uncover previously unconscious aspects of their cognition. Overall, 5 major themes were derived from the interviews: logistic awareness, managing uncertainty, visual fixation behaviors, selective attendance to information, and anticipatory behaviors.

CONCLUSION

The novel approach of cognitive task analysis augmented by eye tracking allowed the derivation of 5 unique cognitive processes underpinning expert performance in leading a resuscitation. An understanding of these cognitive processes has the potential to enhance educational methods and to create new assessment modalities of these previously tacit aspects of expertise in this field.

Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool

In general, researchers attempt to quantify cognitive load using physiologic and psychometric measures. Although the construct measured by both of these metrics is thought to represent overall cognitive load, there is a paucity of studies that compares these techniques to one another. The authors compared data obtained from one physiologic tool (pupillometry) to one psychometric tool (Paas scale) to explore whether they actually measured the construct of cognitive load as purported. Thirty-two participants with a range of resuscitation medicine experience and expertise completed resuscitation-medicine based multiple-choice-questions as well as arithmetic questions. Cognitive load, as measured by both tools, was found to be higher for the more difficult questions as well as for questions that were answered incorrectly (p < 0.001). The group with the least medical experience had higher cognitive load than both the intermediate and experienced groups when answering domain-specific questions (p = 0.023 and p = 0.003 respectively for the physiologic tool; p = 0.006 and p < 0.001 respectively for the psychometric tool). There was a strong positive correlation (Spearman's ρ = 0.827, p < 0.001 for arithmetic questions; Spearman's ρ = 0.606, p < 0.001 for medical questions) between the two cognitive load measurement tools. These findings support the validity argument that both physiologic and psychometric metrics measure the construct of cognitive load.

The Holistic Processing Account of Visual Expertise in Medical Image Perception: A Review

In the field of medical image perception, the holistic processing perspective contends that experts can rapidly extract global information about the image, which can be used to guide their subsequent search of the image (Swensson, 1980; Nodine and Kundel, 1987; Kundel et al., 2007). In this review, we discuss the empirical evidence supporting three different predictions that can be derived from the holistic processing perspective: Expertise in medical image perception is domain-specific, experts use parafoveal and/or peripheral vision to process large regions of the image in parallel, and experts benefit from a rapid initial glimpse of an image. In addition, we discuss a pivotal recent study (Litchfield and Donovan, 2016) that seems to contradict the assumption that experts benefit from a rapid initial glimpse of the image. To reconcile this finding with the existing literature, we suggest that global processing may serve multiple functions that extend beyond the initial glimpse of the image. Finally, we discuss future research directions, and we highlight the connections between the holistic processing account and similar theoretical perspectives and findings from other domains of visual expertise.