1
|
Castner N, Arsiwala-Scheppach L, Mertens S, Krois J, Thaqi E, Kasneci E, Wahl S, Schwendicke F. Expert gaze as a usability indicator of medical AI decision support systems: a preliminary study. NPJ Digit Med 2024; 7:199. [PMID: 39068241 PMCID: PMC11283514 DOI: 10.1038/s41746-024-01192-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2023] [Accepted: 07/12/2024] [Indexed: 07/30/2024] Open
Abstract
Given the current state of medical artificial intelligence (AI) and perceptions towards it, collaborative systems are becoming the preferred choice for clinical workflows. This work aims to address expert interaction with medical AI support systems to gain insight towards how these systems can be better designed with the user in mind. As eye tracking metrics have been shown to be robust indicators of usability, we employ them for evaluating the usability and user interaction with medical AI support systems. We use expert gaze to assess experts' interaction with an AI software for caries detection in bitewing x-ray images. We compared standard viewing of bitewing images without AI support versus viewing where AI support could be freely toggled on and off. We found that experts turned the AI on for roughly 25% of the total inspection task, and generally turned it on halfway through the course of the inspection. Gaze behavior showed that when supported by AI, more attention was dedicated to user interface elements related to the AI support, with more frequent transitions from the image itself to these elements. When considering that expert visual strategy is already optimized for fast and effective image inspection, such interruptions in attention can lead to increased time needed for the overall assessment. Gaze analysis provided valuable insights into an AI's usability for medical image inspection. Further analyses of these tools and how to delineate metrical measures of usability should be developed.
Collapse
Affiliation(s)
- Nora Castner
- Carl Zeiss Vision International GmbH, Tübingen, Germany.
- University of Tübingen, Tübingen, Germany.
| | | | - Sarah Mertens
- Charité - Univesitätsmedizin, Oral Diagnostics, Digital Health and Services Research, Berlin, Germany
| | - Joachim Krois
- Charité - Univesitätsmedizin, Oral Diagnostics, Digital Health and Services Research, Berlin, Germany
| | - Enkeleda Thaqi
- Technical University of Munich, Human-Centered Technologies for Learning, Munich, Germany
| | - Enkelejda Kasneci
- Technical University of Munich, Human-Centered Technologies for Learning, Munich, Germany
| | - Siegfried Wahl
- Carl Zeiss Vision International GmbH, Tübingen, Germany
- Institute for Ophthalmic Research, University of Tübingen, Tübingen, Germany
| | - Falk Schwendicke
- Ludwig Maximilian University, Operative, Preventative and Pediatric Dentistry and Periodontology, Munich, Germany
| |
Collapse
|
2
|
Kok EM, Niehorster DC, van der Gijp A, Rutgers DR, Auffermann WF, van der Schaaf M, Kester L, van Gog T. The effects of gaze-display feedback on medical students' self-monitoring and learning in radiology. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2024:10.1007/s10459-024-10322-6. [PMID: 38555550 DOI: 10.1007/s10459-024-10322-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Accepted: 03/03/2024] [Indexed: 04/02/2024]
Abstract
Self-monitoring is essential for effectively regulating learning, but difficult in visual diagnostic tasks such as radiograph interpretation. Eye-tracking technology can visualize viewing behavior in gaze displays, thereby providing information about visual search and decision-making. We hypothesized that individually adaptive gaze-display feedback improves posttest performance and self-monitoring of medical students who learn to detect nodules in radiographs. We investigated the effects of: (1) Search displays, showing which part of the image was searched by the participant; and (2) Decision displays, showing which parts of the image received prolonged attention in 78 medical students. After a pretest and instruction, participants practiced identifying nodules in 16 cases under search-display, decision-display, or no feedback conditions (n = 26 per condition). A 10-case posttest, without feedback, was administered to assess learning outcomes. After each case, participants provided self-monitoring and confidence judgments. Afterward, participants reported on self-efficacy, perceived competence, feedback use, and perceived usefulness of the feedback. Bayesian analyses showed no benefits of gaze displays for post-test performance, monitoring accuracy (absolute difference between participants' estimated and their actual test performance), completeness of viewing behavior, self-efficacy, and perceived competence. Participants receiving search-displays reported greater feedback utilization than participants receiving decision-displays, and also found the feedback more useful when the gaze data displayed was precise and accurate. As the completeness of search was not related to posttest performance, search displays might not have been sufficiently informative to improve self-monitoring. Information from decision displays was rarely used to inform self-monitoring. Further research should address if and when gaze displays can support learning.
Collapse
Affiliation(s)
- Ellen M Kok
- Department of Education, Utrecht University, P.O. Box 80140, 3508 CS, Utrecht, The Netherlands.
| | - Diederick C Niehorster
- Lund University Humanities Lab, Lund University, Lund, Sweden
- Department of Psychology, Lund University, Lund, Sweden
| | - Anouk van der Gijp
- Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Dirk R Rutgers
- Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands
| | | | - Marieke van der Schaaf
- Utrecht Center for Research and Development in Health Professions Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Liesbeth Kester
- Department of Education, Utrecht University, P.O. Box 80140, 3508 CS, Utrecht, The Netherlands
| | - Tamara van Gog
- Department of Education, Utrecht University, P.O. Box 80140, 3508 CS, Utrecht, The Netherlands
| |
Collapse
|
3
|
Arsiwala-Scheppach LT, Castner N, Rohrer C, Mertens S, Kasneci E, Cejudo Grano de Oro JE, Krois J, Schwendicke F. Gaze patterns of dentists while evaluating bitewing radiographs. J Dent 2023; 135:104585. [PMID: 37301462 DOI: 10.1016/j.jdent.2023.104585] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 05/15/2023] [Accepted: 06/07/2023] [Indexed: 06/12/2023] Open
Abstract
OBJECTIVES Understanding dentists' gaze patterns on radiographs may allow to unravel sources of their limited accuracy and develop strategies to mitigate them. We conducted an eye tracking experiment to characterize dentists' scanpaths and thus their gaze patterns when assessing bitewing radiographs to detect primary proximal carious lesions. METHODS 22 dentists assessed a median of nine bitewing images each, resulting in 170 datasets after excluding data with poor quality of gaze recording. Fixation was defined as an area of attentional focus related to visual stimuli. We calculated time to first fixation, fixation count, average fixation duration, and fixation frequency. Analyses were performed for the entire image and stratified by (1) presence of carious lesions and/or restorations and (2) lesion depth (E1/2: outer/inner enamel; D1-3: outer-inner third of dentin). We also examined the transitional nature of the dentists' gaze. RESULTS Dentists had more fixations on teeth with lesions and/or restorations (median=138 [interquartile range=87, 204]) than teeth without them (32 [15, 66]), p<0.001. Notably, teeth with lesions had longer fixation durations (407 milliseconds [242, 591]) than those with restorations (289 milliseconds [216, 337]), p<0.001. Time to first fixation was longer for teeth with E1 lesions (17,128 milliseconds [8813, 21,540]) than lesions of other depths (p = 0.049). The highest number of fixations were on teeth with D2 lesions (43 [20, 51]) and lowest on teeth with E1 lesions (5 [1, 37]), p<0.001. Generally, a systematic tooth-by-tooth gaze pattern was observed. CONCLUSIONS As hypothesized, while visually inspecting bitewing radiographic images, dentists employed a heightened focus on certain image features/areas, relevant to the assigned task. Also, they generally examined the entire image in a systematic tooth-by-tooth pattern.
Collapse
Affiliation(s)
- Lubaina T Arsiwala-Scheppach
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany; ITU/WHO Focus Group AI on Health, Topic Group Dental Diagnostics and Digital Dentistry, Switzerland.
| | - Nora Castner
- Department of Computer Science, University of Tuebingen, Tuebingen, Germany
| | - Csaba Rohrer
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany
| | - Sarah Mertens
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany
| | - Enkelejda Kasneci
- Department of Computer Science, Technical University of Munich, Germany
| | - Jose Eduardo Cejudo Grano de Oro
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany
| | - Joachim Krois
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany; ITU/WHO Focus Group AI on Health, Topic Group Dental Diagnostics and Digital Dentistry, Switzerland
| | - Falk Schwendicke
- Department of Oral Diagnostics, Digital Health and Health Services Research, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Germany; ITU/WHO Focus Group AI on Health, Topic Group Dental Diagnostics and Digital Dentistry, Switzerland
| |
Collapse
|
4
|
Budeanu RG, Broemmer C, Budeanu AR, Pop M. Comparing the Diagnostic Performance of ECG Gated versus Non-Gated CT Angiography in Ascending Aortic Dissection: A GRRAS Study. Tomography 2022; 8:2426-2434. [PMID: 36287800 PMCID: PMC9609484 DOI: 10.3390/tomography8050201] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2022] [Revised: 09/17/2022] [Accepted: 09/26/2022] [Indexed: 11/27/2022] Open
Abstract
Rationale and Objective: Thoracic CT angiography (CTA) for ascending aortic dissection, a life-threatening emergency, is performed routinely without Electrocardiographic (ECG) gating, therefore allowing the apparition of a pulsation artefact. We aimed to evaluate and compare the diagnostic performance, the inter and intra-reporter agreement of ECG gated CTA and non-ECG gated CTA for detecting ascending aortic dissection, considering their training level. Our hypothesis is that ECG gated CTA has superior diagnostic accuracy for ascending aortic dissection compared to non-gated CTA. Materials and Methods: We collected data using 24 questions survey using clinically validated CT examinations. Sixty-six respondents (medical students, radiology residents, and consultants) blinded to the actual diagnosis independently evaluated the images pertaining to the presence of ascending aortic dissection. The reference standard was represented by clinical and imaging diagnosis. Inter-rater and inter-group concordance was evaluated; the agreement with reference tests was calculated and assessed as a function of reporters’ training level. Results: Reporters’ ascending aortic dissection assessment showed a better correlation with the reference standard in the ECG gated CTA. The inter-rater correlation was higher in the ECG gated CTA compared to non-ECG gated CTA. Observers’ confidence for diagnosing ascending aortic dissection was higher in the ECG gated CTA. Statistically significant differences (p < 0.05) were found between different training levels when assessing non-ECG gated examinations. Conclusions: ECG gated CTA shows a higher diagnostic performance for ascending aortic dissection than non-ECG gated CTA, regardless of the reporters’ training level.
Collapse
Affiliation(s)
| | - Christian Broemmer
- ME1 Department, “George Emil Palade” University of Medicine, Pharmacy, Science and Technology of Targu Mures, 540142 Targu Mures, Romania
| | - Anamaria R. Budeanu
- Emergency County Hospital Târgu Mureș, 540136 Targu Mures, Romania
- Correspondence:
| | - Marian Pop
- ME1 Department, “George Emil Palade” University of Medicine, Pharmacy, Science and Technology of Targu Mures, 540142 Targu Mures, Romania
- Radiology and Medical Imaging Department, Emergency Institute for Cardiovascular Disease and Heart Transplant of Targu Mures, 540136 Targu Mures, Romania
| |
Collapse
|
5
|
LoGiudice AB, Sherbino J, Norman G, Monteiro S, Sibbald M. Intuitive and deliberative approaches for diagnosing 'well' versus 'unwell': evidence from eye tracking, and potential implications for training. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:811-825. [PMID: 33423154 DOI: 10.1007/s10459-020-10023-w] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/22/2020] [Accepted: 12/21/2020] [Indexed: 06/12/2023]
Abstract
Rapidly assessing how ill a patient is based on their immediate presentation-colloquially termed 'eyeballing' in practice-serves a vital role in acute care settings. Yet surprisingly little is known about how this diagnostic skill is learned or how it should be taught. Some authors have pointed to a dual-process model, suggesting that assessments of illness severity are driven by two distinct types of processing: an intuitive, fast, pattern recognition-like process (Type 1) that depends on many prior patient encounters and outcomes being stored in memory; and a deliberate, slow, analytic process (Type 2) characterized by additional data gathering, data scrutiny, or recollection of rules. But prior studies have supported a dual-process model for the assessment of illness severity only insofar as experienced clinicians chiefly displayed what was presumed to be Type 1 processing. Here we further explored a dual-process model by examining whether less experienced clinicians displayed both types of processing when assessing illness severity across a series of cases. Consistent with the model, a dissociation between Type 1 and Type 2 processing was observed through resident reports of deliberation, response times, and three eye tracking metrics associated with diagnostic expertise. We conclude by discussing potential implications for the training of this enigmatic diagnostic skill.
Collapse
Affiliation(s)
- Andrew B LoGiudice
- MacPherson Institute for Leadership, Innovation, and Excellence in Teaching, McMaster University, 1280 Main Street West, Hamilton, ON, L8S 4L8, Canada.
- McMaster Faculty of Health Sciences Program in Education Research, Innovation and Theory (MERIT), Hamilton, Canada.
| | - Jonathan Sherbino
- Department of Medicine, McMaster University, Hamilton, Canada
- McMaster Faculty of Health Sciences Program in Education Research, Innovation and Theory (MERIT), Hamilton, Canada
| | - Geoffrey Norman
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Canada
- McMaster Faculty of Health Sciences Program in Education Research, Innovation and Theory (MERIT), Hamilton, Canada
| | - Sandra Monteiro
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Canada
- McMaster Faculty of Health Sciences Program in Education Research, Innovation and Theory (MERIT), Hamilton, Canada
| | - Matthew Sibbald
- Department of Medicine, McMaster University, Hamilton, Canada
- McMaster Faculty of Health Sciences Program in Education Research, Innovation and Theory (MERIT), Hamilton, Canada
| |
Collapse
|
6
|
Morra M, Braund H, Hall AK, Szulewski A. Cognitive load and processes during chest radiograph interpretation in the emergency department across the spectrum of expertise. AEM EDUCATION AND TRAINING 2021; 5:e10693. [PMID: 34901684 PMCID: PMC8637309 DOI: 10.1002/aet2.10693] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 09/13/2021] [Accepted: 09/14/2021] [Indexed: 06/14/2023]
Abstract
BACKGROUND In the emergency department (ED), chest radiographs (CXRs) provide essential information for clinical diagnostic reasoning. Errors in interpretation by emergency physicians can lead to negative patient outcomes. To aid in teaching this important skill, an understanding of cognitive processes and cognitive load (CL) in CXR interpretation in emergency medicine (EM) personnel is warranted. METHODS This study adopted a concurrent mixed-methods research design. Participant groups included medical students (M), junior (J) and senior (S) EM residents, and attending emergency physicians (P) in the ED at an academic hospital. To elucidate cognitive processes, a real-time cognitive task analysis during CXR interpretation was performed. Interviews were audio recorded, transcribed verbatim, and analyzed thematically. The interview was followed by a questionnaire, where participants rated their CL, stress, and confidence level. RESULTS Levels of CL (M vs. S and M vs. P, p = 0.002; J vs. S, p = 0.004; J vs. P, p = 0.005) and stress (J vs. P, p = 0.002) decreased, while confidence levels increased (M vs. S, p = 0.006; J vs. S, p ≤ 0.001; J vs. P, p = 0.003) as experience level increased. Qualitative analysis of interviews revealed four themes: checking behavior, information reduction, pattern recognition versus systematic viewing, and recognizing scope of practice. Experts commonly utilized checking behavior (e.g., comparison to prior radiographs) and deprioritized task irrelevant data. Experts used a general overview technique as their initial approach as opposed to a systematic viewing approach, and they more readily recognized an EM physicians' scope of practice in this task. CONCLUSION This study characterized differences in cognition that led to increased CL, stress, and lower level of confidence in EM learners during CXR interpretation and provided insight into expertise development in this important skill.
Collapse
Affiliation(s)
- Michael Morra
- School of MedicineQueen’s UniversityKingstonOntarioCanada
| | - Heather Braund
- Office of Professional Development and Educational ScholarshipFaculty of Health SciencesFaculty of EducationQueen’s UniversityKingstonOntarioCanada
| | - Andrew K. Hall
- Department of Emergency MedicineUniversity of OttawaOttawaOntarioCanada
- Royal College of Physicians and Surgeons of CanadaOttawaOntarioCanada
| | - Adam Szulewski
- Departments of Emergency Medicine and PsychologyKingston Health Sciences CentreQueen’s UniversityKingstonOntarioCanada
| |
Collapse
|
7
|
van Montfort D, Kok E, Vincken K, van der Schaaf M, van der Gijp A, Ravesloot C, Rutgers D. Expertise development in volumetric image interpretation of radiology residents: what do longitudinal scroll data reveal? ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:437-466. [PMID: 33030627 PMCID: PMC8041671 DOI: 10.1007/s10459-020-09995-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/03/2019] [Accepted: 09/28/2020] [Indexed: 06/11/2023]
Abstract
The current study used theories on expertise development (the holistic model of image perception and the information reduction hypothesis) as a starting point to identify and explore potentially relevant process measures to monitor and evaluate expertise development in radiology residency training. It is the first to examine expertise development in volumetric image interpretation (i.e., CT scans) within radiology residents using scroll data collected longitudinally over five years of residency training. Consistent with the holistic model of image perception, the percentage of time spent on full runs, i.e. scrolling through more than 50% of the CT-scan slices (global search), decreased within residents over residency training years. Furthermore, the percentage of time spent on question-relevant areas in the CT scans increased within residents over residency training years, consistent with the information reduction hypothesis. Second, we examined if scroll patterns can predict diagnostic accuracy. The percentage of time spent on full runs and the percentage of time spent on question-relevant areas did not predict diagnostic accuracy. Thus, although scroll patterns over training years are consistent with visual expertise theories, they could not be used as predictors of diagnostic accuracy in the current study. Therefore, the relation between scroll patterns and performance needs to be further examined, before process measures can be used to monitor and evaluate expertise development in radiology residency training.
Collapse
Affiliation(s)
- Dorien van Montfort
- Department of Education, Utrecht University, Heidelberglaan 1, 3584CS, Utrecht, The Netherlands
| | - Ellen Kok
- Department of Education, Utrecht University, Heidelberglaan 1, 3584CS, Utrecht, The Netherlands.
| | - Koen Vincken
- Image Sciences Institute, Imaging Dept, University Medical Center, Utrecht, The Netherlands
| | - Marieke van der Schaaf
- Department of Education, Utrecht University, Heidelberglaan 1, 3584CS, Utrecht, The Netherlands
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Anouk van der Gijp
- Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Cécile Ravesloot
- Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Dirk Rutgers
- Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
8
|
Monteiro S, Logiudice A, Sibbald M. The unexplored value of “Normal”: A commentary on the lack of normal cases in high-stakes assessment. ARCHIVES OF MEDICINE AND HEALTH SCIENCES 2021. [DOI: 10.4103/amhs.amhs_106_21] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
|
9
|
Richter J, Scheiter K, Eder TF, Huettig F, Keutel C. How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study. PLoS One 2020; 15:e0243060. [PMID: 33270704 PMCID: PMC7714201 DOI: 10.1371/journal.pone.0243060] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2020] [Accepted: 11/15/2020] [Indexed: 02/06/2023] Open
Abstract
The interpretation of medical images is an error-prone process that may yield severe consequences for patients. In dental medicine panoramic radiography (OPT) is a frequently used diagnostic procedure. OPTs typically contain multiple, diverse anomalies within one image making the diagnostic process very demanding, rendering students’ development of visual expertise a complex task. Radiograph interpretation is typically taught through massed practice; however, it is not known how effective this approach is nor how it changes students’ visual inspection of radiographs. Therefore, this study investigated how massed practice–an instructional method that entails massed learning of one type of material–affects processing of OPTs and the development of diagnostic performance. From 2017 to 2018, 47 dental students in their first clinical semester diagnosed 10 OPTs before and after their regular massed practice training, which is embedded in their curriculum. The OPTs contained between 3 to 26 to-be-identified anomalies. During massed practice they diagnosed 100 dental radiographs without receiving corrective feedback. The authors recorded students’ eye movements and assessed the number of correctly identified and falsely marked low- and high prevalence anomalies before and after massed practice. Massed practice had a positive effect on detecting anomalies especially with low prevalence (p < .001). After massed practice students covered a larger proportion of the OPTs (p < .001), which was positively related to the detection of low-prevalence anomalies (p = .04). Students also focused longer, more frequently, and earlier on low-prevalence anomalies after massed practice (ps < .001). While massed practice improved visual expertise in dental students with limited prior knowledge, there is still substantial room for improvement. The results suggest integrating massed practice with more deliberate practice, where, for example, corrective feedback is provided, and support is adapted to students’ needs.
Collapse
Affiliation(s)
- Juliane Richter
- Leibniz-Institut für Wissensmedien, Tübingen, Germany
- * E-mail:
| | - Katharina Scheiter
- Leibniz-Institut für Wissensmedien, Tübingen, Germany
- Eberhard Karls University of Tübingen, Tübingen, Germany
| | | | - Fabian Huettig
- Department of Prosthodontics, University Hospital for Dentistry, Oral Medicine, and Maxillofacial Surgery at the University Hospital Tübingen, Eberhard Karls University of Tübingen, Tübingen, Germany
| | - Constanze Keutel
- Radiology Department of the University Hospital for Dentistry, Oral Medicine, and Maxillofacial Surgery at the University Hospital Tübingen, Eberhard Karls University of Tübingen, Tübingen, Germany
| |
Collapse
|
10
|
Focal lung pathology detection in radiology: Is there an effect of experience on visual search behavior? Atten Percept Psychophys 2020; 82:2837-2850. [PMID: 32367272 DOI: 10.3758/s13414-020-02033-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
Abstract
In radiology, 60% to 80% of diagnostic errors are perceptual. The use of more efficient visual search behaviors is expected to reduce these errors. We collected eye-tracking data from participants with different levels of experience when interpreting chest X-rays during the completion of a pathology-detection task. Eye-tracking measures were assessed in the context of three existing visual search theories from the literature to understand the association between visual search behavior and underlying processes: the long-term working memory theory, the information-reduction hypothesis, and the holistic model of image perception. The most experienced participants (radiology residents) showed the highest level of performance, albeit their visual search behaviors did not differ from the intermediate group. This suggests that radiology residents better processed the represented information on the X-ray, using a visual search strategy similar to the intermediate group. Since similar visual search resulted in more information extraction in the radiology residents compared with the intermediates, we suggest that this result might support the long-term working memory theory. Furthermore, compared with novices, intermediates and radiology residents fixated longer on areas that were more important to avoid missing any pathology, which possibly confirms the information-reduction hypothesis. Finally, the larger distances between fixations observed in more experienced participants could support the holistic model of image perception. In addition, measures of generic skills were related to a lower time cost for switching between global and local information processing. Our findings suggest that the three theories may be complementary in chest X-ray interpretation. Therefore, a unified theory explaining perceptual-cognitive superiority in radiology is considered.
Collapse
|
11
|
Castner N, Appel T, Eder T, Richter J, Scheiter K, Keutel C, Hüttig F, Duchowski A, Kasneci E. Pupil diameter differentiates expertise in dental radiography visual search. PLoS One 2020; 15:e0223941. [PMID: 32469952 PMCID: PMC7259659 DOI: 10.1371/journal.pone.0223941] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2019] [Accepted: 05/13/2020] [Indexed: 01/22/2023] Open
Abstract
Expert behavior is characterized by rapid information processing abilities, dependent on more structured schemata in long-term memory designated for their domain-specific tasks. From this understanding, expertise can effectively reduce cognitive load on a domain-specific task. However, certain tasks could still evoke different gradations of load even for an expert, e.g., when having to detect subtle anomalies in dental radiographs. Our aim was to measure pupil diameter response to anomalies of varying levels of difficulty in expert and student dentists’ visual examination of panoramic radiographs. We found that students’ pupil diameter dilated significantly from baseline compared to experts, but anomaly difficulty had no effect on pupillary response. In contrast, experts’ pupil diameter responded to varying levels of anomaly difficulty, where more difficult anomalies evoked greater pupil dilation from baseline. Experts thus showed proportional pupillary response indicative of increasing cognitive load with increasingly difficult anomalies, whereas students showed pupillary response indicative of higher cognitive load for all anomalies when compared to experts.
Collapse
Affiliation(s)
- Nora Castner
- Human-Computer Interaction, Institute of Computer Science, University Tübingen, Tübingen, Germany
- * E-mail:
| | - Tobias Appel
- Human-Computer Interaction, Institute of Computer Science, University Tübingen, Tübingen, Germany
| | - Thérése Eder
- Multiple Representations Lab, Leibniz-Institut für Wissensmedien, Tübingen, Germany
| | - Juliane Richter
- Multiple Representations Lab, Leibniz-Institut für Wissensmedien, Tübingen, Germany
| | - Katharina Scheiter
- Multiple Representations Lab, Leibniz-Institut für Wissensmedien, Tübingen, Germany
- University Tübingen, Tübingen, Germany
| | - Constanze Keutel
- Department of Oral- and Maxillofacial Radiology, University Clinic for Dentistry, Oral Medicine, and Maxillofacial Surgery, University of Tübingen, Tübingen, Germany
| | - Fabian Hüttig
- Department of Prosthodontics, University Clinic for Dentistry, Oral Medicine, and Maxillofacial Surgery, University of Tübingen, Tübingen, Germany
| | - Andrew Duchowski
- Visual Computing, Clemson University, Clemson, South Carolina, United States of America
| | - Enkelejda Kasneci
- Human-Computer Interaction, Institute of Computer Science, University Tübingen, Tübingen, Germany
| |
Collapse
|
12
|
[Quality of findings, interpretation, and self-assessment of readers of chest x-rays acquired in one plane depending on the use of an anatomical viewing scheme and reader expertise]. Radiologe 2020; 60:850-862. [PMID: 32248250 DOI: 10.1007/s00117-020-00673-7] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
PURPOSE To objectify effects of an anatomical viewing scheme on the respective correctness of (a) findings, (b) interpretations, and (c) self-assessments of readers in chest radiographs acquired in one plane and the assessment of other influencing factors. MATERIALS AND METHODS In all, 20 radiologists with 3-60 months of full-time radiography experience evaluated 12 chest radiographs of varying difficulty: once with and once without using an anatomical viewing scheme with at least 1 month in between (n = 480). In consensus of 3 radiological experts (a) and (b) were determined by means of a current computed tomography. The self-assessment (c) of readers was queried. RESULTS (a) Findings were either missed or not described in 21%. Another 20% were recognized, but incorrectly described, (b) 62% of interpretations and 31% of derived clinical consequences were wrong and (c) in 39% of items the readers overestimated themselves. Experts were faster and better than novices, but for the scheme usage no further significant differences were detected (p > 0.5, respectively). The most pronounced effect was found in comparison with the routine report produced by the joint evaluation of novices and experts being clearly superior even to the expert study results (a), (b) and (c) alone (p < 0.001, respectively). CONCLUSION Reporting of chest X‑rays acquired in one plane was often incomplete or even wrong, and the evaluators overestimated themselves, which was not influenced by the use of the anatomical viewing scheme. Since errors between the evaluators sometimes differed greatly, duplicate evaluation of the radiographs by two different radiologists, which is already the case in many training facilities, may possibly be advisable in general.
Collapse
|
13
|
Waite S, Farooq Z, Grigorian A, Sistrom C, Kolla S, Mancuso A, Martinez-Conde S, Alexander RG, Kantor A, Macknik SL. A Review of Perceptual Expertise in Radiology-How it develops, How we can test it, and Why humans still matter in the era of Artificial Intelligence. Acad Radiol 2020; 27:26-38. [PMID: 31818384 DOI: 10.1016/j.acra.2019.08.018] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2019] [Revised: 08/26/2019] [Accepted: 08/27/2019] [Indexed: 10/25/2022]
Abstract
As the first step in image interpretation is detection, an error in perception can prematurely end the diagnostic process leading to missed diagnoses. Because perceptual errors of this sort-"failure to detect"-are the most common interpretive error (and cause of litigation) in radiology, understanding the nature of perceptual expertise is essential in decreasing radiology's long-standing error rates. In this article, we review what constitutes a perceptual error, the existing models of radiologic image perception, the development of perceptual expertise and how it can be tested, perceptual learning methods in training radiologists, and why understanding perceptual expertise is still relevant in the era of artificial intelligence. Adding targeted interventions, such as perceptual learning, to existing teaching practices, has the potential to enhance expertise and reduce medical error.
Collapse
|
14
|
Koury HF, Leonard CJ, Carry PM, Lee LMJ. An Expert Derived Feedforward Histology Module Improves Pattern Recognition Efficiency in Novice Students. ANATOMICAL SCIENCES EDUCATION 2019; 12:645-654. [PMID: 30586223 DOI: 10.1002/ase.1854] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/11/2018] [Revised: 12/18/2018] [Accepted: 12/19/2018] [Indexed: 06/09/2023]
Abstract
Histology is a visually oriented, foundational anatomical sciences subject in professional health curricula that has seen a dramatic reduction in educational contact hours and an increase in content migration to a digital platform. While the digital migration of histology laboratories has transformed histology education, few studies have shown the impact of this change on visual literacy development, a critical competency in histology. The objective of this study was to assess whether providing a video clip of an expert's gaze while completing leukocyte identification tasks would increase the efficiency and performance of novices completing similar identification tasks. In a randomized study, one group of novices (n = 9) was provided with training materials that included expert eye gaze, while the other group (n = 12) was provided training materials with identical content, but without the expert eye gaze. Eye movement parameters including fixation rate and total scan path distance, and performance measures including time-to-task-completion and accuracy, were collected during an identification task assessment. Compared to the control group, the average fixation duration was 13.2% higher (P < 0.02) and scan path distance was 35.0% shorter in the experimental group (P = 0.14). Analysis of task performance measures revealed no significant difference between the groups. These preliminary results suggest a more efficient search performed by the experimental group, indicating the potential efficacy of training using an expert's gaze to enhance visual literacy development. With further investigation, such feedforward enhanced training methods could be utilized for histology and other visually oriented subjects.
Collapse
Affiliation(s)
- Hannah F Koury
- Master of Science in Modern Human Anatomy Program, University of Colorado, Graduate School, Aurora, Colorado
| | - Carly J Leonard
- Department of Psychology, University of Colorado Denver, Denver, Colorado
| | - Patrick M Carry
- Musculoskeletal Research Center, Department of Orthopedics, Colorado Children's Hospital, Aurora, Colorado
| | - Lisa M J Lee
- Master of Science in Modern Human Anatomy Program, University of Colorado, Graduate School, Aurora, Colorado
- Department of Cell and Developmental Biology, University of Colorado School of Medicine, Aurora, Colorado
| |
Collapse
|
15
|
Williams LH, Drew T. What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures. Cogn Res Princ Implic 2019; 4:21. [PMID: 31286283 PMCID: PMC6614227 DOI: 10.1186/s41235-019-0171-6] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2019] [Accepted: 05/19/2019] [Indexed: 11/26/2022] Open
Abstract
Interpretation of volumetric medical images represents a rapidly growing proportion of the workload in radiology. However, relatively little is known about the strategies that best guide search behavior when looking for abnormalities in volumetric images. Although there is extensive literature on two-dimensional medical image perception, it is an open question whether the conclusions drawn from these images can be generalized to volumetric images. Importantly, volumetric images have distinct characteristics (e.g., scrolling through depth, smooth-pursuit eye-movements, motion onset cues, etc.) that should be considered in future research. In this manuscript, we will review the literature on medical image perception and discuss relevant findings from basic science that can be used to generate predictions about expertise in volumetric image interpretation. By better understanding search through volumetric images, we may be able to identify common sources of error, characterize the optimal strategies for searching through depth, or develop new training and assessment techniques for radiology residents.
Collapse
|
16
|
Waite S, Grigorian A, Alexander RG, Macknik SL, Carrasco M, Heeger DJ, Martinez-Conde S. Analysis of Perceptual Expertise in Radiology - Current Knowledge and a New Perspective. Front Hum Neurosci 2019; 13:213. [PMID: 31293407 PMCID: PMC6603246 DOI: 10.3389/fnhum.2019.00213] [Citation(s) in RCA: 49] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2019] [Accepted: 06/07/2019] [Indexed: 12/14/2022] Open
Abstract
Radiologists rely principally on visual inspection to detect, describe, and classify findings in medical images. As most interpretive errors in radiology are perceptual in nature, understanding the path to radiologic expertise during image analysis is essential to educate future generations of radiologists. We review the perceptual tasks and challenges in radiologic diagnosis, discuss models of radiologic image perception, consider the application of perceptual learning methods in medical training, and suggest a new approach to understanding perceptional expertise. Specific principled enhancements to educational practices in radiology promise to deepen perceptual expertise among radiologists with the goal of improving training and reducing medical error.
Collapse
Affiliation(s)
- Stephen Waite
- Department of Radiology, SUNY Downstate Medical Center, Brooklyn, NY, United States
| | - Arkadij Grigorian
- Department of Radiology, SUNY Downstate Medical Center, Brooklyn, NY, United States
| | - Robert G. Alexander
- Department of Ophthalmology, SUNY Downstate Medical Center, Brooklyn, NY, United States
- Department of Neurology, SUNY Downstate Medical Center, Brooklyn, NY, United States
- Department of Physiology/Pharmacology, SUNY Downstate Medical Center, Brooklyn, NY, United States
| | - Stephen L. Macknik
- Department of Ophthalmology, SUNY Downstate Medical Center, Brooklyn, NY, United States
- Department of Neurology, SUNY Downstate Medical Center, Brooklyn, NY, United States
- Department of Physiology/Pharmacology, SUNY Downstate Medical Center, Brooklyn, NY, United States
| | - Marisa Carrasco
- Department of Psychology and Center for Neural Science, New York University, New York, NY, United States
| | - David J. Heeger
- Department of Psychology and Center for Neural Science, New York University, New York, NY, United States
| | - Susana Martinez-Conde
- Department of Ophthalmology, SUNY Downstate Medical Center, Brooklyn, NY, United States
- Department of Neurology, SUNY Downstate Medical Center, Brooklyn, NY, United States
- Department of Physiology/Pharmacology, SUNY Downstate Medical Center, Brooklyn, NY, United States
| |
Collapse
|
17
|
Kok EM. Eye tracking: the silver bullet of competency assessment in medical image interpretation? PERSPECTIVES ON MEDICAL EDUCATION 2019; 8:63-64. [PMID: 30949975 PMCID: PMC6468029 DOI: 10.1007/s40037-019-0506-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Affiliation(s)
- Ellen M Kok
- Department of Education, Faculty of Social Sciences, Utrecht University, Utrecht, The Netherlands.
| |
Collapse
|
18
|
den Boer L, van der Schaaf MF, Vincken KL, Mol CP, Stuijfzand BG, van der Gijp A. Volumetric image interpretation in radiology: scroll behavior and cognitive processes. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2018; 23:783-802. [PMID: 29767400 PMCID: PMC6132416 DOI: 10.1007/s10459-018-9828-z] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2017] [Accepted: 05/07/2018] [Indexed: 05/12/2023]
Abstract
The interpretation of medical images is a primary task for radiologists. Besides two-dimensional (2D) images, current imaging technologies allow for volumetric display of medical images. Whereas current radiology practice increasingly uses volumetric images, the majority of studies on medical image interpretation is conducted on 2D images. The current study aimed to gain deeper insight into the volumetric image interpretation process by examining this process in twenty radiology trainees who all completed four volumetric image cases. Two types of data were obtained concerning scroll behaviors and think-aloud data. Types of scroll behavior concerned oscillations, half runs, full runs, image manipulations, and interruptions. Think-aloud data were coded by a framework of knowledge and skills in radiology including three cognitive processes: perception, analysis, and synthesis. Relating scroll behavior to cognitive processes showed that oscillations and half runs coincided more often with analysis and synthesis than full runs, whereas full runs coincided more often with perception than oscillations and half runs. Interruptions were characterized by synthesis and image manipulations by perception. In addition, we investigated relations between cognitive processes and found an overall bottom-up way of reasoning with dynamic interactions between cognitive processes, especially between perception and analysis. In sum, our results highlight the dynamic interactions between these processes and the grounding of cognitive processes in scroll behavior. It suggests, that the types of scroll behavior are relevant to describe how radiologists interact with and manipulate volumetric images.
Collapse
Affiliation(s)
- Larissa den Boer
- Utrecht University, Heidelberglaan 1, 3584 CS, Utrecht, The Netherlands.
| | | | - Koen L Vincken
- University Medical Center Utrecht, Utrecht, The Netherlands
| | - Chris P Mol
- University Medical Center Utrecht, Utrecht, The Netherlands
| | | | | |
Collapse
|
19
|
A new way to look at simulation-based assessment: the relationship between gaze-tracking and exam performance. CAN J EMERG MED 2018; 21:129-137. [DOI: 10.1017/cem.2018.391] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
AbstractObjectiveA key task of the team leader in a medical emergency is effective information gathering. Studying information gathering patterns is readily accomplished with the use of gaze-tracking glasses. This technology was used to generate hypotheses about the relationship between performance scores and expert-hypothesized visual areas of interest in residents across scenarios in simulated medical resuscitation examinations.MethodsEmergency medicine residents wore gaze-tracking glasses during two simulation-based examinations (n=29 and 13 respectively). Blinded experts assessed video-recorded performances using a simulation performance assessment tool that has validity evidence in this context. The relationships between gaze patterns and performance scores were analyzed and potential hypotheses generated. Four scenarios were assessed in this study: diabetic ketoacidosis, bradycardia secondary to beta-blocker overdose, ruptured abdominal aortic aneurysm and metabolic acidosis caused by antifreeze ingestion.ResultsSpecific gaze patterns were correlated with objective performance. High performers were more likely to fixate on task-relevant stimuli and appropriately ignore task-irrelevant stimuli compared with lower performers. For example, shorter latency to fixation on the vital signs in a case of diabetic ketoacidosis was positively correlated with performance (r=0.70, p<0.05). Conversely, total time spent fixating on lab values in a case of ruptured abdominal aortic aneurysm was negatively correlated with performance (r= −0.50, p<0.05).ConclusionsThere are differences between the visual patterns of high and low-performing residents. These findings may allow for better characterization of expertise development in resuscitation medicine and provide a framework for future study of visual behaviours in resuscitation cases.
Collapse
|
20
|
Szulewski A, Gegenfurtner A, Howes DW, Sivilotti MLA, van Merriënboer JJG. Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2017; 22:951-968. [PMID: 27787677 DOI: 10.1007/s10459-016-9725-2] [Citation(s) in RCA: 32] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/25/2016] [Accepted: 10/19/2016] [Indexed: 05/14/2023]
Abstract
In general, researchers attempt to quantify cognitive load using physiologic and psychometric measures. Although the construct measured by both of these metrics is thought to represent overall cognitive load, there is a paucity of studies that compares these techniques to one another. The authors compared data obtained from one physiologic tool (pupillometry) to one psychometric tool (Paas scale) to explore whether they actually measured the construct of cognitive load as purported. Thirty-two participants with a range of resuscitation medicine experience and expertise completed resuscitation-medicine based multiple-choice-questions as well as arithmetic questions. Cognitive load, as measured by both tools, was found to be higher for the more difficult questions as well as for questions that were answered incorrectly (p < 0.001). The group with the least medical experience had higher cognitive load than both the intermediate and experienced groups when answering domain-specific questions (p = 0.023 and p = 0.003 respectively for the physiologic tool; p = 0.006 and p < 0.001 respectively for the psychometric tool). There was a strong positive correlation (Spearman's ρ = 0.827, p < 0.001 for arithmetic questions; Spearman's ρ = 0.606, p < 0.001 for medical questions) between the two cognitive load measurement tools. These findings support the validity argument that both physiologic and psychometric metrics measure the construct of cognitive load.
Collapse
Affiliation(s)
- Adam Szulewski
- Department of Emergency Medicine, Queen's University, Kingston General Hospital, 76 Stuart Street, Kingston, ON, K7L 2V7, Canada
| | - Andreas Gegenfurtner
- Institut für Qualität und Weiterbildung, Technische Hochschule Deggendorf, Edlmairstraße 9, 94469, Deggendorf, Germany
| | - Daniel W Howes
- Departments of Emergency Medicine and Critical Care, Queen's University, Kingston General Hospital, 76 Stuart Street, Kingston, ON, K7L 2V7, Canada.
| | - Marco L A Sivilotti
- Departments of Emergency Medicine and Biomedical and Molecular Sciences, Kingston General Hospital, 76 Stuart Street, Kingston, ON, K7L 2V7, Canada
| | - Jeroen J G van Merriënboer
- Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, Room N.5.06, 6200 MD, Maastricht, The Netherlands
| |
Collapse
|
21
|
Sheridan H, Reingold EM. The Holistic Processing Account of Visual Expertise in Medical Image Perception: A Review. Front Psychol 2017; 8:1620. [PMID: 29033865 PMCID: PMC5627012 DOI: 10.3389/fpsyg.2017.01620] [Citation(s) in RCA: 48] [Impact Index Per Article: 6.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2017] [Accepted: 09/04/2017] [Indexed: 12/11/2022] Open
Abstract
In the field of medical image perception, the holistic processing perspective contends that experts can rapidly extract global information about the image, which can be used to guide their subsequent search of the image (Swensson, 1980; Nodine and Kundel, 1987; Kundel et al., 2007). In this review, we discuss the empirical evidence supporting three different predictions that can be derived from the holistic processing perspective: Expertise in medical image perception is domain-specific, experts use parafoveal and/or peripheral vision to process large regions of the image in parallel, and experts benefit from a rapid initial glimpse of an image. In addition, we discuss a pivotal recent study (Litchfield and Donovan, 2016) that seems to contradict the assumption that experts benefit from a rapid initial glimpse of the image. To reconcile this finding with the existing literature, we suggest that global processing may serve multiple functions that extend beyond the initial glimpse of the image. Finally, we discuss future research directions, and we highlight the connections between the holistic processing account and similar theoretical perspectives and findings from other domains of visual expertise.
Collapse
Affiliation(s)
- Heather Sheridan
- Department of Psychology, University at Albany, State University of New York, Albany, NY, United States
| | - Eyal M. Reingold
- Department of Psychology, University of Toronto, Mississauga, ON, Canada
| |
Collapse
|
22
|
van der Gijp A, Ravesloot CJ, Jarodzka H, van der Schaaf MF, van der Schaaf IC, van Schaik JPJ, Ten Cate TJ. How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2017; 22:765-787. [PMID: 27436353 PMCID: PMC5498587 DOI: 10.1007/s10459-016-9698-1] [Citation(s) in RCA: 85] [Impact Index Per Article: 12.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/17/2015] [Accepted: 07/09/2016] [Indexed: 05/26/2023]
Abstract
Eye tracking research has been conducted for decades to gain understanding of visual diagnosis such as in radiology. For educational purposes, it is important to identify visual search patterns that are related to high perceptual performance and to identify effective teaching strategies. This review of eye-tracking literature in the radiology domain aims to identify visual search patterns associated with high perceptual performance. Databases PubMed, EMBASE, ERIC, PsycINFO, Scopus and Web of Science were searched using 'visual perception' OR 'eye tracking' AND 'radiology' and synonyms. Two authors independently screened search results and included eye tracking studies concerning visual skills in radiology published between January 1, 1994 and July 31, 2015. Two authors independently assessed study quality with the Medical Education Research Study Quality Instrument, and extracted study data with respect to design, participant and task characteristics, and variables. A thematic analysis was conducted to extract and arrange study results, and a textual narrative synthesis was applied for data integration and interpretation. The search resulted in 22 relevant full-text articles. Thematic analysis resulted in six themes that informed the relation between visual search and level of expertise: (1) time on task, (2) eye movement characteristics of experts, (3) differences in visual attention, (4) visual search patterns, (5) search patterns in cross sectional stack imaging, and (6) teaching visual search strategies. Expert search was found to be characterized by a global-focal search pattern, which represents an initial global impression, followed by a detailed, focal search-to-find mode. Specific task-related search patterns, like drilling through CT scans and systematic search in chest X-rays, were found to be related to high expert levels. One study investigated teaching of visual search strategies, and did not find a significant effect on perceptual performance. Eye tracking literature in radiology indicates several search patterns are related to high levels of expertise, but teaching novices to search as an expert may not be effective. Experimental research is needed to find out which search strategies can improve image perception in learners.
Collapse
Affiliation(s)
- A van der Gijp
- Radiology Department, University Medical Center Utrecht, E01.132, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands.
| | - C J Ravesloot
- Radiology Department, University Medical Center Utrecht, E01.132, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - H Jarodzka
- Center for Learning Science and Technologies, Open University of the Netherlands, Heerlen, The Netherlands
| | | | - I C van der Schaaf
- Radiology Department, University Medical Center Utrecht, E01.132, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - J P J van Schaik
- Radiology Department, University Medical Center Utrecht, E01.132, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands
| | - Th J Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
23
|
van der Gijp A, Vincken KL, Boscardin C, Webb EM, Ten Cate OTJ, Naeger DM. The Effect of Teaching Search Strategies on Perceptual Performance. Acad Radiol 2017; 24:762-767. [PMID: 28242103 DOI: 10.1016/j.acra.2017.01.007] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2016] [Revised: 01/16/2017] [Accepted: 01/17/2017] [Indexed: 11/29/2022]
Abstract
RATIONALE AND OBJECTIVES Radiology expertise is dependent on the use of efficient search strategies. The aim of this study is to investigate the effect of teaching search strategies on trainee's accuracy in detecting lung nodules at computed tomography. MATERIALS AND METHODS Two search strategies, "scanning" and "drilling," were tested with a randomized crossover design. Nineteen junior radiology residents were randomized into two groups. Both groups first completed a baseline lung nodule detection test allowing a free search strategy, followed by a test after scanning instruction and drilling instruction or vice versa. True positive (TP) and false positive (FP) scores and scroll behavior were registered. A mixed-design analysis of variance was applied to compare the three search conditions. RESULTS Search strategy instruction had a significant effect on scroll behavior, F(1.3) = 54.2, P < 0.001; TP score, F(2) = 16.1, P < 0.001; and FP score, F(1.3) = 15.3, P < 0.001. Scanning instruction resulted in significantly lower TP scores than drilling instruction (M = 10.7, SD = 5.0 versus M = 16.3, SD = 5.3), t(18) = 4.78, P < 0.001; or free search (M = 15.3, SD = 4.6), t(18) = 4.44, P < 0.001. TP scores for drilling did not significantly differ from free search. FP scores for drilling (M = 7.3, SD = 5.6) were significantly lower than for free search (M = 12.5, SD = 7.8), t(18) = 4.86, P < 0.001. CONCLUSIONS Teaching a drilling strategy is preferable to teaching a scanning strategy for finding lung nodules.
Collapse
Affiliation(s)
- Anouk van der Gijp
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, 505 Parnassus Ave., M-391, San Francisco, CA 94143-0628
| | - Koen L Vincken
- Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Christy Boscardin
- Office of Medical Education, University of California, San Francisco, San Francisco, California
| | - Emily M Webb
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, 505 Parnassus Ave., M-391, San Francisco, CA 94143-0628
| | - Olle Th J Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - David M Naeger
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, 505 Parnassus Ave., M-391, San Francisco, CA 94143-0628.
| |
Collapse
|
24
|
Christensen PA, Lee NE, Thrall MJ, Powell SZ, Chevez-Barrios P, Long SW. RecutClub.com: An Open Source, Whole Slide Image-based Pathology Education System. J Pathol Inform 2017; 8:10. [PMID: 28382224 PMCID: PMC5364738 DOI: 10.4103/jpi.jpi_72_16] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2016] [Accepted: 01/18/2017] [Indexed: 01/02/2023] Open
Abstract
BACKGROUND Our institution's pathology unknown conferences provide educational cases for our residents. However, the cases have not been previously available digitally, have not been collated for postconference review, and were not accessible to a wider audience. Our objective was to create an inexpensive whole slide image (WSI) education suite to address these limitations and improve the education of pathology trainees. MATERIALS AND METHODS We surveyed residents regarding their preference between four unique WSI systems. We then scanned weekly unknown conference cases and study set cases and uploaded them to our custom built WSI viewer located at RecutClub.com. We measured site utilization and conference participation. RESULTS Residents preferred our OpenLayers WSI implementation to Ventana Virtuoso, Google Maps API, and OpenSlide. Over 16 months, we uploaded 1366 cases from 77 conferences and ten study sets, occupying 793.5 GB of cloud storage. Based on resident evaluations, the interface was easy to use and demonstrated minimal latency. Residents are able to review cases from home and from their mobile devices. Worldwide, 955 unique IP addresses from 52 countries have viewed cases in our site. CONCLUSIONS We implemented a low-cost, publicly available repository of WSI slides for resident education. Our trainees are very satisfied with the freedom to preview either the glass slides or WSI and review the WSI postconference. Both local users and worldwide users actively and repeatedly view cases in our study set.
Collapse
Affiliation(s)
- Paul A Christensen
- Department of Pathology and Genomic Medicine, Houston Methodist Hospital, Weill Cornell Medical College of Cornell University, Houston, TX 77030, USA
| | - Nathan E Lee
- Department of Pathology and Genomic Medicine, Houston Methodist Hospital, Weill Cornell Medical College of Cornell University, Houston, TX 77030, USA
| | - Michael J Thrall
- Department of Pathology and Genomic Medicine, Houston Methodist Hospital, Weill Cornell Medical College of Cornell University, Houston, TX 77030, USA
| | - Suzanne Z Powell
- Department of Pathology and Genomic Medicine, Houston Methodist Hospital, Weill Cornell Medical College of Cornell University, Houston, TX 77030, USA
| | - Patricia Chevez-Barrios
- Department of Pathology and Genomic Medicine, Houston Methodist Hospital, Weill Cornell Medical College of Cornell University, Houston, TX 77030, USA
| | - S Wesley Long
- Department of Pathology and Genomic Medicine, Houston Methodist Hospital, Weill Cornell Medical College of Cornell University, Houston, TX 77030, USA
| |
Collapse
|
25
|
Kok EM, Jarodzka H. Before your very eyes: the value and limitations of eye tracking in medical education. MEDICAL EDUCATION 2017; 51:114-122. [PMID: 27580633 DOI: 10.1111/medu.13066] [Citation(s) in RCA: 53] [Impact Index Per Article: 7.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/30/2015] [Revised: 11/09/2015] [Accepted: 03/01/2016] [Indexed: 05/12/2023]
Abstract
CONTEXT Medicine is a highly visual discipline. Physicians from many specialties constantly use visual information in diagnosis and treatment. However, they are often unable to explain how they use this information. Consequently, it is unclear how to train medical students in this visual processing. Eye tracking is a research technique that may offer answers to these open questions, as it enables researchers to investigate such visual processes directly by measuring eye movements. This may help researchers understand the processes that support or hinder a particular learning outcome. AIM In this article, we clarify the value and limitations of eye tracking for medical education researchers. For example, eye tracking can clarify how experience with medical images mediates diagnostic performance and how students engage with learning materials. Furthermore, eye tracking can also be used directly for training purposes by displaying eye movements of experts in medical images. CONCLUSIONS Eye movements reflect cognitive processes, but cognitive processes cannot be directly inferred from eye-tracking data. In order to interpret eye-tracking data properly, theoretical models must always be the basis for designing experiments as well as for analysing and interpreting eye-tracking data. The interpretation of eye-tracking data is further supported by sound experimental design and methodological triangulation.
Collapse
Affiliation(s)
- Ellen M Kok
- Department of Educational Development and Research, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Halszka Jarodzka
- Welten Institute, Research Centre for Learning, Teaching and Technology, Open University of the Netherlands, Heerlen, The Netherlands
- Humanities Laboratory, Lund University, Lund, Sweden
| |
Collapse
|
26
|
Kok EM, Jarodzka H, de Bruin ABH, BinAmir HAN, Robben SGF, van Merriënboer JJG. Systematic viewing in radiology: seeing more, missing less? ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2016; 21:189-205. [PMID: 26228704 PMCID: PMC4749649 DOI: 10.1007/s10459-015-9624-y] [Citation(s) in RCA: 61] [Impact Index Per Article: 7.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/09/2014] [Accepted: 07/16/2015] [Indexed: 05/26/2023]
Abstract
To prevent radiologists from overlooking lesions, radiology textbooks recommend "systematic viewing," a technique whereby anatomical areas are inspected in a fixed order. This would ensure complete inspection (full coverage) of the image and, in turn, improve diagnostic performance. To test this assumption, two experiments were performed. Both experiments investigated the relationship between systematic viewing, coverage, and diagnostic performance. Additionally, the first investigated whether systematic viewing increases with expertise; the second investigated whether novices benefit from full-coverage or systematic viewing training. In Experiment 1, 11 students, ten residents, and nine radiologists inspected five chest radiographs. Experiment 2 had 75 students undergo a training in either systematic, full-coverage (without being systematic) or non-systematic viewing. Eye movements and diagnostic performance were measured throughout both experiments. In Experiment 1, no significant correlations were found between systematic viewing and coverage, r = -.10, p = .62, and coverage and performance, r = -.06, p = .74. Experts were significantly more systematic than students F2,25 = 4.35, p = .02. In Experiment 2, significant correlations were found between systematic viewing and coverage, r = -.35, p < .01, but not between coverage and performance, r = .13, p = .31. Participants in the full-coverage training performed worse compared with both other groups, which did not differ between them, F2,71 = 3.95, p = .02. In conclusion, the data question the assumption that systematic viewing leads to increased coverage, and, consequently, to improved performance. Experts inspected cases more systematically, but students did not benefit from systematic viewing training.
Collapse
Affiliation(s)
- Ellen M Kok
- Department of Educational Development and Research, School of Health Professions Education, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands.
| | - Halszka Jarodzka
- Welten Institute, Research Centre for Learning, Teaching and Technology, Open University of the Netherlands, Heerlen, The Netherlands
- Humanities Laboratory, Lund University, Lund, Sweden
| | - Anique B H de Bruin
- Department of Educational Development and Research, School of Health Professions Education, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Hussain A N BinAmir
- International Master in Medicine, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands
| | - Simon G F Robben
- Department of Radiology, Maastricht University Medical Center, Maastricht, The Netherlands
| | - Jeroen J G van Merriënboer
- Department of Educational Development and Research, School of Health Professions Education, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| |
Collapse
|
27
|
Kok EM, de Bruin AB, Leppink J, van Merriënboer JJ, Robben SG. Case Comparisons: An Efficient Way of Learning Radiology. Acad Radiol 2015; 22:1226-35. [PMID: 26254543 DOI: 10.1016/j.acra.2015.04.012] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2015] [Revised: 04/22/2015] [Accepted: 04/27/2015] [Indexed: 11/26/2022]
Abstract
RATIONALE AND OBJECTIVES Radiologists commonly use comparison films to improve their differential diagnosis. Educational literature suggests that this technique might also be used to bolster the process of learning to interpret radiographs. We investigated the effectiveness of three comparison techniques in medical students, whom we invited to compare cases of the same disease (same-disease comparison), cases of different diseases (different-disease comparison), disease images with normal images (disease/normal comparison), and identical images (no comparison/control condition). Furthermore, we used eye-tracking technology to investigate which elements of the two cases were compared by the students. MATERIALS AND METHODS We randomly assigned 84 medical students to one of four conditions and had them study different diseases on chest radiographs, while their eye movements were being measured. Thereafter, participants took two tests that measured diagnostic performance and their ability to locate diseases, respectively. RESULTS Students studied most efficiently in the same-disease and different-disease comparison conditions: test 1, F(3, 68) = 3.31, P = .025, ηp(2) = 0.128; test 2, F(3, 65) = 2.88, P = .043, ηp(2) = 0.117. We found that comparisons were effected in 91% of all trials (except for the control condition). Comparisons between normal anatomy were particularly common (45.8%) in all conditions. CONCLUSIONS Comparing cases can be an efficient way of learning to interpret radiographs, especially when the comparison technique used is specifically tailored to the learning goal. Eye tracking provided insight into the comparison process, by showing that few comparisons were made between abnormalities, for example.
Collapse
|
28
|
Balslev T, Rasmussen AB, Skajaa T, Nielsen JP, Muijtjens A, De Grave W, Van Merriënboer J. Combining bimodal presentation schemes and buzz groups improves clinical reasoning and learning at morning report. MEDICAL TEACHER 2015; 37:759-766. [PMID: 25496711 DOI: 10.3109/0142159x.2014.986445] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
Morning reports offer opportunities for intensive work-based learning. In this controlled study, we measured learning processes and outcomes with the report of paediatric emergency room patients. Twelve specialists and 12 residents were randomised into four groups and discussed the same two paediatric cases. The groups differed in their presentation modality (verbal only vs. verbal + text) and the use of buzz groups (with vs. without). The verbal interactions were analysed for clinical reasoning processes. Perceptions of learning and judgment of learning were reported in a questionnaire. Diagnostic accuracy was assessed by a 20-item multiple-choice test. Combined bimodal presentation and buzz groups increased the odds ratio of clinical reasoning to occur in the discussion of cases by a factor of 1.90 (p = 0.013), indicating superior reasoning for buzz groups working with bimodal materials. For specialists, a positive effect of bimodal presentation was found on perceptions of learning (p < 0.05), and for residents, a positive effect of buzz groups was found on judgment of learning (p < 0.005). A positive effect of bimodal presentation on diagnostic accuracy was noted in the specialists (p < 0.05). Combined bimodal presentation and buzz group discussion of emergency cases improves clinicians' clinical reasoning and learning.
Collapse
Affiliation(s)
- Thomas Balslev
- a Viborg Regional Hospital , Denmark
- b Aarhus University , Denmark
| | | | | | | | | | | | | |
Collapse
|
29
|
|
30
|
Abstract
It has been just over a century since Gestalt psychologists described the factors that contribute to the holistic processing of visually presented stimuli. Recent research indicates that holistic processing may come at a cost; specifically, the perception of holistic forms may reduce the visibility of constituent parts. In the present experiment, we examined change detection and change identification accuracy with Kanizsa rectangle patterns that were arranged to either form a Gestalt whole or not. Results from an experiment with 62 participants support this trade-off in processing holistic forms. Holistic processing improved the detection of change but obstructed its identification. Results are discussed in terms of both their theoretical significance and their application in areas ranging from baggage screening and the detection of changes in radiological images to the systems that are used to generate composite images of perpetrators on the basis of eyewitness reports.
Collapse
|
31
|
Ayres P, Paas F. Cognitive Load Theory: New Directions and Challenges. APPLIED COGNITIVE PSYCHOLOGY 2012. [DOI: 10.1002/acp.2882] [Citation(s) in RCA: 58] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Affiliation(s)
- Paul Ayres
- School of Education; University of New South Wales; Sydney; Australia
| | | |
Collapse
|