Towards a framework for analysis of eye-tracking studies in the three dimensional environment: a study of visual search by experienced readers of endoluminal CT colonography

Doug Altman, medical statistician par excellence: What can radiologists learn from his legacy?

This narrative review describes our experience of working with Doug Altman, the most highly cited medical statistician in the world. Doug was particularly interested in diagnostics, and imaging studies in particular. We describe how his insights helped improve our own radiological research studies and we provide advice for other researchers hoping to improve their own research practice.

Visual Image Annotation for Bowel Obstruction: Repeatability and Agreement with Manual Annotation and Neural Networks

Bowel obstruction is a common cause of acute abdominal pain. The development of algorithms for automated detection and characterization of bowel obstruction on CT has been limited by the effort required for manual annotation. Visual image annotation with an eye tracking device may mitigate that limitation. The purpose of this study is to assess the agreement between visual and manual annotations for bowel segmentation and diameter measurement, and to assess agreement with convolutional neural networks (CNNs) trained using that data. Sixty CT scans of 50 patients with bowel obstruction from March to June 2022 were retrospectively included and partitioned into training and test data sets. An eye tracking device was used to record 3-dimensional coordinates within the scans, while a radiologist cast their gaze at the centerline of the bowel, and adjusted the size of a superimposed ROI to approximate the diameter of the bowel. For each scan, 59.4 ± 15.1 segments, 847.9 ± 228.1 gaze locations, and 5.8 ± 1.2 m of bowel were recorded. 2d and 3d CNNs were trained using this data to predict bowel segmentation and diameter maps from the CT scans. For comparisons between two repetitions of visual annotation, CNN predictions, and manual annotations, Dice scores for bowel segmentation ranged from 0.69 ± 0.17 to 0.81 ± 0.04 and intraclass correlations [95% CI] for diameter measurement ranged from 0.672 [0.490-0.782] to 0.940 [0.933-0.947]. Thus, visual image annotation is a promising technique for training CNNs to perform bowel segmentation and diameter measurement in CT scans of patients with bowel obstruction.

An Eye Tracking Based Framework for Safety Improvement of Offshore Operations

Offshore drilling operations consist of complex and high-risk processes. Lack of situational awareness in drilling operations has become an important human factor issue that causes safety accidents. Prolonged work shifts and fatigue are some of the crucial issues that impact performance. Eye tracking technology can be used to distinguish the degree of awareness or alertness of participants that might be related to fatigue or onsite distractions. Oculomotor activity can be used to obtain visual cues that can quantify the drilling operators' situational awareness that might enable us to develop warning alarms to alert the driller. Such systems can help reduce accidents and save non-productive time. In this paper, eye movement char-acteristics were investigated to differentiate the situational awareness between a representa-tive expert and a group of novices using a scenario-based Virtual Reality Drilling Simulator. Significant visual oculomotor activity differences were identified between the expert and the novices that indicate an eye-tracking based system can detect the distraction and alert-ness exhibited by the workers. Results show promise on developing a framework which implements a real-time eye tracking technology in various drilling operations at drilling rigs and Real Time Operation Centers to improve process safety.

Characteristics of expert search behavior in volumetric medical image interpretation

Purpose: Experienced radiologists have enhanced global processing ability relative to novices, allowing experts to rapidly detect medical abnormalities without performing an exhaustive search. However, evidence for global processing models is primarily limited to two-dimensional image interpretation, and it is unclear whether these findings generalize to volumetric images, which are widely used in clinical practice. We examined whether radiologists searching volumetric images use methods consistent with global processing models of expertise. In addition, we investigated whether search strategy (scanning/drilling) differs with experience level. Approach: Fifty radiologists with a wide range of experience evaluated chest computed-tomography scans for lung nodules while their eye movements and scrolling behaviors were tracked. Multiple linear regressions were used to determine: (1) how search behaviors differed with years of experience and the number of chest CTs evaluated per week and (2) which search behaviors predicted better performance. Results: Contrary to global processing models based on 2D images, experience was unrelated to measures of global processing (saccadic amplitude, coverage, time to first fixation, search time, and depth passes) in this task. Drilling behavior was associated with better accuracy than scanning behavior when controlling for observer experience. Greater image coverage was a strong predictor of task accuracy. Conclusions: Global processing ability may play a relatively small role in volumetric image interpretation, where global scene statistics are not available to radiologists in a single glance. Rather, in volumetric images, it may be more important to engage in search strategies that support a more thorough search of the image.

Influence of 3D laparoscopic surgery on surgeon's visual pattern and mental workload

Previous studies have found that surgeons perform better in three-dimensional (3D) surgery than in two-dimensional (2D) surgery. However, no comparative studies have revealed the impact of 3D laparoscopic surgery on the surgeon's vision. To explore the effect of laparoscopic surgeons' depth perception during 3D laparoscopic surgery, 10 participants were recruited and performed 4 sets comparative simulated laparoscopic procedures in a virtual simulator, and eye movement signals were acquired, which were used to characteristics the visual differences. Fixation rate and saccade speed were used to characterise the influence of moderating variables for visual characteristics. The results from the data showed significant differences in eye movement behaviour. Compared with 2D laparoscopic surgery, surgeons have more average fixation rate (p-values = 0.001, 0.000, 0.003 and 0.015, respectively) and faster saccade speed (p-values = 0.037, 0.003, 0.073 and 0.105, respectively) in 3D laparoscopic surgery. The results of this study showed that surgeons had more efficient visual search in 3D laparoscopic surgery. At the same time, the results also indicated that surgeon's mental workload in 3D laparoscopic surgery was low. The relevant conclusions of this paper revealed the advantages of 3D laparoscopic surgery through visual efficiency.

Differences in Gaze Fixation Location and Duration Between Resident and Fellowship Sonographers Interpreting a Focused Assessment With Sonography in Trauma

OBJECTIVES

We quantified the gaze fixation duration of resident and fellowship sonographers interpreting a prerecorded focused assessment with sonography in trauma (FAST). We hypothesized that all sonographers would fixate on each relevant anatomic relationship but that the duration of fixation would differ.

METHODS

We conducted a cross-sectional study collecting and analyzing the gaze fixations of a convenience sample of current resident and fellowship sonographers. All sonographers viewed a standardized FAST video, and their gaze fixations were recorded using a Tobii X3-120 eye-tracking bar. Gaze fixations over nine anatomic regions of interest (ROIs) were identified. These were assessed for normality and analyzed using the Wilcoxon rank sum test at an alpha of 0.05 and Bonferroni correction p value of <0.0034. The chi-square test and Pearson's correlation were performed to assess statistical association.

RESULTS

The gaze fixation recordings of 24 resident and eight fellowship sonographers were suitable for analysis. Fourteen of the 24 resident sonographers viewed all ROIs in the FAST, whereas all eight fellowship sonographers viewed each of the nine relevant ROIs. Five ROIs were identified over which at least one resident sonographer did not have a gaze fixation. No statistically significant difference was identified between groups. Resident sonographers gaze fixated over the left upper quadrant (LUQ) splenorenal interface for a median (interquartile range) of 10.64 (9.73-11.60) seconds. The fellowship group viewed the same ROI for 8.43 (6.64-8.95) seconds (p < 0.003). All participants viewed this ROI. No other ROIs had a statistical difference.

CONCLUSION

Five ROIs were identified that were not visually interrogated by all resident sonographers. Only 14 of 24 resident sonographers visually interrogated every area in the FAST, whereas all fellowship sonographers interrogated every ROI. A statistically significant difference was found in gaze fixation duration between resident and fellowship sonographers in one ROI. Further study is required for gaze fixation assessment to become a tool for the interpretation component of point-of-care ultrasound.

Eye Tracking for Deep Learning Segmentation Using Convolutional Neural Networks

Deep learning with convolutional neural networks (CNNs) has experienced tremendous growth in multiple healthcare applications and has been shown to have high accuracy in semantic segmentation of medical (e.g., radiology and pathology) images. However, a key barrier in the required training of CNNs is obtaining large-scale and precisely annotated imaging data. We sought to address the lack of annotated data with eye tracking technology. As a proof of principle, our hypothesis was that segmentation masks generated with the help of eye tracking (ET) would be very similar to those rendered by hand annotation (HA). Additionally, our goal was to show that a CNN trained on ET masks would be equivalent to one trained on HA masks, the latter being the current standard approach. Step 1: Screen captures of 19 publicly available radiologic images of assorted structures within various modalities were analyzed. ET and HA masks for all regions of interest (ROIs) were generated from these image datasets. Step 2: Utilizing a similar approach, ET and HA masks for 356 publicly available T1-weighted postcontrast meningioma images were generated. Three hundred six of these image + mask pairs were used to train a CNN with U-net-based architecture. The remaining 50 images were used as the independent test set. Step 1: ET and HA masks for the nonneurological images had an average Dice similarity coefficient (DSC) of 0.86 between each other. Step 2: Meningioma ET and HA masks had an average DSC of 0.85 between each other. After separate training using both approaches, the ET approach performed virtually identically to HA on the test set of 50 images. The former had an area under the curve (AUC) of 0.88, while the latter had AUC of 0.87. ET and HA predictions had trimmed mean DSCs compared to the original HA maps of 0.73 and 0.74, respectively. These trimmed DSCs between ET and HA were found to be statistically equivalent with a p value of 0.015. We have demonstrated that ET can create segmentation masks suitable for deep learning semantic segmentation. Future work will integrate ET to produce masks in a faster, more natural manner that distracts less from typical radiology clinical workflow.

The Radiologist's Gaze: Mapping Three-Dimensional Visual Search in Computed Tomography of the Abdomen and Pelvis

A radiologist's search pattern can directly influence patient management. A missed finding is a missed opportunity for intervention. Multiple studies have attempted to describe and quantify search patterns but have mainly focused on chest radiographs and chest CTs. Here, we describe and quantify the visual search patterns of 17 radiologists as they scroll through 6 CTs of the abdomen and pelvis. Search pattern tracings varied among individuals and remained relatively consistent per individual between cases. Attendings and trainees had similar eye metric statistics with respect to time to first fixation (TTFF), number of fixations in the region of interest (ROI), fixation duration in ROI, mean saccadic amplitude, or total number of fixations. Attendings had fewer numbers of fixations per second versus trainees (p < 0.001), suggesting efficiency due to expertise. In those cases that were accurately interpreted, TTFF was shorter (p = 0.04), the number of fixations per second and number of fixations in ROI were higher (p = 0.04, p = 0.02, respectively), and fixation duration in ROI was increased (p = 0.02). We subsequently categorized radiologists as "scanners" or "drillers" by both qualitative and quantitative methods and found no differences in accuracy with most radiologists being categorized as "drillers." This study describes visual search patterns of radiologists in interpretation of CTs of the abdomen and pelvis to better approach future endeavors in determining the effects of manipulations such as fatigue, interruptions, and computer-aided detection.

What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures

Interpretation of volumetric medical images represents a rapidly growing proportion of the workload in radiology. However, relatively little is known about the strategies that best guide search behavior when looking for abnormalities in volumetric images. Although there is extensive literature on two-dimensional medical image perception, it is an open question whether the conclusions drawn from these images can be generalized to volumetric images. Importantly, volumetric images have distinct characteristics (e.g., scrolling through depth, smooth-pursuit eye-movements, motion onset cues, etc.) that should be considered in future research. In this manuscript, we will review the literature on medical image perception and discuss relevant findings from basic science that can be used to generate predictions about expertise in volumetric image interpretation. By better understanding search through volumetric images, we may be able to identify common sources of error, characterize the optimal strategies for searching through depth, or develop new training and assessment techniques for radiology residents.

Eye-Tracking Study to Enhance Usability of Molecular Diagnostics Reports in Cancer Precision Medicine

PURPOSE

We conducted usability studies on commercially available molecular diagnostic (MDX) test reports to identify strengths and weaknesses in content and form that drive clinical decision making. Given routine genomic testing in cancer medicine, oncologists must interpret MDX reports as well as evidence concerning clinical utility of biomarkers accurately for treatment or trial selection. This work aims to evaluate effectiveness of MDX reports in facilitating cancer treatment planning.

METHODS

Fourteen clinicians at an academic tertiary care medical facility, with a wide range of experience in oncology and in the use of molecular testing, participated in this study. Three commercially available, widely used, Clinical Laboratory Improvement Amendments (CLIA)-certified, College of American Pathologists (CAP)-accredited test reports (labeled Laboratories A, B, and C) were used. Eye tracking, surveys, and think-aloud protocols were used to collect usability data for these MDX reports focusing on ease of comprehension and actionability.

RESULTS

Clinicians found two primary areas in molecular diagnostic reports most useful for patient care: therapy options with benefit or lack of benefit to patients, including enrolling clinical trials; and pathogenic tumor molecular anomalies detected. Therapeutic implications and therapy classes such as US Food and Drug Administration-approved off-label, on-label, clinical trials were critical for decision making. However, all reports had usability and comprehension issues in these areas and could be improved.

CONCLUSION

Focused usability studies can help drive our understanding of the clinical workflow for use of molecular diagnostic tests in cancer care. This in turn can have major effects on quality of care, outcomes, costs, and patient satisfaction. This study demonstrates the use of specific usability techniques (eye tracking and think-aloud protocols) to help clinical laboratories improve MDX report design in a precision oncology treatment setting.

Volumetric image interpretation in radiology: scroll behavior and cognitive processes

The interpretation of medical images is a primary task for radiologists. Besides two-dimensional (2D) images, current imaging technologies allow for volumetric display of medical images. Whereas current radiology practice increasingly uses volumetric images, the majority of studies on medical image interpretation is conducted on 2D images. The current study aimed to gain deeper insight into the volumetric image interpretation process by examining this process in twenty radiology trainees who all completed four volumetric image cases. Two types of data were obtained concerning scroll behaviors and think-aloud data. Types of scroll behavior concerned oscillations, half runs, full runs, image manipulations, and interruptions. Think-aloud data were coded by a framework of knowledge and skills in radiology including three cognitive processes: perception, analysis, and synthesis. Relating scroll behavior to cognitive processes showed that oscillations and half runs coincided more often with analysis and synthesis than full runs, whereas full runs coincided more often with perception than oscillations and half runs. Interruptions were characterized by synthesis and image manipulations by perception. In addition, we investigated relations between cognitive processes and found an overall bottom-up way of reasoning with dynamic interactions between cognitive processes, especially between perception and analysis. In sum, our results highlight the dynamic interactions between these processes and the grounding of cognitive processes in scroll behavior. It suggests, that the types of scroll behavior are relevant to describe how radiologists interact with and manipulate volumetric images.

Increasing Navigation Speed at Endoluminal CT Colonography Reduces Colonic Visualization and Polyp Identification

As navigation speed increases, gaze of the reader becomes more central and polyp identification rates fall.

Purpose

To investigate the effect of increasing navigation speed on the visual search and decision making during polyp identification for computed tomography (CT) colonography

Materials and Methods

Institutional review board permission was obtained to use deidentified CT colonography data for this prospective reader study. After obtaining informed consent from the readers, 12 CT colonography fly-through examinations that depicted eight polyps were presented at four different fixed navigation speeds to 23 radiologists. Speeds ranged from 1 cm/sec to 4.5 cm/sec. Gaze position was tracked by using an infrared eye tracker, and readers indicated that they saw a polyp by clicking a mouse. Patterns of searching and decision making by speed were investigated graphically and by multilevel modeling.

Results

Readers identified polyps correctly in 56 of 77 (72.7%) of viewings at the slowest speed but in only 137 of 225 (60.9%) of viewings at the fastest speed (P = .004). They also identified fewer false-positive features at faster speeds (42 of 115; 36.5%) of videos at slowest speed, 89 of 345 (25.8%) at fastest, P = .02). Gaze location was highly concentrated toward the central quarter of the screen area at faster speeds (mean gaze points at slowest speed vs fastest speed, 86% vs 97%, respectively).

Conclusion

Faster navigation speed at endoluminal CT colonography led to progressive restriction of visual search patterns. Greater speed also reduced both true-positive and false-positive colorectal polyp identification.

^© RSNA, 2017

Online supplemental material is available for this article.

Impact of Patient Photographs on Radiologists' Visual Search of Chest Radiographs

RATIONALE AND OBJECTIVES

To increase detection of mislabeled medical imaging studies, evidence shows it may be useful to include patient photographs during interpretation. This study examined how inclusion of photographs impacts visual search.

MATERIALS AND METHODS

Ten radiologists participated. Average age was 43.00 years and average years Board-certified was 9.70, with 2 residents, 1 general, 2 abdominal, 4 cardiothoracic, and 1 pediatric radiologist. They viewed 21 portable chest radiographs with and without a simultaneously acquired photograph of the patient while visual search was recorded. Their task was to note placement of lines and tubes.

RESULTS

Presence of the photograph reduced the number of fixations (chest radiograph only mean 98.68; chest with photograph present 80.81; photograph 10.59; p < 0.0001) and total dwell (chest radiograph only mean 30.84 seconds; chest radiograph with photograph present 25.68; photograph 3.93; p < 0.0001) on the chest radiograph as a result of periodically looking at the photograph. Overall viewing time did not increase with addition of the photograph because time not spent on the radiograph was spent on the photograph. On average, readers scanned from the radiograph to the photographs about four times during search. Men and non-cardiothoracic radiologists spent significantly more time scanning all the images, including the photographs. Average preference for having photographs was 6.10 on a 0-10 scale, and neck and chest were preferred as areas to include in the photograph.

CONCLUSION

Photographs may help with certain image interpretation tasks and may help personalize the reading experience for radiologists without increasing interpretation time.

Do prevalence expectations affect patterns of visual search and decision-making in interpreting CT colonography endoluminal videos?

OBJECTIVE

To assess the effect of expected abnormality prevalence on visual search and decision-making in CT colonography (CTC).

METHODS

13 radiologists interpreted endoluminal CTC fly-throughs of the same group of 10 patient cases, 3 times each. Abnormality prevalence was fixed (50%), but readers were told, before viewing each group, that prevalence was either 20%, 50% or 80% in the population from which cases were drawn. Infrared visual search recording was used. Readers indicated seeing a polyp by clicking a mouse. Multilevel modelling quantified the effect of expected prevalence on outcomes.

RESULTS

Differences between expected prevalence were not statistically significant for time to first pursuit of the polyp (median 0.5 s, each prevalence), pursuit rate when no polyp was on screen (median 2.7 s(-1), each prevalence) or number of mouse clicks [mean 0.75/video (20% prevalence), 0.93 (50%), 0.97 (80%)]. There was weak evidence of increased tendency to look outside the central screen area at 80% prevalence and reduction in positive polyp identifications at 20% prevalence.

CONCLUSION

This study did not find a large effect of prevalence information on most visual search metrics or polyp identification in CTC. Further research is required to quantify effects at lower prevalence and in relation to secondary outcome measures.

ADVANCES IN KNOWLEDGE

Prevalence effects in evaluating CTC have not previously been assessed. In this study, providing expected prevalence information did not have a large effect on diagnostic decisions or patterns of visual search.

Using multidimensional scaling to quantify similarity in visual search and beyond

Visual search is one of the most widely studied topics in vision science, both as an independent topic of interest, and as a tool for studying attention and visual cognition. A wide literature exists that seeks to understand how people find things under varying conditions of difficulty and complexity, and in situations ranging from the mundane (e.g., looking for one's keys) to those with significant societal importance (e.g., baggage or medical screening). A primary determinant of the ease and probability of success during search are the similarity relationships that exist in the search environment, such as the similarity between the background and the target, or the likeness of the non-targets to one another. A sense of similarity is often intuitive, but it is seldom quantified directly. This presents a problem in that similarity relationships are imprecisely specified, limiting the capacity of the researcher to examine adequately their influence. In this article, we present a novel approach to overcoming this problem that combines multi-dimensional scaling (MDS) analyses with behavioral and eye-tracking measurements. We propose a method whereby MDS can be repurposed to successfully quantify the similarity of experimental stimuli, thereby opening up theoretical questions in visual search and attention that cannot currently be addressed. These quantifications, in conjunction with behavioral and oculomotor measures, allow for critical observations about how similarity affects performance, information selection, and information processing. We provide a demonstration and tutorial of the approach, identify documented examples of its use, discuss how complementary computer vision methods could also be adopted, and close with a discussion of potential avenues for future application of this technique.

Review of prospects and challenges of eye tracking in volumetric imaging

While eye tracking research in conventional radiography has flourished over the past decades, the number of eye tracking studies that looked at multislice images lags behind. A possible reason for the lack of studies in this area might be that the eye tracking methodology used in the context of conventional radiography cannot be applied one-on-one to volumetric imaging material. Challenges associated with eye tracking in volumetric imaging are particularly associated with the selection of stimulus material, the detection of events in the eye tracking data, the calculation of meaningful eye tracking parameters, and the reporting of abnormalities. However, all of these challenges can be addressed in the design of the experiment. If this is done, eye tracking studies using volumetric imaging material offer almost unlimited opportunity for perception research and are highly relevant as the number of volumetric images that are acquired and interpreted is rising.

The effect of computer-aided detection markers on visual search and reader performance during concurrent reading of CT colonography

OBJECTIVE

We aimed to identify the effect of computer-aided detection (CAD) on visual search and performance in CT Colonography (CTC) of inexperienced and experienced readers.

METHODS

Fifteen endoluminal CTC examinations were recorded, each with one polyp, and two videos were generated, one with and one without a CAD mark. Forty-two readers (17 experienced, 25 inexperienced) interpreted the videos during infrared visual search recording. CAD markers and polyps were treated as regions of interest in data processing. This multi-reader, multi-case study was analysed using multilevel modelling.

RESULTS

CAD drew readers' attention to polyps faster, accelerating identification times: median 'time to first pursuit' was 0.48 s (IQR 0.27 to 0.87 s) with CAD, versus 0.58 s (IQR 0.35 to 1.06 s) without. For inexperienced readers, CAD also held visual attention for longer. All visual search metrics used to assess visual gaze behaviour demonstrated statistically significant differences when "with" and "without" CAD were compared. A significant increase in the number of correct polyp identifications across all readers was seen with CAD (74 % without CAD, 87 % with CAD; p < 0.001).

CONCLUSIONS

CAD significantly alters visual search and polyp identification in readers viewing three-dimensional endoluminal CTC. For polyp and CAD marker pursuit times, CAD generally exerted a larger effect on inexperienced readers.

KEY POINTS

• Visual gaze is attracted by computer-assisted detection (CAD) marks on polyps • Inexperienced readers' gaze is affected more by CAD than experienced readers. • CAD marks could mean that the unannotated endoluminal surface is relatively neglected. • Correct polyp identification is increased significantly by CAD.

Tracking eye gaze during interpretation of endoluminal three-dimensional CT colonography: visual perception of experienced and inexperienced readers

PURPOSE

To identify and compare key stages of the visual process in experienced and inexperienced readers and to examine how these processes are used to search a moving three-dimensional ( 3D three-dimensional ) image and their relationship to false-negative errors.

MATERIALS AND METHODS

Institutional review board research ethics approval was granted to use anonymized computed tomographic (CT) colonographic data from previous studies and to obtain eye-tracking data from volunteers. Sixty-five radiologists (27 experienced, 38 inexperienced) interpreted 23 endoluminal 3D three-dimensional CT colonographic videos. Eye movements were recorded by using eye tracking with a desk-mounted tracker. Readers indicated when they saw a polyp by clicking a computer mouse. Polyp location and boundary on each video frame were quantified and gaze data were related to the polyp boundary for each individual reader and case. Predefined metrics were quantified and used to describe and compare visual search patterns between experienced and inexperienced readers by using multilevel modeling.

RESULTS

Time to first pursuit was significantly shorter in experienced readers (hazard ratio, 1.22 [95% confidence interval: 1.04, 1.44]; P = .017) but other metrics were not significantly different. Regardless of expertise, metrics such as assessment, identification period, and pursuit times were extended in videos where polyps were visible on screen for longer periods of time. In 97% (760 of 787) of observations, readers correctly pursued polyps.

CONCLUSION

Experienced readers had shorter time to first eye pursuit, but many other characteristics of eye tracking were similar between experienced and inexperienced readers. Readers pursued polyps in 97% of observations, which indicated that errors during interpretation of 3D three-dimensional CT colonography in this study occurred in either the discovery or the recognition phase, but rarely in the scanning phase of radiologic image inspection.