Expert gaze as a usability indicator of medical AI decision support systems: a preliminary study

Given the current state of medical artificial intelligence (AI) and perceptions towards it, collaborative systems are becoming the preferred choice for clinical workflows. This work aims to address expert interaction with medical AI support systems to gain insight towards how these systems can be better designed with the user in mind. As eye tracking metrics have been shown to be robust indicators of usability, we employ them for evaluating the usability and user interaction with medical AI support systems. We use expert gaze to assess experts' interaction with an AI software for caries detection in bitewing x-ray images. We compared standard viewing of bitewing images without AI support versus viewing where AI support could be freely toggled on and off. We found that experts turned the AI on for roughly 25% of the total inspection task, and generally turned it on halfway through the course of the inspection. Gaze behavior showed that when supported by AI, more attention was dedicated to user interface elements related to the AI support, with more frequent transitions from the image itself to these elements. When considering that expert visual strategy is already optimized for fast and effective image inspection, such interruptions in attention can lead to increased time needed for the overall assessment. Gaze analysis provided valuable insights into an AI's usability for medical image inspection. Further analyses of these tools and how to delineate metrical measures of usability should be developed.

Impact of artificial intelligence on dentists' gaze during caries detection: A randomized controlled trial

OBJECTIVES

We aimed to understand how artificial intelligence (AI) influences dentists by comparing their gaze behavior when using versus not using an AI software to detect primary proximal carious lesions on bitewing radiographs.

METHODS

22 dentists assessed a median of 18 bitewing images resulting in 170 datasets from dentists without AI and 179 datasets from dentists with AI, after excluding data with poor gaze recording quality. We compared time to first fixation, fixation count, average fixation duration, and fixation frequency between both trial groups. Analyses were performed for the entire image and stratified by (1) presence of carious lesions and/or restorations and (2) lesion depth (E1/2: outer/inner enamel; D1-3 outer-inner third of dentin). We also compared the transitional pattern of the dentists' gaze between the trial groups.

RESULTS

Median time to first fixation was shorter in all groups of teeth for dentists with AI versus without AI, although p>0.05. Dentists with AI had more fixations (median=68, IQR=31, 116) on teeth with restorations compared to dentists without AI (median=47, IQR=19, 100), p = 0.01. In turn, average fixation duration was longer on teeth with caries for the dentists with AI than those without AI; although p>0.05. The visual search strategy employed by dentists with AI was less systematic with a lower proportion of lateral tooth-wise transitions compared to dentists without AI.

CONCLUSIONS

Dentists with AI exhibited more efficient viewing behavior compared to dentists without AI, e.g., lesser time taken to notice caries and/or restorations, more fixations on teeth with restorations, and fixating for shorter durations on teeth without carious lesions and/or restorations.

CLINICAL SIGNIFICANCE

Analysis of dentists' gaze patterns while using AI-generated annotations of carious lesions demonstrates how AI influences their data extraction methods for dental images. Such insights can be exploited to improve, and even customize, AI-based diagnostic tools, thus reducing the dentists' extraneous attentional processing and allowing for more thorough examination of other image areas.

Gaze patterns of dentists while evaluating bitewing radiographs

OBJECTIVES

Understanding dentists' gaze patterns on radiographs may allow to unravel sources of their limited accuracy and develop strategies to mitigate them. We conducted an eye tracking experiment to characterize dentists' scanpaths and thus their gaze patterns when assessing bitewing radiographs to detect primary proximal carious lesions.

METHODS

22 dentists assessed a median of nine bitewing images each, resulting in 170 datasets after excluding data with poor quality of gaze recording. Fixation was defined as an area of attentional focus related to visual stimuli. We calculated time to first fixation, fixation count, average fixation duration, and fixation frequency. Analyses were performed for the entire image and stratified by (1) presence of carious lesions and/or restorations and (2) lesion depth (E1/2: outer/inner enamel; D1-3: outer-inner third of dentin). We also examined the transitional nature of the dentists' gaze.

RESULTS

Dentists had more fixations on teeth with lesions and/or restorations (median=138 [interquartile range=87, 204]) than teeth without them (32 [15, 66]), p<0.001. Notably, teeth with lesions had longer fixation durations (407 milliseconds [242, 591]) than those with restorations (289 milliseconds [216, 337]), p<0.001. Time to first fixation was longer for teeth with E1 lesions (17,128 milliseconds [8813, 21,540]) than lesions of other depths (p = 0.049). The highest number of fixations were on teeth with D2 lesions (43 [20, 51]) and lowest on teeth with E1 lesions (5 [1, 37]), p<0.001. Generally, a systematic tooth-by-tooth gaze pattern was observed.

CONCLUSIONS

As hypothesized, while visually inspecting bitewing radiographic images, dentists employed a heightened focus on certain image features/areas, relevant to the assigned task. Also, they generally examined the entire image in a systematic tooth-by-tooth pattern.

A time slice analysis of dentistry students' visual search strategies and pupil dilation during diagnosing radiographs

Diagnosing orthopantomograms (OPTs: panoramic radiographs) is an essential skill dentistry students acquire during university training. While prior research described experts' visual search behavior in radiology as global-to-focal for chest radiographs and mammography, generalizability to a hybrid search task in OPTs (i.e., searching for multiple, diverse anomalies) remains unclear. Addressing this gap, this study investigated visual search of N = 107 dentistry students while they were diagnosing anomalies in OPTs. Following a global-to-focal expert model, we hypothesized that students would use many, short fixations representing global search in earlier stages, and few, long fixations representing focal search in later stages. Furthermore, pupil dilation and mean fixation duration served as cognitive load measures. We hypothesized that later stages would be characterized by elaboration and a reflective search strategy, leading to higher cognitive load being associated with higher diagnostic performance in late compared to earlier stages. In line with the first hypothesis, students' visual search comprised of a three-stage process that grew increasingly focal in terms of the number of fixations and anomalies fixated. Contrary to the second hypothesis, mean fixation duration during anomaly fixations was positively associated with diagnostic performance across all stages. As OPTs greatly varied in how difficult it was to identify the anomalies contained therein, OPTs with above-average difficulty were sampled for exploratory analysis. Pupil dilation predicted diagnostic performance for difficult OPTs, possibly capturing elaborative cognitive processes and cognitive load compared to mean fixation duration. A visual analysis of fine-grained time slices indicated large cognitive load differences towards the end of trials, showcasing a richness-resolution-trade-off in data sampling crucial for future studies using time-slicing of eye tracking data.

Artificial Intelligence for the Analysis of Workload-Related Changes in Radiologists' Gaze Patterns

Around 60-80% of radiological errors are attributed to overlooked abnormalities, the rate of which increases at the end of work shifts. In this study, we run an experiment to investigate if artificial intelligence (AI) can assist in detecting radiologists' gaze patterns that correlate with fatigue. A retrospective database of lung X-ray images with the reference diagnoses was used. The X-ray images were acquired from 400 subjects with a mean age of 49 ± 17, and 61% men. Four practicing radiologists read these images while their eye movements were recorded. The radiologists passed a series of concentration tests at prearranged breaks of the experiment. A U-Net neural network was adapted to annotate lung anatomy on X-rays and calculate coverage and information gain features from the radiologists' eye movements over lung fields. The lung coverage, information gain, and eye tracker-based features were compared with the cumulative work done (CDW) label for each radiologist. The gaze-traveled distance, X-ray coverage, and lung coverage statistically significantly (p < 0.01) deteriorated with cumulative work done (CWD) for three out of four radiologists. The reading time and information gain over lungs statistically significantly deteriorated for all four radiologists. We discovered a novel AI-based metric blending reading time, speed, and organ coverage, which can be used to predict changes in the fatigue-related image reading patterns.

Dental undergraduate students' ability in the diagnosis of root fractures: Comparative analysis of different imaging methods

OBJECTIVE

The aim of this study was to evaluate the diagnostic ability of dental undergraduate students to detect horizontal and oblique root fractures (ORFs) using different imaging techniques.

MATERIALS AND METHODS

Nine teeth were selected and randomly divided in three groups in order to create a fracture line without fragments separation: control (without fracture), horizontal root fracture (HRF), and ORF. The root fracture was created using perpendicular force and was confirmed by transillumination. A model with two adjacent teeth was created, and different imaging techniques were performed: conventional periapical radiograph; mesially and distally shifted periapical radiographs; cone-beam computer tomography (CBCT). Twenty students that had participation on dental trauma clinic at the year of 2019 were invited to identify root fractures by a five-point scale: (i) fracture definitely not present, (ii) fracture probably not present, (iii) uncertain whether fracture is present or not, (iv) fracture probably present, and (v) fracture definitely present. Data were analyzed by Kappa test for agreement evaluation.

RESULTS

Comparing each student to the gold standard, there was a variation in reproducibility and performance from poor to substantial (0.042-0.667). Reproducibility values ranged from poor to good for all periapical radiographs both in the diagnosis of ORF (-0.33-0.667) and in HRF (0-1).

CONCLUSIONS

In CBCT images, the students' ability was lower in HRF detection in comparison with the oblique ones. The students showed limited capacity to diagnose root fractures; however when CBCT was used, the performance was more satisfactory than when periapical radiographs were used.

Viewing patterns regarding panoramic radiographs with different pathological lesions: an eye-tracking study

OBJECTIVE

The aim of this study was to examine how dental students vary their viewing patterns of panoramic radiographs during different levels of dental education.

METHODS

Two groups of students (total number = 48, n = 24) in different grades (second and fifth clinical semester) were compared. The second clinical semester participated twice, as during the second clinical semester a specific lecture on dental radiology and diagnosis is held. The first viewing took place at the beginning of the semester (2a), the second at the end of it (2e). The fifth semester (5e) represents students shortly before graduation. While viewing 20 panoramic radiographs showing specific pathologies, the eye movement was captured by an eye-tracker. After a maximum of 60 s per image, the students had to report a suspected diagnosis. Every panoramic radiograph included a pathological lesion which was diagnosed by an expert observer who also defined the areas of interest (AOI). The images were presented in the same order to each participant. The metric data recorded by the tracking-system included total time to first fixation, total fixation count, total gaze duration and coordinates of the fixation in and outside an area of interest. In addition, parameters like the completeness of scanning and the suspected diagnosis were analysed. Differences between the groups were assessed for statistical significance and associations between level of different grades, viewing time, completeness of scanning and correctness of diagnosis were computed.

RESULTS

2e was significantly faster (p < 0,001), whereas 5e was significantly (p < 0.001) more likely to diagnose correctly and also to scan more completely. Scanning duration did not significantly influence the correctness of diagnosis. The lower edges of the panoramic radiographs were not scanned as often as the centre of the image. Bony lesions were generally found to be difficult to interpret and significant findings located in the sinus were overlooked the most.

CONCLUSION

The higher semester had a more complete viewing pattern and diagnosed correctly with a higher percentage. After hearing the mentioned lecture, the second semester scanned faster and mentioned the AOI more often but could not make a right diagnosis.

Comparing the Visual Perception According to the Performance Using the Eye-Tracking Technology in High-Fidelity Simulation Settings

Introduction: We used eye-tracking technology to explore the visual perception of clinicians during a high-fidelity simulation scenario. We hypothesized that physicians who were able to successfully manage a critical situation would have a different visual focus compared to those who failed. Methods: A convenience sample of 18 first-year emergency medicine residents were enrolled voluntarily to participate in a high-fidelity scenario involving a patient in shock with a 3rd degree atrioventricular block. Their performance was rated as pass or fail and depended on the proper use of the pacing unit. Participants were wearing pre-calibrated eye-tracking glasses throughout the 9-min scenario and infrared (IR) markers installed in the simulator were used to define various Areas of Interest (AOI). Total View Duration (TVD) and Time to First Fixation (TFF) by the participants were recorded for each AOI and the results were used to produce heat maps. Results: Twelve residents succeeded while six failed the scenario. The TVD for the AOI containing the pacing unit was significantly shorter (median [quartile]) for those who succeeded compared to the ones who failed (42 [31–52] sec vs. 70 [61–90] sec, p = 0.0097). The TFF for the AOI containing the ECG and vital signs monitor was also shorter for the participants who succeeded than for those who failed (22 [6–28] sec vs. 30 [27–77] sec, p = 0.0182). Discussion: There seemed to be a connection between the gaze pattern of residents in a high-fidelity bradycardia simulation and their performance. The participants who succeeded looked at the monitor earlier (diagnosis). They also spent less time fixating the pacing unit, using it promptly to address the bradycardia. This study suggests that eye-tracking technology could be used to explore how visual perception, a key information-gathering element, is tied to decision-making and clinical performance.

How to support dental students in reading radiographs: effects of a gaze-based compare-and-contrast intervention

In dental medicine, interpreting radiographs (i.e., orthopantomograms, OPTs) is an error-prone process, even in experts. Effective intervention methods are therefore needed to support students in improving their image reading skills for OPTs. To this end, we developed a compare-and-contrast intervention, which aimed at supporting students in achieving full coverage when visually inspecting OPTs and, consequently, obtaining a better diagnostic performance. The comparison entailed a static eye movement visualization (heat map) on an OPT showing full gaze coverage from a peer-model (other student) and another heat map showing a student's own gaze behavior. The intervention group (N = 38) compared five such heat map combinations, whereas the control group (N = 23) diagnosed five OPTs. Prior to the experimental variation (pre-test) and after it (post-test), students in both conditions searched for anomalies in OPTs while their gaze was recorded. Results showed that students in the intervention group covered more areas of the OPTs and looked less often and for a shorter amount of time at anomalies after the intervention. Furthermore, they fixated on low-prevalence anomalies earlier and high-prevalence anomalies later during the inspection. However, the students in the intervention group did not show any meaningful improvement in detection rate and made more false positive errors compared to the control group. Thus, the intervention guided visual attention but did not improve diagnostic performance substantially. Exploratory analyses indicated that further interventions should teach knowledge about anomalies rather than focusing on full coverage of radiographs.

Evaluation of diagnostic radiology acumen between 2 groups of clinical dental students

PURPOSE

To evaluate and distinguish if an additional year of clinical experience increases the cognitive ability of dental students to accurately assess and interpret dental radiographs.

METHODS

Radiological acuity was assessed between two groups of clinical dental students at Penn Dental Medicine (PDM). Group 1 was composed of 147 third-year dental students (D3), group 2 was composed of 145 fourth-year dental students (D4). A 65-question test comprising the length and breadth of radiographic anatomy and pathology was administered to both D3 and D4 students. The test was designed to test the participants' knowledge of radiographic technique, anatomy, and differential diagnosis. The null hypothesis was that there would be no significant differences between the two groups.

RESULTS

STATA 15 software (StataCorp LLC, College Station, TX, USA) was used to statistically analyze the findings. Although, the mean correct score for group 1 was higher (60/65) than group 2 (59/65), there was no statistically significant difference between the performance of the groups. On average, group 1 outperformed the clinically more experienced group 2 on an individual question basis. The average overall number of correctly answered items compared to incorrectly answered items reflected this difference.

CONCLUSION(S)

An additional year of clinical dental education does not appear to correlate with any higher radiographic acumen. This may be due to tapering exposure to unique pathology and findings, as more routine findings are encountered daily and classroom instruction during the fourth year of dental school is limited. Early incorporation of new radiographic education tools that are clinically oriented may be one method to increase retention of knowledge accumulated in the initial didactic years of dental education.

How massed practice improves visual expertise in reading panoramic radiographs in dental students: An eye tracking study

The interpretation of medical images is an error-prone process that may yield severe consequences for patients. In dental medicine panoramic radiography (OPT) is a frequently used diagnostic procedure. OPTs typically contain multiple, diverse anomalies within one image making the diagnostic process very demanding, rendering students’ development of visual expertise a complex task. Radiograph interpretation is typically taught through massed practice; however, it is not known how effective this approach is nor how it changes students’ visual inspection of radiographs. Therefore, this study investigated how massed practice–an instructional method that entails massed learning of one type of material–affects processing of OPTs and the development of diagnostic performance. From 2017 to 2018, 47 dental students in their first clinical semester diagnosed 10 OPTs before and after their regular massed practice training, which is embedded in their curriculum. The OPTs contained between 3 to 26 to-be-identified anomalies. During massed practice they diagnosed 100 dental radiographs without receiving corrective feedback. The authors recorded students’ eye movements and assessed the number of correctly identified and falsely marked low- and high prevalence anomalies before and after massed practice. Massed practice had a positive effect on detecting anomalies especially with low prevalence (p < .001). After massed practice students covered a larger proportion of the OPTs (p < .001), which was positively related to the detection of low-prevalence anomalies (p = .04). Students also focused longer, more frequently, and earlier on low-prevalence anomalies after massed practice (p_s < .001). While massed practice improved visual expertise in dental students with limited prior knowledge, there is still substantial room for improvement. The results suggest integrating massed practice with more deliberate practice, where, for example, corrective feedback is provided, and support is adapted to students’ needs.

Pupil diameter differentiates expertise in dental radiography visual search

Expert behavior is characterized by rapid information processing abilities, dependent on more structured schemata in long-term memory designated for their domain-specific tasks. From this understanding, expertise can effectively reduce cognitive load on a domain-specific task. However, certain tasks could still evoke different gradations of load even for an expert, e.g., when having to detect subtle anomalies in dental radiographs. Our aim was to measure pupil diameter response to anomalies of varying levels of difficulty in expert and student dentists’ visual examination of panoramic radiographs. We found that students’ pupil diameter dilated significantly from baseline compared to experts, but anomaly difficulty had no effect on pupillary response. In contrast, experts’ pupil diameter responded to varying levels of anomaly difficulty, where more difficult anomalies evoked greater pupil dilation from baseline. Experts thus showed proportional pupillary response indicative of increasing cognitive load with increasingly difficult anomalies, whereas students showed pupillary response indicative of higher cognitive load for all anomalies when compared to experts.

A comparison of visual identification of dental radiographic and nonradiographic images using eye tracking technology

OBJECTIVES

Eye tracking has been used in medical radiology to understand observers' gaze patterns during radiological diagnosis. This study examines the visual identification ability of junior hospital dental officers (JHDOs) and dental surgery assistants (DSAs) in radiographic and nonradiographic images using eye tracking technology and examines if there is a correlation.

MATERIAL AND METHODS

Nine JHDOs and nine DSAs examined six radiographic images and 16 nonradiographic images using eye tracking. The areas of interest (AOIs) of the radiographic images were rated as easy, medium, and hard, and the nonradiographic images were categorized as pattern recognition, face recognition, and image comparison. The participants were required to identify and locate the AOIs. Data analysis of the two domains, entire slide and AOI, was conducted by evaluating the eye tracking metrics (ETM) and the performance outcomes. ETM consisted of six parameters, and performance outcomes consisted of four parameters.

RESULTS

No significant differences were observed for ETMs for JHDOs and DSAs for both radiographic and nonradiographic images. The JHDOs showed significantly higher percentage in identifying AOIs than DSAs for all the radiographic images (72.7% vs. 36.4%, p = .004) and for the easy categorization of radiographic AOIs (85.7% vs. 42.9%, p = .012). JHDOs with higher correct identification percentage in face recognition had a shorter dwell time in AOIs.

CONCLUSIONS

Although no significant relation was observed between radiographic and nonradiographic images, there were some evidence that visual recognition skills may impact certain attributes of the visual search pattern in radiographic images.

Dental students' ability to detect maxillary sinus abnormalities: A comparison between panoramic radiography and cone-beam computed tomography

Purpose

To compare the diagnostic ability of undergraduate dental students to detect maxillary sinus abnormalities in panoramic radiographs (PR) and cone-beam computed tomography (CBCT).

Materials and Methods

This was a retrospective study based on the evaluation of PR and CBCT images. A pilot study was conducted to determine the number of students eligible to participate in the study. The images were evaluated by 2 students, and 280 maxillary sinuses were assessed using the following categories: normal, mucosal thickening, sinus polyp, antral pseudocyst, nonspecific opacification, periostitis, antrolith, and antrolith associated with mucosal thickening. The reference standard was established by the consensus of 2 oral radiologists based on the CBCT images. The kappa test, receiver operating characteristic curves, and 1-way analysis of variance with the Tukey-Kramer post-hoc test were employed.

Results

Intraobserver and interobserver reliability showed agreement ranging from substantial (0.809) to almost perfect (0.922). The agreement between the students' evaluations and the reference standard was reasonable (0.258) for PR and substantial (0.692) for CBCT. Comparisons of values of sensitivity, specificity, and accuracy showed that CBCT was significantly better (P<0.05).

Conclusion

CBCT was better than PR for the detection of maxillary sinus abnormalities by dental students. However, CBCT should only be requested after a careful analysis of PR by students and more experienced professionals.

A review of factors influencing radiologists' visual search behaviour

This narrative literature review aims to identify the various factors that have significant impact on radiologists' visual search patterns. Identifying the factors that influences readers' visual search behaviour helps to understand their perception and interpretation of medical images, which in turn could lead to the development and implementation of effective strategies that could aid in improving the ability to detect abnormalities. Databases including PubMed, MedLine, Web of Science and ScienceDirect were searched using terms 'visual search', 'eye-tracking', 'radiology OR radiography', 'mammogram OR mammography' published since the early 1960s until June 30, 2016. Some of the factors that have been identified to significantly influence radiologists' visual search patterns were (i) readers' expertise, (ii) Satisfaction of Search, (iii) readers' visual fatigue, (iv) readers' confidence in reporting abnormalities, (v) training received and (vi) readers' prior knowledge. Readers' level of expertise was the factor that has been identified to have the most significant impact on their visual search pattern. Eye-tracking studies have shown the differences in visual search patterns of readers with different levels of experience and not so surprisingly, more experienced readers have shown effective visual search strategies. Readers' expertise has also been found to have significant impact in their confidence in reporting abnormalities and their ability to discriminate lesions from background structures in medical images.

Visual Fixation and Scan Patterns of Dentists Viewing Dental Periapical Radiographs: An Eye Tracking Pilot Study

INTRODUCTION

The visual search patterns of dentists and the areas that attract their attention when interpreting dental periapical radiographs are currently unknown. This research identifies areas and patterns of visual fixation when observing dental periapical radiographs.

METHODS

In an observational study using eye tracking technology and a convenience sample of 44 observers, the interpretations of 4 dental periapical radiographs were recorded using Camtasia Software (TechSmith, Okemos, MI) with a gaze tracking "bubble" denoting where within the radiograph the observers' eyes gazed. The recorded observations included the scanning pattern, the area of first fixation, and revisits of areas. Also noted was whether the area of first fixation or revisit was radiopaque, radiolucent, or of normal radiodensity and whether it was a coronal or radicular area.

RESULTS

The first fixation is more likely to be an area of high contrast that is either radiopaque or radiolucent compared with areas that were normal or of average gray scale. Significantly more revisits occurred on areas that were radiopaque and located in the radicular area. Of the 4 categorized scanning patterns, tooth by tooth scanning predominated.

CONCLUSIONS

When interpreting dental periapical radiographs, significantly more observers initially fixated on areas of the radiograph that were of high contrast (ie, radiopaque or radiolucent) compared with "normal areas." A tooth by tooth scanning pattern was most commonly used.

Evaluation of symptomatic maxillary sinus pathologies using panoramic radiography and cone beam computed tomography-influence of professional training

Background

A comparison of panoramic radiography (PAN) alone and PAN together with small field of view cone beam computed tomography (sFOV-CBCT) for diagnosis of symptomatic pathologies of the maxillary sinus was carried out by clinicians of different experience.

Methods

Corresponding radiographic images (PAN/sFOV-CBCT) of 28 patients with symptomatic maxillary sinus pathologies were chosen and analyzed by two general practitioners (GP), two junior maxillofacial surgeons (MS1), and three senior maxillofacial surgeons (MS2) via questionnaire.

Results

Visibility of maxillary pathologies in PAN was significantly different between the groups (GP 39%, MS1 48%, MS2 61%; p < 0.05). The number of incidental findings varied within examiner groups in PAN with a significant increase in MS2 (p = 0.027). The majority of examiners rated an additional sFOV-CBCT as “reasonable”/“required” with a significant influence of the examining groups (GP 98.2%, MS1 94.6%, MS2 80.9%; p = 0.008). In 58% of cases, an additional sFOV-CBCT was seen as “affecting therapy” with significant differences between the groups (GP 68%, MS1 50%, MS2 55%; p < 0.001).

Conclusions

PAN alone is not sufficient for the evaluation of pathologies of the maxillary sinus. But, depending on the examiners’ clinical experience, it remains a useful diagnostic tool. Along with the observers’ training, significant benefits of an additional sFOV-CBCT for evaluation of symptomatic maxillary sinus pathologies were detected.