Investigating the link between radiologists' gaze, diagnostic decision, and image content

Interobserver agreement during clinical magnetic resonance imaging of the equine foot

BACKGROUND

Agreement between experienced observers for assessment of pathology and assessment confidence are poorly documented for magnetic resonance imaging (MRI) of the equine foot.

OBJECTIVES

To report interobserver agreement for pathology assessment and observer confidence for key anatomical structures of the equine foot during MRI.

STUDY DESIGN

Exploratory clinical study.

METHODS

Ten experienced observers (diploma or associate level) assessed 15 equine foot MRI studies acquired from clinical databases of 3 MRI systems. Observers graded pathology in seven key anatomical structures (Grade 1: no pathology, Grade 2: mild pathology, Grade 3: moderate pathology, Grade 4: severe pathology) and provided a grade for their confidence for each pathology assessment (Grade 1: high confidence, Grade 2: moderate confidence, Grade 3: limited confidence, Grade 4: no confidence). Interobserver agreement for the presence/absence of pathology and agreement for individual grades of pathology were assessed with Fleiss' kappa (k). Overall interobserver agreement for pathology was determined using Fleiss' kappa and Kendall's coefficient of concordance (KCC). The distribution of grading was also visualised with bubble charts.

RESULTS

Interobserver agreement for the presence/absence of pathology of individual anatomical structures was poor-to-fair, except for the navicular bone which had moderate agreement (k = 0.52). Relative agreement for pathology grading (accounting for the ranking of grades) ranged from KCC = 0.19 for the distal interphalangeal joint to KCC = 0.70 for the navicular bone. Agreement was generally greatest at the extremes of pathology. Observer confidence in pathology assessment was generally moderate to high.

MAIN LIMITATIONS

Distribution of pathology varied between anatomical structures due to random selection of clinical MRI studies. Observers had most experience with low-field MRI.

CONCLUSIONS

Even with experienced observers, there can be notable variation in the perceived severity of foot pathology on MRI for individual cases, which could be important in a clinical context.

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

Machine learning (ML) has revolutionized medical image-based diagnostics. In this review, we cover a rapidly emerging field that can be potentially significantly impacted by ML - eye tracking in medical imaging. The review investigates the clinical, algorithmic, and hardware properties of the existing studies. In particular, it evaluates 1) the type of eye-tracking equipment used and how the equipment aligns with study aims; 2) the software required to record and process eye-tracking data, which often requires user interface development, and controller command and voice recording; 3) the ML methodology utilized depending on the anatomy of interest, gaze data representation, and target clinical application. The review concludes with a summary of recommendations for future studies, and confirms that the inclusion of gaze data broadens the ML applicability in Radiology from computer-aided diagnosis (CAD) to gaze-based image annotation, physicians' error detection, fatigue recognition, and other areas of potentially high research and clinical impact.

Shedding light on ai in radiology: A systematic review and taxonomy of eye gaze-driven interpretability in deep learning

X-ray imaging plays a crucial role in diagnostic medicine. Yet, a significant portion of the global population lacks access to this essential technology due to a shortage of trained radiologists. Eye-tracking data and deep learning models can enhance X-ray analysis by mapping expert focus areas, guiding automated anomaly detection, optimizing workflow efficiency, and bolstering training methods for novice radiologists. However, the literature shows contradictory results regarding the usefulness of eye-tracking data in deep-learning architectures for abnormality detection. We argue that these discrepancies between studies in the literature are due to (a) the way eye-tracking data is (or is not) processed, (b) the types of deep learning architectures chosen, and (c) the type of application that these architectures will have. We conducted a systematic literature review using PRISMA to address these contradicting results. We analyzed 60 studies that incorporated eye-tracking data in a deep-learning approach for different application goals in radiology. We performed a comparative analysis to understand if eye gaze data contains feature maps that can be useful under a deep learning approach and whether they can promote more interpretable predictions. To the best of our knowledge, this is the first survey in the area that performs a thorough investigation of eye gaze data processing techniques and their impacts in different deep learning architectures for applications such as error detection, classification, object detection, expertise level analysis, fatigue estimation and human attention prediction in medical imaging data. Our analysis resulted in two main contributions: (1) taxonomy that first divides the literature by task, enabling us to analyze the value eye movement can bring for each case and build guidelines regarding architectures and gaze processing techniques adequate for each application, and (2) an overall analysis of how eye gaze data can promote explainability in radiology.

Predicting Clinician Fixations on Glaucoma OCT Reports via CNN-Based Saliency Prediction Methods

Goal: To predict physician fixations specifically on ophthalmology optical coherence tomography (OCT) reports from eye tracking data using CNN based saliency prediction methods in order to aid in the education of ophthalmologists and ophthalmologists-in-training. Methods: Fifteen ophthalmologists were recruited to each examine 20 randomly selected OCT reports and evaluate the likelihood of glaucoma for each report on a scale of 0-100. Eye movements were collected using a Pupil Labs Core eye-tracker. Fixation heat maps were generated using fixation data. Results: A model trained with traditional saliency mapping resulted in a correlation coefficient (CC) value of 0.208, a Normalized Scanpath Saliency (NSS) value of 0.8172, a Kullback-Leibler (KLD) value of 2.573, and a Structural Similarity Index (SSIM) of 0.169. Conclusions: The TranSalNet model was able to predict fixations within certain regions of the OCT report with reasonable accuracy, but more data is needed to improve model accuracy. Future steps include increasing data collection, improving quality of data, and modifying the model architecture.

Machine learning classification of diagnostic accuracy in pathologists interpreting breast biopsies

OBJECTIVE

This study explores the feasibility of using machine learning to predict accurate versus inaccurate diagnoses made by pathologists based on their spatiotemporal viewing behavior when evaluating digital breast biopsy images.

MATERIALS AND METHODS

The study gathered data from 140 pathologists of varying experience levels who each reviewed a set of 14 digital whole slide images of breast biopsy tissue. Pathologists' viewing behavior, including zooming and panning actions, was recorded during image evaluation. A total of 30 features were extracted from the viewing behavior data, and 4 machine learning algorithms were used to build classifiers for predicting diagnostic accuracy.

RESULTS

The Random Forest classifier demonstrated the best overall performance, achieving a test accuracy of 0.81 and area under the receiver-operator characteristic curve of 0.86. Features related to attention distribution and focus on critical regions of interest were found to be important predictors of diagnostic accuracy. Further including case-level and pathologist-level information incrementally improved classifier performance.

DISCUSSION

Results suggest that pathologists' viewing behavior during digital image evaluation can be leveraged to predict diagnostic accuracy, affording automated feedback and decision support systems based on viewing behavior to aid in training and, ultimately, clinical practice. They also carry implications for basic research examining the interplay between perception, thought, and action in diagnostic decision-making.

CONCLUSION

The classifiers developed herein have potential applications in training and clinical settings to provide timely feedback and support to pathologists during diagnostic decision-making. Further research could explore the generalizability of these findings to other medical domains and varied levels of expertise.

Visual Image Annotation for Bowel Obstruction: Repeatability and Agreement with Manual Annotation and Neural Networks

Bowel obstruction is a common cause of acute abdominal pain. The development of algorithms for automated detection and characterization of bowel obstruction on CT has been limited by the effort required for manual annotation. Visual image annotation with an eye tracking device may mitigate that limitation. The purpose of this study is to assess the agreement between visual and manual annotations for bowel segmentation and diameter measurement, and to assess agreement with convolutional neural networks (CNNs) trained using that data. Sixty CT scans of 50 patients with bowel obstruction from March to June 2022 were retrospectively included and partitioned into training and test data sets. An eye tracking device was used to record 3-dimensional coordinates within the scans, while a radiologist cast their gaze at the centerline of the bowel, and adjusted the size of a superimposed ROI to approximate the diameter of the bowel. For each scan, 59.4 ± 15.1 segments, 847.9 ± 228.1 gaze locations, and 5.8 ± 1.2 m of bowel were recorded. 2d and 3d CNNs were trained using this data to predict bowel segmentation and diameter maps from the CT scans. For comparisons between two repetitions of visual annotation, CNN predictions, and manual annotations, Dice scores for bowel segmentation ranged from 0.69 ± 0.17 to 0.81 ± 0.04 and intraclass correlations [95% CI] for diameter measurement ranged from 0.672 [0.490-0.782] to 0.940 [0.933-0.947]. Thus, visual image annotation is a promising technique for training CNNs to perform bowel segmentation and diameter measurement in CT scans of patients with bowel obstruction.

Visual Interpretation of Vocal Fold Paralysis in Flexible Laryngoscopy Using Eye Tracking Technology

OBJECTIVES

Interpretation of laryngoscopy is an important diagnostic skill in otolaryngology. There is, however, limited understanding of the specific visual strategies used while assessing flexible laryngoscopy video. Eye-tracking technology allows for objective study of eye movements during dynamic tasks. The purpose of the present study was to explore visual gaze strategies during laryngoscopy interpretation of unilateral vocal fold paralysis (UVFP) across clinician experience from novice to expert.

METHODS

Thirty individuals were shown five flexible laryngoscopy videos, each 10 seconds long. After viewing each video, participants reported their impressions of "left vocal fold paralysis," "right vocal fold paralysis," or "no vocal fold paralysis." Eye tracking data were collected and analyzed for duration of fixation and number of fixations on select areas of interest (AOI). Diagnostic accuracy and visual gaze patterns were compared between novice, experienced, and expert groups.

RESULTS

Diagnostic accuracy among learners in the novice group was significantly lower than those in the more experienced groups (P = 0.04). All groups demonstrated similar visual gaze patterns when viewing the video with normal bilateral vocal fold mobility, spending the greatest percentage of time viewing the trachea. There were differences among groups when viewing the videos of left or right VFP, but the trachea was always in the top three structures for greatest fixation duration and highest number of fixations.

CONCLUSIONS

Eye-tracking is a novel tool in the setting of laryngoscopy interpretation. With further study it has the potential to be useful for the training of otolaryngology learners to improve diagnostic skills.

Creation and validation of a chest X-ray dataset with eye-tracking and report dictation for AI development

We developed a rich dataset of Chest X-Ray (CXR) images to assist investigators in artificial intelligence. The data were collected using an eye-tracking system while a radiologist reviewed and reported on 1,083 CXR images. The dataset contains the following aligned data: CXR image, transcribed radiology report text, radiologist's dictation audio and eye gaze coordinates data. We hope this dataset can contribute to various areas of research particularly towards explainable and multimodal deep learning/machine learning methods. Furthermore, investigators in disease classification and localization, automated radiology report generation, and human-machine interaction can benefit from these data. We report deep learning experiments that utilize the attention maps produced by the eye gaze dataset to show the potential utility of this dataset.

Contemporary training methods in regional anaesthesia: fundamentals and innovations

Over the past two decades, regional anaesthesia and medical education as a whole have undergone a renaissance. Significant changes in our teaching methods and clinical practice have been influenced by improvements in our theoretical understanding as well as by technological innovations. More recently, there has been a focus on using foundational education principles to teach regional anaesthesia, and the evidence on how to best teach and assess trainees is growing. This narrative review will discuss fundamentals and innovations in regional anaesthesia training. We present the fundamentals in regional anaesthesia training, specifically the current state of simulation-based education, deliberate practice and curriculum design based on competency-based progression. Moving into the future, we present the latest innovations in web-based learning, emerging technologies for teaching and assessment and new developments in alternate reality learning systems.

Computational modeling of human reasoning processes for interpretable visual knowledge: a case study with radiographers

Visual reasoning is critical in many complex visual tasks in medicine such as radiology or pathology. It is challenging to explicitly explain reasoning processes due to the dynamic nature of real-time human cognition. A deeper understanding of such reasoning processes is necessary for improving diagnostic accuracy and computational tools. Most computational analysis methods for visual attention utilize black-box algorithms which lack explainability and are therefore limited in understanding the visual reasoning processes. In this paper, we propose a computational method to quantify and dissect visual reasoning. The method characterizes spatial and temporal features and identifies common and contrast visual reasoning patterns to extract significant gaze activities. The visual reasoning patterns are explainable and can be compared among different groups to discover strategy differences. Experiments with radiographers of varied levels of expertise on 10 levels of visual tasks were conducted. Our empirical observations show that the method can capture the temporal and spatial features of human visual attention and distinguish expertise level. The extracted patterns are further examined and interpreted to showcase key differences between expertise levels in the visual reasoning processes. By revealing task-related reasoning processes, this method demonstrates potential for explaining human visual understanding.

Eye Tracking for Deep Learning Segmentation Using Convolutional Neural Networks

Deep learning with convolutional neural networks (CNNs) has experienced tremendous growth in multiple healthcare applications and has been shown to have high accuracy in semantic segmentation of medical (e.g., radiology and pathology) images. However, a key barrier in the required training of CNNs is obtaining large-scale and precisely annotated imaging data. We sought to address the lack of annotated data with eye tracking technology. As a proof of principle, our hypothesis was that segmentation masks generated with the help of eye tracking (ET) would be very similar to those rendered by hand annotation (HA). Additionally, our goal was to show that a CNN trained on ET masks would be equivalent to one trained on HA masks, the latter being the current standard approach. Step 1: Screen captures of 19 publicly available radiologic images of assorted structures within various modalities were analyzed. ET and HA masks for all regions of interest (ROIs) were generated from these image datasets. Step 2: Utilizing a similar approach, ET and HA masks for 356 publicly available T1-weighted postcontrast meningioma images were generated. Three hundred six of these image + mask pairs were used to train a CNN with U-net-based architecture. The remaining 50 images were used as the independent test set. Step 1: ET and HA masks for the nonneurological images had an average Dice similarity coefficient (DSC) of 0.86 between each other. Step 2: Meningioma ET and HA masks had an average DSC of 0.85 between each other. After separate training using both approaches, the ET approach performed virtually identically to HA on the test set of 50 images. The former had an area under the curve (AUC) of 0.88, while the latter had AUC of 0.87. ET and HA predictions had trimmed mean DSCs compared to the original HA maps of 0.73 and 0.74, respectively. These trimmed DSCs between ET and HA were found to be statistically equivalent with a p value of 0.015. We have demonstrated that ET can create segmentation masks suitable for deep learning semantic segmentation. Future work will integrate ET to produce masks in a faster, more natural manner that distracts less from typical radiology clinical workflow.

Computational framework for fusing eye movements and spoken narratives for image annotation

Despite many recent advances in the field of computer vision, there remains a disconnect between how computers process images and how humans understand them. To begin to bridge this gap, we propose a framework that integrates human-elicited gaze and spoken language to label perceptually important regions in an image. Our work relies on the notion that gaze and spoken narratives can jointly model how humans inspect and analyze images. Using an unsupervised bitext alignment algorithm originally developed for machine translation, we create meaningful mappings between participants' eye movements over an image and their spoken descriptions of that image. The resulting multimodal alignments are then used to annotate image regions with linguistic labels. The accuracy of these labels exceeds that of baseline alignments obtained using purely temporal correspondence between fixations and words. We also find differences in system performances when identifying image regions using clustering methods that rely on gaze information rather than image features. The alignments produced by our framework can be used to create a database of low-level image features and high-level semantic annotations corresponding to perceptually important image regions. The framework can potentially be applied to any multimodal data stream and to any visual domain. To this end, we provide the research community with access to the computational framework.

Association of Electronic Health Record Use With Physician Fatigue and Efficiency

IMPORTANCE

The use of electronic health records (EHRs) is directly associated with physician burnout. An underlying factor associated with burnout may be EHR-related fatigue owing to insufficient user-centered interface design and suboptimal usability.

OBJECTIVE

To examine the association between EHR use and fatigue, as measured by pupillometry, and efficiency, as measured by mouse clicks, time, and number of EHR screens, among intensive care unit (ICU) physicians completing a simulation activity in a prominent EHR.

DESIGN, SETTING, AND PARTICIPANTS

A cross-sectional, simulation-based EHR usability assessment of a leading EHR system was conducted from March 20 to April 5, 2018, among 25 ICU physicians and physician trainees at a southeastern US academic medical center. Participants completed 4 simulation patient cases in the EHR that involved information retrieval and task execution while wearing eye-tracking glasses. Fatigue was quantified through continuous eye pupil data; EHR efficiency was characterized through task completion time, mouse clicks, and EHR screen visits. Data were analyzed from June 1, 2018, to August 31, 2019.

MAIN OUTCOMES AND MEASURES

Primary outcomes were physician fatigue, measured by pupillometry (with lower scores indicating greater fatigue), and EHR efficiency, measured by task completion times, number of mouse clicks, and number of screens visited during EHR simulation.

RESULTS

The 25 ICU physicians (13 women; mean [SD] age, 32.1 [6.1] years) who completed a simulation exercise involving 4 patient cases (mean [SD] completion time, 34:43 [11:41] minutes) recorded a total of 14 hours and 27 minutes of EHR activity. All physician participants experienced physiological fatigue at least once during the exercise, and 20 of 25 participants (80%) experienced physiological fatigue within the first 22 minutes of EHR use. Physicians who experienced EHR-related fatigue in 1 patient case were less efficient in the subsequent patient case, as demonstrated by longer task completion times (r = -0.521; P = .007), higher numbers of mouse clicks (r = -0.562; P = .003), and more EHR screen visits (r = -0.486; P = .01).

CONCLUSIONS AND RELEVANCE

This study reports high rates of fatigue among ICU physicians during short periods of EHR simulation, which were negatively associated with EHR efficiency and included a carryover association across patient cases. More research is needed to investigate the underlying causes of EHR-associated fatigue, to support user-centered EHR design, and to inform safe EHR use policies and guidelines.

Factors affecting perception of the normal pediatric appendix on sonography

Background

To determine if an inherent perception skill along with sonographer experience, knowledge base, scanning time play a role in the identification of the normal appendix in the pediatric population. This is a retrospective review of pediatric (< 18 years old) patients with a clinical suspicion of acute appendicitis presenting to the emergency department of two affiliated academic tertiary care hospitals over a 1-year time span. All patients had a sonogram performed by 1/15 sonographers or by 1/8 on-call radiology residents. Those with a normal or non-visualized appendix with subsequent discharge from ER were included in the study. Patient demographics, minutes spent scanning, and sonographer years of experience in general abdominal scanning and residents level of training were recorded.

Results

Of the 127 patients included in the study, 51 (40%) were male and 76 (60%) were female, with a mean age of 11.8 ± 4.2 years. Sonographers who failed to see a normal appendix had less experience (median 8 years) than those who did visualize the appendix (median 15 years), p ≤ 0.001. Longer time spent scanning was also associated with visualizing a normal appendix (20.4 versus 29.1 min, p = 0.001). In multivariable logistic regression, more time spent scanning (OR 1.04, 95% CI 1.01, 1.07, p = 0.012) and increased sonographer experience (OR 1.07, 95% CI 1.02, 1.13, p = 0.012) resulted in greater odds of perceiving the appendix. The top 4 were significantly more likely to visualize the appendix (88.0%) than all of the other combined (20.8%, p < 0.001), and they also had substantially more experience (median 15 years versus 8 years, p < 0.001). Overall, sonographers were more likely to see a normal appendix (61%) than the residents (14%), p < 0.001.

Conclusion

Sonography to rule out appendicitis in the pediatric patient is in general most successful when performed by experienced sonographers with adequate time to perform the scan. Triaging patients to those sonographers who have displayed optimal perceptual ability of the normal appendix may help optimize patient care and hospital resources. Having experienced sonographers available after hours would allow for optimal care in the setting of ‘query’ appendicitis.

Investigating the effect of clinical history before electrocardiogram interpretation on the visual behavior and interpretation accuracy of clinicians

We examine the impact of the presentation of a patient's clinical history on subsequent visual appraisal and interpretation accuracy of electrocardiograms (ECGs). Healthcare-practitioners (N = 31) skilled in 12-lead ECG interpretation took part in a repeated-measures experiment with counterbalancing viewing 9 ECGs on a computer screen in two separate conditions: with/without an associated patient-history. A Hellinger-distance calculation was applied using a permutation test to eye-movement transitions at two granularity levels: between the ECG leads, and between smaller grid-cells, whose size was determined via data-driven clustering of the fixation points. Findings indicate that presentation of clinical-history does affect accuracy of interpretation in one ECG. Visual-behavior differed as a function of both history presentation and accuracy when considering transitions between the data-driven grid units (using a fine granularity, and able to show attention to parts of the waveform). Differences in visual-behavior at waveform level demonstrate an influence of patient-history and expertise that are not detected at the lead level. Visual-behaviour differs according to whether a patient-history is presented, and whether a clinician provides an accurate interpretation. This difference is evident in how the waveform itself is viewed, and is less present at the coarse granularity of visual transitions between leads. To understand how clinicians interpret ECGs, and potentially other medical images, visual transitions should be considered at a fine level of granularity, determined in a data-driven fashion.

Visual search in breast imaging

Breast cancer is the most common cancer among females worldwide and large volumes of breast images are produced and interpreted annually. As long as radiologists interpret these images, the diagnostic accuracy will be limited by human factors and both false-positive and false-negative errors might occur. By understanding visual search in breast images, we may be able to identify causes of diagnostic errors, find ways to reduce them, and also provide a better education to radiology residents. Many visual search studies in breast radiology have been devoted to mammography. These studies showed that 70% of missed lesions on mammograms attract radiologists' visual attention and that a plethora of different reasons, such as satisfaction of search, incorrect background sampling, and incorrect first impression can cause diagnostic errors in the interpretation of mammograms. Recently, highly accurate tools, which rely on both eye-tracking data and the content of the mammogram, have been proposed to provide feedback to the radiologists. Improving these tools and determining the optimal pathway to integrate them in the radiology workflow could be a possible line of future research. Moreover, in the past few years deep learning has led to improving diagnostic accuracy of computerized diagnostic tools and visual search studies will be required to understand how radiologists interact with the prompts from these tools, and to identify the best way to utilize them. Visual search in other breast imaging modalities, such as breast ultrasound and digital breast tomosynthesis, have so far received less attention, probably due to associated complexities of eye-tracking monitoring and analysing the data. For example, in digital breast tomosynthesis, scrolling through the image results in longer trials, adds a new factor to the study's complexity and makes calculation of gaze parameters more difficult. However, considering the wide utilization of three-dimensional imaging modalities, more visual search studies involving reading stack-view examinations are required in the future. To conclude, in the past few decades visual search studies provided extensive understanding about underlying reasons for diagnostic errors in breast radiology and characterized differences between experts' and novices' visual search patterns. Further visual search studies are required to investigate radiologists' interaction with relatively newer imaging modalities and artificial intelligence tools.

Characterizing Diagnostic Search Patterns in Digital Breast Pathology: Scanners and Drillers

Following a baseline demographic survey, 87 pathologists interpreted 240 digital whole slide images of breast biopsy specimens representing a range of diagnostic categories from benign to atypia, ductal carcinoma in situ, and invasive cancer. A web-based viewer recorded pathologists' behaviors while interpreting a subset of 60 randomly selected and randomly ordered slides. To characterize diagnostic search patterns, we used the viewport location, time stamp, and zoom level data to calculate four variables: average zoom level, maximum zoom level, zoom level variance, and scanning percentage. Two distinct search strategies were confirmed: scanning is characterized by panning at a constant zoom level, while drilling involves zooming in and out at various locations. Statistical analysis was applied to examine the associations of different visual interpretive strategies with pathologist characteristics, diagnostic accuracy, and efficiency. We found that females scanned more than males, and age was positively correlated with scanning percentage, while the facility size was negatively correlated. Throughout 60 cases, the scanning percentage and total interpretation time per slide decreased, and these two variables were positively correlated. The scanning percentage was not predictive of diagnostic accuracy. Increasing average zoom level, maximum zoom level, and zoom variance were correlated with over-interpretation.

A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning

Computer aided diagnosis (CAD) tools help radiologists to reduce diagnostic errors such as missing tumors and misdiagnosis. Vision researchers have been analyzing behaviors of radiologists during screening to understand how and why they miss tumors or misdiagnose. In this regard, eye-trackers have been instrumental in understanding visual search processes of radiologists. However, most relevant studies in this aspect are not compatible with realistic radiology reading rooms. In this study, we aim to develop a paradigm shifting CAD system, called collaborative CAD (C-CAD), that unifies CAD and eye-tracking systems in realistic radiology room settings. We first developed an eye-tracking interface providing radiologists with a real radiology reading room experience. Second, we propose a novel algorithm that unifies eye-tracking data and a CAD system. Specifically, we present a new graph based clustering and sparsification algorithm to transform eye-tracking data (gaze) into a graph model to interpret gaze patterns quantitatively and qualitatively. The proposed C-CAD collaborates with radiologists via eye-tracking technology and helps them to improve their diagnostic decisions. The C-CAD uses radiologists' search efficiency by processing their gaze patterns. Furthermore, the C-CAD incorporates a deep learning algorithm in a newly designed multi-task learning platform to segment and diagnose suspicious areas simultaneously. The proposed C-CAD system has been tested in a lung cancer screening experiment with multiple radiologists, reading low dose chest CTs. Promising results support the efficiency, accuracy and applicability of the proposed C-CAD system in a real radiology room setting. We have also shown that our framework is generalizable to more complex applications such as prostate cancer screening with multi-parametric magnetic resonance imaging (mp-MRI).

Modeling sequential context effects in diagnostic interpretation of screening mammograms

Prior research has shown that physicians’ medical decisions can be influenced by sequential context, particularly in cases where successive stimuli exhibit similar characteristics when analyzing medical images. This type of systematic error is known to psychophysicists as sequential context effect as it indicates that judgments are influenced by features of and decisions about the preceding case in the sequence of examined cases, rather than being based solely on the peculiarities unique to the present case. We determine if radiologists experience some form of context bias, using screening mammography as the use case. To this end, we explore correlations between previous perceptual behavior and diagnostic decisions and current decisions. We hypothesize that a radiologist’s visual search pattern and diagnostic decisions in previous cases are predictive of the radiologist’s current diagnostic decisions. To test our hypothesis, we tasked 10 radiologists of varied experience to conduct blind reviews of 100 four-view screening mammograms. Eye-tracking data and diagnostic decisions were collected from each radiologist under conditions mimicking clinical practice. Perceptual behavior was quantified using the fractal dimension of gaze scanpath, which was computed using the Minkowski–Bouligand box-counting method. To test the effect of previous behavior and decisions, we conducted a multifactor fixed-effects ANOVA. Further, to examine the predictive value of previous perceptual behavior and decisions, we trained and evaluated a predictive model for radiologists’ current diagnostic decisions. ANOVA tests showed that previous visual behavior, characterized by fractal analysis, previous diagnostic decisions, and image characteristics of previous cases are significant predictors of current diagnostic decisions. Additionally, predictive modeling of diagnostic decisions showed an overall improvement in prediction error when the model is trained on additional information about previous perceptual behavior and diagnostic decisions.

Eye-tracking technology in medical education: A systematic review

BACKGROUND

Eye-tracking technology is an established research tool within allied industries such as advertising, psychology and aerospace. This review aims to consolidate literature describing the evidence for use of eye-tracking as an adjunct to traditional teaching methods in medical education.

METHODS

A systematic literature review was conducted in line with STORIES guidelines. A search of EMBASE, OVID MEDLINE, PsycINFO, TRIP database, and Science Direct was conducted until January 2017. Studies describing the use of eye-tracking in the training, assessment, and feedback of clinicians were included in the review.

RESULTS

Thirty-three studies were included in the final qualitative synthesis. Three studies were based on the use of gaze training, three studies on the changes in gaze behavior during the learning curve, 17 studies on clinical assessment and six studies focused on the use of eye-tracking methodology as a feedback tool. The studies demonstrated feasibility and validity in the use of eye-tracking as a training and assessment method.

CONCLUSIONS

Overall, eye-tracking methodology has contributed significantly to the training, assessment, and feedback practices used in the clinical setting. The technology provides reliable quantitative data, which can be interpreted to give an indication of clinical skill, provide training solutions and aid in feedback and reflection. This review provides a detailed summary of evidence relating to eye-tracking methodology and its uses as a training method, changes in visual gaze behavior during the learning curve, eye-tracking methodology for proficiency assessment and its uses as a feedback tool.

Variations in the functional visual field for detection of lung nodules on chest computed tomography: Impact of nodule size, distance, and local lung complexity

PURPOSE

To explore the characteristics that impact lung nodule detection by peripheral vision when searching for lung nodules on chest CT-scans.

METHODS

This study was approved by the local IRB and is HIPAA compliant. A simulated primary (1°) target mass (2 × 2 × 5 cm) was embedded into 5 cm thick subvolumes (SV) extracted from three unenhanced lung MDCT scans (64 row, 1.25 mm thickness, 0.7 mm increment). One of 30 solid, secondary nodules with either 3-4 mm and 5-8 mm diameters were embedded into 192 of 207 SVs. The secondary nodule was placed at a random depth within each SV, a transverse distance of 2.5, 5, 7.5, or 10 mm, and along one of eight rays cast every 45° from the center of the 1° mass. Video recordings of transverse paging in cranio-caudal direction were created for each SV (frame rate three sections/sec). Six radiologists observed each cine-loop once while gaze-tracking hardware assured that gaze was centered on the 1° mass. Each radiologist assigned a confidence rating (0-5) to the detection of a secondary nodule and indicated its location. Detection sensitivity was analyzed relative to secondary nodule size, transverse distance, radial orientation, and lung complexity. Lung complexity was characterized by the number of particles (connected pixels) and the sum of the area of all particles above a -500 HU threshold within regions of interest around the 1° mass and secondary nodule.

RESULTS

Using a proportional odds logistic regression model and eliminating redundant predictors, models fit individually to each reader resulted in the following decreasing order of association based on greatest reduction in Akaike Information Criterion: secondary nodule diameter (6/6 readers, P < 0.001), distance from central mass (6/6 readers, P < 0.001), lung complexity particle count (5/6 readers, P = 0.05), and lung complexity particle area (3/6 readers, P = 0.03). Substantial inter-reader differences in sensitivity to decreasing nodule diameter, distance, and complexity characteristics were observed.

CONCLUSIONS

Of the investigated parameters, secondary nodule size, distance from the gaze center and lung complexity (particle number and area) significantly impact nodule detection with peripheral vision.

iCAP: An Individualized Model Combining Gaze Parameters and Image-Based Features to Predict Radiologists' Decisions While Reading Mammograms

This study introduces an individualized tool for identifying mammogram interpretation errors, called eye-Computer Assisted Perception (iCAP). iCAP consists of two modules, one which processes areas marked by radiologists as suspicious for cancer and classifies these as False Positive (FP) or True Positive (TP) decisions, while the second module classifies fixated but not marked locations as False Negative (FN) or True-Negative (TN) decisions. iCAP relies on both radiologists' gaze-related parameters, extracted from eye tracking data, and image-based features. In order to evaluate iCAP, eye tracking data from eight breast radiologists reading 120 two-view digital mammograms were collected. Fifty-nine cases had biopsy proven cancer. For each radiologist, a user-specific support vector machine model was built to classify the radiologist' s reported areas as TPs or FPs and fixated locations as TNs or FNs. The performances of the classifiers were evaluated by utilizing leave-one-out cross validation. iCAP was tested retrospectively in a simulated scenario in which it was assumed that the radiologists would accept all iCAP decisions. Using iCAP led to an average increase of 12%±6% in the number of correctly localized cancer and an average decrease of 44.5%±22.7% in the number of FPs per image.

Fractal analysis of visual search activity for mass detection during mammographic screening

PURPOSE

The objective of this study was to assess the complexity of human visual search activity during mammographic screening using fractal analysis and to investigate its relationship with case and reader characteristics.

METHODS

The study was performed for the task of mammographic screening with simultaneous viewing of four coordinated breast views as typically done in clinical practice. Eye-tracking data and diagnostic decisions collected for 100 mammographic cases (25 normal, 25 benign, 50 malignant) from 10 readers (three board certified radiologists and seven Radiology residents), formed the corpus for this study. The fractal dimension of the readers' visual scanning pattern was computed with the Minkowski-Bouligand box-counting method and used as a measure of gaze complexity. Individual factor and group-based interaction ANOVA analysis was performed to study the association between fractal dimension, case pathology, breast density, and reader experience level. The consistency of the observed trends depending on gaze data representation was also examined.

RESULTS

Case pathology, breast density, reader experience level, and individual reader differences are all independent predictors of the complexity of visual scanning pattern when screening for breast cancer. No higher order effects were found to be significant.

CONCLUSIONS

Fractal characterization of visual search behavior during mammographic screening is dependent on case properties and image reader characteristics.

Visual Interpretation of Plain Radiographs in Orthopaedics Using Eye-Tracking Technology

BACKGROUND

Despite the importance of radiographic interpretation in orthopaedics, there not a clear understanding of the specific visual strategies used while analyzing a plain film. Eyetracking technology allows for the objective study of eye movements while performing a dynamic task, such as reading X-rays. Our study looks to elucidate objective differences in image interpretation between novice and experienced orthopaedic trainees using this novel technology.

METHODS

Novice and experienced orthopaedic trainees (N=23) were asked to interpret AP pelvis films, searching for unilateral acetabular fractures while eye-movements were assessed for pattern of gaze, fixation on regions of interest, and time of fixation at regions of interest. Participants were asked to label radiographs as "fractured" or "not fractured." If "fractured", the participant was asked to determine the fracture pattern. A control condition employed Ekman faces and participants judged gender and facial emotion. Data were analyzed for variation in eye movements between participants, accuracy of responses, and response time.

RESULTS

Accuracy: There was no significant difference by level of training for accurately identifing fracture images (p=0.3255). There was a significant association between higher level of training and correctly identifying non-fractured images (p=0.0155); greater training was also associated with more success in identifying the correct Judet-Letournel classification (p=0.0029). Response Time: Greater training was associated with faster response times (p=0.0009 for fracture images and 0.0012 for non-fractured images). Fixation Duration: There was no correlation of average fixation duration with experience (p=0.9632). Regions of Interest (ROIs): More experience was associated with an average of two fewer fixated ROIs (p=0.0047). Number of Fixations: Increased experience was associated with fewer fixations overall (p=0.0007).

CONCLUSIONS

Experience has a significant impact on both accuracy and efficiency in interpreting plain films. Greater training is associated with a shift toward a more efficient and thorough assessment of plain radiographs. Eyetracking is a useful descriptive tool in the setting of plain film interpretation.

CLINICAL RELEVANCE

We propose further assessment of eye movements in larger populations of orthopaedic surgeons, including staff orthopaedists. Describing the differences between novice and expert interpretation may provide insight into ways to accelerate the learning process in young orthopaedists.

Tired in the Reading Room: The Influence of Fatigue in Radiology

Commonly conflated with sleepiness, fatigue is a distinct multidimensional condition with physical and mental effects. Fatigue in health care providers and any secondary effects on patient care are an important societal concern. As medical image interpretation is highly dependent on visual input, visual fatigue is of particular interest to radiologists. Humans analyze their surroundings with rapid eye movements called saccades, and fatigue decreases saccadic velocity. Oculomotor parameters may, therefore, be an objective and reproducible metric of fatigue and eye movement analysis can provide valuable insight into the etiology of fatigue-related error.

Does Expectation of Abnormality Affect the Search Pattern of Radiologists When Looking for Pulmonary Nodules?

This experiment investigated whether there might be an effect on the visual search strategy of radiologists during image interpretation of the same adult chest radiographs when given different clinical information. Each of 17 experienced radiologists was asked to interpret a set of 57 (10 abnormal) posteroanterior chest images to identify the presence of pulmonary lesions using differing clinical information (leading to unknown, low and high expectations of prevalence). Eye position metrics (search time, dwell time and time to first fixation) were compared for normal and abnormal images, as well as between conditions. For all images, there was a significantly longer search time at high prevalence expectation compared to low prevalence expectation (W = 75.19, P = <0.0001). Mann-Whitney analysis of the abnormal images demonstrated that the dwell time on correctly identified lesions was significantly shorter at low prevalence expectation compared to both unknown (U = 364.5, P = 0.02) and high prevalence expectation (U = 397.0, P = 0.0002). Visual search patterns of radiologists appear to be affected by changing a priori information where such information fosters an expectation of abnormality.

Feasibility of utilizing a commercial eye tracker to assess electronic health record use during patient simulation

Numerous reports describe unintended consequences of electronic health record implementation. Having previously described physicians’ failures to recognize patient safety issues within our electronic health record simulation environment, we now report on our use of eye and screen-tracking technology to understand factors associated with poor error recognition during an intensive care unit–based electronic health record simulation. We linked performance on the simulation to standard eye and screen-tracking readouts including number of fixations, saccades, mouse clicks and screens visited. In addition, we developed an overall Composite Eye Tracking score which measured when, where and how often each safety item was viewed. For 39 participants, the Composite Eye Tracking score correlated with performance on the simulation (p = 0.004). Overall, the improved performance was associated with a pattern of rapid scanning of data manifested by increased number of screens visited (p = 0.001), mouse clicks (p = 0.03) and saccades (p = 0.004). Eye tracking can be successfully integrated into electronic health record–based simulation and provides a surrogate measure of cognitive decision making and electronic health record usability.

Identifying visual search patterns in eye gaze data; gaining insights into physician visual workflow

Importance and Objectives As health information technologies become more prevalent in physician workflow, it is increasingly important to understand how physicians are using and interacting with these systems. This includes understanding how physicians search for information presented through health information technology systems. Eye tracking technologies provide a useful technique to understand how physicians visually search for information. However, analyzing eye tracking data can be challenging and is often done by measuring summative metrics, such as total time looking at a specific area and first-order transitions.

Methods In this paper, we propose an algorithmic approach to identify different visual search patterns. We demonstrate this approach by identifying common visual search patterns from physicians using a simulated prototype emergency department patient tracking system.

Results and Conclusions We evaluate and compare the visual search pattern results to first-order transition results. We discuss the benefits and limitations of this approach and insights from this initial evaluation.

Decision Making and Variation in Radiation Exposure Factor Selection by Radiologic Technologists

The goal of radiographic imaging is to produce a diagnostically useful image while minimizing patient radiation dose. This study aimed to review variations in exposure factor selection by radiologic technologists for virtual patients with varying body mass index characteristics. Eleven technologists were asked to assign exposure parameters (kVp, mAs, source-to-image receptor distance, and grid use) to 10 computer-generated patient images for each of four radiographic examinations (anteroposterior [AP] shoulder; AP lumbar spine; lateral lumbar spine; AP portable chest). The virtual patients represented five body mass index categories-underweight, healthy weight, overweight, obese, and superobese. As participants assigned exposures, their visual patterns were recorded by a Tobii TX300 eye-tracker. Significant (P < .05) correlation was found between radiographer age/experience and assignment of mAs for AP shoulder and lumbar examinations. Greater age/experience correlated with higher mAs for the AP shoulder examination, but with lower values for lumbar examinations. Strong correlations also existed between times to first fixations on relevant anatomic areas, and kVp/mAs values existed for the AP portable chest examination. Exposure selection differences related to age/experience highlight inconsistencies in the practice of exposure parameter setting. The reason for these inconsistencies requires further investigation, and how to address deficiencies in practice requires consideration to optimize safe patient care. Because of the small sample size used, further research into the relationship between visual factors and individual examinations is suggested, after the findings regarding the AP portable chest examination.

Using High-Fidelity Simulation and Eye Tracking to Characterize EHR Workflow Patterns among Hospital Physicians

Modern EHR systems are complex, and end-user behavior and training are highly variable. The need for clinicians to access key clinical data is a critical patient safety issue. This study used a mixed methods approach employing a high-fidelity EHR simulation environment, eye and screen tracking, surveys, and semi-structured interviews to characterize typical EHR usage by hospital physicians (hospitalists) as they encounter a new patient. The main findings were: 1) There were strong similarities across the groups in the information types the physicians looked at most frequently, 2) While there was no overall difference in case duration between the groups, we observed two distinct workflow types between the groups with respect to gathering information in the EHR and creating a note, and 3) A majority of the case time was devoted to note composition in both groups. This has implications for EHR interface design and raises further questions about what individual user workflows exist in the EHR.

Trends in biomedical informatics: automated topic analysis of JAMIA articles

Biomedical Informatics is a growing interdisciplinary field in which research topics and citation trends have been evolving rapidly in recent years. To analyze these data in a fast, reproducible manner, automation of certain processes is needed. JAMIA is a "generalist" journal for biomedical informatics. Its articles reflect the wide range of topics in informatics. In this study, we retrieved Medical Subject Headings (MeSH) terms and citations of JAMIA articles published between 2009 and 2014. We use tensors (i.e., multidimensional arrays) to represent the interaction among topics, time and citations, and applied tensor decomposition to automate the analysis. The trends represented by tensors were then carefully interpreted and the results were compared with previous findings based on manual topic analysis. A list of most cited JAMIA articles, their topics, and publication trends over recent years is presented. The analyses confirmed previous studies and showed that, from 2012 to 2014, the number of articles related to MeSH terms Methods, Organization & Administration, and Algorithms increased significantly both in number of publications and citations. Citation trends varied widely by topic, with Natural Language Processing having a large number of citations in particular years, and Medical Record Systems, Computerized remaining a very popular topic in all years.

Using computer-extracted image features for modeling of error-making patterns in detection of mammographic masses among radiology residents

PURPOSE

Mammography is the most widely accepted and utilized screening modality for early breast cancer detection. Providing high quality mammography education to radiology trainees is essential, since excellent interpretation skills are needed to ensure the highest benefit of screening mammography for patients. The authors have previously proposed a computer-aided education system based on trainee models. Those models relate human-assessed image characteristics to trainee error. In this study, the authors propose to build trainee models that utilize features automatically extracted from images using computer vision algorithms to predict likelihood of missing each mass by the trainee. This computer vision-based approach to trainee modeling will allow for automatically searching large databases of mammograms in order to identify challenging cases for each trainee.

METHODS

The authors' algorithm for predicting the likelihood of missing a mass consists of three steps. First, a mammogram is segmented into air, pectoral muscle, fatty tissue, dense tissue, and mass using automated segmentation algorithms. Second, 43 features are extracted using computer vision algorithms for each abnormality identified by experts. Third, error-making models (classifiers) are applied to predict the likelihood of trainees missing the abnormality based on the extracted features. The models are developed individually for each trainee using his/her previous reading data. The authors evaluated the predictive performance of the proposed algorithm using data from a reader study in which 10 subjects (7 residents and 3 novices) and 3 experts read 100 mammographic cases. Receiver operating characteristic (ROC) methodology was applied for the evaluation.

RESULTS

The average area under the ROC curve (AUC) of the error-making models for the task of predicting which masses will be detected and which will be missed was 0.607 (95% CI,0.564-0.650). This value was statistically significantly different from 0.5 (p<0.0001). For the 7 residents only, the AUC performance of the models was 0.590 (95% CI,0.537-0.642) and was also significantly higher than 0.5 (p=0.0009). Therefore, generally the authors' models were able to predict which masses were detected and which were missed better than chance.

CONCLUSIONS

The authors proposed an algorithm that was able to predict which masses will be detected and which will be missed by each individual trainee. This confirms existence of error-making patterns in the detection of masses among radiology trainees. Furthermore, the proposed methodology will allow for the optimized selection of difficult cases for the trainees in an automatic and efficient manner.

Modeling false positive error making patterns in radiology trainees for improved mammography education

INTRODUCTION

While mammography notably contributes to earlier detection of breast cancer, it has its limitations, including a large number of false positive exams. Improved radiology education could potentially contribute to alleviating this issue. Toward this goal, in this paper we propose an algorithm for modeling of false positive error making among radiology trainees. Identifying troublesome locations for the trainees could focus their training and in turn improve their performance.

METHODS

The algorithm proposed in this paper predicts locations that are likely to result in a false positive error for each trainee based on the previous annotations made by the trainee. The algorithm consists of three steps. First, the suspicious false positive locations are identified in mammograms by Difference of Gaussian filter and suspicious regions are segmented by computer vision-based segmentation algorithms. Second, 133 features are extracted for each suspicious region to describe its distinctive characteristics. Third, a random forest classifier is applied to predict the likelihood of the trainee making a false positive error using the extracted features. The random forest classifier is trained using previous annotations made by the trainee. We evaluated the algorithm using data from a reader study in which 3 experts and 10 trainees interpreted 100 mammographic cases.

RESULTS

The algorithm was able to identify locations where the trainee will commit a false positive error with accuracy higher than an algorithm that selects such locations randomly. Specifically, our algorithm found false positive locations with 40% accuracy when only 1 location was selected for all cases for each trainee and 12% accuracy when 10 locations were selected. The accuracies for randomly identified locations were both 0% for these two scenarios.

CONCLUSIONS

In this first study on the topic, we were able to build computer models that were able to find locations for which a trainee will make a false positive error in images that were not previously seen by the trainee. Presenting the trainees with such locations rather than randomly selected ones may improve their educational outcomes.

Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: elucidation with eye tracking

PURPOSE

To determine the effectiveness of radiologists' search, recognition, and acceptance of lung nodules on computed tomographic (CT) images by using eye tracking.

MATERIALS AND METHODS

This study was performed with a protocol approved by the institutional review board. All study subjects provided informed consent, and all private health information was protected in accordance with HIPAA. A remote eye tracker was used to record time-varying gaze paths while 13 radiologists interpreted 40 lung CT images with an average of 3.9 synthetic nodules (5-mm diameter) embedded randomly in the lung parenchyma. The radiologists' gaze volumes ( GV gaze volume s) were defined as the portion of the lung parenchyma within 50 pixels (approximately 3 cm) of all gaze points. The fraction of the total lung volume encompassed within the GV gaze volume s, the fraction of lung nodules encompassed within each GV gaze volume (search effectiveness), the fraction of lung nodules within the GV gaze volume detected by the reader (recognition-acceptance effectiveness), and overall sensitivity of lung nodule detection were measured.

RESULTS

Detected nodules were within 50 pixels of the nearest gaze point for 990 of 992 correct detections. On average, radiologists searched 26.7% of the lung parenchyma in 3 minutes and 16 seconds and encompassed between 86 and 143 of 157 nodules within their GV gaze volume s. Once encompassed within their GV gaze volume , the average sensitivity of nodule recognition and acceptance ranged from 47 of 100 nodules to 103 of 124 nodules (sensitivity, 0.47-0.82). Overall sensitivity ranged from 47 to 114 of 157 nodules (sensitivity, 0.30-0.73) and showed moderate correlation (r = 0.62, P = .02) with the fraction of lung volume searched.

CONCLUSION

Relationships between reader search, recognition and acceptance, and overall lung nodule detection rate can be studied with eye tracking. Radiologists appear to actively search less than half of the lung parenchyma, with substantial interreader variation in volume searched, fraction of nodules included within the search volume, sensitivity for nodules within the search volume, and overall detection rate.

Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT

We describe a framework to model visual semantics of liver lesions in CT images in order to predict the visual semantic terms (VST) reported by radiologists in describing these lesions. Computational models of VST are learned from image data using linear combinations of high-order steerable Riesz wavelets and support vector machines (SVM). In a first step, these models are used to predict the presence of each semantic term that describes liver lesions. In a second step, the distances between all VST models are calculated to establish a nonhierarchical computationally-derived ontology of VST containing inter-term synonymy and complementarity. A preliminary evaluation of the proposed framework was carried out using 74 liver lesions annotated with a set of 18 VSTs from the RadLex ontology. A leave-one-patient-out cross-validation resulted in an average area under the ROC curve of 0.853 for predicting the presence of each VST. The proposed framework is expected to foster human-computer synergies for the interpretation of radiological images while using rotation-covariant computational models of VSTs to 1) quantify their local likelihood and 2) explicitly link them with pixel-based image content in the context of a given imaging domain.

Radiology resident mammography training: interpretation difficulty and error-making patterns

RATIONALE AND OBJECTIVES

The purpose of this study was to better understand the concept of mammography difficulty and how it affects radiology resident performance.

MATERIALS AND METHODS

Seven radiology residents and three expert breast imagers reviewed 100 mammograms, consisting of bilateral medial lateral oblique and craniocaudal views, using a research workstation. The cases consisted of normal, benign, and malignant findings. Participants identified abnormalities and scored the difficulty and malignant potential for each case. Resident performance (sensitivity, specificity, and area under the receiver operating characteristic curve [AUC]) was calculated for self- and expert-assessed high and low difficulties.

RESULTS

For cases classified by self-assessed difficulty, the resident AUCs were 0.667 for high difficulty and 0.771 for low difficulty cases (P = .010). Resident sensitivities were 0.707 for high and 0.614 for low difficulty cases (P = .113). Resident specificities were 0.583 for high and 0.905 for low difficulty cases (P < .001). For cases classified by expert-assessed difficulty, the resident AUCs were 0.583 for high and 0.783 for low difficulty cases (P = .001). Resident sensitivities were 0.558 for high and 0.796 for low difficulty cases (P < .001). Resident specificities were 0.714 for high and 0.740 for low difficulty cases (P = .807).

CONCLUSIONS

Increased self- and expert-assessed difficulty is associated with a decrease in resident performance in mammography. However, while this lower performance is due to a decrease in specificity for self-assessed difficulty, it is due to a decrease in sensitivity for expert-assessed difficulty. These trends suggest that educators should provide a mix of self- and expert-assessed difficult cases in educational materials to maximize the effect of training on resident performance and confidence.