Eye-tracking of nodule detection in lung CT volumetric data

Different Evaluation Strategies of Oncological CT Examinations with Regard to Professional Experience: A Clinical Study Using Eye-tracking

Contrast-enhanced CT is the standard imaging technique in oncological objectives. Rates of missed pathologies depend on work experience of the respective radiologists. Thus the aim of this study is to analyze the eye movements of professionals while reading CT images in order to evaluate whether the eye-fixation patterns and search strategies of experienced radiologists could explain higher detection rates of pathologies and whether such patterns can be learned.Anonymized images of 10 patients were presented to three medical students and six radiologists with different levels of work experience. During image analysis, ocular fixation positions were recorded using an eye-tracking software tool. The CT scans were analyzed retrospectively, considering the individual course of disease with the issue of successful detection of all pathologies. Visual attention and dwell time of ocular fixation on clinically important abnormalities or areas with pathological findings, general search patterns, and time efficiency were assessed. For statistical analysis, interobserver variability and accuracy of lesion detection were evaluated taking into account individual experience.The results revealed that observer sensitivity depends on work experience due to a more systematic order of inspection and a well-known course of disease, e.g. in case of metastatic spread. The areas of missed pathologies mostly included secondary findings. Inexperienced readers changed the stratification considerably more often and required more time for reporting or detecting pathologies.Our results suggest that experienced radiological physicians reduce their amount of missed findings by looking more systematically at images and by applying a more targeted inspection of clinically important regions. · CT interpretation by radiology residents is faster and less error-prone compared to postgraduate residents. · systematic image analysis is trainable. · engrams tend to be acquired through experience. · Kluge S, Schülke C, Ites HC et al. Different Evaluation Strategies of Oncological CT Examinations with Regard to Professional Experience: A Clinical Study Using Eye-tracking. Fortschr Röntgenstr 2024; DOI 10.1055/a-2452-2180.

Impact of a digital training platform and tailored education on the chest image interpretation performance of healthcare professionals

INTRODUCTION

With the use of expert consensus a digital training tool was developed which proved useful when teaching radiographers how to interpret chest images. The training tool included A) a search strategy and B) an educational video programme to communicate the search strategies using eye tracking technology.

METHODS

A multi-reader multi-case study was undertaken to assess the effectiveness of a training tool and study day. The interventions were designed to cover a range of potential pathological presentations. Participants, physiotherapists and nurse practitioners working at a cardiothoracic Intensive Care Unit (ICU), were asked to interpret 20 chest images at the beginning of the study and following access to each intervention. Participants received access to the training tool at different times for a period of 4-6 weeks. A study day was then be provided to all participants and interpretations of a different dataset were completed by all. Each participant was asked to complete a questionnaire to gain perceptions of the training provided.

RESULTS

Twenty-eight participants interpreted a total of 1680 chest radiographs. Improvements in specificity were noted across the participants. Sensitivity fell in both groups following both training interventions.

CONCLUSION

Face to face learning and digital components are potentially useful in professional development and revision in chest x-ray interpretation for non-medical healthcare professionals working in an ICU setting.

IMPLICATIONS FOR PRACTICE

The training tool and study day may be useful as image interpretation revision aids or to accompany formal methods of education.

Visual Image Annotation for Bowel Obstruction: Repeatability and Agreement with Manual Annotation and Neural Networks

Bowel obstruction is a common cause of acute abdominal pain. The development of algorithms for automated detection and characterization of bowel obstruction on CT has been limited by the effort required for manual annotation. Visual image annotation with an eye tracking device may mitigate that limitation. The purpose of this study is to assess the agreement between visual and manual annotations for bowel segmentation and diameter measurement, and to assess agreement with convolutional neural networks (CNNs) trained using that data. Sixty CT scans of 50 patients with bowel obstruction from March to June 2022 were retrospectively included and partitioned into training and test data sets. An eye tracking device was used to record 3-dimensional coordinates within the scans, while a radiologist cast their gaze at the centerline of the bowel, and adjusted the size of a superimposed ROI to approximate the diameter of the bowel. For each scan, 59.4 ± 15.1 segments, 847.9 ± 228.1 gaze locations, and 5.8 ± 1.2 m of bowel were recorded. 2d and 3d CNNs were trained using this data to predict bowel segmentation and diameter maps from the CT scans. For comparisons between two repetitions of visual annotation, CNN predictions, and manual annotations, Dice scores for bowel segmentation ranged from 0.69 ± 0.17 to 0.81 ± 0.04 and intraclass correlations [95% CI] for diameter measurement ranged from 0.672 [0.490-0.782] to 0.940 [0.933-0.947]. Thus, visual image annotation is a promising technique for training CNNs to perform bowel segmentation and diameter measurement in CT scans of patients with bowel obstruction.

Characteristics of expert search behavior in volumetric medical image interpretation

Purpose: Experienced radiologists have enhanced global processing ability relative to novices, allowing experts to rapidly detect medical abnormalities without performing an exhaustive search. However, evidence for global processing models is primarily limited to two-dimensional image interpretation, and it is unclear whether these findings generalize to volumetric images, which are widely used in clinical practice. We examined whether radiologists searching volumetric images use methods consistent with global processing models of expertise. In addition, we investigated whether search strategy (scanning/drilling) differs with experience level. Approach: Fifty radiologists with a wide range of experience evaluated chest computed-tomography scans for lung nodules while their eye movements and scrolling behaviors were tracked. Multiple linear regressions were used to determine: (1) how search behaviors differed with years of experience and the number of chest CTs evaluated per week and (2) which search behaviors predicted better performance. Results: Contrary to global processing models based on 2D images, experience was unrelated to measures of global processing (saccadic amplitude, coverage, time to first fixation, search time, and depth passes) in this task. Drilling behavior was associated with better accuracy than scanning behavior when controlling for observer experience. Greater image coverage was a strong predictor of task accuracy. Conclusions: Global processing ability may play a relatively small role in volumetric image interpretation, where global scene statistics are not available to radiologists in a single glance. Rather, in volumetric images, it may be more important to engage in search strategies that support a more thorough search of the image.

The cognitive and perceptual processes that affect observer performance in lung cancer detection: a scoping review

INTRODUCTION

Early detection of malignant pulmonary nodules through screening has been shown to reduce lung cancer-related mortality by 20%. However, perceptual and cognitive factors that affect nodule detection are poorly understood. This review examines the cognitive and visual processes of various observers, with a particular focus on radiologists, during lung nodule detection.

METHODS

Four databases (Medline, Embase, Scopus and PubMed) were searched to extract studies on eye-tracking in pulmonary nodule detection. Studies were included if they used eye-tracking to assess the search and detection of lung nodules in computed tomography or 2D radiographic imaging. Data were charted according to identified themes and synthesised using a thematic narrative approach.

RESULTS

The literature search yielded 25 articles and five themes were discovered: 1 - functional visual field and satisfaction of search, 2 - expert search patterns, 3 - error classification through dwell time, 4 - the impact of the viewing environment and 5 - the effect of prevalence expectation on search. Functional visual field reduced to 2.7° in 3D imaging compared to 5° in 2D radiographs. Although greater visual coverage improved nodule detection, incomplete search was not responsible for missed nodules. Most radiological errors during lung nodule detection were decision-making errors (30%-45%). Dwell times associated with false-positive (FP) decisions informed feedback systems to improve diagnosis. Interruptions did not influence diagnostic performance; however, it increased viewing time by 8% and produced a 23.1% search continuation accuracy. Comparative scanning was found to increase the detection of low contrast nodules. Prevalence expectation did not directly affect diagnostic accuracy; however, decision-making time increased by 2.32 seconds with high prevalence expectations.

CONCLUSION

Visual and cognitive factors influence pulmonary nodule detection. Insights gained from eye-tracking can inform advancements in lung screening. Further exploration of eye-tracking in lung screening, particularly with low-dose computed tomography (LDCT), will benefit the future of lung cancer screening.

Automatic Lung Nodule Detection Combined With Gaze Information Improves Radiologists' Screening Performance

Early diagnosis of lung cancer via computed tomography can significantly reduce the morbidity and mortality rates associated with the pathology. However, searching lung nodules is a high complexity task, which affects the success of screening programs. Whilst computer-aided detection systems can be used as second observers, they may bias radiologists and introduce significant time overheads. With this in mind, this study assesses the potential of using gaze information for integrating automatic detection systems in the clinical practice. For that purpose, 4 radiologists were asked to annotate 20 scans from a public dataset while being monitored by an eye tracker device, and an automatic lung nodule detection system was developed. Our results show that radiologists follow a similar search routine and tend to have lower fixation periods in regions where finding errors occur. The overall detection sensitivity of the specialists was 0.67±0.07, whereas the system achieved 0.69. Combining the annotations of one radiologist with the automatic system significantly improves the detection performance to similar levels of two annotators. Filtering automatic detection candidates only for low fixation regions still significantly improves the detection sensitivity without increasing the number of false-positives.

What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures

Interpretation of volumetric medical images represents a rapidly growing proportion of the workload in radiology. However, relatively little is known about the strategies that best guide search behavior when looking for abnormalities in volumetric images. Although there is extensive literature on two-dimensional medical image perception, it is an open question whether the conclusions drawn from these images can be generalized to volumetric images. Importantly, volumetric images have distinct characteristics (e.g., scrolling through depth, smooth-pursuit eye-movements, motion onset cues, etc.) that should be considered in future research. In this manuscript, we will review the literature on medical image perception and discuss relevant findings from basic science that can be used to generate predictions about expertise in volumetric image interpretation. By better understanding search through volumetric images, we may be able to identify common sources of error, characterize the optimal strategies for searching through depth, or develop new training and assessment techniques for radiology residents.

Development of competence in volumetric image interpretation in radiology residents

BACKGROUND

During residency, radiology residents learn to interpret volumetric radiological images. The development of their competence for volumetric image interpretation, as opposed to 2D image reading, is not completely understood. The purpose of the present study was to investigate how competence for volumetric image interpretation develops in radiology residents and how this compares with competence development for 2D image interpretation, by studying resident scores on image-based items in digital radiology tests.

METHODS

We reviewed resident scores on volumetric and 2D image-based test items in 9 consecutive semi-annual digital radiology tests that were carried out from November 2013 to April 2018. We assessed percentage-correct sum scores for all test items about volumetric images and for all test items about 2D images in each test as well as for all residents across the 9 tests (i.e. 4.5 years of test materials). We used a paired t-test to analyze whether scores differed between volumetric and 2D image-based test items in individual residents in postgraduate year (PGY) 0-5, subdivided in 10 half-year phases (PGY 0-0.5, 0.5-1.0, 1.0-1.5 et cetera).

RESULTS

The percentage-correct scores on volumetric and 2D image-based items showed a comparable trend of development, increasing in the first half of residency and flattening off in the second half. Chance-corrected scores were generally lower in volumetric than in 2D items (on average 1-5% points). In PGY 1.5-4.5, this score difference was statistically significant (p-values ranging from 0.02 to < 0.001), with the largest difference found in PGY 2.5 (mean: 5% points; 95% CI: -7.3 - -3.4). At the end of training in PGY 5, there was no statistically significant score difference between both item types.

CONCLUSIONS

The development of competence in volumetric image interpretation fits a similar curvilinear growth curve during radiology residency as 2D image interpretation competence in digital radiology tests. Although residents performed significantly lower on volumetric than 2D items in PGY 1.5-4.5, we consider the magnitude of this difference as relatively small for our educational setting and we suggest that throughout radiology training there are no relevant differences in the development of both types of competences, as investigated by digital radiology tests.

Local complexity metrics to quantify the effect of anatomical noise on detectability of lung nodules in chest CT imaging

The purpose of this study is to (1) develop metrics to characterize the regional anatomical complexity of the lungs, and (2) relate these metrics with lung nodule detection in chest CT. A free-scrolling reader-study with virtually inserted nodules (13 radiologists × 157 total nodules = 2041 responses) is used to characterize human detection performance. Metrics of complexity based on the local density and orientation of distracting vasculature are developed for two-dimensional (2-D) and three-dimensional (3-D) considerations of the image volume. Assessed characteristics included the distribution of 2-D/3-D vessel structures of differing orientation (dubbed "2-D/3-D and dot-like/line-like distractor indices"), contiguity of inserted nodules with local vasculature, mean local gray-level surrounding each nodule, the proportion of lung voxels to total voxels in each section, and 3-D distance of each nodule from the trachea bifurcation. A generalized linear mixed-effects statistical model is used to determine the influence of each these metrics on nodule detectability. In order of decreasing effect size: 3-D line-like distractor index, 2-D line-like distractor index, 2-D dot-like distractor index, local mean gray-level, contiguity with 2-D dots, lung area, and contiguity with 3-D lines all significantly affect detectability ( P < 0.05 ). These data demonstrate that local lung complexity degrades detection of lung nodules.

How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology

Eye tracking research has been conducted for decades to gain understanding of visual diagnosis such as in radiology. For educational purposes, it is important to identify visual search patterns that are related to high perceptual performance and to identify effective teaching strategies. This review of eye-tracking literature in the radiology domain aims to identify visual search patterns associated with high perceptual performance. Databases PubMed, EMBASE, ERIC, PsycINFO, Scopus and Web of Science were searched using 'visual perception' OR 'eye tracking' AND 'radiology' and synonyms. Two authors independently screened search results and included eye tracking studies concerning visual skills in radiology published between January 1, 1994 and July 31, 2015. Two authors independently assessed study quality with the Medical Education Research Study Quality Instrument, and extracted study data with respect to design, participant and task characteristics, and variables. A thematic analysis was conducted to extract and arrange study results, and a textual narrative synthesis was applied for data integration and interpretation. The search resulted in 22 relevant full-text articles. Thematic analysis resulted in six themes that informed the relation between visual search and level of expertise: (1) time on task, (2) eye movement characteristics of experts, (3) differences in visual attention, (4) visual search patterns, (5) search patterns in cross sectional stack imaging, and (6) teaching visual search strategies. Expert search was found to be characterized by a global-focal search pattern, which represents an initial global impression, followed by a detailed, focal search-to-find mode. Specific task-related search patterns, like drilling through CT scans and systematic search in chest X-rays, were found to be related to high expert levels. One study investigated teaching of visual search strategies, and did not find a significant effect on perceptual performance. Eye tracking literature in radiology indicates several search patterns are related to high levels of expertise, but teaching novices to search as an expert may not be effective. Experimental research is needed to find out which search strategies can improve image perception in learners.

Foveated Model Observers to predict human performance in 3D images

We evaluate 3D search requires model observers that take into account the peripheral human visual processing (foveated models) to predict human observer performance. We show that two different 3D tasks, free search and location-known detection, influence the relative human visual detectability of two signals of different sizes in synthetic backgrounds mimicking the noise found in 3D digital breast tomosynthesis. One of the signals resembled a microcalcification (a small and bright sphere), while the other one was designed to look like a mass (a larger Gaussian blob). We evaluated current standard models observers (Hotelling; Channelized Hotelling; non-prewhitening matched filter with eye filter, NPWE; and non-prewhitening matched filter model, NPW) and showed that they incorrectly predict the relative detectability of the two signals in 3D search. We propose a new model observer (3D Foveated Channelized Hotelling Observer) that incorporates the properties of the visual system over a large visual field (fovea and periphery). We show that the foveated model observer can accurately predict the rank order of detectability of the signals in 3D images for each task. Together, these results motivate the use of a new generation of foveated model observers for predicting image quality for search tasks in 3D imaging modalities such as digital breast tomosynthesis or computed tomography.

Visual-search observers for assessing tomographic x-ray image quality

PURPOSE

Mathematical model observers commonly used for diagnostic image-quality assessments in x-ray imaging research are generally constrained to relatively simple detection tasks due to their need for statistical prior information. Visual-search (VS) model observers that employ morphological features in sequential search and analysis stages have less need for such information and fewer task constraints. The authors compared four VS observers against human observers and an existing scanning model observer in a pilot study that quantified how mass detection and localization in simulated digital breast tomosynthesis (DBT) can be affected by the number P of acquired projections.

METHODS

Digital breast phantoms with embedded spherical masses provided single-target cases for a localization receiver operating characteristic (LROC) study. DBT projection sets based on an acquisition arc of 60° were generated for values of P between 3 and 51. DBT volumes were reconstructed using filtered backprojection with a constant 3D Butterworth postfilter; extracted 2D slices were used as test images. Three imaging physicists participated as observers. A scanning channelized nonprewhitening (CNPW) observer had knowledge of the mean lesion-absent images. The VS observers computed an initial single-feature search statistic that identified candidate locations as local maxima of either a template matched-filter (MF) image or a gradient-template MF (GMF) image. Search inefficiencies that modified the statistic were also considered. Subsequent VS candidate analyses were carried out with (i) the CNPW statistical discriminant and (ii) the discriminant computed from GMF training images. These location-invariant discriminants did not utilize covariance information. All observers read 36 training images and 108 study images per P value. Performance was scored in terms of area under the LROC curve.

RESULTS

Average human-observer performance was stable for P between 7 and 35. In the absence of search inefficiencies, the VS models based on the GMF analysis provided the best correlation (Pearson ρ ≥ 0.62) with the human results. The CNPW-based VS observers deviated from the humans primarily at lower values of P. In this limited study, search inefficiencies allowed for good quantitative agreement with the humans for most of the VS observers.

CONCLUSIONS

The computationally efficient training requirements for the VS observer are suitable for high-resolution imaging, indicating that the observer framework has the potential to overcome important task limitations of current model observers for x-ray applications.

Radiology resident MR and CT image analysis skill assessment using an interactive volumetric simulation tool - the RadioLOG project

OBJECTIVE

Assess the use of a volumetric simulation tool for the evaluation of radiology resident MR and CT interpretation skills.

MATERIAL AND METHODS

Forty-three participants were evaluated with a software allowing the visualisation of multiple volumetric image series. There were 7 medical students, 28 residents and 8 senior radiologists among the participants. Residents were divided into two sub-groups (novice and advanced). The test was composed of 15 exercises on general radiology and lasted 45 min. Participants answered a questionnaire on their experience with the test using a 5-point Likert scale. This study was approved by the dean of the medical school and did not require ethics committee approval.

RESULTS

The reliability of the test was good with a Cronbach alpha value of 0.9. Test scores were significantly different in all sub-groups studies (p < 0.0225). The relation between test scores and the year of residency was logarithmic (R² = 0.974). Participants agreed that the test reflected their radiological practice (3.9 ± 0.9 on a 5-point scale) and was better than the conventional evaluation methods (4.6 ± 0.5 on a 5-point scale).

CONCLUSION

This software provides a high quality evaluation tool for the assessment of the interpretation skills in radiology residents.

KEY POINTS

• This tool allows volumetric image analysis of MR and CT studies. • A high reliability test could be created with this tool. • Test scores were strongly associated with the examinee expertise level. • Examinees positively evaluated the authenticity and usability of this tool.