Accuracy is in the eyes of the pathologist: The visual interpretive process and diagnostic accuracy with digital whole slide images

Eye tracking in digital pathology: A comprehensive literature review

Eye tracking has been used for decades in attempt to understand the cognitive processes of individuals. From memory access to problem-solving to decision-making, such insight has the potential to improve workflows and the education of students to become experts in relevant fields. Until recently, the traditional use of microscopes in pathology made eye tracking exceptionally difficult. However, the digital revolution of pathology from conventional microscopes to digital whole slide images allows for new research to be conducted and information to be learned with regards to pathologist visual search patterns and learning experiences. This has the promise to make pathology education more efficient and engaging, ultimately creating stronger and more proficient generations of pathologists to come. The goal of this review on eye tracking in pathology is to characterize and compare the visual search patterns of pathologists. The PubMed and Web of Science databases were searched using 'pathology' AND 'eye tracking' synonyms. A total of 22 relevant full-text articles published up to and including 2023 were identified and included in this review. Thematic analysis was conducted to organize each study into one or more of the 10 themes identified to characterize the visual search patterns of pathologists: (1) effect of experience, (2) fixations, (3) zooming, (4) panning, (5) saccades, (6) pupil diameter, (7) interpretation time, (8) strategies, (9) machine learning, and (10) education. Expert pathologists were found to have higher diagnostic accuracy, fewer fixations, and shorter interpretation times than pathologists with less experience. Further, literature on eye tracking in pathology indicates that there are several visual strategies for diagnostic interpretation of digital pathology images, but no evidence of a superior strategy exists. The educational implications of eye tracking in pathology have also been explored but the effect of teaching novices how to search as an expert remains unclear. In this article, the main challenges and prospects of eye tracking in pathology are briefly discussed along with their implications to the field.

Machine learning classification of diagnostic accuracy in pathologists interpreting breast biopsies

OBJECTIVE

This study explores the feasibility of using machine learning to predict accurate versus inaccurate diagnoses made by pathologists based on their spatiotemporal viewing behavior when evaluating digital breast biopsy images.

MATERIALS AND METHODS

The study gathered data from 140 pathologists of varying experience levels who each reviewed a set of 14 digital whole slide images of breast biopsy tissue. Pathologists' viewing behavior, including zooming and panning actions, was recorded during image evaluation. A total of 30 features were extracted from the viewing behavior data, and 4 machine learning algorithms were used to build classifiers for predicting diagnostic accuracy.

RESULTS

The Random Forest classifier demonstrated the best overall performance, achieving a test accuracy of 0.81 and area under the receiver-operator characteristic curve of 0.86. Features related to attention distribution and focus on critical regions of interest were found to be important predictors of diagnostic accuracy. Further including case-level and pathologist-level information incrementally improved classifier performance.

DISCUSSION

Results suggest that pathologists' viewing behavior during digital image evaluation can be leveraged to predict diagnostic accuracy, affording automated feedback and decision support systems based on viewing behavior to aid in training and, ultimately, clinical practice. They also carry implications for basic research examining the interplay between perception, thought, and action in diagnostic decision-making.

CONCLUSION

The classifiers developed herein have potential applications in training and clinical settings to provide timely feedback and support to pathologists during diagnostic decision-making. Further research could explore the generalizability of these findings to other medical domains and varied levels of expertise.

Inflamed immune phenotype predicts favorable clinical outcomes of immune checkpoint inhibitor therapy across multiple cancer types

BACKGROUND

The inflamed immune phenotype (IIP), defined by enrichment of tumor-infiltrating lymphocytes (TILs) within intratumoral areas, is a promising tumor-agnostic biomarker of response to immune checkpoint inhibitor (ICI) therapy. However, it is challenging to define the IIP in an objective and reproducible manner during manual histopathologic examination. Here, we investigate artificial intelligence (AI)-based immune phenotypes capable of predicting ICI clinical outcomes in multiple solid tumor types.

METHODS

Lunit SCOPE IO is a deep learning model which determines the immune phenotype of the tumor microenvironment based on TIL analysis. We evaluated the correlation between the IIP and ICI treatment outcomes in terms of objective response rates (ORR), progression-free survival (PFS), and overall survival (OS) in a cohort of 1,806 ICI-treated patients representing over 27 solid tumor types retrospectively collected from multiple institutions.

RESULTS

We observed an overall IIP prevalence of 35.2% and significantly more favorable ORRs (26.3% vs 15.8%), PFS (median 5.3 vs 3.1 months, HR 0.68, 95% CI 0.61 to 0.76), and OS (median 25.3 vs 13.6 months, HR 0.66, 95% CI 0.57 to 0.75) after ICI therapy in IIP compared with non-IIP patients, respectively (p<0.001 for all comparisons). On subgroup analysis, the IIP was generally prognostic of favorable PFS across major patient subgroups, with the exception of the microsatellite unstable/mismatch repair deficient subgroup.

CONCLUSION

The AI-based IIP may represent a practical, affordable, clinically actionable, and tumor-agnostic biomarker prognostic of ICI therapy response across diverse tumor types.

State-of-the-Art of Breast Cancer Diagnosis in Medical Images via Convolutional Neural Networks (CNNs)

Early detection of breast cancer is crucial for a better prognosis. Various studies have been conducted where tumor lesions are detected and localized on images. This is a narrative review where the studies reviewed are related to five different image modalities: histopathological, mammogram, magnetic resonance imaging (MRI), ultrasound, and computed tomography (CT) images, making it different from other review studies where fewer image modalities are reviewed. The goal is to have the necessary information, such as pre-processing techniques and CNN-based diagnosis techniques for the five modalities, readily available in one place for future studies. Each modality has pros and cons, such as mammograms might give a high false positive rate for radiographically dense breasts, while ultrasounds with low soft tissue contrast result in early-stage false detection, and MRI provides a three-dimensional volumetric image, but it is expensive and cannot be used as a routine test. Various studies were manually reviewed using particular inclusion and exclusion criteria; as a result, 91 recent studies that classify and detect tumor lesions on breast cancer images from 2017 to 2022 related to the five image modalities were included. For histopathological images, the maximum accuracy achieved was around 99 % , and the maximum sensitivity achieved was 97.29 % by using DenseNet, ResNet34, and ResNet50 architecture. For mammogram images, the maximum accuracy achieved was 96.52 % using a customized CNN architecture. For MRI, the maximum accuracy achieved was 98.33 % using customized CNN architecture. For ultrasound, the maximum accuracy achieved was around 99 % by using DarkNet-53, ResNet-50, G-CNN, and VGG. For CT, the maximum sensitivity achieved was 96 % by using Xception architecture. Histopathological and ultrasound images achieved higher accuracy of around 99 % by using ResNet34, ResNet50, DarkNet-53, G-CNN, and VGG compared to other modalities for either of the following reasons: use of pre-trained architectures with pre-processing techniques, use of modified architectures with pre-processing techniques, use of two-stage CNN, and higher number of studies available for Artificial Intelligence (AI)/machine learning (ML) researchers to reference. One of the gaps we found is that only a single image modality is used for CNN-based diagnosis; in the future, a multiple image modality approach can be used to design a CNN architecture with higher accuracy.

Assessment of Primary Human Liver Cancer Cells by Artificial Intelligence-Assisted Raman Spectroscopy

We investigated the possibility of using Raman spectroscopy assisted by artificial intelligence methods to identify liver cancer cells and distinguish them from their Non-Tumor counterpart. To this aim, primary liver cells (40 Tumor and 40 Non-Tumor cells) obtained from resected hepatocellular carcinoma (HCC) tumor tissue and the adjacent non-tumor area (negative control) were analyzed by Raman micro-spectroscopy. Preliminarily, the cells were analyzed morphologically and spectrally. Then, three machine learning approaches, including multivariate models and neural networks, were simultaneously investigated and successfully used to analyze the cells' Raman data. The results clearly demonstrate the effectiveness of artificial intelligence (AI)-assisted Raman spectroscopy for Tumor cell classification and prediction with an accuracy of nearly 90% of correct predictions on a single spectrum.

Webcam-based eye-tracking to measure visual expertise of medical students during online histology training

Objectives

Visual expertise is essential for image-based tasks that rely on visual cues, such as in radiology or histology. Studies suggest that eye movements are related to visual expertise and can be measured by near-infrared eye-tracking. With the popularity of device-embedded webcam eye-tracking technology, cost-effective use in educational contexts has recently become amenable. This study investigated the feasibility of such methodology in a curricular online-only histology course during the 2021 summer term.

Methods

At two timepoints (t1 and t2), third-semester medical students were asked to diagnose a series of histological slides while their eye movements were recorded. Students' eye metrics, performance and behavioral measures were analyzed using variance analyses and multiple regression models.

Results

First, webcam-eye tracking provided eye movement data with satisfactory quality (mean accuracy=115.7 px±31.1). Second, the eye movement metrics reflected the students' proficiency in finding relevant image sections (fixation count on relevant areas=6.96±1.56 vs. irrelevant areas=4.50±1.25). Third, students' eye movement metrics successfully predicted their performance (R2adj=0.39, p<0.001).

Conclusion

This study supports the use of webcam-eye-tracking expanding the range of educational tools available in the (digital) classroom. As the students' interest in using the webcam eye-tracking was high, possible areas of implementation will be discussed.

From Image to Diagnosis: Characterizing Sources of Error in Histopathologic Interpretation

An accurate histopathologic diagnosis on surgical biopsy material is necessary for the clinical management of patients and has important implications for research, clinical trial design/enrollment, and public health education. This study used a mixed methods approach to isolate sources of diagnostic error while residents and attending pathologists interpreted digitized breast biopsy slides. Ninety participants, including pathology residents and attending physicians at major United States medical centers reviewed a set of 14 digitized whole-slide images of breast biopsies. Each case had a consensus-defined diagnosis and critical region of interest (cROI) representing the most significant pathology on the slide. Participants were asked to view unmarked digitized slides, draw their participant region of interest (pROI), describe its features, and render a diagnosis. Participants' review behavior was tracked using case viewer software and an eye-tracking device. Diagnostic accuracy was calculated in comparison to the consensus diagnosis. We measured the frequency of errors emerging during 4 interpretive phases: (1) detecting the cROI, (2) recognizing its relevance, (3) using the correct terminology to describe findings in the pROI, and (4) making a diagnostic decision. According to eye-tracking data, trainees and attending pathologists were very likely (∼94% of the time) to find the cROI when inspecting a slide. However, trainees were less likely to consider the cROI relevant to their diagnosis. Pathology trainees (41% of cases) were more likely to use incorrect terminology to describe pROI features than attending pathologists (21% of cases). Failure to accurately describe features was the only factor strongly associated with an incorrect diagnosis. Identifying where errors emerge in the interpretive and/or descriptive process and working on building organ-specific feature recognition and verbal fluency in describing those features are critical steps for achieving competency in diagnostic decision making.

Synoptic Reporting in Clinical Placental Pathology: A Preliminary Investigation Into Report Findings and Interobserver Agreement

INTRODUCTION

Placental pathology is key for investigating adverse pregnancy outcomes, however, lack of standardization in reporting has limited clinical utility. We evaluated a novel placental pathology synoptic report, comparing its robustness to narrative reports, and assessed interobserver agreement.

METHODS

100 singleton placentas were included. Histology slides were examined by 2 senior perinatal pathologists and 2 pathology residents using a synoptic report (32 lesions). Historical narrative reports were compared to synoptic reports. Kappa scores were calculated for interobserver agreement between senior, resident, and senior vs resident pathologists.

RESULTS

Synoptic reporting detected 169 (51.4%) lesion instances initially not included in historical reports. Amongst senior pathologists, 64% of all lesions examined demonstrated fair-to-excellent agreement (Kappa ≥0.41), with only 26% of Kappas ≥0.41 amongst those examined by resident pathologists. Well-characterized lesions (e.g., chorioamnionitis) demonstrated higher agreement, with lower agreement for uncommon lesions and those previously shown to have poor consensus.

DISCUSSION

Synoptic reporting is one proposed method to address issues in placenta pathology reporting. The synoptic report generally identifies more lesions compared to the narrative report, however clinical significance remains unclear. Interobserver agreement is likely related to differential in experience. Further efforts to improve overall standardization of placenta pathology reporting are needed.

Should low-risk DCIS lose the cancer label? An evidence review

BACKGROUND

Population mammographic screening for breast cancer has led to large increases in the diagnosis and treatment of ductal carcinoma in situ (DCIS). Active surveillance has been proposed as a management strategy for low-risk DCIS to mitigate against potential overdiagnosis and overtreatment. However, clinicians and patients remain reluctant to choose active surveillance, even within a trial setting. Re-calibration of the diagnostic threshold for low-risk DCIS and/or use of a label that does not include the word 'cancer' might encourage the uptake of active surveillance and other conservative treatment options. We aimed to identify and collate relevant epidemiological evidence to inform further discussion on these ideas.

METHODS

We searched PubMed and EMBASE databases for low-risk DCIS studies in four categories: (1) natural history; (2) subclinical cancer found at autopsy; (3) diagnostic reproducibility (two or more pathologist interpretations at a single time point); and (4) diagnostic drift (two or more pathologist interpretations at different time points). Where we identified a pre-existing systematic review, the search was restricted to studies published after the inclusion period of the review. Two authors screened records, extracted data, and performed risk of bias assessment. We undertook a narrative synthesis of the included evidence within each category.

RESULTS

Natural History (n = 11): one systematic review and nine primary studies were included, but only five provided evidence on the prognosis of women with low-risk DCIS. These studies reported that women with low-risk DCIS had comparable outcomes whether or not they had surgery. The risk of invasive breast cancer in patients with low-risk DCIS ranged from 6.5% (7.5 years) to 10.8% (10 years). The risk of dying from breast cancer in patients with low-risk DCIS ranged from 1.2 to 2.2% (10 years). Subclinical cancer at autopsy (n = 1): one systematic review of 13 studies estimated the mean prevalence of subclinical in situ breast cancer to be 8.9%. Diagnostic reproducibility (n = 13): two systematic reviews and 11 primary studies found at most moderate agreement in differentiating low-grade DCIS from other diagnoses. Diagnostic drift: no studies found.

CONCLUSION

Epidemiological evidence supports consideration of relabelling and/or recalibrating diagnostic thresholds for low-risk DCIS. Such diagnostic changes would need agreement on the definition of low-risk DCIS and improved diagnostic reproducibility.

Zoom behavior during visual search modulates pupil diameter and reflects adaptive control states

Adaptive gain theory proposes that the dynamic shifts between exploration and exploitation control states are modulated by the locus coeruleus-norepinephrine system and reflected in tonic and phasic pupil diameter. This study tested predictions of this theory in the context of a societally important visual search task: the review and interpretation of digital whole slide images of breast biopsies by physicians (pathologists). As these medical images are searched, pathologists encounter difficult visual features and intermittently zoom in to examine features of interest. We propose that tonic and phasic pupil diameter changes during image review may correspond to perceived difficulty and dynamic shifts between exploration and exploitation control states. To examine this possibility, we monitored visual search behavior and tonic and phasic pupil diameter while pathologists (N = 89) interpreted 14 digital images of breast biopsy tissue (1,246 total images reviewed). After viewing the images, pathologists provided a diagnosis and rated the level of difficulty of the image. Analyses of tonic pupil diameter examined whether pupil dilation was associated with pathologists' difficulty ratings, diagnostic accuracy, and experience level. To examine phasic pupil diameter, we parsed continuous visual search data into discrete zoom-in and zoom-out events, including shifts from low to high magnification (e.g., 1× to 10×) and the reverse. Analyses examined whether zoom-in and zoom-out events were associated with phasic pupil diameter change. Results demonstrated that tonic pupil diameter was associated with image difficulty ratings and zoom level, and phasic pupil diameter showed constriction upon zoom-in events, and dilation immediately preceding a zoom-out event. Results are interpreted in the context of adaptive gain theory, information gain theory, and the monitoring and assessment of physicians' diagnostic interpretive processes.

Automated analysis of whole slide digital skin biopsy images

A rapidly increasing rate of melanoma diagnosis has been noted over the past three decades, and nearly 1 in 4 skin biopsies are diagnosed as melanocytic lesions. The gold standard for diagnosis of melanoma is the histopathological examination by a pathologist to analyze biopsy material at both the cellular and structural levels. A pathologist's diagnosis is often subjective and prone to variability, while deep learning image analysis methods may improve and complement current diagnostic and prognostic capabilities. Mitoses are important entities when reviewing skin biopsy cases as their presence carries prognostic information; thus, their precise detection is an important factor for clinical care. In addition, semantic segmentation of clinically important structures in skin biopsies might help the diagnosis pipeline with an accurate classification. We aim to provide prognostic and diagnostic information on skin biopsy images, including the detection of cellular level entities, segmentation of clinically important tissue structures, and other important factors toward the accurate diagnosis of skin biopsy images. This paper is an overview of our work on analysis of digital whole slide skin biopsy images, including mitotic figure (mitosis) detection, semantic segmentation, diagnosis, and analysis of pathologists' viewing patterns, and with new work on melanocyte detection. Deep learning has been applied to our methods for all the detection, segmentation, and diagnosis work. In our studies, deep learning is proven superior to prior approaches to skin biopsy analysis. Our work on analysis of pathologists' viewing patterns is the only such work in the skin biopsy literature. Our work covers the whole spectrum from low-level entities through diagnosis and understanding what pathologists do in performing their diagnoses.

On Smart Gaze based Annotation of Histopathology Images for Training of Deep Convolutional Neural Networks

Unavailability of large training datasets is a bottleneck that needs to be overcome to realize the true potential of deep learning in histopathology applications. Although slide digitization via whole slide imaging scanners has increased the speed of data acquisition, labeling of virtual slides requires a substantial time investment from pathologists. Eye gaze annotations have the potential to speed up the slide labeling process. This work explores the viability and timing comparisons of eye gaze labeling compared to conventional manual labeling for training object detectors. Challenges associated with gaze based labeling and methods to refine the coarse data annotations for subsequent object detection are also discussed. Results demonstrate that gaze tracking based labeling can save valuable pathologist time and delivers good performance when employed for training a deep object detector. Using the task of localization of Keratin Pearls in cases of oral squamous cell carcinoma as a test case, we compare the performance gap between deep object detectors trained using hand-labelled and gaze-labelled data. On average, compared to 'Bounding-box' based hand-labeling, gaze-labeling required 57.6% less time per label and compared to 'Freehand' labeling, gaze-labeling required on average 85% less time per label.

Breast cancer or surrounding normal tissue? A successful discrimination by FTIR or Raman microspectroscopy

Breast cancer is a type of cancer with the highest incidence worldwide in 2021, with early diagnosis and rapid treatment intervention being the reasons for the decreasing mortality rate associated with the disease.

Evaluation of Influence Factors on the Visual Inspection Performance of Aircraft Engine Blades

Background—There are various influence factors that affect visual inspection of aircraft engine blades including type of inspection, defect type, severity level, blade perspective and background colour. The effect of those factors on the inspection performance was assessed. Method—The inspection accuracy of fifty industry practitioners was measured for 137 blade images, leading to N = 6850 observations. The data were statistically analysed to identify the significant factors. Subsequent evaluation of the eye tracking data provided additional insights into the inspection process. Results—Inspection accuracies in borescope inspections were significantly lower compared to piece-part inspection at 63.8% and 82.6%, respectively. Airfoil dents (19.0%), cracks (11.0%), and blockage (8.0%) were the most difficult defects to detect, while nicks (100.0%), tears (95.5%), and tip curls (89.0%) had the highest detection rates. The classification accuracy was lowest for airfoil dents (5.3%), burns (38.4%), and tears (44.9%), while coating loss (98.1%), nicks (90.0%), and blockage (87.5%) were most accurately classified. Defects of severity level S1 (72.0%) were more difficult to detect than increased severity levels S2 (92.8%) and S3 (99.0%). Moreover, visual perspectives perpendicular to the airfoil led to better inspection rates (up to 87.5%) than edge perspectives (51.0% to 66.5%). Background colour was not a significant factor. The eye tracking results of novices showed an unstructured search path, characterised by numerous fixations, leading to longer inspection times. Experts in contrast applied a systematic search strategy with focus on the edges, and showed a better defect discrimination ability. This observation was consistent across all stimuli, thus independent of the influence factors. Conclusions—Eye tracking identified the challenges of the inspection process and errors made. A revised inspection framework was proposed based on insights gained, and support the idea of an underlying mental model.

A New Look into Cancer-A Review on the Contribution of Vibrational Spectroscopy on Early Diagnosis and Surgery Guidance

Simple Summary

Cancer is a leading cause of death worldwide, with the detection of the disease in its early stages, as well as a correct assessment of the tumour margins, being paramount for a successful recovery. While breast cancer is one of most common types of cancer, head and neck cancer is one of the types of cancer with a lower prognosis and poor aesthetic results. Vibrational spectroscopy detects molecular vibrations, being sensitive to different sample compositions, even when the difference was slight. The use of spectroscopy in biomedicine has been extensively explored, since it allows a broader assessment of the biochemical fingerprint of several diseases. This literature review covers the most recent advances in breast and head and neck cancer early diagnosis and intraoperative margin assessment, through Raman and Fourier transform infrared spectroscopies. The rising field of spectral histopathology was also approached. The authors aimed at expounding in a more concise and simple way the challenges faced by clinicians and how vibrational spectroscopy has evolved to respond to those needs for the two types of cancer with the highest potential for improvement regarding an early diagnosis, surgical margin assessment and histopathology.

Abstract

In 2020, approximately 10 million people died of cancer, rendering this disease the second leading cause of death worldwide. Detecting cancer in its early stages is paramount for patients’ prognosis and survival. Hence, the scientific and medical communities are engaged in improving both therapeutic strategies and diagnostic methodologies, beyond prevention. Optical vibrational spectroscopy has been shown to be an ideal diagnostic method for early cancer diagnosis and surgical margins assessment, as a complement to histopathological analysis. Being highly sensitive, non-invasive and capable of real-time molecular imaging, Raman and Fourier transform infrared (FTIR) spectroscopies give information on the biochemical profile of the tissue under analysis, detecting the metabolic differences between healthy and cancerous portions of the same sample. This constitutes tremendous progress in the field, since the cancer-prompted morphological alterations often occur after the biochemical imbalances in the oncogenic process. Therefore, the early cancer-associated metabolic changes are unnoticed by the histopathologist. Additionally, Raman and FTIR spectroscopies significantly reduce the subjectivity linked to cancer diagnosis. This review focuses on breast and head and neck cancers, their clinical needs and the progress made to date using vibrational spectroscopy as a diagnostic technique prior to surgical intervention and intraoperative margin assessment.

Molecular tissue profiling by MALDI imaging: recent progress and applications in cancer research

Matrix-assisted laser desorption/ionization (MALDI) imaging is an emergent technology that has been increasingly adopted in cancer research. MALDI imaging is capable of providing global molecular mapping of the abundance and spatial information of biomolecules directly in the tissues without labeling. It enables the characterization of a wide spectrum of analytes, including proteins, peptides, glycans, lipids, drugs, and metabolites and is well suited for both discovery and targeted analysis. An advantage of MALDI imaging is that it maintains tissue integrity, which allows correlation with histological features. It has proven to be a valuable tool for probing tumor heterogeneity and has been increasingly applied to interrogate molecular events associated with cancer. It provides unique insights into both the molecular content and spatial details that are not accessible by other techniques, and it has allowed considerable progress in the field of cancer research. In this review, we first provide an overview of the MALDI imaging workflow and approach. We then highlight some useful applications in various niches of cancer research, followed by a discussion of the challenges, recent developments and future prospect of this technique in the field.

Assessment of the Effect of Cleanliness on the Visual Inspection of Aircraft Engine Blades: An Eye Tracking Study

Background-The visual inspection of aircraft parts such as engine blades is crucial to ensure safe aircraft operation. There is a need to understand the reliability of such inspections and the factors that affect the results. In this study, the factor 'cleanliness' was analysed among other factors. Method-Fifty industry practitioners of three expertise levels inspected 24 images of parts with a variety of defects in clean and dirty conditions, resulting in a total of N = 1200 observations. The data were analysed statistically to evaluate the relationships between cleanliness and inspection performance. Eye tracking was applied to understand the search strategies of different levels of expertise for various part conditions. Results-The results show an inspection accuracy of 86.8% and 66.8% for clean and dirty blades, respectively. The statistical analysis showed that cleanliness and defect type influenced the inspection accuracy, while expertise was surprisingly not a significant factor. In contrast, inspection time was affected by expertise along with other factors, including cleanliness, defect type and visual acuity. Eye tracking revealed that inspectors (experts) apply a more structured and systematic search with less fixations and revisits compared to other groups. Conclusions-Cleaning prior to inspection leads to better results. Eye tracking revealed that inspectors used an underlying search strategy characterised by edge detection and differentiation between surface deposits and other types of damage, which contributed to better performance.

Artificial Intelligence-based Analytics for Diagnosis of Small Bowel Enteropathies and Black Box Feature Detection

OBJECTIVES

Striking histopathological overlap between distinct but related conditions poses a disease diagnostic challenge. There is a major clinical need to develop computational methods enabling clinicians to translate heterogeneous biomedical images into accurate and quantitative diagnostics. This need is particularly salient with small bowel enteropathies; environmental enteropathy (EE) and celiac disease (CD). We built upon our preliminary analysis by developing an artificial intelligence (AI)-based image analysis platform utilizing deep learning convolutional neural networks (CNNs) for these enteropathies.

METHODS

Data for the secondary analysis was obtained from three primary studies at different sites. The image analysis platform for EE and CD was developed using CNNs including one with multizoom architecture. Gradient-weighted class activation mappings (Grad-CAMs) were used to visualize the models' decision-making process for classifying each disease. A team of medical experts simultaneously reviewed the stain color normalized images done for bias reduction and Grad-CAMs to confirm structural preservation and biomedical relevance, respectively.

RESULTS

Four hundred and sixty-one high-resolution biopsy images from 150 children were acquired. Median age (interquartile range) was 37.5 (19.0-121.5) months with a roughly equal sex distribution; 77 males (51.3%). ResNet50 and shallow CNN demonstrated 98% and 96% case-detection accuracy, respectively, which increased to 98.3% with an ensemble. Grad-CAMs demonstrated models' ability to learn different microscopic morphological features for EE, CD, and controls.

CONCLUSIONS

Our AI-based image analysis platform demonstrated high classification accuracy for small bowel enteropathies which was capable of identifying biologically relevant microscopic features and emulating human pathologist decision-making process. Grad-CAMs illuminated the otherwise "black box" of deep learning in medicine, allowing for increased physician confidence in adopting these new technologies in clinical practice.

Scale-Aware Transformers for Diagnosing Melanocytic Lesions

Diagnosing melanocytic lesions is one of the most challenging areas of pathology with extensive intra- and inter-observer variability. The gold standard for a diagnosis of invasive melanoma is the examination of histopathological whole slide skin biopsy images by an experienced dermatopathologist. Digitized whole slide images offer novel opportunities for computer programs to improve the diagnostic performance of pathologists. In order to automatically classify such images, representations that reflect the content and context of the input images are needed. In this paper, we introduce a novel self-attention-based network to learn representations from digital whole slide images of melanocytic skin lesions at multiple scales. Our model softly weighs representations from multiple scales, allowing it to discriminate between diagnosis-relevant and -irrelevant information automatically. Our experiments show that our method outperforms five other state-of-the-art whole slide image classification methods by a significant margin. Our method also achieves comparable performance to 187 practicing U.S. pathologists who interpreted the same cases in an independent study. To facilitate relevant research, full training and inference code is made publicly available at https://github.com/meredith-wenjunwu/ScATNet.

Multi-resolution open-top light-sheet microscopy to enable efficient 3D pathology workflows

Open-top light-sheet (OTLS) microscopes have been developed for user-friendly and versatile high-throughput 3D microscopy of thick specimens. As with all imaging modalities, spatial resolution trades off with imaging and analysis times. A hierarchical multi-scale imaging workflow would therefore be of value for many volumetric microscopy applications. We describe a compact multi-resolution OTLS microscope, enabled by a novel solid immersion meniscus lens (SIMlens), which allows users to rapidly transition between air-based objectives for low- and high-resolution 3D imaging. We demonstrate the utility of this system by showcasing an efficient 3D analysis workflow for a diagnostic pathology application.

Radio-pathomic mapping model generated using annotations from five pathologists reliably distinguishes high-grade prostate cancer

Purpose: Our study predictively maps epithelium density in magnetic resonance imaging (MRI) space while varying the ground truth labels provided by five pathologists to quantify the downstream effects of interobserver variability. Approach: Clinical imaging and postsurgical tissue from 48 recruited prospective patients were used in our study. Tissue was sliced to match the MRI orientation and whole-mount slides were stained and digitized. Data from 28 patients ( n = 33 slides) were sent to five pathologists to be annotated. Slides from the remaining 20 patients ( n = 123 slides) were annotated by one of the five pathologists. Interpathologist variability was measured using Krippendorff's alpha. Pathologist-specific radiopathomic mapping models were trained using a partial least-squares regression using MRI values to predict epithelium density, a known marker for disease severity. An analysis of variance characterized intermodel means difference in epithelium density. A consensus model was created and evaluated using a receiver operator characteristic classifying high grade versus low grade and benign, and was statistically compared to apparent diffusion coefficient (ADC). Results: Interobserver variability ranged from low to acceptable agreement (0.31 to 0.69). There was a statistically significant difference in mean predicted epithelium density values ( p < 0.001 ) between the five models. The consensus model outperformed ADC (areas under the curve = 0.80 and 0.71, respectively, p < 0.05 ). Conclusion: We demonstrate that radiopathomic maps of epithelium density are sensitive to the pathologist annotating the dataset; however, it is unclear if these differences are clinically significant. The consensus model produced the best maps, matched the performance of the best individual model, and outperformed ADC.

Report on computational assessment of Tumor Infiltrating Lymphocytes from the International Immuno-Oncology Biomarker Working Group

Assessment of tumor-infiltrating lymphocytes (TILs) is increasingly recognized as an integral part of the prognostic workflow in triple-negative (TNBC) and HER2-positive breast cancer, as well as many other solid tumors. This recognition has come about thanks to standardized visual reporting guidelines, which helped to reduce inter-reader variability. Now, there are ripe opportunities to employ computational methods that extract spatio-morphologic predictive features, enabling computer-aided diagnostics. We detail the benefits of computational TILs assessment, the readiness of TILs scoring for computational assessment, and outline considerations for overcoming key barriers to clinical translation in this arena. Specifically, we discuss: 1. ensuring computational workflows closely capture visual guidelines and standards; 2. challenges and thoughts standards for assessment of algorithms including training, preanalytical, analytical, and clinical validation; 3. perspectives on how to realize the potential of machine learning models and to overcome the perceptual and practical limits of visual scoring.

Difference in eye movements during gait analysis between professionals and trainees

INTRODUCTION

Observational gait analysis is a widely used skill in physical therapy. Meanwhile, the skill has not been investigated using objective assessments. The present study investigated the differences in eye movement between professionals and trainees, while observing gait analysis.

METHODS

The participants included in this study were 26 professional physical therapists and 26 physical therapist trainees. The participants, wearing eye tracker systems, were asked to describe gait abnormalities of a patient as much as possible. The eye movement parameters of interest were fixation count, average fixation duration, and total fixation duration.

RESULTS

The number of gait abnormalities described was significantly higher in professionals than in trainees, overall and in limbs of the patient. The fixation count was significantly higher in professionals when compared to trainees. Additionally, the average fixation duration and total fixation duration were significantly shorter in professionals. Conversely, in trunks, the number of gait abnormalities and eye movements showed no significant differences between groups.

CONCLUSIONS

Professionals require shorter fixation durations on areas of interest than trainees, while describing a higher number of gait abnormalities.

The Use of Digital Microscopy to Compare the Thicknesses of Normal Corneas and Ex Vivo Rejected Corneal Grafts with a Focus on the Descemet's Membrane

Objective

To compare the thickness of corneal layers, specifically the Descemet's membrane (DM), in normal corneas and in failed grafts due to rejection (FGRs) using the digital histopathology and to propose a model for the measurement of corneal layers using this method.

Methods

This is a prospective, cross-sectional study performed at the MUHC-McGill University Ocular Pathology & Translational Research Laboratory (McGill University, Montreal, Canada). Histopathological sections of 25 normal human corneas and 40 FGRs were fully digitalized and examined. Inclusion criteria: samples diagnosed as normal corneas or FGRs, from patients older than 18 years of age. Exclusion criteria: histopathological sections without adequate tissue or missing epidemiological information. For each sample, the thicknesses of the epithelium, stroma, and DM were acquired. From a perpendicular plane of reference, two central measurements and two nasal and two temporal peripheral measurements were obtained.

Results

There were differences between the normal and FGR groups in the mean central thickness of the epithelium (p < 0.001), the nasal and temporal stromal regions (p < 0.001), and of the DM in the nasal and temporal regions (p < 0.001). Compared with the extremities of the sample (nasal and temporal), the mean thickness of the DM in normal corneas was lower in the central region (p < 0.001), and this difference was not found in the FGR group.

Conclusions

Normal corneas have a thinner epithelium in the central region than the FGR group. In addition, the stroma and DM thicknesses of the nasal and temporal periphery were significantly higher in normal corneas than in those from the FGR group. The digital microscopy protocol applied in this study may be useful for further research studies regarding cornea and other tissues.

An Expert Derived Feedforward Histology Module Improves Pattern Recognition Efficiency in Novice Students

Histology is a visually oriented, foundational anatomical sciences subject in professional health curricula that has seen a dramatic reduction in educational contact hours and an increase in content migration to a digital platform. While the digital migration of histology laboratories has transformed histology education, few studies have shown the impact of this change on visual literacy development, a critical competency in histology. The objective of this study was to assess whether providing a video clip of an expert's gaze while completing leukocyte identification tasks would increase the efficiency and performance of novices completing similar identification tasks. In a randomized study, one group of novices (n = 9) was provided with training materials that included expert eye gaze, while the other group (n = 12) was provided training materials with identical content, but without the expert eye gaze. Eye movement parameters including fixation rate and total scan path distance, and performance measures including time-to-task-completion and accuracy, were collected during an identification task assessment. Compared to the control group, the average fixation duration was 13.2% higher (P < 0.02) and scan path distance was 35.0% shorter in the experimental group (P = 0.14). Analysis of task performance measures revealed no significant difference between the groups. These preliminary results suggest a more efficient search performed by the experimental group, indicating the potential efficacy of training using an expert's gaze to enhance visual literacy development. With further investigation, such feedforward enhanced training methods could be utilized for histology and other visually oriented subjects.

Multiple metastases of bones and sigmoid colon after mastectomy for ductal carcinoma in situ of the breast: a case report

BACKGROUND

The prognosis of ductal carcinoma in situ (DCIS) is reportedly well. Extremely rare patients with DCIS develop distant breast cancer metastasis without locoregional or contralateral recurrence. This is the first report of multiple bones and sigmoid colon metastases from DCIS after mastectomy.

CASE PRESENTATION

A 43-year-old woman was diagnosed with DCIS, and she received mastectomy, followed by endocrine therapy and target therapy. During the following-up, convulsions and pain on the legs were complaint. Therefore, Computed Tomography (CT) on bones and positron emission tomography (PET) for whole body were examined in order. Multiple bones and sigmoid colon were under the suspect of metastases, which were then verified by biopsy in the left ilium and colonoscopy respectively.

CONCLUSIONS

This case reveals the heterogeneous behavior and the potential poor outcome of DCIS, regular examination and surveillance are necessary even though the distant metastasis rate in DCIS is low.

Eye tracking in cytotechnology education: "visualizing" students becoming experts

INTRODUCTION

This study reports the potential of eye-tracking technology in determining screening skills of cytotechnology (CT) students while examining digital images (DI).

MATERIALS AND METHODS

Twenty-five static DI of gynecologic cytology specimens were serially displayed on a computer monitor for evaluation by 16 CT students and 3 cytotechnologists at 3 locations. During evaluation, participant's eye movements were monitored with a Mirametrix S2 eye tracker (iMotions, Boston, MA) and EyeWorks software (Eyetracking, Solana Beach, CA). Students completed the protocol at: Period1 (P1)-4 months, Period2 (P2)-7 months, Period3 (P3)-11 months during their 1-year training; and the cytotechnologists only once. A general linear mixed model was used to analyze the results.

RESULTS

The proportion of agreement on interpretations for cytotechnologists, students during P1, and students during P3 were 0.83, 0.62, and 0.70 respectively. The mean task duration in seconds for cytotechnologists, students during P1, and students during P3 were 21.1, 34.6, and 24.9 respectively. The mean number of fixation points for cytotechnologists, students during P1, and students during P3 were 14.5, 52.2, and 35.3, respectively. The mean number of gaze observations of cytotechnologists, students during P1, and students during P3 on region of interest (ROI) 1 were 77.93, 181.12, and 123.83, respectively; and, ROI 2 were 38.90, 142.79, and 92.46, respectively.

CONCLUSIONS

This study demonstrated that students had decreased time, number of fixation points, gaze observations on ROI, and increased agreement with the reference interpretations at the end of the training program, indicating that their screening skills were progressing towards the level of practicing cytotechnologists.

Eye-tracking for assessing medical image interpretation: A pilot feasibility study comparing novice vs expert cardiologists

INTRODUCTION

As specialized medical professionals such as radiologists, pathologists, and cardiologists gain education and experience, their diagnostic efficiency and accuracy change, and they show altered eye movement patterns during medical image interpretation. Existing research in this area is limited to interpretation of static medical images, such as digitized whole slide biopsies, making it difficult to understand how expertise development might manifest during dynamic image interpretation, such as with angiograms or volumetric scans.

METHODS

A two-group (novice, expert) comparative pilot study examined the feasibility and utility of tracking and interpreting eye movement patterns while cardiologists viewed video-based coronary angiograms. A non-invasive eye tracking system recorded cardiologists' (n = 8) visual behaviour while they viewed and diagnosed a series of eight angiogram videos. Analyses assessed frame-by-frame video navigation behaviour, eye fixation behaviour, and resulting diagnostic decision making.

RESULTS

Relative to novices, expert cardiologists demonstrated shorter and less variable video review times, fewer eye fixations and saccadic eye movements, and less time spent paused on individual video frames. Novices showed repeated eye fixations on critical image frames and regions, though these were not predictive of accurate diagnostic decisions.

DISCUSSION

These preliminary results demonstrate interpretive decision errors among novices, suggesting they identify and process critical diagnostic features, but sometimes fail to accurately interpret those features. Results also showcase the feasibility of tracking and understanding eye movements during video-based coronary angiogram interpretation and suggest that eye tracking may be valuable for informing assessments of competency progression during medical education and training.

A review of eye tracking for understanding and improving diagnostic interpretation

Inspecting digital imaging for primary diagnosis introduces perceptual and cognitive demands for physicians tasked with interpreting visual medical information and arriving at appropriate diagnoses and treatment decisions. The process of medical interpretation and diagnosis involves a complex interplay between visual perception and multiple cognitive processes, including memory retrieval, problem-solving, and decision-making. Eye-tracking technologies are becoming increasingly available in the consumer and research markets and provide novel opportunities to learn more about the interpretive process, including differences between novices and experts, how heuristics and biases shape visual perception and decision-making, and the mechanisms underlying misinterpretation and misdiagnosis. The present review provides an overview of eye-tracking technology, the perceptual and cognitive processes involved in medical interpretation, how eye tracking has been employed to understand medical interpretation and promote medical education and training, and some of the promises and challenges for future applications of this technology.

Cancer diagnosis through a tandem of classifiers for digitized histopathological slides

The current research study is concerned with the automated differentiation between histopathological slides from colon tissues with respect to four classes (healthy tissue and cancerous of grades 1, 2 or 3) through an optimized ensemble of predictors. Six distinct classifiers with prediction accuracies ranging from 87% to 95% are considered for the task. The proposed method of combining them takes into account the probabilities of the individual classifiers for each sample to be assigned to any of the four classes, optimizes weights for each technique by differential evolution and attains an accuracy that is significantly better than the individual results. Moreover, a degree of confidence is defined that would allow the pathologists to separate the data into two distinct sets, one that is correctly classified with a high level of confidence and the rest that would need their further attention. The tandem is also validated on other benchmark data sets. The proposed methodology proves to be efficient in improving the classification accuracy of each algorithm taken separately and performs reasonably well on other data sets, even with default weights. In addition, by establishing a degree of confidence the method becomes more viable for use by actual practitioners.

Detection and classification of cancer in whole slide breast histopathology images using deep convolutional networks

Generalizability of algorithms for binary cancer vs. no cancer classification is unknown for clinically more significant multi-class scenarios where intermediate categories have different risk factors and treatment strategies. We present a system that classifies whole slide images (WSI) of breast biopsies into five diagnostic categories. First, a saliency detector that uses a pipeline of four fully convolutional networks, trained with samples from records of pathologists' screenings, performs multi-scale localization of diagnostically relevant regions of interest in WSI. Then, a convolutional network, trained from consensus-derived reference samples, classifies image patches as non-proliferative or proliferative changes, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma. Finally, the saliency and classification maps are fused for pixel-wise labeling and slide-level categorization. Experiments using 240 WSI showed that both saliency detector and classifier networks performed better than competing algorithms, and the five-class slide-level accuracy of 55% was not statistically different from the predictions of 45 pathologists. We also present example visualizations of the learned representations for breast cancer diagnosis.

An efficient architecture to support digital pathology in standard medical imaging repositories

In the past decade, digital pathology and whole-slide imaging (WSI) have been gaining momentum with the proliferation of digital scanners from different manufacturers. The literature reports significant advantages associated with the adoption of digital images in pathology, namely, improvements in diagnostic accuracy and better support for telepathology. Moreover, it also offers new clinical and research applications. However, numerous barriers have been slowing the adoption of WSI, among which the most important are performance issues associated with storage and distribution of huge volumes of data, and lack of interoperability with other hospital information systems, most notably Picture Archive and Communications Systems (PACS) based on the DICOM standard. This article proposes an architecture of a Web Pathology PACS fully compliant with DICOM standard communications and data formats. The solution includes a PACS Archive responsible for storing whole-slide imaging data in DICOM WSI format and offers a communication interface based on the most recent DICOM Web services. The second component is a zero-footprint viewer that runs in any web-browser. It consumes data using the PACS archive standard web services. Moreover, it features a tiling engine especially suited to deal with the WSI image pyramids. These components were designed with special focus on efficiency and usability. The performance of our system was assessed through a comparative analysis of the state-of-the-art solutions. The results demonstrate that it is possible to have a very competitive solution based on standard workflows.