Tailored for Real-World: A Whole Slide Image Classification System Validated on Uncurated Multi-Site Data Emulating the Prospective Pathology Workload

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Artificial intelligence for nonmelanoma skin cancer

Nonmelanoma skin cancers (NMSCs) are among the top five most common cancers globally. NMSC is an area with great potential for novel application of diagnostic tools including artificial intelligence (AI). In this scoping review, we aimed to describe the applications of AI in the diagnosis and treatment of NMSC. Twenty-nine publications described AI applications to dermatopathology including lesion classification and margin assessment. Twenty-five publications discussed AI use in clinical image analysis, showing that algorithms are not superior to dermatologists and may rely on unbalanced, nonrepresentative, and nontransparent training data sets. Sixteen publications described the use of AI in cutaneous surgery for NMSC including use in margin assessment during excisions and Mohs surgery, as well as predicting procedural complexity. Eleven publications discussed spectroscopy, confocal microscopy, thermography, and the AI algorithms that analyze and interpret their data. Ten publications pertained to AI applications for the discovery and use of NMSC biomarkers. Eight publications discussed the use of smartphones and AI, specifically how they enable clinicians and patients to have increased access to instant dermatologic assessments but with varying accuracies. Five publications discussed large language models and NMSC, including how they may facilitate or hinder patient education and medical decision-making. Three publications pertaining to the skin of color and AI for NMSC discussed concerns regarding limited diverse data sets for the training of convolutional neural networks. AI demonstrates tremendous potential to improve diagnosis, patient and clinician education, and management of NMSC. Despite excitement regarding AI, data sets are often not transparently reported, may include low-quality images, and may not include diverse skin types, limiting generalizability. AI may serve as a tool to increase access to dermatology services for patients in rural areas and save health care dollars. These benefits can only be achieved, however, with consideration of potential ethical costs.

Region of Interest Detection in Melanocytic Skin Tumor Whole Slide Images - Nevus & Melanoma

Automated region of interest detection in histopathological image analysis is a challenging and important topic with tremendous potential impact on clinical practice. The deep-learning methods used in computational pathology may help us to reduce costs and increase the speed and accuracy of cancer diagnosis. We started with the UNC Melanocytic Tumor Dataset cohort that contains 160 hematoxylin and eosin whole-slide images of primary melanomas (86) and nevi (74). We randomly assigned 80% (134) as a training set and built an in-house deep-learning method to allow for classification, at the slide level, of nevi and melanomas. The proposed method performed well on the other 20% (26) test dataset; the accuracy of the slide classification task was 92.3% and our model also performed well in terms of predicting the region of interest annotated by the pathologists, showing excellent performance of our model on melanocytic skin tumors. Even though we tested the experiments on the skin tumor dataset, our work could also be extended to other medical image detection problems to benefit the clinical evaluation and diagnosis of different tumors.

Detection and subtyping of basal cell carcinoma in whole-slide histopathology using weakly-supervised learning

The frequency of basal cell carcinoma (BCC) cases is putting an increasing strain on dermatopathologists. BCC is the most common type of skin cancer, and its incidence is increasing rapidly worldwide. AI can play a significant role in reducing the time and effort required for BCC diagnostics and thus improve the overall efficiency of the process. To train such an AI system in a fully-supervised fashion however, would require a large amount of pixel-level annotation by already strained dermatopathologists. Therefore, in this study, our primary objective was to develop a weakly-supervised for the identification of basal cell carcinoma (BCC) and the stratification of BCC into low-risk and high-risk categories within histopathology whole-slide images (WSI). We compared Clustering-constrained Attention Multiple instance learning (CLAM) with StreamingCLAM and hypothesized that the latter would be the superior approach. A total of 5147 images were used to train and validate the models, which were subsequently tested on an internal set of 949 images and an external set of 183 images. The labels for training were automatically extracted from free-text pathology reports using a rule-based approach. All data has been made available through the COBRA dataset. The results showed that both the CLAM and StreamingCLAM models achieved high performance for the detection of BCC, with an area under the ROC curve (AUC) of 0.994 and 0.997, respectively, on the internal test set and 0.983 and 0.993 on the external dataset. Furthermore, the models performed well on risk stratification, with AUC values of 0.912 and 0.931, respectively, on the internal set, and 0.851 and 0.883 on the external set. In every single metric the StreamingCLAM model outperformed the CLAM model or is on par. The performance of both models was comparable to that of two pathologists who scored 240 BCC positive slides. Additionally, in the public test set, StreamingCLAM demonstrated a comparable AUC of 0.958, markedly superior to CLAM's 0.803. This difference was statistically significant and emphasized the strength and better adaptability of the StreamingCLAM approach.

Multiple instance learning for digital pathology: A review of the state-of-the-art, limitations & future potential

Digital whole slides images contain an enormous amount of information providing a strong motivation for the development of automated image analysis tools. Particularly deep neural networks show high potential with respect to various tasks in the field of digital pathology. However, a limitation is given by the fact that typical deep learning algorithms require (manual) annotations in addition to the large amounts of image data, to enable effective training. Multiple instance learning exhibits a powerful tool for training deep neural networks in a scenario without fully annotated data. These methods are particularly effective in the domain of digital pathology, due to the fact that labels for whole slide images are often captured routinely, whereas labels for patches, regions, or pixels are not. This potential resulted in a considerable number of publications, with the vast majority published in the last four years. Besides the availability of digitized data and a high motivation from the medical perspective, the availability of powerful graphics processing units exhibits an accelerator in this field. In this paper, we provide an overview of widely and effectively used concepts of (deep) multiple instance learning approaches and recent advancements. We also critically discuss remaining challenges as well as future potential.

ChatGPT as an aid for pathological diagnosis of cancer

Diagnostic workup of cancer patients is highly reliant on the science of pathology using cytopathology, histopathology, and other ancillary techniques like immunohistochemistry and molecular cytogenetics. Data processing and learning by means of artificial intelligence (AI) has become a spearhead for the advancement of medicine, with pathology and laboratory medicine being no exceptions. ChatGPT, an artificial intelligence (AI)-based chatbot, that was recently launched by OpenAI, is currently a talk of the town, and its role in cancer diagnosis is also being explored meticulously. Pathology workflow by integration of digital slides, implementation of advanced algorithms, and computer-aided diagnostic techniques extend the frontiers of the pathologist's view beyond a microscopic slide and enables effective integration, assimilation, and utilization of knowledge that is beyond human limits and boundaries. Despite of it's numerous advantages in the pathological diagnosis of cancer, it comes with several challenges like integration of digital slides with input language parameters, problems of bias, and legal issues which have to be addressed and worked up soon so that we as a pathologists diagnosing malignancies are on the same band wagon and don't miss the train.

Advances in melanoma: epidemiology, diagnosis, and prognosis

Unraveling the multidimensional complexities of melanoma has required concerted efforts by dedicated community of researchers and clinicians battling against this deadly form of skin cancer. Remarkable advances have been made in the realm of epidemiology, classification, diagnosis, and therapy of melanoma. The treatment of advanced melanomas has entered the golden era as targeted personalized therapies have emerged that have significantly altered the mortality rate. A paradigm shift in the approach to melanoma classification, diagnosis, prognosis, and staging is underway, fueled by discoveries of genetic alterations in melanocytic neoplasms. A morphologic clinicopathologic classification of melanoma is expected to be replaced by a more precise molecular based one. As validated, convenient, and cost-effective molecular-based tests emerge, molecular diagnostics will play a greater role in the clinical and histologic diagnosis of melanoma. Artificial intelligence augmented clinical and histologic diagnosis of melanoma is expected to make the process more streamlined and efficient. A more accurate model of prognosis and staging of melanoma is emerging based on molecular understanding melanoma. This contribution summarizes the recent advances in melanoma epidemiology, classification, diagnosis, and prognosis.

Applying an artificial intelligence deep learning approach to routine dermatopathological diagnosis of basal cell carcinoma

BACKGROUND

Institutes of dermatopathology are faced with considerable challenges including a continuously rising numbers of submitted specimens and a shortage of specialized health care practitioners. Basal cell carcinoma (BCC) is one of the most common tumors in the fair-skinned western population and represents a major part of samples submitted for histological evaluation. Digitalizing glass slides has enabled the application of artificial intelligence (AI)-based procedures. To date, these methods have found only limited application in routine diagnostics. The aim of this study was to establish an AI-based model for automated BCC detection.

PATIENTS AND METHODS

In three dermatopathological centers, daily routine practice BCC cases were digitalized. The diagnosis was made both conventionally by analog microscope and digitally through an AI-supported algorithm based on a U-Net architecture neural network.

RESULTS

In routine practice, the model achieved a sensitivity of 98.23% (center 1) and a specificity of 98.51%. The model generalized successfully without additional training to samples from the other centers, achieving similarly high accuracies in BCC detection (sensitivities of 97.67% and 98.57% and specificities of 96.77% and 98.73% in centers 2 and 3, respectively). In addition, automated AI-based basal cell carcinoma subtyping and tumor thickness measurement were established.

CONCLUSIONS

AI-based methods can detect BCC with high accuracy in a routine clinical setting and significantly support dermatopathological work.

Einsatz künstlicher Intelligenz mittels Deep Learning in der dermatopathologischen Routinediagnostik des Basalzellkarzinoms: Applying an artificial intelligence deep learning approach to routine dermatopathological diagnosis of basal cell carcinoma

Hintergrund

Dermatopathologische Institute stehen aufgrund immer höherer Anforderungen bei andererseits schwindenden Ressourcen vor zunehmenden Herausforderungen. Basalzellkarzinome stellen einen Großteil des Einsendeguts mit entsprechendem Arbeitsaufwand dar. Gleichzeitig ermöglicht die Digitalisierung von Glasobjektträgern den Einsatz künstlicher Intelligenz (KI)‐basierter Verfahren in der Dermatopathologie. Bislang haben diese Verfahren keinen Einzug in die Routinediagnostik gefunden. Ziel dieser Studie war daher, den Einsatz eines KI‐basierten Modells zur automatisierten Basalzellkarzinom‐Erkennung zu etablieren.

Patienten und Methodik

In drei dermatopathologischen Zentren wurden während des täglichen Routinebetriebs Basalzellkarzinom‐Fälle digitalisiert und sowohl klassisch am Mikroskop als auch mittels KI‐basierter Methodik basierend auf neuronalen Netzen mit U‐Net‐Architektur befundet.

Ergebnisse

Im Routinebetrieb erzielte das Modell eine Sensitivität von 98,23 % und eine Spezifität von 98,51 % (Zentrum 1). Das Modell konnte übergangslos in den anderen Zentren Einsatz finden und erreichte ähnlich hohe Genauigkeiten in der Basalzellkarzinom‐Erkennung (Sensitivität von 97,67 % beziehungsweise 98,57 %, Spezifität von 96,77 % beziehungsweise 98,73 %). Zusätzlich wurden eine automatisierte, KI‐basierte Basalzellkarzinom‐Subtypisierung und Tumordickenmessung etabliert.

Schlussfolgerungen

KI‐basierte Verfahren können mit einer hohen Genauigkeit im Routinebetrieb Basalzellkarzinome erkennen und signifikant die dermatopathologische Arbeit unterstützen.

Performance of Automated Classification of Diagnostic Entities in Dermatopathology Validated on Multisite Data Representing the Real-World Variability of Pathology Workload

CONTEXT.—

More people receive a diagnosis of skin cancer each year in the United States than all other cancers combined. Many patients around the globe do not have access to highly trained dermatopathologists, whereas some biopsy diagnoses of patients who do have access result in disagreements between such specialists. Mechanomind has developed software based on a deep-learning algorithm to classify 40 different diagnostic dermatopathology entities to improve diagnostic accuracy and to enable improvements in turnaround times and effort allocation.

OBJECTIVE.—

To assess the value of machine learning for microscopic tissue evaluation in dermatopathology.

DESIGN.—

A retrospective study comparing diagnoses of hematoxylin and eosin-stained glass slides rendered by 2 senior board-certified pathologists not involved in algorithm creation with the machine learning algorithm's classification was conducted. A total of 300 glass slides (1 slide per patient's case) from 4 hospitals in the United States and Africa with common variations in tissue preparation, staining, and scanning methods were included in the study.

RESULTS.—

The automated algorithm demonstrated sensitivity of 89 of 91 (97.8%), 107 of 107 (100%), and 101 of 102 (99%), as well as specificity of 204 of 209 (97.6%), 189 of 193 (97.9%), and 198 of 198 (100%) while identifying melanoma, nevi, and basal cell carcinoma in whole slide images, respectively.

CONCLUSIONS.—

Appropriately trained deep learning image analysis algorithms demonstrate high specificity and high sensitivity sufficient for use in screening, quality assurance, and workload distribution in anatomic pathology.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Artificial intelligence in digital pathology of cutaneous lymphomas: a review of the current state and future perspectives

Primary cutaneous lymphomas (CLs) represent a heterogeneous group of T-cell lymphomas and B-cell lymphomas that present in the skin without evidence of extracutaneous involvement at time of diagnosis. CLs are largely distinct from their systemic counterparts in clinical presentation, histopathology, and biological behavior and, therefore, require different therapeutic management. Additional diagnostic burden is added by the fact that several benign inflammatory dermatoses mimic CL subtypes, requiring clinicopathological correlation for definitive diagnosis. Due to the heterogeneity and rarity of CL, adjunct diagnostic tools are welcomed, especially by pathologists without expertise in this field or with limited access to a centralized specialist panel. The transition into digital pathology workflows enables artificial intelligence (AI)-based analysis of patients' whole-slide pathology images (WSIs). AI can be used to automate manual processes in histopathology but, more importantly, can be applied to complex diagnostic tasks, especially suitable for rare disease like CL. To date, AI-based applications for CL have been minimally explored in literature. However, in other skin cancers and systemic lymphomas, disciplines that are recognized here as the building blocks for CLs, several studies demonstrated promising results using AI for disease diagnosis and subclassification, cancer detection, specimen triaging, and outcome prediction. Additionally, AI allows discovery of novel biomarkers or may help to quantify established biomarkers. This review summarizes and blends applications of AI in pathology of skin cancer and lymphoma and proposes how these findings can be applied to diagnostics of CL.

Multilayer outperforms single-layer slide scanning in AI-based classification of whole slide images with low-burden acid-fast mycobacteria (AFB)

Manual screening of Ziehl-Neelsen (ZN)-stained slides that are negative or contain rare acid-fast mycobacteria (AFB) is labor-intensive and requires repetitive refocusing to visualize AFB candidates under the microscope. Whole slide image (WSI) scanners have enabled implementation of AI to classify digital ZN-stained slides as AFB+ or AFB-. By default, these scanners acquire a single-layer WSI. However, some scanners can acquire a multilayer WSI with a z-stack and an extended focus image layer embedded. We developed a parameterized WSI classification pipeline to assess whether multilayer imaging improves ZN-stained slide classification accuracy. A CNN built into the pipeline classified tiles in each image layer to form an AFB probability score heatmap. Features extracted from the heatmap were then entered into a WSI classifier. 46 AFB+ and 88 AFB- single-layer WSIs were used for the classifier training. 15 AFB+ (with rare microorganisms) and 5 AFB- multilayer WSIs comprised the test set. Parameters in the pipeline included: (a) a WSI representation: z-stack of image layers, middle image layer (a single image layer equivalent) or an extended focus image layer, (b) 4 methods of aggregating AFB probability scores across the z-stack, (c) 3 classifiers, (d) 3 AFB probability thresholds, and (e) 9 feature vector types extracted from the aggregated AFB probability heatmaps. Balanced accuracy (BACC) was used to measure the pipeline performance for all parameter combinations. Analysis of Covariance (ANCOVA) was used to statistically evaluate the effect of each parameter on the BACC. After adjusting for other factors, a significant effect of the WSI representation (p-value < 1.99E-76), classifier type (p-value < 1.73E-21), and AFB threshold (p-value = 0.003) was observed on the BACC. The feature type had no significant effect (p-value = 0.459) on the BACC. WSIs represented by the middle layer, extended focus layer and the z-stack followed by the weighted averaging of AFB probability scores were classified with the average BACC of 58.80%, 68.64%, and 77.28%, respectively. The multilayer WSIs represented by the z-stack with the weighted averaging of AFB probability scores were classified by a Random Forest classifier with the average BACC of 83.32%. Low classification accuracy of WSIs represented by the middle layer suggests that they contain fewer features permitting identification of AFB than the multilayer WSIs. Our results indicate that single-layer acquisition can introduce a bias (sampling error) into the WSI. This bias can be mitigated by the multilayer or the extended focus acquisitions.

Artificial intelligence for basal cell carcinoma: diagnosis and distinction from histological mimics

We trained an artificial intelligence (AI) algorithm to identify basal cell carcinoma (BCC), and to distinguish BCC from histological mimics. A total of 1061 glass slides were collected: 616 containing BCC and 445 without BCC. BCC slides were collected prospectively, reflecting the range of specimen types and morphological variety encountered in routine pathology practice. Benign and malignant histological mimics of BCC were selected prospectively and retrospectively, including cases considered diagnostically challenging for pathologists. Glass slides were digitally scanned to create a whole slide image (WSI), which was divided into patches representing a tissue area of 65,535 μm². Pathologists annotated the data, yielding 87,205 patches labelled BCC present and 1,688,697 patches labelled BCC absent. The COMPASS model (COntext-aware Multi-scale tool for Pathologists Assessing SlideS) based on Convolutional Neural Networks, was trained to provide a probability of BCC being present at the patch level and the slide level. The test set comprised 246 slides, 147 of which contained BCC. The COMPASS AI model demonstrated high accuracy, classifying WSIs as containing BCC with a sensitivity of 98.0% and a specificity of 97.0%, representing 240 WSIs classified correctly, three false positives, and three false negatives. Using BCC as a proof of concept, we demonstrate how AI can account for morphological variation within an entity, and accurately distinguish from histologically similar entities. Our study highlights the potential for AI in routine pathology practice.

Interpretable classification of pathology whole-slide images using attention based context-aware graph convolutional neural network

BACKGROUND AND OBJECTIVE

Whole slide image (WSI) classification and lesion localization within giga-pixel slide are challenging tasks in computational pathology that requires context-aware representations of histological features to adequately infer nidus. The existing weakly supervised learning methods mainly treat different locations in the slide as independent regions and cannot learn potential nonlinear interactions between instances based on i.i.d assumption, resulting in the model unable to effectively utilize context-ware information to predict the labels of WSIs and locate the region of interest (ROI).

METHODS

Here, we propose an interpretable classification model named bidirectional Attention-based Multiple Instance Learning Graph Convolutional Network (ABMIL-GCN), which hierarchically aggregates context-aware features of instances into a global representation in a topology fashion to predict the slide labels and localize the region of lymph node metastasis in WSIs.

RESULTS

We verified the superiority of this method on the Camelyon16 dataset, and the results show that the average predicted ACC and AUC of the proposed model after flooding optimization can reach 90.89% and 0.9149, respectively. The average accuracy and ACC score are improved by more than 7% and 4% compared with the existing state-of-the-art algorithms.

CONCLUSIONS

The results demonstrate that context-aware GCN outperforms existing weakly supervised learning methods by introducing spatial correlations between the neighbor image patches, which also addresses the 'accuracy-interpretability trade-off' problem. The framework provides a novel paradigm for the clinical application of computer-aided diagnosis and intelligent systems.

Basic Elements of Artificial Intelligence Tools in the Diagnosis of Cutaneous Melanoma

Cutaneous melanoma (CM) incidence has dramatically increased in the last years. Early diagnosis is of paramount importance in terms of prognosis. Artificial Intelligence (AI) tools are being proposed for clinicians and pathologists as an adjunct support in the diagnostic process. We described herein an overview of the most important parameters that a potential AI tool should take into consideration in histopathology to evaluate a skin lesion. First of all, recognition of a melanocytic or non-melanocytic nature. Furthermore, melanocytic lesions should be stratified according to at least four parameters: silhouette and asymmetry; identification and spatial distribution of the cells; mitosis count; presence of ulceration. According to the number of parameters the AI tools might stratify the risk of CM and prioritize the pathologist's work.

Predictive uncertainty estimation for out-of-distribution detection in digital pathology

Machine learning model deployment in clinical practice demands real-time risk assessment to identify situations in which the model is uncertain. Once deployed, models should be accurate for classes seen during training while providing informative estimates of uncertainty to flag abnormalities and unseen classes for further analysis. Although recent developments in uncertainty estimation have resulted in an increasing number of methods, a rigorous empirical evaluation of their performance on large-scale digital pathology datasets is lacking. This work provides a benchmark for evaluating prevalent methods on multiple datasets by comparing the uncertainty estimates on both in-distribution and realistic near and far out-of-distribution (OOD) data on a whole-slide level. To this end, we aggregate uncertainty values from patch-based classifiers to whole-slide level uncertainty scores. We show that results found in classical computer vision benchmarks do not always translate to the medical imaging setting. Specifically, we demonstrate that deep ensembles perform best at detecting far-OOD data but can be outperformed on a more challenging near-OOD detection task by multi-head ensembles trained for optimal ensemble diversity. Furthermore, we demonstrate the harmful impact OOD data can have on the performance of deployed machine learning models. Overall, we show that uncertainty estimates can be used to discriminate in-distribution from OOD data with high AUC scores. Still, model deployment might require careful tuning based on prior knowledge of prospective OOD data.

Artificial intelligence in Dermatopathology

INTRODUCTION

Ever evolving research in medical field has reached an exciting stage with advent of newer technologies. With the introduction of digital microscopy, pathology has transitioned to become more digitally oriented speciality. The potential of artificial intelligence (AI) in dermatopathology is to aid the diagnosis, and it requires dermatopathologists' guidance for efficient functioning of artificial intelligence.

METHOD

Comprehensive literature search was performed using electronic online databases "PubMed" and "Google Scholar." Articles published in English language were considered for the review.

RESULTS

Convolutional neural network, a type of deep neural network, is considered as an ideal tool in image recognition, processing, classification, and segmentation. Implementation of AI in tumor pathology is involved in the diagnosis, grading, staging, and prognostic prediction as well as in identification of genetic or pathological features. In this review, we attempt to discuss the use of AI in dermatopathology, the attitude of patients and clinicians, its challenges, limitation, and potential opportunities in future implementation.

Diagnostic and Prognostic Deep Learning Applications for Histological Assessment of Cutaneous Melanoma

Melanoma is among the most devastating human malignancies. Accurate diagnosis and prognosis are essential to offer optimal treatment. Histopathology is the gold standard for establishing melanoma diagnosis and prognostic features. However, discrepancies often exist between pathologists, and analysis is costly and time-consuming. Deep-learning algorithms are deployed to improve melanoma diagnosis and prognostication from histological images of melanoma. In recent years, the development of these machine-learning tools has accelerated, and machine learning is poised to become a clinical tool to aid melanoma histology. Nevertheless, a review of the advances in machine learning in melanoma histology was lacking. We performed a comprehensive literature search to provide a complete overview of the recent advances in machine learning in the assessment of melanoma based on hematoxylin eosin digital pathology images. In our work, we review 37 recent publications, compare the methods and performance of the reviewed studies, and highlight the variety of promising machine-learning applications in melanoma histology.

Artificial intelligence: Its role in dermatopathology

Artificial intelligence (AI), a major frontier in the field of medical research, can potentially lead to a paradigm shift in clinical practice. A type of artificial intelligence system known as convolutional neural network points to the possible utility of deep learning in dermatopathology. Though pathology has been traditionally restricted to microscopes and glass slides, recent advancement in digital pathological imaging has led to a transition making it a potential branch for the implementation of artificial intelligence. The current application of artificial intelligence in dermatopathology is to complement the diagnosis and requires a well-trained dermatopathologist's guidance for better designing and development of deep learning algorithms. Here we review the recent advances of artificial intelligence in dermatopathology, its applications in disease diagnosis and in research, along with its limitations and future potential.

High-fidelity detection, subtyping, and localization of five skin neoplasms using supervised and semi-supervised learning

Background

Skin cancers are the most common malignancies diagnosed worldwide. While the early detection and treatment of pre-cancerous and cancerous skin lesions can dramatically improve outcomes, factors such as a global shortage of pathologists, increased workloads, and high rates of diagnostic discordance underscore the need for techniques that improve pathology workflows. Although AI models are now being used to classify lesions from whole slide images (WSIs), diagnostic performance rarely surpasses that of expert pathologists.

Objectives

The objective of the present study was to create an AI model to detect and classify skin lesions with a higher degree of sensitivity than previously demonstrated, with potential to match and eventually surpass expert pathologists to improve clinical workflows.

Methods

We combined supervised learning (SL) with semi-supervised learning (SSL) to produce an end-to-end multi-level skin detection system that not only detects 5 main types of skin lesions with high sensitivity and specificity, but also subtypes, localizes, and provides margin status to evaluate the proximity of the lesion to non-epidermal margins. The Supervised Training Subset consisted of 2188 random WSIs collected by the PathologyWatch (PW) laboratory between 2013 and 2018, while the Weakly Supervised Subset consisted of 5161 WSIs from daily case specimens. The Validation Set consisted of 250 curated daily case WSIs obtained from the PW tissue archives and included 50 "mimickers". The Testing Set (3821 WSIs) was composed of non-curated daily case specimens collected from July 20, 2021 to August 20, 2021 from PW laboratories.

Results

The performance characteristics of our AI model (i.e., Mihm) were assessed retrospectively by running the Testing Set through the Mihm Evaluation Pipeline. Our results show that the sensitivity of Mihm in classifying melanocytic lesions, basal cell carcinoma, and atypical squamous lesions, verruca vulgaris, and seborrheic keratosis was 98.91% (95% CI: 98.27%, 99.55%), 97.24% (95% CI: 96.15%, 98.33%), 95.26% (95% CI: 93.79%, 96.73%), 93.50% (95% CI: 89.14%, 97.86%), and 86.91% (95% CI: 82.13%, 91.69%), respectively. Additionally, our multi-level (i.e., patch-level, ROI-level, and WSI-level) detection algorithm includes a qualitative feature that subtypes lesions, an AI overlay in the front-end digital display that localizes diagnostic ROIs, and reports on margin status by detecting overlap between lesions and non-epidermal tissue margins.

Conclusions

Our AI model, developed in collaboration with dermatopathologists, detects 5 skin lesion types with higher sensitivity than previously published AI models, and provides end users with information such as subtyping, localization, and margin status in a front-end digital display. Our end-to-end system has the potential to improve pathology workflows by increasing diagnostic accuracy, expediting the course of patient care, and ultimately improving patient outcomes.

Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sections

Basal cell carcinoma (BCC), squamous cell carcinoma (SqCC) and melanoma are among the most common cancer types. Correct diagnosis based on histological evaluation after biopsy or excision is paramount for adequate therapy stratification. Deep learning on histological slides has been suggested to complement and improve routine diagnostics, but publicly available curated and annotated data and usable models trained to distinguish common skin tumors are rare and often lack heterogeneous non-tumor categories. A total of 16 classes from 386 cases were manually annotated on scanned histological slides, 129,364 100 x 100 µm (~395 x 395 px) image tiles were extracted and split into a training, validation and test set. An EfficientV2 neuronal network was trained and optimized to classify image categories. Cross entropy loss, balanced accuracy and Matthews correlation coefficient were used for model evaluation. Image and patient data were assessed with confusion matrices. Application of the model to an external set of whole slides facilitated localization of melanoma and non-tumor tissue. Automated differentiation of BCC, SqCC, melanoma, naevi and non-tumor tissue structures was possible, and a high diagnostic accuracy was achieved in the validation (98%) and test (97%) set. In summary, we provide a curated dataset including the most common neoplasms of the skin and various anatomical compartments to enable researchers to train, validate and improve deep learning models. Automated classification of skin tumors by deep learning techniques is possible with high accuracy, facilitates tumor localization and has the potential to support and improve routine diagnostics.

Deep learning-based framework for slide-based histopathological image analysis

Digital pathology coupled with advanced machine learning (e.g., deep learning) has been changing the paradigm of whole-slide histopathological images (WSIs) analysis. Major applications in digital pathology using machine learning include automatic cancer classification, survival analysis, and subtyping from pathological images. While most pathological image analyses are based on patch-wise processing due to the extremely large size of histopathology images, there are several applications that predict a single clinical outcome or perform pathological diagnosis per slide (e.g., cancer classification, survival analysis). However, current slide-based analyses are task-dependent, and a general framework of slide-based analysis in WSI has been seldom investigated. We propose a novel slide-based histopathology analysis framework that creates a WSI representation map, called HipoMap, that can be applied to any slide-based problems, coupled with convolutional neural networks. HipoMap converts a WSI of various shapes and sizes to structured image-type representation. Our proposed HipoMap outperformed existing methods in intensive experiments with various settings and datasets. HipoMap showed the Area Under the Curve (AUC) of 0.96±0.026 (5% improved) in the experiments for lung cancer classification, and c-index of 0.787±0.013 (3.5% improved) and coefficient of determination ([Formula: see text]) of 0.978±0.032 (24% improved) in survival analysis and survival prediction with TCGA lung cancer data respectively, as a general framework of slide-based analysis with a flexible capability. The results showed significant improvement comparing to the current state-of-the-art methods on each task. We further discussed experimental results of HipoMap as pathological viewpoints and verified the performance using publicly available TCGA datasets. A Python package is available at https://pypi.org/project/hipomap , and the package can be easily installed using Python PIP. The open-source codes in Python are available at: https://github.com/datax-lab/HipoMap .

[Deep learning-based fully automated intelligent and precise diagnosis for melanocytic lesions]

Melanocytic lesions occur on the surface of the skin, in which the malignant type is melanoma with a high fatality rate, seriously endangering human health. The histopathological analysis is the gold standard for diagnosis of melanocytic lesions. In this study, a fully automated intelligent diagnosis method based on deep learning was proposed to classify the pathological whole slide images (WSI) of melanocytic lesions. Firstly, the color normalization based on CycleGAN neural network was performed on multi-center pathological WSI; Secondly, ResNet-152 neural network-based deep convolutional network prediction model was built using 745 WSI; Then, a decision fusion model was cascaded, which calculates the average prediction probability of each WSI; Finally, the diagnostic performance of the proposed method was verified by internal and external test sets containing 182 and 54 WSI, respectively. Experimental results showed that the overall diagnostic accuracy of the proposed method reached 94.12% in the internal test set and exceeded 90% in the external test set. Furthermore, the color normalization method adopted was superior to the traditional color statistics-based and staining separation-based methods in terms of structure preservation and artifact suppression. The results demonstrate that the proposed method can achieve high precision and strong robustness in pathological WSI classification of melanocytic lesions, which has the potential in promoting the clinical application of computer-aided pathological diagnosis.

Current Developments of Artificial Intelligence in Digital Pathology and Its Future Clinical Applications in Gastrointestinal Cancers

The implementation of DP will revolutionize current practice by providing pathologists with additional tools and algorithms to improve workflow. Furthermore, DP will open up opportunities for development of AI-based tools for more precise and reproducible diagnosis through computational pathology. One of the key features of AI is its capability to generate perceptions and recognize patterns beyond the human senses. Thus, the incorporation of AI into DP can reveal additional morphological features and information. At the current rate of AI development and adoption of DP, the interest in computational pathology is expected to rise in tandem. There have already been promising developments related to AI-based solutions in prostate cancer detection; however, in the GI tract, development of more sophisticated algorithms is required to facilitate histological assessment of GI specimens for early and accurate diagnosis. In this review, we aim to provide an overview of the current histological practices in AP laboratories with respect to challenges faced in image preprocessing, present the existing AI-based algorithms, discuss their limitations and present clinical insight with respect to the application of AI in early detection and diagnosis of GI cancer.

Evaluation of a Deep Learning Approach to Differentiate Bowen's Disease and Seborrheic Keratosis

Background: Some of the most common cutaneous neoplasms are Bowen’s disease and seborrheic keratosis, a malignant and a benign proliferation, respectively. These entities represent a significant fraction of a dermatopathologists’ workload, and in some cases, histological differentiation may be challenging. The potential of deep learning networks to distinguish these diseases is assessed. Methods: In total, 1935 whole-slide images from three institutions were scanned on two different slide scanners. A U-Net-based segmentation deep learning algorithm was trained on data from one of the centers to differentiate Bowen’s disease, seborrheic keratosis, and normal tissue, learning from annotations performed by dermatopathologists. Optimal thresholds for the class distinction of diagnoses were extracted and assessed on a test set with data from all three institutions. Results: We aimed to diagnose Bowen’s diseases with the highest sensitivity. A good performance was observed across all three centers, underlining the model’s robustness. In one of the centers, the distinction between Bowen’s disease and all other diagnoses was achieved with an AUC of 0.9858 and a sensitivity of 0.9511. Seborrheic keratosis was detected with an AUC of 0.9764 and a sensitivity of 0.9394. Nevertheless, distinguishing irritated seborrheic keratosis from Bowen’s disease remained challenging. Conclusions: Bowen’s disease and seborrheic keratosis could be correctly identified by the evaluated deep learning model on test sets from three different centers, two of which were not involved in training, and AUC scores > 0.97 were obtained. The method proved robust to changes in the staining solution and scanner model. We believe this demonstrates that deep learning algorithms can aid in clinical routine; however, the results should be confirmed by qualified histopathologists.

Development of an Image Analysis-Based Prognosis Score Using Google's Teachable Machine in Melanoma

BACKGROUND

The increasing number of melanoma patients makes it necessary to establish new strategies for prognosis assessment to ensure follow-up care. Deep-learning-based image analysis of primary melanoma could be a future component of risk stratification.

OBJECTIVES

To develop a risk score for overall survival based on image analysis through artificial intelligence (AI) and validate it in a test cohort.

METHODS

Hematoxylin and eosin (H&E) stained sections of 831 melanomas, diagnosed from 2012-2015 were photographed and used to perform deep-learning-based group classification. For this purpose, the freely available software of Google's teachable machine was used. Five hundred patient sections were used as the training cohort, and 331 sections served as the test cohort.

RESULTS

Using Google's Teachable Machine, a prognosis score for overall survival could be developed that achieved a statistically significant prognosis estimate with an AUC of 0.694 in a ROC analysis based solely on image sections of approximately 250 × 250 µm. The prognosis group "low-risk" (n = 230) showed an overall survival rate of 93%, whereas the prognosis group "high-risk" (n = 101) showed an overall survival rate of 77.2%.

CONCLUSIONS

The study supports the possibility of using deep learning-based classification systems for risk stratification in melanoma. The AI assessment used in this study provides a significant risk estimate in melanoma, but it does not considerably improve the existing risk classification based on the TNM classification.

Streaming Convolutional Neural Networks for End-to-End Learning With Multi-Megapixel Images

Due to memory constraints on current hardware, most convolution neural networks (CNN) are trained on sub-megapixel images. For example, most popular datasets in computer vision contain images much less than a megapixel in size (0.09MP for ImageNet and 0.001MP for CIFAR-10). In some domains such as medical imaging, multi-megapixel images are needed to identify the presence of disease accurately. We propose a novel method to directly train convolutional neural networks using any input image size end-to-end. This method exploits the locality of most operations in modern convolutional neural networks by performing the forward and backward pass on smaller tiles of the image. In this work, we show a proof of concept using images of up to 66-megapixels (8192×8192), saving approximately 50GB of memory per image. Using two public challenge datasets, we demonstrate that CNNs can learn to extract relevant information from these large images and benefit from increasing resolution. We improved the area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for metastasis detection in breast cancer (CAMELYON17). We also obtained a Spearman correlation metric approaching state-of-the-art performance on the TUPAC16 dataset, from 0.485 (1MP) to 0.570 (16MP). Code to reproduce a subset of the experiments is available at https://github.com/DIAGNijmegen/StreamingCNN.

Artificial intelligence modelling in differentiating core biopsies of fibroadenoma from phyllodes tumor

Breast fibroepithelial lesions (FEL) are biphasic tumors which consist of benign fibroadenomas (FAs) and the rarer phyllodes tumors (PTs). FAs and PTs have overlapping features, but have different clinical management, which makes correct core biopsy diagnosis important. This study used whole-slide images (WSIs) of 187 FA and 100 PT core biopsies, to investigate the potential role of artificial intelligence (AI) in FEL diagnosis. A total of 9228 FA patches and 6443 PT patches was generated from WSIs of the training subset, with each patch being 224 × 224 pixel in size. Our model employed a two-stage architecture comprising a convolutional neural network (CNN) component for feature extraction from the patches, and a recurrent neural network (RNN) component for whole-slide classification using activation values from the global average pooling layer in the CNN model. It achieved an overall slide-level accuracy of 87.5%, with accuracies of 80% and 95% for FA and PT slides respectively. This affirms the potential role of AI in diagnostic discrimination between FA and PT on core biopsies which may be further refined for use in routine practice.

Artificial intelligence as a tool for diagnosis in digital pathology whole slide images: A systematic review

Digital pathology had a recent growth, stimulated by the implementation of digital whole slide images (WSIs) in clinical practice, and the pathology field faces shortage of pathologists in the last few years. This scenario created fronts of research applying artificial intelligence (AI) to help pathologists. One of them is the automated diagnosis, helping in the clinical decision support, increasing efficiency and quality of diagnosis. However, the complexity nature of the WSIs requires special treatments to create a reliable AI model for diagnosis. Therefore, we systematically reviewed the literature to analyze and discuss all the methods and results in AI in digital pathology performed in WSIs on H&E stain, investigating the capacity of AI as a diagnostic support tool for the pathologist in the routine real-world scenario. This review analyzes 26 studies, reporting in detail all the best methods to apply AI as a diagnostic tool, as well as the main limitations, and suggests new ideas to improve the AI field in digital pathology as a whole. We hope that this study could lead to a better use of AI as a diagnostic tool in pathology, helping future researchers in the development of new studies and projects.

Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology

Artificial intelligence (AI) solutions that automatically extract information from digital histology images have shown great promise for improving pathological diagnosis. Prior to routine use, it is important to evaluate their predictive performance and obtain regulatory approval. This assessment requires appropriate test datasets. However, compiling such datasets is challenging and specific recommendations are missing. A committee of various stakeholders, including commercial AI developers, pathologists, and researchers, discussed key aspects and conducted extensive literature reviews on test datasets in pathology. Here, we summarize the results and derive general recommendations on compiling test datasets. We address several questions: Which and how many images are needed? How to deal with low-prevalence subsets? How can potential bias be detected? How should datasets be reported? What are the regulatory requirements in different countries? The recommendations are intended to help AI developers demonstrate the utility of their products and to help pathologists and regulatory agencies verify reported performance measures. Further research is needed to formulate criteria for sufficiently representative test datasets so that AI solutions can operate with less user intervention and better support diagnostic workflows in the future.

Deep Learning Approaches and Applications in Toxicologic Histopathology: Current Status and Future Perspectives

Whole slide imaging enables the use of a wide array of digital image analysis tools that are revolutionizing pathology. Recent advances in digital pathology and deep convolutional neural networks have created an enormous opportunity to improve workflow efficiency, provide more quantitative, objective, and consistent assessments of pathology datasets, and develop decision support systems. Such innovations are already making their way into clinical practice. However, the progress of machine learning - in particular, deep learning (DL) - has been rather slower in nonclinical toxicology studies. Histopathology data from toxicology studies are critical during the drug development process that is required by regulatory bodies to assess drug-related toxicity in laboratory animals and its impact on human safety in clinical trials. Due to the high volume of slides routinely evaluated, low-throughput, or narrowly performing DL methods that may work well in small-scale diagnostic studies or for the identification of a single abnormality are tedious and impractical for toxicologic pathology. Furthermore, regulatory requirements around good laboratory practice are a major hurdle for the adoption of DL in toxicologic pathology. This paper reviews the major DL concepts, emerging applications, and examples of DL in toxicologic pathology image analysis. We end with a discussion of specific challenges and directions for future research.

Artificial intelligence in dermatology for the clinician

As medicine enters the era of artificial intelligence (AI)-augmented practice, dermatology is beginning to witness the integration of AI into the daily practice, particularly in the areas of diagnosis, prognosis, and treatment of skin diseases. Many of the current electronic medical records that dermatologists have incorporated provide guidance in billing, a form of AI at work. The recent advances in visual recognition AI make application and integration of the technology particularly suited for perceptual specialties such as radiology and dermatology. In dermatology, AI is poised to improve the efficiency and accuracy of traditional diagnostic approaches, including visual examination, skin biopsy, and histopathologic examination. This review highlights the current progress of AI in dermatology and provides a basic overview of the technology.

Deep learning-enabled whole slide imaging (DeepWSI): oil-immersion quality using dry objectives, longer depth of field, higher system throughput, and better functionality

Whole slide imaging (WSI) has moved the traditional manual slide inspection process to the era of digital pathology. A typical WSI system translates the sample to different positions and captures images using a high numerical aperture (NA) objective lens. Performing oil-immersion microscopy is a major obstacle for WSI as it requires careful liquid handling during the scanning process. Switching between dry objective and oil-immersion lens is often impossible as it disrupts the acquisition process. For a high-NA objective lens, the sub-micron depth of field also poses a challenge to acquiring in-focus images of samples with uneven topography. Additionally, it implies a small field of view for each tile, thus limiting the system throughput and resulting in a long acquisition time. Here we report a deep learning-enabled WSI platform, termed DeepWSI, to substantially improve the system performance and imaging throughput. With this platform, we show that images captured with a regular dry objective lens can be transformed into images comparable to that of a 1.4-NA oil immersion lens. Blurred images with defocus distance from -5 µm to +5 µm can be virtually refocused to the in-focus plane post measurement. We demonstrate an equivalent data throughput of >2 gigapixels per second, the highest among existing WSI systems. Using the same deep neural network, we also report a high-resolution virtual staining strategy and demonstrate it for Fourier ptychographic WSI. The DeepWSI platform may provide a turnkey solution for developing high-performance diagnostic tools for digital pathology.

Deep learning for bone marrow cell detection and classification on whole-slide images

Bone marrow (BM) examination is an essential step in both diagnosing and managing numerous hematologic disorders. BM nucleated differential count (NDC) analysis, as part of BM examination, holds the most fundamental and crucial information. However, there are many challenges to perform automated BM NDC analysis on whole-slide images (WSIs), including large dimensions of data to process, complicated cell types with subtle differences. To the authors best knowledge, this is the first study on fully automatic BM NDC using WSIs with 40x objective magnification, which can replace traditional manual counting relying on light microscopy via oil-immersion 100x objective lens with a total 1000x magnification. In this study, we develop an efficient and fully automatic hierarchical deep learning framework for BM NDC WSI analysis in seconds. The proposed hierarchical framework consists of (1) a deep learning model for rapid localization of BM particles and cellular trails generating regions of interest (ROI) for further analysis, (2) a patch-based deep learning model for cell identification of 16 cell types, including megakaryocytes, mitotic cells, and four stages of erythroblasts which have not been demonstrated in previous studies before, and (3) a fast stitching model for integrating patch-based results and producing final outputs. In evaluation, the proposed method is firstly tested on a dataset with a total of 12,426 annotated cells using cross validation, achieving high recall and accuracy of 0.905 ± 0.078 and 0.989 ± 0.006, respectively, and taking only 44 seconds to perform BM NDC analysis for a WSI. To further examine the generalizability of our model, we conduct an evaluation on the second independent dataset with a total of 3005 cells, and the results show that the proposed method also obtains high recall and accuracy of 0.842 and 0.988, respectively. In comparison with the existing small-image-based benchmark methods, the proposed method demonstrates superior performance in recall, accuracy and computational time.

Identification of nodal micrometastasis in colorectal cancer using deep learning on annotation-free whole-slide images

Detection of nodal micrometastasis (tumor size: 0.2-2.0 mm) is challenging for pathologists due to the small size of metastatic foci. Since lymph nodes with micrometastasis are counted as positive nodes, detecting micrometastasis is crucial for accurate pathologic staging of colorectal cancer. Previously, deep learning algorithms developed with manually annotated images performed well in identifying micrometastasis of breast cancer in sentinel lymph nodes. However, the process of manual annotation is labor intensive and time consuming. Multiple instance learning was later used to identify metastatic breast cancer without manual annotation, but its performance appears worse in detecting micrometastasis. Here, we developed a deep learning model using whole-slide images of regional lymph nodes of colorectal cancer with only a slide-level label (either a positive or negative slide). The training, validation, and testing sets included 1963, 219, and 1000 slides, respectively. A supercomputer TAIWANIA 2 was used to train a deep learning model to identify metastasis. At slide level, our algorithm performed well in identifying both macrometastasis (tumor size > 2.0 mm) and micrometastasis with an area under the receiver operating characteristics curve (AUC) of 0.9993 and 0.9956, respectively. Since most of our slides had more than one lymph node, we then tested the performance of our algorithm on 538 single-lymph node images randomly cropped from the testing set. At single-lymph node level, our algorithm maintained good performance in identifying macrometastasis and micrometastasis with an AUC of 0.9944 and 0.9476, respectively. Visualization using class activation mapping confirmed that our model identified nodal metastasis based on areas of tumor cells. Our results demonstrate for the first time that micrometastasis could be detected by deep learning on whole-slide images without manual annotation.

Machine Learning in Oncology: What Should Clinicians Know?

The volume and complexity of scientific and clinical data in oncology have grown markedly over recent years, including but not limited to the realms of electronic health data, radiographic and histologic data, and genomics. This growth holds promise for a deeper understanding of malignancy and, accordingly, more personalized and effective oncologic care. Such goals require, however, the development of new methods to fully make use of the wealth of available data. Improvements in computer processing power and algorithm development have positioned machine learning, a branch of artificial intelligence, to play a prominent role in oncology research and practice. This review provides an overview of the basics of machine learning and highlights current progress and challenges in applying this technology to cancer diagnosis, prognosis, and treatment recommendations, including a discussion of current takeaways for clinicians.

Automated Cervical Digitized Histology Whole-Slide Image Analysis Toolbox

Background:

Cervical intraepithelial neoplasia (CIN) is regarded as a potential precancerous state of the uterine cervix. Timely and appropriate early treatment of CIN can help reduce cervical cancer mortality. Accurate estimation of CIN grade correlated with human papillomavirus type, which is the primary cause of the disease, helps determine the patient's risk for developing the disease. Colposcopy is used to select women for biopsy. Expert pathologists examine the biopsied cervical epithelial tissue under a microscope. The examination can take a long time and is prone to error and often results in high inter-and intra-observer variability in outcomes.

Methodology:

We propose a novel image analysis toolbox that can automate CIN diagnosis using whole slide image (digitized biopsies) of cervical tissue samples. The toolbox is built as a four-step deep learning model that detects the epithelium regions, segments the detected epithelial portions, analyzes local vertical segment regions, and finally classifies each epithelium block with localized attention. We propose an epithelium detection network in this study and make use of our earlier research on epithelium segmentation and CIN classification to complete the design of the end-to-end CIN diagnosis toolbox.

Results:

The results show that automated epithelium detection and segmentation for CIN classification yields comparable results to manually segmented epithelium CIN classification.

Conclusion:

This highlights the potential as a tool for automated digitized histology slide image analysis to assist expert pathologists.

Detection of Prostate Cancer in Whole-Slide Images Through End-to-End Training With Image-Level Labels

Prostate cancer is the most prevalent cancer among men in Western countries, with 1.1 million new diagnoses every year. The gold standard for the diagnosis of prostate cancer is a pathologists' evaluation of prostate tissue. To potentially assist pathologists deep / learning / based cancer detection systems have been developed. Many of the state-of-the-art models are patch / based convolutional neural networks, as the use of entire scanned slides is hampered by memory limitations on accelerator cards. Patch-based systems typically require detailed, pixel-level annotations for effective training. However, such annotations are seldom readily available, in contrast to the clinical reports of pathologists, which contain slide-level labels. As such, developing algorithms which do not require manual pixel-wise annotations, but can learn using only the clinical report would be a significant advancement for the field. In this paper, we propose to use a streaming implementation of convolutional layers, to train a modern CNN (ResNet / 34) with 21 million parameters end-to-end on 4712 prostate biopsies. The method enables the use of entire biopsy images at high-resolution directly by reducing the GPU memory requirements by 2.4 TB. We show that modern CNNs, trained using our streaming approach, can extract meaningful features from high-resolution images without additional heuristics, reaching similar performance as state-of-the-art patch-based and multiple-instance learning methods. By circumventing the need for manual annotations, this approach can function as a blueprint for other tasks in histopathological diagnosis. The source code to reproduce the streaming models is available at https://github.com/DIAGNijmegen/ pathology-streaming-pipeline.

Artificial neural networks and pathologists recognize basal cell carcinomas based on different histological patterns

Recent advances in artificial intelligence, particularly in the field of deep learning, have enabled researchers to create compelling algorithms for medical image analysis. Histological slides of basal cell carcinomas (BCCs), the most frequent skin tumor, are accessed by pathologists on a daily basis and are therefore well suited for automated prescreening by neural networks for the identification of cancerous regions and swift tumor classification.In this proof-of-concept study, we implemented an accurate and intuitively interpretable artificial neural network (ANN) for the detection of BCCs in histological whole-slide images (WSIs). Furthermore, we identified and compared differences in the diagnostic histological features and recognition patterns relevant for machine learning algorithms vs. expert pathologists.An attention-ANN was trained with WSIs of BCCs to identify tumor regions (n = 820). The diagnosis-relevant regions used by the ANN were compared to regions of interest for pathologists, detected by eye-tracking techniques.This ANN accurately identified BCC tumor regions on images of histologic slides (area under the ROC curve: 0.993, 95% CI: 0.990-0.995; sensitivity: 0.965, 95% CI: 0.951-0.979; specificity: 0.910, 95% CI: 0.859-0.960). The ANN implicitly calculated a weight matrix, indicating the regions of a histological image that are important for the prediction of the network. Interestingly, compared to pathologists' eye-tracking results, machine learning algorithms rely on significantly different recognition patterns for tumor identification (p < 10^-4).To conclude, we found on the example of BCC WSIs, that histopathological images can be efficiently and interpretably analyzed by state-of-the-art machine learning techniques. Neural networks and machine learning algorithms can potentially enhance diagnostic precision in digital pathology and uncover hitherto unused classification patterns.

Deeply Supervised UNet for Semantic Segmentation to Assist Dermatopathological Assessment of Basal Cell Carcinoma

Accurate and fast assessment of resection margins is an essential part of a dermatopathologist's clinical routine. In this work, we successfully develop a deep learning method to assist the dermatopathologists by marking critical regions that have a high probability of exhibiting pathological features in whole slide images (WSI). We focus on detecting basal cell carcinoma (BCC) through semantic segmentation using several models based on the UNet architecture. The study includes 650 WSI with 3443 tissue sections in total. Two clinical dermatopathologists annotated the data, marking tumor tissues' exact location on 100 WSI. The rest of the data, with ground-truth sectionwise labels, are used to further validate and test the models. We analyze two different encoders for the first part of the UNet network and two additional training strategies: (a) deep supervision, (b) linear combination of decoder outputs, and obtain some interpretations about what the network's decoder does in each case. The best model achieves over 96%, accuracy, sensitivity, and specificity on the Test set.

Combining CNN-based histologic whole slide image analysis and patient data to improve skin cancer classification

BACKGROUND

Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses.

OBJECTIVES

We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone.

METHODS

We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier.

RESULTS

The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%.

CONCLUSION

In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy.

Multiplexed Immunohistochemistry and Digital Pathology as the Foundation for Next-Generation Pathology in Melanoma: Methodological Comparison and Future Clinical Applications

The state-of-the-art for melanoma treatment has recently witnessed an enormous revolution, evolving from a chemotherapeutic, "one-drug-for-all" approach, to a tailored molecular- and immunological-based approach with the potential to make personalized therapy a reality. Nevertheless, methods still have to improve a lot before these can reliably characterize all the tumoral features that make each patient unique. While the clinical introduction of next-generation sequencing has made it possible to match mutational profiles to specific targeted therapies, improving response rates to immunotherapy will similarly require a deep understanding of the immune microenvironment and the specific contribution of each component in a patient-specific way. Recent advancements in artificial intelligence and single-cell profiling of resected tumor samples are paving the way for this challenging task. In this review, we provide an overview of the state-of-the-art in artificial intelligence and multiplexed immunohistochemistry in pathology, and how these bear the potential to improve diagnostics and therapy matching in melanoma. A major asset of in-situ single-cell profiling methods is that these preserve the spatial distribution of the cells in the tissue, allowing researchers to not only determine the cellular composition of the tumoral microenvironment, but also study tissue sociology, making inferences about specific cell-cell interactions and visualizing distinctive cellular architectures - all features that have an impact on anti-tumoral response rates. Despite the many advantages, the introduction of these approaches requires the digitization of tissue slides and the development of standardized analysis pipelines which pose substantial challenges that need to be addressed before these can enter clinical routine.

Deep learning in cancer pathology: a new generation of clinical biomarkers

Clinical workflows in oncology rely on predictive and prognostic molecular biomarkers. However, the growing number of these complex biomarkers tends to increase the cost and time for decision-making in routine daily oncology practice; furthermore, biomarkers often require tumour tissue on top of routine diagnostic material. Nevertheless, routinely available tumour tissue contains an abundance of clinically relevant information that is currently not fully exploited. Advances in deep learning (DL), an artificial intelligence (AI) technology, have enabled the extraction of previously hidden information directly from routine histology images of cancer, providing potentially clinically useful information. Here, we outline emerging concepts of how DL can extract biomarkers directly from histology images and summarise studies of basic and advanced image analysis for cancer histology. Basic image analysis tasks include detection, grading and subtyping of tumour tissue in histology images; they are aimed at automating pathology workflows and consequently do not immediately translate into clinical decisions. Exceeding such basic approaches, DL has also been used for advanced image analysis tasks, which have the potential of directly affecting clinical decision-making processes. These advanced approaches include inference of molecular features, prediction of survival and end-to-end prediction of therapy response. Predictions made by such DL systems could simplify and enrich clinical decision-making, but require rigorous external validation in clinical settings.

Artificial intelligence in dermatopathology: Diagnosis, education, and research

Artificial intelligence (AI) utilizes computer algorithms to carry out tasks with human-like intelligence. Convolutional neural networks, a type of deep learning AI, can classify basal cell carcinoma, seborrheic keratosis, and conventional nevi, highlighting the potential for deep learning algorithms to improve diagnostic workflow in dermatopathology of highly routine diagnoses. Additionally, convolutional neural networks can support the diagnosis of melanoma and may help predict disease outcomes. Capabilities of machine learning in dermatopathology can extend beyond clinical diagnosis to education and research. Intelligent tutoring systems can teach visual diagnoses in inflammatory dermatoses, with measurable cognitive effects on learners. Natural language interfaces can instruct dermatopathology trainees to produce diagnostic reports that capture relevant detail for diagnosis in compliance with guidelines. Furthermore, deep learning can power computation- and population-based research. However, there are many limitations of deep learning that need to be addressed before broad incorporation into clinical practice. The current potential of AI in dermatopathology is to supplement diagnosis, and dermatopathologist guidance is essential for the development of useful deep learning algorithms. Herein, the recent progress of AI in dermatopathology is reviewed with emphasis on how deep learning can influence diagnosis, education, and research.

Quality control stress test for deep learning-based diagnostic model in digital pathology

Digital pathology provides a possibility for computational analysis of histological slides and automatization of routine pathological tasks. Histological slides are very heterogeneous concerning staining, sections' thickness, and artifacts arising during tissue processing, cutting, staining, and digitization. In this study, we digitally reproduce major types of artifacts. Using six datasets from four different institutions digitized by different scanner systems, we systematically explore artifacts' influence on the accuracy of the pre-trained, validated, deep learning-based model for prostate cancer detection in histological slides. We provide evidence that any histological artifact dependent on severity can lead to a substantial loss in model performance. Strategies for the prevention of diagnostic model accuracy losses in the context of artifacts are warranted. Stress-testing of diagnostic models using synthetically generated artifacts might be an essential step during clinical validation of deep learning-based algorithms.

Learning fine-grained estimation of physiological states from coarse-grained labels by distribution restoration

Due to its importance in clinical science, the estimation of physiological states (e.g., the severity of pathological tremor) has aroused growing interest in machine learning community. While the physiological state is a continuous variable, its continuity is lost when the physiological state is quantized into a few discrete classes during recording and labeling. The discreteness introduces misalignment between the true value and its label, meaning that these labels are unfortunately imprecise and coarse-grained. Most previous work did not consider the inaccuracy and directly utilized the coarse labels to train the machine learning algorithms, whose predictions are also coarse-grained. In this work, we propose to learn a precise, fine-grained estimation of physiological states using these coarse-grained ground truths. Established on mathematical rigorous proof, we utilize imprecise labels to restore the probabilistic distribution of precise labels in an approximate order-preserving fashion, then the deep neural network learns from this distribution and offers fine-grained estimation. We demonstrate the effectiveness of our approach in assessing the pathological tremor in Parkinson's Disease and estimating the systolic blood pressure from bioelectrical signals.