Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study

Accurate recognition of colorectal cancer with semi-supervised deep learning on pathological images

Machine-assisted pathological recognition has been focused on supervised learning (SL) that suffers from a significant annotation bottleneck. We propose a semi-supervised learning (SSL) method based on the mean teacher architecture using 13,111 whole slide images of colorectal cancer from 8803 subjects from 13 independent centers. SSL (~3150 labeled, ~40,950 unlabeled; ~6300 labeled, ~37,800 unlabeled patches) performs significantly better than the SL. No significant difference is found between SSL (~6300 labeled, ~37,800 unlabeled) and SL (~44,100 labeled) at patch-level diagnoses (area under the curve (AUC): 0.980 ± 0.014 vs. 0.987 ± 0.008, P value = 0.134) and patient-level diagnoses (AUC: 0.974 ± 0.013 vs. 0.980 ± 0.010, P value = 0.117), which is close to human pathologists (average AUC: 0.969). The evaluation on 15,000 lung and 294,912 lymph node images also confirm SSL can achieve similar performance as that of SL with massive annotations. SSL dramatically reduces the annotations, which has great potential to effectively build expert-level pathological artificial intelligence platforms in practice.

The role of machine learning applications in diagnosing and assessing critical and non-critical CHD: a scoping review

Machine learning uses historical data to make predictions about new data. It has been frequently applied in healthcare to optimise diagnostic classification through discovery of hidden patterns in data that may not be obvious to clinicians. Congenital Heart Defect (CHD) machine learning research entails one of the most promising clinical applications, in which timely and accurate diagnosis is essential. The objective of this scoping review is to summarise the application and clinical utility of machine learning techniques used in paediatric cardiology research, specifically focusing on approaches aiming to optimise diagnosis and assessment of underlying CHD. Out of 50 full-text articles identified between 2015 and 2021, 40% focused on optimising the diagnosis and assessment of CHD. Deep learning and support vector machine were the most commonly used algorithms, accounting for an overall diagnostic accuracy > 0.80. Clinical applications primarily focused on the classification of auscultatory heart sounds, transthoracic echocardiograms, and cardiac MRIs. The range of these applications and directions of future research are discussed in this scoping review.

Interpretable Diagnosis for Whole-Slide Melanoma Histology Images Using Convolutional Neural Network

At present, deep learning-based medical image diagnosis had achieved high performance in several diseases. However, the black-box nature of the convolutional neural network (CNN) limits their role in diagnosis. In this study, a novel interpretable diagnosis pipeline using the CNN model was proposed. Furthermore, a sizeable melanoma database that contains 841 digital whole-slide images (WSIs) was built to train and evaluate the model. The model achieved strong melanoma classification ability (0.962 areas under the receiver operating characteristic, 0.887 sensitivity, and 0.925 specificity). Moreover, the proposed model outperformed the existing schemes in terms of accuracy that is 20 pathologists (0.933 vs 0.732 accuracy). Finally, the gradient-weighted class activation mapping (Grad-CAM) method was used to show the inner logic of the proposed model and its feasibility to improve diagnosis process in healthcare. The mechanism of feature heat maps which is visualized through a saliency mapping has demonstrated that features learned or extracted by the proposed model are compatible with the accepted pathological features. Conclusively, the proposed model provides a rapid and accurate diagnosis by locating the distinctive features of melanoma to build doctors' trust in the CNNs' diagnosis results.

Deep embeddings and logistic regression for rapid active learning in histopathological images

BACKGROUND AND OBJECTIVE

Recognizing different tissue components is one of the most fundamental and essential works in digital pathology. Current methods are often based on convolutional neural networks (CNNs), which need numerous annotated samples for training. Creating large-scale histopathological datasets is labor-intensive, where interactive data annotation is a potential solution.

METHODS

We propose DELR (Deep Embedding-based Logistic Regression) to enable rapid model training and inference for histopathological image analysis. DELR utilizes a pretrained CNN to encode images as compact embeddings with low computational cost. The embeddings are then used to train a Logistic Regression model efficiently. We implemented DELR in an active learning framework, and validated it on three histopathological problems (binary, 4-category, and 8-category classification challenge for lung, breast, and colorectal cancer, respectively). We also investigated the influence of active learning strategy and type of the encoder.

RESULTS

On all the three datasets, DELR can achieve an area under curve (AUC) metric higher than 0.95 with only 100 image patches per class. Although its AUC is slightly lower than a fine-tuned CNN counterpart, DELR can be 536, 316, and 1481 times faster after pre-encoding. Moreover, DELR is proved to be compatible with a variety of active learning strategies and encoders.

CONCLUSIONS

DELR can achieve comparable accuracy to CNN with rapid running speed. These advantages make it a potential solution for real-time interactive data annotation.

A deep learning quantified stroma-immune score to predict survival of patients with stage II-III colorectal cancer

Background

Profound heterogeneity in prognosis has been observed in colorectal cancer (CRC) patients with intermediate levels of disease (stage II–III), advocating the identification of valuable biomarkers that could improve the prognostic stratification. This study aims to develop a deep learning-based pipeline for fully automatic quantification of immune infiltration within the stroma region on immunohistochemical (IHC) whole-slide images (WSIs) and further analyze its prognostic value in CRC.

Methods

Patients from two independent cohorts were divided into three groups: the development group (N = 200), the internal (N = 134), and the external validation group (N = 90). We trained a convolutional neural network for tissue classification of CD3 and CD8 stained WSIs. A scoring system, named stroma-immune score, was established by quantifying the density of CD3⁺ and CD8⁺ T-cells infiltration in the stroma region.

Results

Patients with higher stroma-immune scores had much longer survival. In the development group, 5-year survival rates of the low and high scores were 55.7% and 80.8% (hazard ratio [HR] for high vs. low 0.39, 95% confidence interval [CI] 0.24–0.63, P < 0.001). These results were confirmed in the internal and external validation groups with 5-year survival rates of low and high scores were 57.1% and 78.8%, 63.9% and 88.9%, respectively (internal: HR for high vs. low 0.49, 95% CI 0.28–0.88, P = 0.017; external: HR for high vs. low 0.35, 95% CI 0.15–0.83, P = 0.018). The combination of stroma-immune score and tumor-node-metastasis (TNM) stage showed better discrimination ability for survival prediction than using the TNM stage alone.

Conclusions

We proposed a stroma-immune score via a deep learning-based pipeline to quantify CD3⁺ and CD8⁺ T-cells densities within the stroma region on WSIs of CRC and further predict survival.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12935-021-02297-w.

Artificial intelligence in gastroenterology: A state-of-the-art review

The development of artificial intelligence (AI) has increased dramatically in the last 20 years, with clinical applications progressively being explored for most of the medical specialties. The field of gastroenterology and hepatology, substantially reliant on vast amounts of imaging studies, is not an exception. The clinical applications of AI systems in this field include the identification of premalignant or malignant lesions (e.g., identification of dysplasia or esophageal adenocarcinoma in Barrett’s esophagus, pancreatic malignancies), detection of lesions (e.g., polyp identification and classification, small-bowel bleeding lesion on capsule endoscopy, pancreatic cystic lesions), development of objective scoring systems for risk stratification, predicting disease prognosis or treatment response [e.g., determining survival in patients post-resection of hepatocellular carcinoma), determining which patients with inflammatory bowel disease (IBD) will benefit from biologic therapy], or evaluation of metrics such as bowel preparation score or quality of endoscopic examination. The objective of this comprehensive review is to analyze the available AI-related studies pertaining to the entirety of the gastrointestinal tract, including the upper, middle and lower tracts; IBD; the hepatobiliary system; and the pancreas, discussing the findings and clinical applications, as well as outlining the current limitations and future directions in this field.

Automated Diagnosis and Localization of Melanoma from Skin Histopathology Slides Using Deep Learning: A Multicenter Study

In traditional hospital systems, diagnosis and localization of melanoma are the critical challenges for pathological analysis, treatment instructions, and prognosis evaluation particularly in skin diseases. In literature, various studies have been reported to address these issues; however, a prominent smart diagnosis system is needed to be developed for the smart healthcare system. In this study, a deep learning-enabled diagnostic system is proposed and implemented that it has the capacity to automatically detect malignant melanoma in whole slide images (WSIs). In this system, the convolutional neural network (CNN), sophisticated statistical method, and image processing algorithms were integrated and implemented to locate benign and malignant lesions which are extremely useful in the diagnoses process of melanoma disease. To verify the exceptional performance of the proposed scheme, it is implemented in a multicenter database, which has 701 WSIs (641 WSIs from Central South University Xiangya Hospital (CSUXH) and 60 WSIs from the Cancer Genome Atlas (TCGA)). Experimental results have verified that the proposed system has achieved an area under the receiver operating characteristic curve (AUROC) of 0.971. Furthermore, the lesion area on the WSIs is represented by its degree of malignancy. These results show that the proposed system has the capacity to fully automate the diagnosis and localization problem of the melanoma in the smart healthcare systems.

Deep learning can predict lymph node status directly from histology in colorectal cancer

BACKGROUND

Lymph node status is a prognostic marker and strongly influences therapeutic decisions in colorectal cancer (CRC).

OBJECTIVES

The objective of the study is to investigate whether image features extracted by a deep learning model from routine histological slides and/or clinical data can be used to predict CRC lymph node metastasis (LNM).

METHODS

Using histological whole slide images (WSIs) of primary tumours of 2431 patients in the DACHS cohort, we trained a convolutional neural network to predict LNM. In parallel, we used clinical data derived from the same cases in logistic regression analyses. Subsequently, the slide-based artificial intelligence predictor (SBAIP) score was included in the regression. WSIs and data from 582 patients of the TCGA cohort were used as the external test set.

RESULTS

On the internal test set, the SBAIP achieved an area under receiver operating characteristic (AUROC) of 71.0%, the clinical classifier achieved an AUROC of 67.0% and a combination of the two classifiers yielded an improvement to 74.1%. Whereas the clinical classifier's performance remained stable on the TCGA set, performance of the SBAIP dropped to an AUROC of 61.2%. Performance of the clinical classifier depended strongly on the T stage.

CONCLUSION

Deep learning-based image analysis may help predict LNM of patients with CRC using routine histological slides. Combination with clinical data such as T stage might be useful. Strategies to increase performance of the SBAIP on external images should be investigated.

Self-supervised driven consistency training for annotation efficient histopathology image analysis

Training a neural network with a large labeled dataset is still a dominant paradigm in computational histopathology. However, obtaining such exhaustive manual annotations is often expensive, laborious, and prone to inter and intra-observer variability. While recent self-supervised and semi-supervised methods can alleviate this need by learning unsupervised feature representations, they still struggle to generalize well to downstream tasks when the number of labeled instances is small. In this work, we overcome this challenge by leveraging both task-agnostic and task-specific unlabeled data based on two novel strategies: (i) a self-supervised pretext task that harnesses the underlying multi-resolution contextual cues in histology whole-slide images to learn a powerful supervisory signal for unsupervised representation learning; (ii) a new teacher-student semi-supervised consistency paradigm that learns to effectively transfer the pretrained representations to downstream tasks based on prediction consistency with the task-specific unlabeled data. We carry out extensive validation experiments on three histopathology benchmark datasets across two classification and one regression based tasks, i.e., tumor metastasis detection, tissue type classification, and tumor cellularity quantification. Under limited-label data, the proposed method yields tangible improvements, which is close to or even outperforming other state-of-the-art self-supervised and supervised baselines. Furthermore, we empirically show that the idea of bootstrapping the self-supervised pretrained features is an effective way to improve the task-specific semi-supervised learning on standard benchmarks. Code and pretrained models are made available at: https://github.com/srinidhiPY/SSL_CR_Histo.

The Crohn's-like lymphoid reaction density: a new artificial intelligence quantified prognostic immune index in colon cancer

BACKGROUND

The Crohn's-like lymphoid reaction (CLR) is manifested as peritumoral lymphocytes aggregation in colon cancer, which is a major component of the host immune response to cancer. However, the lack of a unified and objective CLR evaluation standard limits its clinical application. We, therefore, developed a deep learning model for the fully automated CLR density quantification on routine hematoxylin and eosin (HE)-stained whole-slide images (WSIs) and further investigated its prognostic validity for patient stratification.

METHODS

The CLR density was calculated by using a deep learning method on HE-stained WSIs. A training (N = 279) and a validation (N = 194) cohorts were used to evaluate the prognostic value of CLR density for overall survival (OS).

RESULT

The fully automated quantified CLR density was an independent prognostic factor, with high CLR density associated with increased OS in the discovery (HR 0.58, 95% CI 0.38-0.89, P = 0.012) and validation cohort (0.45, 0.23-0.88, 0.020). Integrating CLR density into a Cox model with other risk factors showed improved prognostic capability.

CONCLUSION

We developed a new immune indicator (CLR density) quantified by a deep learning method to evaluate the lymphocytes aggregation in colon cancer. The CLR density was demonstrated its predictive value for OS in two independent cohorts. This approach allows for the objective and standardized quantification while reducing pathologists' workload. Therefore, this fully automated standardized method of CLR evaluation had potential clinical value.

Machine learning in gastrointestinal surgery

Machine learning (ML) is a collection of algorithms allowing computers to learn directly from data without predetermined equations. It is used widely to analyze "big data". In gastrointestinal surgery, surgeons deal with various data such as clinical parameters, surgical videos, and pathological images, to stratify surgical risk, perform safe surgery and predict patient prognosis. In the current "big data" era, the accelerating accumulation of a large amount of data drives studies using ML algorithms. Three subfields of ML are supervised learning, unsupervised learning, and reinforcement learning. In this review, we summarize applications of ML to surgical practice in the preoperative, intraoperative, and postoperative phases of care. Prediction and stratification using ML is promising; however, the current overarching concern is the availability of ML models. Information systems that can manage "big data" and integrate ML models into electronic health records are essential to incorporate ML into daily practice. ML is fundamental technology to meaningfully process data that exceeds the capacity of the human mind to comprehend. The accelerating accumulation of a large amount of data is changing the nature of surgical practice fundamentally. Artificial intelligence (AI), represented by ML, is being incorporated into daily surgical practice.

Application of an indocyanine green-mediated fluorescence imaging navigation system in detecting mice tumors

Background

Surgical operation plays an important role in the treatment of cancer. The success of the operation lies in the complete removal of the primary and disseminated tumor tissue while preserving the normal tissue. The development of optical molecular image navigation technology has provided a new option for intraoperative tumor visualization. In this study, a fluorescence imaging navigation system was used to detect the diameter of mice tumors and provide experimental evidence for the further development of digital diagnosis and treatment equipment.

Methods

The minimum detection concentration in vitro of the fluorescence imaging navigation system for indocyanine green (ICG) was first detected, then 120 female Institute of Cancer Research (ICR) mice and 120 female BALB/c nude mice were randomly divided into three groups by weight, high-dose (H, 4 mg/kg), middle-dose (M, 2 mg/kg), and low-dose (L, 1 mg/kg) groups of ICG solution. After inoculating solid tumors, high, medium, and low doses of ICG were injected via the tail vein, and the tumor diameter was measured by a fluorescence imaging navigation system and vernier caliper within 24 hours of injection.

Results

The minimum detectable diameter of the system could reach 0.2 mm compared with the vernier caliper, and the actual measurement error was within 0.2 mm.

Conclusions

A fluorescence imaging navigation system has high accuracy and sensitivity in the application of tumor detection, which may assist the clinical diagnosis and treatment of tumors.

Method of Tumor Pathological Micronecrosis Quantification Via Deep Learning From Label Fuzzy Proportions

The presence of necrosis is associated with tumor progression and patient outcomes in many cancers, but existing analyses rarely adopt quantitative methods because the manual quantification of histopathological features is too expensive. We aim to accurately identify necrotic regions on hematoxylin and eosin (HE)-stained slides and to calculate the ratio of necrosis with minimal annotations on the images. An adaptive method named Learning from Label Fuzzy Proportions (LLFP) was introduced to histopathological image analysis. Two datasets of liver cancer HE slides were collected to verify the feasibility of the method by training on the internal set using cross validation and performing validation on the external set, along with ensemble learning to improve performance. The models from cross validation performed relatively stably in identifying necrosis, with a Concordance Index of the Slide Necrosis Score (CISNS) of 0.9165±0.0089 in the internal test set. The integration model improved the CISNS to 0.9341 and achieved a CISNS of 0.8278 on the external set. There were significant differences in survival (p = 0.0060) between the three groups divided according to the calculated necrosis ratio. The proposed method can build an integration model good at distinguishing necrosis and capable of clinical assistance as an automatic tool to stratify patients with different risks or as a cluster tool for the quantification of histopathological features. We presented a method effective for identifying histopathological features and suggested that the extent of necrosis, especially micronecrosis, in liver cancer is related to patient outcomes.

Classifying Microscopic Acute and Old Myocardial Infarction Using Convolutional Neural Networks

ABSTRACT

Convolutional neural network (CNN) has advanced in recent years and translated from research into medical practice, most notably in clinical radiology and histopathology. Research on CNNs in forensic/postmortem pathology is almost exclusive to postmortem computed tomography despite the wealth of research into CNNs in surgical/anatomical histopathology. This study was carried out to investigate whether CNNs are able to identify and age myocardial infarction (a common example of forensic/postmortem histopathology) from histology slides. As a proof of concept, this study compared 4 CNNs commonly used in surgical/anatomical histopathology to identify normal myocardium from myocardial infarction. A total of 150 images of the myocardium (50 images each for normal myocardium, acute myocardial infarction, and old myocardial infarction) were used to train and test each CNN. One of the CNNs used (InceptionResNet v2) was able to show a greater than 95% accuracy in classifying normal myocardium from acute and old myocardial infarction. The result of this study is promising and demonstrates that CNN technology has potential applications as a screening and computer-assisted diagnostics tool in forensic/postmortem histopathology.

A Comprehensive Review on Radiomics and Deep Learning for Nasopharyngeal Carcinoma Imaging

Nasopharyngeal carcinoma (NPC) is one of the most common malignant tumours of the head and neck, and improving the efficiency of its diagnosis and treatment strategies is an important goal. With the development of the combination of artificial intelligence (AI) technology and medical imaging in recent years, an increasing number of studies have been conducted on image analysis of NPC using AI tools, especially radiomics and artificial neural network methods. In this review, we present a comprehensive overview of NPC imaging research based on radiomics and deep learning. These studies depict a promising prospect for the diagnosis and treatment of NPC. The deficiencies of the current studies and the potential of radiomics and deep learning for NPC imaging are discussed. We conclude that future research should establish a large-scale labelled dataset of NPC images and that studies focused on screening for NPC using AI are necessary.

Technological advances in cancer immunity: from immunogenomics to single-cell analysis and artificial intelligence

Immunotherapies play critical roles in cancer treatment. However, given that only a few patients respond to immune checkpoint blockades and other immunotherapeutic strategies, more novel technologies are needed to decipher the complicated interplay between tumor cells and the components of the tumor immune microenvironment (TIME). Tumor immunomics refers to the integrated study of the TIME using immunogenomics, immunoproteomics, immune-bioinformatics, and other multi-omics data reflecting the immune states of tumors, which has relied on the rapid development of next-generation sequencing. High-throughput genomic and transcriptomic data may be utilized for calculating the abundance of immune cells and predicting tumor antigens, referring to immunogenomics. However, as bulk sequencing represents the average characteristics of a heterogeneous cell population, it fails to distinguish distinct cell subtypes. Single-cell-based technologies enable better dissection of the TIME through precise immune cell subpopulation and spatial architecture investigations. In addition, radiomics and digital pathology-based deep learning models largely contribute to research on cancer immunity. These artificial intelligence technologies have performed well in predicting response to immunotherapy, with profound significance in cancer therapy. In this review, we briefly summarize conventional and state-of-the-art technologies in the field of immunogenomics, single-cell and artificial intelligence, and present prospects for future research.

Deep learning-based predictive biomarker of pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer

Background

Pathological complete response (pCR) is considered a surrogate endpoint for favorable survival in breast cancer patients treated with neoadjuvant chemotherapy (NAC). Predictive biomarkers of treatment response are crucial for guiding treatment decisions. With the hypothesis that histological information on tumor biopsy images could predict NAC response in breast cancer, we proposed a novel deep learning (DL)-based biomarker that predicts pCR from images of hematoxylin and eosin (H&E)-stained tissue and evaluated its predictive performance.

Methods

In total, 540 breast cancer patients receiving standard NAC were enrolled. Based on H&E-stained images, DL methods were employed to automatically identify tumor epithelium and predict pCR by scoring the identified tumor epithelium to produce a histopathological biomarker, the pCR-score. The predictive performance of the pCR-score was assessed and compared with that of conventional biomarkers including stromal tumor-infiltrating lymphocytes (sTILs) and subtype.

Results

The pCR-score derived from H&E staining achieved an area under the curve (AUC) of 0.847 in predicting pCR directly, and achieved accuracy, F1 score, and AUC of 0.853, 0.503, and 0.822 processed by the logistic regression method, respectively, higher than either sTILs or subtype; a prediction model of pCR constructed by integrating sTILs, subtype and pCR-score yielded a mean AUC of 0.890, outperforming the baseline sTIL-subtype model by 0.051 (0.839, P  =  0.001).

Conclusion

The DL-based pCR-score from histological images is predictive of pCR better than sTILs and subtype, and holds the great potentials for a more accurate stratification of patients for NAC.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12967-021-03020-z.

Gastrointestinal cancer classification and prognostication from histology using deep learning: Systematic review

BACKGROUND

Gastrointestinal cancers account for approximately 20% of all cancer diagnoses and are responsible for 22.5% of cancer deaths worldwide. Artificial intelligence-based diagnostic support systems, in particular convolutional neural network (CNN)-based image analysis tools, have shown great potential in medical computer vision. In this systematic review, we summarise recent studies reporting CNN-based approaches for digital biomarkers for characterization and prognostication of gastrointestinal cancer pathology.

METHODS

Pubmed and Medline were screened for peer-reviewed papers dealing with CNN-based gastrointestinal cancer analyses from histological slides, published between 2015 and 2020.Seven hundred and ninety titles and abstracts were screened, and 58 full-text articles were assessed for eligibility.

RESULTS

Sixteen publications fulfilled our inclusion criteria dealing with tumor or precursor lesion characterization or prognostic and predictive biomarkers: 14 studies on colorectal or rectal cancer, three studies on gastric cancer and none on esophageal cancer. These studies were categorised according to their end-points: polyp characterization, tumor characterization and patient outcome. Regarding the translation into clinical practice, we identified several studies demonstrating generalization of the classifier with external tests and comparisons with pathologists, but none presenting clinical implementation.

CONCLUSIONS

Results of recent studies on CNN-based image analysis in gastrointestinal cancer pathology are promising, but studies were conducted in observational and retrospective settings. Large-scale trials are needed to assess performance and predict clinical usefulness. Furthermore, large-scale trials are required for approval of CNN-based prediction models as medical devices.

Classification and Visualisation of Normal and Abnormal Radiographs; A Comparison between Eleven Convolutional Neural Network Architectures

This paper investigates the classification of radiographic images with eleven convolutional neural network (CNN) architectures (GoogleNet, VGG-19, AlexNet, SqueezeNet, ResNet-18, Inception-v3, ResNet-50, VGG-16, ResNet-101, DenseNet-201 and Inception-ResNet-v2). The CNNs were used to classify a series of wrist radiographs from the Stanford Musculoskeletal Radiographs (MURA) dataset into two classes-normal and abnormal. The architectures were compared for different hyper-parameters against accuracy and Cohen's kappa coefficient. The best two results were then explored with data augmentation. Without the use of augmentation, the best results were provided by Inception-ResNet-v2 (Mean accuracy = 0.723, Mean kappa = 0.506). These were significantly improved with augmentation to Inception-ResNet-v2 (Mean accuracy = 0.857, Mean kappa = 0.703). Finally, Class Activation Mapping was applied to interpret activation of the network against the location of an anomaly in the radiographs.

Deep learning for colon cancer histopathological images analysis

Nowadays, digital pathology plays a major role in the diagnosis and prognosis of tumours. Unfortunately, existing methods remain limited when faced with the high resolution and size of Whole Slide Images (WSIs) coupled with the lack of richly annotated datasets. Regarding the ability of the Deep Learning (DL) methods to cope with the large scale applications, such models seem like an appealing solution for tissue classification and segmentation in histopathological images. This paper focuses on the use of DL architectures to classify and highlight colon cancer regions in a sparsely annotated histopathological data context. First, we review and compare state-of-the-art Convolutional Neural networks (CNN) including the AlexNet, vgg, ResNet, DenseNet and Inception models. To cope with the shortage of rich WSI datasets, we have resorted to the use of transfer learning techniques. This strategy comes with the hallmark of relying on a large size computer vision dataset (ImageNet) to train the network and generate a rich collection of learnt features. The testing and evaluation of such models on our AiCOLO colon cancer dataset ensure accurate patch-level classification results reaching up to 96.98% accuracy rate with ResNet. The CNN models have also been tested and evaluated with the CRC-5000, nct-crc-he-100k and merged datasets. ResNet respectively achieves 96.77%, 99.76% and 99.98% for the three publicly available datasets. Then, we present a pixel-wise segmentation strategy for colon cancer WSIs through the use of both UNet and SegNet models. We introduce a multi-step training strategy as a remedy for the sparse annotation of histopathological images. UNet and SegNet are used and tested in different training scenarios including data augmentation and transfer learning and ensure up to 76.18% and 81.22% accuracy rates. Besides, we test our training strategy and models on the CRC-5000, nct-crc-he-100k and Warwick datasets. Respective accuracy rates of 98.66%, 99.12% and 78.39% were achieved by SegNet. Finally, we analyze the existing models to discover the most suitable network and the most effective training strategy for our colon tumour segmentation case study.¹.

Deep Learning of Histopathological Features for the Prediction of Tumour Molecular Genetics

Advanced diagnostics are enabling cancer treatments to become increasingly tailored to the individual through developments in immunotherapies and targeted therapies. However, long turnaround times and high costs of molecular testing hinder the widespread implementation of targeted cancer treatments. Meanwhile, gold-standard histopathological assessment carried out by a trained pathologist is widely regarded as routine and mandatory in most cancers. Recently, methods have been developed to mine hidden information from histopathological slides using deep learning applied to scanned and digitized slides; deep learning comprises a collection of computational methods which learn patterns in data in order to make predictions. Such methods have been reported to be successful in a variety of cancers for predicting the presence of biomarkers such as driver mutations, tumour mutational burden, and microsatellite instability. This information could prove valuable to pathologists and oncologists in clinical decision making for cancer treatment and triage for in-depth sequencing. In addition to identifying molecular features, deep learning has been applied to predict prognosis and treatment response in certain cancers. Despite reported successes, many challenges remain before the clinical implementation of such diagnostic strategies in the clinical setting is possible. This review aims to outline recent developments in the field of deep learning for predicting molecular genetics from histopathological slides, as well as to highlight limitations and pitfalls of working with histopathology slides in deep learning.

Machine Learning and Artificial Intelligence-driven Spatial Analysis of the Tumor Immune Microenvironment in Pathology Slides

A better understanding of the tumor immune microenvironment (TIME) could lead to accurate diagnosis, prognosis, and treatment stratification. Although molecular analyses at the tissue and/or single cell level could reveal the cellular status of the tumor microenvironment, these approaches lack information related to spatial-level cellular distribution, co-organization, and cell-cell interaction in the TIME. With the emergence of computational pathology coupled with machine learning (ML) and artificial intelligence (AI), ML- and AI-driven spatial TIME analyses of pathology images could revolutionize our understanding of the highly heterogeneous and complex molecular architecture of the TIME. In this review we highlight recent studies on spatial TIME analysis of pathology slides using state-of-the-art ML and AI algorithms. PATIENT SUMMARY: This mini-review reports recent advances in machine learning and artificial intelligence for spatial analysis of the tumor immune microenvironment in pathology slides. This information can help in understanding the spatial heterogeneity and organization of cells in patient tumors.

Artificial Intelligence for solid tumor diagnosis in digital pathology

Tumor diagnosis relies on the visual examination of histological slides by pathologists through a microscope eyepiece. Digital pathology, the digitalization of histological slides at high magnification with slides scanners, has raised the opportunity to extract quantitative information thanks to image analysis. In the last decade, medical image analysis has made exceptional progress due to the development of artificial intelligence (AI) algorithms. AI has been successfully used in the field of medical imaging and more recently in digital pathology. The feasibility and usefulness of AI assisted pathology tasks have been demonstrated in the very last years and we can expect those developments to be applied on routine histopathology in the future. In this review, we will describe and illustrate this technique and present the most recent applications in the field of tumor histopathology.

Role of artificial intelligence in multidisciplinary imaging diagnosis of gastrointestinal diseases

The use of artificial intelligence-based tools is regarded as a promising approach to increase clinical efficiency in diagnostic imaging, improve the interpretability of results, and support decision-making for the detection and prevention of diseases. Radiology, endoscopy and pathology images are suitable for deep-learning analysis, potentially changing the way care is delivered in gastroenterology. The aim of this review is to examine the key aspects of different neural network architectures used for the evaluation of gastrointestinal conditions, by discussing how different models behave in critical tasks, such as lesion detection or characterization (i.e. the distinction between benign and malignant lesions of the esophagus, the stomach and the colon). To this end, we provide an overview on recent achievements and future prospects in deep learning methods applied to the analysis of radiology, endoscopy and histologic whole-slide images of the gastrointestinal tract.

The impact of site-specific digital histology signatures on deep learning model accuracy and bias

The Cancer Genome Atlas (TCGA) is one of the largest biorepositories of digital histology. Deep learning (DL) models have been trained on TCGA to predict numerous features directly from histology, including survival, gene expression patterns, and driver mutations. However, we demonstrate that these features vary substantially across tissue submitting sites in TCGA for over 3,000 patients with six cancer subtypes. Additionally, we show that histologic image differences between submitting sites can easily be identified with DL. Site detection remains possible despite commonly used color normalization and augmentation methods, and we quantify the image characteristics constituting this site-specific digital histology signature. We demonstrate that these site-specific signatures lead to biased accuracy for prediction of features including survival, genomic mutations, and tumor stage. Furthermore, ethnicity can also be inferred from site-specific signatures, which must be accounted for to ensure equitable application of DL. These site-specific signatures can lead to overoptimistic estimates of model performance, and we propose a quadratic programming method that abrogates this bias by ensuring models are not trained and validated on samples from the same site.

Deep learning approach to predict sentinel lymph node status directly from routine histology of primary melanoma tumours

AIM

Sentinel lymph node status is a central prognostic factor for melanomas. However, the surgical excision involves some risks for affected patients. In this study, we therefore aimed to develop a digital biomarker that can predict lymph node metastasis non-invasively from digitised H&E slides of primary melanoma tumours.

METHODS

A total of 415 H&E slides from primary melanoma tumours with known sentinel node (SN) status from three German university hospitals and one private pathological practice were digitised (150 SN positive/265 SN negative). Two hundred ninety-one slides were used to train artificial neural networks (ANNs). The remaining 124 slides were used to test the ability of the ANNs to predict sentinel status. ANNs were trained and/or tested on data sets that were matched or not matched between SN-positive and SN-negative cases for patient age, ulceration, and tumour thickness, factors that are known to correlate with lymph node status.

RESULTS

The best accuracy was achieved by an ANN that was trained and tested on unmatched cases (61.8% ± 0.2%) area under the receiver operating characteristic (AUROC). In contrast, ANNs that were trained and/or tested on matched cases achieved (55.0% ± 3.5%) AUROC or less.

CONCLUSION

Our results indicate that the image classifier can predict lymph node status to some, albeit so far not clinically relevant, extent. It may do so by mostly detecting equivalents of factors on histological slides that are already known to correlate with lymph node status. Our results provide a basis for future research with larger data cohorts.

Deep Learning Techniques for the Classification of Colorectal Cancer Tissue

It is very important to make an objective evaluation of colorectal cancer histological images. Current approaches are generally based on the use of different combinations of textual features and classifiers to assess the classification performance, or transfer learning to classify different organizational types. However, since histological images contain multiple tissue types and characteristics, classification is still challenging. In this study, we proposed the best classification methodology based on the selected optimizer and modified the parameters of CNN methods. Then, we used deep learning technology to distinguish between healthy and diseased large intestine tissues. Firstly, we trained a neural network and compared the network architecture optimizers. Secondly, we modified the parameters of the network layer to optimize the superior architecture. Finally, we compared our well-trained deep learning methods on two different histological image open datasets, which comprised 5000 H&E images of colorectal cancer. The other dataset was composed of nine organizational categories of 100,000 images with an external validation of 7180 images. The results showed that the accuracy of the recognition of histopathological images was significantly better than that of existing methods. Therefore, this method is expected to have great potential to assist physicians to make clinical diagnoses and reduce the number of disparate assessments based on the use of artificial intelligence to classify colorectal cancer tissue.

Neural networks and robotic microneedles enable autonomous extraction of plant metabolites

Plant metabolites comprise a wide range of extremely important chemicals. In many cases, like savory spices, they combine distinctive functional properties-deterrence against herbivory-with an unmistakable flavor. Others have remarkable therapeutic qualities, for instance, the malaria drug artemisinin, or mechanical properties, for example natural rubber. We present a breakthrough in plant metabolite extraction technology. Using a neural network, we teach a computer how to recognize metabolite-rich cells of the herbal plant rosemary (Rosmarinus officinalis) and automatically extract the chemicals using a microrobot while leaving the rest of the plant undisturbed. Our approach obviates the need for chemical and mechanical separation and enables the extraction of plant metabolites that currently lack proper methods for efficient biomass use. Computer code required to train the neural network, identify regions of interest, and control the micromanipulator is available as part of the Supplementary Material.

Deep learning radiomics-based prediction of distant metastasis in patients with locally advanced rectal cancer after neoadjuvant chemoradiotherapy: A multicentre study

BACKGROUND

Accurate predictions of distant metastasis (DM) in locally advanced rectal cancer (LARC) patients receiving neoadjuvant chemoradiotherapy (nCRT) are helpful in developing appropriate treatment plans. This study aimed to perform DM prediction through deep learning radiomics.

METHODS

We retrospectively sampled 235 patients receiving nCRT with the minimum 36 months' postoperative follow-up from three hospitals. Through transfer learning, a deep learning radiomic signature (DLRS) based on multiparametric magnetic resonance imaging (MRI) was constructed. A nomogram was established integrating deep MRI information and clinicopathologic factors for better prediction. Harrell's concordance index (C-index) and time-dependent receiver operating characteristic (ROC) were used as performance metrics. Furthermore, the risk of DM in patients with different response to nCRT was evaluated with the nomogram.

FINDINGS

DLRS performed well in DM prediction, with a C-index of 0·747 and an area under curve (AUC) at three years of 0·894 in the validation cohort. The performance of nomogram was better, with a C-index of 0·775. In addition, the nomogram could stratify patients with different responses to nCRT into high- and low-risk groups of DM (P < 0·05).

INTERPRETATION

MRI-based deep learning radiomics had potential in predicting the DM of LARC patients receiving nCRT and could help evaluate the risk of DM in patients who have different responses to nCRT.

FUNDING

The funding bodies that contributed to this study are listed in the Acknowledgements section.

The value of the tumour-stroma ratio for predicting neoadjuvant chemoradiotherapy response in locally advanced rectal cancer: a case control study

BACKGROUND

The tumour-stroma ratio (TSR) is recognized as a practical prognostic factor in colorectal cancer. However, TSR assessment generally utilizes surgical specimens. This study aims to investigate whether the TSR evaluated from preoperative biopsy specimens by a semi-automatic quantification method can predict the response after neoadjuvant chemoradiotherapy (nCRT) of patients with locally advanced rectal cancer (LARC).

METHODS

A total of 248 consecutive patients diagnosed with LARC and treated with nCRT followed by resection were included. Haematoxylin and eosin (HE)-stained sections of biopsy specimens were collected, and the TSR was evaluated by a semi-automatic quantification method and was divided into three categories, using the cut-offs determined in the whole cohort to balance the proportion of patients in each category. The response to nCRT was evaluated on the primary tumour resection specimen by an expert pathologist using the four-tier tumour regression grade (TRG) system.

RESULTS

The TSR can discriminate patients that are major-responders (TRG 0-1) from patients that are non-responders (TRG 2-3). Patients were divided into stroma-low (33.5%), stroma-intermediate (33.9%), and stroma-high (32.7%) groups using 56.3 and 72.8% as the cutoffs. In the stroma-low group, 58 (69.9%) patients were major-responders, and only 39 (48.1%) patients were considered major-responders in the stroma-high group (P = 0.018). Multivariate analysis showed that the TSR was the only pre-treatment predictor of response to nCRT (adjusted odds ratio 0.40, 95% confidence interval 0.21-0.76, P = 0.002).

CONCLUSION

An elevated TSR in preoperative biopsy specimens is an independent predictor of nCRT response in LARC. This semi-automatic quantified TSR could be easily translated into routine pathologic assessment due to its reproducibility and reliability.

Requirements for implementation of artificial intelligence in the practice of gastrointestinal pathology

Tremendous advances in artificial intelligence (AI) in medical image analysis have been achieved in recent years. The integration of AI is expected to cause a revolution in various areas of medicine, including gastrointestinal (GI) pathology. Currently, deep learning algorithms have shown promising benefits in areas of diagnostic histopathology, such as tumor identification, classification, prognosis prediction, and biomarker/genetic alteration prediction. While AI cannot substitute pathologists, carefully constructed AI applications may increase workforce productivity and diagnostic accuracy in pathology practice. Regardless of these promising advances, unlike the areas of radiology or cardiology imaging, no histopathology-based AI application has been approved by a regulatory authority or for public reimbursement. Thus, implying that there are still some obstacles to be overcome before AI applications can be safely and effectively implemented in real-life pathology practice. The challenges have been identified at different stages of the development process, such as needs identification, data curation, model development, validation, regulation, modification of daily workflow, and cost-effectiveness balance. The aim of this review is to present challenges in the process of AI development, validation, and regulation that should be overcome for its implementation in real-life GI pathology practice.

A generalized deep learning framework for whole-slide image segmentation and analysis

Histopathology tissue analysis is considered the gold standard in cancer diagnosis and prognosis. Whole-slide imaging (WSI), i.e., the scanning and digitization of entire histology slides, are now being adopted across the world in pathology labs. Trained histopathologists can provide an accurate diagnosis of biopsy specimens based on WSI data. Given the dimensionality of WSIs and the increase in the number of potential cancer cases, analyzing these images is a time-consuming process. Automated segmentation of tumorous tissue helps in elevating the precision, speed, and reproducibility of research. In the recent past, deep learning-based techniques have provided state-of-the-art results in a wide variety of image analysis tasks, including the analysis of digitized slides. However, deep learning-based solutions pose many technical challenges, including the large size of WSI data, heterogeneity in images, and complexity of features. In this study, we propose a generalized deep learning-based framework for histopathology tissue analysis to address these challenges. Our framework is, in essence, a sequence of individual techniques in the preprocessing-training-inference pipeline which, in conjunction, improve the efficiency and the generalizability of the analysis. The combination of techniques we have introduced includes an ensemble segmentation model, division of the WSI into smaller overlapping patches while addressing class imbalances, efficient techniques for inference, and an efficient, patch-based uncertainty estimation framework. Our ensemble consists of DenseNet-121, Inception-ResNet-V2, and DeeplabV3Plus, where all the networks were trained end to end for every task. We demonstrate the efficacy and improved generalizability of our framework by evaluating it on a variety of histopathology tasks including breast cancer metastases (CAMELYON), colon cancer (DigestPath), and liver cancer (PAIP). Our proposed framework has state-of-the-art performance across all these tasks and is ranked within the top 5 currently for the challenges based on these datasets. The entire framework along with the trained models and the related documentation are made freely available at GitHub and PyPi. Our framework is expected to aid histopathologists in accurate and efficient initial diagnosis. Moreover, the estimated uncertainty maps will help clinicians to make informed decisions and further treatment planning or analysis.

Artificial intelligence-based pathology for gastrointestinal and hepatobiliary cancers

Artificial intelligence (AI) can extract complex information from visual data. Histopathology images of gastrointestinal (GI) and liver cancer contain a very high amount of information which human observers can only partially make sense of. Complementing human observers, AI allows an in-depth analysis of digitised histological slides of GI and liver cancer and offers a wide range of clinically relevant applications. First, AI can automatically detect tumour tissue, easing the exponentially increasing workload on pathologists. In addition, and possibly exceeding pathologist's capacities, AI can capture prognostically relevant tissue features and thus predict clinical outcome across GI and liver cancer types. Finally, AI has demonstrated its capacity to infer molecular and genetic alterations of cancer tissues from histological digital slides. These are likely only the first of many AI applications that will have important clinical implications. Thus, pathologists and clinicians alike should be aware of the principles of AI-based pathology and its ability to solve clinically relevant problems, along with its limitations and biases.

Deep learning-based tumor microenvironment analysis in colon adenocarcinoma histopathological whole-slide images

BACKGROUND AND OBJECTIVE

Colon cancer is a fatal disease, and a comprehensive understanding of the tumor microenvironment (TME) could lead to better risk stratification, prognosis prediction, and therapy management. In this paper, we focused on the automatic evaluation of TME in giga-pixel digital histopathology whole-slide images.

METHODS

A convolutional neural network is used to recognize nine different content presented in colon cancer whole-slide images. Several implementation details, including the foreground filtering and stain normalization are discussed. Based on the whole-slide segmentation, several TME descriptors are quantified and correlated with the clinical outcome by Kaplan-Meier analysis and Cox regression. Specifically, the stroma, tumor, necrosis, and lymphocyte components are discussed.

RESULTS

We validated the method on colon adenocarcinoma cases from The Cancer Genome Atlas project. The result shows that the stroma is an independent predictor of progression-free interval (PFI) after corrected by age and pathological stage, with a hazard ratio of 1.665 (95%CI: 1.110~2.495, p = 0.014). High-level necrosis component and lymphocytes component tend to be correlated with poor PFI, with a hazard ratio of 1.552 (95%CI: 0.943~2.554, p = 0.084) and 1.512 (95%CI: 0.979~2.336, p = 0.062), respectively.

CONCLUSIONS

The result reveals the complex role of the tumor microenvironment in colon adenocarcinoma, and the quantified descriptors are potential predictors of disease progression. The method could be considered for risk stratification and targeted therapy and extend to other types of cancer, leading to a better understanding of the tumor microenvironment.

State of machine and deep learning in histopathological applications in digestive diseases

Machine learning (ML)- and deep learning (DL)-based imaging modalities have exhibited the capacity to handle extremely high dimensional data for a number of computer vision tasks. While these approaches have been applied to numerous data types, this capacity can be especially leveraged by application on histopathological images, which capture cellular and structural features with their high-resolution, microscopic perspectives. Already, these methodologies have demonstrated promising performance in a variety of applications like disease classification, cancer grading, structure and cellular localizations, and prognostic predictions. A wide range of pathologies requiring histopathological evaluation exist in gastroenterology and hepatology, indicating these as disciplines highly targetable for integration of these technologies. Gastroenterologists have also already been primed to consider the impact of these algorithms, as development of real-time endoscopic video analysis software has been an active and popular field of research. This heightened clinical awareness will likely be important for future integration of these methods and to drive interdisciplinary collaborations on emerging studies. To provide an overview on the application of these methodologies for gastrointestinal and hepatological histopathological slides, this review will discuss general ML and DL concepts, introduce recent and emerging literature using these methods, and cover challenges moving forward to further advance the field.

Spatially Constrained Context-Aware Hierarchical Deep Correlation Filters for Nucleus Detection in Histology Images

Nucleus detection in histology images is a fundamental step for cellular-level analysis in computational pathology. In clinical practice, quantitative nuclear morphology can be used for diagnostic decision making, prognostic stratification, and treatment outcome prediction. Nucleus detection is a challenging task because of large variations in the shape of different types of nucleus such as nuclear clutter, heterogeneous chromatin distribution, and irregular and fuzzy boundaries. To address these challenges, we aim to accurately detect nuclei using spatially constrained context-aware correlation filters using hierarchical deep features extracted from multiple layers of a pre-trained network. During training, we extract contextual patches around each nucleus which are used as negative examples while the actual nucleus patch is used as a positive example. In order to spatially constrain the correlation filters, we propose to construct a spatial structural graph across different nucleus components encoding pairwise similarities. The correlation filters are constrained to act as eigenvectors of the Laplacian of the spatial graphs enforcing these to capture the nucleus structure. A novel objective function is proposed by embedding graph-based structural information as well as the contextual information within the discriminative correlation filter framework. The learned filters are constrained to be orthogonal to both the contextual patches and the spatial graph-Laplacian basis to improve the localization and discriminative performance. The proposed objective function trains a hierarchy of correlation filters on different deep feature layers to capture the heterogeneity in nuclear shape and texture. The proposed algorithm is evaluated on three publicly available datasets and compared with 15 current state-of-the-art methods demonstrating competitive performance in terms of accuracy, speed, and generalization.

Exploring prognostic indicators in the pathological images of hepatocellular carcinoma based on deep learning

OBJECTIVE

Tumour pathology contains rich information, including tissue structure and cell morphology, that reflects disease progression and patient survival. However, phenotypic information is subtle and complex, making the discovery of prognostic indicators from pathological images challenging.

DESIGN

An interpretable, weakly supervised deep learning framework incorporating prior knowledge was proposed to analyse hepatocellular carcinoma (HCC) and explore new prognostic phenotypes on pathological whole-slide images (WSIs) from the Zhongshan cohort of 1125 HCC patients (2451 WSIs) and TCGA cohort of 320 HCC patients (320 WSIs). A 'tumour risk score (TRS)' was established to evaluate patient outcomes, and then risk activation mapping (RAM) was applied to visualise the pathological phenotypes of TRS. The multi-omics data of The Cancer Genome Atlas(TCGA) HCC were used to assess the potential pathogenesis underlying TRS.

RESULTS

Survival analysis revealed that TRS was an independent prognosticator in both the Zhongshan cohort (p<0.0001) and TCGA cohort (p=0.0003). The predictive ability of TRS was superior to and independent of clinical staging systems, and TRS could evenly stratify patients into up to five groups with significantly different prognoses. Notably, sinusoidal capillarisation, prominent nucleoli and karyotheca, the nucleus/cytoplasm ratio and infiltrating inflammatory cells were identified as the main underlying features of TRS. The multi-omics data of TCGA HCC hint at the relevance of TRS to tumour immune infiltration and genetic alterations such as the FAT3 and RYR2 mutations.

CONCLUSION

Our deep learning framework is an effective and labour-saving method for decoding pathological images, providing a valuable means for HCC risk stratification and precise patient treatment.

Triage-driven diagnosis of Barrett's esophagus for early detection of esophageal adenocarcinoma using deep learning

Deep learning methods have been shown to achieve excellent performance on diagnostic tasks, but how to optimally combine them with expert knowledge and existing clinical decision pathways is still an open challenge. This question is particularly important for the early detection of cancer, where high-volume workflows may benefit from (semi-)automated analysis. Here we present a deep learning framework to analyze samples of the Cytosponge-TFF3 test, a minimally invasive alternative to endoscopy, for detecting Barrett's esophagus, which is the main precursor of esophageal adenocarcinoma. We trained and independently validated the framework on data from two clinical trials, analyzing a combined total of 4,662 pathology slides from 2,331 patients. Our approach exploits decision patterns of gastrointestinal pathologists to define eight triage classes of varying priority for manual expert review. By substituting manual review with automated review in low-priority classes, we can reduce pathologist workload by 57% while matching the diagnostic performance of experienced pathologists.

Perspectives in pathomics in head and neck cancer

PURPOSE OF REVIEW

Pathology is the cornerstone of cancer care. Pathomics, which represents the use of artificial intelligence in digital pathology, is an emerging and promising field that will revolutionize medical and surgical pathology in the coming years. This review provides an overview of pathomics, its current and future applications and its most relevant applications in Head and Neck cancer care.

RECENT FINDINGS

The number of studies investigating the use of artificial intelligence in pathology is rapidly growing, especially as the utilization of deep learning has shown great potential with Whole Slide Images. Even though numerous steps still remain before its clinical use, Pathomics has been used for varied applications comprising of computer-assisted diagnosis, molecular anomalies prediction, tumor microenvironment and biomarker identification as well as prognosis evaluation. The majority of studies were performed on the most frequent cancers, notably breast, prostate, and lung. Interesting results were also found in Head and Neck cancers.

SUMMARY

Even if its use in Head and Neck cancer care is still low, Pathomics is a powerful tool to improve diagnosis, identify prognostic factors and new biomarkers. Important challenges lie ahead before its use in a clinical practice, notably the lack of information on how AI makes its decisions, the slow deployment of digital pathology, and the need for extensively validated data in order to obtain authorities approval. Regardless, pathomics will most likely improve pathology in general, including Head and Neck cancer care in the coming years.

Deep learning in histopathology: the path to the clinic

Machine learning techniques have great potential to improve medical diagnostics, offering ways to improve accuracy, reproducibility and speed, and to ease workloads for clinicians. In the field of histopathology, deep learning algorithms have been developed that perform similarly to trained pathologists for tasks such as tumor detection and grading. However, despite these promising results, very few algorithms have reached clinical implementation, challenging the balance between hope and hype for these new techniques. This Review provides an overview of the current state of the field, as well as describing the challenges that still need to be addressed before artificial intelligence in histopathology can achieve clinical value.

Employing machine learning and microscopy images of AIB1-stained biopsy material to assess the 5-year survival of patients with colorectal cancer

Our purpose was to employ microscopy images of amplified in breast cancer 1 (AIB1)-stained biopsy material of patients with colorectal cancer (CRC) to: (a) find statistically significant differences (SSDs) in the texture and color of the epithelial gland tissue, between 5-year survivors and non-survivors after the first diagnosis and (b) employ machine learning (ML) methods for predicting the CRC-patient 5-year survival. We collected biopsy material from 54 patients with diagnosed CRC from the archives of the University Hospital of Patras, Greece. Twenty-six of the patients had survived 5 years after the first diagnosis. We selected regions of interest containing the epithelial gland at different microscope lens magnifications. We computed 69 textural and color features. Furthermore, we identified features with SSDs between the two groups of patients and we designed a supervised ML system for predicting the CRC-patient 5-year survival. Additionally, we employed the VGG16 pretrained convolution neural network to extract deep learning (DL) features, the support vector machines classifier, and the bootstrap cross-validation method for boosting the accuracy of predicting 5-year survival. Fourteen features sustained SSDs between the two groups of patients. The supervised ML system achieved 87% accuracy in predicting 5-year survival. In comparison, the DL system, using images from all magnifications, gave 97% classification accuracy. Glandular texture in 5-year non-survivors appeared to be of lower contrast, coarseness, roughness, local pixel correlation, and lower AIB1 variation, all indicating loss of textural definition. The supervised ML system revealed useful information regarding features that discriminate between 5-year survivors and non-survivors while the DL system displayed superior accuracy by employing DL features.

Artificial intelligence in gastroenterology and hepatology: Status and challenges

Originally proposed by John McCarthy in 1955, artificial intelligence (AI) has achieved a breakthrough and revolutionized the processing methods of clinical medicine with the increasing workloads of medical records and digital images. Doctors are paying attention to AI technologies for various diseases in the fields of gastroenterology and hepatology. This review will illustrate AI technology procedures for medical image analysis, including data processing, model establishment, and model validation. Furthermore, we will summarize AI applications in endoscopy, radiology, and pathology, such as detecting and evaluating lesions, facilitating treatment, and predicting treatment response and prognosis with excellent model performance. The current challenges for AI in clinical application include potential inherent bias in retrospective studies that requires larger samples for validation, ethics and legal concerns, and the incomprehensibility of the output results. Therefore, doctors and researchers should cooperate to address the current challenges and carry out further investigations to develop more accurate AI tools for improved clinical applications.

Feasibility of deep learning-based fully automated classification of microsatellite instability in tissue slides of colorectal cancer

High levels of microsatellite instability (MSI-H) occurs in about 15% of sporadic colorectal cancer (CRC) and is an important predictive marker for response to immune checkpoint inhibitors. To test the feasibility of a deep learning (DL)-based classifier as a screening tool for MSI status, we built a fully automated DL-based MSI classifier using pathology whole-slide images (WSIs) of CRCs. On small image patches of The Cancer Genome Atlas (TCGA) CRC WSI dataset, tissue/non-tissue, normal/tumor and MSS/MSI-H classifiers were applied sequentially for the fully automated prediction of the MSI status. The classifiers were also tested on an independent cohort. Furthermore, to test how the expansion of the training data affects the performance of the DL-based classifier, additional classifier trained on both TCGA and external datasets was tested. The areas under the receiver operating characteristic curves were 0.892 and 0.972 for the TCGA and external datasets, respectively, by a classifier trained on both datasets. The performance of the DL-based classifier was much better than that of previously reported histomorphology-based methods. We speculated that about 40% of CRC slides could be screened for MSI status without molecular testing by the DL-based classifier. These results demonstrated that the DL-based method has potential as a screening tool to discriminate molecular alteration in tissue slides.

Development and initial validation of a deep learning algorithm to quantify histological features in colorectal carcinoma including tumour budding/poorly differentiated clusters

AIMS

To develop and validate a deep learning algorithm to quantify a broad spectrum of histological features in colorectal carcinoma.

METHODS AND RESULTS

A deep learning algorithm was trained on haematoxylin and eosin-stained slides from tissue microarrays of colorectal carcinomas (N = 230) to segment colorectal carcinoma digitised images into 13 regions and one object. The segmentation algorithm demonstrated moderate to almost perfect agreement with interpretations by gastrointestinal pathologists, and was applied to an independent test cohort of digitised whole slides of colorectal carcinoma (N = 136). The algorithm correctly classified mucinous and high-grade tumours, and identified significant differences between mismatch repair-proficient and mismatch repair-deficient (MMRD) tumours with regard to mucin, inflammatory stroma, and tumour-infiltrating lymphocytes (TILs). A cutoff of >44.4 TILs per mm² carcinoma gave a sensitivity of 88% and a specificity of 73% in classifying MMRD carcinomas. Algorithm measures of tumour budding (TB) and poorly differentiated clusters (PDCs) outperformed TB grade derived from routine sign-out, and compared favourably with manual counts of TB/PDCs with regard to lymphatic, venous and perineural invasion. Comparable associations were seen between algorithm measures of TB/PDCs and manual counts of TB/PDCs for lymph node metastasis (all P < 0.001); however, stronger correlations were seen between the proportion of positive lymph nodes and algorithm measures of TB/PDCs. Stronger associations were also seen between distant metastasis and algorithm measures of TB/PDCs (P = 0.004) than between distant metastasis and TB (P = 0.04) and TB/PDC counts (P = 0.06).

CONCLUSIONS

Our results highlight the potential of deep learning to identify and quantify a broad spectrum of histological features in colorectal carcinoma.

Deep Learning Prediction of Metastasis in Locally Advanced Colon Cancer Using Binary Histologic Tumor Images

In this study, we developed the Binary ImaGe Colon Metastasis classifier (BIg-CoMet), a semi-guided approach for the stratification of colon cancer patients into two risk groups for the occurrence of distant metastasis, using an InceptionResNetV2-based deep learning model trained on binary images. We enrolled 291 colon cancer patients with pT3 and pT4 adenocarcinomas and converted one cytokeratin-stained representative tumor section per case into a binary image. Image augmentation and dropout layers were incorporated to avoid overfitting. In a validation collective (n = 128), BIg-CoMet was able to discriminate well between patients with and without metastasis (AUC: 0.842, 95% CI: 0.774-0.911). Further, the Kaplan-Meier curves of the metastasis-free survival showed a highly significant worse clinical course for the high-risk group (log-rank test: p < 0.001), and we demonstrated superiority over other established risk factors. A multivariable Cox regression analysis adjusted for confounders supported the use of risk groups as a prognostic factor for the occurrence of metastasis (hazard ratio (HR): 5.4, 95% CI: 2.5-11.7, p < 0.001). BIg-CoMet achieved good performance for both UICC subgroups, especially for UICC III (n = 53), with a positive predictive value of 80%. Our study demonstrates the ability to stratify colon cancer patients via a semi-guided process on images that primarily reflect tumor architecture.

Interpretable survival prediction for colorectal cancer using deep learning

Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease-specific survival for stage II and III colorectal cancer using 3652 cases (27,300 slides). When evaluated on two validation datasets containing 1239 cases (9340 slides) and 738 cases (7140 slides), respectively, the DLS achieved a 5-year disease-specific survival AUC of 0.70 (95% CI: 0.66–0.73) and 0.69 (95% CI: 0.64–0.72), and added significant predictive value to a set of nine clinicopathologic features. To interpret the DLS, we explored the ability of different human-interpretable features to explain the variance in DLS scores. We observed that clinicopathologic features such as T-category, N-category, and grade explained a small fraction of the variance in DLS scores (R² = 18% in both validation sets). Next, we generated human-interpretable histologic features by clustering embeddings from a deep-learning-based image-similarity model and showed that they explained the majority of the variance (R² of 73–80%). Furthermore, the clustering-derived feature most strongly associated with high DLS scores was also highly prognostic in isolation. With a distinct visual appearance (poorly differentiated tumor cell clusters adjacent to adipose tissue), this feature was identified by annotators with 87.0–95.5% accuracy. Our approach can be used to explain predictions from a prognostic deep learning model and uncover potentially-novel prognostic features that can be reliably identified by people for future validation studies.

Deep Transfer Learning Approach for Automatic Recognition of Drug Toxicity and Inhibition of SARS-CoV-2

Severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) causes COVID-19 and is responsible for the ongoing pandemic. Screening of potential antiviral drugs against SARS-CoV-2 depend on in vitro experiments, which are based on the quantification of the virus titer. Here, we used virus-induced cytopathic effects (CPE) in brightfield microscopy of SARS-CoV-2-infected monolayers to quantify the virus titer. Images were classified using deep transfer learning (DTL) that fine-tune the last layers of a pre-trained Resnet18 (ImageNet). To exclude toxic concentrations of potential drugs, the network was expanded to include a toxic score (TOX) that detected cell death (CPETOXnet). With this analytic tool, the inhibitory effects of chloroquine, hydroxychloroquine, remdesivir, and emetine were validated. Taken together we developed a simple method and provided open access implementation to quantify SARS-CoV-2 titers and drug toxicity in experimental settings, which may be adaptable to assays with other viruses. The quantification of virus titers from brightfield images could accelerate the experimental approach for antiviral testing.

Artificial Intelligence & Tissue Biomarkers: Advantages, Risks and Perspectives for Pathology

Tissue Biomarkers are information written in the tissue and used in Pathology to recognize specific subsets of patients with diagnostic, prognostic or predictive purposes, thus representing the key elements of Personalized Medicine. The advent of Artificial Intelligence (AI) promises to further reinforce the role of Pathology in the scenario of Personalized Medicine: AI-based devices are expected to standardize the evaluation of tissue biomarkers and also to discover novel information, which would otherwise be ignored by human review, and use them to make specific predictions. In this review we will present how AI has been used to support Tissue Biomarkers evaluation in the specific field of Pathology, give an insight to the intriguing field of AI-based biomarkers and discuss possible advantages, risk and perspectives for Pathology.

Targeted drug delivery strategies for precision medicines

Progress in the field of precision medicine has changed the landscape of cancer therapy. Precision medicine is propelled by technologies that enable molecular profiling, genomic analysis, and optimized drug design to tailor treatments for individual patients. Although precision medicines have resulted in some clinical successes, the use of many potential therapeutics has been hindered by pharmacological issues, including toxicities and drug resistance. Drug delivery materials and approaches have now advanced to a point where they can enable the modulation of a drug's pharmacological parameters without compromising the desired effect on molecular targets. Specifically, they can modulate a drug's pharmacokinetics, stability, absorption, and exposure to tumours and healthy tissues, and facilitate the administration of synergistic drug combinations. This Review highlights recent progress in precision therapeutics and drug delivery, and identifies opportunities for strategies to improve the therapeutic index of cancer drugs, and consequently, clinical outcomes.

Applying artificial intelligence in the microbiome for gastrointestinal diseases: A review

For a long time, gut bacteria have been recognized for their important roles in the occurrence and progression of gastrointestinal diseases like colorectal cancer, and the ever-increasing amounts of microbiome data combined with other high-quality clinical and imaging datasets are leading the study of gastrointestinal diseases into an era of biomedical big data. The "omics" technologies used for microbiome analysis continuously evolve, and the machine learning or artificial intelligence technologies are key to extract the relevant information from microbiome data. This review intends to provide a focused summary of recent research and applications of microbiome big data and to discuss the use of artificial intelligence to combat gastrointestinal diseases.