Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles

On image search in histopathology

Pathology images of histopathology can be acquired from camera-mounted microscopes or whole-slide scanners. Utilizing similarity calculations to match patients based on these images holds significant potential in research and clinical contexts. Recent advancements in search technologies allow for implicit quantification of tissue morphology across diverse primary sites, facilitating comparisons, and enabling inferences about diagnosis, and potentially prognosis, and predictions for new patients when compared against a curated database of diagnosed and treated cases. In this article, we comprehensively review the latest developments in image search technologies for histopathology, offering a concise overview tailored for computational pathology researchers seeking effective, fast, and efficient image search methods in their work.

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Network analysis of histopathological image features and genomics data improving prognosis performance in clear cell renal cell carcinoma

INTRODUCTION

Clear cell renal cell carcinoma is the most common type of kidney cancer, but the prediction of prognosis remains a challenge.

METHODS

We collected whole-slide histopathological images, corresponding clinical and genetic information from the The Cancer Imaging Archive and The Cancer Genome Atlas databases and randomly divided patients into training (n = 197) and validation (n = 84) cohorts. After feature extraction by CellProfiler, we used 2 different machine learning techniques (Least Absolute Shrinkage and Selector Operation-regularized Cox and Support Vector Machine-Recursive Feature Elimination) and weighted gene co-expression network analysis to select prognosis-related image features and genes, respectively. These features and genes were integrated into a joint model using random forest and used to create a nomogram that combines other predictive indicators.

RESULTS

A total of 4 overlapped features were identified, represented by the computed histopathological risk score in the random forest model, and showed predictive value for overall survival (test set: 1-year area under the curves (AUC) = 0.726, 3-year AUC = 0.727, and 5-year AUC = 0.764). The histopathological-genetic risk score (HGRS) integrating the genetic information computed performed better than the model that used image features only (test set: 1-year AUC = 0.682, 3-year AUC = 0.734, and 5-year AUC = 0.78). The nomogram (gender, stage, and HGRS) achieved the highest net benefit according to decision curve analysis compared to HGRS or clinical model.

CONCLUSION

This study developed a histopathological-genetic-related nomogram by combining histopathological features and clinical predictors, providing a more comprehensive prognostic assessment for clear cell renal cell carcinoma patients.

Computer-aided diagnosis system for grading brain tumor using histopathology images based on color and texture features

BACKGROUND

Cancer pathology shows disease development and associated molecular features. It provides extensive phenotypic information that is cancer-predictive and has potential implications for planning treatment. Based on the exceptional performance of computational approaches in the field of digital pathogenic, the use of rich phenotypic information in digital pathology images has enabled us to identify low-level gliomas (LGG) from high-grade gliomas (HGG). Because the differences between the textures are so slight, utilizing just one feature or a small number of features produces poor categorization results.

METHODS

In this work, multiple feature extraction methods that can extract distinct features from the texture of histopathology image data are used to compare the classification outcomes. The successful feature extraction algorithms GLCM, LBP, multi-LBGLCM, GLRLM, color moment features, and RSHD have been chosen in this paper. LBP and GLCM algorithms are combined to create LBGLCM. The LBGLCM feature extraction approach is extended in this study to multiple scales using an image pyramid, which is defined by sampling the image both in space and scale. The preprocessing stage is first used to enhance the contrast of the images and remove noise and illumination effects. The feature extraction stage is then carried out to extract several important features (texture and color) from histopathology images. Third, the feature fusion and reduction step is put into practice to decrease the number of features that are processed, reducing the computation time of the suggested system. The classification stage is created at the end to categorize various brain cancer grades. We performed our analysis on the 821 whole-slide pathology images from glioma patients in the Cancer Genome Atlas (TCGA) dataset. Two types of brain cancer are included in the dataset: GBM and LGG (grades II and III). 506 GBM images and 315 LGG images are included in our analysis, guaranteeing representation of various tumor grades and histopathological features.

RESULTS

The fusion of textural and color characteristics was validated in the glioma patients using the 10-fold cross-validation technique with an accuracy equals to 95.8%, sensitivity equals to 96.4%, DSC equals to 96.7%, and specificity equals to 97.1%. The combination of the color and texture characteristics produced significantly better accuracy, which supported their synergistic significance in the predictive model. The result indicates that the textural characteristics can be an objective, accurate, and comprehensive glioma prediction when paired with conventional imagery.

CONCLUSION

The results outperform current approaches for identifying LGG from HGG and provide competitive performance in classifying four categories of glioma in the literature. The proposed model can help stratify patients in clinical studies, choose patients for targeted therapy, and customize specific treatment schedules.

SPLICE: Streamlining Digital Pathology Image Processing

Digital pathology and the integration of artificial intelligence (AI) models have revolutionized histopathology, opening new opportunities. With the increasing availability of whole-slide images (WSIs), demand is growing for efficient retrieval, processing, and analysis of relevant images from vast biomedical archives. However, processing WSIs presents challenges due to their large size and content complexity. Full computer digestion of WSIs is impractical, and processing all patches individually is prohibitively expensive. In this article, we propose an unsupervised patching algorithm, Sequential Patching Lattice for Image Classification and Enquiry (SPLICE). This novel approach condenses a histopathology WSI into a compact set of representative patches, forming a collage of WSI while minimizing redundancy. SPLICE prioritizes patch quality and uniqueness by sequentially analyzing a WSI and selecting nonredundant representative features. We evaluated SPLICE for search and match applications, showing its improved accuracy, reduced computation time, and storage requirements compared with existing state-of-the-art methods. As an unsupervised method, SPLICE effectively reduces storage requirements for representing tissue images by 50%. This reduction enables numerous algorithms in computational pathology to operate much more efficiently, paving the way for accelerated adoption of digital pathology.

Dual-stream multi-dependency graph neural network enables precise cancer survival analysis

Histopathology image-based survival prediction aims to provide a precise assessment of cancer prognosis and can inform personalized treatment decision-making in order to improve patient outcomes. However, existing methods cannot automatically model the complex correlations between numerous morphologically diverse patches in each whole slide image (WSI), thereby preventing them from achieving a more profound understanding and inference of the patient status. To address this, here we propose a novel deep learning framework, termed dual-stream multi-dependency graph neural network (DM-GNN), to enable precise cancer patient survival analysis. Specifically, DM-GNN is structured with the feature updating and global analysis branches to better model each WSI as two graphs based on morphological affinity and global co-activating dependencies. As these two dependencies depict each WSI from distinct but complementary perspectives, the two designed branches of DM-GNN can jointly achieve the multi-view modeling of complex correlations between the patches. Moreover, DM-GNN is also capable of boosting the utilization of dependency information during graph construction by introducing the affinity-guided attention recalibration module as the readout function. This novel module offers increased robustness against feature perturbation, thereby ensuring more reliable and stable predictions. Extensive benchmarking experiments on five TCGA datasets demonstrate that DM-GNN outperforms other state-of-the-art methods and offers interpretable prediction insights based on the morphological depiction of high-attention patches. Overall, DM-GNN represents a powerful and auxiliary tool for personalized cancer prognosis from histopathology images and has great potential to assist clinicians in making personalized treatment decisions and improving patient outcomes.

Prognostic stratification of glioblastoma patients by unsupervised clustering of morphology patterns on whole slide images furthering our disease understanding

Introduction

Glioblastoma (GBM) is a highly aggressive malignant tumor of the central nervous system that displays varying molecular and morphological profiles, leading to challenging prognostic assessments. Stratifying GBM patients according to overall survival (OS) from H&E-stained whole slide images (WSI) using advanced computational methods is challenging, but with direct clinical implications.

Methods

This work is focusing on GBM (IDH-wildtype, CNS WHO Gr.4) cases, identified from the TCGA-GBM and TCGA-LGG collections after considering the 2021 WHO classification criteria. The proposed approach starts with patch extraction in each WSI, followed by comprehensive patch-level curation to discard artifactual content, i.e., glass reflections, pen markings, dust on the slide, and tissue tearing. Each patch is then computationally described as a feature vector defined by a pre-trained VGG16 convolutional neural network. Principal component analysis provides a feature representation of reduced dimensionality, further facilitating identification of distinct groups of morphology patterns, via unsupervised k-means clustering.

Results

The optimal number of clusters, according to cluster reproducibility and separability, is automatically determined based on the rand index and silhouette coefficient, respectively. Our proposed approach achieved prognostic stratification accuracy of 83.33% on a multi-institutional independent unseen hold-out test set with sensitivity and specificity of 83.33%.

Discussion

We hypothesize that the quantification of these clusters of morphology patterns, reflect the tumor's spatial heterogeneity and yield prognostic relevant information to distinguish between short and long survivors using a decision tree classifier. The interpretability analysis of the obtained results can contribute to furthering and quantifying our understanding of GBM and potentially improving our diagnostic and prognostic predictions.

Predicting gastric cancer tumor mutational burden from histopathological images using multimodal deep learning

Tumor mutational burden (TMB) is a significant predictive biomarker for selecting patients that may benefit from immune checkpoint inhibitor therapy. Whole exome sequencing is a common method for measuring TMB; however, its clinical application is limited by the high cost and time-consuming wet-laboratory experiments and bioinformatics analysis. To address this challenge, we downloaded multimodal data of 326 gastric cancer patients from The Cancer Genome Atlas, including histopathological images, clinical data and various molecular data. Using these data, we conducted a comprehensive analysis to investigate the relationship between TMB, clinical factors, gene expression and image features extracted from hematoxylin and eosin images. We further explored the feasibility of predicting TMB levels, i.e. high and low TMB, by utilizing a residual network (Resnet)-based deep learning algorithm for histopathological image analysis. Moreover, we developed a multimodal fusion deep learning model that combines histopathological images with omics data to predict TMB levels. We evaluated the performance of our models against various state-of-the-art methods using different TMB thresholds and obtained promising results. Specifically, our histopathological image analysis model achieved an area under curve (AUC) of 0.749. Notably, the multimodal fusion model significantly outperformed the model that relied only on histopathological images, with the highest AUC of 0.971. Our findings suggest that histopathological images could be used with reasonable accuracy to predict TMB levels in gastric cancer patients, while multimodal deep learning could achieve even higher levels of accuracy. This study sheds new light on predicting TMB in gastric cancer patients.

Marker-controlled watershed algorithm and fuzzy C-means clustering machine learning: automated segmentation of glioblastoma from MRI images in a case series

Introduction and importance

Automated segmentation of glioblastoma multiforme (GBM) from MRI images is crucial for accurate diagnosis and treatment planning. This paper presents a new and innovative approach for automating the segmentation of GBM from MRI images using the marker-controlled watershed segmentation (MCWS) algorithm.

Case presentation and methods

The technique involves several image processing techniques, including adaptive thresholding, morphological filtering, gradient magnitude calculation, and regional maxima identification. The MCWS algorithm efficiently segments images based on local intensity structures using the watershed transform, and fuzzy c-means (FCM) clustering improves segmentation accuracy. The presented approach achieved improved segmentation accuracy in detecting and segmenting GBM tumours from axial T2-weighted (T2-w) MRI images, as demonstrated by the mean characteristics performance metrics for GBM segmentation (sensitivity: 0.9905, specificity: 0.9483, accuracy: 0.9508, precision: 0.5481, F_measure: 0.7052, and jaccard: 0.9340).

Clinical discussion

The results of this study underline the importance of reliable and accurate image segmentation for effective diagnosis and treatment planning of GBM tumours.

Conclusion

The MCWS technique provides an effective and efficient approach for the segmentation of challenging medical images.

A multi-class brain tumor grading system based on histopathological images using a hybrid YOLO and RESNET networks

Gliomas are primary brain tumors caused by glial cells. These cancers' classification and grading are crucial for prognosis and treatment planning. Deep learning (DL) can potentially improve the digital pathology investigation of brain tumors. In this paper, we developed a technique for visualizing a predictive tumor grading model on histopathology pictures to help guide doctors by emphasizing characteristics and heterogeneity in forecasts. The proposed technique is a hybrid model based on YOLOv5 and ResNet50. The function of YOLOv5 is to localize and classify the tumor in large histopathological whole slide images (WSIs). The suggested technique incorporates ResNet into the feature extraction of the YOLOv5 framework, and the detection results show that our hybrid network is effective for identifying brain tumors from histopathological images. Next, we estimate the glioma grades using the extreme gradient boosting classifier. The high-dimensional characteristics and nonlinear interactions present in histopathology images are well-handled by this classifier. DL techniques have been used in previous computer-aided diagnosis systems for brain tumor diagnosis. However, by combining the YOLOv5 and ResNet50 architectures into a hybrid model specifically designed for accurate tumor localization and predictive grading within histopathological WSIs, our study presents a new approach that advances the field. By utilizing the advantages of both models, this creative integration goes beyond traditional techniques to produce improved tumor localization accuracy and thorough feature extraction. Additionally, our method ensures stable training dynamics and strong model performance by integrating ResNet50 into the YOLOv5 framework, addressing concerns about gradient explosion. The proposed technique is tested using the cancer genome atlas dataset. During the experiments, our model outperforms the other standard ways on the same dataset. Our results indicate that the proposed hybrid model substantially impacts tumor subtype discrimination between low-grade glioma (LGG) II and LGG III. With 97.2% of accuracy, 97.8% of precision, 98.6% of sensitivity, and the Dice similarity coefficient of 97%, the proposed model performs well in classifying four grades. These results outperform current approaches for identifying LGG from high-grade glioma and provide competitive performance in classifying four categories of glioma in the literature.

Explainable survival analysis with uncertainty using convolution-involved vision transformer

Image-based precision medicine research is able to help doctors make better decisions on treatments. Among all kinds of medical images, a special form is called Whole Slide Image (WSI), which is used for diagnosing patients with cancer, aiming to enable more accurate survival prediction with its high resolution. However, One unique challenge of the WSI-based prediction models is processing the gigabyte-size or even terabyte-size WSIs, which would make most models computationally infeasible. Although existing models mostly use a pre-selected subset of key patches or patch clusters as input, they might discard some important morphology information, making the prediction inferior. Another challenge is improving the prediction models' explainability, which is crucial to help doctors understand the predictions given by the models and make faithful decisions with high confidence. To address the above two challenges, in this work, we propose a novel explainable survival prediction model based on Vision Transformer. Specifically, we adopt dual-channel convolutional layers to utilize the complete WSIs for more accurate predictions. We also introduce the aleatoric uncertainty into our model to understand its limitation and avoid overconfidence in using the prediction results. Additionally, we present a post-hoc explainable method to identify the most salient patches and distinct morphology features as supporting evidence for predictions. Evaluations of two large cancer datasets show that our proposed model is able to make survival predictions more effectively and has better explainability for cancer diagnosis.

Tumor Mutation Burden-Related Histopathologic Features for Predicting Overall Survival in Gliomas Using Graph Deep Learning

Tumor mutation burden (TMB) is a potential biomarker for evaluating the prognosis and response to immune checkpoint inhibitors, but its costly and time-consuming method of measurement limits its widespread application. This study aimed to identify the TMB-related histopathologic features from hematoxylin and eosin slides and explore their prognostic value in gliomas. TMB-related features were detected using a graph convolutional neural network from whole-slide images of patients from The Cancer Genome Atlas data set (619 patients), and the correlation between features and TMB was evaluated in an external validation set (237 patients). TMB-related features were used for predicting overall survival (OS) of patients to investigate whether these features have potential for prognostic prediction. Moreover, biological pathways underlying the prognostic value of the features were further explored. Histopathologic features derived from whole-slide images were significantly associated with patient TMB (P = 0.007 in the external validation set). TMB-related features showed excellent performance for OS prediction, and patients with lower-grade gliomas could be further stratified into different risk groups according to the features (P = 0.00013; hazard ratio, 4.004). Pathways involved in the cell cycle and execution of immune response were enriched in patients with higher OS risk. The TMB-related features could be used to estimate TMB and aid in prognostic risk stratification of patients with glioma with dysregulated biological pathways.

Surformer: An interpretable pattern-perceptive survival transformer for cancer survival prediction from histopathology whole slide images

BACKGROUND AND OBJECTIVE

High-resolution histopathology whole slide images (WSIs) contain abundant valuable information for cancer prognosis. However, most computational pathology methods for survival prediction have weak interpretability and cannot explain the decision-making processes reasonably. To address this issue, we propose a highly interpretable neural network termed pattern-perceptive survival transformer (Surformer) for cancer survival prediction from WSIs.

METHODS

Notably, Surformer can quantify specific histological patterns through bag-level labels without any patch/cell-level auxiliary information. Specifically, the proposed ratio-reserved cross-attention module (RRCA) generates global and local features with the learnable prototypes (pglobal, plocals) as detectors and quantifies the patches correlative to each plocal in the form of ratio factors (rfs). Afterward, multi-head self&cross-attention modules proceed with the computation for feature enhancement against noise. Eventually, the designed disentangling loss function guides multiple local features to focus on distinct patterns, thereby assisting rfs from RRCA in achieving more explicit histological feature quantification.

RESULTS

Extensive experiments on five TCGA datasets illustrate that Surformer outperforms existing state-of-the-art methods. In addition, we highlight its interpretation by visualizing rfs distribution across high-risk and low-risk cohorts and retrieving and analyzing critical histological patterns contributing to the survival prediction.

CONCLUSIONS

Surformer is expected to be exploited as a useful tool for performing histopathology image data-driven analysis and gaining new insights for interpreting the associations between such images and patient survival states.

Exploration and Enhancement of Classifiers in the Detection of Lung Cancer from Histopathological Images

Lung cancer is a prevalent malignancy that impacts individuals of all genders and is often diagnosed late due to delayed symptoms. To catch it early, researchers are developing algorithms to study lung cancer images. The primary objective of this work is to propose a novel approach for the detection of lung cancer using histopathological images. In this work, the histopathological images underwent preprocessing, followed by segmentation using a modified approach of KFCM-based segmentation and the segmented image intensity values were dimensionally reduced using Particle Swarm Optimization (PSO) and Grey Wolf Optimization (GWO). Algorithms such as KL Divergence and Invasive Weed Optimization (IWO) are used for feature selection. Seven different classifiers such as SVM, KNN, Random Forest, Decision Tree, Softmax Discriminant, Multilayer Perceptron, and BLDC were used to analyze and classify the images as benign or malignant. Results were compared using standard metrics, and kappa analysis assessed classifier agreement. The Decision Tree Classifier with GWO feature extraction achieved good accuracy of 85.01% without feature selection and hyperparameter tuning approaches. Furthermore, we present a methodology to enhance the accuracy of the classifiers by employing hyperparameter tuning algorithms based on Adam and RAdam. By combining features from GWO and IWO, and using the RAdam algorithm, the Decision Tree classifier achieves the commendable accuracy of 91.57%.

Lung Cancer Classification in Histopathology Images Using Multiresolution Efficient Nets

Histopathological images are very effective for investigating the status of various biological structures and diagnosing diseases like cancer. In addition, digital histopathology increases diagnosis precision and provides better image quality and more detail for the pathologist with multiple viewing options and team annotations. As a result of the benefits above, faster treatment is available, increasing therapy success rates and patient recovery and survival chances. However, the present manual examination of these images is tedious and time-consuming for pathologists. Therefore, reliable automated techniques are needed to effectively classify normal and malignant cancer images. This paper applied a deep learning approach, namely, EfficientNet and its variants from B0 to B7. We used different image resolutions for each model, from 224 × 224 pixels to 600 × 600 pixels. We also applied transfer learning and parameter tuning techniques to improve the results and overcome the overfitting problem. We collected the dataset from the Lung and Colon Cancer Histopathological Image LC25000 image dataset. The dataset acquisition consists of 25,000 histopathology images of five classes (lung adenocarcinoma, lung squamous cell carcinoma, benign lung tissue, colon adenocarcinoma, and colon benign tissue). Then, we performed preprocessing on the dataset to remove the noisy images and bring them into a standard format. The model's performance was evaluated in terms of classification accuracy and loss. We have achieved good accuracy results for all variants; however, the results of EfficientNetB2 stand excellent, with an accuracy of 97% for 260 × 260 pixels resolution images.

Explainable Convolutional Neural Networks for Brain Cancer Detection and Localisation

Brain cancer is widely recognised as one of the most aggressive types of tumors. In fact, approximately 70% of patients diagnosed with this malignant cancer do not survive. In this paper, we propose a method aimed to detect and localise brain cancer, starting from the analysis of magnetic resonance images. The proposed method exploits deep learning, in particular convolutional neural networks and class activation mapping, in order to provide explainability by highlighting the areas of the medical image related to brain cancer (from the model point of view). We evaluate the proposed method with 3000 magnetic resonances using a free available dataset. The results we obtained are encouraging. We reach an accuracy ranging from 97.83% to 99.67% in brain cancer detection by exploiting four different models: VGG16, ResNet50, Alex_Net, and MobileNet, thus showing the effectiveness of the proposed method.

High-Order Correlation-Guided Slide-Level Histology Retrieval With Self-Supervised Hashing

Histopathological Whole Slide Images (WSIs) play a crucial role in cancer diagnosis. It is of significant importance for pathologists to search for images sharing similar content with the query WSI, especially in the case-based diagnosis. While slide-level retrieval could be more intuitive and practical in clinical applications, most methods are designed for patch-level retrieval. A few recently unsupervised slide-level methods only focus on integrating patch features directly, without perceiving slide-level information, and thus severely limits the performance of WSI retrieval. To tackle the issue, we propose a High-Order Correlation-Guided Self-Supervised Hashing-Encoding Retrieval (HSHR) method. Specifically, we train an attention-based hash encoder with slide-level representation in a self-supervised manner, enabling it to generate more representative slide-level hash codes of cluster centers and assign weights for each. These optimized and weighted codes are leveraged to establish a similarity-based hypergraph, in which a hypergraph-guided retrieval module is adopted to explore high-order correlations in the multi-pairwise manifold to conduct WSI retrieval. Extensive experiments on multiple TCGA datasets with over 24,000 WSIs spanning 30 cancer subtypes demonstrate that HSHR achieves state-of-the-art performance compared with other unsupervised histology WSI retrieval methods.

The value of deep neural networks in the pathological classification of thyroid tumors

BACKGROUND

To explore the distinguishing diagnostic value and clinical application potential of deep neural networks (DNN) for pathological images of thyroid tumors.

METHODS

A total of 799 pathological thyroid images of 559 patients with thyroid tumors were retrospectively analyzed. The pathological types included papillary thyroid carcinoma (PTC), medullary thyroid carcinoma (MTC), follicular thyroid carcinoma (FTC), adenomatous goiter, adenoma, and normal thyroid gland. The dataset was divided into a training set and a test set. Resnet50, Resnext50, EfficientNet, and Densenet121 were trained using the training set data and tested with the test set data to determine the diagnostic efficiency of different pathology types and to further analyze the causes of misdiagnosis.

RESULTS

The recall, precision, negative predictive value (NPV), accuracy, specificity, and F1 scores of the four models ranged from 33.33% to 100.00%. The area under curve (AUC) ranged from 0.822 to 0.994, and the Kappa coefficient ranged from 0.7508 to 0.7713. However, the performance of diagnosing FTC, adenoma, and adenomatous goiter was slightly inferior to other types of pathological tissues.

CONCLUSION

The DNN model achieved satisfactory results in the task of classifying thyroid tumors by learning thyroid pathology images. These results indicate the potential of the DNN model for the efficient diagnosis of thyroid tumor histopathology.

Artificial Intelligence-Assisted Classification of Gliomas Using Whole Slide Images

CONTEXT.—

Glioma is the most common primary brain tumor in adults. The diagnosis and grading of different pathological subtypes of glioma is essential in treatment planning and prognosis.

OBJECTIVE.—

To propose a deep learning-based approach for the automated classification of glioma histopathology images. Two classification methods, the ensemble method based on 2 binary classifiers and the multiclass method using a single multiclass classifier, were implemented to classify glioma images into astrocytoma, oligodendroglioma, and glioblastoma, according to the 5th edition of the World Health Organization classification of central nervous system tumors, published in 2021.

DESIGN.—

We tested 2 different deep neural network architectures (VGG19 and ResNet50) and extensively validated the proposed approach based on The Cancer Genome Atlas data set (n = 700). We also studied the effects of stain normalization and data augmentation on the glioma classification task.

RESULTS.—

With the binary classifiers, our model could distinguish astrocytoma and oligodendroglioma (combined) from glioblastoma with an accuracy of 0.917 (area under the curve [AUC] = 0.976) and astrocytoma from oligodendroglioma (accuracy = 0.821, AUC = 0.865). The multiclass method (accuracy = 0.861, AUC = 0.961) outperformed the ensemble method (accuracy = 0.847, AUC = 0.933) with the best performance displayed by the ResNet50 architecture.

CONCLUSIONS.—

With the high performance of our model (>80%), the proposed method can assist pathologists and physicians to support examination and differential diagnosis of glioma histopathology images, with the aim to expedite personalized medical care.

Classification of colorectal cancer consensus molecular subtypes using attention-based multi-instance learning network on whole-slide images

Colorectal cancer (CRC) is the third most common and second most lethal cancer globally. It is highly heterogeneous with different clinical-pathological characteristics, prognostic status, and therapy responses. Thus, the precise diagnosis of CRC subtypes is of great significance for improving the prognosis and survival of CRC patients. Nowadays, the most commonly used molecular-level CRC classification system is the Consensus Molecular Subtypes (CMSs). In this study, we applied a weakly supervised deep learning method, named attention-based multi-instance learning (MIL), on formalin-fixed paraffin-embedded (FFPE) whole-slide images (WSIs) to distinguish CMS1 subtype from CMS2, CMS3, and CMS4 subtypes, as well as distinguish CMS4 from CMS1, CMS2, and CMS3 subtypes. The advantage of MIL is training a bag of the tiled instance with bag-level labels only. Our experiment was performed on 1218 WSIs obtained from The Cancer Genome Atlas (TCGA). We constructed three convolutional neural network-based structures for model training and evaluated the ability of the max-pooling operator and mean-pooling operator on aggregating bag-level scores. The results showed that the 3-layer model achieved the best performance in both comparison groups. When compared CMS1 with CMS234, max-pooling reached the ACC of 83.86 % and the mean-pooling operator reached the AUC of 0.731. While comparing CMS4 with CMS123, mean-pooling reached the ACC of 74.26 % and max-pooling reached the AUC of 0.609. Our results implied that WSIs could be utilized to classify CMSs, and manual pixel-level annotation is not a necessity for computational pathology imaging analysis.

Region of interest (ROI) selection using vision transformer for automatic analysis using whole slide images

Selecting regions of interest (ROI) is a common step in medical image analysis across all imaging modalities. An ROI is a subset of an image appropriate for the intended analysis and identified manually by experts. In modern pathology, the analysis involves processing multidimensional and high resolution whole slide image (WSI) tiles automatically with an overwhelming quantity of structural and functional information. Despite recent improvements in computing capacity, analyzing such a plethora of data is challenging but vital to accurate analysis. Automatic ROI detection can significantly reduce the number of pixels to be processed, speed the analysis, improve accuracy and reduce dependency on pathologists. In this paper, we present an ROI detection method for WSI and demonstrated it for human epidermal growth factor receptor 2 (HER2) grading for breast cancer patients. Existing HER2 grading relies on manual ROI selection, which is tedious, time-consuming and suffers from inter-observer and intra-observer variability. This study found that the HER2 grade changes with ROI selection. We proposed an ROI detection method using Vision Transformer and investigated the role of image magnification for ROI detection. This method yielded an accuracy of 99% using 20 × WSI and 97% using 10 × WSI for the ROI detection. In the demonstration, the proposed method increased the diagnostic agreement to 99.3% with the clinical scores and reduced the time to 15 seconds for automated HER2 grading.

Contribution of whole slide imaging-based deep learning in the assessment of intraoperative and postoperative sections in neuropathology

The pathological diagnosis of intracranial germinoma (IG), oligodendroglioma, and low-grade astrocytoma on intraoperative frozen section (IFS) and hematoxylin-eosin (HE)-staining section directly determines patients' treatment options, but it is a difficult task for pathologists. We aimed to investigate whether whole-slide imaging (WSI)-based deep learning can contribute new precision to the diagnosis of IG, oligodendroglioma, and low-grade astrocytoma. Two types of WSIs (500 IFSs and 832 HE-staining sections) were collected from 379 patients at multiple medical centers. Patients at Center 1 were split into the training, testing, and internal validation sets (3:1:1), while the other centers were the external validation sets. First, we subdivided WSIs into small tiles and selected tissue tiles using a tissue tile selection model. Then a tile-level classification model was established, and the majority voting method was used to determine the final diagnoses. Color jitter was applied to the tiles so that the deep learning (DL) models could adapt to the variations in the staining. Last, we investigated the effectiveness of model assistance. The internal validation accuracies of the IFS and HE models were 93.9% and 95.3%, respectively. The external validation accuracies of the IFS and HE models were 82.0% and 76.9%, respectively. Furthermore, the IFS and HE models can predict Ki-67 positive cell areas with R² of 0.81 and 0.86, respectively. With model assistance, the IFS and HE diagnosis accuracy of pathologists improved from 54.6%-69.7% and 53.5%-83.7% to 87.9%-93.9% and 86.0%-90.7%, respectively. Both the IFS model and the HE model can differentiate the three tumors, predict the expression of Ki-67, and improve the diagnostic accuracy of pathologists. The use of our model can assist clinicians in providing patients with optimal and timely treatment options.

Intraoperative cytological diagnosis of brain tumours: A preliminary study using a deep learning model

BACKGROUND

Intraoperative pathological diagnosis of central nervous system (CNS) tumours is essential to planning patient management in neuro-oncology. Frozen section slides and cytological preparations provide architectural and cellular information that is analysed by pathologists to reach an intraoperative diagnosis. Progress in the fields of artificial intelligence and machine learning means that AI systems have significant potential for the provision of highly accurate real-time diagnosis in cytopathology.

OBJECTIVE

To investigate the efficiency of machine-learning models in the intraoperative cytological diagnosis of CNS tumours.

MATERIALS AND METHODS

We trained a deep neural network to classify biopsy material for intraoperative tissue diagnosis of four major brain lesions. Overall, 205 medical images were obtained from squash smear slides of histologically correlated cases, with 18 high-grade and 11 low-grade gliomas, 17 metastatic carcinomas, and 9 non-neoplastic pathological brain tissue samples. The neural network model was trained and evaluated using 5-fold cross-validation.

RESULTS

The model achieved 95% and 97% diagnostic accuracy in the patch-level classification and patient-level classification tasks, respectively.

CONCLUSIONS

We conclude that deep learning-based classification of cytological preparations may be a promising complementary method for the rapid and accurate intraoperative diagnosis of CNS tumours.

Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology

Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.

Self-supervised learning-based Multi-Scale feature Fusion Network for survival analysis from whole slide images

Understanding prognosis and mortality is critical for evaluating the treatment plan of patients. Advances in digital pathology and deep learning techniques have made it practical to perform survival analysis in whole slide images (WSIs). Current methods are usually based on a multi-stage framework which includes patch sampling, feature extraction and prediction. However, the random patch sampling strategy is highly unstable and prone to sampling non-ROI. Feature extraction typically relies on hand-crafted features or convolutional neural networks (CNNs) pre-trained on ImageNet, while the artificial error or domain gaps may affect the survival prediction performance. Besides, the limited information representation of local sampling patches will create a bottleneck limitation on the effectiveness of prediction. To address the above challenges, we propose a novel patch sampling strategy based on image information entropy and construct a Multi-Scale feature Fusion Network (MSFN) based on self-supervised feature extractor. Specifically, we adopt image information entropy as a criterion to select representative sampling patches, thereby avoiding the noise interference caused by random to blank regions. Meanwhile, we pretrain the feature extractor utilizing self-supervised learning mechanism to improve the efficiency of feature extraction. Furthermore, a global-local feature fusion prediction network based on the attention mechanism is constructed to improve the survival prediction effect of WSIs with comprehensive multi-scale information representation. The proposed method is validated by adequate experiments and achieves competitive results on both of the most popular WSIs survival analysis datasets, TCGA-GBM and TCGA-LUSC. Code and trained models are made available at: https://github.com/Mercuriiio/MSFN.

A semi-supervised learning approach with consistency regularization for tumor histopathological images analysis

Introduction

Manual inspection of histopathological images is important in clinical cancer diagnosis. Pathologists implement pathological diagnosis and prognostic evaluation through the microscopic examination of histopathological slices. This entire process is time-consuming, laborious, and challenging for pathologists. The modern use of whole-slide imaging, which scans histopathology slides to digital slices, and analysis using computer-aided diagnosis is an essential problem.

Methods

To solve the problem of difficult labeling of histopathological data, and improve the flexibility of histopathological analysis in clinical applications, we herein propose a semi-supervised learning algorithm coupled with consistency regularization strategy, called"Semi- supervised Histopathology Analysis Network"(Semi-His-Net), for automated normal-versus-tumor and subtype classifications. Specifically, when inputted disturbing versions of the same image, the model should predict similar outputs. Based on this, the model itself can assign artificial labels to unlabeled data for subsequent model training, thereby effectively reducing the labeled data required for training.

Results

Our Semi-His-Net is able to classify patches from breast cancer histopathological images into normal tissue and three other different tumor subtypes, achieving an accuracy was 90%. The average AUC of cross-classification between tumors reached 0.893.

Discussion

To overcome the limitations of visual inspection by pathologists for histopathology images, such as long time and low repeatability, we have developed a deep learning-based framework (Semi-His-Net) for automatic classification subdivision of the subtypes contained in the whole pathological images. This learning-based framework has great potential to improve the efficiency and repeatability of histopathological image diagnosis.

Stain normalization using score-based diffusion model through stain separation and overlapped moving window patch strategies

Hematoxylin and eosin (H&E) staining is the gold standard modality for diagnosis in medicine. However, the dosage ratio of hematoxylin to eosin in H&E staining has not been standardized yet. Additionally, H&E stains fade out at various speeds. Therefore, the staining quality could differ among each image, and stain normalization is a critical preprocessing approach for training deep learning (DL) models, especially in long-term and/or multicenter digital pathology studies. However, conventional methods for stain normalization have some significant drawbacks, such as collapsing in the structure and/or texture of tissue. In addition, conventional methods must require a reference patch or slide. Meanwhile, DL-based methods have a risk of overfitting and/or grid artifacts. We developed a score-based diffusion model of colorization for stain normalization. However, mistransfer, in which the model confuses hematoxylin with eosin, can occur using a score-based diffusion model due to its high diversity nature. To overcome this mistransfer, we propose a stain separation method using sparse non-negative matrix factorization (SNMF), which can decompose pathology slide into Hematoxylin and Eosin to normalize each stain component. Furthermore, inpainting with overlapped moving window patches was used to prevent grid artifacts of whole slide image normalization. Our method can normalize the whole slide pathology images through this stain normalization pipeline with decent performance.

Radio-pathomic approaches in pediatric neuro-oncology: Opportunities and challenges

With medical software platforms moving to cloud environments with scalable storage and computing, the translation of predictive artificial intelligence (AI) models to aid in clinical decision-making and facilitate personalized medicine for cancer patients is becoming a reality. Medical imaging, namely radiologic and histologic images, has immense analytical potential in neuro-oncology, and models utilizing integrated radiomic and pathomic data may yield a synergistic effect and provide a new modality for precision medicine. At the same time, the ability to harness multi-modal data is met with challenges in aggregating data across medical departments and institutions, as well as significant complexity in modeling the phenotypic and genotypic heterogeneity of pediatric brain tumors. In this paper, we review recent pathomic and integrated pathomic, radiomic, and genomic studies with clinical applications. We discuss current challenges limiting translational research on pediatric brain tumors and outline technical and analytical solutions. Overall, we propose that to empower the potential residing in radio-pathomics, systemic changes in cross-discipline data management and end-to-end software platforms to handle multi-modal data sets are needed, in addition to embracing modern AI-powered approaches. These changes can improve the performance of predictive models, and ultimately the ability to advance brain cancer treatments and patient outcomes through the development of such models.

Differential diagnosis of thyroid nodule capsules using random forest guided selection of image features

Microscopic evaluation of tissue sections stained with hematoxylin and eosin is the current gold standard for diagnosing thyroid pathology. Digital pathology is gaining momentum providing the pathologist with additional cues to traditional routes when placing a diagnosis, therefore it is extremely important to develop new image analysis methods that can extract image features with diagnostic potential. In this work, we use histogram and texture analysis to extract features from microscopic images acquired on thin thyroid nodule capsules sections and demonstrate how they enable the differential diagnosis of thyroid nodules. Targeted thyroid nodules are benign (i.e., follicular adenoma) and malignant (i.e., papillary thyroid carcinoma and its sub-type arising within a follicular adenoma). Our results show that the considered image features can enable the quantitative characterization of the collagen capsule surrounding thyroid nodules and provide an accurate classification of the latter's type using random forest.

Su-MICL: Severity-Guided Multiple Instance Curriculum Learning for Histopathology Image Interpretable Classification

Histopathology image classification plays a critical role in clinical diagnosis. However, due to the absence of clinical interpretability, most existing image-level classifiers remain impractical. To acquire the essential interpretability, lesion-level diagnosis is also provided, relying on detailed lesion-level annotations. Although the multiple-instance learning (MIL)-based approach can identify lesions by only utilizing image-level annotations, it requires overly strict prior information and has limited accuracy in lesion-level tasks. Here, we present a novel severity-guided multiple instance curriculum learning (Su-MICL) strategy to avoid tedious labeling. The proposed Su-MICL is under a MIL framework with a neglected prior: disease severity to define the learning difficulty of training images. Based on the difficulty degree, a curriculum is developed to train a model utilizing images from easy to hard. The experimental results for two histopathology image datasets demonstrate that Su-MICL achieves comparable performance to the state-of-the-art weakly supervised methods for image-level classification, and its performance for identifying lesions is closest to the supervised learning method. Without tedious lesion labeling, the Su-MICL approach can provide an interpretable diagnosis, as well as an effective insight to aid histopathology image diagnosis.

Artificial intelligence techniques for neuropathological diagnostics and research

Artificial intelligence (AI) research began in theoretical neurophysiology, and the resulting classical paper on the McCulloch-Pitts mathematical neuron was written in a psychiatry department almost 80 years ago. However, the application of AI in digital neuropathology is still in its infancy. Rapid progress is now being made, which prompted this article. Human brain diseases represent distinct system states that fall outside the normal spectrum. Many differ not only in functional but also in structural terms, and the morphology of abnormal nervous tissue forms the traditional basis of neuropathological disease classifications. However, only a few countries have the medical specialty of neuropathology, and, given the sheer number of newly developed histological tools that can be applied to the study of brain diseases, a tremendous shortage of qualified hands and eyes at the microscope is obvious. Similarly, in neuroanatomy, human observers no longer have the capacity to process the vast amounts of connectomics data. Therefore, it is reasonable to assume that advances in AI technology and, especially, whole-slide image (WSI) analysis will greatly aid neuropathological practice. In this paper, we discuss machine learning (ML) techniques that are important for understanding WSI analysis, such as traditional ML and deep learning, introduce a recently developed neuropathological AI termed PathoFusion, and present thoughts on some of the challenges that must be overcome before the full potential of AI in digital neuropathology can be realized.

Multiregional Radiomic Signatures Based on Functional Parametric Maps from DCE-MRI for Preoperative Identification of Estrogen Receptor and Progesterone Receptor Status in Breast Cancer

Radiomics based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) has been used for breast estrogen receptor (ER) and progesterone receptor (PR) status evaluation. However, the radiomic features of peritumoral regions were not thoroughly analyzed. This study aimed to establish and validate the multiregional radiomic signatures (RSs) for the preoperative identification of the ER and PR status in breast cancer. A total of 443 patients with breast cancer were divided into training (n = 356) and validation (n = 87) sets. Radiomic features were extracted from intra- and peritumoral regions on six functional parametric maps from DCE-MRI. A two-sample t-test, least absolute shrinkage and selection operator regression, and stepwise were used for feature selections. Three RSs for predicting the ER and PR status were constructed using a logistic regression model based on selected intratumoral, peritumoral, and combined intra- and peritumoral radiomic features. The area under the receiver operator characteristic curve (AUC) was used to assess the discriminative performance of three RSs. The AUCs of intra- and peritumoral RSs for identifying the ER status were 0.828/0.791 and 0.755/0.733 in the training and validation sets, respectively. For predicting the PR status, intra- and peritumoral RSs resulted in AUCs of 0.816/0.749 and 0.806/0.708 in the training and validation sets, respectively. Multiregional RSs achieved the best AUCs among three RSs for evaluating the ER (0.851 and 0.833) and PR (0.848 and 0.763) status. In conclusion, multiregional RSs based on functional parametric maps from DCE-MRI showed promising results for preoperatively evaluating the ER and PR status in breast cancer patients. Further studies using a larger cohort from multiple centers are necessary to confirm the reliability of the established models before clinical application.

Deep Learning Techniques for the Effective Prediction of Alzheimer’s Disease: A Comprehensive Review

“Alzheimer’s disease” (AD) is a neurodegenerative disorder in which the memory shrinks and neurons die. “Dementia” is described as a gradual decline in mental, psychological, and interpersonal qualities that hinders a person’s ability to function autonomously. AD is the most common degenerative brain disease. Among the first signs of AD are missing recent incidents or conversations. “Deep learning” (DL) is a type of “machine learning” (ML) that allows computers to learn by doing, much like people do. DL techniques can attain cutting-edge precision, beating individuals in certain cases. A large quantity of tagged information with multi-layered “neural network” architectures is used to perform analysis. Because significant advancements in computed tomography have resulted in sizable heterogeneous brain signals, the use of DL for the timely identification as well as automatic classification of AD has piqued attention lately. With these considerations in mind, this paper provides an in-depth examination of the various DL approaches and their implementations for the identification and diagnosis of AD. Diverse research challenges are also explored, as well as current methods in the field.

Current Developments of Artificial Intelligence in Digital Pathology and Its Future Clinical Applications in Gastrointestinal Cancers

The implementation of DP will revolutionize current practice by providing pathologists with additional tools and algorithms to improve workflow. Furthermore, DP will open up opportunities for development of AI-based tools for more precise and reproducible diagnosis through computational pathology. One of the key features of AI is its capability to generate perceptions and recognize patterns beyond the human senses. Thus, the incorporation of AI into DP can reveal additional morphological features and information. At the current rate of AI development and adoption of DP, the interest in computational pathology is expected to rise in tandem. There have already been promising developments related to AI-based solutions in prostate cancer detection; however, in the GI tract, development of more sophisticated algorithms is required to facilitate histological assessment of GI specimens for early and accurate diagnosis. In this review, we aim to provide an overview of the current histological practices in AP laboratories with respect to challenges faced in image preprocessing, present the existing AI-based algorithms, discuss their limitations and present clinical insight with respect to the application of AI in early detection and diagnosis of GI cancer.

BM-Net: CNN-Based MobileNet-V3 and Bilinear Structure for Breast Cancer Detection in Whole Slide Images

Breast cancer is one of the most common types of cancer and is the leading cause of cancer-related death. Diagnosis of breast cancer is based on the evaluation of pathology slides. In the era of digital pathology, these slides can be converted into digital whole slide images (WSIs) for further analysis. However, due to their sheer size, digital WSIs diagnoses are time consuming and challenging. In this study, we present a lightweight architecture that consists of a bilinear structure and MobileNet-V3 network, bilinear MobileNet-V3 (BM-Net), to analyze breast cancer WSIs. We utilized the WSI dataset from the ICIAR2018 Grand Challenge on Breast Cancer Histology Images (BACH) competition, which contains four classes: normal, benign, in situ carcinoma, and invasive carcinoma. We adopted data augmentation techniques to increase diversity and utilized focal loss to remove class imbalance. We achieved high performance, with 0.88 accuracy in patch classification and an average 0.71 score, which surpassed state-of-the-art models. Our BM-Net shows great potential in detecting cancer in WSIs and is a promising clinical tool.

Unsupervised Learning Based on Multiple Descriptors for WSIs Diagnosis

An automatic pathological diagnosis is a challenging task because histopathological images with different cellular heterogeneity representations are sometimes limited. To overcome this, we investigated how the holistic and local appearance features with limited information can be fused to enhance the analysis performance. We propose an unsupervised deep learning model for whole-slide image diagnosis, which uses stacked autoencoders simultaneously feeding multiple-image descriptors such as the histogram of oriented gradients and local binary patterns along with the original image to fuse the heterogeneous features. The pre-trained latent vectors are extracted from each autoencoder, and these fused feature representations are utilized for classification. We observed that training with additional descriptors helps the model to overcome the limitations of multiple variants and the intricate cellular structure of histopathology data by various experiments. Our model outperforms existing state-of-the-art approaches by achieving the highest accuracies of 87.2 for ICIAR2018, 94.6 for Dartmouth, and other significant metrics for public benchmark datasets. Our model does not rely on a specific set of pre-trained features based on classifiers to achieve high performance. Unsupervised spaces are learned from the number of independent multiple descriptors and can be used with different variants of classifiers to classify cancer diseases from whole-slide images. Furthermore, we found that the proposed model classifies the types of breast and lung cancer similar to the viewpoint of pathologists by visualization. We also designed our whole-slide image processing toolbox to extract and process the patches from whole-slide images.

Effective active learning in digital pathology: A case study in tumor infiltrating lymphocytes

BACKGROUND AND OBJECTIVE

Deep learning methods have demonstrated remarkable performance in pathology image analysis, but they require a large amount of annotated training data from expert pathologists. The aim of this study is to minimize the data annotation need in these analyses.

METHODS

Active learning (AL) is an iterative approach to training deep learning models. It was used in our context with a Tumor Infiltrating Lymphocytes (TIL) classification task to minimize annotation. State-of-the-art AL methods were evaluated with the TIL application and we have proposed and evaluated a more efficient and effective AL acquisition method. The proposed method uses data grouping based on imaging features and model prediction uncertainty to select meaningful training samples (image patches).

RESULTS

An experimental evaluation with a collection of cancer tissue images shows that: (i) Our approach reduces the number of patches required to attain a given AUC as compared to other approaches, and (ii) our optimization (subpooling) leads to AL execution time improvement of about 2.12×.

CONCLUSIONS

This strategy enabled TIL based deep learning analyses using smaller annotation demand. We expect this approach may be used to build other analyses in digital pathology with fewer training samples.

Building Efficient CNN Architectures for Histopathology Images Analysis: A Case-Study in Tumor-Infiltrating Lymphocytes Classification

Background

Deep learning methods have demonstrated remarkable performance in pathology image analysis, but they are computationally very demanding. The aim of our study is to reduce their computational cost to enable their use with large tissue image datasets.

Methods

We propose a method called Network Auto-Reduction (NAR) that simplifies a Convolutional Neural Network (CNN) by reducing the network to minimize the computational cost of doing a prediction. NAR performs a compound scaling in which the width, depth, and resolution dimensions of the network are reduced together to maintain a balance among them in the resulting simplified network. We compare our method with a state-of-the-art solution called ResRep. The evaluation is carried out with popular CNN architectures and a real-world application that identifies distributions of tumor-infiltrating lymphocytes in tissue images.

Results

The experimental results show that both ResRep and NAR are able to generate simplified, more efficient versions of ResNet50 V2. The simplified versions by ResRep and NAR require 1.32× and 3.26× fewer floating-point operations (FLOPs), respectively, than the original network without a loss in classification power as measured by the Area under the Curve (AUC) metric. When applied to a deeper and more computationally expensive network, Inception V4, NAR is able to generate a version that requires 4× lower than the original version with the same AUC performance.

Conclusions

NAR is able to achieve substantial reductions in the execution cost of two popular CNN architectures, while resulting in small or no loss in model accuracy. Such cost savings can significantly improve the use of deep learning methods in digital pathology. They can enable studies with larger tissue image datasets and facilitate the use of less expensive and more accessible graphics processing units (GPUs), thus reducing the computing costs of a study.

Differential diagnostic value of the ResNet50, random forest, and DS ensemble models for papillary thyroid carcinoma and other thyroid nodules

Objective To explore the differential diagnostic efficiency of the residual network (ResNet)50, random forest (RF), and DS ensemble models for papillary thyroid carcinoma (PTC) and other pathological types of thyroid nodules.

Methods This study retrospectively analyzed 559 patients with thyroid nodules and collected thyroid pathological images and auxiliary examination results (laboratory and ultrasound results) to construct datasets. The pathological image dataset was used to train a ResNet50 model, the text dataset was used to train a random forest (RF) model, and a DS ensemble model was constructed from the results of the two models. The differential diagnostic values of the three models for PTC and other types of thyroid nodules were then compared.

Results The DS ensemble model had the highest sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve (85.87%, 97.18%, 93.77%, and 0.982, respectively).

Conclusions Compared with Resnet50 and the RF models trained only on imaging data or text information, respectively, the DS ensemble model showed better diagnostic value for PTC.

Automated bone marrow cytology using deep learning to generate a histogram of cell types

Background

Bone marrow cytology is required to make a hematological diagnosis, influencing critical clinical decision points in hematology. However, bone marrow cytology is tedious, limited to experienced reference centers and associated with inter-observer variability. This may lead to a delayed or incorrect diagnosis, leaving an unmet need for innovative supporting technologies.

Methods

We develop an end-to-end deep learning-based system for automated bone marrow cytology. Starting with a bone marrow aspirate digital whole slide image, our system rapidly and automatically detects suitable regions for cytology, and subsequently identifies and classifies all bone marrow cells in each region. This collective cytomorphological information is captured in a representation called Histogram of Cell Types (HCT) quantifying bone marrow cell class probability distribution and acting as a cytological patient fingerprint.

Results

Our system achieves high accuracy in region detection (0.97 accuracy and 0.99 ROC AUC), and cell detection and cell classification (0.75 mean average precision, 0.78 average F1-score, Log-average miss rate of 0.31).

Conclusions

HCT has potential to eventually support more efficient and accurate diagnosis in hematology, supporting AI-enabled computational pathology.

Identifying and counting cells in bone marrow samples, known as cytology, is critical for the diagnosis of blood disorders. This is a complex and labor-intensive process, with some variation in how hematopathologists interpret these samples. Here, we develop an artificial intelligence system for automated bone marrow cytology, which automatically detects and identifies all types of cells found in the bone marrow. This information is summarized in a chart that we call the Histogram of Cell Types (HCT), a new way to represent complex information generated in bone marrow cytology. Our system achieves high accuracy and precision in classifying the different types of bone marrow cells as a HCT. This tool may eventually help clinicians to make more efficient and accurate diagnoses.

Moosavi Tayebi et al. develop a deep learning-based computational pathology tool for automated bone marrow cytology from whole slide images. Their approach generates a histogram of cell types present within the bone marrow aspirate to aid in diagnostic haematopathology.

A weakly supervised deep learning-based method for glioma subtype classification using WSI and mpMRIs

Accurate glioma subtype classification is critical for the treatment management of patients with brain tumors. Developing an automatically computer-aided algorithm for glioma subtype classification is challenging due to many factors. One of the difficulties is the label constraint. Specifically, each case is simply labeled the glioma subtype without precise annotations of lesion regions information. In this paper, we propose a novel hybrid fully convolutional neural network (CNN)-based method for glioma subtype classification using both whole slide imaging (WSI) and multiparametric magnetic resonance imagings (mpMRIs). It is comprised of two methods: a WSI-based method and a mpMRIs-based method. For the WSI-based method, we categorize the glioma subtype using a 2D CNN on WSIs. To overcome the label constraint issue, we extract the truly representative patches for the glioma subtype classification in a weakly supervised fashion. For the mpMRIs-based method, we develop a 3D CNN-based method by analyzing the mpMRIs. The mpMRIs-based method consists of brain tumor segmentation and classification. Finally, to enhance the robustness of the predictions, we fuse the WSI-based and mpMRIs-based results guided by a confidence index. The experimental results on the validation dataset in the competition of CPM-RadPath 2020 show the comprehensive judgments from both two modalities can achieve better performance than the ones by solely using WSI or mpMRIs. Furthermore, our result using the proposed method ranks the third place in the CPM-RadPath 2020 in the testing phase. The proposed method demonstrates a competitive performance, which is creditable to the success of weakly supervised approach and the strategy of label agreement from multi-modality data.

Artificial intelligence as a tool for diagnosis in digital pathology whole slide images: A systematic review

Digital pathology had a recent growth, stimulated by the implementation of digital whole slide images (WSIs) in clinical practice, and the pathology field faces shortage of pathologists in the last few years. This scenario created fronts of research applying artificial intelligence (AI) to help pathologists. One of them is the automated diagnosis, helping in the clinical decision support, increasing efficiency and quality of diagnosis. However, the complexity nature of the WSIs requires special treatments to create a reliable AI model for diagnosis. Therefore, we systematically reviewed the literature to analyze and discuss all the methods and results in AI in digital pathology performed in WSIs on H&E stain, investigating the capacity of AI as a diagnostic support tool for the pathologist in the routine real-world scenario. This review analyzes 26 studies, reporting in detail all the best methods to apply AI as a diagnostic tool, as well as the main limitations, and suggests new ideas to improve the AI field in digital pathology as a whole. We hope that this study could lead to a better use of AI as a diagnostic tool in pathology, helping future researchers in the development of new studies and projects.

Detection of Structure Characteristics and Its Discontinuity Based Field Programmable Gate Array Processor in Cancer Cell by Wavelet Transform

Digital clinical histopathology is one of the crucial techniques for precise cancer cell diagnosing in modern medicine. The Synovial Sarcoma (SS) cancer cell patterns seem to be a spindle shaped cell (SSC) structure and it is very difficult to identify the exact oval shaped cell structure through pathologist’s eye perception. Meanwhile, there is necessitating for monitoring and securing the successful and effective image data processing in the the huge network data which is also a complex one. A field programmable Gate Array (FPGA) was regarded as a necessary one for this. In this work, based on FPGA a Cancer Cell classification is made for the regulation and execution. Hence, mathematically the SSC regularity structures and its discontinuities are measured by the holder exponent (HE) function. In this research work, HE values have been determined by Wavelet Transform Modulus Maxima (WTMM) and Wavelet Leader (WL) methods with basis function of Haar wavelet based on FPGA Processor. The quantitative parameters such as Mean of Asymptotic Discontinuity (MAD), Mean of Removable Discontinuity (MRD) and Number of Discontinuity Points (NDPs) have been considered to determine the prediction of discontinuity detection between WTMM and WL methods. With the help of receiver operating characteristics (ROC) curve, the significant difference of discontinuity detection performance between both the methods has been analyzed. From the experimental results, it is clear that the WL method is more practically feasible and it gives satisfactory performance, in terms of sensitivity and specificity percentage values, which are 80.56% and 59.46%, respectively in the blue color components of the SNR 20 dB noise image.

Reduced and stable feature sets selection with random forest for neurons segmentation in histological images of macaque brain

In preclinical research, histology images are produced using powerful optical microscopes to digitize entire sections at cell scale. Quantification of stained tissue relies on machine learning driven segmentation. However, such methods require multiple additional information, or features, which are increasing the quantity of data to process. As a result, the quantity of features to deal with represents a drawback to process large series or massive histological images rapidly in a robust manner. Existing feature selection methods can reduce the amount of required information but the selected subsets lack reproducibility. We propose a novel methodology operating on high performance computing (HPC) infrastructures and aiming at finding small and stable sets of features for fast and robust segmentation of high-resolution histological images. This selection has two steps: (1) selection at features families scale (an intermediate pool of features, between spaces and individual features) and (2) feature selection performed on pre-selected features families. We show that the selected sets of features are stables for two different neuron staining. In order to test different configurations, one of these dataset is a mono-subject dataset and the other is a multi-subjects dataset to test different configurations. Furthermore, the feature selection results in a significant reduction of computation time and memory cost. This methodology will allow exhaustive histological studies at a high-resolution scale on HPC infrastructures for both preclinical and clinical research.

Artificial Intelligence in Brain Tumour Surgery-An Emerging Paradigm

Artificial intelligence (AI) platforms have the potential to cause a paradigm shift in brain tumour surgery. Brain tumour surgery augmented with AI can result in safer and more effective treatment. In this review article, we explore the current and future role of AI in patients undergoing brain tumour surgery, including aiding diagnosis, optimising the surgical plan, providing support during the operation, and better predicting the prognosis. Finally, we discuss barriers to the successful clinical implementation, the ethical concerns, and we provide our perspective on how the field could be advanced.

MCUa: Multi-level Context and Uncertainty aware Dynamic Deep Ensemble for Breast Cancer Histology Image Classification

Breast histology image classification is a crucial step in the early diagnosis of breast cancer. In breast pathological diagnosis, Convolutional Neural Networks (CNNs) have demonstrated great success using digitized histology slides. However, tissue classification is still challenging due to the high visual variability of the large-sized digitized samples and the lack of contextual information. In this paper, we propose a novel CNN, called Multi-level Context and Uncertainty aware (MCUa) dynamic deep learning ensemble model. MCUa model consists of several multi-level context-aware models to learn the spatial dependency between image patches in a layer-wise fashion. It exploits the high sensitivity to the multi-level contextual information using an uncertainty quantification component to accomplish a novel dynamic ensemble model. MCUa model has achieved a high accuracy of 98.11% on a breast cancer histology image dataset. Experimental results show the superior effectiveness of the proposed solution compared to the state-of-the-art histology classification models.

Digital Pathology-based Study of Cell- and Tissue-level Morphologic Features in Serous Borderline Ovarian Tumor and High-grade Serous Ovarian Cancer

Background:

Serous borderline ovarian tumor (SBOT) and high-grade serous ovarian cancer (HGSOC) are two distinct subtypes of epithelial ovarian tumors, with markedly different biologic background, behavior, prognosis, and treatment. However, the histologic diagnosis of serous ovarian tumors can be subjectively variable and labor-intensive as multiple tumor slides/blocks need to be thoroughly examined to search for these features.

Materials and Methods:

We developed a novel informatics system to facilitate objective and scalable diagnosis screening for SBOT and HGSOC. The system was built upon Groovy scripts and QuPath to enable interactive annotation and data exchange.

Results:

The system was used to successfully detect cellular boundaries and extract an expanded set of cellular features representing cell- and tissue-level characteristics. The performance of cell-level classification for both tumor and stroma cells achieved >90% accuracy. The performance of differentiating HGSOC versus SBOT achieved 91%–95% accuracy for 6485 imaging patches which have sufficient tumor and stroma cells (minimum of ten each) and 97% accuracy for classifying patients when aggregating the results to whole-slide image based on consensus.

Conclusions:

Cellular features digitally extracted from pathological images can be used for cell classification and SBOT v. HGSOC differentiation. Introducing digital pathology into ovarian cancer research could be beneficial to discover potential clinical implications. A larger cohort is required to further evaluate the system.