Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

ImmuNet: a segmentation-free machine learning pipeline for immune landscape phenotyping in tumors by multiplex imaging

Tissue specimens taken from primary tumors or metastases contain important information for diagnosis and treatment of cancer patients. Multiplex imaging allows in situ visualization of heterogeneous cell populations, such as immune cells, in tissue samples. Most image processing pipelines first segment cell boundaries and then measure marker expression to assign cell phenotypes. In dense tissue environments, this segmentation-first approach can be inaccurate due to segmentation errors or overlapping cells. Here, we introduce the machine-learning pipeline "ImmuNet", which identifies positions and phenotypes of cells without segmenting them. ImmuNet is easy to train: human annotators only need to click on an immune cell and score its expression of each marker-drawing a full cell outline is not required. We trained and evaluated ImmuNet on multiplex images from human tonsil, lung cancer, prostate cancer, melanoma, and bladder cancer tissue samples and found it to consistently achieve error rates below 5%-10% across tissue types, cell types, and tissue densities, outperforming a segmentation-based baseline method. Furthermore, we externally validate ImmuNet results by comparing them to flow cytometric cell count measurements from the same tissue. In summary, ImmuNet is an effective, simpler alternative to segmentation-based approaches when only cell positions and phenotypes, but not their shapes, are required for downstream analyses. Thus, ImmuNet helps researchers to analyze cell positions in multiplex tissue images more easily and accurately.

Deformable multi-level feature network applied to nucleus segmentation

Introduction

The nucleus plays a crucial role in medical diagnosis, and accurate nucleus segmentation is essential for disease assessment. However, existing methods have limitations in handling the diversity of nuclei and differences in staining conditions, restricting their practical application.

Methods

A novel deformable multi-level feature network (DMFNet) is proposed for nucleus segmentation. This network is based on convolutional neural network and divides feature processing and mask generation into two levels. At the feature level, deformable convolution is used to enhance feature extraction ability, and multi-scale features are integrated through a balanced feature pyramid. At the mask level, a one-stage framework is adopted to directly perform instance segmentation based on location.

Results

Experimental results on the MoNuSeg 2018 dataset show that the mean average precision (mAP) and mean average recall (mAR) of DMFNet reach 37.8% and 47.4% respectively, outperforming many current advanced methods. Ablation experiments verified the effectiveness of each module of the network.

Discussion

DMFNet provides an effective solution for nucleus segmentation and has important application value in medical image analysis.

Improving generalization capability of deep learning-based nuclei instance segmentation by non-deterministic train time and deterministic test time stain normalization

With the advent of digital pathology and microscopic systems that can scan and save whole slide histological images automatically, there is a growing trend to use computerized methods to analyze acquired images. Among different histopathological image analysis tasks, nuclei instance segmentation plays a fundamental role in a wide range of clinical and research applications. While many semi- and fully-automatic computerized methods have been proposed for nuclei instance segmentation, deep learning (DL)-based approaches have been shown to deliver the best performances. However, the performance of such approaches usually degrades when tested on unseen datasets. In this work, we propose a novel method to improve the generalization capability of a DL-based automatic segmentation approach. Besides utilizing one of the state-of-the-art DL-based models as a baseline, our method incorporates non-deterministic train time and deterministic test time stain normalization, and ensembling to boost the segmentation performance. We trained the model with one single training set and evaluated its segmentation performance on seven test datasets. Our results show that the proposed method provides up to 4.9%, 5.4%, and 5.9% better average performance in segmenting nuclei based on Dice score, aggregated Jaccard index, and panoptic quality score, respectively, compared to the baseline segmentation model.

Intelligent sensing for the autonomous manipulation of microrobots toward minimally invasive cell surgery

The physiology and pathogenesis of biological cells have drawn enormous research interest. Benefiting from the rapid development of microfabrication and microelectronics, miniaturized robots with a tool size below micrometers have widely been studied for manipulating biological cells in vitro and in vivo. Traditionally, the complex physiological environment and biological fragility require human labor interference to fulfill these tasks, resulting in high risks of irreversible structural or functional damage and even clinical risk. Intelligent sensing devices and approaches have been recently integrated within robotic systems for environment visualization and interaction force control. As a consequence, microrobots can be autonomously manipulated with visual and interaction force feedback, greatly improving accuracy, efficiency, and damage regulation for minimally invasive cell surgery. This review first explores advanced tactile sensing in the aspects of sensing principles, design methodologies, and underlying physics. It also comprehensively discusses recent progress on visual sensing, where the imaging instruments and processing methods are summarized and analyzed. It then introduces autonomous micromanipulation practices utilizing visual and tactile sensing feedback and their corresponding applications in minimally invasive surgery. Finally, this work highlights and discusses the remaining challenges of current robotic micromanipulation and their future directions in clinical trials, providing valuable references about this field.

Automated cell detection for immediate early gene-expressing neurons using inhomogeneous background subtraction in fluorescent images

Although many methods for automated fluorescent-labeled cell detection have been proposed, not all of them assume a highly inhomogeneous background arising from complex biological structures. Here, we propose an automated cell detection algorithm that accounts for and subtracts the inhomogeneous background by avoiding high-intensity pixels in the blur filtering calculation. Cells were detected by intensity thresholding in the background-subtracted image, and the algorithm's performance was tested on NeuN- and c-Fos-stained images in the mouse prefrontal cortex and hippocampal dentate gyrus. In addition, applications in c-Fos positive cell counting and the quantification for the expression level in double-labeled cells were demonstrated. Our method of automated detection after background assumption (ADABA) offers the advantage of high-throughput and unbiased analysis in regions with complex biological structures that produce inhomogeneous background.

Highlights

- We proposed a method to assume and subtract inhomogeneous background pattern. (79/85) - Cells were automatically detected in the background-subtracted image. (71/85) - The automated detection results corresponded with the manual detection. (73/85) - Detection of IEG positive cells and overlapping with neural marker were demonstrated. (85/85).

Fluorescent carbon nanodot-based rapid cell nucleus staining and application for diagnosis

Cell nucleus status decides the activities of corresponding cells, making its rapid and effective staining important for revealing the actual condition of biological environment in life science and related fields. In this study, fast staining of cell nucleus is realized by fluorescent carbon nanodots (CDs). The staining mechanism is due to the positively charged CD surface-induced cell membrane penetration, which facilitates the CD-nucleus binding via electrostatic attraction. The size of cell nucleus is easily measured with fluorescence imaging technique. In addition, the CD-based cell nucleus stain is applied for discriminating the normal and cancer cells by determining the cell-to-nucleus ratio with fluorescence images.

Optical microscopic imaging, manipulation, and analysis methods for morphogenesis research

Morphogenesis is a developmental process of organisms being shaped through complex and cooperative cellular movements. To understand the interplay between genetic programs and the resulting multicellular morphogenesis, it is essential to characterize the morphologies and dynamics at the single-cell level and to understand how physical forces serve as both signaling components and driving forces of tissue deformations. In recent years, advances in microscopy techniques have led to improvements in imaging speed, resolution and depth. Concurrently, the development of various software packages has supported large-scale, analyses of challenging images at the single-cell resolution. While these tools have enhanced our ability to examine dynamics of cells and mechanical processes during morphogenesis, their effective integration requires specialized expertise. With this background, this review provides a practical overview of those techniques. First, we introduce microscopic techniques for multicellular imaging and image analysis software tools with a focus on cell segmentation and tracking. Second, we provide an overview of cutting-edge techniques for mechanical manipulation of cells and tissues. Finally, we introduce recent findings on morphogenetic mechanisms and mechanosensations that have been achieved by effectively combining microscopy, image analysis tools and mechanical manipulation techniques.

Spatial Pattern Analysis using Closest Events (SPACE)-A Nearest Neighbor Point Pattern Analysis Framework for Assessing Spatial Relationships from Digital Images

The quantitative description of biological structures is a valuable yet difficult task in the life sciences. This is commonly accomplished by imaging samples using fluorescence microscopy and analyzing resulting images using Pearson's correlation or Manders' co-occurrence intensity-based colocalization paradigms. Though conceptually and computationally simple, these approaches are critically flawed due to their reliance on signal overlap, sensitivity to cursory signal qualities, and inability to differentiate true and incidental colocalization. Point pattern analysis provides a framework for quantitative characterization of spatial relationships between spatial patterns using the distances between observations rather than their overlap, thus overcoming these issues. Here we introduce an image analysis tool called Spatial Pattern Analysis using Closest Events (SPACE) that leverages nearest neighbor-based point pattern analysis to characterize the spatial relationship of fluorescence microscopy signals from image data. The utility of SPACE is demonstrated by assessing the spatial association between mRNA and cell nuclei from confocal images of cardiac myocytes. Additionally, we use synthetic and empirical images to characterize the sensitivity of SPACE to image segmentation parameters and cursory image qualities such as signal abundance and image resolution. Ultimately, SPACE delivers performance superior to traditional colocalization methods and offers a valuable addition to the microscopist's toolbox.

Using Deep Learning to Automate Eosinophil Counting in Pediatric Ulcerative Colitis Histopathological Images

Background

Accurate identification of inflammatory cells from mucosal histopathology images is important in diagnosing ulcerative colitis. The identification of eosinophils in the colonic mucosa has been associated with disease course. Cell counting is not only time-consuming but can also be subjective to human biases. In this study we developed an automatic eosinophilic cell counting tool from mucosal histopathology images, using deep learning.

Method

Four pediatric IBD pathologists from two North American pediatric hospitals annotated 530 crops from 143 standard-of-care hematoxylin and eosin (H & E) rectal mucosal biopsies. A 305/75 split was used for training/validation to develop and optimize a U-Net based deep learning model, and 150 crops were used as a test set. The U-Net model was then compared to SAU-Net, a state-of-the-art U-Net variant. We undertook post-processing steps, namely, (1) the pixel-level probability threshold, (2) the minimum number of clustered pixels to designate a cell, and (3) the connectivity. Experiments were run to optimize model parameters using AUROC and cross-entropy loss as the performance metrics.

Results

The F1-score was 0.86 (95%CI:0.79-0.91) (Precision: 0.77 (95%CI:0.70-0.83), Recall: 0.96 (95%CI:0.93-0.99)) to identify eosinophils as compared to an F1-score of 0.2 (95%CI:0.13-0.26) for SAU-Net (Precision: 0.38 (95%CI:0.31-0.46), Recall: 0.13 (95%CI:0.08-0.19)). The inter-rater reliability was 0.96 (95%CI:0.93-0.97). The correlation between two pathologists and the algorithm was 0.89 (95%CI:0.82-0.94) and 0.88 (95%CI:0.80-0.94) respectively.

Conclusion

Our results indicate that deep learning-based automated eosinophilic cell counting can obtain a robust level of accuracy with a high degree of concordance with manual expert annotations.

Super resolution-based methodology for self-supervised segmentation of microscopy images

Data-driven Artificial Intelligence (AI)/Machine learning (ML) image analysis approaches have gained a lot of momentum in analyzing microscopy images in bioengineering, biotechnology, and medicine. The success of these approaches crucially relies on the availability of high-quality microscopy images, which is often a challenge due to the diverse experimental conditions and modes under which these images are obtained. In this study, we propose the use of recent ML-based image super-resolution (SR) techniques for improving the image quality of microscopy images, incorporating them into multiple ML-based image analysis tasks, and describing a comprehensive study, investigating the impact of SR techniques on the segmentation of microscopy images. The impacts of four Generative Adversarial Network (GAN)- and transformer-based SR techniques on microscopy image quality are measured using three well-established quality metrics. These SR techniques are incorporated into multiple deep network pipelines using supervised, contrastive, and non-contrastive self-supervised methods to semantically segment microscopy images from multiple datasets. Our results show that the image quality of microscopy images has a direct influence on the ML model performance and that both supervised and self-supervised network pipelines using SR images perform better by 2%-6% in comparison to baselines, not using SR. Based on our experiments, we also establish that the image quality improvement threshold range [20-64] for the complemented Perception-based Image Quality Evaluator(PIQE) metric can be used as a pre-condition by domain experts to incorporate SR techniques to significantly improve segmentation performance. A plug-and-play software platform developed to integrate SR techniques with various deep networks using supervised and self-supervised learning methods is also presented.

Bio-net dataset: AI-based diagnostic solutions using peripheral blood smear images

Peripheral blood smear examination is one of the basic steps in the evaluation of different blood cells. It is a confirmatory step after an automated complete blood count analysis. Manual microscopy is time-consuming and requires professional laboratory expertise. Therefore, the turn-around time for peripheral smear in a health care center is approximately 3-4 hours. To avoid the traditional method of manual counting under the microscope a computerized automation of peripheral blood smear examination has been adopted, which is a challenging task in medical diagnostics. In recent times, deep learning techniques have overcome the challenges associated with human microscopic evaluation of peripheral smears and this has led to reduced cost and precise diagnosis. However, their application can be significantly improved by the availability of annotated datasets. This study presents a large customized annotated blood cell dataset (named the Bio-Net dataset from healthy individuals) and blood cell detection and counting in the peripheral blood smear images. A mini-version of the dataset for specialized WBC-based image processing tasks is also equipped to classify the healthy and mature WBCs in their respective classes. An object detection algorithm called You Only Look Once (YOLO) with a refashion disposition has been trained on the novel dataset to automatically detect and classify blood cells into RBCs, WBCs, and platelets and compare the results with other publicly available datasets to highlight the versatility. In short the introduction of the Bio-Net dataset and AI-powered detection and counting offers a significant potential for advancement in biomedical research for analyzing and understanding biological data.

Perinatal Western-style diet exposure associated with decreased microglial counts throughout the arcuate nucleus of the hypothalamus in Japanese macaques

Perinatal exposure to a high-fat, high-sugar Western-style diet (WSD) is associated with altered neural circuitry in the melanocortin system. This association may have an underlying inflammatory component, as consumption of a WSD during pregnancy can lead to an elevated inflammatory environment. Our group previously demonstrated that prenatal WSD exposure was associated with increased markers of inflammation in the placenta and fetal hypothalamus in Japanese macaques. In this follow-up study, we sought to determine whether this heightened inflammatory state persisted into the postnatal period, as prenatal exposure to inflammation has been shown to reprogram offspring immune function and long-term neuroinflammation would present a potential means for prolonged disruptions to microglia-mediated neuronal circuit formation. Neuroinflammation was approximated in 1-yr-old offspring by counting resident microglia and peripherally derived macrophages in the region of the hypothalamus examined in the fetal study, the arcuate nucleus (ARC). Microglia and macrophages were immunofluorescently stained with their shared marker, ionized calcium-binding adapter molecule 1 (Iba1), and quantified in 11 regions along the rostral-caudal axis of the ARC. A mixed-effects model revealed main effects of perinatal diet (P = 0.011) and spatial location (P = 0.003) on Iba1-stained cell count. Perinatal WSD exposure was associated with a slight decrease in the number of Iba1-stained cells, and cells were more densely located in the center of the ARC. These findings suggest that the heightened inflammatory state experienced in utero does not persist postnatally. This inflammatory response trajectory could have important implications for understanding how neurodevelopmental disorders progress.NEW & NOTEWORTHY Prenatal Western-style diet exposure is associated with increased microglial activity in utero. However, we found a potentially neuroprotective reduction in microglia count during early postnatal development. This trajectory could inform the timing of disruptions to microglia-mediated neuronal circuit formation. Additionally, this is the first study in juvenile macaques to characterize the distribution of microglia along the rostral-caudal axis of the arcuate nucleus of the hypothalamus. Nearby neuronal populations may be greater targets during inflammatory insults.

An interactive nuclei segmentation framework with Voronoi diagrams and weighted convex difference for cervical cancer pathology images

Objective.Nuclei segmentation is crucial for pathologists to accurately classify and grade cancer. However, this process faces significant challenges, such as the complex background structures in pathological images, the high-density distribution of nuclei, and cell adhesion.Approach.In this paper, we present an interactive nuclei segmentation framework that increases the precision of nuclei segmentation. Our framework incorporates expert monitoring to gather as much prior information as possible and accurately segment complex nucleus images through limited pathologist interaction, where only a small portion of the nucleus locations in each image are labeled. The initial contour is determined by the Voronoi diagram generated from the labeled points, which is then input into an optimized weighted convex difference model to regularize partition boundaries in an image. Specifically, we provide theoretical proof of the mathematical model, stating that the objective function monotonically decreases. Furthermore, we explore a postprocessing stage that incorporates histograms, which are simple and easy to handle and prevent arbitrariness and subjectivity in individual choices.Main results.To evaluate our approach, we conduct experiments on both a cervical cancer dataset and a nasopharyngeal cancer dataset. The experimental results demonstrate that our approach achieves competitive performance compared to other methods.Significance.The Voronoi diagram in the paper serves as prior information for the active contour, providing positional information for individual cells. Moreover, the active contour model achieves precise segmentation results while offering mathematical interpretability.

Automatic recognition of white blood cell images with memory efficient superpixel metric GNN: SMGNN

An automatic recognizing system of white blood cells can assist hematologists in the diagnosis of many diseases, where accuracy and efficiency are paramount for computer-based systems. In this paper, we presented a new image processing system to recognize the five types of white blood cells in peripheral blood with marked improvement in efficiency when juxtaposed against mainstream methods. The prevailing deep learning segmentation solutions often utilize millions of parameters to extract high-level image features and neglect the incorporation of prior domain knowledge, which consequently consumes substantial computational resources and increases the risk of overfitting, especially when limited medical image samples are available for training. To address these challenges, we proposed a novel memory-efficient strategy that exploits graph structures derived from the images. Specifically, we introduced a lightweight superpixel-based graph neural network (GNN) and broke new ground by introducing superpixel metric learning to segment nucleus and cytoplasm. Remarkably, our proposed segmentation model superpixel metric graph neural network (SMGNN) achieved state of the art segmentation performance while utilizing at most 10000$ \times $ less than the parameters compared to existing approaches. The subsequent segmentation-based cell type classification processes showed satisfactory results that such automatic recognizing algorithms are accurate and efficient to execeute in hematological laboratories. Our code is publicly available at https://github.com/jyh6681/SPXL-GNN.

Quantitative Analysis of Whole-Mount Fluorescence-Stained Tumor Spheroids in Phenotypic Drug Screens

Three-dimensional cell cultures, such as spheroids or organoids, serve as important models for drug screening purposes. Optical tissue clearing (OTC) enhances the visualization of fluorescence stainings and enables in toto microscopy of 3D cell culture models. Furthermore, subsequent automated image analysis tools convert qualitative confocal image sets into quantitative data. In this chapter, we describe a detailed protocol for preparation of HT29 cancer spheroids, 3D in toto immunostaining, glycerol-based OTC, whole-mount imaging, and semi-automated downstream image processing and segmentation for nuclear image analysis using open-source software.

Learning with limited target data to detect cells in cross-modality images

Deep neural networks have achieved excellent cell or nucleus quantification performance in microscopy images, but they often suffer from performance degradation when applied to cross-modality imaging data. Unsupervised domain adaptation (UDA) based on generative adversarial networks (GANs) has recently improved the performance of cross-modality medical image quantification. However, current GAN-based UDA methods typically require abundant target data for model training, which is often very expensive or even impossible to obtain for real applications. In this paper, we study a more realistic yet challenging UDA situation, where (unlabeled) target training data is limited and previous work seldom delves into cell identification. We first enhance a dual GAN with task-specific modeling, which provides additional supervision signals to assist with generator learning. We explore both single-directional and bidirectional task-augmented GANs for domain adaptation. Then, we further improve the GAN by introducing a differentiable, stochastic data augmentation module to explicitly reduce discriminator overfitting. We examine source-, target-, and dual-domain data augmentation for GAN enhancement, as well as joint task and data augmentation in a unified GAN-based UDA framework. We evaluate the framework for cell detection on multiple public and in-house microscopy image datasets, which are acquired with different imaging modalities, staining protocols and/or tissue preparations. The experiments demonstrate that our method significantly boosts performance when compared with the reference baseline, and it is superior to or on par with fully supervised models that are trained with real target annotations. In addition, our method outperforms recent state-of-the-art UDA approaches by a large margin on different datasets.

Nuclei instance segmentation using a transformer-based graph convolutional network and contextual information augmentation

Nucleus instance segmentation is an important task in medical image analysis involving cell-level pathological analysis and is of great significance for many biomedical applications, such as disease diagnosis and drug screening. However, the high-density and tight-contact between cells is a common feature of most cell images, which poses a great technical challenge for nuclei instance segmentation. The latest research focuses on CNN-based methods for nuclei instance segmentation, which typically rely on bounding box regression and non-maximum suppression to locate nuclei. However, this frequently results in poor local bounding boxes for nuclei that are adhered or clustered together. In response to the challenges of high-density and tight-contact in cellular images, we propose a novel end-to-end nuclei instance segmentation model. Specifically, we first employ the Swin Transformer as the backbone network of our model, which captures global multi-scale information by combining the global modelling capability of transformers and the local modelling capability of convolutional neural networks (CNNs). Additionally, we integrate a graph convolutional feature fusion module (GCFM), that combines deep and shallow features to learn an affinity matrix. The module also adopts graph convolution to guide the network in learning the object-level local information. Finally, we design a hybrid dilated convolution module (HDC) and insert it into the backbone network to enhance the contextual information over a large range. These components assist the network in extracting rich features. The experimental results demonstrate that our algorithm outperforms several state-of-the-art models on the DSB2018 and LIVECell datasets.

Democratizing Pathological Image Segmentation with Lay Annotators via Molecular-empowered Learning

Multi-class cell segmentation in high-resolution Giga-pixel whole slide images (WSI) is critical for various clinical applications. Training such an AI model typically requires labor-intensive pixel-wise manual annotation from experienced domain experts (e.g., pathologists). Moreover, such annotation is error-prone when differentiating fine-grained cell types (e.g., podocyte and mesangial cells) via the naked human eye. In this study, we assess the feasibility of democratizing pathological AI deployment by only using lay annotators (annotators without medical domain knowledge). The contribution of this paper is threefold: (1) We proposed a molecular-empowered learning scheme for multi-class cell segmentation using partial labels from lay annotators; (2) The proposed method integrated Giga-pixel level molecular-morphology cross-modality registration, molecular-informed annotation, and molecular-oriented segmentation model, so as to achieve significantly superior performance via 3 lay annotators as compared with 2 experienced pathologists; (3) A deep corrective learning (learning with imperfect label) method is proposed to further improve the segmentation performance using partially annotated noisy data. From the experimental results, our learning method achieved F1 = 0.8496 using molecular-informed annotations from lay annotators, which is better than conventional morphology-based annotations (F1 = 0.7015) from experienced pathologists. Our method democratizes the development of a pathological segmentation deep model to the lay annotator level, which consequently scales up the learning process similar to a non-medical computer vision task. The official implementation and cell annotations are publicly available at https://github.com/hrlblab/MolecularEL.

Toward Source-Free Cross Tissues Histopathological Cell Segmentation via Target-Specific Finetuning

Recognition and quantitative analytics of histopathological cells are the golden standard for diagnosing multiple cancers. Despite recent advances in deep learning techniques that have been widely investigated for the automated segmentation of various types of histopathological cells, the heavy dependency on specific histopathological image types with sufficient supervised annotations, as well as the limited access to clinical data in hospitals, still pose significant challenges in the application of computer-aided diagnosis in pathology. In this paper, we focus on the model generalization of cell segmentation towards cross-tissue histopathological images. Remarkably, a novel target-specific finetuning-based self-supervised domain adaptation framework is proposed to transfer the cell segmentation model to unlabeled target datasets, without access to source datasets and annotations. When performed on the target unlabeled histopathological image set, the proposed method only needs to tune very few parameters of the pre-trained model in a self-supervised manner. Considering the morphological properties of pathological cells, we introduce two constraint terms at both local and global levels into this framework to access more reliable predictions. The proposed cross-domain framework is validated on three different types of histopathological tissues, showing promising performance in self-supervised cell segmentation. Additionally, the whole framework can be further applied to clinical tools in pathology without accessing the original training image data. The code and dataset are released at: https://github.com/NeuronXJTU/SFDA-CellSeg.

Optimized detection and segmentation of nuclei in gastric cancer images using stain normalization and blurred artifact removal

Histological analysis with microscopy is the gold standard to diagnose and stage cancer, where slides or whole slide images are analyzed for cell morphological and spatial features by pathologists. The nuclei of cancerous cells are characterized by nonuniform chromatin distribution, irregular shapes, and varying size. As nucleus area and shape alone carry prognostic value, detection and segmentation of nuclei are among the most important steps in disease grading. However, evaluation of nuclei is a laborious, time-consuming, and subjective process with large variation among pathologists. Recent advances in digital pathology have allowed significant applications in nuclei detection, segmentation, and classification, but automated image analysis is greatly affected by staining factors, scanner variability, and imaging artifacts, requiring robust image preprocessing, normalization, and segmentation methods for clinically satisfactory results. In this paper, we aimed to evaluate and compare the digital image analysis techniques used in clinical pathology and research in the setting of gastric cancer. A literature review was conducted to evaluate potential methods of improving nuclei detection. Digitized images of 35 patients from a retrospective cohort of gastric adenocarcinoma at Oulu University Hospital in 1987-2016 were annotated for nuclei (n = 9085) by expert pathologists and 14 images of different cancer types from public TCGA dataset with annotated nuclei (n = 7000) were used as a comparison to evaluate applicability in other cancer types. The detection and segmentation accuracy with the selected color normalization and stain separation techniques were compared between the methods. The extracted information can be supplemented by patient's medical data and fed to the existing statistical clinical tools or subjected to subsequent AI-assisted classification and prediction models. The performance of each method is evaluated by several metrics against the annotations done by expert pathologists. The F1-measure of 0.854 ± 0.068 is achieved with color normalization for the gastric cancer dataset, and 0.907 ± 0.044 with color deconvolution for the public dataset, showing comparable results to the earlier state-of-the-art works. The developed techniques serve as a basis for further research on application and interpretability of AI-assisted tools for gastric cancer diagnosis.

Histopathological Images Analysis and Predictive Modeling Implemented in Digital Pathology-Current Affairs and Perspectives

In modern clinical practice, digital pathology has an essential role, being a technological necessity for the activity in the pathological anatomy laboratories. The development of information technology has majorly facilitated the management of digital images and their sharing for clinical use; the methods to analyze digital histopathological images, based on artificial intelligence techniques and specific models, quantify the required information with significantly higher consistency and precision compared to that provided by optical microscopy. In parallel, the unprecedented advances in machine learning facilitate, through the synergy of artificial intelligence and digital pathology, the possibility of diagnosis based on image analysis, previously limited only to certain specialties. Therefore, the integration of digital images into the study of pathology, combined with advanced algorithms and computer-assisted diagnostic techniques, extends the boundaries of the pathologist's vision beyond the microscopic image and allows the specialist to use and integrate his knowledge and experience adequately. We conducted a search in PubMed on the topic of digital pathology and its applications, to quantify the current state of knowledge. We found that computer-aided image analysis has a superior potential to identify, extract and quantify features in more detail compared to the human pathologist's evaluating possibilities; it performs tasks that exceed its manual capacity, and can produce new diagnostic algorithms and prediction models applicable in translational research that are able to identify new characteristics of diseases based on changes at the cellular and molecular level.

Self-Supervised Segmentation of 3D Fluorescence Microscopy Images Using CycleGAN

In recent years, deep learning models have been extensively applied for the segmentation of microscopy images to efficiently and accurately quantify and characterize cells, nuclei, and other biological structures. However, typically these are supervised models that require large amounts of training data that are manually annotated to create the ground-truth. Since manual annotation of these segmentation masks is difficult and time-consuming, specially in 3D, we sought to develop a self-supervised segmentation method.Our method is based on an image-to-image translation model, the CycleGAN, which we use to learn the mapping from the fluorescence microscopy images domain to the segmentation domain. We exploit the fact that CycleGAN does not require paired data and train the model using synthetic masks, instead of manually labeled masks. These masks are created automatically based on the approximate shapes and sizes of the nuclei and Golgi, thus manual image segmentation is not needed in our proposed approach.The experimental results obtained with the proposed CycleGAN model are compared with two well-known supervised segmentation models: 3D U-Net [1] and Vox2Vox [2]. The CycleGAN model led to the following results: Dice coefficient of 78.07% for the nuclei class and 67.73% for the Golgi class with a difference of only 1.4% and 0.61% compared to the best results obtained with the supervised models Vox2Vox and 3D U-Net, respectively. Moreover, training and testing the CycleGAN model is about 5.78 times faster in comparison with the 3D U-Net model. Our results show that without manual annotation effort we can train a model that performs similarly to supervised models for the segmentation of organelles in 3D microscopy images.Clinical relevance- Segmentation of cell organelles in microscopy images is an important step to extract several features, such as the morphology, density, size, shape and texture of these organelles. These quantitative analyses provide valuable information to classify and diagnose diseases, and to study biological processes.

Lung cancer lesion detection in histopathology images using graph-based sparse PCA network

Early detection of lung cancer is critical for improvement of patient survival. To address the clinical need for efficacious treatments, genetically engineered mouse models (GEMM) have become integral in identifying and evaluating the molecular underpinnings of this complex disease that may be exploited as therapeutic targets. Assessment of GEMM tumor burden on histopathological sections performed by manual inspection is both time consuming and prone to subjective bias. Therefore, an interplay of needs and challenges exists for computer-aided diagnostic tools, for accurate and efficient analysis of these histopathology images. In this paper, we propose a simple machine learning approach called the graph-based sparse principal component analysis (GS-PCA) network, for automated detection of cancerous lesions on histological lung slides stained by hematoxylin and eosin (H&E). Our method comprises four steps: 1) cascaded graph-based sparse PCA, 2) PCA binary hashing, 3) block-wise histograms, and 4) support vector machine (SVM) classification. In our proposed architecture, graph-based sparse PCA is employed to learn the filter banks of the multiple stages of a convolutional network. This is followed by PCA hashing and block histograms for indexing and pooling. The meaningful features extracted from this GS-PCA are then fed to an SVM classifier. We evaluate the performance of the proposed algorithm on H&E slides obtained from an inducible K-ras^G12D lung cancer mouse model using precision/recall rates, F_β-score, Tanimoto coefficient, and area under the curve (AUC) of the receiver operator characteristic (ROC) and show that our algorithm is efficient and provides improved detection accuracy compared to existing algorithms.

Hematologic Cancer Detection Using White Blood Cancerous Cells Empowered with Transfer Learning and Image Processing

Lymphoma and leukemia are fatal syndromes of cancer that cause other diseases and affect all types of age groups including male and female, and disastrous and fatal blood cancer causes an increased savvier death ratio. Both lymphoma and leukemia are associated with the damage and rise of immature lymphocytes, monocytes, neutrophils, and eosinophil cells. So, in the health sector, the early prediction and treatment of blood cancer is a major issue for survival rates. Nowadays, there are various manual techniques to analyze and predict blood cancer using the microscopic medical reports of white blood cell images, which is very steady for prediction and causes a major ratio of deaths. Manual prediction and analysis of eosinophils, lymphocytes, monocytes, and neutrophils are very difficult and time-consuming. In previous studies, they used numerous deep learning and machine learning techniques to predict blood cancer, but there are still some limitations in these studies. So, in this article, we propose a model of deep learning empowered with transfer learning and indulge in image processing techniques to improve the prediction results. The proposed transfer learning model empowered with image processing incorporates different levels of prediction, analysis, and learning procedures and employs different learning criteria like learning rate and epochs. The proposed model used numerous transfer learning models with varying parameters for each model and cloud techniques to choose the best prediction model, and the proposed model used an extensive set of performance techniques and procedures to predict the white blood cells which cause cancer to incorporate image processing techniques. So, after extensive procedures of AlexNet, MobileNet, and ResNet with both image processing and without image processing techniques with numerous learning criteria, the stochastic gradient descent momentum incorporated with AlexNet is outperformed with the highest prediction accuracy of 97.3% and the misclassification rate is 2.7% with image processing technique. The proposed model gives good results and can be applied for smart diagnosing of blood cancer using eosinophils, lymphocytes, monocytes, and neutrophils.

Densely Convolutional Spatial Attention Network for nuclei segmentation of histological images for computational pathology

Introduction

Automatic nuclear segmentation in digital microscopic tissue images can aid pathologists to extract high-quality features for nuclear morphometrics and other analyses. However, image segmentation is a challenging task in medical image processing and analysis. This study aimed to develop a deep learning-based method for nuclei segmentation of histological images for computational pathology.

Methods

The original U-Net model sometime has a caveat in exploring significant features. Herein, we present the Densely Convolutional Spatial Attention Network (DCSA-Net) model based on U-Net to perform the segmentation task. Furthermore, the developed model was tested on external multi-tissue dataset - MoNuSeg. To develop deep learning algorithms for well-segmenting nuclei, a large quantity of data are mandatory, which is expensive and less feasible. We collected hematoxylin and eosin-stained image data sets from two hospitals to train the model with a variety of nuclear appearances. Because of the limited number of annotated pathology images, we introduced a small publicly accessible data set of prostate cancer (PCa) with more than 16,000 labeled nuclei. Nevertheless, to construct our proposed model, we developed the DCSA module, an attention mechanism for capturing useful information from raw images. We also used several other artificial intelligence-based segmentation methods and tools to compare their results to our proposed technique.

Results

To prioritize the performance of nuclei segmentation, we evaluated the model's outputs based on the Accuracy, Dice coefficient (DC), and Jaccard coefficient (JC) scores. The proposed technique outperformed the other methods and achieved superior nuclei segmentation with accuracy, DC, and JC of 96.4% (95% confidence interval [CI]: 96.2 - 96.6), 81.8 (95% CI: 80.8 - 83.0), and 69.3 (95% CI: 68.2 - 70.0), respectively, on the internal test data set.

Conclusion

Our proposed method demonstrates superior performance in segmenting cell nuclei of histological images from internal and external datasets, and outperforms many standard segmentation algorithms used for comparative analysis.

A Compound Loss Function With Shape Aware Weight Map for Microscopy Cell Segmentation

Microscopy cell segmentation is a crucial step in biological image analysis and a challenging task. In recent years, deep learning has been widely used to tackle this task, with promising results. A critical aspect of training complex neural networks for this purpose is the selection of the loss function, as it affects the learning process. In the field of cell segmentation, most of the recent research in improving the loss function focuses on addressing the problem of inter-class imbalance. Despite promising achievements, more work is needed, as the challenge of cell segmentation is not only the inter-class imbalance but also the intra-class imbalance (the cost imbalance between the false positives and false negatives of the inference model), the segmentation of cell minutiae, and the missing annotations. To deal with these challenges, in this paper, we propose a new compound loss function employing a shape aware weight map. The proposed loss function is inspired by Youden's J index to handle the problem of inter-class imbalance and uses a focal cross-entropy term to penalize the intra-class imbalance and weight easy/hard samples. The proposed shape aware weight map can handle the problem of missing annotations and facilitate valid segmentation of cell minutiae. Results of evaluations on all ten 2D+time datasets from the public cell tracking challenge demonstrate 1) the superiority of the proposed loss function with the shape aware weight map, and 2) that the performance of recent deep learning-based cell segmentation methods can be improved by using the proposed compound loss function.

Biomarkers of Tumor Heterogeneity in Glioblastoma Multiforme Cohort of TCGA

Tumor Whole Slide Images (WSI) are often heterogeneous, which hinders the discovery of biomarkers in the presence of confounding clinical factors. In this study, we present a pipeline for identifying biomarkers from the Glioblastoma Multiforme (GBM) cohort of WSIs from TCGA archive. The GBM cohort endures many technical artifacts while the discovery of GBM biomarkers is challenged because "age" is the single most confounding factor for predicting outcomes. The proposed approach relies on interpretable features (e.g., nuclear morphometric indices), effective similarity metrics for heterogeneity analysis, and robust statistics for identifying biomarkers. The pipeline first removes artifacts (e.g., pen marks) and partitions each WSI into patches for nuclear segmentation via an extended U-Net for subsequent quantitative representation. Given the variations in fixation and staining that can artificially modulate hematoxylin optical density (HOD), we extended Navab's Lab method to normalize images and reduce the impact of batch effects. The heterogeneity of each WSI is then represented either as probability density functions (PDF) per patient or as the composition of a dictionary predicted from the entire cohort of WSIs. For PDF- or dictionary-based methods, morphometric subtypes are constructed based on distances computed from optimal transport and linkage analysis or consensus clustering with Euclidean distances, respectively. For each inferred subtype, Kaplan-Meier and/or the Cox regression model are used to regress the survival time. Since age is the single most important confounder for predicting survival in GBM and there is an observed violation of the proportionality assumption in the Cox model, we use both age and age-squared coupled with the Likelihood ratio test and forest plots for evaluating competing statistics. Next, the PDF- and dictionary-based methods are combined to identify biomarkers that are predictive of survival. The combined model has the advantage of integrating global (e.g., cohort scale) and local (e.g., patient scale) attributes of morphometric heterogeneity, coupled with robust statistics, to reveal stable biomarkers. The results indicate that, after normalization of the GBM cohort, mean HOD, eccentricity, and cellularity are predictive of survival. Finally, we also stratified the GBM cohort as a function of EGFR expression and published genomic subtypes to reveal genomic-dependent morphometric biomarkers.

Which Pixel to Annotate: A Label-Efficient Nuclei Segmentation Framework

Recently deep neural networks, which require a large amount of annotated samples, have been widely applied in nuclei instance segmentation of H&E stained pathology images. However, it is inefficient and unnecessary to label all pixels for a dataset of nuclei images which usually contain similar and redundant patterns. Although unsupervised and semi-supervised learning methods have been studied for nuclei segmentation, very few works have delved into the selective labeling of samples to reduce the workload of annotation. Thus, in this paper, we propose a novel full nuclei segmentation framework that chooses only a few image patches to be annotated, augments the training set from the selected samples, and achieves nuclei segmentation in a semi-supervised manner. In the proposed framework, we first develop a novel consistency-based patch selection method to determine which image patches are the most beneficial to the training. Then we introduce a conditional single-image GAN with a component-wise discriminator, to synthesize more training samples. Lastly, our proposed framework trains an existing segmentation model with the above augmented samples. The experimental results show that our proposed method could obtain the same-level performance as a fully-supervised baseline by annotating less than 5% pixels on some benchmarks.

Enhanced Pathology Image Quality with Restore-Generative Adversarial Network

Whole slide imaging is becoming a routine procedure in clinical diagnosis. Advanced image analysis techniques have been developed to assist pathologists in disease diagnosis, staging, subtype classification, and risk stratification. Recently, deep learning algorithms have achieved state-of-the-art performances in various imaging analysis tasks, including tumor region segmentation, nuclei detection, and disease classification. However, widespread clinical use of these algorithms is hampered by their performances often degrading due to image quality issues commonly seen in real-world pathology imaging data such as low resolution, blurring regions, and staining variation. Restore-Generative Adversarial Network (GAN), a deep learning model, was developed to improve the imaging qualities by restoring blurred regions, enhancing low resolution, and normalizing staining colors. The results demonstrate that Restore-GAN can significantly improve image quality, which leads to improved model robustness and performance for existing deep learning algorithms in pathology image analysis. Restore-GAN has the potential to be used to facilitate the applications of deep learning models in digital pathology analyses.

Super-resolution and segmentation deep learning for breast cancer histopathology image analysis

Traditionally, a high-performance microscope with a large numerical aperture is required to acquire high-resolution images. However, the images' size is typically tremendous. Therefore, they are not conveniently managed and transferred across a computer network or stored in a limited computer storage system. As a result, image compression is commonly used to reduce image size resulting in poor image resolution. Here, we demonstrate custom convolution neural networks (CNNs) for both super-resolution image enhancement from low-resolution images and characterization of both cells and nuclei from hematoxylin and eosin (H&E) stained breast cancer histopathological images by using a combination of generator and discriminator networks so-called super-resolution generative adversarial network-based on aggregated residual transformation (SRGAN-ResNeXt) to facilitate cancer diagnosis in low resource settings. The results provide high enhancement in image quality where the peak signal-to-noise ratio and structural similarity of our network results are over 30 dB and 0.93, respectively. The derived performance is superior to the results obtained from both the bicubic interpolation and the well-known SRGAN deep-learning methods. In addition, another custom CNN is used to perform image segmentation from the generated high-resolution breast cancer images derived with our model with an average Intersection over Union of 0.869 and an average dice similarity coefficient of 0.893 for the H&E image segmentation results. Finally, we propose the jointly trained SRGAN-ResNeXt and Inception U-net Models, which applied the weights from the individually trained SRGAN-ResNeXt and inception U-net models as the pre-trained weights for transfer learning. The jointly trained model's results are progressively improved and promising. We anticipate these custom CNNs can help resolve the inaccessibility of advanced microscopes or whole slide imaging (WSI) systems to acquire high-resolution images from low-performance microscopes located in remote-constraint settings.

Weakly Supervised Deep Nuclei Segmentation With Sparsely Annotated Bounding Boxes for DNA Image Cytometry

Nuclei segmentation is an essential step in DNA ploidy analysis by image-based cytometry (DNA-ICM) which is widely used in cytopathology and allows an objective measurement of DNA content (ploidy). The routine fully supervised learning-based method requires often tedious and expensive pixel-wise labels. In this paper, we propose a novel weakly supervised nuclei segmentation framework which exploits only sparsely annotated bounding boxes, without any segmentation labels. The key is to integrate the traditional image segmentation and self-training into fully supervised instance segmentation. We first leverage the traditional segmentation to generate coarse masks for each box-annotated nucleus to supervise the training of a teacher model, which is then responsible for both the refinement of these coarse masks and pseudo labels generation of unlabeled nuclei. These pseudo labels and refined masks along with the original manually annotated bounding boxes jointly supervise the training of student model. Both teacher and student share the same architecture and especially the student is initialized by the teacher. We have extensively evaluated our method with both our DNA-ICM dataset and public cytopathological dataset. Without bells and whistles, our method outperforms all existing weakly supervised entries on both datasets. Code and our DNA-ICM dataset are publicly available at https://github.com/CVIU-CSU/Weakly-Supervised-Nuclei-Segmentation.

Imaging Flow Cytometry of Multi-Nuclearity

Multi-nuclearity is a common feature for cells in different cancers. Also, analysis of multi-nuclearity in cultured cells is widely used for evaluating the toxicity of different drugs. Multi-nuclear cells in cancer and under drug treatments form from aberrations in cell division and/or cytokinesis. These cells are a hallmark of cancer progression, and the abundance of multi-nucleated cells often correlates with poor prognosis.The use of standard bright field or fluorescent microscopy to analyze multi-nuclearity at the quantitative level is laborious and can suffer from user bias. Automated slide-scanning microscopy can eliminate scorer bias and improve data collection. However, this method has limitations, such as insufficient visibility of multiple nuclei in the cells attached to the substrate at low magnification.Since quantification of multi-nuclear cells using microscopic methods might be difficult, imaging flow cytometry (IFC) is a method of choice for this. We describe the experimental protocol for the preparation of the samples of multi-nucleated cells from the attached cultures and the algorithm for the analysis of these cells by IFC. Images of multi-nucleated cells obtained after mitotic arrest induced by taxol, as well as cells obtained after cytokinesis blockade by cytochalasin D treatment, can be acquired at a maximal resolution of IFC. We suggest two algorithms for the discrimination of single-nucleus and multi-nucleated cells. The advantages and disadvantages of IFC analysis of multi-nuclear cells in comparison with microscopy are discussed.

A comprehensive review of computational and image analysis techniques for quantitative evaluation of striated muscle tissue architecture

Unbiased evaluation of morphology is crucial to understanding development, mechanics, and pathology of striated muscle tissues. Indeed, the ability of striated muscles to contract and the strength of their contraction is dependent on their tissue-, cellular-, and cytoskeletal-level organization. Accordingly, the study of striated muscles often requires imaging and assessing aspects of their architecture at multiple different spatial scales. While an expert may be able to qualitatively appraise tissues, it is imperative to have robust, repeatable tools to quantify striated myocyte morphology and behavior that can be used to compare across different labs and experiments. There has been a recent effort to define the criteria used by experts to evaluate striated myocyte architecture. In this review, we will describe metrics that have been developed to summarize distinct aspects of striated muscle architecture in multiple different tissues, imaged with various modalities. Additionally, we will provide an overview of metrics and image processing software that needs to be developed. Importantly to any lab working on striated muscle platforms, characterization of striated myocyte morphology using the image processing pipelines discussed in this review can be used to quantitatively evaluate striated muscle tissues and contribute to a robust understanding of the development and mechanics of striated muscles.

A tumor vasculature-based imaging biomarker for predicting response and survival in patients with lung cancer treated with checkpoint inhibitors

Tumor vasculature is a key component of the tumor microenvironment that can influence tumor behavior and therapeutic resistance. We present a new imaging biomarker, quantitative vessel tortuosity (QVT), and evaluate its association with response and survival in patients with non-small cell lung cancer (NSCLC) treated with immune checkpoint inhibitor (ICI) therapies. A total of 507 cases were used to evaluate different aspects of the QVT biomarkers. QVT features were extracted from computed tomography imaging of patients before and after ICI therapy to capture the tortuosity, curvature, density, and branching statistics of the nodule vasculature. Our results showed that QVT features were prognostic of OS (HR = 3.14, 0.95% CI = 1.2 to 9.68, P = 0.0006, C-index = 0.61) and could predict ICI response with AUCs of 0.66, 0.61, and 0.67 on three validation sets. Our study shows that QVT imaging biomarker could potentially aid in predicting and monitoring response to ICI in patients with NSCLC.

Guiding Irregular Nuclear Morphology on Nanopillar Arrays for Malignancy Differentiation in Tumor Cells

For more than a century, abnormal nuclei in tumor cells, presenting subnuclear invaginations and folds on the nuclear envelope, have been known to be associated with high malignancy and poor prognosis. However, current nuclear morphology analysis focuses on the features of the entire nucleus, overlooking the malignancy-related subnuclear features in nanometer scale. The main technical challenge is to probe such tiny and randomly distributed features inside cells. We here employ nanopillar arrays to guide subnuclear features into ordered patterns, enabling their quantification as a strong indicator of cell malignancy. Both breast and liver cancer cells were validated as well as the quantification of nuclear abnormality heterogeneity. The alterations of subnuclear patterns were also explored as effective readouts for drug treatment. We envision that this nanopillar-enabled quantification of subnuclear abnormal features in tumor cells opens a new angle in characterizing malignant cells and studying the unique nuclear biology in cancer.

Segmentation of Tissues and Proliferating Cells in Light-Sheet Microscopy Images of Mouse Embryos Using Convolutional Neural Networks

A variety of genetic mutations affect cell proliferation during organism development, leading to structural birth defects. However, the mechanisms by which these alterations influence the development of the face remain unclear. Cell proliferation and its relation to shape variation can be studied using Light-Sheet Microscopy (LSM) imaging across a range of developmental time points using mouse models. The aim of this work was to develop and evaluate accurate automatic methods based on convolutional neural networks (CNNs) for: (i) tissue segmentation (neural ectoderm and mesenchyme), (ii) cell segmentation in nuclear-stained images, and (iii) segmentation of proliferating cells in phospho-Histone H3 (pHH3)-stained LSM images of mouse embryos. For training and evaluation of the CNN models, 155 to 176 slices from 10 mouse embryo LSM images with corresponding manual segmentations were available depending on the segmentation task. Three U-net CNN models were trained optimizing their loss functions, among other hyper-parameters, depending on the segmentation task. The tissue segmentation achieved a macro-average F-score of 0.84, whereas the inter-observer value was 0.89. The cell segmentation achieved a Dice score of 0.57 and 0.56 for nuclear-stained and pHH3-stained images, respectively, whereas the corresponding inter-observer Dice scores were 0.39 and 0.45, respectively. The proposed pipeline using the U-net CNN architecture can accelerate LSM image analysis and together with the annotated datasets can serve as a reference for comparison of more advanced LSM image segmentation methods in future.

Global Research Trends of Artificial Intelligence on Histopathological Images: A 20-Year Bibliometric Analysis

Cancer has become a major threat to global health care. With the development of computer science, artificial intelligence (AI) has been widely applied in histopathological images (HI) analysis. This study analyzed the publications of AI in HI from 2001 to 2021 by bibliometrics, exploring the research status and the potential popular directions in the future. A total of 2844 publications from the Web of Science Core Collection were included in the bibliometric analysis. The country/region, institution, author, journal, keyword, and references were analyzed by using VOSviewer and CiteSpace. The results showed that the number of publications has grown rapidly in the last five years. The USA is the most productive and influential country with 937 publications and 23,010 citations, and most of the authors and institutions with higher numbers of publications and citations are from the USA. Keyword analysis showed that breast cancer, prostate cancer, colorectal cancer, and lung cancer are the tumor types of greatest concern. Co-citation analysis showed that classification and nucleus segmentation are the main research directions of AI-based HI studies. Transfer learning and self-supervised learning in HI is on the rise. This study performed the first bibliometric analysis of AI in HI from multiple indicators, providing insights for researchers to identify key cancer types and understand the research trends of AI application in HI.

Deep learning applied to analyze patterns from evaporated droplets of Viscum album extracts

This paper introduces a deep learning based methodology for analyzing the self-assembled, fractal-like structures formed in evaporated droplets. To this end, an extensive image database of such structures of the plant extract Viscum album Quercus\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$10^{-3}$$\end{document}10-3 was used, prepared by three different mixing procedures (turbulent, laminar, and diffusion based). The proposed pattern analysis approach is based on two stages: (1) automatic selection of patches that exhibit rich texture along the database; and (2) clustering of patches in accordance with prevalent texture by means of a Dense Convolutional Neural Network. The fractality of the patterns in each cluster is verified through Local Connected Fractal Dimension histograms. Experiments with Gray-Level Co-Occurrence matrices are performed to determine the benefit of the proposed approach in comparison with well established image analysis techniques. For the investigated plant extract, significant differences were found between the production modalities; whereas the patterns obtained by laminar flow showed the highest fractal structure, the patterns obtained by the application of turbulent mixture exhibited the lowest fractality. Our approach is the first to analyze, at the pure image level, the clustering properties of regions of interest within a database of evaporated droplets. This allows a greater description and differentiation of the patterns formed through different mixing procedures.

A deep learning–based algorithm for tall cell detection in papillary thyroid carcinoma

Introduction

According to the World Health Organization, the tall cell variant (TCV) is an aggressive subtype of papillary thyroid carcinoma (PTC) comprising at least 30% epithelial cells two to three times as tall as they are wide. In practice, applying this definition is difficult causing substantial interobserver variability. We aimed to train a deep learning algorithm to detect and quantify the proportion of tall cells (TCs) in PTC.

Methods

We trained the deep learning algorithm using supervised learning, testing it on an independent dataset, and further validating it on an independent set of 90 PTC samples from patients treated at the Hospital District of Helsinki and Uusimaa between 2003 and 2013. We compared the algorithm-based TC percentage to the independent scoring by a human investigator and how those scorings associated with disease outcomes. Additionally, we assessed the TC score in 71 local and distant tumor relapse samples from patients with aggressive disease.

Results

In the test set, the deep learning algorithm detected TCs with a sensitivity of 93.7% and a specificity of 94.5%, whereas the sensitivity fell to 90.9% and specificity to 94.1% for non-TC areas. In the validation set, the deep learning algorithm TC scores correlated with a diminished relapse-free survival using cutoff points of 10% (p = 0.044), 20% (p < 0.01), and 30% (p = 0.036). The visually assessed TC score did not statistically significantly predict survival at any of the analyzed cutoff points. We observed no statistically significant difference in the TC score between primary tumors and relapse tumors determined by the deep learning algorithm or visually.

Conclusions

We present a novel deep learning–based algorithm to detect tall cells, showing that a high deep learning–based TC score represents a statistically significant predictor of less favorable relapse-free survival in PTC.

SAU-Net: A Unified Network for Cell Counting in 2D and 3D Microscopy Images

Image-based cell counting is a fundamental yet challenging task with wide applications in biological research. In this paper, we propose a novel unified deep network framework designed to solve this problem for various cell types in both 2D and 3D images. Specifically, we first propose SAU-Net for cell counting by extending the segmentation network U-Net with a Self-Attention module. Second, we design an extension of Batch Normalization (BN) to facilitate the training process for small datasets. In addition, a new 3D benchmark dataset based on the existing mouse blastocyst (MBC) dataset is developed and released to the community. Our SAU-Net achieves state-of-the-art results on four benchmark 2D datasets - synthetic fluorescence microscopy (VGG) dataset, Modified Bone Marrow (MBM) dataset, human subcutaneous adipose tissue (ADI) dataset, and Dublin Cell Counting (DCC) dataset, and the new 3D dataset, MBC. The BN extension is validated using extensive experiments on the 2D datasets, since GPU memory constraints preclude use of 3D datasets. The source code is available at https://github.com/mzlr/sau-net.

Non-invasive measurement of nuclear relative stiffness from quantitative analysis of microscopy data

The connection between the properties of a cell tissue and those of the single constituent cells remains to be elucidated. At the purely mechanical level, the degree of rigidity of different cellular components, such as the nucleus and the cytoplasm, modulates the interplay between the cell inner processes and the external environment, while simultaneously mediating the mechanical interactions between neighboring cells. Being able to quantify the correlation between single-cell and tissue properties would improve our mechanobiological understanding of cell tissues. Here we develop a methodology to quantitatively extract a set of structural and motility parameters from the analysis of time-lapse movies of nuclei belonging to jammed and flocking cell monolayers. We then study in detail the correlation between the dynamical state of the tissue and the deformation of the nuclei. We observe that the nuclear deformation rate linearly correlates with the local divergence of the velocity field, which leads to a non-invasive estimate of the elastic modulus of the nucleus relative to the one of the cytoplasm. We also find that nuclei belonging to flocking monolayers, subjected to larger mechanical perturbations, are about two time stiffer than nuclei belonging to dynamically arrested monolayers, in agreement with atomic force microscopy results. Our results demonstrate a non-invasive route to the determination of nuclear relative stiffness for cells in a monolayer.

NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer

BACKGROUND

Deep learning enables accurate high-resolution mapping of cells and tissue structures that can serve as the foundation of interpretable machine-learning models for computational pathology. However, generating adequate labels for these structures is a critical barrier, given the time and effort required from pathologists.

RESULTS

This article describes a novel collaborative framework for engaging crowds of medical students and pathologists to produce quality labels for cell nuclei. We used this approach to produce the NuCLS dataset, containing >220,000 annotations of cell nuclei in breast cancers. This builds on prior work labeling tissue regions to produce an integrated tissue region- and cell-level annotation dataset for training that is the largest such resource for multi-scale analysis of breast cancer histology. This article presents data and analysis results for single and multi-rater annotations from both non-experts and pathologists. We present a novel workflow that uses algorithmic suggestions to collect accurate segmentation data without the need for laborious manual tracing of nuclei. Our results indicate that even noisy algorithmic suggestions do not adversely affect pathologist accuracy and can help non-experts improve annotation quality. We also present a new approach for inferring truth from multiple raters and show that non-experts can produce accurate annotations for visually distinctive classes.

CONCLUSIONS

This study is the most extensive systematic exploration of the large-scale use of wisdom-of-the-crowd approaches to generate data for computational pathology applications.

Chronic Lymphocytic Leukemia Progression Diagnosis with Intrinsic Cellular Patterns via Unsupervised Clustering

Simple Summary

Distinguishing between chronic lymphocytic leukemia (CLL), accelerated CLL (aCLL), and full-blown transformation to diffuse large B-cell lymphoma (Richter transformation; RT) has significant clinical implications. Identifying cellular phenotypes via unsupervised clustering provides the most robust analytic performance in analyzing digitized pathology slides. This study serves as a proof of concept that using an unsupervised machine learning scheme can enhance diagnostic accuracy.

Abstract

Identifying the progression of chronic lymphocytic leukemia (CLL) to accelerated CLL (aCLL) or transformation to diffuse large B-cell lymphoma (Richter transformation; RT) has significant clinical implications as it prompts a major change in patient management. However, the differentiation between these disease phases may be challenging in routine practice. Unsupervised learning has gained increased attention because of its substantial potential in data intrinsic pattern discovery. Here, we demonstrate that cellular feature engineering, identifying cellular phenotypes via unsupervised clustering, provides the most robust analytic performance in analyzing digitized pathology slides (accuracy = 0.925, AUC = 0.978) when compared to alternative approaches, such as mixed features, supervised features, unsupervised/mixed/supervised feature fusion and selection, as well as patch-based convolutional neural network (CNN) feature extraction. We further validate the reproducibility and robustness of unsupervised feature extraction via stability and repeated splitting analysis, supporting its utility as a diagnostic aid in identifying CLL patients with histologic evidence of disease progression. The outcome of this study serves as proof of principle using an unsupervised machine learning scheme to enhance the diagnostic accuracy of the heterogeneous histology patterns that pathologists might not easily see.

Automatic Cancer Cell Taxonomy Using an Ensemble of Deep Neural Networks

Microscopic image-based analysis has been intensively performed for pathological studies and diagnosis of diseases. However, mis-authentication of cell lines due to misjudgments by pathologists has been recognized as a serious problem. To address this problem, we propose a deep-learning-based approach for the automatic taxonomy of cancer cell types. A total of 889 bright-field microscopic images of four cancer cell lines were acquired using a benchtop microscope. Individual cells were further segmented and augmented to increase the image dataset. Afterward, deep transfer learning was adopted to accelerate the classification of cancer types. Experiments revealed that the deep-learning-based methods outperformed traditional machine-learning-based methods. Moreover, the Wilcoxon signed-rank test showed that deep ensemble approaches outperformed individual deep-learning-based models (p < 0.001) and were in effect to achieve the classification accuracy up to 97.735%. Additional investigation with the Wilcoxon signed-rank test was conducted to consider various network design choices, such as the type of optimizer, type of learning rate scheduler, degree of fine-tuning, and use of data augmentation. Finally, it was found that the using data augmentation and updating all the weights of a network during fine-tuning improve the overall performance of individual convolutional neural network models.

DenseRes-Unet: Segmentation of overlapped/clustered nuclei from multi organ histopathology images

Cancer is the second deadliest disease globally that can affect any human body organ. Early detection of cancer can increase the chances of survival in humans. Morphometric appearances of histopathology images make it difficult to segment nuclei effectively. We proposed a model to segment overlapped nuclei from H&E stained images. U-Net model achieved state-of-the-art performance in many medical image segmentation tasks; however, we modified the U-Net to learn a distinct set of consistent features. In this paper, we proposed the DenseRes-Unet model by integrating dense blocks in the last layers of the encoder block of U-Net, focused on relevant features from previous layers of the model. Moreover, we take advantage of residual connections with Atrous blocks instead of conventional skip connections, which helps to reduce the semantic gap between encoder and decoder paths. The distance map and binary threshold techniques intensify the nuclei interior and contour information in the images, respectively. The distance map is used to detect the center point of nuclei; moreover, it differentiates among nuclei interior boundary and core area. The distance map lacks a contour problem, which is resolved by using a binary threshold. Binary threshold helps to enhance the pixels around nuclei. Afterward, we fed images into the proposed DenseRes-Unet model, a deep, fully convolutional network to segment nuclei in the images. We have evaluated our model on four publicly available datasets for Nuclei segmentation to validate the model's performance. Our proposed model achieves 89.77% accuracy 90.36% F1-score, and 78.61% Aggregated Jaccard Index (AJI) on Multi organ Nucleus Segmentation (MoNuSeg).