Automatic segmentation for cell images based on bottleneck detection and ellipse fitting

An efficient heuristic for geometric analysis of cell deformations

Sickle cell disease causes erythrocytes to become sickle-shaped, affecting their movement in the bloodstream and reducing oxygen delivery. It has a high global prevalence and places a significant burden on healthcare systems, especially in resource-limited regions. Automated classification of sickle cells in blood images is crucial, allowing the specialist to reduce the effort required and avoid errors when quantifying the deformed cells and assessing the severity of a crisis. Recent studies have proposed various erythrocyte representation and classification methods (Jennifer et al., 2023 [1]). Since classification depends solely on cell shape, a suitable approach models erythrocytes as closed planar curves in shape space (Epifanio et al., 2020). This approach employs elastic distances between shapes, which are invariant under rotations, translations, scaling, and reparameterizations, ensuring consistent distance measurements regardless of the curves' position, starting point, or traversal speed. While previous methods exploiting shape space distances had achieved high accuracy, we refined the model by considering the geometric characteristics of healthy and sickled erythrocytes. Our method proposes (1) to employ a fixed parameterization based on the major axis of each cell to compute distances and (2) to align each cell with two templates using this parameterization before computing distances. Aligning shapes to templates before distance computation, a concept successfully applied in areas such as molecular dynamics (Richmond et al., 2004 [2]), and using a fixed parameterization, instead of minimizing distances across all possible parameterizations, simplifies calculations. This strategy achieves 96.03% accuracy rate in both supervised classification and unsupervised clustering. Our method ensures efficient erythrocyte classification, maintaining or improving accuracy over shape space models while significantly reducing computational costs.

Homopolymer switches mediate adaptive mutability in mismatch repair-deficient colorectal cancer

Mismatch repair (MMR)-deficient cancer evolves through the stepwise erosion of coding homopolymers in target genes. Curiously, the MMR genes MutS homolog 6 (MSH6) and MutS homolog 3 (MSH3) also contain coding homopolymers, and these are frequent mutational targets in MMR-deficient cancers. The impact of incremental MMR mutations on MMR-deficient cancer evolution is unknown. Here we show that microsatellite instability modulates DNA repair by toggling hypermutable mononucleotide homopolymer runs in MSH6 and MSH3 through stochastic frameshift switching. Spontaneous mutation and reversion modulate subclonal mutation rate, mutation bias and HLA and neoantigen diversity. Patient-derived organoids corroborate these observations and show that MMR homopolymer sequences drift back into reading frame in the absence of immune selection, suggesting a fitness cost of elevated mutation rates. Combined experimental and simulation studies demonstrate that subclonal immune selection favors incremental MMR mutations. Overall, our data demonstrate that MMR-deficient colorectal cancers fuel intratumor heterogeneity by adapting subclonal mutation rate and diversity to immune selection.

BoNuS: Boundary Mining for Nuclei Segmentation With Partial Point Labels

Nuclei segmentation is a fundamental prerequisite in the digital pathology workflow. The development of automated methods for nuclei segmentation enables quantitative analysis of the wide existence and large variances in nuclei morphometry in histopathology images. However, manual annotation of tens of thousands of nuclei is tedious and time-consuming, which requires significant amount of human effort and domain-specific expertise. To alleviate this problem, in this paper, we propose a weakly-supervised nuclei segmentation method that only requires partial point labels of nuclei. Specifically, we propose a novel boundary mining framework for nuclei segmentation, named BoNuS, which simultaneously learns nuclei interior and boundary information from the point labels. To achieve this goal, we propose a novel boundary mining loss, which guides the model to learn the boundary information by exploring the pairwise pixel affinity in a multiple-instance learning manner. Then, we consider a more challenging problem, i.e., partial point label, where we propose a nuclei detection module with curriculum learning to detect the missing nuclei with prior morphological knowledge. The proposed method is validated on three public datasets, MoNuSeg, CPM, and CoNIC datasets. Experimental results demonstrate the superior performance of our method to the state-of-the-art weakly-supervised nuclei segmentation methods. Code: https://github.com/hust-linyi/bonus.

CellViT: Vision Transformers for precise cell segmentation and classification

Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in combination with large scale pre-training in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F1-detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT.

An approach of separating the overlapped cells or nuclei based on the outer Canny edges and morphological erosion

In biomedicine, the automatic processing of medical microscope images plays a key role in the subsequent analysis and diagnosis. Cell or nucleus segmentation is one of the most challenging tasks for microscope image processing. Due to the frequently occurred overlapping, few segmentation methods can achieve satisfactory segmentation accuracy yet. In this paper, we propose an approach to separate the overlapped cells or nuclei based on the outer Canny edges and morphological erosion. The threshold selection is first used to segment the foreground and background of cell or nucleus images. For each binary connected domain in the segmentation image, an intersection based edge selection method is proposed to choose the outer Canny edges of the overlapped cells or nuclei. The outer Canny edges are used to generate a binary cell or nucleus image that is then used to compute the cell or nucleus seeds by the proposed morphological erosion method. The nuclei of the Human U2OS cells, the mouse NIH3T3 cells and the synthetic cells are used for evaluating our proposed approach. The quantitative quantification accuracy is computed by the Dice score and 95.53% is achieved by the proposed approach. Both the quantitative and the qualitative comparisons show that the accuracy of the proposed approach is better than those of the area constrained morphological erosion (ACME) method, the iterative erosion (IE) method, the morphology and watershed (MW) method, the Generalized Laplacian of Gaussian filters (GLGF) method and ellipse fitting (EF) method in separating the cells or nuclei in three publicly available datasets.

Nuclei instance segmentation using a transformer-based graph convolutional network and contextual information augmentation

Nucleus instance segmentation is an important task in medical image analysis involving cell-level pathological analysis and is of great significance for many biomedical applications, such as disease diagnosis and drug screening. However, the high-density and tight-contact between cells is a common feature of most cell images, which poses a great technical challenge for nuclei instance segmentation. The latest research focuses on CNN-based methods for nuclei instance segmentation, which typically rely on bounding box regression and non-maximum suppression to locate nuclei. However, this frequently results in poor local bounding boxes for nuclei that are adhered or clustered together. In response to the challenges of high-density and tight-contact in cellular images, we propose a novel end-to-end nuclei instance segmentation model. Specifically, we first employ the Swin Transformer as the backbone network of our model, which captures global multi-scale information by combining the global modelling capability of transformers and the local modelling capability of convolutional neural networks (CNNs). Additionally, we integrate a graph convolutional feature fusion module (GCFM), that combines deep and shallow features to learn an affinity matrix. The module also adopts graph convolution to guide the network in learning the object-level local information. Finally, we design a hybrid dilated convolution module (HDC) and insert it into the backbone network to enhance the contextual information over a large range. These components assist the network in extracting rich features. The experimental results demonstrate that our algorithm outperforms several state-of-the-art models on the DSB2018 and LIVECell datasets.

Cell segmentation in fluorescence microscopy images based on multi-scale histogram thresholding

Cell segmentation from fluorescent microscopy images plays an important role in various applications, such as disease mechanism assessment and drug discovery research. Exiting segmentation methods often adopt image binarization as the first step, through which the foreground cell is separated from the background so that the subsequent processing steps can be greatly facilitated. To pursue this goal, a histogram thresholding can be performed on the input image, which first applies a Gaussian smoothing to suppress the jaggedness of the histogram curve and then exploits Rosin's method to determine a threshold for conducting image binarization. However, an inappropriate amount of smoothing could lead to the inaccurate segmentation of cells. To address this crucial problem, a multi-scale histogram thresholding (MHT) technique is proposed in the present paper, where the scale refers to the standard deviation of the Gaussian that determines the amount of smoothing. To be specific, the image histogram is smoothed at three chosen scales first, and then the smoothed histogram curves are fused to conduct image binarization via thresholding. To further improve the segmentation accuracy and overcome the difficulty of extracting overlapping cells, our proposed MHT technique is incorporated into a multi-scale cell segmentation framework, in which a region-based ellipse fitting technique is adopted to identify overlapping cells. Extensive experimental results obtained on benchmark datasets show that the new method can deliver superior performance compared to the current state-of-the-arts.

Dual Encoder Attention U-net for Nuclei Segmentation

Nuclei segmentation in whole slide images (WSIs) stained with Hematoxylin and Eosin (H&E) dye, is a key step in computational pathology which aims to automate the laborious process of manual counting and segmentation. Nuclei segmentation is a challenging problem that involves challenges such as touching nuclei resolution, small-sized nuclei, size, and shape variations. With the advent of deep learning, convolution neural networks (CNNs) have shown a powerful ability to extract effective representations from microscopic H&E images. We propose a novel dual encoder Attention U-net (DEAU) deep learning architecture and pseudo hard attention gating mechanism, to enhance the attention to target instances. We added a new secondary encoder to the attention U-net to capture the best attention for a given input. Since H captures nuclei information, we propose a stain-separated H channel as input to the secondary encoder. The role of the secondary encoder is to transform attention prior to different spatial resolutions while learning significant attention information. The proposed DEAU performance was evaluated on three publicly available H&E data sets for nuclei segmentation from different research groups. Experimental results show that our approach outperforms other attention-based approaches for nuclei segmentation.

Tertiary lymphoid structures (TLS) identification and density assessment on H&E-stained digital slides of lung cancer

Tertiary lymphoid structures (TLS) are ectopic aggregates of lymphoid cells in inflamed, infected, or tumoral tissues that are easily recognized on an H&E histology slide as discrete entities, distinct from lymphocytes. TLS are associated with improved cancer prognosis but there is no standardised method available to quantify their presence. Previous studies have used immunohistochemistry to determine the presence of specific cells as a marker of the TLS. This has now been proven to be an underestimate of the true number of TLS. Thus, we propose a methodology for the automated identification and quantification of TLS, based on H&E slides. We subsequently determined the mathematical criteria defining a TLS. TLS regions were identified through a deep convolutional neural network and segmentation of lymphocytes was performed through an ellipsoidal model. This methodology had a 92.87% specificity at 95% sensitivity, 88.79% specificity at 98% sensitivity and 84.32% specificity at 99% sensitivity level based on 144 TLS annotated H&E slides implying that the automated approach was able to reproduce the histopathologists’ assessment with great accuracy. We showed that the minimum number of lymphocytes within TLS is 45 and the minimum TLS area is 6,245μm². Furthermore, we have shown that the density of the lymphocytes is more than 3 times those outside of the TLS. The mean density and standard deviation of lymphocytes within a TLS area are 0.0128/μm² and 0.0026/μm² respectively compared to 0.004/μm² and 0.001/μm² in non-TLS regions. The proposed methodology shows great potential for automated identification and quantification of the TLS density on digital H&E slides.

A generic approach for cell segmentation based on Gabor filtering and area-constrained ultimate erosion

Nowadays, the demand for segmenting different types of cells imaged by microscopes is increased tremendously. The requirements for the segmentation accuracy are becoming stricter. Because of the great diversity of cells, no traditional methods could segment various types of cells with adequate accuracy. In this paper, we aim to propose a generic approach that is capable of segmenting various types of cells robustly and counting the total number of cells accurately. To this end, we utilize the gradients of cells instead of intensity for cell segmentation because the gradients are less affected by the global intensity variations. To improve the segmentation accuracy, we utilize the Gabor filter to increase the intensity uniformity of the gradient image. To get the optimal segmentation, we utilize the slope difference distribution based threshold selection method to segment the Gabor filtered gradient image. At last, we propose an area-constrained ultimate erosion method to separate the connected cells robustly. Twelve types of cells are used to test the proposed approach in this paper. Experimental results showed that the proposed approach is very promising in meeting the strict accuracy requirements for many applications.

An Augmented Cell Segmentation in Fluorescent in Situ Hybridization Images

Fluorescence in situ hybridization (FISH) surpass previously available technology to become a foremost biological assay, which can provide reliable imaging biomarkers to diagnose cancer and genetic disorders in the cellular level. In order to guarantee the validity of the quality analysis in cell images, it is significant to accurately segment the cell touching regions. We previously structured a mini-U-net to precisely capture cell regions, but this method sometimes can not separate multiple cells that are attached to each other. This work aims to solve this matter by applying cell identification results to provide more accurate prior information for the watershed to describe the cell boundaries. Validation results on 458 cells showed that Dice coefficients and intersection over union were improved from 81.92% to 83.98% and from 68.34% to 73.83% (p=0.03), respectively. The improved results indicated that cell identification is an effective means to handie the cell touching and produce more accurate cell segmentation.

Clumped Nuclei Segmentation with Adjacent Point Match and Local Shape-Based Intensity Analysis in Fluorescence Microscopy Images

Highly clumped nuclei captured in fluorescence microscopy images are commonly observed in a wide spectrum of tissue-related biomedical investigations. To ensure the quality of downstream biomedical analyses, it is essential to accurately segment clustered nuclei. However, this presents a technical challenge as fluorescence intensity alone is often insufficient for recovering the true nuclei boundaries. In this paper, we propose an segmentation algorithm that identifies point pair connection candidates and evaluates adjacent point connections with a formulated ellipse fitting quality indicator. After connection relationships are determined, we recover the resulting dividing paths by following points with specific eigenvalues from the image Hessian in a constrained searching space. We validate our algorithm with 560 image patches from two classes of tumor regions of seven brain tumor patients. Both qualitative and quantitative experimental results suggest that our algorithm is promising for dividing overlapped nuclei in fluorescence microscopy images widely used in various biomedical research.

Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images

Nuclear segmentation and classification within Haematoxylin & Eosin stained histology images is a fundamental prerequisite in the digital pathology work-flow. The development of automated methods for nuclear segmentation and classification enables the quantitative analysis of tens of thousands of nuclei within a whole-slide pathology image, opening up possibilities of further analysis of large-scale nuclear morphometry. However, automated nuclear segmentation and classification is faced with a major challenge in that there are several different types of nuclei, some of them exhibiting large intra-class variability such as the nuclei of tumour cells. Additionally, some of the nuclei are often clustered together. To address these challenges, we present a novel convolutional neural network for simultaneous nuclear segmentation and classification that leverages the instance-rich information encoded within the vertical and horizontal distances of nuclear pixels to their centres of mass. These distances are then utilised to separate clustered nuclei, resulting in an accurate segmentation, particularly in areas with overlapping instances. Then, for each segmented instance the network predicts the type of nucleus via a devoted up-sampling branch. We demonstrate state-of-the-art performance compared to other methods on multiple independent multi-tissue histology image datasets. As part of this work, we introduce a new dataset of Haematoxylin & Eosin stained colorectal adenocarcinoma image tiles, containing 24,319 exhaustively annotated nuclei with associated class labels.

A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images

This paper addresses the task of nuclei segmentation in high-resolution histopathology images. We propose an automatic end-to-end deep neural network algorithm for segmentation of individual nuclei. A nucleus-boundary model is introduced to predict nuclei and their boundaries simultaneously using a fully convolutional neural network. Given a color-normalized image, the model directly outputs an estimated nuclei map and a boundary map. A simple, fast, and parameter-free post-processing procedure is performed on the estimated nuclei map to produce the final segmented nuclei. An overlapped patch extraction and assembling method is also designed for seamless prediction of nuclei in large whole-slide images. We also show the effectiveness of data augmentation methods for nuclei segmentation task. Our experiments showed our method outperforms prior state-of-the-art methods. Moreover, it is efficient that one 1000×1000 image can be segmented in less than 5 s. This makes it possible to precisely segment the whole-slide image in acceptable time. The source code is available at https://github.com/easycui/nuclei_segmentation . Graphical Abstract The neural network for nuclei segmentation.

A novel generic dictionary-based denoising method for improving noisy and densely packed nuclei segmentation in 3D time-lapse fluorescence microscopy images

Time-lapse fluorescence microscopy is an essential technique for quantifying various characteristics of cellular processes, i.e. cell survival, migration, and differentiation. To perform high-throughput quantification of cellular processes, nuclei segmentation and tracking should be performed in an automated manner. Nevertheless, nuclei segmentation and tracking are challenging tasks due to embedded noise, intensity inhomogeneity, shape variation as well as a weak boundary of nuclei. Although several nuclei segmentation approaches have been reported in the literature, dealing with embedded noise remains the most challenging part of any segmentation algorithm. We propose a novel denoising algorithm, based on sparse coding, that can both enhance very faint and noisy nuclei signal but simultaneously detect nuclei position accurately. Furthermore our method is based on a limited number of parameters, with only one being critical, which is the approximate size of the objects of interest. We also show that our denoising method coupled with classical segmentation method works properly in the context of the most challenging cases. To evaluate the performance of the proposed method, we tested our method on two datasets from the cell tracking challenge. Across all datasets, the proposed method achieved satisfactory results with 96:96% recall for the C. elegans dataset. Besides, in the Drosophila dataset, our method achieved very high recall (99:3%).