A novel deep learning-based 3D cell segmentation framework for future image-based disease detection

Continuous distribution of cancer cells in the cell cycle unveiled by AI-segmented imaging of 37,000 HeLa FUCCI cells

Classification of live or fixed cells based on their unlabeled microscopic images would be a powerful tool for cell biology and pathology. For such software, the first step is the generation of a ground truth database that can be used for training and testing AI classification algorithms. The Application of cells expressing fluorescent reporter proteins allows the building of ground truth datasets in a straightforward way. In this study, we present an automated imaging pipeline utilizing the Cellpose algorithm for the precise cell segmentation and measurement of fluorescent cellular intensities across multiple channels. We analyzed the cell cycle of HeLa-FUCCI cells expressing fluorescent red and green reporter proteins at various levels depending on the cell cycle state. To build the dataset, 37,000 fixed cells were automatically scanned using a standard motorized microscope, capturing phase contrast and fluorescent red/green images. The fluorescent pixel intensity of each cell was integrated to calculate the total fluorescence of cells based on cell segmentation in the phase contrast channel. It resulted in a precise intensity value for each cell in both channels. Furthermore, we conducted a comparative analysis of Cellpose 1.0 and Cellpose 2.0 in cell segmentation performance. Cellpose 2.0 demonstrated notable improvements, achieving a significantly reduced false positive rate of 2.7 % and 1.4 % false negative. The cellular fluorescence was visualized in a 2D plot (map) based on the red and green intensities of the FUCCI construct revealing the continuous distribution of cells in the cell cycle. This 2D map enables the selection and potential isolation of single cells in a specific phase. In the corresponding heatmap, two clusters appeared representing cells in the red and green states. Our pipeline allows the high-throughput and accurate measurement of cellular fluorescence providing extensive statistical information on thousands of cells with potential applications in developmental and cancer biology. Furthermore, our method can be used to build ground truth datasets automatically for training and testing AI cell classification. Our automated pipeline can be used to analyze thousands of cells within 2 h after putting the sample onto the microscope.

A general algorithm for consensus 3D cell segmentation from 2D segmented stacks

Cell segmentation is the fundamental task. Only by segmenting, can we define the quantitative spatial unit for collecting measurements to draw biological conclusions. Deep learning has revolutionized 2D cell segmentation, enabling generalized solutions across cell types and imaging modalities. This has been driven by the ease of scaling up image acquisition, annotation and computation. However 3D cell segmentation, which requires dense annotation of 2D slices still poses significant challenges. Labelling every cell in every 2D slice is prohibitive. Moreover it is ambiguous, necessitating cross-referencing with other orthoviews. Lastly, there is limited ability to unambiguously record and visualize 1000's of annotated cells. Here we develop a theory and toolbox, u-Segment3D for 2D-to-3D segmentation, compatible with any 2D segmentation method. Given optimal 2D segmentations, u-Segment3D generates the optimal 3D segmentation without data training, as demonstrated on 11 real life datasets, >70,000 cells, spanning single cells, cell aggregates and tissue.

Automated segmentation and recognition of C. elegans whole-body cells

MOTIVATION

Accurate segmentation and recognition of C.elegans cells are critical for various biological studies, including gene expression, cell lineages, and cell fates analysis at single-cell level. However, the highly dense distribution, similar shapes, and inhomogeneous intensity profiles of whole-body cells in 3D fluorescence microscopy images make automatic cell segmentation and recognition a challenging task. Existing methods either rely on additional fiducial markers or only handle a subset of cells. Given the difficulty or expense associated with generating fiducial features in many experimental settings, a marker-free approach capable of reliably segmenting and recognizing C.elegans whole-body cells is highly desirable.

RESULTS

We report a new pipeline, called automated segmentation and recognition (ASR) of cells, and applied it to 3D fluorescent microscopy images of L1-stage C.elegans with 558 whole-body cells. A novel displacement vector field based deep learning model is proposed to address the problem of reliable segmentation of highly crowded cells with blurred boundary. We then realize the cell recognition by encoding and exploiting statistical priors on cell positions and structural similarities of neighboring cells. To the best of our knowledge, this is the first method successfully applied to the segmentation and recognition of C.elegans whole-body cells. The ASR-segmentation module achieves an F1-score of 0.8956 on a dataset of 116 C.elegans image stacks with 64 728 cells (accuracy 0.9880, AJI 0.7813). Based on the segmentation results, the ASR recognition module achieved an average accuracy of 0.8879. We also show ASR's applicability to other cell types, e.g. platynereis and rat kidney cells.

AVAILABILITY AND IMPLEMENTATION

The code is available at https://github.com/reaneyli/ASR.

3DCellComposer - A Versatile Pipeline Utilizing 2D Cell Segmentation Methods for 3D Cell Segmentation

Background

Cell segmentation is crucial in bioimage informatics, as its accuracy directly impacts conclusions drawn from cellular analyses. While many approaches to 2D cell segmentation have been described, 3D cell segmentation has received much less attention. 3D segmentation faces significant challenges, including limited training data availability due to the difficulty of the task for human annotators, and inherent three-dimensional complexity. As a result, existing 3D cell segmentation methods often lack broad applicability across different imaging modalities.

Results

To address this, we developed a generalizable approach for using 2D cell segmentation methods to produce accurate 3D cell segmentations. We implemented this approach in 3DCellComposer, a versatile, open-source package that allows users to choose any existing 2D segmentation model appropriate for their tissue or cell type(s) without requiring any additional training. Importantly, we have enhanced our open source CellSegmentationEvaluator quality evaluation tool to support 3D images. It provides metrics that allow selection of the best approach for a given imaging source and modality, without the need for human annotations to assess performance. Using these metrics, we demonstrated that our approach produced high-quality 3D segmentations of tissue images, and that it could outperform an existing 3D segmentation method on the cell culture images with which it was trained.

Conclusions

3DCellComposer, when paired with well-trained 2D segmentation models, provides an important alternative to acquiring human-annotated 3D images for new sample types or imaging modalities and then training 3D segmentation models using them. It is expected to be of significant value for large scale projects such as the Human BioMolecular Atlas Program.

Deep Learning-Based 3D Single-Cell Imaging Analysis Pipeline Enables Quantification of Cell-Cell Interaction Dynamics in the Tumor Microenvironment

The three-dimensional (3D) tumor microenvironment (TME) comprises multiple interacting cell types that critically impact tumor pathology and therapeutic response. Efficient 3D imaging assays and analysis tools could facilitate profiling and quantifying distinctive cell-cell interaction dynamics in the TMEs of a wide spectrum of human cancers. Here, we developed a 3D live-cell imaging assay using confocal microscopy of patient-derived tumor organoids and a software tool, SiQ-3D (single-cell image quantifier for 3D), that optimizes deep learning (DL)-based 3D image segmentation, single-cell phenotype classification, and tracking to automatically acquire multidimensional dynamic data for different interacting cell types in the TME. An organoid model of tumor cells interacting with natural killer cells was used to demonstrate the effectiveness of the 3D imaging assay to reveal immuno-oncology dynamics as well as the accuracy and efficiency of SiQ-3D to extract quantitative data from large 3D image datasets. SiQ-3D is Python-based, publicly available, and customizable to analyze data from both in vitro and in vivo 3D imaging. The DL-based 3D imaging analysis pipeline can be employed to study not only tumor interaction dynamics with diverse cell types in the TME but also various cell-cell interactions involved in other tissue/organ physiology and pathology.

SIGNIFICANCE

A 3D single-cell imaging pipeline that quantifies cancer cell interaction dynamics with other TME cell types using primary patient-derived samples can elucidate how cell-cell interactions impact tumor behavior and treatment responses.

Facilitating cell segmentation with the projection-enhancement network

Contemporary approaches to instance segmentation in cell science use 2D or 3D convolutional networks depending on the experiment and data structures. However, limitations in microscopy systems or efforts to prevent phototoxicity commonly require recording sub-optimally sampled data that greatly reduces the utility of such 3D data, especially in crowded sample space with significant axial overlap between objects. In such regimes, 2D segmentations are both more reliable for cell morphology and easier to annotate. In this work, we propose the projection enhancement network (PEN), a novel convolutional module which processes the sub-sampled 3D data and produces a 2D RGB semantic compression, and is trained in conjunction with an instance segmentation network of choice to produce 2D segmentations. Our approach combines augmentation to increase cell density using a low-density cell image dataset to train PEN, and curated datasets to evaluate PEN. We show that with PEN, the learned semantic representation in CellPose encodes depth and greatly improves segmentation performance in comparison to maximum intensity projection images as input, but does not similarly aid segmentation in region-based networks like Mask-RCNN. Finally, we dissect the segmentation strength against cell density of PEN with CellPose on disseminated cells from side-by-side spheroids. We present PEN as a data-driven solution to form compressed representations of 3D data that improve 2D segmentations from instance segmentation networks.

Contributions of deep learning to automated numerical modelling of the interaction of electric fields and cartilage tissue based on 3D images

Electric fields find use in tissue engineering but also in sensor applications besides the broad classical application range. Accurate numerical models of electrical stimulation devices can pave the way for effective therapies in cartilage regeneration. To this end, the dielectric properties of the electrically stimulated tissue have to be known. However, knowledge of the dielectric properties is scarce. Electric field-based methods such as impedance spectroscopy enable determining the dielectric properties of tissue samples. To develop a detailed understanding of the interaction of the employed electric fields and the tissue, fine-grained numerical models based on tissue-specific 3D geometries are considered. A crucial ingredient in this approach is the automated generation of numerical models from biomedical images. In this work, we explore classical and artificial intelligence methods for volumetric image segmentation to generate model geometries. We find that deep learning, in particular the StarDist algorithm, permits fast and automatic model geometry and discretisation generation once a sufficient amount of training data is available. Our results suggest that already a small number of 3D images (23 images) is sufficient to achieve 80% accuracy on the test data. The proposed method enables the creation of high-quality meshes without the need for computer-aided design geometry post-processing. Particularly, the computational time for the geometrical model creation was reduced by half. Uncertainty quantification as well as a direct comparison between the deep learning and the classical approach reveal that the numerical results mainly depend on the cell volume. This result motivates further research into impedance sensors for tissue characterisation. The presented approach can significantly improve the accuracy and computational speed of image-based models of electrical stimulation for tissue engineering applications.

In-silico and in-vitro morphometric analysis of intestinal organoids

Organoids offer a powerful model to study cellular self-organisation, the growth of specific tissue morphologies in-vitro, and to assess potential medical therapies. However, the intrinsic mechanisms of these systems are not entirely understood yet, which can result in variability of organoids due to differences in culture conditions and basement membrane extracts used. Improving the standardisation of organoid cultures is essential for their implementation in clinical protocols. Developing tools to assess and predict the behaviour of these systems may produce a more robust and standardised biological model to perform accurate clinical studies. Here, we developed an algorithm to automate crypt-like structure counting on intestinal organoids in both in-vitro and in-silico images. In addition, we modified an existing two-dimensional agent-based mathematical model of intestinal organoids to better describe the system physiology, and evaluated its ability to replicate budding structures compared to new experimental data we generated. The crypt-counting algorithm proved useful in approximating the average number of budding structures found in our in-vitro intestinal organoid culture images on days 3 and 7 after seeding. Our changes to the in-silico model maintain the potential to produce simulations that replicate the number of budding structures found on days 5 and 7 of in-vitro data. The present study aims to aid in quantifying key morphological structures and provide a method to compare both in-vitro and in-silico experiments. Our results could be extended later to 3D in-silico models.

Microsystem Advances through Integration with Artificial Intelligence

Microfluidics is a rapidly growing discipline that involves studying and manipulating fluids at reduced length scale and volume, typically on the scale of micro- or nanoliters. Under the reduced length scale and larger surface-to-volume ratio, advantages of low reagent consumption, faster reaction kinetics, and more compact systems are evident in microfluidics. However, miniaturization of microfluidic chips and systems introduces challenges of stricter tolerances in designing and controlling them for interdisciplinary applications. Recent advances in artificial intelligence (AI) have brought innovation to microfluidics from design, simulation, automation, and optimization to bioanalysis and data analytics. In microfluidics, the Navier-Stokes equations, which are partial differential equations describing viscous fluid motion that in complete form are known to not have a general analytical solution, can be simplified and have fair performance through numerical approximation due to low inertia and laminar flow. Approximation using neural networks trained by rules of physical knowledge introduces a new possibility to predict the physicochemical nature. The combination of microfluidics and automation can produce large amounts of data, where features and patterns that are difficult to discern by a human can be extracted by machine learning. Therefore, integration with AI introduces the potential to revolutionize the microfluidic workflow by enabling the precision control and automation of data analysis. Deployment of smart microfluidics may be tremendously beneficial in various applications in the future, including high-throughput drug discovery, rapid point-of-care-testing (POCT), and personalized medicine. In this review, we summarize key microfluidic advances integrated with AI and discuss the outlook and possibilities of combining AI and microfluidics.

Enhanced contrast synchrotron X-ray microtomography for describing skeleton-associated soft tissue defects in zebrafish mutants

Detailed histological analyses are desirable for zebrafish mutants that are models for human skeletal diseases, but traditional histological techniques are limited to two-dimensional thin sections with orientations highly dependent on careful sample preparation. On the other hand, techniques that provide three-dimensional (3D) datasets including µCT scanning are typically limited to visualizing the bony skeleton and lack histological resolution. We combined diffusible iodine-based contrast enhancement (DICE) and propagation phase-contrast synchrotron radiation micro-computed tomography (PPC-SRµCT) to image late larval and juvenile zebrafish, obtaining high-quality 3D virtual histology datasets of the mineralized skeleton and surrounding soft tissues. To demonstrate this technique, we used virtual histological thin sections and 3D segmentation to qualitatively and quantitatively compare wild-type zebrafish and nkx3.2 -/- mutants to characterize novel soft-tissue phenotypes in the muscles and tendons of the jaw and ligaments of the Weberian apparatus, as well as the sinus perilymphaticus associated with the inner ear. We could observe disrupted fiber organization and tendons of the adductor mandibulae and protractor hyoideus muscles associated with the jaws, and show that despite this, the overall muscle volumes appeared unaffected. Ligaments associated with the malformed Weberian ossicles were mostly absent in nkx3.2 -/- mutants, and the sinus perilymphaticus was severely constricted or absent as a result of the fused exoccipital and basioccipital elements. These soft-tissue phenotypes have implications for the physiology of nkx3.2 -/- zebrafish, and demonstrate the promise of DICE-PPC-SRµCT for histopathological investigations of bone-associated soft tissues in small-fish skeletal disease models and developmental studies more broadly.

Evaluation of artificial intelligence techniques in disease diagnosis and prediction

A broad range of medical diagnoses is based on analyzing disease images obtained through high-tech digital devices. The application of artificial intelligence (AI) in the assessment of medical images has led to accurate evaluations being performed automatically, which in turn has reduced the workload of physicians, decreased errors and times in diagnosis, and improved performance in the prediction and detection of various diseases. AI techniques based on medical image processing are an essential area of research that uses advanced computer algorithms for prediction, diagnosis, and treatment planning, leading to a remarkable impact on decision-making procedures. Machine Learning (ML) and Deep Learning (DL) as advanced AI techniques are two main subfields applied in the healthcare system to diagnose diseases, discover medication, and identify patient risk factors. The advancement of electronic medical records and big data technologies in recent years has accompanied the success of ML and DL algorithms. ML includes neural networks and fuzzy logic algorithms with various applications in automating forecasting and diagnosis processes. DL algorithm is an ML technique that does not rely on expert feature extraction, unlike classical neural network algorithms. DL algorithms with high-performance calculations give promising results in medical image analysis, such as fusion, segmentation, recording, and classification. Support Vector Machine (SVM) as an ML method and Convolutional Neural Network (CNN) as a DL method is usually the most widely used techniques for analyzing and diagnosing diseases. This review study aims to cover recent AI techniques in diagnosing and predicting numerous diseases such as cancers, heart, lung, skin, genetic, and neural disorders, which perform more precisely compared to specialists without human error. Also, AI's existing challenges and limitations in the medical area are discussed and highlighted.

Zebrafish as a Mainstream Model for In Vivo Systems Pharmacology and Toxicology

Pharmacology and toxicology are part of a much broader effort to understand the relationship between chemistry and biology. While biomedicine has necessarily focused on specific cases, typically of direct human relevance, there are real advantages in pursuing more systematic approaches to characterizing how health and disease are influenced by small molecules and other interventions. In this context, the zebrafish is now established as the representative screenable vertebrate and, through ongoing advances in the available scale of genome editing and automated phenotyping, is beginning to address systems-level solutions to some biomedical problems. The addition of broader efforts to integrate information content across preclinical model organisms and the incorporation of rigorous analytics, including closed-loop deep learning, will facilitate efforts to create systems pharmacology and toxicology with the ability to continuously optimize chemical biological interactions around societal needs. In this review, we outline progress toward this goal.

Obtaining 3D super-resolution images by utilizing rotationally symmetric structures and 2D-to-3D transformation

Super-resolution imaging techniques have provided unprecedentedly detailed information by surpassing the diffraction-limited resolution of light microscopy. However, in order to derive high quality spatial resolution, many of these techniques require high laser power, extended imaging time, dedicated sample preparation, or some combination of the three. These constraints are particularly evident when considering three-dimensional (3D) super-resolution imaging. As a result, high-speed capture of 3D super-resolution information of structures and dynamic processes within live cells remains both desirable and challenging. Recently, a highly effective approach to obtain 3D super-resolution information was developed that can be employed in commonly available laboratory microscopes. This development makes it both scientifically possible and financially feasible to obtain super-resolution 3D information under certain conditions. This is accomplished by converting 2D single-molecule localization data captured at high speed within subcellular structures and rotationally symmetric organelles. Here, a high-speed 2D single-molecule tracking and post-localization technique, known as single-point edge-excitation sub-diffraction (SPEED) microcopy, along with its 2D-to-3D transformation algorithm is detailed with special emphasis on the mathematical principles and Monte Carlo simulation validation of the technique.

Network Neuroscience Untethered: Brain-Wide Immediate Early Gene Expression for the Analysis of Functional Connectivity in Freely Behaving Animals

Studying how spatially discrete neuroanatomical regions across the brain interact is critical to advancing our understanding of the brain. Traditional neuroimaging techniques have led to many important discoveries about the nature of these interactions, termed functional connectivity. However, in animal models these traditional neuroimaging techniques have generally been limited to anesthetized or head-fixed setups or examination of small subsets of neuroanatomical regions. Using the brain-wide expression density of immediate early genes (IEG), we can assess brain-wide functional connectivity underlying a wide variety of behavioural tasks in freely behaving animal models. Here, we provide an overview of the necessary steps required to perform IEG-based analyses of functional connectivity. We also outline important considerations when designing such experiments and demonstrate the implications of these considerations using an IEG-based network dataset generated for the purpose of this review.

E-DU: Deep neural network for multimodal medical image segmentation based on semantic gap compensation

BACKGROUND

U-Net includes encoder, decoder and skip connection structures. It has become the benchmark network in medical image segmentation. However, the direct fusion of low-level and high-level convolution features with semantic gaps by traditional skip connections may lead to problems such as fuzzy generated feature maps and target region segmentation errors.

OBJECTIVE

We use spatial enhancement filtering technology to compensate for the semantic gap and propose an enhanced dense U-Net (E-DU), aiming to apply it to multimodal medical image segmentation to improve the segmentation performance and efficiency.

METHODS

Before combining encoder and decoder features, we replace the traditional skip connection with a multiscale denoise enhancement (MDE) module. The encoder features need to be deeply convolved by the spatial enhancement filter and then combined with the decoder features. We propose a simple and efficient deep full convolution network structure E-DU, which can not only fuse semantically various features but also denoise and enhance the feature map.

RESULTS

We performed experiments on medical image segmentation datasets with seven image modalities and combined MDE with various baseline networks to perform ablation studies. E-DU achieved the best segmentation results on evaluation indicators such as DSC on the U-Net family, with DSC values of 97.78, 97.64, 95.31, 94.42, 94.93, 98.85, and 98.38 (%), respectively. The addition of the MDE module to the attention mechanism network improves segmentation performance and efficiency, reflecting its generalization performance. In comparison to advanced methods, our method is also competitive.

CONCLUSION

Our proposed MDE module has a good segmentation effect and operating efficiency, and it can be easily extended to multiple modal medical segmentation datasets. Our idea and method can achieve clinical multimodal medical image segmentation and make full use of image information to provide clinical decision support. It has great application value and promotion prospects.

On the Variability in Cell and Nucleus Shapes

Cell morphology is an important regulator of cell function. Many abnormalities in cellular behavior can be discerned from changes in the shape of the cell and its organelles, typically the nucleus. Two major challenges for developing such phenotypic assays are reconstructing 3D surfaces of individual cells and nuclei from confocal images and developing characterizations of these surfaces for comparisons. We demonstrate two algorithms - 3D active contours and 3D condensed-attention UNet - to segment cells and nuclei from confocal images. The cell and nuclear surfaces are then converted into vectors using a reversible, spherical transform - i.e., shapes can be recovered from the vectors. Typical methods for characterizing shapes using size, shape, and image parameters such as area, volume, shape factor, solidity, and pixel intensities are not amenable to such reverse transformation. Our vector representation's principal component analysis shows that the significant modes of variability among cell and nucleus shapes are scaling and flattening. We benchmark these modes using a known mechanical model for nucleus morphology. Subsequent modes alter the eccentricity of the nucleus and translate and rotate it with respect to the cell. Our vector-space representation of cell and nucleus shape helps physically interpret the variability sources. It may further help to guide mechanical models and identify molecular mechanisms driving cell and nuclear shape changes.

Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging

Complex intracellular organizations are commonly represented by dividing the metabolic process of cells into different organelles. Therefore, identifying sub-cellular organelle architecture is significant for understanding intracellular structural properties, specific functions, and biological processes in cells. However, the discrimination of these structures in the natural organizational environment and their functional consequences are not clear. In this article, we propose a new pixel-level multimodal fusion (PLMF) deep network which can be used to predict the location of cellular organelle using label-free cell optical microscopy images followed by deep-learning-based automated image denoising. It provides valuable insights that can be of tremendous help in improving the specificity of label-free cell optical microscopy by using the Transformer-Unet network to predict the ground truth imaging which corresponds to different sub-cellular organelle architectures. The new prediction method proposed in this article combines the advantages of a transformer's global prediction and CNN's local detail analytic ability of background features for label-free cell optical microscopy images, so as to improve the prediction accuracy. Our experimental results showed that the PLMF network can achieve over 0.91 Pearson's correlation coefficient (PCC) correlation between estimated and true fractions on lung cancer cell-imaging datasets. In addition, we applied the PLMF network method on the cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new way for the time-resolved study of subcellular components in different cells, especially for cancer cells.

Multiple Parallel Fusion Network for Predicting Protein Subcellular Localization from Stimulated Raman Scattering (SRS) Microscopy Images in Living Cells

Stimulated Raman Scattering Microscopy (SRS) is a powerful tool for label-free detailed recognition and investigation of the cellular and subcellular structures of living cells. Determining subcellular protein localization from the cell level of SRS images is one of the basic goals of cell biology, which can not only provide useful clues for their functions and biological processes but also help to determine the priority and select the appropriate target for drug development. However, the bottleneck in predicting subcellular protein locations of SRS cell imaging lies in modeling complicated relationships concealed beneath the original cell imaging data owing to the spectral overlap information from different protein molecules. In this work, a multiple parallel fusion network, MPFnetwork, is proposed to study the subcellular locations from SRS images. This model used a multiple parallel fusion model to construct feature representations and combined multiple nonlinear decomposing algorithms as the automated subcellular detection method. Our experimental results showed that the MPFnetwork could achieve over 0.93 dice correlation between estimated and true fractions on SRS lung cancer cell datasets. In addition, we applied the MPFnetwork method to cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new method for the time-resolved study of subcellular components in different cells, especially cancer cells.