Learning cellular morphology with neural networks

Large-scale foundation models and generative AI for BigData neuroscience

Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.

NEURD: automated proofreading and feature extraction for connectomics

We are now in the era of millimeter-scale electron microscopy (EM) volumes collected at nanometer resolution (Shapson-Coe et al., 2021; Consortium et al., 2021). Dense reconstruction of cellular compartments in these EM volumes has been enabled by recent advances in Machine Learning (ML) (Lee et al., 2017; Wu et al., 2021; Lu et al., 2021; Macrina et al., 2021). Automated segmentation methods can now yield exceptionally accurate reconstructions of cells, but despite this accuracy, laborious post-hoc proofreading is still required to generate large connectomes free of merge and split errors. The elaborate 3-D meshes of neurons produced by these segmentations contain detailed morphological information, from the diameter, shape, and branching patterns of axons and dendrites, down to the fine-scale structure of dendritic spines. However, extracting information about these features can require substantial effort to piece together existing tools into custom workflows. Building on existing open-source software for mesh manipulation, here we present "NEURD", a software package that decomposes each meshed neuron into a compact and extensively-annotated graph representation. With these feature-rich graphs, we implement workflows to automate a variety of tasks that would otherwise require extensive manual effort, such as state of the art automated post-hoc proofreading of merge errors, cell classification, spine detection, axon-dendritic proximities, and computation of other features. These features enable many downstream analyses of neural morphology and connectivity, making these new massive and complex datasets more accessible to neuroscience researchers focused on a variety of scientific questions.

Morphological entropy encodes cellular migration strategies on multiple length scales

Cell migration is crucial for numerous physiological and pathological processes. A cell adapts its morphology, including the overall and nuclear morphology, in response to various cues in complex microenvironments, such as topotaxis and chemotaxis during migration. Thus, the dynamics of cellular morphology can encode migration strategies, from which diverse migration mechanisms can be inferred. However, deciphering the mechanisms behind cell migration encoded in morphology dynamics remains a challenging problem. Here, we present a powerful universal metric, the Cell Morphological Entropy (CME), developed by combining parametric morphological analysis with Shannon entropy. The utility of CME, which accurately quantifies the complex cellular morphology at multiple length scales through the deviation from a perfectly circular shape, is illustrated using a variety of normal and tumor cell lines in different in vitro microenvironments. Our results show how geometric constraints affect the MDA-MB-231 cell nucleus, the emerging interactions of MCF-10A cells migrating on collagen gel, and the critical transition from proliferation to invasion in tumor spheroids. The analysis demonstrates that the CME-based approach provides an effective and physically interpretable tool to measure morphology in real-time across multiple length scales. It provides deeper insight into cell migration and contributes to the understanding of different behavioral modes and collective cell motility in more complex microenvironments.

Current Progress and Challenges in Large-Scale 3D Mitochondria Instance Segmentation

In this paper, we present the results of the MitoEM challenge on mitochondria 3D instance segmentation from electron microscopy images, organized in conjunction with the IEEE-ISBI 2021 conference. Our benchmark dataset consists of two large-scale 3D volumes, one from human and one from rat cortex tissue, which are 1,986 times larger than previously used datasets. At the time of paper submission, 257 participants had registered for the challenge, 14 teams had submitted their results, and six teams participated in the challenge workshop. Here, we present eight top-performing approaches from the challenge participants, along with our own baseline strategies. Posterior to the challenge, annotation errors in the ground truth were corrected without altering the final ranking. Additionally, we present a retrospective evaluation of the scoring system which revealed that: 1) challenge metric was permissive with the false positive predictions; and 2) size-based grouping of instances did not correctly categorize mitochondria of interest. Thus, we propose a new scoring system that better reflects the correctness of the segmentation results. Although several of the top methods are compared favorably to our own baselines, substantial errors remain unsolved for mitochondria with challenging morphologies. Thus, the challenge remains open for submission and automatic evaluation, with all volumes available for download.

Multi-layered maps of neuropil with segmentation-guided contrastive learning

Maps of the nervous system that identify individual cells along with their type, subcellular components and connectivity have the potential to elucidate fundamental organizational principles of neural circuits. Nanometer-resolution imaging of brain tissue provides the necessary raw data, but inferring cellular and subcellular annotation layers is challenging. We present segmentation-guided contrastive learning of representations (SegCLR), a self-supervised machine learning technique that produces representations of cells directly from 3D imagery and segmentations. When applied to volumes of human and mouse cortex, SegCLR enables accurate classification of cellular subcompartments and achieves performance equivalent to a supervised approach while requiring 400-fold fewer labeled examples. SegCLR also enables inference of cell types from fragments as small as 10 μm, which enhances the utility of volumes in which many neurites are truncated at boundaries. Finally, SegCLR enables exploration of layer 5 pyramidal cell subtypes and automated large-scale analysis of synaptic partners in mouse visual cortex.

A robust approach to 3D neuron shape representation for quantification and classification

We consider the problem of finding an accurate representation of neuron shapes, extracting sub-cellular features, and classifying neurons based on neuron shapes. In neuroscience research, the skeleton representation is often used as a compact and abstract representation of neuron shapes. However, existing methods are limited to getting and analyzing "curve" skeletons which can only be applied for tubular shapes. This paper presents a 3D neuron morphology analysis method for more general and complex neuron shapes. First, we introduce the concept of skeleton mesh to represent general neuron shapes and propose a novel method for computing mesh representations from 3D surface point clouds. A skeleton graph is then obtained from skeleton mesh and is used to extract sub-cellular features. Finally, an unsupervised learning method is used to embed the skeleton graph for neuron classification. Extensive experiment results are provided and demonstrate the robustness of our method to analyze neuron morphology.

Virtual Fly Brain-An interactive atlas of the Drosophila nervous system

As a model organism, Drosophila is uniquely placed to contribute to our understanding of how brains control complex behavior. Not only does it have complex adaptive behaviors, but also a uniquely powerful genetic toolkit, increasingly complete dense connectomic maps of the central nervous system and a rapidly growing set of transcriptomic profiles of cell types. But this also poses a challenge: Given the massive amounts of available data, how are researchers to Find, Access, Integrate and Reuse (FAIR) relevant data in order to develop an integrated anatomical and molecular picture of circuits, inform hypothesis generation, and find reagents for experiments to test these hypotheses? The Virtual Fly Brain (virtualflybrain.org) web application & API provide a solution to this problem, using FAIR principles to integrate 3D images of neurons and brain regions, connectomics, transcriptomics and reagent expression data covering the whole CNS in both larva and adult. Users can search for neurons, neuroanatomy and reagents by name, location, or connectivity, via text search, clicking on 3D images, search-by-image, and queries by type (e.g., dopaminergic neuron) or properties (e.g., synaptic input in the antennal lobe). Returned results include cross-registered 3D images that can be explored in linked 2D and 3D browsers or downloaded under open licenses, and extensive descriptions of cell types and regions curated from the literature. These solutions are potentially extensible to cover similar atlasing and data integration challenges in vertebrates.

MorphoFeatures for unsupervised exploration of cell types, tissues, and organs in volume electron microscopy

Electron microscopy (EM) provides a uniquely detailed view of cellular morphology, including organelles and fine subcellular ultrastructure. While the acquisition and (semi-)automatic segmentation of multicellular EM volumes are now becoming routine, large-scale analysis remains severely limited by the lack of generally applicable pipelines for automatic extraction of comprehensive morphological descriptors. Here, we present a novel unsupervised method for learning cellular morphology features directly from 3D EM data: a neural network delivers a representation of cells by shape and ultrastructure. Applied to the full volume of an entire three-segmented worm of the annelid Platynereis dumerilii, it yields a visually consistent grouping of cells supported by specific gene expression profiles. Integration of features across spatial neighbours can retrieve tissues and organs, revealing, for example, a detailed organisation of the animal foregut. We envision that the unbiased nature of the proposed morphological descriptors will enable rapid exploration of very different biological questions in large EM volumes, greatly increasing the impact of these invaluable, but costly resources.

SyConn2: dense synaptic connectivity inference for volume electron microscopy

The ability to acquire ever larger datasets of brain tissue using volume electron microscopy leads to an increasing demand for the automated extraction of connectomic information. We introduce SyConn2, an open-source connectome analysis toolkit, which works with both on-site high-performance compute environments and rentable cloud computing clusters. SyConn2 was tested on connectomic datasets with more than 10 million synapses, provides a web-based visualization interface and makes these data amenable to complex anatomical and neuronal connectivity queries.

SyConn2 is a machine learning-based framework for inferring and analyzing the connectomes contained in a volume electron microscopy dataset of brain tissue, for example from the zebra finch.

A review of pathobiological mechanisms and potential application of medicinal plants for vascular aging: focus on endothelial cell senescence

Endothelial cell (EC) senescence plays a pivotal role in aging and is essential for the pathomechanism of aging-related diseases. Drugs targeting cellular senescence, such as senolytic or senomorphic drugs, may prevent aging and age-related diseases, but these bullets remain undeveloped to target EC senescence. Some medicinal plants may have an anti-senescence property but remain undiscovered. Deep learning has become an emerging approach for drug discovery by simply analyzing cellular morphology-based deep learning. This precious tool would be useful for screening the herb candidate in senescent EC rejuvenescence. Of note, several medicinal plants that can be found in Indonesia such as Curcuma longa L., Piper retrofractum, Guazuma ulmifolia Lam, Centella asiatica (L.) Urb., and Garcinia mangostana L. might potentially possess an anti-senescence effect. This review highlighted the importance of targeting EC senescence, the use of deep learning for medicinal plant screening, and some potential anti-senescence plants originating from Indonesia.

Ecomechanics and the Rules of Life: a Critical Conduit Between the Physical and Natural Sciences

Nature provides the parameters, or boundaries, within which organisms must cope in order to survive. Therefore, ecological conditions have an unequivocal influence on the ability of organisms to perform the necessary functions for survival. Biomechanics brings together physics and biology to understand how an organism will function under a suite of conditions. Despite a relatively rich recent history linking physiology and morphology with ecology, less attention has been paid to the linkage between biomechanics and ecology. This linkage, however, could provide key insights into patterns and processes of evolution. Ecomechanics, also known as ecological biomechanics or mechanical ecology, is not necessarily new, but has received far less attention than ecophysiology or ecomorphology. Here, we briefly review the history of ecomechanics, and then identify what we believe are grand challenges for the discipline and how they can inform some of the most pressing questions in science today, such as how organisms will cope with global change.

Actomyosin-Assisted Pulling of Lipid Nanotubes from Lipid Vesicles and Cells

Molecular motors are pivotal for intracellular transport as well as cell motility and have great potential to be put to use outside cells. Here, we exploit engineered motor proteins in combination with self-assembly of actin filaments to actively pull lipid nanotubes from giant unilamellar vesicles (GUVs). In particular, actin filaments are bound to the outer GUV membrane and the GUVs are seeded on a heavy meromyosin-coated substrate. Upon addition of ATP, hollow lipid nanotubes with a length of tens of micrometer are pulled from single GUVs due to the motor activity. We employ the same mechanism to pull lipid nanotubes from different types of cells. We find that the length and number of nanotubes critically depends on the cell type, whereby suspension cells form bigger networks than adherent cells. This suggests that molecular machines can be used to exert forces on living cells to probe membrane-to-cortex attachment.

Efficient and Accurate Synapse Detection With Selective Structured Illumination Microscopy on the Putative Regions of Interest of Ultrathin Serial Sections

Critical determinants of synaptic functions include subcellular locations, input sources, and specific molecular characteristics. However, there is not yet a reliable and efficient method that can detect synapses. Electron microscopy is a gold-standard method to detect synapses due to its exceedingly high spatial resolution. However, it requires laborious and time-consuming sample preparation and lengthy imaging time with limited labeling methods. Recent advances in various fluorescence microscopy methods have highlighted fluorescence microscopy as a substitute for electron microscopy in reliable synapse detection in a large volume of neural circuits. In particular, array tomography has been verified as a useful tool for neural circuit reconstruction. To further improve array tomography, we developed a novel imaging method, called “structured illumination microscopy on the putative region of interest on ultrathin sections”, which enables efficient and accurate detection of synapses-of-interest. Briefly, based on low-magnification conventional fluorescence microscopy images, synapse candidacy was determined. Subsequently, the coordinates of the regions with candidate synapses were imaged using super-resolution structured illumination microscopy. Using this system, synapses from the high-order thalamic nucleus, the posterior medial nucleus in the barrel cortex were rapidly and accurately imaged.

Learning and Segmenting Dense Voxel Embeddings for 3D Neuron Reconstruction

We show dense voxel embeddings learned via deep metric learning can be employed to produce a highly accurate segmentation of neurons from 3D electron microscopy images. A "metric graph" on a set of edges between voxels is constructed from the dense voxel embeddings generated by a convolutional network. Partitioning the metric graph with long-range edges as repulsive constraints yields an initial segmentation with high precision, with substantial accuracy gain for very thin objects. The convolutional embedding net is reused without any modification to agglomerate the systematic splits caused by complex "self-contact" motifs. Our proposed method achieves state-of-the-art accuracy on the challenging problem of 3D neuron reconstruction from the brain images acquired by serial section electron microscopy. Our alternative, object-centered representation could be more generally useful for other computational tasks in automated neural circuit reconstruction.

Prediction of RNA subcellular localization: Learning from heterogeneous data sources

RNA subcellular localization has recently emerged as a widespread phenomenon, which may apply to the majority of RNAs. The two main sources of data for characterization of RNA localization are sequence features and microscopy images, such as obtained from single-molecule fluorescent in situ hybridization-based techniques. Although such imaging data are ideal for characterization of RNA distribution, these techniques remain costly, time-consuming, and technically challenging. Given these limitations, imaging data exist only for a limited number of RNAs. We argue that the field of RNA localization would greatly benefit from complementary techniques able to characterize location of RNA. Here we discuss the importance of RNA localization and the current methodology in the field, followed by an introduction on prediction of location of molecules. We then suggest a machine learning approach based on the integration between imaging localization data and sequence-based data to assist in characterization of RNA localization on a transcriptome level.

Niwaki Instead of Random Forests: Targeted Serial Sectioning Scanning Electron Microscopy With Reimaging Capabilities for Exploring Central Nervous System Cell Biology and Pathology

Ultrastructural analysis of discrete neurobiological structures by volume scanning electron microscopy (SEM) often constitutes a "needle-in-the-haystack" problem and therefore relies on sophisticated search strategies. The appropriate SEM approach for a given relocation task not only depends on the desired final image quality but also on the complexity and required accuracy of the screening process. Block-face SEM techniques like Focused Ion Beam or serial block-face SEM are "one-shot" imaging runs by nature and, thus, require precise relocation prior to acquisition. In contrast, "multi-shot" approaches conserve the sectioned tissue through the collection of serial sections onto solid support and allow reimaging. These tissue libraries generated by Array Tomography or Automated Tape Collecting Ultramicrotomy can be screened at low resolution to target high resolution SEM. This is particularly useful if a structure of interest is rare or has been predetermined by correlated light microscopy, which can assign molecular, dynamic and functional information to an ultrastructure. As such approaches require bridging mm to nm scales, they rely on tissue trimming at different stages of sample processing. Relocation is facilitated by endogenous or exogenous landmarks that are visible by several imaging modalities, combined with appropriate registration strategies that allow overlaying images of various sources. Here, we discuss the opportunities of using multi-shot serial sectioning SEM approaches, as well as suitable trimming and registration techniques, to slim down the high-resolution imaging volume to the actual structure of interest and hence facilitate ambitious targeted volume SEM projects.

NuMorph: Tools for cortical cellular phenotyping in tissue-cleared whole-brain images

Tissue-clearing methods allow every cell in the mouse brain to be imaged without physical sectioning. However, the computational tools currently available for cell quantification in cleared tissue images have been limited to counting sparse cell populations in stereotypical mice. Here, we introduce NuMorph, a group of analysis tools to quantify all nuclei and nuclear markers within the mouse cortex after clearing and imaging by light-sheet microscopy. We apply NuMorph to investigate two distinct mouse models: a Topoisomerase 1 (Top1) model with severe neurodegenerative deficits and a Neurofibromin 1 (Nf1) model with a more subtle brain overgrowth phenotype. In each case, we identify differential effects of gene deletion on individual cell-type counts and distribution across cortical regions that manifest as alterations of gross brain morphology. These results underline the value of whole-brain imaging approaches, and the tools are widely applicable for studying brain structure phenotypes at cellular resolution.

Connecting chemistry and biology through molecular descriptors

Through the representation of small molecule structures as numerical descriptors and the exploitation of the similarity principle, chemoinformatics has made paramount contributions to drug discovery, from unveiling mechanisms of action and repurposing approved drugs to de novo crafting of molecules with desired properties and tailored targets. Yet, the inherent complexity of biological systems has fostered the implementation of large-scale experimental screenings seeking a deeper understanding of the targeted proteins, the disrupted biological processes and the systemic responses of cells to chemical perturbations. After this wealth of data, a new generation of data-driven descriptors has arisen providing a rich portrait of small molecule characteristics that goes beyond chemical properties. Here, we give an overview of biologically relevant descriptors, covering chemical compounds, proteins and other biological entities, such as diseases and cell lines, while aligning them to the major contributions in the field from disciplines, such as natural language processing or computer vision. We now envision a new scenario for chemical and biological entities where they both are translated into a common numerical format. In this computational framework, complex connections between entities can be unveiled by means of simple arithmetic operations, such as distance measures, additions, and subtractions.

Exploring the value of pleural fluid biomarkers for complementary pleural effusion disease examination

OBJECTIVE

Pleural fluid biomarkers are beneficial for the complementary diagnosis of pleural effusion etiologies. This study focuses on the multidimensional evaluation of deep learning to investigate the pleural effusion biomarkers value and the diagnostic utility of combining these markers, in distinguishing pleural effusion etiologies.

METHODS

Pleural effusion were divided into three groups according to the diagnosis and treatment guidelines: malignant pleural effusion (MPE), parapneumonic effusion (PPE), and congestive heart failure (CHF). First, the value of the biomarker was analyzed by a receiver operating characteristic (ROC) curve. Then by utilizing deep learning and entropy weight method (EWM), the clinical value of biomarkers was computed multidimensionally for complementary diagnosis of pleural effusion diseases.

RESULTS

There were significant differences in the six biomarkers, TP, ADA, CEA, CYFRA211, NSE, MNC% (p < 0.05) and no significant differences in three physical characteristics including color, transparency, specific gravity and six other biomarkers such as WBC, PNC%, MTC%, pH level, GLU, LDH (p > 0.05) among the three pleural effusion groups. The comprehensive test of pleural fluid biomarkers based on deep learning is of high accuracy. The clinical value of cytomorphology biomarkers WBC, MNC %, PNC %, MTC % was higher among pleural fluid biomarkers.

CONCLUSION

The clinical value of multi-dimensional analysis of biomarkers by deep learning and entropy weight method is different from the ROC curve analysis. It is suggested that during the clinical examination process, more attention should be paid to the cell morphology biomarkers, but the physical properties of the pleural fluid are less clinical significance.

Red blood cell phenotyping from 3D confocal images using artificial neural networks

The investigation of cell shapes mostly relies on the manual classification of 2D images, causing a subjective and time consuming evaluation based on a portion of the cell surface. We present a dual-stage neural network architecture for analyzing fine shape details from confocal microscopy recordings in 3D. The system, tested on red blood cells, uses training data from both healthy donors and patients with a congenital blood disease, namely hereditary spherocytosis. Characteristic shape features are revealed from the spherical harmonics spectrum of each cell and are automatically processed to create a reproducible and unbiased shape recognition and classification. The results show the relation between the particular genetic mutation causing the disease and the shape profile. With the obtained 3D phenotypes, we suggest our method for diagnostics and theragnostics of blood diseases. Besides the application employed in this study, our algorithms can be easily adapted for the 3D shape phenotyping of other cell types and extend their use to other applications, such as industrial automated 3D quality control.

Positive effect of Astragaloside IV on neurite outgrowth via talin-dependent integrin signaling and microfilament force

Integrin plays a prominent role in neurite outgrowth by transmitting both mechanical and chemical signals. Integrin expression is closely associated with Astragaloside IV (AS-IV), the main component extracted from Astragali radix, which has a positive effect on neural-protection. However, the relationship between AS-IV and neurite outgrowth has not been studied exhaustively to date. The present study investigated the underlying mechanism of AS-IV on neurite outgrowth. Longer neurites have been observed in SH-SY5Y cells or cortical neurons after AS-IV treatment. Furthermore, AS-IV not only increased the expression of integrin β but also activated it. The AS-IV-induced increased integrin activity was attributed to the integrin-activating protein talin. Application of the actin force probe showed that AS-IV led to an increase in intracellular microfilament force during neurite growth. Furthermore, in response to AS-IV, the microfilament force was regulated by talin and integrin activity during neurite growth. These results suggest that AS-IV has the ability to increase intracellular structural force and facilitate neurite elongation by integrin signaling, which highlights its therapeutic potential for neurite outgrowth.

MitoEM Dataset: Large-scale 3D Mitochondria Instance Segmentation from EM Images

Electron microscopy (EM) allows the identification of intracellular organelles such as mitochondria, providing insights for clinical and scientific studies. However, public mitochondria segmentation datasets only contain hundreds of instances with simple shapes. It is unclear if existing methods achieving human-level accuracy on these small datasets are robust in practice. To this end, we introduce the MitoEM dataset, a 3D mitochondria instance segmentation dataset with two (30μm)3 volumes from human and rat cortices respectively, 3, 600× larger than previous benchmarks. With around 40K instances, we find a great diversity of mitochondria in terms of shape and density. For evaluation, we tailor the implementation of the average precision (AP) metric for 3D data with a 45× speedup. On MitoEM, we find existing instance segmentation methods often fail to correctly segment mitochondria with complex shapes or close contacts with other instances. Thus, our MitoEM dataset poses new challenges to the field. We release our code and data: https://donglaiw.github.io/page/mitoEM/index.html.