Automatic Morphological Reconstruction of Neurons from Multiphoton and Confocal Microscopy Images Using 3D Tubular Models

Deep learning in mesoscale brain image analysis: A review

Mesoscale microscopy images of the brain contain a wealth of information which can help us understand the working mechanisms of the brain. However, it is a challenging task to process and analyze these data because of the large size of the images, their high noise levels, the complex morphology of the brain from the cellular to the regional and anatomical levels, the inhomogeneous distribution of fluorescent labels in the cells and tissues, and imaging artifacts. Due to their impressive ability to extract relevant information from images, deep learning algorithms are widely applied to microscopy images of the brain to address these challenges and they perform superiorly in a wide range of microscopy image processing and analysis tasks. This article reviews the applications of deep learning algorithms in brain mesoscale microscopy image processing and analysis, including image synthesis, image segmentation, object detection, and neuron reconstruction and analysis. We also discuss the difficulties of each task and possible directions for further research.

SNAP: a structure-based neuron morphology reconstruction automatic pruning pipeline

Background

Neuron morphology analysis is an essential component of neuron cell-type definition. Morphology reconstruction represents a bottleneck in high-throughput morphology analysis workflow, and erroneous extra reconstruction owing to noise and entanglements in dense neuron regions restricts the usability of automated reconstruction results. We propose SNAP, a structure-based neuron morphology reconstruction pruning pipeline, to improve the usability of results by reducing erroneous extra reconstruction and splitting entangled neurons.

Methods

For the four different types of erroneous extra segments in reconstruction (caused by noise in the background, entanglement with dendrites of close-by neurons, entanglement with axons of other neurons, and entanglement within the same neuron), SNAP incorporates specific statistical structure information into rules for erroneous extra segment detection and achieves pruning and multiple dendrite splitting.

Results

Experimental results show that this pipeline accomplishes pruning with satisfactory precision and recall. It also demonstrates good multiple neuron-splitting performance. As an effective tool for post-processing reconstruction, SNAP can facilitate neuron morphology analysis.

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. We observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

Lung nodule pre-diagnosis and insertion path planning for chest CT images

Medical image processing has proven to be effective and feasible for assisting oncologists in diagnosing lung, thyroid, and other cancers, especially at early stage. However, there is no reliable method for the recognition, screening, classification, and detection of nodules, and even deep learning-based methods have limitations. In this study, we mainly explored the automatic pre-diagnosis of lung nodules with the aim of accurately identifying nodules in chest CT images, regardless of the benign and malignant nodules, and the insertion path planning of suspected malignant nodules, used for further diagnosis by robotic-based biopsy puncture. The overall process included lung parenchyma segmentation, classification and pre-diagnosis, 3-D reconstruction and path planning, and experimental verification. First, accurate lung parenchyma segmentation in chest CT images was achieved using digital image processing technologies, such as adaptive gray threshold, connected area labeling, and mathematical morphological boundary repair. Multi-feature weight assignment was then adopted to establish a multi-level classification criterion to complete the classification and pre-diagnosis of pulmonary nodules. Next, 3-D reconstruction of lung regions was performed using voxelization, and on its basis, a feasible local optimal insertion path with an insertion point could be found by avoiding sternums and/or key tissues in terms of the needle-inserting path. Finally, CT images of 900 patients from Lung Image Database Consortium and Image Database Resource Initiative were chosen to verify the validity of pulmonary nodule diagnosis. Our previously designed surgical robotic system and a custom thoracic model were used to validate the effectiveness of the insertion path. This work can not only assist doctors in completing the pre-diagnosis of pulmonary nodules but also provide a reference for clinical biopsy puncture of suspected malignant nodules considered by doctors.

Neuron tracing from light microscopy images: automation, deep learning and bench testing

MOTIVATION

Large-scale neuronal morphologies are essential to neuronal typing, connectivity characterization and brain modeling. It is widely accepted that automation is critical to the production of neuronal morphology. Despite previous survey papers about neuron tracing from light microscopy data in the last decade, thanks to the rapid development of the field, there is a need to update recent progress in a review focusing on new methods and remarkable applications.

RESULTS

This review outlines neuron tracing in various scenarios with the goal to help the community understand and navigate tools and resources. We describe the status, examples and accessibility of automatic neuron tracing. We survey recent advances of the increasingly popular deep-learning enhanced methods. We highlight the semi-automatic methods for single neuron tracing of mammalian whole brains as well as the resulting datasets, each containing thousands of full neuron morphologies. Finally, we exemplify the commonly used datasets and metrics for neuron tracing bench testing.

Structure-Guided Segmentation for 3D Neuron Reconstruction

Digital reconstruction of neuronal morphologies in 3D microscopy images is critical in the field of neuroscience. However, most existing automatic tracing algorithms cannot obtain accurate neuron reconstruction when processing 3D neuron images contaminated by strong background noises or containing weak filament signals. In this paper, we present a 3D neuron segmentation network named Structure-Guided Segmentation Network (SGSNet) to enhance weak neuronal structures and remove background noises. The network contains a shared encoding path but utilizes two decoding paths called Main Segmentation Branch (MSB) and Structure-Detection Branch (SDB), respectively. MSB is trained on binary labels to acquire the 3D neuron image segmentation maps. However, the segmentation results in challenging datasets often contain structural errors, such as discontinued segments of the weak-signal neuronal structures and missing filaments due to low signal-to-noise ratio (SNR). Therefore, SDB is presented to detect the neuronal structures by regressing neuron distance transform maps. Furthermore, a Structure Attention Module (SAM) is designed to integrate the multi-scale feature maps of the two decoding paths, and provide contextual guidance of structural features from SDB to MSB to improve the final segmentation performance. In the experiments, we evaluate our model in two challenging 3D neuron image datasets, the BigNeuron dataset and the Extended Whole Mouse Brain Sub-image (EWMBS) dataset. When using different tracing methods on the segmented images produced by our method rather than other state-of-the-art segmentation methods, the distance scores gain 42.48% and 35.83% improvement in the BigNeuron dataset and 37.75% and 23.13% in the EWMBS dataset.

NeuroRetriever: Automatic Neuron Segmentation for Connectome Assembly

Segmenting individual neurons from a large number of noisy raw images is the first step in building a comprehensive map of neuron-to-neuron connections for predicting information flow in the brain. Thousands of fluorescence-labeled brain neurons have been imaged. However, mapping a complete connectome remains challenging because imaged neurons are often entangled and manual segmentation of a large population of single neurons is laborious and prone to bias. In this study, we report an automatic algorithm, NeuroRetriever, for unbiased large-scale segmentation of confocal fluorescence images of single neurons in the adult Drosophila brain. NeuroRetriever uses a high-dynamic-range thresholding method to segment three-dimensional morphology of single neurons based on branch-specific structural features. Applying NeuroRetriever to automatically segment single neurons in 22,037 raw brain images, we successfully retrieved 28,125 individual neurons validated by human segmentation. Thus, automated NeuroRetriever will greatly accelerate 3D reconstruction of the single neurons for constructing the complete connectomes.

Neuron Image Segmentation via Learning Deep Features and Enhancing Weak Neuronal Structures

Neuron morphology reconstruction (tracing) in 3D volumetric images is critical for neuronal research. However, most existing neuron tracing methods are not applicable in challenging datasets where the neuron images are contaminated by noises or containing weak filament signals. In this paper, we present a two-stage 3D neuron segmentation approach via learning deep features and enhancing weak neuronal structures, to reduce the impact of image noise in the data and enhance the weak-signal neuronal structures. In the first stage, we train a voxel-wise multi-level fully convolutional network (FCN), which specializes in learning deep features, to obtain the 3D neuron image segmentation maps in an end-to-end manner. In the second stage, a ray-shooting model is employed to detect the discontinued segments in segmentation results of the first-stage, and the local neuron diameter of the broken point is estimated and direction of the filamentary fragment is detected by rayburst sampling algorithm. Then, a Hessian-repair model is built to repair the broken structures, by enhancing weak neuronal structures in a fibrous structure determined by the estimated local neuron diameter and the filamentary fragment direction. Experimental results demonstrate that our proposed segmentation approach achieves better segmentation performance than other state-of-the-art methods for 3D neuron segmentation. Compared with the neuron reconstruction results on the segmented images produced by other segmentation methods, the proposed approach gains 47.83% and 34.83% improvement in the average distance scores. The average Precision and Recall rates of the branch point detection with our proposed method are 38.74% and 22.53% higher than the detection results without segmentation.

3D Neuron Microscopy Image Segmentation via the Ray-Shooting Model and a DC-BLSTM Network

The morphology reconstruction (tracing) of neurons in 3D microscopy images is important to neuroscience research. However, this task remains very challenging because of the low signal-to-noise ratio (SNR) and the discontinued segments of neurite patterns in the images. In this paper, we present a neuronal structure segmentation method based on the ray-shooting model and the Long Short-Term Memory (LSTM)-based network to enhance the weak-signal neuronal structures and remove background noise in 3D neuron microscopy images. Specifically, the ray-shooting model is used to extract the intensity distribution features within a local region of the image. And we design a neural network based on the dual channel bidirectional LSTM (DC-BLSTM) to detect the foreground voxels according to the voxel-intensity features and boundary-response features extracted by multiple ray-shooting models that are generated in the whole image. This way, we transform the 3D image segmentation task into multiple 1D ray/sequence segmentation tasks, which makes it much easier to label the training samples than many existing Convolutional Neural Network (CNN) based 3D neuron image segmentation methods. In the experiments, we evaluate the performance of our method on the challenging 3D neuron images from two datasets, the BigNeuron dataset and the Whole Mouse Brain Sub-image (WMBS) dataset. Compared with the neuron tracing results on the segmented images produced by other state-of-the-art neuron segmentation methods, our method improves the distance scores by about 32% and 27% in the BigNeuron dataset, and about 38% and 27% in the WMBS dataset.

Neuronal Population Reconstruction From Ultra-Scale Optical Microscopy Images via Progressive Learning

Reconstruction of neuronal populations from ultra-scale optical microscopy (OM) images is essential to investigate neuronal circuits and brain mechanisms. The noises, low contrast, huge memory requirement, and high computational cost pose significant challenges in the neuronal population reconstruction. Recently, many studies have been conducted to extract neuron signals using deep neural networks (DNNs). However, training such DNNs usually relies on a huge amount of voxel-wise annotations in OM images, which are expensive in terms of both finance and labor. In this paper, we propose a novel framework for dense neuronal population reconstruction from ultra-scale images. To solve the problem of high cost in obtaining manual annotations for training DNNs, we propose a progressive learning scheme for neuronal population reconstruction (PLNPR) which does not require any manual annotations. Our PLNPR scheme consists of a traditional neuron tracing module and a deep segmentation network that mutually complement and progressively promote each other. To reconstruct dense neuronal populations from a terabyte-sized ultra-scale image, we introduce an automatic framework which adaptively traces neurons block by block and fuses fragmented neurites in overlapped regions continuously and smoothly. We build a dataset "VISoR-40" which consists of 40 large-scale OM image blocks from cortical regions of a mouse. Extensive experimental results on our VISoR-40 dataset and the public BigNeuron dataset demonstrate the effectiveness and superiority of our method on neuronal population reconstruction and single neuron reconstruction. Furthermore, we successfully apply our method to reconstruct dense neuronal populations from an ultra-scale mouse brain slice. The proposed adaptive block propagation and fusion strategies greatly improve the completeness of neurites in dense neuronal population reconstruction.

Retrieving similar substructures on 3D neuron reconstructions

Since manual tracing is time consuming and the performance of automatic tracing is unstable, it is still a challenging task to generate accurate neuron reconstruction efficiently and effectively. One strategy is generating a reconstruction automatically and then amending its inaccurate parts manually. Aiming at finding inaccurate substructures efficiently, we propose a pipeline to retrieve similar substructures on one or more neuron reconstructions, which are very similar to a marked problematic substructure. The pipeline consists of four steps: getting a marked substructure, constructing a query substructure, generating candidate substructures and retrieving most similar substructures. The retrieval procedure was tested on 163 gold standard reconstructions provided by the BigNeuron project and a reconstruction of a mouse’s large neuron. Experimental results showed that the implementation of the proposed methods is very efficient and all retrieved substructures are very similar to the marked one in numbers of nodes and branches, and degree of curvature.

Automated Neuron Reconstruction from 3D Fluorescence Microscopy Images Using Sequential Monte Carlo Estimation

Microscopic images of neuronal cells provide essential structural information about the key constituents of the brain and form the basis of many neuroscientific studies. Computational analyses of the morphological properties of the captured neurons require first converting the structural information into digital tree-like reconstructions. Many dedicated computational methods and corresponding software tools have been and are continuously being developed with the aim to automate this step while achieving human-comparable reconstruction accuracy. This pursuit is hampered by the immense diversity and intricacy of neuronal morphologies as well as the often low quality and ambiguity of the images. Here we present a novel method we developed in an effort to improve the robustness of digital reconstruction against these complicating factors. The method is based on probabilistic filtering by sequential Monte Carlo estimation and uses prediction and update models designed specifically for tracing neuronal branches in microscopic image stacks. Moreover, it uses multiple probabilistic traces to arrive at a more robust, ensemble reconstruction. The proposed method was evaluated on fluorescence microscopy image stacks of single neurons and dense neuronal networks with expert manual annotations serving as the gold standard, as well as on synthetic images with known ground truth. The results indicate that our method performs well under varying experimental conditions and compares favorably to state-of-the-art alternative methods.

Multiphoton intravital microscopy in small animals: motion artefact challenges and technical solutions

Since its invention 29 years ago, two‐photon laser‐scanning microscopy has evolved from a promising imaging technique, to an established widely available imaging modality used throughout the biomedical research community. The establishment of two‐photon microscopy as the preferred method for imaging fluorescently labelled cells and structures in living animals can be attributed to the biophysical mechanism by which the generation of fluorescence is accomplished. The use of powerful lasers capable of delivering infrared light pulses within femtosecond intervals, facilitates the nonlinear excitation of fluorescent molecules only at the focal plane and determines by objective lens position. This offers numerous benefits for studies of biological samples at high spatial and temporal resolutions with limited photo‐damage and superior tissue penetration. Indeed, these attributes have established two‐photon microscopy as the ideal method for live‐animal imaging in several areas of biology and have led to a whole new field of study dedicated to imaging biological phenomena in intact tissues and living organisms. However, despite its appealing features, two‐photon intravital microscopy is inherently limited by tissue motion from heartbeat, respiratory cycles, peristalsis, muscle/vascular tone and physiological functions that change tissue geometry. Because these movements impede temporal and spatial resolution, they must be properly addressed to harness the full potential of two‐photon intravital microscopy and enable accurate data analysis and interpretation. In addition, the sources and features of these motion artefacts are varied, sometimes unpredictable and unique to specific organs and multiple complex strategies have previously been devised to address them. This review will discuss these motion artefacts requirement and technical solutions for their correction and after intravital two‐photon microscopy.

Deep Learning-Based Single-Cell Optical Image Studies

Optical imaging technology that has the advantages of high sensitivity and cost-effectiveness greatly promotes the progress of nondestructive single-cell studies. Complex cellular image analysis tasks such as three-dimensional reconstruction call for machine-learning technology in cell optical image research. With the rapid developments of high-throughput imaging flow cytometry, big data cell optical images are always obtained that may require machine learning for data analysis. In recent years, deep learning has been prevalent in the field of machine learning for large-scale image processing and analysis, which brings a new dawn for single-cell optical image studies with an explosive growth of data availability. Popular deep learning techniques offer new ideas for multimodal and multitask single-cell optical image research. This article provides an overview of the basic knowledge of deep learning and its applications in single-cell optical image studies. We explore the feasibility of applying deep learning techniques to single-cell optical image analysis, where popular techniques such as transfer learning, multimodal learning, multitask learning, and end-to-end learning have been reviewed. Image preprocessing and deep learning model training methods are then summarized. Applications based on deep learning techniques in the field of single-cell optical image studies are reviewed, which include image segmentation, super-resolution image reconstruction, cell tracking, cell counting, cross-modal image reconstruction, and design and control of cell imaging systems. In addition, deep learning in popular single-cell optical imaging techniques such as label-free cell optical imaging, high-content screening, and high-throughput optical imaging cytometry are also mentioned. Finally, the perspectives of deep learning technology for single-cell optical image analysis are discussed. © 2020 International Society for Advancement of Cytometry.

Gotta Trace 'em All: A Mini-Review on Tools and Procedures for Segmenting Single Neurons Toward Deciphering the Structural Connectome

Decoding the morphology and physical connections of all the neurons populating a brain is necessary for predicting and studying the relationships between its form and function, as well as for documenting structural abnormalities in neuropathies. Digitizing a complete and high-fidelity map of the mammalian brain at the micro-scale will allow neuroscientists to understand disease, consciousness, and ultimately what it is that makes us humans. The critical obstacle for reaching this goal is the lack of robust and accurate tools able to deal with 3D datasets representing dense-packed cells in their native arrangement within the brain. This obliges neuroscientist to manually identify the neurons populating an acquired digital image stack, a notably time-consuming procedure prone to human bias. Here we review the automatic and semi-automatic algorithms and software for neuron segmentation available in the literature, as well as the metrics purposely designed for their validation, highlighting their strengths and limitations. In this direction, we also briefly introduce the recent advances in tissue clarification that enable significant improvements in both optical access of neural tissue and image stack quality, and which could enable more efficient segmentation approaches. Finally, we discuss new methods and tools for processing tissues and acquiring images at sub-cellular scales, which will require new robust algorithms for identifying neurons and their sub-structures (e.g., spines, thin neurites). This will lead to a more detailed structural map of the brain, taking twenty-first century cellular neuroscience to the next level, i.e., the Structural Connectome.

FMST: an Automatic Neuron Tracing Method Based on Fast Marching and Minimum Spanning Tree

Neuron reconstruction is an important technique in computational neuroscience. Although there are many reconstruction algorithms, few can generate robust results. In this paper, we propose a reconstruction algorithm called fast marching spanning tree (FMST). FMST is based on a minimum spanning tree method (MST) and improve its performance in two aspects: faster implementation and no loss of small branches. The contributions of the proposed method are as follows. Firstly, the Euclidean distance weight of edges in MST is improved to be a more reasonable value, which is related to the probability of the existence of an edge. Secondly, a strategy of pruning nodes is presented, which is based on the radius of a node's inscribed ball. Thirdly, separate branches of broken neuron reconstructions can be merged into a single tree. FMST and many other state of the art reconstruction methods were implemented on two datasets: 120 Drosophila neurons and 163 neurons with gold standard reconstructions. Qualitative and quantitative analysis on experimental results demonstrates that the performance of FMST is good compared with many existing methods. Especially, on the 91 fruitfly neurons with gold standard and evaluated by five metrics, FMST is one of two methods with best performance among all 27 state of the art reconstruction methods. FMST is a good and practicable neuron reconstruction algorithm, and can be implemented in Vaa3D platform as a neuron tracing plugin.

Automated 3D Soma Segmentation with Morphological Surface Evolution for Neuron Reconstruction

The automatic neuron reconstruction is important since it accelerates the collection of 3D neuron models for the neuronal morphological studies. The majority of the previous neuron reconstruction methods only focused on tracing neuron fibres without considering the somatic surface. Thus, topological errors often present around the soma area in the results obtained by these tracing methods. Segmentation of the soma structures can be embedded in the existing neuron tracing methods to reduce such topological errors. In this paper, we present a novel method to segment the soma structures with complex geometry. It can be applied along with the existing methods in a fully automated pipeline. An approximate bounding block is firstly estimated based on a geodesic distance transform. Then the soma segmentation is obtained by evolving the surface with a set of morphological operators inside the initial bounding region. By evaluating the methods against the challenging images released by the BigNeuron project, we showed that the proposed method can outperform the existing soma segmentation methods regarding the accuracy. We also showed that the soma segmentation can be used for enhancing the results of existing neuron tracing methods.

Identifying Weak Signals in Inhomogeneous Neuronal Images for Large-Scale Tracing of Sparsely Distributed Neurites

Tracing neurites constitutes the core of neuronal morphology reconstruction, a key step toward neuronal circuit mapping. Modern optical-imaging techniques allow observation of nearly complete mouse neuron morphologies across brain regions or even the whole brain. However, high-level automation reconstruction of neurons, i.e., the reconstruction with a few of manual edits requires discrimination of weak foreground points from the inhomogeneous background. We constructed an identification model, where empirical observations made from neuronal images were summarized into rules for designing feature vectors that to classify foreground and background, and a support vector machine (SVM) was used to learn these feature vectors. We embedded this constructed SVM classifier into a previously developed tool, SparseTracer, to obtain SparseTracer-Learned Feature Vector (ST-LFV). ST-LFV can trace sparsely distributed neurites with weak signals (contrast-to-noise ratio < 1.5) against an inhomogeneous background in datasets imaged by widely used light-microscopy techniques like confocal microscopy and two-photon microscopy. Moreover, 12 sub-blocks were extracted from different brain regions. The average recall and precision rates were 99% and 97%, respectively. These results indicated that ST-LFV is well suited for weak signal identification with varying image characteristics. We also applied ST-LFV to trace long-range neurites from images where neurites are sparsely distributed but their image intensities are weak in some cases. When tracing this long-range neurites, manual edit was required once to obtain results equivalent to the ground truth, compared with 20 times of manual edits required by SparseTracer. This improvement in the level of automatic reconstruction indicates that ST-LFV has the potential to rapidly reconstruct sparsely distributed neurons at the large scale.

PAT-Probabilistic Axon Tracking for Densely Labeled Neurons in Large 3-D Micrographs

A major goal of contemporary neuroscience research is to map the structural connectivity of mammalian brain using microscopy imaging data. In this context, the reconstruction of densely labeled axons from two-photon microscopy images is a challenging and important task. The visually overlapping, crossing, and often strongly distorted images of the axons allow many ambiguous interpretations to be made. We address the problem of tracking axons in densely labeled samples of neurons in large image data sets acquired from marmoset brains. Our high-resolution images were acquired using two-photon microscopy and they provided whole brain coverage, occupying terabytes of memory. Both the image distortions and the large data set size frequently make it impractical to apply present-day neuron tracing algorithms to such data due to the optimization of such algorithms to the precise tracing of either single or sparse sets of neurons. Thus, new tracking techniques are needed. We propose a probabilistic axon tracking algorithm (PAT). PAT tackles the tracking of axons in two steps: locally (L-PAT) and globally (G-PAT). L-PAT is a probabilistic tracking algorithm that can tackle distorted, cluttered images of densely labeled axons. L-PAT divides a large micrograph into smaller image stacks. It then processes each image stack independently before mapping the axons in each image to a sparse model of axon trajectories. G-PAT merges the sparse L-PAT models into a single global model of axon trajectories by minimizing a global objective function using a probabilistic optimization method. We demonstrate the superior performance of PAT over standard approaches on synthetic data. Furthermore, we successfully apply PAT to densely labeled axons in large images acquired from marmoset brains.

Cocaine-Induced Preference Conditioning: a Machine Vision Perspective

Existing work on drug-induced synaptic changes has shown that the expression of perineuronal nets (PNNs) at the cerebellar cortex can be regulated by cocaine-related memory. However, these studies on animals have mostly relied on limited manually-driven procedures, and lack some more rigorous statistical approaches and more automated techniques. In this work, established methods from computer vision and machine learning are considered to build stronger evidence of those previous findings. To that end, an image descriptor is designed to characterize PNNs images; unsupervised learning (clustering) is used to automatically find distinctive patterns of PNNs; and supervised learning (classification) is adopted for predicting the experiment group of the mice from their PNN images. Experts in neurobiology, who were not aware of the underlying computational procedures, were asked to describe the patterns emerging from the automatically found clusters, and their descriptions were found to align surprisingly well with the two types of PNN images revealed from previous studies, namely strong and weak PNNs. Furthermore, when the set of PNN images corresponding to every mice in the saline (control) group and the conditioned (experimental) group were characterized using a bag-of-words representation, and subject to supervised learning (saline vs conditioned mice), the high classification results suggest the ability of the proposed representation and procedures in recognizing these groups. Therefore, despite the limited size of the dataset (1,032 PNN images of 6 saline and 6 conditioned mice), the results support existing evidence on the drug-related brain plasticity, while providing higher objectivity.

3-D Human Sperm Flagellum Tracing in Low SNR Fluorescence Images

Tracing tubular structures from biomedical images is important for a wide range of applications. Particularly, the spermatozoon is an essential cell whose flagella have a tubular form. Its main function is to fertilize the egg, and the flagellum is fundamental to achieve this task which depends importantly on the dynamics of intracellular calcium ([Ca²⁺]_i). Measuring [Ca²⁺]_i along the flagellum in 3-D is not a simple matter since it requires: 1) sophisticated fluorescence imaging techniques dealing with low intensity and signal to noise ratio (SNR) and 2) tracing the flagellum's centerline. Most of the algorithms proposed to trace tubular structures have been developed for multi-branch structures not being adequate for single tubular structures with low SNR. Taking into account the prior knowledge that the flagellum is constituted by a single tubular structure, we propose an automatic method to trace and track multiple single tubular structures from 3-D images. First, an algorithm based on one-class classification allows enhancement of the flagellum. This enhanced 3-D image permits guiding an iterative centerline algorithm toward the flagellum's centerline. Each sperm is assigned an ID to keep track of it in 3-D . Our algorithm was quantitatively evaluated using a ground truth 564 semi-manual traces (six 3-D image stacks) comparing them to those obtained from state-of-the-art tubular structure centerline extraction algorithms. The qualitative and quantitative results show that our algorithm is extracting similar traces as compared with ground truth, and it is more robust and accurate to trace the flagellum's centerline than multi-branch algorithms.

Visualization of Neuronal Structures in Wide-Field Microscopy Brain Images

Wide-field microscopes are commonly used in neurobiology for experimental studies of brain samples. Available visualization tools are limited to electron, two-photon, and confocal microscopy datasets, and current volume rendering techniques do not yield effective results when used with wide-field data. We present a workflow for the visualization of neuronal structures in wide-field microscopy images of brain samples. We introduce a novel gradient-based distance transform that overcomes the out-of-focus blur caused by the inherent design of wide-field microscopes. This is followed by the extraction of the 3D structure of neurites using a multi-scale curvilinear filter and cell-bodies using a Hessian-based enhancement filter. The response from these filters is then applied as an opacity map to the raw data. Based on the visualization challenges faced by domain experts, our workflow provides multiple rendering modes to enable qualitative analysis of neuronal structures, which includes separation of cell-bodies from neurites and an intensity-based classification of the structures. Additionally, we evaluate our visualization results against both a standard image processing deconvolution technique and a confocal microscopy image of the same specimen. We show that our method is significantly faster and requires less computational resources, while producing high quality visualizations. We deploy our workflow in an immersive gigapixel facility as a paradigm for the processing and visualization of large, high-resolution, wide-field microscopy brain datasets.

Automated sorting of neuronal trees in fluorescent images of neuronal networks using NeuroTreeTracer

Fluorescence confocal microscopy has become increasingly more important in neuroscience due to its applications in image-based screening and profiling of neurons. Multispectral confocal imaging is useful to simultaneously probe for distribution of multiple analytes over networks of neurons. However, current automated image analysis algorithms are not designed to extract single-neuron arbors in images where neurons are not separated, hampering the ability map fluorescence signals at the single cell level. To overcome this limitation, we introduce NeuroTreeTracer - a novel image processing framework aimed at automatically extracting and sorting single-neuron traces in fluorescent images of multicellular neuronal networks. This method applies directional multiscale filters for automated segmentation of neurons and soma detection, and includes a novel tracing routine that sorts neuronal trees in the image by resolving network connectivity even when neurites appear to intersect. By extracting each neuronal tree, NeuroTreetracer enables to automatically quantify the spatial distribution of analytes of interest in the subcellular compartments of individual neurons. This software is released open-source and freely available with the goal to facilitate applications in neuron screening and profiling.

SparseTracer: the Reconstruction of Discontinuous Neuronal Morphology in Noisy Images

Digital reconstruction of a single neuron occupies an important position in computational neuroscience. Although many novel methods have been proposed, recent advances in molecular labeling and imaging systems allow for the production of large and complicated neuronal datasets, which pose many challenges for neuron reconstruction, especially when discontinuous neuronal morphology appears in a strong noise environment. Here, we develop a new pipeline to address this challenge. Our pipeline is based on two methods, one is the region-to-region connection (RRC) method for detecting the initial part of a neurite, which can effectively gather local cues, i.e., avoid the whole image analysis, and thus boosts the efficacy of computation; the other is constrained principal curves method for completing the neurite reconstruction, which uses the past reconstruction information of a neurite for current reconstruction and thus can be suitable for tracing discontinuous neurites. We investigate the reconstruction performances of our pipeline and some of the best state-of-the-art algorithms on the experimental datasets, indicating the superiority of our method in reconstructing sparsely distributed neurons with discontinuous neuronal morphologies in noisy environment. We show the strong ability of our pipeline in dealing with the large-scale image dataset. We validate the effectiveness in dealing with various kinds of image stacks including those from the DIADEM challenge and BigNeuron project.

Rivulet: 3D Neuron Morphology Tracing with Iterative Back-Tracking

The digital reconstruction of single neurons from 3D confocal microscopic images is an important tool for understanding the neuron morphology and function. However the accurate automatic neuron reconstruction remains a challenging task due to the varying image quality and the complexity in the neuronal arborisation. Targeting the common challenges of neuron tracing, we propose a novel automatic 3D neuron reconstruction algorithm, named Rivulet, which is based on the multi-stencils fast-marching and iterative back-tracking. The proposed Rivulet algorithm is capable of tracing discontinuous areas without being interrupted by densely distributed noises. By evaluating the proposed pipeline with the data provided by the Diadem challenge and the recent BigNeuron project, Rivulet is shown to be robust to challenging microscopic imagestacks. We discussed the algorithm design in technical details regarding the relationships between the proposed algorithm and the other state-of-the-art neuron tracing algorithms.

Automated neuron tracing using probability hypothesis density filtering

Motivation

The functionality of neurons and their role in neuronal networks is tightly connected to the cell morphology. A fundamental problem in many neurobiological studies aiming to unravel this connection is the digital reconstruction of neuronal cell morphology from microscopic image data. Many methods have been developed for this, but they are far from perfect, and better methods are needed.

Results

Here we present a new method for tracing neuron centerlines needed for full reconstruction. The method uses a fundamentally different approach than previous methods by considering neuron tracing as a Bayesian multi-object tracking problem. The problem is solved using probability hypothesis density filtering. Results of experiments on 2D and 3D fluorescence microscopy image datasets of real neurons indicate the proposed method performs comparably or even better than the state of the art.

Availability and Implementation

Software implementing the proposed neuron tracing method was written in the Java programming language as a plugin for the ImageJ platform. Source code is freely available for non-commercial use at https://bitbucket.org/miroslavradojevic/phd .

Contact

meijering@imagescience.org.

Supplementary information

Supplementary data are available at Bioinformatics online.

Semi-automatic 3D morphological reconstruction of neurons with densely branching morphology: Application to retinal AII amacrine cells imaged with multi-photon excitation microscopy

BACKGROUND

Accurate reconstruction of the morphology of single neurons is important for morphometric studies and for developing compartmental models. However, manual morphological reconstruction can be extremely time-consuming and error-prone and algorithms for automatic reconstruction can be challenged when applied to neurons with a high density of extensively branching processes.

NEW METHOD

We present a procedure for semi-automatic reconstruction specifically adapted for densely branching neurons such as the AII amacrine cell found in mammalian retinas. We used whole-cell recording to fill AII amacrine cells in rat retinal slices with fluorescent dyes and acquired digital image stacks with multi-photon excitation microscopy. Our reconstruction algorithm combines elements of existing procedures, with segmentation based on adaptive thresholding and reconstruction based on a minimal spanning tree. We improved this workflow with an algorithm that reconnects neuron segments that are disconnected after adaptive thresholding, using paths extracted from the image stacks with the Fast Marching method.

RESULTS

By reducing the likelihood that disconnected segments were incorrectly connected to neighboring segments, our procedure generated excellent morphological reconstructions of AII amacrine cells.

COMPARISON WITH EXISTING METHODS

Reconstructing an AII amacrine cell required about 2h computing time, compared to 2-4days for manual reconstruction. To evaluate the performance of our method relative to manual reconstruction, we performed detailed analysis using a measure of tree structure similarity (DIADEM score), the degree of projection area overlap (Dice coefficient), and branch statistics.

CONCLUSIONS

We expect our procedure to be generally useful for morphological reconstruction of neurons filled with fluorescent dyes.

Fuzzy-Logic Based Detection and Characterization of Junctions and Terminations in Fluorescence Microscopy Images of Neurons

Digital reconstruction of neuronal cell morphology is an important step toward understanding the functionality of neuronal networks. Neurons are tree-like structures whose description depends critically on the junctions and terminations, collectively called critical points, making the correct localization and identification of these points a crucial task in the reconstruction process. Here we present a fully automatic method for the integrated detection and characterization of both types of critical points in fluorescence microscopy images of neurons. In view of the majority of our current studies, which are based on cultured neurons, we describe and evaluate the method for application to two-dimensional (2D) images. The method relies on directional filtering and angular profile analysis to extract essential features about the main streamlines at any location in an image, and employs fuzzy logic with carefully designed rules to reason about the feature values in order to make well-informed decisions about the presence of a critical point and its type. Experiments on simulated as well as real images of neurons demonstrate the detection performance of our method. A comparison with the output of two existing neuron reconstruction methods reveals that our method achieves substantially higher detection rates and could provide beneficial information to the reconstruction process.

Multi-scale segmentation of neurons based on one-class classification

BACKGROUND

High resolution multiphoton and confocal microscopy has allowed the acquisition of large amounts of data to be analyzed by neuroscientists. However, manual processing of these images has become infeasible. Thus, there is a need to create automatic methods for the morphological reconstruction of 3D neuronal image stacks.

NEW METHOD

An algorithm to extract the 3D morphology from a neuron is presented. The main contribution of the paper is the segmentation of the neuron from the background. Our segmentation method is based on one-class classification where the 3D image stack is analyzed at different scales. First, a multi-scale approach is proposed to compute the Laplacian of the 3D image stack. The Laplacian is used to select a training set consisting of background points. A decision function is learned for each scale from the training set that allows determining how similar an unlabeled point is to the points in the background class. Foreground points (dendrites and axons) are assigned as those points that are rejected as background. Finally, the morphological reconstruction of the neuron is extracted by applying a state-of-the-art centerline tracing algorithm on the segmentation.

RESULTS

Quantitative and qualitative results on several datasets demonstrate the ability of our algorithm to accurately and robustly segment and trace neurons.

COMPARISON WITH EXISTING METHOD(S)

Our method was compared to state-of-the-art neuron tracing algorithms.

CONCLUSIONS

Our approach allows segmentation of thin and low contrast dendrites that are usually difficult to segment. Compared to our previous approach, this algorithm is more accurate and much faster.

A Graph-Theoretical Approach for Tracing Filamentary Structures in Neuronal and Retinal Images

The aim of this study is about tracing filamentary structures in both neuronal and retinal images. It is often crucial to identify single neurons in neuronal networks, or separate vessel tree structures in retinal blood vessel networks, in applications such as drug screening for neurological disorders or computer-aided diagnosis of diabetic retinopathy. Both tasks are challenging as the same bottleneck issue of filament crossovers is commonly encountered, which essentially hinders the ability of existing systems to conduct large-scale drug screening or practical clinical usage. To address the filament crossovers' problem, a two-step graph-theoretical approach is proposed in this paper. The first step focuses on segmenting filamentary pixels out of the background. This produces a filament segmentation map used as input for the second step, where they are further separated into disjointed filaments. Key to our approach is the idea that the problem can be reformulated as label propagation over directed graphs, such that the graph is to be partitioned into disjoint sub-graphs, or equivalently, each of the neurons (vessel trees) is separated from the rest of the neuronal (vessel) network. This enables us to make the interesting connection between the tracing problem and the digraph matrix-forest theorem in algebraic graph theory for the first time. Empirical experiments on neuronal and retinal image datasets demonstrate the superior performance of our approach over existing methods.

Neuron anatomy structure reconstruction based on a sliding filter

BACKGROUND

Reconstruction of neuron anatomy structure is a challenging and important task in neuroscience. However, few algorithms can automatically reconstruct the full structure well without manual assistance, making it essential to develop new methods for this task.

METHODS

This paper introduces a new pipeline for reconstructing neuron anatomy structure from 3-D microscopy image stacks. This pipeline is initialized with a set of seeds that were detected by our proposed Sliding Volume Filter (SVF), given a non-circular cross-section of a neuron cell. Then, an improved open curve snake model combined with a SVF external force is applied to trace the full skeleton of the neuron cell. A radius estimation method based on a 2D sliding band filter is developed to fit the real edge of the cross-section of the neuron cell. Finally, a surface reconstruction method based on non-parallel curve networks is used to generate the neuron cell surface to finish this pipeline.

RESULTS

The proposed pipeline has been evaluated using publicly available datasets. The results show that the proposed method achieves promising results in some datasets from the DIgital reconstruction of Axonal and DEndritic Morphology (DIADEM) challenge and new BigNeuron project.

CONCLUSION

The new pipeline works well in neuron tracing and reconstruction. It can achieve higher efficiency, stability and robustness in neuron skeleton tracing. Furthermore, the proposed radius estimation method and applied surface reconstruction method can obtain more accurate neuron anatomy structures.

BigNeuron: Large-Scale 3D Neuron Reconstruction from Optical Microscopy Images

Understanding the structure of single neurons is critical for understanding how they function within neural circuits. BigNeuron is a new community effort that combines modern bioimaging informatics, recent leaps in labeling and microscopy, and the widely recognized need for openness and standardization to provide a community resource for automated reconstruction of dendritic and axonal morphology of single neurons.