Fully-Automatic Synapse Prediction and Validation on a Large Data Set

Neurotransmitter classification from electron microscopy images at synaptic sites in Drosophila melanogaster

High-resolution electron microscopy of nervous systems has enabled the reconstruction of synaptic connectomes. However, we do not know the synaptic sign for each connection (i.e., whether a connection is excitatory or inhibitory), which is implied by the released transmitter. We demonstrate that artificial neural networks can predict transmitter types for presynapses from electron micrographs: a network trained to predict six transmitters (acetylcholine, glutamate, GABA, serotonin, dopamine, octopamine) achieves an accuracy of 87% for individual synapses, 94% for neurons, and 91% for known cell types across a D. melanogaster whole brain. We visualize the ultrastructural features used for prediction, discovering subtle but significant differences between transmitter phenotypes. We also analyze transmitter distributions across the brain and find that neurons that develop together largely express only one fast-acting transmitter (acetylcholine, glutamate, or GABA). We hope that our publicly available predictions act as an accelerant for neuroscientific hypothesis generation for the fly.

SynapseCLR: Uncovering features of synapses in primary visual cortex through contrastive representation learning

3D electron microscopy (EM) connectomics image volumes are surpassing 1 mm³, providing information-dense, multi-scale visualizations of brain circuitry and necessitating scalable analysis techniques. We present SynapseCLR, a self-supervised contrastive learning method for 3D EM data, and use it to extract features of synapses from mouse visual cortex. SynapseCLR feature representations separate synapses by appearance and functionally important structural annotations. We demonstrate SynapseCLR's utility for valuable downstream tasks, including one-shot identification of defective synapse segmentations, dataset-wide similarity-based querying, and accurate imputation of annotations for unlabeled synapses, using manual annotation of only 0.2% of the dataset's synapses. In particular, excitatory versus inhibitory neuronal types can be assigned with >99.8% accuracy to individual synapses and highly truncated neurites, enabling neurite-enhanced connectomics analysis. Finally, we present a data-driven, unsupervised study of synaptic structural variation on the representation manifold, revealing its intrinsic axes of variation and showing that representations contain inhibitory subtype information.

Finding the right type of cell

A new method allows researchers to automatically assign cells into different cell types and tissues, a step which is critical for understanding complex organisms.

Local shape descriptors for neuron segmentation

We present an auxiliary learning task for the problem of neuron segmentation in electron microscopy volumes. The auxiliary task consists of the prediction of local shape descriptors (LSDs), which we combine with conventional voxel-wise direct neighbor affinities for neuron boundary detection. The shape descriptors capture local statistics about the neuron to be segmented, such as diameter, elongation, and direction. On a study comparing several existing methods across various specimen, imaging techniques, and resolutions, auxiliary learning of LSDs consistently increases segmentation accuracy of affinity-based methods over a range of metrics. Furthermore, the addition of LSDs promotes affinity-based segmentation methods to be on par with the current state of the art for neuron segmentation (flood-filling networks), while being two orders of magnitudes more efficient-a critical requirement for the processing of future petabyte-sized datasets.

En bloc preparation of Drosophila brains enables high-throughput FIB-SEM connectomics

Deriving the detailed synaptic connections of an entire nervous system is the unrealized goal of the nascent field of connectomics. For the fruit fly Drosophila, in particular, we need to dissect the brain, connectives, and ventral nerve cord as a single continuous unit, fix and stain it, and undertake automated segmentation of neuron membranes. To achieve this, we designed a protocol using progressive lowering of temperature dehydration (PLT), a technique routinely used to preserve cellular structure and antigenicity. We combined PLT with low temperature en bloc staining (LTS) and recover fixed neurons as round profiles with darkly stained synapses, suitable for machine segmentation and automatic synapse detection. Here we report three different PLT-LTS methods designed to meet the requirements for FIB-SEM imaging of the Drosophila brain. These requirements include: good preservation of ultrastructural detail, high level of en bloc staining, artifact-free microdissection, and smooth hot-knife cutting to reduce the brain to dimensions suited to FIB-SEM. In addition to PLT-LTS, we designed a jig to microdissect and pre-fix the fly's delicate brain and central nervous system. Collectively these methods optimize morphological preservation, allow us to image the brain usually at 8 nm per voxel, and simultaneously speed the formerly slow rate of FIB-SEM imaging.

Automatic classification and neurotransmitter prediction of synapses in electron microscopy

This paper presents a deep-learning-based workflow to detect synapses and predict their neurotransmitter type in the primitive chordate Ciona intestinalis (Ciona) electron microscopic (EM) images. Identifying synapses from EM images to build a full map of connections between neurons is a labor-intensive process and requires significant domain expertise. Automation of synapse classification would hasten the generation and analysis of connectomes. Furthermore, inferences concerning neuron type and function from synapse features are in many cases difficult to make. Finding the connection between synapse structure and function is an important step in fully understanding a connectome. Class Activation Maps derived from the convolutional neural network provide insights on important features of synapses based on cell type and function. The main contribution of this work is in the differentiation of synapses by neurotransmitter type through the structural information in their EM images. This enables the prediction of neurotransmitter types for neurons in Ciona, which were previously unknown. The prediction model with code is available on GitHub.

How innovations in methodology offer new prospects for volume electron microscopy

Detailed knowledge of biological structure has been key in understanding biology at several levels of organisation, from organs to cells and proteins. Volume electron microscopy (volume EM) provides high resolution 3D structural information about tissues on the nanometre scale. However, the throughput rate of conventional electron microscopes has limited the volume size and number of samples that can be imaged. Recent improvements in methodology are currently driving a revolution in volume EM, making possible the structural imaging of whole organs and small organisms. In turn, these recent developments in image acquisition have created or stressed bottlenecks in other parts of the pipeline, like sample preparation, image analysis and data management. While the progress in image analysis is stunning due to the advent of automatic segmentation and server‐based annotation tools, several challenges remain. Here we discuss recent trends in volume EM, emerging methods for increasing throughput and implications for sample preparation, image analysis and data management.

Edge-colored directed subgraph enumeration on the connectome

Following significant advances in image acquisition, synapse detection, and neuronal segmentation in connectomics, researchers have extracted an increasingly diverse set of wiring diagrams from brain tissue. Neuroscientists frequently represent these wiring diagrams as graphs with nodes corresponding to a single neuron and edges indicating synaptic connectivity. The edges can contain "colors" or "labels", indicating excitatory versus inhibitory connections, among other things. By representing the wiring diagram as a graph, we can begin to identify motifs, the frequently occurring subgraphs that correspond to specific biological functions. Most analyses on these wiring diagrams have focused on hypothesized motifs-those we expect to find. However, one of the goals of connectomics is to identify biologically-significant motifs that we did not previously hypothesize. To identify these structures, we need large-scale subgraph enumeration to find the frequencies of all unique motifs. Exact subgraph enumeration is a computationally expensive task, particularly in the edge-dense wiring diagrams. Furthermore, most existing methods do not differentiate between types of edges which can significantly affect the function of a motif. We propose a parallel, general-purpose subgraph enumeration strategy to count motifs in the connectome. Next, we introduce a divide-and-conquer community-based subgraph enumeration strategy that allows for enumeration per brain region. Lastly, we allow for differentiation of edges by types to better reflect the underlying biological properties of the graph. We demonstrate our results on eleven connectomes and publish for future analyses extensive overviews for the 26 trillion subgraphs enumerated that required approximately 9.25 years of computation time.

Automated Synapse Detection Method for Cerebellar Connectomics

The connectomic analyses of large-scale volumetric electron microscope (EM) images enable the discovery of hidden neural connectivity. While the technologies for neuronal reconstruction of EM images are under rapid progress, the technologies for synapse detection are lagging behind. Here, we propose a method that automatically detects the synapses in the 3D EM images, specifically for the mouse cerebellar molecular layer (CML). The method aims to accurately detect the synapses between the reconstructed neuronal fragments whose types can be identified. It extracts the contacts between the reconstructed neuronal fragments and classifies them as synaptic or non-synaptic with the help of type information and two deep learning artificial intelligences (AIs). The method can also assign the pre- and postsynaptic sides of a synapse and determine excitatory and inhibitory synapse types. The accuracy of this method is estimated to be 0.955 in F1-score for a test volume of CML containing 508 synapses. To demonstrate the usability, we measured the size and number of the synapses in the volume and investigated the subcellular connectivity between the CML neuronal fragments. The basic idea of the method to exploit tissue-specific properties can be extended to other brain regions.

RealNeuralNetworks.jl: An Integrated Julia Package for Skeletonization, Morphological Analysis, and Synaptic Connectivity Analysis of Terabyte-Scale 3D Neural Segmentations

Benefiting from the rapid development of electron microscopy imaging and deep learning technologies, an increasing number of brain image datasets with segmentation and synapse detection are published. Most of the automated segmentation methods label voxels rather than producing neuron skeletons directly. A further skeletonization step is necessary for quantitative morphological analysis. Currently, several tools are published for skeletonization as well as morphological and synaptic connectivity analysis using different computer languages and environments. Recently the Julia programming language, notable for elegant syntax and high performance, has gained rapid adoption in the scientific computing community. Here, we present a Julia package, called RealNeuralNetworks.jl, for efficient sparse skeletonization, morphological analysis, and synaptic connectivity analysis. Based on a large-scale Zebrafish segmentation dataset, we illustrate the software features by performing distributed skeletonization in Google Cloud, clustering the neurons using the NBLAST algorithm, combining morphological similarity and synaptic connectivity to study their relationship. We demonstrate that RealNeuralNetworks.jl is suitable for use in terabyte-scale electron microscopy image segmentation datasets.

FlyWire: online community for whole-brain connectomics

Due to advances in automated image acquisition and analysis, whole-brain connectomes with 100,000 or more neurons are on the horizon. Proofreading of whole-brain automated reconstructions will require many person-years of effort, due to the huge volumes of data involved. Here we present FlyWire, an online community for proofreading neural circuits in a Drosophila melanogaster brain and explain how its computational and social structures are organized to scale up to whole-brain connectomics. Browser-based three-dimensional interactive segmentation by collaborative editing of a spatially chunked supervoxel graph makes it possible to distribute proofreading to individuals located virtually anywhere in the world. Information in the edit history is programmatically accessible for a variety of uses such as estimating proofreading accuracy or building incentive systems. An open community accelerates proofreading by recruiting more participants and accelerates scientific discovery by requiring information sharing. We demonstrate how FlyWire enables circuit analysis by reconstructing and analyzing the connectome of mechanosensory neurons.

Automatic detection of synaptic partners in a whole-brain Drosophila electron microscopy data set

We develop an automatic method for synaptic partner identification in insect brains and use it to predict synaptic partners in a whole-brain electron microscopy dataset of the fruit fly. The predictions can be used to infer a connectivity graph with high accuracy, thus allowing fast identification of neural pathways. To facilitate circuit reconstruction using our results, we develop CIRCUITMAP, a user interface add-on for the circuit annotation tool CATMAID.

Light microscopy based approach for mapping connectivity with molecular specificity

Mapping neuroanatomy is a foundational goal towards understanding brain function. Electron microscopy (EM) has been the gold standard for connectivity analysis because nanoscale resolution is necessary to unambiguously resolve synapses. However, molecular information that specifies cell types is often lost in EM reconstructions. To address this, we devise a light microscopy approach for connectivity analysis of defined cell types called spectral connectomics. We combine multicolor labeling (Brainbow) of neurons with multi-round immunostaining Expansion Microscopy (miriEx) to simultaneously interrogate morphology, molecular markers, and connectivity in the same brain section. We apply this strategy to directly link inhibitory neuron cell types with their morphologies. Furthermore, we show that correlative Brainbow and endogenous synaptic machinery immunostaining can define putative synaptic connections between neurons, as well as map putative inhibitory and excitatory inputs. We envision that spectral connectomics can be applied routinely in neurobiology labs to gain insights into normal and pathophysiological neuroanatomy.

A connectome and analysis of the adult Drosophila central brain

The neural circuits responsible for animal behavior remain largely unknown. We summarize new methods and present the circuitry of a large fraction of the brain of the fruit fly Drosophila melanogaster. Improved methods include new procedures to prepare, image, align, segment, find synapses in, and proofread such large data sets. We define cell types, refine computational compartments, and provide an exhaustive atlas of cell examples and types, many of them novel. We provide detailed circuits consisting of neurons and their chemical synapses for most of the central brain. We make the data public and simplify access, reducing the effort needed to answer circuit questions, and provide procedures linking the neurons defined by our analysis with genetic reagents. Biologically, we examine distributions of connection strengths, neural motifs on different scales, electrical consequences of compartmentalization, and evidence that maximizing packing density is an important criterion in the evolution of the fly's brain.

Two Stream Active Query Suggestion for Active Learning in Connectomics

For large-scale vision tasks in biomedical images, the labeled data is often limited to train effective deep models. Active learning is a common solution, where a query suggestion method selects representative unlabeled samples for annotation, and the new labels are used to improve the base model. However, most query suggestion models optimize their learnable parameters only on the limited labeled data and consequently become less effective for the more challenging unlabeled data. To tackle this, we propose a two-stream active query suggestion approach. In addition to the supervised feature extractor, we introduce an unsupervised one optimized on all raw images to capture diverse image features, which can later be improved by fine-tuning on new labels. As a use case, we build an end-to-end active learning framework with our query suggestion method for 3D synapse detection and mitochondria segmentation in connectomics. With the framework, we curate, to our best knowledge, the largest connectomics dataset with dense synapses and mitochondria annotation. On this new dataset, our method outperforms previous state-of-the-art methods by 3.1% for synapse and 3.8% for mitochondria in terms of region-of-interest proposal accuracy. We also apply our method to image classification, where it outperforms previous approaches on CIFAR-10 under the same limited annotation budget. The project page is https://zudi-lin.github.io/projects/#two_stream_active.

Automatic Reconstruction of Mitochondria and Endoplasmic Reticulum in Electron Microscopy Volumes by Deep Learning

Together, mitochondria and the endoplasmic reticulum (ER) occupy more than 20% of a cell's volume, and morphological abnormality may lead to cellular function disorders. With the rapid development of large-scale electron microscopy (EM), manual contouring and three-dimensional (3D) reconstruction of these organelles has previously been accomplished in biological studies. However, manual segmentation of mitochondria and ER from EM images is time consuming and thus unable to meet the demands of large data analysis. Here, we propose an automated pipeline for mitochondrial and ER reconstruction, including the mitochondrial and ER contact sites (MAMs). We propose a novel recurrent neural network to detect and segment mitochondria and a fully residual convolutional network to reconstruct the ER. Based on the sparse distribution of synapses, we use mitochondrial context information to rectify the local misleading results and obtain 3D mitochondrial reconstructions. The experimental results demonstrate that the proposed method achieves state-of-the-art performance.

Convolutional nets for reconstructing neural circuits from brain images acquired by serial section electron microscopy

Neural circuits can be reconstructed from brain images acquired by serial section electron microscopy. Image analysis has been performed by manual labor for half a century, and efforts at automation date back almost as far. Convolutional nets were first applied to neuronal boundary detection a dozen years ago, and have now achieved impressive accuracy on clean images. Robust handling of image defects is a major outstanding challenge. Convolutional nets are also being employed for other tasks in neural circuit reconstruction: finding synapses and identifying synaptic partners, extending or pruning neuronal reconstructions, and aligning serial section images to create a 3D image stack. Computational systems are being engineered to handle petavoxel images of cubic millimeter brain volumes.

Comparisons between the ON- and OFF-edge motion pathways in the Drosophila brain

Understanding the circuit mechanisms behind motion detection is a long-standing question in visual neuroscience. In Drosophila melanogaster, recently discovered synapse-level connectomes in the optic lobe, particularly in ON-pathway (T4) receptive-field circuits, in concert with physiological studies, suggest a motion model that is increasingly intricate when compared with the ubiquitous Hassenstein-Reichardt model. By contrast, our knowledge of OFF-pathway (T5) has been incomplete. Here, we present a conclusive and comprehensive connectome that, for the first time, integrates detailed connectivity information for inputs to both the T4 and T5 pathways in a single EM dataset covering the entire optic lobe. With novel reconstruction methods using automated synapse prediction suited to such a large connectome, we successfully corroborate previous findings in the T4 pathway and comprehensively identify inputs and receptive fields for T5. Although the two pathways are probably evolutionarily linked and exhibit many similarities, we uncover interesting differences and interactions that may underlie their distinct functional properties.

SynEM, automated synapse detection for connectomics

Nerve tissue contains a high density of chemical synapses, about 1 per µm3 in the mammalian cerebral cortex. Thus, even for small blocks of nerve tissue, dense connectomic mapping requires the identification of millions to billions of synapses. While the focus of connectomic data analysis has been on neurite reconstruction, synapse detection becomes limiting when datasets grow in size and dense mapping is required. Here, we report SynEM, a method for automated detection of synapses from conventionally en-bloc stained 3D electron microscopy image stacks. The approach is based on a segmentation of the image data and focuses on classifying borders between neuronal processes as synaptic or non-synaptic. SynEM yields 97% precision and recall in binary cortical connectomes with no user interaction. It scales to large volumes of cortical neuropil, plausibly even whole-brain datasets. SynEM removes the burden of manual synapse annotation for large densely mapped connectomes.

FIJI Macro 3D ART VeSElecT: 3D Automated Reconstruction Tool for Vesicle Structures of Electron Tomograms

Automatic image reconstruction is critical to cope with steadily increasing data from advanced microscopy. We describe here the Fiji macro 3D ART VeSElecT which we developed to study synaptic vesicles in electron tomograms. We apply this tool to quantify vesicle properties (i) in embryonic Danio rerio 4 and 8 days past fertilization (dpf) and (ii) to compare Caenorhabditis elegans N2 neuromuscular junctions (NMJ) wild-type and its septin mutant (unc-59(e261)). We demonstrate development-specific and mutant-specific changes in synaptic vesicle pools in both models. We confirm the functionality of our macro by applying our 3D ART VeSElecT on zebrafish NMJ showing smaller vesicles in 8 dpf embryos then 4 dpf, which was validated by manual reconstruction of the vesicle pool. Furthermore, we analyze the impact of C. elegans septin mutant unc-59(e261) on vesicle pool formation and vesicle size. Automated vesicle registration and characterization was implemented in Fiji as two macros (registration and measurement). This flexible arrangement allows in particular reducing false positives by an optional manual revision step. Preprocessing and contrast enhancement work on image-stacks of 1nm/pixel in x and y direction. Semi-automated cell selection was integrated. 3D ART VeSElecT removes interfering components, detects vesicles by 3D segmentation and calculates vesicle volume and diameter (spherical approximation, inner/outer diameter). Results are collected in color using the RoiManager plugin including the possibility of manual removal of non-matching confounder vesicles. Detailed evaluation considered performance (detected vesicles) and specificity (true vesicles) as well as precision and recall. We furthermore show gain in segmentation and morphological filtering compared to learning based methods and a large time gain compared to manual segmentation. 3D ART VeSElecT shows small error rates and its speed gain can be up to 68 times faster in comparison to manual annotation. Both automatic and semi-automatic modes are explained including a tutorial.