A scalable and modular automated pipeline for stitching of large electron microscopy datasets

Expanding the field of view - a simple approach for interactive visualisation of electron microscopy data

The unparalleled resolving power of electron microscopy is both a blessing and a curse. At 30,000× magnification, 1 µm corresponds to 3 cm in the image and the field of view is only a few micrometres or less, resulting in an inevitable reduction in the spatial data available in an image. Consequently, the gain in resolution is at the cost of loss of the contextual 'reference space', which is crucial for understanding the embedded structures of interest. This problem is particularly pronounced in immunoelectron microscopy, where the detection of a gold particle is crucial for the localisation of specific molecules. The common solution of presenting high-magnification and overview images side by side often insufficiently represents the cellular environment. To address these limitations, we propose here an interactive visualization strategy inspired by digital maps and GPS modules which enables seamless transitions between different magnifications by dynamically linking virtual low magnification overview images with primary high-resolution data. By enabling dynamic browsing, it offers the potential for a deeper understanding of cellular landscapes leading to more comprehensive analysis of the primary ultrastructural data.

Local-Peak Scale-Invariant Feature Transform for Fast and Random Image Stitching

Image stitching aims to construct a wide field of view with high spatial resolution, which cannot be achieved in a single exposure. Typically, conventional image stitching techniques, other than deep learning, require complex computation and are thus computationally expensive, especially for stitching large raw images. In this study, inspired by the multiscale feature of fluid turbulence, we developed a fast feature point detection algorithm named local-peak scale-invariant feature transform (LP-SIFT), based on the multiscale local peaks and scale-invariant feature transform method. By combining LP-SIFT and RANSAC in image stitching, the stitching speed can be improved by orders compared with the original SIFT method. Benefiting from the adjustable size of the interrogation window, the LP-SIFT algorithm demonstrates comparable or even less stitching time than the other commonly used algorithms, while achieving comparable or even better stitching results. Nine large images (over 2600 × 1600 pixels), arranged randomly without prior knowledge, can be stitched within 158.94 s. The algorithm is highly practical for applications requiring a wide field of view in diverse application scenes, e.g., terrain mapping, biological analysis, and even criminal investigation.

Fast imaging of millimeter-scale areas with beam deflection transmission electron microscopy

Serial section transmission electron microscopy (TEM) has proven to be one of the leading methods for millimeter-scale 3D imaging of brain tissues at nanoscale resolution. It is important to further improve imaging efficiency to acquire larger and more brain volumes. We report here a threefold increase in the speed of TEM by using a beam deflecting mechanism to enable highly efficient acquisition of multiple image tiles (nine) for each motion of the mechanical stage. For millimeter-scale areas, the duty cycle of imaging doubles to more than 30%, yielding a net average imaging rate of 0.3 gigapixels per second. If fully utilized, an array of four beam deflection TEMs should be capable of imaging a dataset of cubic millimeter scale in five weeks.

A scalable and modular computational pipeline for axonal connectomics: automated tracing and assembly of axons across serial sections

Progress in histological methods and in microscope technology has enabled dense staining and imaging of axons over large brain volumes, but tracing axons over such volumes requires new computational tools for 3D reconstruction of data acquired from serial sections. We have developed a computational pipeline for automated tracing and volume assembly of densely stained axons imaged over serial sections, which leverages machine learning-based segmentation to enable stitching and alignment with the axon traces themselves. We validated this segmentation-driven approach to volume assembly and alignment of individual axons over centimeter-scale serial sections and show the application of the output traces for analysis of local orientation and for proofreading over aligned volumes. The pipeline is scalable, and combined with recent advances in experimental approaches, should enable new studies of mesoscale connectivity and function over the whole human brain.

Large-scale neurophysiology and single-cell profiling in human neuroscience

Advances in large-scale single-unit human neurophysiology, single-cell RNA sequencing, spatial transcriptomics and long-term ex vivo tissue culture of surgically resected human brain tissue have provided an unprecedented opportunity to study human neuroscience. In this Perspective, we describe the development of these paradigms, including Neuropixels and recent brain-cell atlas efforts, and discuss how their convergence will further investigations into the cellular underpinnings of network-level activity in the human brain. Specifically, we introduce a workflow in which functionally mapped samples of human brain tissue resected during awake brain surgery can be cultured ex vivo for multi-modal cellular and functional profiling. We then explore how advances in human neuroscience will affect clinical practice, and conclude by discussing societal and ethical implications to consider. Potential findings from the field of human neuroscience will be vast, ranging from insights into human neurodiversity and evolution to providing cell-type-specific access to study and manipulate diseased circuits in pathology. This Perspective aims to provide a unifying framework for the field of human neuroscience as we welcome an exciting era for understanding the functional cytoarchitecture of the human brain.

FAST-EM array tomography: a workflow for multibeam volume electron microscopy

Elucidating the 3D nanoscale structure of tissues and cells is essential for understanding the complexity of biological processes. Electron microscopy (EM) offers the resolution needed for reliable interpretation, but the limited throughput of electron microscopes has hindered its ability to effectively image large volumes. We report a workflow for volume EM with FAST-EM, a novel multibeam scanning transmission electron microscope that speeds up acquisition by scanning the sample in parallel with 64 electron beams. FAST-EM makes use of optical detection to separate the signals of the individual beams. The acquisition and 3D reconstruction of ultrastructural data from multiple biological samples is demonstrated. The results show that the workflow is capable of producing large reconstructed volumes with high resolution and contrast to address biological research questions within feasible acquisition time frames.

Cell-type-specific inhibitory circuitry from a connectomic census of mouse visual cortex

Mammalian cortex features a vast diversity of neuronal cell types, each with characteristic anatomical, molecular and functional properties. Synaptic connectivity powerfully shapes how each cell type participates in the cortical circuit, but mapping connectivity rules at the resolution of distinct cell types remains difficult. Here, we used millimeter-scale volumetric electron microscopy1 to investigate the connectivity of all inhibitory neurons across a densely-segmented neuronal population of 1352 cells spanning all layers of mouse visual cortex, producing a wiring diagram of inhibitory connections with more than 70,000 synapses. Taking a data-driven approach inspired by classical neuroanatomy, we classified inhibitory neurons based on the relative targeting of dendritic compartments and other inhibitory cells and developed a novel classification of excitatory neurons based on the morphological and synaptic input properties. The synaptic connectivity between inhibitory cells revealed a novel class of disinhibitory specialist targeting basket cells, in addition to familiar subclasses. Analysis of the inhibitory connectivity onto excitatory neurons found widespread specificity, with many interneurons exhibiting differential targeting of certain subpopulations spatially intermingled with other potential targets. Inhibitory targeting was organized into "motif groups," diverse sets of cells that collectively target both perisomatic and dendritic compartments of the same excitatory targets. Collectively, our analysis identified new organizing principles for cortical inhibition and will serve as a foundation for linking modern multimodal neuronal atlases with the cortical wiring diagram.

Petascale pipeline for precise alignment of images from serial section electron microscopy

The reconstruction of neural circuits from serial section electron microscopy (ssEM) images is being accelerated by automatic image segmentation methods. Segmentation accuracy is often limited by the preceding step of aligning 2D section images to create a 3D image stack. Precise and robust alignment in the presence of image artifacts is challenging, especially as datasets are attaining the petascale. We present a computational pipeline for aligning ssEM images with several key elements. Self-supervised convolutional nets are trained via metric learning to encode and align image pairs, and they are used to initialize iterative fine-tuning of alignment. A procedure called vector voting increases robustness to image artifacts or missing image data. For speedup the series is divided into blocks that are distributed to computational workers for alignment. The blocks are aligned to each other by composing transformations with decay, which achieves a global alignment without resorting to a time-consuming global optimization. We apply our pipeline to a whole fly brain dataset, and show improved accuracy relative to prior state of the art. We also demonstrate that our pipeline scales to a cubic millimeter of mouse visual cortex. Our pipeline is publicly available through two open source Python packages.

msemalign: a pipeline for serial section multibeam scanning electron microscopy volume alignment

Serial section multibeam scanning electron microscopy (ssmSEM) is currently among the fastest technologies available for acquiring 3D anatomical data spanning relatively large neural tissue volumes, on the order of 1 mm3 or larger, at a resolution sufficient to resolve the fine detail of neuronal morphologies and synapses. These petabyte-scale volumes can be analyzed to create connectomes, datasets that contain detailed anatomical information including synaptic connectivity, neuronal morphologies and distributions of cellular organelles. The mSEM acquisition process creates hundreds of millions of individual image tiles for a single cubic-millimeter-sized dataset and these tiles must be aligned to create 3D volumes. Here we introduce msemalign, an alignment pipeline that strives for scalability and design simplicity. The pipeline can align petabyte-scale datasets such that they contain smooth transitions as the dataset is navigated in all directions, but critically that does so in a fashion that minimizes the overall magnitude of section distortions relative to the originally acquired micrographs.

An intelligent workflow for sub-nanoscale 3D reconstruction of intact synapses from serial section electron tomography

BACKGROUND

As an extension of electron tomography (ET), serial section electron tomography (serial section ET) aims to align the tomographic images of multiple thick tissue sections together, to break through the volume limitation of the single section and preserve the sub-nanoscale voxel size. It could be applied to reconstruct the intact synapse, which expands about one micrometer and contains nanoscale vesicles. However, there are several drawbacks of the existing serial section ET methods. First, locating and imaging regions of interest (ROIs) in serial sections during the shooting process is time-consuming. Second, the alignment of ET volumes is difficult due to the missing information caused by section cutting and imaging. Here we report a workflow to simplify the acquisition of ROIs in serial sections, automatically align the volume of serial section ET, and semi-automatically reconstruct the target synaptic structure.

RESULTS

We propose an intelligent workflow to reconstruct the intact synapse with sub-nanometer voxel size. Our workflow includes rapid localization of ROIs in serial sections, automatic alignment, restoration, assembly of serial ET volumes, and semi-automatic target structure segmentation. For the localization and acquisition of ROIs in serial sections, we use affine transformations to calculate their approximate position based on their relative location in orderly placed sections. For the alignment of consecutive ET volumes with significantly distinct appearances, we use multi-scale image feature matching and the elastic with belief propagation (BP-Elastic) algorithm to align them from coarse to fine. For the restoration of the missing information in ET, we first estimate the number of lost images based on the pixel changes of adjacent volumes after alignment. Then, we present a missing information generation network that is appropriate for small-sample of ET volume using pre-training interpolation network and distillation learning. And we use it to generate the missing information to achieve the whole volume reconstruction. For the reconstruction of synaptic ultrastructures, we use a 3D neural network to obtain them quickly. In summary, our workflow can quickly locate and acquire ROIs in serial sections, automatically align, restore, assemble serial sections, and obtain the complete segmentation result of the target structure with minimal manual manipulation. Multiple intact synapses in wild-type rat were reconstructed at a voxel size of 0.664 nm/voxel to demonstrate the effectiveness of our workflow.

CONCLUSIONS

Our workflow contributes to obtaining intact synaptic structures at the sub-nanometer scale through serial section ET, which contains rapid ROI locating, automatic alignment, volume reconstruction, and semi-automatic synapse reconstruction. We have open-sourced the relevant code in our workflow, so it is easy to apply it to other labs and obtain complete 3D ultrastructures which size is similar to intact synapses with sub-nanometer voxel size.