Segmentation of Neurons from Fluorescence Calcium Recordings Beyond Real-time

SomaSeg: a robust neuron identification framework for two-photon imaging video

Objective.Accurate neuron identification is fundamental to the analysis of neuronal population dynamics and signal extraction in fluorescence videos. However, several factors such as severe imaging noise, out-of-focus neuropil contamination, and adjacent neuron overlap would impair the performance of neuron identification algorithms and lead to errors in neuron shape and calcium activity extraction, or ultimately compromise the reliability of analysis conclusions.Approach.To address these challenges, we developed a novel cascade framework named SomaSeg. This framework integrates Duffing denoising and neuropil contamination defogging for video enhancement, and an overlapping instance segmentation network for stacked neurons differentiating.Main results.Compared with the state-of-the-art neuron identification methods, both simulation and actual experimental results demonstrate that SomaSeg framework is robust to noise, insensitive to out-of-focus contamination and effective in dealing with overlapping neurons in actual complex imaging scenarios.Significance.The SomaSeg framework provides a widely applicable solution for two-photon video processing, which enhances the reliability of neuron identification and exhibits value in distinguishing visually confusing neurons.

Accurate neuron segmentation method for one-photon calcium imaging videos combining convolutional neural networks and clustering

One-photon fluorescent calcium imaging helps understand brain functions by recording large-scale neural activities in freely moving animals. Automatic, fast, and accurate active neuron segmentation algorithms are essential to extract and interpret information from these videos. One-photon imaging videos' low resolution, high noise, and high background fluctuation pose significant challenges. Here, we develop a software pipeline to address the challenges of processing one-photon calcium imaging videos. We extend our previous two-photon active neuron segmentation algorithm, Shallow U-Net Neuron Segmentation (SUNS), to better suppress background fluctuations in one-photon videos. We also develop additional neuron extraction (ANE) to locate small or dim neurons missed by SUNS. To train our segmentation method, we create ground truth neurons by developing a manual labeling pipeline assisted with semi-automatic refinement. Our method is more accurate and faster than state-of-the-art techniques when processing simulated videos and multiple experimental datasets acquired over various brain regions with different imaging conditions.

Closed-loop experiments and brain machine interfaces with multiphoton microscopy

In the field of neuroscience, the importance of constructing closed-loop experimental systems has increased in conjunction with technological advances in measuring and controlling neural activity in live animals. We provide an overview of recent technological advances in the field, focusing on closed-loop experimental systems where multiphoton microscopy-the only method capable of recording and controlling targeted population activity of neurons at a single-cell resolution in vivo-works through real-time feedback. Specifically, we present some examples of brain machine interfaces (BMIs) using in vivo two-photon calcium imaging and discuss applications of two-photon optogenetic stimulation and adaptive optics to real-time BMIs. We also consider conditions for realizing future optical BMIs at the synaptic level, and their possible roles in understanding the computational principles of the brain.

Neurophotonics: a comprehensive review, current challenges and future trends

The human brain, with its vast network of billions of neurons and trillions of synapses (connections) between diverse cell types, remains one of the greatest mysteries in science and medicine. Despite extensive research, an understanding of the underlying mechanisms that drive normal behaviors and response to disease states is still limited. Advancement in the Neuroscience field and development of therapeutics for related pathologies requires innovative technologies that can provide a dynamic and systematic understanding of the interactions between neurons and neural circuits. In this work, we provide an up-to-date overview of the evolution of neurophotonic approaches in the last 10 years through a multi-source, literature analysis. From an initial corpus of 243 papers retrieved from Scopus, PubMed and WoS databases, we have followed the PRISMA approach to select 56 papers in the area. Following a full-text evaluation of these 56 scientific articles, six main areas of applied research were identified and discussed: (1) Advanced optogenetics, (2) Multimodal neural interfaces, (3) Innovative therapeutics, (4) Imaging devices and probes, (5) Remote operations, and (6) Microfluidic platforms. For each area, the main technologies selected are discussed according to the photonic principles applied, the neuroscience application evaluated and the more indicative results of efficiency and scientific potential. This detailed analysis is followed by an outlook of the main challenges tackled over the last 10 years in the Neurophotonics field, as well as the main technological advances regarding specificity, light delivery, multimodality, imaging, materials and system designs. We conclude with a discussion of considerable challenges for future innovation and translation in Neurophotonics, from light delivery within the brain to physical constraints and data management strategies.

NeuroSeg-III: efficient neuron segmentation in two-photon Ca2+ imaging data using self-supervised learning

Two-photon Ca2+ imaging technology increasingly plays an essential role in neuroscience research. However, the requirement for extensive professional annotation poses a significant challenge to improving the performance of neuron segmentation models. Here, we present NeuroSeg-III, an innovative self-supervised learning approach specifically designed to achieve fast and precise segmentation of neurons in imaging data. This approach consists of two modules: a self-supervised pre-training network and a segmentation network. After pre-training the encoder of the segmentation network via a self-supervised learning method without any annotated data, we only need to fine-tune the segmentation network with a small amount of annotated data. The segmentation network is designed with YOLOv8s, FasterNet, efficient multi-scale attention mechanism (EMA), and bi-directional feature pyramid network (BiFPN), which enhanced the model's segmentation accuracy while reducing the computational cost and parameters. The generalization of our approach was validated across different Ca2+ indicators and scales of imaging data. Significantly, the proposed neuron segmentation approach exhibits exceptional speed and accuracy, surpassing the current state-of-the-art benchmarks when evaluated using a publicly available dataset. The results underscore the effectiveness of NeuroSeg-III, with employing an efficient training strategy tailored for two-photon Ca2+ imaging data and delivering remarkable precision in neuron segmentation.

A Semi-supervised Pipeline for Accurate Neuron Segmentation with Fewer Ground Truth Labels

Recent advancements in two-photon calcium imaging have enabled scientists to record the activity of thousands of neurons with cellular resolution. This scope of data collection is crucial to understanding the next generation of neuroscience questions, but analyzing these large recordings requires automated methods for neuron segmentation. Supervised methods for neuron segmentation achieve state of-the-art accuracy and speed but currently require large amounts of manually generated ground truth training labels. We reduced the required number of training labels by designing a semi-supervised pipeline. Our pipeline used neural network ensembling to generate pseudolabels to train a single shallow U-Net. We tested our method on three publicly available datasets and compared our performance to three widely used segmentation methods. Our method outperformed other methods when trained on a small number of ground truth labels and could achieve state-of-the-art accuracy after training on approximately a quarter of the number of ground truth labels as supervised methods. When trained on many ground truth labels, our pipeline attained higher accuracy than that of state-of-the-art methods. Overall, our work will help researchers accurately process large neural recordings while minimizing the time and effort needed to generate manual labels.

Pupil engineering for extended depth-of-field imaging in a fluorescence miniscope

Significance

Fluorescence head-mounted microscopes, i.e., miniscopes, have emerged as powerful tools to analyze in-vivo neural populations but exhibit a limited depth-of-field (DoF) due to the use of high numerical aperture (NA) gradient refractive index (GRIN) objective lenses.

Aim

We present extended depth-of-field (EDoF) miniscope, which integrates an optimized thin and lightweight binary diffractive optical element (DOE) onto the GRIN lens of a miniscope to extend the DoF by 2.8× between twin foci in fixed scattering samples.

Approach

We use a genetic algorithm that considers the GRIN lens' aberration and intensity loss from scattering in a Fourier optics-forward model to optimize a DOE and manufacture the DOE through single-step photolithography. We integrate the DOE into EDoF-Miniscope with a lateral accuracy of 70  μm to produce high-contrast signals without compromising the speed, spatial resolution, size, or weight.

Results

We characterize the performance of EDoF-Miniscope across 5- and 10-μm fluorescent beads embedded in scattering phantoms and demonstrate that EDoF-Miniscope facilitates deeper interrogations of neuronal populations in a 100-μm-thick mouse brain sample and vessels in a whole mouse brain sample.

Conclusions

Built from off-the-shelf components and augmented by a customizable DOE, we expect that this low-cost EDoF-Miniscope may find utility in a wide range of neural recording applications.

maskNMF: A denoise-sparsen-detect approach for extracting neural signals from dense imaging data

A number of calcium imaging methods have been developed to monitor the activity of large populations of neurons. One particularly promising approach, Bessel imaging, captures neural activity from a volume by projecting within the imaged volume onto a single imaging plane, therefore effectively mixing signals and increasing the number of neurons imaged per pixel. These signals must then be computationally demixed to recover the desired neural activity. Unfortunately, currently-available demixing methods can perform poorly in the regime of high imaging density (i.e., many neurons per pixel). In this work we introduce a new pipeline (maskNMF) for demixing dense calcium imaging data. The main idea is to first denoise and temporally sparsen the observed video; this enhances signal strength and reduces spatial overlap significantly. Next we detect neurons in the sparsened video using a neural network trained on a library of neural shapes. These shapes are derived from segmented electron microscopy images input into a Bessel imaging model; therefore no manual selection of "good" neural shapes from the functional data is required here. After cells are detected, we use a constrained non-negative matrix factorization approach to demix the activity, using the detected cells' shapes to initialize the factorization. We test the resulting pipeline on both simulated and real datasets and find that it is able to achieve accurate demixing on denser data than was previously feasible, therefore enabling faithful imaging of larger neural populations. The method also provides good results on more "standard" two-photon imaging data. Finally, because much of the pipeline operates on a significantly compressed version of the raw data and is highly parallelizable, the algorithm is fast, processing large datasets faster than real time.

FIOLA: an accelerated pipeline for fluorescence imaging online analysis

Optical microscopy methods such as calcium and voltage imaging enable fast activity readout of large neuronal populations using light. However, the lack of corresponding advances in online algorithms has slowed progress in retrieving information about neural activity during or shortly after an experiment. This gap not only prevents the execution of real-time closed-loop experiments, but also hampers fast experiment-analysis-theory turnover for high-throughput imaging modalities. Reliable extraction of neural activity from fluorescence imaging frames at speeds compatible with indicator dynamics and imaging modalities poses a challenge. We therefore developed FIOLA, a framework for fluorescence imaging online analysis that extracts neuronal activity from calcium and voltage imaging movies at speeds one order of magnitude faster than state-of-the-art methods. FIOLA exploits algorithms optimized for parallel processing on GPUs and CPUs. We demonstrate reliable and scalable performance of FIOLA on both simulated and real calcium and voltage imaging datasets. Finally, we present an online experimental scenario to provide guidance in setting FIOLA parameters and to highlight the trade-offs of our approach.

Sharing neurophysiology data from the Allen Brain Observatory

Nullius in verba ('trust no one'), chosen as the motto of the Royal Society in 1660, implies that independently verifiable observations-rather than authoritative claims-are a defining feature of empirical science. As the complexity of modern scientific instrumentation has made exact replications prohibitive, sharing data is now essential for ensuring the trustworthiness of one's findings. While embraced in spirit by many, in practice open data sharing remains the exception in contemporary systems neuroscience. Here, we take stock of the Allen Brain Observatory, an effort to share data and metadata associated with surveys of neuronal activity in the visual system of laboratory mice. Data from these surveys have been used to produce new discoveries, to validate computational algorithms, and as a benchmark for comparison with other data, resulting in over 100 publications and preprints to date. We distill some of the lessons learned about open surveys and data reuse, including remaining barriers to data sharing and what might be done to address these.

Rapid detection of neurons in widefield calcium imaging datasets after training with synthetic data

Widefield microscopy can provide optical access to multi-millimeter fields of view and thousands of neurons in mammalian brains at video rate. However, tissue scattering and background contamination results in signal deterioration, making the extraction of neuronal activity challenging, laborious and time consuming. Here we present our deep-learning-based widefield neuron finder (DeepWonder), which is trained by simulated functional recordings and effectively works on experimental data to achieve high-fidelity neuronal extraction. Equipped with systematic background contribution priors, DeepWonder conducts neuronal inference with an order-of-magnitude-faster speed and improved accuracy compared with alternative approaches. DeepWonder removes background contaminations and is computationally efficient. Specifically, DeepWonder accomplishes 50-fold signal-to-background ratio enhancement when processing terabytes-scale cortex-wide functional recordings, with over 14,000 neurons extracted in 17 h.

NeuroSeg-II: A deep learning approach for generalized neuron segmentation in two-photon Ca2+ imaging

The development of two-photon microscopy and Ca²⁺ indicators has enabled the recording of multiscale neuronal activities in vivo and thus advanced the understanding of brain functions. However, it is challenging to perform automatic, accurate, and generalized neuron segmentation when processing a large amount of imaging data. Here, we propose a novel deep-learning-based neural network, termed as NeuroSeg-II, to conduct automatic neuron segmentation for in vivo two-photon Ca²⁺ imaging data. This network architecture is based on Mask region-based convolutional neural network (R-CNN) but has enhancements of an attention mechanism and modified feature hierarchy modules. We added an attention mechanism module to focus the computation on neuron regions in imaging data. We also enhanced the feature hierarchy to extract feature information at diverse levels. To incorporate both spatial and temporal information in our data processing, we fused the images from average projection and correlation map extracting the temporal information of active neurons, and the integrated information was expressed as two-dimensional (2D) images. To achieve a generalized neuron segmentation, we conducted a hybrid learning strategy by training our model with imaging data from different labs, including multiscale data with different Ca²⁺ indicators. The results showed that our approach achieved promising segmentation performance across different imaging scales and Ca²⁺ indicators, even including the challenging data of large field-of-view mesoscopic images. By comparing state-of-the-art neuron segmentation methods for two-photon Ca²⁺ imaging data, we showed that our approach achieved the highest accuracy with a publicly available dataset. Thus, NeuroSeg-II enables good segmentation accuracy and a convenient training and testing process.

Review on data analysis methods for mesoscale neural imaging in vivo

Significance: Mesoscale neural imaging in vivo has gained extreme popularity in neuroscience for its capacity of recording large-scale neurons in action. Optical imaging with single-cell resolution and millimeter-level field of view in vivo has been providing an accumulated database of neuron-behavior correspondence. Meanwhile, optical detection of neuron signals is easily contaminated by noises, background, crosstalk, and motion artifacts, while neural-level signal processing and network-level coordinate are extremely complicated, leading to laborious and challenging signal processing demands. The existing data analysis procedure remains unstandardized, which could be daunting to neophytes or neuroscientists without computational background. Aim: We hope to provide a general data analysis pipeline of mesoscale neural imaging shared between imaging modalities and systems. Approach: We divide the pipeline into two main stages. The first stage focuses on extracting high-fidelity neural responses at single-cell level from raw images, including motion registration, image denoising, neuron segmentation, and signal extraction. The second stage focuses on data mining, including neural functional mapping, clustering, and brain-wide network deduction. Results: Here, we introduce the general pipeline of processing the mesoscale neural images. We explain the principles of these procedures and compare different approaches and their application scopes with detailed discussions about the shortcomings and remaining challenges. Conclusions: There are great challenges and opportunities brought by the large-scale mesoscale data, such as the balance between fidelity and efficiency, increasing computational load, and neural network interpretability. We believe that global circuits on single-neuron level will be more extensively explored in the future.

Review of data processing of functional optical microscopy for neuroscience

Functional optical imaging in neuroscience is rapidly growing with the development of optical systems and fluorescence indicators. To realize the potential of these massive spatiotemporal datasets for relating neuronal activity to behavior and stimuli and uncovering local circuits in the brain, accurate automated processing is increasingly essential. We cover recent computational developments in the full data processing pipeline of functional optical microscopy for neuroscience data and discuss ongoing and emerging challenges.

Deep-learning-augmented computational miniature mesoscope

Fluorescence microscopy is essential to study biological structures and dynamics. However, existing systems suffer from a trade-off between field of view (FOV), resolution, and system complexity, and thus cannot fulfill the emerging need for miniaturized platforms providing micron-scale resolution across centimeter-scale FOVs. To overcome this challenge, we developed a computational miniature mesoscope (CM2) that exploits a computational imaging strategy to enable single-shot, 3D high-resolution imaging across a wide FOV in a miniaturized platform. Here, we present CM2 V2, which significantly advances both the hardware and computation. We complement the 3 × 3 microlens array with a hybrid emission filter that improves the imaging contrast by 5×, and design a 3D-printed free-form collimator for the LED illuminator that improves the excitation efficiency by 3×. To enable high-resolution reconstruction across a large volume, we develop an accurate and efficient 3D linear shift-variant (LSV) model to characterize spatially varying aberrations. We then train a multimodule deep learning model called CM2Net, using only the 3D-LSV simulator. We quantify the detection performance and localization accuracy of CM2Net to reconstruct fluorescent emitters under different conditions in simulation. We then show that CM2Net generalizes well to experiments and achieves accurate 3D reconstruction across a ~7-mm FOV and 800-μm depth, and provides ~6-μm lateral and ~25-μm axial resolution. This provides an ~8× better axial resolution and ~1400× faster speed compared to the previous model-based algorithm. We anticipate this simple, low-cost computational miniature imaging system will be useful for many large-scale 3D fluorescence imaging applications.

SmaRT2P: a software for generating and processing smart line recording trajectories for population two-photon calcium imaging

Two-photon fluorescence calcium imaging allows recording the activity of large neural populations with subcellular spatial resolution, but it is typically characterized by low signal-to-noise ratio (SNR) and poor accuracy in detecting single or few action potentials when large number of neurons are imaged. We recently showed that implementing a smart line scanning approach using trajectories that optimally sample the regions of interest increases both the SNR fluorescence signals and the accuracy of single spike detection in population imaging in vivo. However, smart line scanning requires highly specialised software to design recording trajectories, interface with acquisition hardware, and efficiently process acquired data. Furthermore, smart line scanning needs optimized strategies to cope with movement artefacts and neuropil contamination. Here, we develop and validate SmaRT2P, an open-source, user-friendly and easy-to-interface Matlab-based software environment to perform optimized smart line scanning in two-photon calcium imaging experiments. SmaRT2P is designed to interface with popular acquisition software (e.g., ScanImage) and implements novel strategies to detect motion artefacts, estimate neuropil contamination, and minimize their impact on functional signals extracted from neuronal population imaging. SmaRT2P is structured in a modular way to allow flexibility in the processing pipeline, requiring minimal user intervention in parameter setting. The use of SmaRT2P for smart line scanning has the potential to facilitate the functional investigation of large neuronal populations with increased SNR and accuracy in detecting the discharge of single and few action potentials.

Computational Methods for Neuron Segmentation in Two-Photon Calcium Imaging Data: A Survey

Calcium imaging has rapidly become a methodology of choice for real-time in vivo neuron analysis. Its application to large sets of data requires automated tools to annotate and segment cells, allowing scalable image segmentation under reproducible criteria. In this paper, we review and summarize the most recent methods for computational segmentation of calcium imaging. The contributions of the paper are three-fold: we provide an overview of the main algorithms taxonomized in three categories (signal processing, matrix factorization and machine learning-based approaches), we highlight the main advantages and disadvantages of each category and we provide a summary of the performance of the methods that have been tested on public benchmarks (with links to the public code when available).

A fully water coupled oblique light-sheet microscope

Recently developed descanned versions of the oblique light-sheet microscope promise to enable high-frame rate volumetric imaging in a variety of convenient preparations. The efficiency of these microscopes depends on the implementation of the objective coupling that turns the intermediate imaging plane. In this work, we developed a fully immersed coupling strategy between the middle and end objectives of the oblique light-sheet microscope to enable facile alignment and high efficiency coupling. Our design outperformed conventional designs that used only air objectives in resolution and light-collection power. We further demonstrated our design’s ability to capture large fields-of-view when paired with a camera with built-in electronic binning. We simultaneously imaged the forebrain and hindbrain of larval zebrafish and found clusters of activity localized to each region of the brain.

Decontaminate Traces From Fluorescence Calcium Imaging Videos Using Targeted Non-negative Matrix Factorization

Fluorescence microscopy and genetically encoded calcium indicators help understand brain function by recording large-scale in vivo videos in assorted animal models. Extracting the fluorescent transients that represent active periods of individual neurons is a key step when analyzing imaging videos. Non-specific calcium sources and background adjacent to segmented neurons contaminate the neurons' temporal traces with false transients. We developed and characterized a novel method, temporal unmixing of calcium traces (TUnCaT), to quickly and accurately unmix the calcium signals of neighboring neurons and background. Our algorithm used background subtraction to remove the false transients caused by background fluctuations, and then applied targeted non-negative matrix factorization to remove the false transients caused by neighboring calcium sources. TUnCaT was more accurate than existing algorithms when processing multiple experimental and simulated datasets. TUnCaT's speed was faster than or comparable to existing algorithms.

Neurophotonic tools for microscopic measurements and manipulation: status report

Neurophotonics was launched in 2014 coinciding with the launch of the BRAIN Initiative focused on development of technologies for advancement of neuroscience. For the last seven years, Neurophotonics' agenda has been well aligned with this focus on neurotechnologies featuring new optical methods and tools applicable to brain studies. While the BRAIN Initiative 2.0 is pivoting towards applications of these novel tools in the quest to understand the brain, this status report reviews an extensive and diverse toolkit of novel methods to explore brain function that have emerged from the BRAIN Initiative and related large-scale efforts for measurement and manipulation of brain structure and function. Here, we focus on neurophotonic tools mostly applicable to animal studies. A companion report, scheduled to appear later this year, will cover diffuse optical imaging methods applicable to noninvasive human studies. For each domain, we outline the current state-of-the-art of the respective technologies, identify the areas where innovation is needed, and provide an outlook for the future directions.