Reinforcing neuron extraction and spike inference in calcium imaging using deep self-supervised denoising

Neurophotonics beyond the surface: unmasking the brain's complexity exploiting optical scattering

The intricate nature of the brain necessitates the application of advanced probing techniques to comprehensively study and understand its working mechanisms. Neurophotonics offers minimally invasive methods to probe the brain using optics at cellular and even molecular levels. However, multiple challenges persist, especially concerning imaging depth, field of view, speed, and biocompatibility. A major hindrance to solving these challenges in optics is the scattering nature of the brain. This perspective highlights the potential of complex media optics, a specialized area of study focused on light propagation in materials with intricate heterogeneous optical properties, in advancing and improving neuronal readouts for structural imaging and optical recordings of neuronal activity. Key strategies include wavefront shaping techniques and computational imaging and sensing techniques that exploit scattering properties for enhanced performance. We discuss the potential merger of the two fields as well as potential challenges and perspectives toward longer term in vivo applications.

Deep learning based decoding of single local field potential events

How is information processed in the cerebral cortex? In most cases, recorded brain activity is averaged over many (stimulus) repetitions, which erases the fine-structure of the neural signal. However, the brain is obviously a single-trial processor. Thus, we here demonstrate that an unsupervised machine learning approach can be used to extract meaningful information from electro-physiological recordings on a single-trial basis. We use an auto-encoder network to reduce the dimensions of single local field potential (LFP) events to create interpretable clusters of different neural activity patterns. Strikingly, certain LFP shapes correspond to latency differences in different recording channels. Hence, LFP shapes can be used to determine the direction of information flux in the cerebral cortex. Furthermore, after clustering, we decoded the cluster centroids to reverse-engineer the underlying prototypical LFP event shapes. To evaluate our approach, we applied it to both extra-cellular neural recordings in rodents, and intra-cranial EEG recordings in humans. Finally, we find that single channel LFP event shapes during spontaneous activity sample from the realm of possible stimulus evoked event shapes. A finding which so far has only been demonstrated for multi-channel population coding.

An Amygdalar-Vagal-Glandular Circuit Controls the Intestinal Microbiome

Psychological states can regulate intestinal mucosal immunity by altering the gut microbiome. However, the link between the brain and microbiome composition remains elusive. We show that Brunner's glands in the duodenal submucosa couple brain activity to intestinal bacterial homeostasis. Brunner's glands mediated the enrichment of gut probiotic species in response to stimulation of abdominal vagal fibers. Cell-specific ablation of the glands triggered transmissible dysbiosis associated with an immunodeficiency syndrome that led to mortality upon gut infection with pathogens. The syndrome could be largely prevented by oral or intra-intestinal administration of probiotics. In the forebrain, we identified a vagally-mediated, polysynaptic circuit connecting the glands of Brunner to the central nucleus of the amygdala. Intra-vital imaging revealed that excitation of central amygdala neurons activated Brunner's glands and promoted the growth of probiotic populations. Our findings unveil a vagal-glandular neuroimmune circuitry that may be targeted for the modulation of the gut microbiome. The glands of Brunner may be the critical cells that regulate the levels of Lactobacilli species in the intestine.

Long-term intravital subcellular imaging with confocal scanning light-field microscopy

Long-term observation of subcellular dynamics in living organisms is limited by background fluorescence originating from tissue scattering or dense labeling. Existing confocal approaches face an inevitable tradeoff among parallelization, resolution and phototoxicity. Here we present confocal scanning light-field microscopy (csLFM), which integrates axially elongated line-confocal illumination with the rolling shutter in scanning light-field microscopy (sLFM). csLFM enables high-fidelity, high-speed, three-dimensional (3D) imaging at near-diffraction-limit resolution with both optical sectioning and low phototoxicity. By simultaneous 3D excitation and detection, the excitation intensity can be reduced below 1 mW mm-2, with 15-fold higher signal-to-background ratio over sLFM. We imaged subcellular dynamics over 25,000 timeframes in optically challenging environments in different species, such as migrasome delivery in mouse spleen, retractosome generation in mouse liver and 3D voltage imaging in Drosophila. Moreover, csLFM facilitates high-fidelity, large-scale neural recording with reduced crosstalk, leading to high orientation selectivity to visual stimuli, similar to two-photon microscopy, which aids understanding of neural coding mechanisms.

Signal2signal: Pushing the Spatiotemporal Resolution to the Limit by Single Chemical Hyperspectral Imaging

There is growing interest in developing a high-performance self-supervised denoising algorithm for real-time chemical hyperspectral imaging. With a good understanding of the working function of the zero-shot Noise2Noise-based denoising algorithm, we developed a self-supervised Signal2Signal (S2S) algorithm for real-time denoising with a single chemical hyperspectral image. Owing to the accurate distinction and capture of the weak signal from the random fluctuating noise, S2S displays excellent denoising performance, even for the hyperspectral image with a spectral signal-to-noise ratio (SNR) as low as 1.12. Under this condition, both the image clarity and the spatial resolution could be significantly improved and present an almost identical pattern with a spectral SNR of 7.87. The feasibility of real-time denoising during imaging was well demonstrated, and S2S was applied to monitor the photoinduced exfoliation of transition metal dichalcogenide, which is hard to accomplish by confocal Raman spectroscopy. In general, the real-time denoising capability of S2S offers an easy way toward in situ/in vivo/operando research with much improved spatial and temporal resolution. S2S is open-source at https://github.com/3331822w/Signal2signal and will be accessible online at https://ramancloud.xmu.edu.cn/tutorial.

A Large Field-of-view, Single-cell-resolution Two- and Three-Photon Microscope for Deep Imaging

In vivo imaging of large-scale neuron activity plays a pivotal role in unraveling the function of the brain's network. Multiphoton microscopy, a powerful tool for deep-tissue imaging, has received sustained interest in advancing its speed, field of view and imaging depth. However, to avoid thermal damage in scattering biological tissue, field of view decreases exponentially as imaging depth increases. We present a suite of innovations to overcome constraints on the field of view in three-photon microscopy and to perform deep imaging that is inaccessible to two-photon microscopy. These innovations enable us to image neuronal activities in a ~3.5-mm diameter field-of-view at 4 Hz with single-cell resolution and in the deepest cortical layer of mouse brains. We further demonstrate simultaneous large field-of-view two-photon and three-photon imaging, subcortical imaging in the mouse brain, and whole-brain imaging in adult zebrafish. The demonstrated techniques can be integrated into any multiphoton microscope for large-field-of-view and system-level neural circuit research.

Pretraining a foundation model for generalizable fluorescence microscopy-based image restoration

Fluorescence microscopy-based image restoration has received widespread attention in the life sciences and has led to significant progress, benefiting from deep learning technology. However, most current task-specific methods have limited generalizability to different fluorescence microscopy-based image restoration problems. Here, we seek to improve generalizability and explore the potential of applying a pretrained foundation model to fluorescence microscopy-based image restoration. We provide a universal fluorescence microscopy-based image restoration (UniFMIR) model to address different restoration problems, and show that UniFMIR offers higher image restoration precision, better generalization and increased versatility. Demonstrations on five tasks and 14 datasets covering a wide range of microscopy imaging modalities and biological samples demonstrate that the pretrained UniFMIR can effectively transfer knowledge to a specific situation via fine-tuning, uncover clear nanoscale biomolecular structures and facilitate high-quality imaging. This work has the potential to inspire and trigger new research highlights for fluorescence microscopy-based image restoration.

SpiDe-Sr: blind super-resolution network for precise cell segmentation and clustering in spatial proteomics imaging

Spatial proteomics elucidates cellular biochemical changes with unprecedented topological level. Imaging mass cytometry (IMC) is a high-dimensional single-cell resolution platform for targeted spatial proteomics. However, the precision of subsequent clinical analysis is constrained by imaging noise and resolution. Here, we propose SpiDe-Sr, a super-resolution network embedded with a denoising module for IMC spatial resolution enhancement. SpiDe-Sr effectively resists noise and improves resolution by 4 times. We demonstrate SpiDe-Sr respectively with cells, mouse and human tissues, resulting 18.95%/27.27%/21.16% increase in peak signal-to-noise ratio and 15.95%/31.63%/15.52% increase in cell extraction accuracy. We further apply SpiDe-Sr to study the tumor microenvironment of a 20-patient clinical breast cancer cohort with 269,556 single cells, and discover the invasion of Gram-negative bacteria is positively correlated with carcinogenesis markers and negatively correlated with immunological markers. Additionally, SpiDe-Sr is also compatible with fluorescence microscopy imaging, suggesting SpiDe-Sr an alternative tool for microscopy image super-resolution.

Multiscale biochemical mapping of the brain through deep-learning-enhanced high-throughput mass spectrometry

Spatial omics technologies can reveal the molecular intricacy of the brain. While mass spectrometry imaging (MSI) provides spatial localization of compounds, comprehensive biochemical profiling at a brain-wide scale in three dimensions by MSI with single-cell resolution has not been achieved. We demonstrate complementary brain-wide and single-cell biochemical mapping using MEISTER, an integrative experimental and computational mass spectrometry (MS) framework. Our framework integrates a deep-learning-based reconstruction that accelerates high-mass-resolving MS by 15-fold, multimodal registration creating three-dimensional (3D) molecular distributions and a data integration method fitting cell-specific mass spectra to 3D datasets. We imaged detailed lipid profiles in tissues with millions of pixels and in large single-cell populations acquired from the rat brain. We identified region-specific lipid contents and cell-specific localizations of lipids depending on both cell subpopulations and anatomical origins of the cells. Our workflow establishes a blueprint for future development of multiscale technologies for biochemical characterization of the brain.

Multibeam continuous axial scanning two-photon microscopy for in vivo volumetric imaging in mouse brain

This study presents an alternative approach for two-photon volumetric imaging that combines multibeam lateral scanning with continuous axial scanning using a confocal spinning-disk scanner and an electrically focus tunable lens. Using this proposed system, the brain of a living mouse could be imaged at a penetration depth of over 450 μm from the surface. In vivo volumetric Ca2+ imaging at a volume rate of 1.5 Hz within a depth range of 130-200 μm, was segmented with an axial pitch of approximately 5-µm and revealed spontaneous activity of neurons with their 3D positions. This study offers a practical microscope design equipped with compact scanners, a simple control system, and readily adjustable imaging parameters, which is crucial for the widespread adoption of two-photon volumetric imaging.

Model-based correction of rapid thermal confounds in fluorescence neuroimaging of targeted perturbation

Significance

An array of techniques for targeted neuromodulation is emerging, with high potential in brain research and therapy. Calcium imaging or other forms of functional fluorescence imaging are central solutions for monitoring cortical neural responses to targeted neuromodulation, but often are confounded by thermal effects that are inter-mixed with neural responses.

Aim

Here, we develop and demonstrate a method for effectively suppressing fluorescent thermal transients from calcium responses.

Approach

We use high precision phased-array 3 MHz focused ultrasound delivery integrated with fiberscope-based widefield fluorescence to monitor cortex-wide calcium changes. Our approach for detecting the neural activation first takes advantage of the high inter-hemispheric correlation of resting state Ca 2 + dynamics and then removes the ultrasound-induced thermal effect by subtracting its simulated spatio-temporal signature from the processed profile.

Results

The focused 350    μ m -sized ultrasound stimulus triggered rapid localized activation events dominated by transient thermal responses produced by ultrasound. By employing bioheat equation to model the ultrasound heat deposition, we can recover putative neural responses to ultrasound.

Conclusions

The developed method for canceling transient thermal fluorescence quenching could also find applications with optical stimulation techniques to monitor thermal effects and disentangle them from neural responses. This approach may help deepen our understanding of the mechanisms and macroscopic effects of ultrasound neuromodulation, further paving the way for tailoring the stimulation regimes toward specific applications.

Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy

Interrogation of subcellular biological dynamics occurring in a living cell often requires noninvasive imaging of the fragile cell with high spatiotemporal resolution across all three dimensions. It thereby poses big challenges to modern fluorescence microscopy implementations because the limited photon budget in a live-cell imaging task makes the achievable performance of conventional microscopy approaches compromise between their spatial resolution, volumetric imaging speed, and phototoxicity. Here, we incorporate a two-stage view-channel-depth (VCD) deep-learning reconstruction strategy with a Fourier light-field microscope based on diffractive optical element to realize fast 3D super-resolution reconstructions of intracellular dynamics from single diffraction-limited 2D light-filed measurements. This VCD-enabled Fourier light-filed imaging approach (F-VCD), achieves video-rate (50 volumes per second) 3D imaging of intracellular dynamics at a high spatiotemporal resolution of ~180 nm × 180 nm × 400 nm and strong noise-resistant capability, with which light field images with a signal-to-noise ratio (SNR) down to -1.62 dB could be well reconstructed. With this approach, we successfully demonstrate the 4D imaging of intracellular organelle dynamics, e.g., mitochondria fission and fusion, with ~5000 times of observation.

Bio-friendly long-term subcellular dynamic recording by self-supervised image enhancement microscopy

Fluorescence microscopy has become an indispensable tool for revealing the dynamic regulation of cells and organelles. However, stochastic noise inherently restricts optical interrogation quality and exacerbates observation fidelity when balancing the joint demands of high frame rate, long-term recording and low phototoxicity. Here we propose DeepSeMi, a self-supervised-learning-based denoising framework capable of increasing signal-to-noise ratio by over 12 dB across various conditions. With the introduction of newly designed eccentric blind-spot convolution filters, DeepSeMi effectively denoises images with no loss of spatiotemporal resolution. In combination with confocal microscopy, DeepSeMi allows for recording organelle interactions in four colors at high frame rates across tens of thousands of frames, monitoring migrasomes and retractosomes over a half day, and imaging ultra-phototoxicity-sensitive Dictyostelium cells over thousands of frames. Through comprehensive validations across various samples and instruments, we prove DeepSeMi to be a versatile and biocompatible tool for breaking the shot-noise limit.

Spatial redundancy transformer for self-supervised fluorescence image denoising

Fluorescence imaging with high signal-to-noise ratios has become the foundation of accurate visualization and analysis of biological phenomena. However, the inevitable noise poses a formidable challenge to imaging sensitivity. Here we provide the spatial redundancy denoising transformer (SRDTrans) to remove noise from fluorescence images in a self-supervised manner. First, a sampling strategy based on spatial redundancy is proposed to extract adjacent orthogonal training pairs, which eliminates the dependence on high imaging speed. Second, we designed a lightweight spatiotemporal transformer architecture to capture long-range dependencies and high-resolution features at low computational cost. SRDTrans can restore high-frequency information without producing oversmoothed structures and distorted fluorescence traces. Finally, we demonstrate the state-of-the-art denoising performance of SRDTrans on single-molecule localization microscopy and two-photon volumetric calcium imaging. SRDTrans does not contain any assumptions about the imaging process and the sample, thus can be easily extended to various imaging modalities and biological applications.

Live-cell imaging in the deep learning era

Live imaging is a powerful tool, enabling scientists to observe living organisms in real time. In particular, when combined with fluorescence microscopy, live imaging allows the monitoring of cellular components with high sensitivity and specificity. Yet, due to critical challenges (i.e., drift, phototoxicity, dataset size), implementing live imaging and analyzing the resulting datasets is rarely straightforward. Over the past years, the development of bioimage analysis tools, including deep learning, is changing how we perform live imaging. Here we briefly cover important computational methods aiding live imaging and carrying out key tasks such as drift correction, denoising, super-resolution imaging, artificial labeling, tracking, and time series analysis. We also cover recent advances in self-driving microscopy.

Integrative Multiscale Biochemical Mapping of the Brain via Deep-Learning-Enhanced High-Throughput Mass Spectrometry

Elucidating the spatial-biochemical organization of the brain across different scales produces invaluable insight into the molecular intricacy of the brain. While mass spectrometry imaging (MSI) provides spatial localization of compounds, comprehensive chemical profiling at a brain-wide scale in three dimensions by MSI with single-cell resolution has not been achieved. We demonstrate complementary brain-wide and single-cell biochemical mapping via MEISTER, an integrative experimental and computational mass spectrometry framework. MEISTER integrates a deep-learning-based reconstruction that accelerates high-mass-resolving MS by 15-fold, multimodal registration creating 3D molecular distributions, and a data integration method fitting cell-specific mass spectra to 3D data sets. We imaged detailed lipid profiles in tissues with data sets containing millions of pixels, and in large single-cell populations acquired from the rat brain. We identified region-specific lipid contents, and cell-specific localizations of lipids depending on both cell subpopulations and anatomical origins of the cells. Our workflow establishes a blueprint for future developments of multiscale technologies for biochemical characterization of the brain.

Advances in cellular resolution microscopy for brain imaging in rats

Rats are used in neuroscience research because of their physiological similarities with humans and accessibility as model organisms, trainability, and behavioral repertoire. In particular, rats perform a wide range of sophisticated social, cognitive, motor, and learning behaviors within the contexts of both naturalistic and laboratory environments. Further progress in neuroscience can be facilitated by using advanced imaging methods to measure the complex neural and physiological processes during behavior in rats. However, compared with the mouse, the rat nervous system offers a set of challenges, such as larger brain size, decreased neuron density, and difficulty with head restraint. Here, we review recent advances in in vivo imaging techniques in rats with a special focus on open-source solutions for calcium imaging. Finally, we provide suggestions for both users and developers of in vivo imaging systems for rats.

Addressing annotation and data scarcity when designing machine learning strategies for neurophotonics

Machine learning has revolutionized the way data are processed, allowing information to be extracted in a fraction of the time it would take an expert. In the field of neurophotonics, machine learning approaches are used to automatically detect and classify features of interest in complex images. One of the key challenges in applying machine learning methods to the field of neurophotonics is the scarcity of available data and the complexity associated with labeling them, which can limit the performance of data-driven algorithms. We present an overview of various strategies, such as weakly supervised learning, active learning, and domain adaptation that can be used to address the problem of labeled data scarcity in neurophotonics. We provide a comprehensive overview of the strengths and limitations of each approach and discuss their potential applications to bioimaging datasets. In addition, we highlight how different strategies can be combined to increase model performance on those datasets. The approaches we describe can help to improve the accessibility of machine learning-based analysis with limited number of annotated images for training and can enable researchers to extract more meaningful insights from small datasets.

Enhancing Total Optical Throughput of Microscopy with Deep Learning for Intravital Observation

The significance of performing large-depth dynamic microscopic imaging in vivo for life science research cannot be overstated. However, the optical throughput of the microscope limits the available information per unit of time, i.e., it is difficult to obtain both high spatial and temporal resolution at once. Here, a method is proposed to construct a kind of intravital microscopy with high optical throughput, by making near-infrared-II (NIR-II, 900-1880 nm) wide-field fluorescence microscopy learn from two-photon fluorescence microscopy based on a scale-recurrent network. Using this upgraded NIR-II fluorescence microscope, vessels in the opaque brain of a rodent are reconstructed three-dimensionally. Five-fold axial and thirteen-fold lateral resolution improvements are achieved without sacrificing temporal resolution and light utilization. Also, tiny cerebral vessel dilatations in early acute respiratory failure mice are observed, with this high optical throughput NIR-II microscope at an imaging speed of 30 fps.

Self super-resolution of optical coherence tomography images based on deep learning

As a medical imaging modality, many researches have been devoted to improving the resolution of optical coherence tomography (OCT). We developed a deep-learning based OCT self super-resolution (OCT-SSR) pipeline to improve the axial resolution of OCT images based on the high-resolution and low-resolution spectral data collected by the OCT system. In this pipeline, the enhanced super-resolution asymmetric generative adversarial networks were built to improve the network outputs without increasing the complexity. The feasibility and effectiveness of the approach were demonstrated by experimental results on the images of the biological samples collected by the home-made spectral-domain OCT and swept-source OCT systems. More importantly, we found the sidelobes in the original images can be obviously suppressed while improving the resolution based on the OCT-SSR method, which can help to reduce pseudo-signal in OCT imaging when non-Gaussian spectra light source is used. We believe that the OCT-SSR method has broad prospects in breaking the limitation of the source bandwidth on the axial resolution of the OCT system.

Molecular profiling of sponge deflation reveals an ancient relaxant-inflammatory response

A hallmark of animals is the coordination of whole-body movement. Neurons and muscles are central to this, yet coordinated movements also exist in sponges that lack these cell types. Sponges are sessile animals with a complex canal system for filter-feeding. They undergo whole-body movements resembling "contractions" that lead to canal closure and water expulsion. Here, we combine 3D optical coherence microscopy, pharmacology, and functional proteomics to elucidate anatomy, molecular physiology, and control of these movements. We find them driven by the relaxation of actomyosin stress fibers in epithelial canal cells, which leads to whole-body deflation via collapse of the incurrent and expansion of the excurrent system, controlled by an Akt/NO/PKG/A pathway. A concomitant increase in reactive oxygen species and secretion of proteinases and cytokines indicate an inflammation-like state reminiscent of vascular endothelial cells experiencing oscillatory shear stress. This suggests an ancient relaxant-inflammatory response of perturbed fluid-carrying systems in animals.

High-speed low-light in vivo two-photon voltage imaging of large neuronal populations

Monitoring spiking activity across large neuronal populations at behaviorally relevant timescales is critical for understanding neural circuit function. Unlike calcium imaging, voltage imaging requires kilohertz sampling rates that reduce fluorescence detection to near shot-noise levels. High-photon flux excitation can overcome photon-limited shot noise, but photobleaching and photodamage restrict the number and duration of simultaneously imaged neurons. We investigated an alternative approach aimed at low two-photon flux, which is voltage imaging below the shot-noise limit. This framework involved developing positive-going voltage indicators with improved spike detection (SpikeyGi and SpikeyGi2); a two-photon microscope ('SMURF') for kilohertz frame rate imaging across a 0.4 mm × 0.4 mm field of view; and a self-supervised denoising algorithm (DeepVID) for inferring fluorescence from shot-noise-limited signals. Through these combined advances, we achieved simultaneous high-speed deep-tissue imaging of more than 100 densely labeled neurons over 1 hour in awake behaving mice. This demonstrates a scalable approach for voltage imaging across increasing neuronal populations.

Improved Workflow for Analysis of Vascular Myocyte Time-Series and Line-Scan Ca2+ Imaging Datasets

Intracellular Ca2+ signals are key for the regulation of cellular processes ranging from myocyte contraction, hormonal secretion, neural transmission, cellular metabolism, transcriptional regulation, and cell proliferation. Measurement of cellular Ca2+ is routinely performed using fluorescence microscopy with biological indicators. Analysis of deterministic signals is reasonably straightforward as relevant data can be discriminated based on the timing of cellular responses. However, analysis of stochastic, slower oscillatory events, as well as rapid subcellular Ca2+ responses, takes considerable time and effort which often includes visual analysis by trained investigators, especially when studying signals arising from cells embedded in complex tissues. The purpose of the current study was to determine if full-frame time-series and line-scan image analysis workflow of Fluo-4 generated Ca2+ fluorescence data from vascular myocytes could be automated without introducing errors. This evaluation was addressed by re-analyzing a published "gold standard" full-frame time-series dataset through visual analysis of Ca2+ signals from recordings made in pulmonary arterial myocytes of en face arterial preparations. We applied a combination of data driven and statistical approaches with comparisons to our published data to assess the fidelity of the various approaches. Regions of interest with Ca2+ oscillations were detected automatically post hoc using the LCPro plug-in for ImageJ. Oscillatory signals were separated based on event durations between 4 and 40 s. These data were filtered based on cutoffs obtained from multiple methods and compared to the published manually curated "gold standard" dataset. Subcellular focal and rapid Ca2+ "spark" events from line-scan recordings were examined using SparkLab 5.8, which is a custom automated detection and analysis program. After filtering, the number of true positives, false positives, and false negatives were calculated through comparisons to visually derived "gold standard" datasets. Positive predictive value, sensitivity, and false discovery rates were calculated. There were very few significant differences between the automated and manually curated results with respect to quality of the oscillatory and Ca2+ spark events, and there were no systematic biases in the data curation or filtering techniques. The lack of statistical difference in event quality between manual data curation and statistically derived critical cutoff techniques leads us to believe that automated analysis techniques can be reliably used to analyze spatial and temporal aspects to Ca2+ imaging data, which will improve experiment workflow.

Two-photon synthetic aperture microscopy for minimally invasive fast 3D imaging of native subcellular behaviors in deep tissue

Holistic understanding of physio-pathological processes requires noninvasive 3D imaging in deep tissue across multiple spatial and temporal scales to link diverse transient subcellular behaviors with long-term physiogenesis. Despite broad applications of two-photon microscopy (TPM), there remains an inevitable tradeoff among spatiotemporal resolution, imaging volumes, and durations due to the point-scanning scheme, accumulated phototoxicity, and optical aberrations. Here, we harnessed the concept of synthetic aperture radar in TPM to achieve aberration-corrected 3D imaging of subcellular dynamics at a millisecond scale for over 100,000 large volumes in deep tissue, with three orders of magnitude reduction in photobleaching. With its advantages, we identified direct intercellular communications through migrasome generation following traumatic brain injury, visualized the formation process of germinal center in the mouse lymph node, and characterized heterogeneous cellular states in the mouse visual cortex, opening up a horizon for intravital imaging to understand the organizations and functions of biological systems at a holistic level.

Rapid detection of neurons in widefield calcium imaging datasets after training with synthetic data

Widefield microscopy can provide optical access to multi-millimeter fields of view and thousands of neurons in mammalian brains at video rate. However, tissue scattering and background contamination results in signal deterioration, making the extraction of neuronal activity challenging, laborious and time consuming. Here we present our deep-learning-based widefield neuron finder (DeepWonder), which is trained by simulated functional recordings and effectively works on experimental data to achieve high-fidelity neuronal extraction. Equipped with systematic background contribution priors, DeepWonder conducts neuronal inference with an order-of-magnitude-faster speed and improved accuracy compared with alternative approaches. DeepWonder removes background contaminations and is computationally efficient. Specifically, DeepWonder accomplishes 50-fold signal-to-background ratio enhancement when processing terabytes-scale cortex-wide functional recordings, with over 14,000 neurons extracted in 17 h.

Real-time denoising enables high-sensitivity fluorescence time-lapse imaging beyond the shot-noise limit

A fundamental challenge in fluorescence microscopy is the photon shot noise arising from the inevitable stochasticity of photon detection. Noise increases measurement uncertainty and limits imaging resolution, speed and sensitivity. To achieve high-sensitivity fluorescence imaging beyond the shot-noise limit, we present DeepCAD-RT, a self-supervised deep learning method for real-time noise suppression. Based on our previous framework DeepCAD, we reduced the number of network parameters by 94%, memory consumption by 27-fold and processing time by a factor of 20, allowing real-time processing on a two-photon microscope. A high imaging signal-to-noise ratio can be acquired with tenfold fewer photons than in standard imaging approaches. We demonstrate the utility of DeepCAD-RT in a series of photon-limited experiments, including in vivo calcium imaging of mice, zebrafish larva and fruit flies, recording of three-dimensional (3D) migration of neutrophils after acute brain injury and imaging of 3D dynamics of cortical ATP release. DeepCAD-RT will facilitate the morphological and functional interrogation of biological dynamics with a minimal photon budget.

Genetically encoded fluorescent sensors for imaging neuronal dynamics in vivo

The brain relies on many forms of dynamic activities in individual neurons, from synaptic transmission to electrical activity and intracellular signaling events. Monitoring these neuronal activities with high spatiotemporal resolution in the context of animal behavior is a necessary step to achieve a mechanistic understanding of brain function. With the rapid development and dissemination of highly optimized genetically encoded fluorescent sensors, a growing number of brain activities can now be visualized in vivo. To date, cellular calcium imaging, which has been largely used as a proxy for electrical activity, has become a mainstay in systems neuroscience. While challenges remain, voltage imaging of neural populations is now possible. In addition, it is becoming increasingly practical to image over half a dozen neurotransmitters, as well as certain intracellular signaling and metabolic activities. These new capabilities enable neuroscientists to test previously unattainable hypotheses and questions. This review summarizes recent progress in the development and delivery of genetically encoded fluorescent sensors, and highlights example applications in the context of in vivo imaging.

Selfrec-Net: self-supervised deep learning approach for the reconstruction of Cherenkov-excited luminescence scanned tomography

As an emerging imaging technique, Cherenkov-excited luminescence scanned tomography (CELST) can recover a high-resolution 3D distribution of quantum emission fields within tissue using X-ray excitation for deep penetrance. However, its reconstruction is an ill-posed and under-conditioned inverse problem because of the diffuse optical emission signal. Deep learning based image reconstruction has shown very good potential for solving these types of problems, however they suffer from a lack of ground-truth image data to confirm when used with experimental data. To overcome this, a self-supervised network cascaded by a 3D reconstruction network and the forward model, termed Selfrec-Net, was proposed to perform CELST reconstruction. Under this framework, the boundary measurements are input to the network to reconstruct the distribution of the quantum field and the predicted measurements are subsequently obtained by feeding the reconstructed result to the forward model. The network was trained by minimizing the loss between the input measurements and the predicted measurements rather than the reconstructed distributions and the corresponding ground truths. Comparative experiments were carried out on both numerical simulations and physical phantoms. For singular luminescent targets, the results demonstrate the effectiveness and robustness of the proposed network, and comparable performance can be attained to a state-of-the-art deep supervised learning algorithm, where the accuracy of the emission yield and localization of the objects was far superior to iterative reconstruction methods. Reconstruction of multiple objects is still reasonable with high localization accuracy, although with limits to the emission yield accuracy as the distribution becomes more complex. Overall though the reconstruction of Selfrec-Net provides a self-supervised way to recover the location and emission yield of molecular distributions in murine model tissues.

NeuroSeg-II: A deep learning approach for generalized neuron segmentation in two-photon Ca2+ imaging

The development of two-photon microscopy and Ca²⁺ indicators has enabled the recording of multiscale neuronal activities in vivo and thus advanced the understanding of brain functions. However, it is challenging to perform automatic, accurate, and generalized neuron segmentation when processing a large amount of imaging data. Here, we propose a novel deep-learning-based neural network, termed as NeuroSeg-II, to conduct automatic neuron segmentation for in vivo two-photon Ca²⁺ imaging data. This network architecture is based on Mask region-based convolutional neural network (R-CNN) but has enhancements of an attention mechanism and modified feature hierarchy modules. We added an attention mechanism module to focus the computation on neuron regions in imaging data. We also enhanced the feature hierarchy to extract feature information at diverse levels. To incorporate both spatial and temporal information in our data processing, we fused the images from average projection and correlation map extracting the temporal information of active neurons, and the integrated information was expressed as two-dimensional (2D) images. To achieve a generalized neuron segmentation, we conducted a hybrid learning strategy by training our model with imaging data from different labs, including multiscale data with different Ca²⁺ indicators. The results showed that our approach achieved promising segmentation performance across different imaging scales and Ca²⁺ indicators, even including the challenging data of large field-of-view mesoscopic images. By comparing state-of-the-art neuron segmentation methods for two-photon Ca²⁺ imaging data, we showed that our approach achieved the highest accuracy with a publicly available dataset. Thus, NeuroSeg-II enables good segmentation accuracy and a convenient training and testing process.

Advances in Optical Tools to Study Taste Sensation

Taste sensation is the process of converting chemical identities in food into a neural code of the brain. Taste information is initially formed in the taste buds on the tongue, travels through the afferent gustatory nerves to the sensory ganglion neurons, and finally reaches the multiple taste centers of the brain. In the taste field, optical tools to observe cellular-level functions play a pivotal role in understanding how taste information is processed along a pathway. In this review, we introduce recent advances in the optical tools used to study the taste transduction pathways.

Deep Learning-Powered Bessel-Beam Multiparametric Photoacoustic Microscopy

Enabling simultaneous and high-resolution quantification of the total concentration of hemoglobin ( [Formula: see text]), oxygen saturation of hemoglobin (sO2), and cerebral blood flow (CBF), multi-parametric photoacoustic microscopy (PAM) has emerged as a promising tool for functional and metabolic imaging of the live mouse brain. However, due to the limited depth of focus imposed by the Gaussian-beam excitation, the quantitative measurements become inaccurate when the imaging object is out of focus. To address this problem, we have developed a hardware-software combined approach by integrating Bessel-beam excitation and conditional generative adversarial network (cGAN)-based deep learning. Side-by-side comparison of the new cGAN-powered Bessel-beam multi-parametric PAM against the conventional Gaussian-beam multi-parametric PAM shows that the new system enables high-resolution, quantitative imaging of [Formula: see text], sO2, and CBF over a depth range of [Formula: see text] in the live mouse brain, with errors 13-58 times lower than those of the conventional system. Better fulfilling the rigid requirement of light focusing for accurate hemodynamic measurements, the deep learning-powered Bessel-beam multi-parametric PAM may find applications in large-field functional recording across the uneven brain surface and beyond (e.g., tumor imaging).

Self-supervised learning enables 3D digital subtraction angiography reconstruction from ultra-sparse 2D projection views: A multicenter study

3D digital subtraction angiography (DSA) reconstruction from rotational 2D projection X-ray angiography is an important basis for diagnosis and treatment of intracranial aneurysms (IAs). The gold standard requires approximately 133 different projection views for 3D reconstruction. A method to significantly reduce the radiation dosage while ensuring the reconstruction quality is yet to be developed. We propose a self-supervised learning method to realize 3D-DSA reconstruction using ultra-sparse 2D projections. 202 cases (100 from one hospital for training and testing, 102 from two other hospitals for external validation) suspected to be suffering from IAs were conducted to analyze the reconstructed images. Two radiologists scored the reconstructed images from internal and external datasets using eight projections and identified all 82 lesions with high diagnostic confidence. The radiation dosages are approximately 1/16.7 compared with the gold standard method. Our proposed method can help develop a revolutionary 3D-DSA reconstruction method for use in clinic.

Review on data analysis methods for mesoscale neural imaging in vivo

Significance: Mesoscale neural imaging in vivo has gained extreme popularity in neuroscience for its capacity of recording large-scale neurons in action. Optical imaging with single-cell resolution and millimeter-level field of view in vivo has been providing an accumulated database of neuron-behavior correspondence. Meanwhile, optical detection of neuron signals is easily contaminated by noises, background, crosstalk, and motion artifacts, while neural-level signal processing and network-level coordinate are extremely complicated, leading to laborious and challenging signal processing demands. The existing data analysis procedure remains unstandardized, which could be daunting to neophytes or neuroscientists without computational background. Aim: We hope to provide a general data analysis pipeline of mesoscale neural imaging shared between imaging modalities and systems. Approach: We divide the pipeline into two main stages. The first stage focuses on extracting high-fidelity neural responses at single-cell level from raw images, including motion registration, image denoising, neuron segmentation, and signal extraction. The second stage focuses on data mining, including neural functional mapping, clustering, and brain-wide network deduction. Results: Here, we introduce the general pipeline of processing the mesoscale neural images. We explain the principles of these procedures and compare different approaches and their application scopes with detailed discussions about the shortcomings and remaining challenges. Conclusions: There are great challenges and opportunities brought by the large-scale mesoscale data, such as the balance between fidelity and efficiency, increasing computational load, and neural network interpretability. We believe that global circuits on single-neuron level will be more extensively explored in the future.

Review of data processing of functional optical microscopy for neuroscience

Functional optical imaging in neuroscience is rapidly growing with the development of optical systems and fluorescence indicators. To realize the potential of these massive spatiotemporal datasets for relating neuronal activity to behavior and stimuli and uncovering local circuits in the brain, accurate automated processing is increasingly essential. We cover recent computational developments in the full data processing pipeline of functional optical microscopy for neuroscience data and discuss ongoing and emerging challenges.

Three-dimensional fluorescence microscopy through virtual refocusing using a recursive light propagation network

Three-dimensional fluorescence microscopy has an intrinsic performance limit set by the number of photons that can be collected from the sample in a given time interval. Here, we extend our earlier work - a recursive light propagation network (RLP-Net) - which is a computational microscopy technique that overcomes such limitations through virtual refocusing that enables volume reconstruction from two adjacent 2-D wide-field fluorescence images. RLP-Net employs a recursive inference scheme in which the network progressively predicts the subsequent planes along the axial direction. This recursive inference scheme reflects that the law of physics for the light propagation remains spatially invariant and therefore a fixed function (i.e., a neural network) for a short distance light propagation can be recursively applied for a longer distance light propagation. In addition, we employ a self-supervised denoising method to enable accurate virtual light propagation over a long distance. We demonstrate the capability of our method through high-speed volumetric imaging of neuronal activity of a live zebrafish brain. The source code used in the paper is available at https://github.com/NICALab/rlpnet.

Fast, efficient, and accurate neuro-imaging denoising via supervised deep learning

Volumetric functional imaging is widely used for recording neuron activities in vivo, but there exist tradeoffs between the quality of the extracted calcium traces, imaging speed, and laser power. While deep-learning methods have recently been applied to denoise images, their applications to downstream analyses, such as recovering high-SNR calcium traces, have been limited. Further, these methods require temporally-sequential pre-registered data acquired at ultrafast rates. Here, we demonstrate a supervised deep-denoising method to circumvent these tradeoffs for several applications, including whole-brain imaging, large-field-of-view imaging in freely moving animals, and recovering complex neurite structures in C. elegans. Our framework has 30× smaller memory footprint, and is fast in training and inference (50–70 ms); it is highly accurate and generalizable, and further, trained with only small, non-temporally-sequential, independently-acquired training datasets (∼500 pairs of images). We envision that the framework will enable faster and long-term imaging experiments necessary to study neuronal mechanisms of many behaviors.

Volumetric functional imaging is widely used for recording neuron activities in vivo for many experimental organisms. Here the authors report supervised deep-denoising methods for improved whole-brain imaging, large field-of-view imaging in freely moving animals, and recovering complex neurite structures in C. elegans.

An interactive time series image analysis software for dendritic spines

Live fluorescence imaging has demonstrated the dynamic nature of dendritic spines, with changes in shape occurring both during development and in response to activity. The structure of a dendritic spine correlates with its functional efficacy. Learning and memory studies have shown that a great deal of the information stored by a neuron is contained in the synapses. High precision tracking of synaptic structures can give hints about the dynamic nature of memory and help us understand how memories evolve both in biological and artificial neural networks. Experiments that aim to investigate the dynamics behind the structural changes of dendritic spines require the collection and analysis of large time-series datasets. In this paper, we present an open-source software called SpineS for automatic longitudinal structural analysis of dendritic spines with additional features for manual intervention to ensure optimal analysis. We have tested the algorithm on in-vitro, in-vivo, and simulated datasets to demonstrate its performance in a wide range of possible experimental scenarios.

An inter-organ neural circuit for appetite suppression

Glucagon-like peptide-1 (GLP-1) is a signal peptide released from enteroendocrine cells of the lower intestine. GLP-1 exerts anorectic and antimotility actions that protect the body against nutrient malabsorption. However, little is known about how intestinal GLP-1 affects distant organs despite rapid enzymatic inactivation. We show that intestinal GLP-1 inhibits gastric emptying and eating via intestinofugal neurons, a subclass of myenteric neurons that project to abdominal sympathetic ganglia. Remarkably, cell-specific ablation of intestinofugal neurons eliminated intestinal GLP-1 effects, and their chemical activation functioned as a GLP-1 mimetic. GLP-1 sensing by intestinofugal neurons then engaged a sympatho-gastro-spinal-reticular-hypothalamic pathway that links abnormal stomach distension to craniofacial programs for food rejection. Within this pathway, cell-specific activation of discrete neuronal populations caused systemic GLP-1-like effects. These molecularly identified, delimited enteric circuits may be targeted to ameliorate the abdominal bloating and loss of appetite typical of gastric motility disorders.

Deep learning based domain adaptation for mitochondria segmentation on EM volumes

BACKGROUND AND OBJECTIVE

Accurate segmentation of electron microscopy (EM) volumes of the brain is essential to characterize neuronal structures at a cell or organelle level. While supervised deep learning methods have led to major breakthroughs in that direction during the past years, they usually require large amounts of annotated data to be trained, and perform poorly on other data acquired under similar experimental and imaging conditions. This is a problem known as domain adaptation, since models that learned from a sample distribution (or source domain) struggle to maintain their performance on samples extracted from a different distribution or target domain. In this work, we address the complex case of deep learning based domain adaptation for mitochondria segmentation across EM datasets from different tissues and species.

METHODS

We present three unsupervised domain adaptation strategies to improve mitochondria segmentation in the target domain based on (1) state-of-the-art style transfer between images of both domains; (2) self-supervised learning to pre-train a model using unlabeled source and target images, and then fine-tune it only with the source labels; and (3) multi-task neural network architectures trained end-to-end with both labeled and unlabeled images. Additionally, to ensure good generalization in our models, we propose a new training stopping criterion based on morphological priors obtained exclusively in the source domain. The code and its documentation are publicly available at https://github.com/danifranco/EM_domain_adaptation.

RESULTS

We carried out all possible cross-dataset experiments using three publicly available EM datasets. We evaluated our proposed strategies and those of others based on the mitochondria semantic labels predicted on the target datasets.

CONCLUSIONS

The methods introduced here outperform the baseline methods and compare favorably to the state of the art. In the absence of validation labels, monitoring our proposed morphology-based metric is an intuitive and effective way to stop the training process and select in average optimal models.

Imaging in focus: An introduction to denoising bioimages in the era of deep learning

Fluorescence microscopy enables the direct observation of previously hidden dynamic processes of life, allowing profound insights into mechanisms of health and disease. However, imaging of live samples is fundamentally limited by the toxicity of the illuminating light and images are often acquired using low light conditions. As a consequence, images can become very noisy which severely complicates their interpretation. In recent years, deep learning (DL) has emerged as a very successful approach to remove this noise while retaining the useful signal. Unlike classical algorithms which use well-defined mathematical functions to remove noise, DL methods learn to denoise from example data, providing a powerful content-aware approach. In this review, we first describe the different types of noise that typically corrupt fluorescence microscopy images and introduce the denoising task. We then present the main DL-based denoising methods and their relative advantages and disadvantages. We aim to provide insights into how DL-based denoising methods operate and help users choose the most appropriate tools for their applications.

Calcium imaging analysis - how far have we come?

Techniques for calcium imaging were first demonstrated in the mid-1970s, whilst tools to analyse these markers of cellular activity are still being developed and improved today. For image analysis, custom tools were developed within labs and until relatively recently, software packages were not widely available between researchers. We will discuss some of the most popular methods for calcium imaging analysis that are now widely available and describe why these protocols are so effective. We will also describe some of the newest innovations in the field that are likely to benefit researchers, particularly as calcium imaging is often an inherently low signal-to-noise method. Although calcium imaging analysis has seen recent advances, particularly following the rise of machine learning, we will end by highlighting the outstanding requirements and questions that hinder further progress and pose the question of how far we have come in the past sixty years and what can be expected for future development in the field.