Rapid image deconvolution and multiview fusion for optical microscopy

Image processing tools for petabyte-scale light sheet microscopy data

Light sheet microscopy is a powerful technique for high-speed three-dimensional imaging of subcellular dynamics and large biological specimens. However, it often generates datasets ranging from hundreds of gigabytes to petabytes in size for a single experiment. Conventional computational tools process such images far slower than the time to acquire them and often fail outright due to memory limitations. To address these challenges, we present PetaKit5D, a scalable software solution for efficient petabyte-scale light sheet image processing. This software incorporates a suite of commonly used processing tools that are optimized for memory and performance. Notable advancements include rapid image readers and writers, fast and memory-efficient geometric transformations, high-performance Richardson-Lucy deconvolution and scalable Zarr-based stitching. These features outperform state-of-the-art methods by over one order of magnitude, enabling the processing of petabyte-scale image data at the full teravoxel rates of modern imaging cameras. The software opens new avenues for biological discoveries through large-scale imaging experiments.

Artificial Intelligence Tools in Pediatric Urology: A Comprehensive Review of Recent Advances

Artificial intelligence (AI) is providing novel answers to long-standing clinical problems, and it is quickly changing pediatric urology. This thorough analysis focuses on current developments in AI technologies that improve pediatric urology diagnosis, treatment planning, and surgery results. Deep learning algorithms help detect problems with previously unheard-of precision in disorders including hydronephrosis, pyeloplasty, and vesicoureteral reflux, where AI-powered prediction models have demonstrated promising outcomes in boosting diagnostic accuracy. AI-enhanced image processing methods have significantly improved the quality and interpretation of medical images. Examples of these methods are deep-learning-based segmentation and contrast limited adaptive histogram equalization (CLAHE). These methods guarantee higher precision in the identification and classification of pediatric urological disorders, and AI-driven ground truth construction approaches aid in the standardization of and improvement in training data, resulting in more resilient and consistent segmentation models. AI is being used for surgical support as well. AI-assisted navigation devices help with difficult operations like pyeloplasty by decreasing complications and increasing surgical accuracy. AI also helps with long-term patient monitoring, predictive analytics, and customized treatment strategies, all of which improve results for younger patients. However, there are practical, ethical, and legal issues with AI integration in pediatric urology that need to be carefully navigated. To close knowledge gaps, more investigation is required, especially in the areas of AI-driven surgical methods and standardized ground truth datasets for pediatric radiologic image segmentation. In the end, AI has the potential to completely transform pediatric urology by enhancing patient care, increasing the effectiveness of treatments, and spurring more advancements in this exciting area.

Calculating point spread functions: methods, pitfalls, and solutions

The knowledge of the exact structure of the optical system point spread function (PSF) enables a high-quality image reconstruction in fluorescence microscopy. Accurate PSF models account for the vector nature of light and the phase and amplitude modifications. Most existing real-space-based PSF models fall into a sampling pitfall near the center position, yielding to the violation of energy conservation. In this work, we present a novel, to the best of our knowledge, Fourier-based techniques for computing vector PSF and compare them to the state-of-the-art. Our methods are shown to satisfy the physical condition of the imaging process. They are reproducible, computationally efficient, easy to implement, and easy to modify to represent various imaging modalities.

Expansion-assisted selective plane illumination microscopy for nanoscale imaging of centimeter-scale tissues

Recent advances in tissue processing, labeling, and fluorescence microscopy are providing unprecedented views of the structure of cells and tissues at sub-diffraction resolutions and near single molecule sensitivity, driving discoveries in diverse fields of biology, including neuroscience. Biological tissue is organized over scales of nanometers to centimeters. Harnessing molecular imaging across intact, three-dimensional samples on this scale requires new types of microscopes with larger fields of view and working distance, as well as higher throughput. We present a new expansion-assisted selective plane illumination microscope (ExA-SPIM) with aberration-free 1×1×3 μm optical resolution over a large field of view (10.6×8.0 mm 2 ) and working distance (35 mm) at speeds up to 946 megavoxels/sec. Combined with new tissue clearing and expansion methods, the microscope allows imaging centimeter-scale samples with 250×250×750 nm optical resolution (4× expansion), including entire mouse brains, with high contrast and without sectioning. We illustrate ExA-SPIM by reconstructing individual neurons across the mouse brain, imaging cortico-spinal neurons in the macaque motor cortex, and visualizing axons in human white matter.

Deep learning-based aberration compensation improves contrast and resolution in fluorescence microscopy

Optical aberrations hinder fluorescence microscopy of thick samples, reducing image signal, contrast, and resolution. Here we introduce a deep learning-based strategy for aberration compensation, improving image quality without slowing image acquisition, applying additional dose, or introducing more optics into the imaging path. Our method (i) introduces synthetic aberrations to images acquired on the shallow side of image stacks, making them resemble those acquired deeper into the volume and (ii) trains neural networks to reverse the effect of these aberrations. We use simulations and experiments to show that applying the trained 'de-aberration' networks outperforms alternative methods, providing restoration on par with adaptive optics techniques; and subsequently apply the networks to diverse datasets captured with confocal, light-sheet, multi-photon, and super-resolution microscopy. In all cases, the improved quality of the restored data facilitates qualitative image inspection and improves downstream image quantitation, including orientational analysis of blood vessels in mouse tissue and improved membrane and nuclear segmentation in C. elegans embryos.

Intelligent Beam Optimization for Light-Sheet Fluorescence Microscopy through Deep Learning

Light-sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times, enabling high-resolution 3-dimensional imaging of large tissue-cleared samples. Inherent to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, which only illuminates a thin section of the sample. Therefore, substantial efforts are dedicated to identifying slender, nondiffracting beam profiles that yield uniform and high-contrast images. An ongoing debate concerns the identification of optimal illumination beams for different samples: Gaussian, Bessel, Airy patterns, and/or others. However, comparisons among different beam profiles are challenging as their optimization objectives are often different. Given that our large imaging datasets (approximately 0.5 TB of images per sample) are already analyzed using deep learning models, we envisioned a different approach to the problem by designing an illumination beam tailored to boost the performance of the deep learning model. We hypothesized that integrating the physical LSFM illumination model (after passing it through a variable phase mask) into the training of a cell detection network would achieve this goal. Here, we report that joint optimization continuously updates the phase mask and results in improved image quality for better cell detection. The efficacy of our method is demonstrated through both simulations and experiments that reveal substantial enhancements in imaging quality compared to the traditional Gaussian light sheet. We discuss how designing microscopy systems through a computational approach provides novel insights for advancing optical design that relies on deep learning models for the analysis of imaging datasets.

Deconwolf enables high-performance deconvolution of widefield fluorescence microscopy images

Microscopy-based spatially resolved omic methods are transforming the life sciences. However, these methods rely on high numerical aperture objectives and cannot resolve crowded molecular targets, limiting the amount of extractable biological information. To overcome these limitations, here we develop Deconwolf, an open-source, user-friendly software for high-performance deconvolution of widefield fluorescence microscopy images, which efficiently runs on laptop computers. Deconwolf enables accurate quantification of crowded diffraction limited fluorescence dots in DNA and RNA fluorescence in situ hybridization images and allows robust detection of individual transcripts in tissue sections imaged with ×20 air objectives. Deconvolution of in situ spatial transcriptomics images with Deconwolf increased the number of transcripts identified more than threefold, while the application of Deconwolf to images obtained by fluorescence in situ sequencing of barcoded Oligopaint probes drastically improved chromosome tracing. Deconwolf greatly facilitates the use of deconvolution in many bioimaging applications.

Universal inverse modeling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

Live-cell single-molecule fluorescence microscopy for protruding organelles reveals regulatory mechanisms of MYO7A-driven cargo transport in stereocilia of inner ear hair cells

Stereocilia are unidirectional F-actin-based cylindrical protrusions on the apical surface of inner ear hair cells and function as biological mechanosensors of sound and acceleration. Development of functional stereocilia requires motor activities of unconventional myosins to transport proteins necessary for elongating the F-actin cores and to assemble the mechanoelectrical transduction (MET) channel complex. However, how each myosin localizes in stereocilia using the energy from ATP hydrolysis is only partially understood. In this study, we develop a methodology for live-cell single-molecule fluorescence microscopy of organelles protruding from the apical surface using a dual-view light-sheet microscope, diSPIM. We demonstrate that MYO7A, a component of the MET machinery, traffics as a dimer in stereocilia. Movements of MYO7A are restricted when scaffolded by the plasma membrane and F-actin as mediated by MYO7A's interacting partners. Here, we discuss the technical details of our methodology and its future applications including analyses of cargo transportation in various organelles.

Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy

Computational super-resolution methods, including conventional analytical algorithms and deep learning models, have substantially improved optical microscopy. Among them, supervised deep neural networks have demonstrated outstanding performance, however, demanding abundant high-quality training data, which are laborious and even impractical to acquire due to the high dynamics of living cells. Here, we develop zero-shot deconvolution networks (ZS-DeconvNet) that instantly enhance the resolution of microscope images by more than 1.5-fold over the diffraction limit with 10-fold lower fluorescence than ordinary super-resolution imaging conditions, in an unsupervised manner without the need for either ground truths or additional data acquisition. We demonstrate the versatile applicability of ZS-DeconvNet on multiple imaging modalities, including total internal reflection fluorescence microscopy, three-dimensional wide-field microscopy, confocal microscopy, two-photon microscopy, lattice light-sheet microscopy, and multimodal structured illumination microscopy, which enables multi-color, long-term, super-resolution 2D/3D imaging of subcellular bioprocesses from mitotic single cells to multicellular embryos of mouse and C. elegans.

Live-cell single-molecule fluorescence microscopy for protruding organelles reveals regulatory mechanisms of MYO7A-driven cargo transport in stereocilia of inner ear hair cells

Stereocilia are unidirectional F-actin-based cylindrical protrusions on the apical surface of inner ear hair cells and function as biological mechanosensors of sound and acceleration. Development of functional stereocilia requires motor activities of unconventional myosins to transport proteins necessary for elongating the F-actin cores and to assemble the mechanoelectrical transduction (MET) channel complex. However, how each myosin localizes in stereocilia using the energy from ATP hydrolysis is only partially understood. In this study, we develop a methodology for live-cell single-molecule fluorescence microscopy of organelles protruding from the apical surface using a dual-view light-sheet microscope, diSPIM. We demonstrate that MYO7A, a component of the MET machinery, traffics as a dimer in stereocilia. Movements of MYO7A are restricted when scaffolded by the plasma membrane and F-actin as mediated by MYO7A's interacting partners. Here, we discuss the technical details of our methodology and its future applications including analyses of cargo transportation in various organelles.

Light-Sheet Imaging to Reveal Cardiac Structure in Rodent Hearts

Light-sheet microscopy (LSM) plays a pivotal role in comprehending the intricate three-dimensional (3D) structure of the heart, providing crucial insights into fundamental cardiac physiology and pathologic responses. We hereby delve into the development and implementation of the LSM technique to elucidate the micro-architecture of the heart in mouse models. The methodology integrates a customized LSM system with tissue clearing techniques, mitigating light scattering within cardiac tissues for volumetric imaging. The combination of conventional LSM with image stitching and multiview deconvolution approaches allows for the capture of the entire heart. To address the inherent trade-off between axial resolution and field of view (FOV), we further introduce an axially swept light-sheet microscopy (ASLM) method to minimize out-of-focus light and uniformly illuminate the heart across the propagation direction. In the meanwhile, tissue clearing methods such as iDISCO enhance light penetration, facilitating the visualization of deep structures and ensuring a comprehensive examination of the myocardium throughout the entire heart. The combination of the proposed LSM and tissue clearing methods presents a promising platform for researchers in resolving cardiac structures in rodent hearts, holding great potential for the understanding of cardiac morphogenesis and remodeling.

Transgenic Tg(Kcnj10-ZsGreen) fluorescent reporter mice allow visualization of intermediate cells in the stria vascularis

The stria vascularis (SV) is a stratified epithelium in the lateral wall of the mammalian cochlea, responsible for both endolymphatic ion homeostasis and generation of the endocochlear potential (EP) critical for normal hearing. The SV has three layers consisting predominantly of basal, intermediate, and marginal cells. Intermediate and marginal cells form an intricate interdigitated network of cell projections making discrimination of the cells challenging. To enable intermediate cell visualization, we engineered by BAC transgenesis, reporter mouse lines expressing ZsGreen fluorescent protein under the control of Kcnj10 promoter and regulatory sequences. Kcnj10 encodes KCNJ10 protein (also known as Kir4.1 or Kir1.2), an ATP-sensitive inwardly-rectifying potassium channel critical to EP generation, highly expressed in SV intermediate cells. In these transgenic mice, ZsGreen fluorescence mimics Kcnj10 endogenous expression in the cochlea and was detected in the intermediate cells of the SV, in the inner phalangeal cells, Hensen's, Deiters' and pillar cells, in a subset of spiral ganglion neurons, and in glial cells. We show that expression of the transgene in hemizygous mice does not alter auditory function, nor EP. These transgenic Tg(Kcnj10-ZsGreen) mice allow live and fixed tissue visualization of ZsGreen-expressing intermediate cells and will facilitate future studies of stria vascularis cell function.

Harnessing artificial intelligence to reduce phototoxicity in live imaging

Fluorescence microscopy is essential for studying living cells, tissues and organisms. However, the fluorescent light that switches on fluorescent molecules also harms the samples, jeopardizing the validity of results - particularly in techniques such as super-resolution microscopy, which demands extended illumination. Artificial intelligence (AI)-enabled software capable of denoising, image restoration, temporal interpolation or cross-modal style transfer has great potential to rescue live imaging data and limit photodamage. Yet we believe the focus should be on maintaining light-induced damage at levels that preserve natural cell behaviour. In this Opinion piece, we argue that a shift in role for AIs is needed - AI should be used to extract rich insights from gentle imaging rather than recover compromised data from harsh illumination. Although AI can enhance imaging, our ultimate goal should be to uncover biological truths, not just retrieve data. It is essential to prioritize minimizing photodamage over merely pushing technical limits. Our approach is aimed towards gentle acquisition and observation of undisturbed living systems, aligning with the essence of live-cell fluorescence microscopy.

Noise learning of instruments for high-contrast, high-resolution and fast hyperspectral microscopy and nanoscopy

The low scattering efficiency of Raman scattering makes it challenging to simultaneously achieve good signal-to-noise ratio (SNR), high imaging speed, and adequate spatial and spectral resolutions. Here, we report a noise learning (NL) approach that estimates the intrinsic noise distribution of each instrument by statistically learning the noise in the pixel-spatial frequency domain. The estimated noise is then removed from the noisy spectra. This enhances the SNR by ca. 10 folds, and suppresses the mean-square error by almost 150 folds. NL allows us to improve the positioning accuracy and spatial resolution and largely eliminates the impact of thermal drift on tip-enhanced Raman spectroscopic nanoimaging. NL is also applicable to enhance SNR in fluorescence and photoluminescence imaging. Our method manages the ground truth spectra and the instrumental noise simultaneously within the training dataset, which bypasses the tedious labelling of huge dataset required in conventional deep learning, potentially shifting deep learning from sample-dependent to instrument-dependent.

Pro-regenerative biomaterials recruit immunoregulatory dendritic cells after traumatic injury

During wound healing and surgical implantation, the body establishes a delicate balance between immune activation to fight off infection and clear debris and immune tolerance to control reactivity against self-tissue. Nonetheless, how such a balance is achieved is not well understood. Here we describe that pro-regenerative biomaterials for muscle injury treatment promote the proliferation of a BATF3-dependent CD103+XCR1+CD206+CD301b+ dendritic cell population associated with cross-presentation and self-tolerance. Upregulation of E-cadherin, the ligand for CD103, and XCL-1 in injured tissue suggests a mechanism for cell recruitment to trauma. Muscle injury recruited natural killer cells that produced Xcl1 when stimulated with fragmented extracellular matrix. Without cross-presenting cells, T-cell activation increases, pro-regenerative macrophage polarization decreases and there are alterations in myogenesis, adipogenesis, fibrosis and increased muscle calcification. These results, previously observed in cancer progression, suggest a fundamental mechanism of immune regulation in trauma and material implantation with implications for both short- and long-term injury recovery.

Enhancing Light-Sheet Fluorescence Microscopy Illumination Beams through Deep Design Optimization

Light sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times for imaging of tissue-cleared specimen. This allows for high-resolution 3D imaging of large tissue volumes. Inherently to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, with the notion that the illumination beam only illuminates a thin section that is being imaged. Therefore, substantial efforts are dedicated to identifying slender, non-diffracting beam profiles that can yield uniform and high-contrast images. An ongoing debate concerns the employment of the most optimal illumination beam; Gaussian, Bessel, Airy patterns and/or others. Comparisons among different beam profiles is challenging as their optimization objective is often different. Given that our large imaging datasets (~0.5TB images per sample) is already analyzed using deep learning models, we envisioned a different approach to this problem by hypothesizing that we can tailor the illumination beam to boost the deep learning models performance. We achieve this by integrating the physical LSFM illumination model after passing through a variable phase mask into the training of a cell detection network. Here we report that the joint optimization continuously updates the phase mask, improving the image quality for better cell detection. Our method's efficacy is demonstrated through both simulations and experiments, revealing substantial enhancements in imaging quality compared to traditional Gaussian light sheet. We offer valuable insights for designing microscopy systems through a computational approach that exhibits significant potential for advancing optics design that relies on deep learning models for analysis of imaging datasets.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

[Oral panorama reconstruction method based on pre-segmentation and Bezier function]

For patients with partial jaw defects, cysts and dental implants, doctors need to take panoramic X-ray films or manually draw dental arch lines to generate Panorama images in order to observe their complete dentition information during oral diagnosis. In order to solve the problems of additional burden for patients to take panoramic X-ray films and time-consuming issue for doctors to manually segment dental arch lines, this paper proposes an automatic panorama reconstruction method based on cone beam computerized tomography (CBCT). The V-network (VNet) is used to pre-segment the teeth and the background to generate the corresponding binary image, and then the Bezier curve is used to define the best dental arch curve to generate the oral panorama. In addition, this research also addressed the issues of mistakenly recognizing the teeth and jaws as dental arches, incomplete coverage of the dental arch area by the generated dental arch lines, and low robustness, providing intelligent methods for dental diagnosis and improve the work efficiency of doctors.

Transgenic Tg(Kcnj10-ZsGreen) Fluorescent Reporter Mice Allow Visualization of Intermediate Cells in the Stria Vascularis

The stria vascularis (SV) is a stratified epithelium in the lateral wall of the mammalian cochlea, responsible for both endolymphatic ion homeostasis and generation of the endocochlear potential (EP) critical for normal hearing. The SV has three layers consisting predominantly of basal, intermediate, and marginal cells. Intermediate and marginal cells form an intricate interdigitated network of cell projections making discrimination of the cells challenging. To enable intermediate cell visualization, we engineered by BAC transgenesis, reporter mouse lines expressing ZsGreen fluorescent protein under the control of Kcnj10 promoter and regulatory sequences. Kcnj10 encodes KCNJ10 protein (also known as Kir4.1 or Kir1.2), an ATP-sensitive inwardly-rectifying potassium channel critical to EP generation, highly expressed in SV intermediate cells. In these transgenic mice, ZsGreen fluorescence mimics Kcnj10 endogenous expression in the cochlea and was detected in the intermediate cells of the SV, in the inner phalangeal cells, Hensen's, Deiters' and pillar cells, in a subset of spiral ganglion neurons, and in glial cells. We show that expression of the transgene in hemizygous mice does not alter auditory function, nor EP These transgenic Tg(Kcnj10-ZsGreen) mice allow live and fixed tissue visualization of ZsGreen-expressing intermediate cells and will facilitate future studies of stria vascularis cell function.

Experimental and theoretical model for the origin of coiling of cellular protrusions around fibers

Protrusions at the leading-edge of a cell play an important role in sensing the extracellular cues during cellular spreading and motility. Recent studies provided indications that these protrusions wrap (coil) around the extracellular fibers. However, the physics of this coiling process, and the mechanisms that drive it, are not well understood. We present a combined theoretical and experimental study of the coiling of cellular protrusions on fibers of different geometry. Our theoretical model describes membrane protrusions that are produced by curved membrane proteins that recruit the protrusive forces of actin polymerization, and identifies the role of bending and adhesion energies in orienting the leading-edges of the protrusions along the azimuthal (coiling) direction. Our model predicts that the cell's leading-edge coils on fibers with circular cross-section (above some critical radius), but the coiling ceases for flattened fibers of highly elliptical cross-section. These predictions are verified by 3D visualization and quantitation of coiling on suspended fibers using Dual-View light-sheet microscopy (diSPIM). Overall, we provide a theoretical framework, supported by experiments, which explains the physical origin of the coiling phenomenon.

Three-dimensional structured illumination microscopy with enhanced axial resolution

The axial resolution of three-dimensional structured illumination microscopy (3D SIM) is limited to ∼300 nm. Here we present two distinct, complementary methods to improve axial resolution in 3D SIM with minimal or no modification to the optical system. We show that placing a mirror directly opposite the sample enables four-beam interference with higher spatial frequency content than 3D SIM illumination, offering near-isotropic imaging with ∼120-nm lateral and 160-nm axial resolution. We also developed a deep learning method achieving ∼120-nm isotropic resolution. This method can be combined with denoising to facilitate volumetric imaging spanning dozens of timepoints. We demonstrate the potential of these advances by imaging a variety of cellular samples, delineating the nanoscale distribution of vimentin and microtubule filaments, observing the relative positions of caveolar coat proteins and lysosomal markers and visualizing cytoskeletal dynamics within T cells in the early stages of immune synapse formation.

Optimal sparsity allows reliable system-aware restoration of fluorescence microscopy images

Fluorescence microscopy is one of the most indispensable and informative driving forces for biological research, but the extent of observable biological phenomena is essentially determined by the content and quality of the acquired images. To address the different noise sources that can degrade these images, we introduce an algorithm for multiscale image restoration through optimally sparse representation (MIRO). MIRO is a deterministic framework that models the acquisition process and uses pixelwise noise correction to improve image quality. Our study demonstrates that this approach yields a remarkable restoration of the fluorescence signal for a wide range of microscopy systems, regardless of the detector used (e.g., electron-multiplying charge-coupled device, scientific complementary metal-oxide semiconductor, or photomultiplier tube). MIRO improves current imaging capabilities, enabling fast, low-light optical microscopy, accurate image analysis, and robust machine intelligence when integrated with deep neural networks. This expands the range of biological knowledge that can be obtained from fluorescence microscopy.

Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy

One intrinsic yet critical issue that troubles the field of fluorescence microscopy ever since its introduction is the unmatched resolution in the lateral and axial directions (i.e., resolution anisotropy), which severely deteriorates the quality, reconstruction, and analysis of 3D volume images. By leveraging the natural anisotropy, we present a deep self-learning method termed Self-Net that significantly improves the resolution of axial images by using the lateral images from the same raw dataset as rational targets. By incorporating unsupervised learning for realistic anisotropic degradation and supervised learning for high-fidelity isotropic recovery, our method can effectively suppress the hallucination with substantially enhanced image quality compared to previously reported methods. In the experiments, we show that Self-Net can reconstruct high-fidelity isotropic 3D images from organelle to tissue levels via raw images from various microscopy platforms, e.g., wide-field, laser-scanning, or super-resolution microscopy. For the first time, Self-Net enables isotropic whole-brain imaging at a voxel resolution of 0.2 × 0.2 × 0.2 μm3, which addresses the last-mile problem of data quality in single-neuron morphology visualization and reconstruction with minimal effort and cost. Overall, Self-Net is a promising approach to overcoming the inherent resolution anisotropy for all classes of 3D fluorescence microscopy.

Augmented Super-Resolution Radial Fluctuations (aSRRF) Pushing the Limits of Structured Illumination Microscopy

Structured illumination microscopy (SIM) is a versatile super-resolution technique known for its compatibility with a wide range of probes and fast implementation. While 3D SIM is capable of achieving a spatial resolution of ∼120 nm laterally and ∼300 nm axially, attempting to further enhance the resolution through methods such as nonlinear SIM or 4-beam SIM introduces complexities in optical configurations, increased phototoxicity, and reduced temporal resolution. Here, we have developed a novel method that combines SIM with augmented super-resolution radial fluctuations (aSRRF) utilizing a single image through image augmentation. By applying aSRRF reconstruction to SIM images, we can enhance the SIM resolution to ∼50 nm isotopically, without requiring any modifications to the optical system or sample acquisition process. Additionaly, we have incorporated the aSRRF approach into an ImageJ plugin and demonstrated its versatility across various fluorescence microscopy images, showcasing a remarkable two-fold resolution increase.

Airy beam light sheet microscopy boosted by deep learning deconvolution

Common light sheet microscopy comes with a trade-off between light sheet width defining the optical sectioning and the usable field of view arising from the divergence of the illuminating Gaussian beam. To overcome this, low-diverging Airy beams have been introduced. Airy beams, however, exhibit side lobes degrading image contrast. Here, we constructed an Airy beam light sheet microscope, and developed a deep learning image deconvolution to remove the effects of the side lobes without knowledge of the point spread function. Using a generative adversarial network and high-quality training data, we significantly enhanced image contrast and improved the performance of a bicubic upscaling. We evaluated the performance with fluorescently labeled neurons in mouse brain tissue samples. We found that deep learning-based deconvolution was about 20-fold faster than the standard approach. The combination of Airy beam light sheet microscopy and deep learning deconvolution allows imaging large volumes rapidly and with high quality.

Label-free cleared tissue microscopy and machine learning for 3D histopathology of biomaterial implants

Tissue clearing of whole intact organs has enhanced imaging by enabling the exploration of tissue structure at a subcellular level in three-dimensional space. Although clearing and imaging of the whole organ have been used to study tissue biology, the microenvironment in which cells evolve to adapt to biomaterial implants or allografts in the body is poorly understood. Obtaining high-resolution information from complex cell-biomaterial interactions with volumetric landscapes represents a key challenge in the fields of biomaterials and regenerative medicine. To provide a new approach to examine how tissue responds to biomaterial implants, we apply cleared tissue light-sheet microscopy and three-dimensional reconstruction to utilize the wealth of autofluorescence information for visualizing and contrasting anatomical structures. This study demonstrates the adaptability of the clearing and imaging technique to provide sub-cellular resolution (0.6 μm isotropic) 3D maps of various tissue types, using samples from fully intact peritoneal organs to volumetric muscle loss injury specimens. Specifically, in the volumetric muscle loss injury model, we provide 3D visualization of the implanted extracellular matrix biomaterial in the wound bed of the quadricep muscle groups and further apply computational-driven image classification to analyze the autofluorescence spectrum at multiple emission wavelengths to categorize tissue types at the injured site interacting with the biomaterial scaffolds.

Partially interpretable image deconvolution framework based on the Richardson-Lucy model

Fluorescence microscopy typically suffers from aberration induced by system and sample, which could be circumvented by image deconvolution. We proposed a novel, to the best of our knowledge, Richardson-Lucy (RL) model-driven deconvolution framework to improve reconstruction performance and speed. Two kinds of neural networks within this framework were devised, which are partially interpretable compared with previous deep learning methods. We first introduce RL into deep feature space, which has superior generalizability to the convolutional neural networks (CNN). We further accelerate it with an unmatched backprojector, providing a five times faster reconstruction speed than classic RL. Our deconvolution approaches outperform both CNN and traditional methods regarding image quality for blurred images caused by out-of-focus or imaging system aberration.

Efficient 3D light-sheet imaging of very large-scale optically cleared human brain and prostate tissue samples

The ability to image human tissue samples in 3D, with both cellular resolution and a large field of view (FOV), can improve fundamental and clinical investigations. Here, we demonstrate the feasibility of light-sheet imaging of ~5 cm³ sized formalin fixed human brain and up to ~7 cm³ sized formalin fixed paraffin embedded (FFPE) prostate cancer samples, processed with the FFPE-MASH protocol. We present a light-sheet microscopy prototype, the cleared-tissue dual view Selective Plane Illumination Microscope (ct-dSPIM), capable of fast 3D high-resolution acquisitions of cm³ scale cleared tissue. We used mosaic scans for fast 3D overviews of entire tissue samples or higher resolution overviews of large ROIs with various speeds: (a) Mosaic 16 (16.4 µm isotropic resolution, ~1.7 h/cm³), (b) Mosaic 4 (4.1 µm isotropic resolution, ~ 5 h/cm³) and (c) Mosaic 0.5 (0.5 µm near isotropic resolution, ~15.8 h/cm³). We could visualise cortical layers and neurons around the border of human brain areas V1&V2, and could demonstrate suitable imaging quality for Gleason score grading in thick prostate cancer samples. We show that ct-dSPIM imaging is an excellent technique to quantitatively assess entire MASH prepared large-scale human tissue samples in 3D, with considerable future clinical potential.

Optimised generalized polarisation analysis of C-laurdan reveals clear order differences between T cell membrane compartments

Heterogenous packing of plasma membrane lipids is important for cellular processes like signalling, adhesion and sorting of membrane components. Solvatochromic membrane fluorophores that respond to changes from liquid-ordered (l_o) phase to liquid-disordered (l_d) by red shifts in their emission spectra are often used to assess lipid packing. Their response can be quantified using generalized polarisation (GP) using fluorescence microscopy images from two emission ranges, preferably from a region of interest (ROI) limited to a specific membrane compartment. However, image quality is limited by Poisson noise and convolution by the point spread function of the imaging system. Examining GP-analysis of C-laurdan labelled T cells using the image restoration procedure deconvolution, we demonstrate that deconvolution substantially improves the image resolution by making the plasma membrane clearly discernible and facilitating plasma membrane ROI selection. We conclude that automatic ROI selection has advantages over manual ROI selection when it comes to reproducibility and speed, but reliable GP-measurements can also be obtained by manually demarcated ROIs. We find that deconvolution enhances the difference in GP-values between the plasma and intracellular membranes and demonstrate that moving an intensity defined plasma membrane ROI outwards from the cell further improves this differentiation. By systematically changing the key deconvolution regularization parameter signal to noise, we establish a protocol for deconvolution optimisation applicable to any solvatochromic dye and imaging system. The image processing and ROI selection protocol presented improves both the resolution and precision of GP-measurement and will enable detection of smaller changes in membrane order than is currently achievable.

SPITFIR(e): a supermaneuverable algorithm for fast denoising and deconvolution of 3D fluorescence microscopy images and videos

Modern fluorescent microscopy imaging is still limited by the optical aberrations and the photon budget available in the specimen. A direct consequence is the necessity to develop flexible and "off-road" algorithms in order to recover structural details and improve spatial resolution, which is critical when restraining the illumination to low levels in order to limit photo-damages. Here, we report SPITFIR(e) a flexible method designed to accurately and quickly restore 2D-3D fluorescence microscopy images and videos (4D images). We designed a generic sparse-promoting regularizer to subtract undesirable out-of-focus background and we developed a primal-dual algorithm for fast optimization. SPITFIR(e) is a "swiss-knife" method for practitioners as it adapts to any microscopy techniques, to various sources of signal degradation (noise, blur), to variable image contents, as well as to low signal-to-noise ratios. Our method outperforms existing state-of-the-art algorithms, and is more flexible than supervised deep-learning methods requiring ground truth datasets. The performance, the flexibility, and the ability to push the spatiotemporal resolution limit of sub-diffracted fluorescence microscopy techniques are demonstrated on experimental datasets acquired with various microscopy techniques from 3D spinning-disk confocal up to lattice light sheet microscopy.

Analysis of Spatial Molecular Data

Digital analysis of pathology whole-slide images has been recently gaining interest in the context of cancer diagnosis and treatment. In particular, deep learning methods have demonstrated significant potential in supporting pathology analysis, recently detecting molecular traits never before recognized in pathology H&E whole-slide images (WSIs). Alongside these advancements in the digital analysis of WSIs, it is becoming increasingly evident that both spatial and overall tumor heterogeneity may be significant determinants of cancer prognosis and treatment outcome. In this chapter, we describe methods that aim to support these two elements. We describe both an end-to-end deep learning pipeline for producing limited spatial transcriptomics from WSIs with associated bulk gene expression data, as well as an algorithm for quantifying spatial tumor heterogeneity based on the results of this pipeline.

Practical considerations for quantitative light sheet fluorescence microscopy

Fluorescence microscopy has evolved from a purely observational tool to a platform for quantitative, hypothesis-driven research. As such, the demand for faster and less phototoxic imaging modalities has spurred a rapid growth in light sheet fluorescence microscopy (LSFM). By restricting the excitation to a thin plane, LSFM reduces the overall light dose to a specimen while simultaneously improving image contrast. However, the defining characteristics of light sheet microscopes subsequently warrant unique considerations in their use for quantitative experiments. In this Perspective, we outline many of the pitfalls in LSFM that can compromise analysis and confound interpretation. Moreover, we offer guidance in addressing these caveats when possible. In doing so, we hope to provide a useful resource for life scientists seeking to adopt LSFM to quantitatively address complex biological hypotheses.

Experimentally unsupervised deconvolution for light-sheet microscopy with propagation-invariant beams

Deconvolution is a challenging inverse problem, particularly in techniques that employ complex engineered point-spread functions, such as microscopy with propagation-invariant beams. Here, we present a deep-learning method for deconvolution that, in lieu of end-to-end training with ground truths, is trained using known physics of the imaging system. Specifically, we train a generative adversarial network with images generated with the known point-spread function of the system, and combine this with unpaired experimental data that preserve perceptual content. Our method rapidly and robustly deconvolves and super-resolves microscopy images, demonstrating a two-fold improvement in image contrast to conventional deconvolution methods. In contrast to common end-to-end networks that often require 1000-10,000s paired images, our method is experimentally unsupervised and can be trained solely on a few hundred regions of interest. We demonstrate its performance on light-sheet microscopy with propagation-invariant Airy beams in oocytes, preimplantation embryos and excised brain tissue, as well as illustrate its utility for Bessel-beam LSM. This method aims to democratise learned methods for deconvolution, as it does not require data acquisition outwith the conventional imaging protocol.

Incorporating the image formation process into deep learning improves network performance

We present Richardson-Lucy network (RLN), a fast and lightweight deep learning method for three-dimensional fluorescence microscopy deconvolution. RLN combines the traditional Richardson-Lucy iteration with a fully convolutional network structure, establishing a connection to the image formation process and thereby improving network performance. Containing only roughly 16,000 parameters, RLN enables four- to 50-fold faster processing than purely data-driven networks with many more parameters. By visual and quantitative analysis, we show that RLN provides better deconvolution, better generalizability and fewer artifacts than other networks, especially along the axial dimension. RLN outperforms classic Richardson-Lucy deconvolution on volumes contaminated with severe out of focus fluorescence or noise and provides four- to sixfold faster reconstructions of large, cleared-tissue datasets than classic multi-view pipelines. We demonstrate RLN's performance on cells, tissues and embryos imaged with widefield-, light-sheet-, confocal- and super-resolution microscopy.

Multiscale fluorescence imaging of living samples

Fluorescence microscopy is a highly effective tool for interrogating biological structure and function, particularly when imaging across multiple spatiotemporal scales. Here we survey recent innovations and applications in the relatively understudied area of multiscale fluorescence imaging of living samples. We discuss fundamental challenges in live multiscale imaging and describe successful examples that highlight the power of this approach. We attempt to synthesize general strategies from these test cases, aiming to help accelerate progress in this exciting area.

Review on data analysis methods for mesoscale neural imaging in vivo

Significance: Mesoscale neural imaging in vivo has gained extreme popularity in neuroscience for its capacity of recording large-scale neurons in action. Optical imaging with single-cell resolution and millimeter-level field of view in vivo has been providing an accumulated database of neuron-behavior correspondence. Meanwhile, optical detection of neuron signals is easily contaminated by noises, background, crosstalk, and motion artifacts, while neural-level signal processing and network-level coordinate are extremely complicated, leading to laborious and challenging signal processing demands. The existing data analysis procedure remains unstandardized, which could be daunting to neophytes or neuroscientists without computational background. Aim: We hope to provide a general data analysis pipeline of mesoscale neural imaging shared between imaging modalities and systems. Approach: We divide the pipeline into two main stages. The first stage focuses on extracting high-fidelity neural responses at single-cell level from raw images, including motion registration, image denoising, neuron segmentation, and signal extraction. The second stage focuses on data mining, including neural functional mapping, clustering, and brain-wide network deduction. Results: Here, we introduce the general pipeline of processing the mesoscale neural images. We explain the principles of these procedures and compare different approaches and their application scopes with detailed discussions about the shortcomings and remaining challenges. Conclusions: There are great challenges and opportunities brought by the large-scale mesoscale data, such as the balance between fidelity and efficiency, increasing computational load, and neural network interpretability. We believe that global circuits on single-neuron level will be more extensively explored in the future.

Image Segmentation and Quantification of Droplet dPCR Based on Thermal Bubble Printing Technology

Thermal inkjet printing can generate more than 300,000 droplets of picoliter scale within one second stably, and the image analysis workflow is used to quantify the positive and negative values of the droplets. In this paper, the SimpleBlobDetector detection algorithm is used to identify and localize droplets with a volume of 24 pL in bright field images and suppress bright spots and scratches when performing droplet location identification. The polynomial surface fitting of the pixel grayscale value of the fluorescence channel image can effectively compensate and correct the image vignetting caused by the optical path, and the compensated fluorescence image can accurately classify positive and negative droplets by the k-means clustering algorithm. 20 µL of the sample solution in the result reading chip can produce more than 100,000 effective droplets. The effective droplet identification correct rate of 20 images of random statistical samples can reach more than 99% and the classification accuracy of positive and negative droplets can reach more than 98% on average. This paper overcomes the problem of effectively classifying positive and negative droplets caused by the poor image quality of photographed picolitre ddPCR droplets caused by optical hardware limitations.

Stereotyped behavioral maturation and rhythmic quiescence in C.elegans embryos

Systematic analysis of rich behavioral recordings is being used to uncover how circuits encode complex behaviors. Here, we apply this approach to embryos. What are the first embryonic behaviors and how do they evolve as early neurodevelopment ensues? To address these questions, we present a systematic description of behavioral maturation for Caenorhabditis elegans embryos. Posture libraries were built using a genetically encoded motion capture suit imaged with light-sheet microscopy and annotated using custom tracking software. Analysis of cell trajectories, postures, and behavioral motifs revealed a stereotyped developmental progression. Early movement is dominated by flipping between dorsal and ventral coiling, which gradually slows into a period of reduced motility. Late-stage embryos exhibit sinusoidal waves of dorsoventral bends, prolonged bouts of directed motion, and a rhythmic pattern of pausing, which we designate slow wave twitch (SWT). Synaptic transmission is required for late-stage motion but not for early flipping nor the intervening inactive phase. A high-throughput behavioral assay and calcium imaging revealed that SWT is elicited by the rhythmic activity of a quiescence-promoting neuron (RIS). Similar periodic quiescent states are seen prenatally in diverse animals and may play an important role in promoting normal developmental outcomes.

Image Reconstruction in Light-Sheet Microscopy: Spatially Varying Deconvolution and Mixed Noise

We study the problem of deconvolution for light-sheet microscopy, where the data is corrupted by spatially varying blur and a combination of Poisson and Gaussian noise. The spatial variation of the point spread function of a light-sheet microscope is determined by the interaction between the excitation sheet and the detection objective PSF. We introduce a model of the image formation process that incorporates this interaction and we formulate a variational model that accounts for the combination of Poisson and Gaussian noise through a data fidelity term consisting of the infimal convolution of the single noise fidelities, first introduced in L. Calatroni et al. (SIAM J Imaging Sci 10(3):1196-1233, 2017). We establish convergence rates and a discrepancy principle for the infimal convolution fidelity and the inverse problem is solved by applying the primal-dual hybrid gradient (PDHG) algorithm in a novel way. Numerical experiments performed on simulated and real data show superior reconstruction results in comparison with other methods.

Blind deconvolution in autocorrelation inversion for multiview light-sheet microscopy

Combining the information coming from multiview acquisitions is a problem of great interest in light-sheet microscopy. Aligning the views and increasing the resolution of their fusion can be challenging, especially if the setup is not fully calibrated. Here, we tackle these issues by proposing a new reconstruction method based on autocorrelation inversion that avoids alignment procedures. On top of this, we add a blind deconvolution step to improve the resolution of the final reconstruction. Our method permits us to achieve inherently aligned, highly resolved reconstructions while, at the same time, estimating the unknown point-spread function of the system. RESEARCH HIGHLIGHTS: We tackle the problem of multiview light-sheet deconvolution with a blind approach of autocorrelation inversion Our method recovers the object and PSF, requires no alignment and calibration, and enhances the reconstruction of the specimen.

FCE-Net: a fast image contrast enhancement method based on deep learning for biomedical optical images

Optical imaging is an important tool for exploring and understanding structures of biological tissues. However, due to the heterogeneity of biological tissues, the intensity distribution of the signal is not uniform and contrast is normally degraded in the raw image. It is difficult to be used for subsequent image analysis and information extraction directly. Here, we propose a fast image contrast enhancement method based on deep learning called Fast Contrast Enhancement Network (FCE-Net). We divided network into dual-path to simultaneously obtain spatial information and large receptive field. And we introduced the spatial attention mechanism to enhance the inter-spatial relationship. We showed that the cell counting task of mouse brain images processed by FCE-Net was with average precision rate of 97.6% ± 1.6%, and average recall rate of 98.4% ± 1.4%. After processing with FCE-Net, the images from vascular extraction (DRIVE) dataset could be segmented with spatial attention U-Net (SA-UNet) to achieve state-of-the-art performance. By comparing FCE-Net with previous methods, we demonstrated that FCE-Net could obtain higher accuracy while maintaining the processing speed. The images with size of 1024 × 1024 pixels could be processed by FCE-Net with 37fps based on our workstation. Our method has great potential for further image analysis and information extraction from large-scale or dynamic biomedical optical images.

Using Light-Sheet Microscopy to Study Spontaneous Activity in the Developing Lateral-Line System

Hair cells are the sensory receptors in the auditory and vestibular systems of all vertebrates, and in the lateral-line system of aquatic vertebrates. The purpose of this work is to explore the zebrafish lateral-line system as a model to study and understand spontaneous activity in vivo. Our work applies genetically encoded calcium indicators along with light-sheet fluorescence microscopy to visualize spontaneous calcium activity in the developing lateral-line system. Consistent with our previous work, we show that spontaneous calcium activity is present in developing lateral-line hair cells. We now show that supporting cells that surround hair cells, and cholinergic efferent terminals that directly contact hair cells are also spontaneously active. Using two-color functional imaging we demonstrate that spontaneous activity in hair cells does not correlate with activity in either supporting cells or cholinergic terminals. We find that during lateral-line development, hair cells autonomously generate spontaneous events. Using localized calcium indicators, we show that within hair cells, spontaneous calcium activity occurs in two distinct domains—the mechanosensory bundle and the presynapse. Further, spontaneous activity in the mechanosensory bundle ultimately drives spontaneous calcium influx at the presynapse. Comprehensively, our results indicate that in developing lateral-line hair cells, autonomously generated spontaneous activity originates with spontaneous mechanosensory events.

Answering some questions about structured illumination microscopy

Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood 'super-resolution' method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

Simulation-driven learning: a deep learning approach for image scanning microscopy via physical imaging models

Image reconstruction based on deep learning has become an effective tool in fluorescence microscopy. Most deep learning reconstruction methods ignore the mechanism of the imaging process where a large number of datasets are required. In addition, a lot of time is spent solving the aliasing problem from multi-scaled image pairs for data pre-processing. Here we demonstrate an improved generative adversarial network for image scanning microscopy (ISM) that can be trained by simulation data and has good generalization. Based on physical imaging models, this method can generate matching image pairs from simulation images and uses them as datasets for network training, without capturing a large number of real ISM images and avoiding image alignment preprocessing. Simulation and experimental results show that this simulation data-driven method improves the imaging quality of conventional microscopic images and reduces the cost of experiments. This method provides inspiration for optimizing network generalizability of the deep learning network.