Practical sensorless aberration estimation for 3D microscopy with deep learning

Deep learning-based aberration compensation improves contrast and resolution in fluorescence microscopy

Optical aberrations hinder fluorescence microscopy of thick samples, reducing image signal, contrast, and resolution. Here we introduce a deep learning-based strategy for aberration compensation, improving image quality without slowing image acquisition, applying additional dose, or introducing more optics into the imaging path. Our method (i) introduces synthetic aberrations to images acquired on the shallow side of image stacks, making them resemble those acquired deeper into the volume and (ii) trains neural networks to reverse the effect of these aberrations. We use simulations and experiments to show that applying the trained 'de-aberration' networks outperforms alternative methods, providing restoration on par with adaptive optics techniques; and subsequently apply the networks to diverse datasets captured with confocal, light-sheet, multi-photon, and super-resolution microscopy. In all cases, the improved quality of the restored data facilitates qualitative image inspection and improves downstream image quantitation, including orientational analysis of blood vessels in mouse tissue and improved membrane and nuclear segmentation in C. elegans embryos.

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

HoloTile light engine: new digital holographic modalities and applications

HoloTile is a patented computer generated holography approach with the aim of reducing the speckle noise caused by the overlap of the non-trivial physical extent of the point spread function in Fourier holographic systems from adjacent frequency components. By combining tiling of phase-only of rapidly generated sub-holograms with a PSF-shaping phase profile, each frequency component-or output 'pixel'- in the Fourier domain is shaped to a desired non-overlapping profile. In this paper, we show the high-resolution, speckle-reduced reconstructions that can be achieved with HoloTile, as well as present new HoloTile modalities, including an expanded list of PSF options with new key properties. In addition, we discuss numerous applications for which HoloTile, its rapid hologram generation, and the new PSF options may be an ideal fit, including optical trapping and manipulation of particles, volumetric additive printing, information transfer and quantum communication.

Universal adaptive optics for microscopy through embedded neural network control

The resolution and contrast of microscope imaging is often affected by aberrations introduced by imperfect optical systems and inhomogeneous refractive structures in specimens. Adaptive optics (AO) compensates these aberrations and restores diffraction limited performance. A wide range of AO solutions have been introduced, often tailored to a specific microscope type or application. Until now, a universal AO solution - one that can be readily transferred between microscope modalities - has not been deployed. We propose versatile and fast aberration correction using a physics-based machine learning assisted wavefront-sensorless AO control (MLAO) method. Unlike previous ML methods, we used a specially constructed neural network (NN) architecture, designed using physical understanding of the general microscope image formation, that was embedded in the control loop of different microscope systems. The approach means that not only is the resulting NN orders of magnitude simpler than previous NN methods, but the concept is translatable across microscope modalities. We demonstrated the method on a two-photon, a three-photon and a widefield three-dimensional (3D) structured illumination microscope. Results showed that the method outperformed commonly-used modal-based sensorless AO methods. We also showed that our ML-based method was robust in a range of challenging imaging conditions, such as 3D sample structures, specimen motion, low signal to noise ratio and activity-induced fluorescence fluctuations. Moreover, as the bespoke architecture encapsulated physical understanding of the imaging process, the internal NN configuration was no-longer a "black box", but provided physical insights on internal workings, which could influence future designs.

Phase retrieval for the generation of arbitrary intensity distributions using an optofluidic phase shifter

An optofluidic phase shifter can be used to generate virtually arbitrary intensity patterns, but only if the phase shift generated by the controllably deformed fluidic surface can be appropriately defined. To enable this functionality, we present two phase retrieval algorithms based on neural networks and least-squares optimization which are used to determine the necessary phase profile to generate a desired target intensity pattern with high accuracy. We demonstrate the utility of the algorithms by showing experimentally the ability of an optofluidic phase shifter to generate arbitrary complex intensity distributions.

Adaptive optics for optical microscopy [Invited]

Optical microscopy is widely used to visualize fine structures. When applied to bioimaging, its performance is often degraded by sample-induced aberrations. In recent years, adaptive optics (AO), originally developed to correct for atmosphere-associated aberrations, has been applied to a wide range of microscopy modalities, enabling high- or super-resolution imaging of biological structure and function in complex tissues. Here, we review classic and recently developed AO techniques and their applications in optical microscopy.

The Lattice Geometry of Walsh-Function-Based Adaptive Optics

We show that there is an intrinsic link between the use of Walsh aberration modes in adaptive optics (AO) and the mathematics of lattices. The discrete and binary nature of these modes means that there are infinite combinations of Walsh mode coefficients that can optimally correct the same aberration. Finding such a correction is hence a poorly conditioned optimisation problem that can be difficult to solve. This can be mitigated by confining the AO correction space defined in Walsh mode coefficients to the fundamental Voronoi cell of a lattice. By restricting the correction space in this way, one can ensure there is only one set of Walsh coefficients that corresponds to the optimum correction aberration. This property is used to enable the design of efficient estimation algorithms to solve the inverse problem of finding correction aberrations from a sequence of measurements in a wavefront sensorless AO system. The benefit of this approach is illustrated using a neural-network-based estimator.

Open-Source Biomedical Image Analysis Models: A Meta-Analysis and Continuous Survey

Open-source research software has proven indispensable in modern biomedical image analysis. A multitude of open-source platforms drive image analysis pipelines and help disseminate novel analytical approaches and algorithms. Recent advances in machine learning allow for unprecedented improvement in these approaches. However, these novel algorithms come with new requirements in order to remain open source. To understand how these requirements are met, we have collected 50 biomedical image analysis models and performed a meta-analysis of their respective papers, source code, dataset, and trained model parameters. We concluded that while there are many positive trends in openness, only a fraction of all publications makes all necessary elements available to the research community.

Active Aberration Correction with Adaptive Coefficient SPGD Algorithm for Laser Scanning Confocal Microscope

A laser scanning confocal microscope (LSCM) is an effective scientific instrument for studying sub-micron structures, and it has been widely used in the field of biological detection. However, the illumination depth of LSCMs is limited due to the optical aberrations introduced by living biological tissue, which acts as an optical medium with a non-uniform refractive index, resulting in a significant dispersion of the focus of LSCM illumination light and, hence, a loss in the resolution of the image. In this study, to minimize the effect of optical aberrations, an image-based adaptive optics technology using an optimized stochastic parallel gradient descent (SPGD) algorithm with an adaptive coefficient is applied to the optical path of an LSCM system. The effectiveness of the proposed aberration correction approach is experimentally evaluated in the LSCM system. The results illustrate that the proposed adaptive optics system with an adaptive coefficient SPGD algorithm can effectively reduce the interference caused by aberrations during depth imaging.

Self-calibrated general model-based wavefront sensorless adaptive optics for both point-like and extended objects

The deformable mirror (DM) in conventional model-based wavefront sensorless adaptive optics (WFSless AO) must be calibrated in advance by an additional WFS in order to precisely generate predetermined bias modes with known amplitudes. Although the WFS is unnecessary during correction, it will increase system complexity and may be unavailable in real applications. In this paper, the model-based WFSless AO algorithms, either for point-like or extended objects, are generalized to a unified form and the calibration problem comes down to the measurement of a Gram matrix. We proposed a novel self-calibration procedure to obtain the Gram matrix without using a WFS. The calibrated Gram matrix can be used directly for simultaneous correction if using the influence functions of DM as the bias modes, requiring N+1 images to correct N modes. Alternatively, orthogonal or gradient-orthogonal mirror modes obtained from the eigenvectors of the Gram matrix can be used as the modal basis to implement independent sequential correction that requires 2N images to correct N modes. Simulations and experiments have been done to verify the feasibility of proposed self-calibration and correction methods for both point-like and extended objects in a WFSless AO system.

High-accuracy, direct aberration determination using self-attention-armed deep convolutional neural networks

Optical microscopes have long been essential for many scientific disciplines. However, the resolution and contrast of such microscopic images are dramatically affected by aberrations. In this study, compacted with adaptive optics, we propose a machine learning technique, called the "phase-retrieval deep convolutional neural networks (PRDCNNs)". This aberration determination architecture is direct and exhibits high accuracy and certain generalization ability. Notably, its performance surpasses those of similar, existing methods, with fewer fluctuations and greater robustness against noise. We anticipate future application of the proposed PRDCNNs to super-resolution microscopes. This article is protected by copyright. All rights reserved.

Light Sheet Illumination for 3D Single-Molecule Super-Resolution Imaging of Neuronal Synapses

The function of the neuronal synapse depends on the dynamics and interactions of individual molecules at the nanoscale. With the development of single-molecule super-resolution microscopy over the last decades, researchers now have a powerful and versatile imaging tool for mapping the molecular mechanisms behind the biological function. However, imaging of thicker samples, such as mammalian cells and tissue, in all three dimensions is still challenging due to increased fluorescence background and imaging volumes. The combination of single-molecule imaging with light sheet illumination is an emerging approach that allows for imaging of biological samples with reduced fluorescence background, photobleaching, and photodamage. In this review, we first present a brief overview of light sheet illumination and previous super-resolution techniques used for imaging of neurons and synapses. We then provide an in-depth technical review of the fundamental concepts and the current state of the art in the fields of three-dimensional single-molecule tracking and super-resolution imaging with light sheet illumination. We review how light sheet illumination can improve single-molecule tracking and super-resolution imaging in individual neurons and synapses, and we discuss emerging perspectives and new innovations that have the potential to enable and improve single-molecule imaging in brain tissue.

Experimental determination of shift-less aberration bases for sensorless adaptive optics in nonlinear microscopy

Adaptive optics can improve the performance of optical systems and devices by correcting phase aberrations. While in most applications wavefront sensing is employed to drive the adaptive optics correction, some microscopy methods may require sensorless optimization of the wavefront. In these cases, the correction is performed by describing the aberration as a linear combination of a base of influence functions, optimizing an image quality metric as a function of the coefficients. The influence functions base is generally chosen to either efficiently represent the adaptive device used or to describe generic wavefronts in an orthogonal fashion. A rarely discussed problem is that most correction bases have elements which introduce, together with a correction of the aberration, a shift of the imaging field of view in three dimensions. While simple methods to solve the problem are available for linear microscopy methods, nonlinear microscopy techniques such as multiphoton or second harmonic generation microscopy require non-trivial base determination. In this paper, we discuss the problem, and we present a method for calibrating a shift-less base on a spatial light modulator for two-photon microscopy.

Deep learning enables confocal laser-scanning microscopy with enhanced resolution

Theoretical resolution enhancement of confocal laser-scanning microscopy (CLSM) is sacrificed for the best compromise between optical sectioning and the signal-to-noise ratio (SNR). The pixel reassignment reconstruction algorithm can improve the effective spatial resolution of CLSM to its theoretical limit. However, current implementations are not versatile and are time-consuming or technically complex. Here we present a parameter-free post-processing strategy for laser-scanning microscopy based on deep learning, which enables a spatial resolution enhancement by a factor of ∼1.3, compared to conventional CLSM. To speed up the training process for experimental data, transfer learning, combined with a hybrid dataset consisting of simulated synthetic and experimental images, is employed. The overall resolution and SNR improvement, validated by quantitative evaluation metrics, allowed us to correctly infer the fine structures of real experimental images.

Adaptive optics in super-resolution microscopy

Fluorescence microscopy has become a routine tool in biology for interrogating life activities with minimal perturbation. While the resolution of fluorescence microscopy is in theory governed only by the diffraction of light, the resolution obtainable in practice is also constrained by the presence of optical aberrations. The past two decades have witnessed the advent of super-resolution microscopy that overcomes the diffraction barrier, enabling numerous biological investigations at the nanoscale. Adaptive optics, a technique borrowed from astronomical imaging, has been applied to correct for optical aberrations in essentially every microscopy modality, especially in super-resolution microscopy in the last decade, to restore optimal image quality and resolution. In this review, we briefly introduce the fundamental concepts of adaptive optics and the operating principles of the major super-resolution imaging techniques. We highlight some recent implementations and advances in adaptive optics for active and dynamic aberration correction in super-resolution microscopy.

Differentiable model-based adaptive optics for two-photon microscopy

Aberrations limit scanning fluorescence microscopy when imaging in scattering materials such as biological tissue. Model-based approaches for adaptive optics take advantage of a computational model of the optical setup. Such models can be combined with the optimization techniques of machine learning frameworks to find aberration corrections, as was demonstrated for focusing a laser beam through aberrations onto a camera [Opt. Express2826436 (26436)10.1364/OE.403487]. Here, we extend this approach to two-photon scanning microscopy. The developed sensorless technique finds corrections for aberrations in scattering samples and will be useful for a range of imaging application, for example in brain tissue.

Blind Deconvolution Based on Compressed Sensing with bi-l0-l2-norm Regularization in Light Microscopy Image

Blind deconvolution of light microscopy images could improve the ability of distinguishing cell-level substances. In this study, we investigated the blind deconvolution framework for a light microscope image, which combines the benefits of bi-l₀-l₂-norm regularization with compressed sensing and conjugated gradient algorithms. Several existing regularization approaches were limited by staircase artifacts (or cartooned artifacts) and noise amplification. Thus, we implemented our strategy to overcome these problems using the bi-l₀-l₂-norm regularization proposed. It was investigated through simulations and experiments using optical microscopy images including the background noise. The sharpness was improved through the successful image restoration while minimizing the noise amplification. In addition, quantitative factors of the restored images, including the intensity profile, root-mean-square error (RMSE), edge preservation index (EPI), structural similarity index measure (SSIM), and normalized noise power spectrum, were improved compared to those of existing or comparative images. In particular, the results of using the proposed method showed RMSE, EPI, and SSIM values of approximately 0.12, 0.81, and 0.88 when compared with the reference. In addition, RMSE, EPI, and SSIM values in the restored image were proven to be improved by about 5.97, 1.26, and 1.61 times compared with the degraded image. Consequently, the proposed method is expected to be effective for image restoration and to reduce the cost of a high-performance light microscope.

Adaptive optics for high-resolution imaging

Adaptive optics (AO) is a technique that corrects for optical aberrations. It was originally proposed to correct for the blurring effect of atmospheric turbulence on images in ground-based telescopes and was instrumental in the work that resulted in the Nobel prize-winning discovery of a supermassive compact object at the centre of our galaxy. When AO is used to correct for the eye's imperfect optics, retinal changes at the cellular level can be detected, allowing us to study the operation of the visual system and to assess ocular health in the microscopic domain. By correcting for sample-induced blur in microscopy, AO has pushed the boundaries of imaging in thick tissue specimens, such as when observing neuronal processes in the brain. In this primer, we focus on the application of AO for high-resolution imaging in astronomy, vision science and microscopy. We begin with an overview of the general principles of AO and its main components, which include methods to measure the aberrations, devices for aberration correction, and how these components are linked in operation. We present results and applications from each field along with reproducibility considerations and limitations. Finally, we discuss future directions.

Super-resolution Microscopy with Single Molecules in Biology and Beyond-Essentials, Current Trends, and Future Challenges

Single-molecule super-resolution microscopy has developed from a specialized technique into one of the most versatile and powerful imaging methods of the nanoscale over the past two decades. In this perspective, we provide a brief overview of the historical development of the field, the fundamental concepts, the methodology required to obtain maximum quantitative information, and the current state of the art. Then, we will discuss emerging perspectives and areas where innovation and further improvement are needed. Despite the tremendous progress, the full potential of single-molecule super-resolution microscopy is yet to be realized, which will be enabled by the research ahead of us.