Wavefront correction for adaptive optics with reflected light and deep neural networks

Universal adaptive optics for microscopy through embedded neural network control

The resolution and contrast of microscope imaging is often affected by aberrations introduced by imperfect optical systems and inhomogeneous refractive structures in specimens. Adaptive optics (AO) compensates these aberrations and restores diffraction limited performance. A wide range of AO solutions have been introduced, often tailored to a specific microscope type or application. Until now, a universal AO solution - one that can be readily transferred between microscope modalities - has not been deployed. We propose versatile and fast aberration correction using a physics-based machine learning assisted wavefront-sensorless AO control (MLAO) method. Unlike previous ML methods, we used a specially constructed neural network (NN) architecture, designed using physical understanding of the general microscope image formation, that was embedded in the control loop of different microscope systems. The approach means that not only is the resulting NN orders of magnitude simpler than previous NN methods, but the concept is translatable across microscope modalities. We demonstrated the method on a two-photon, a three-photon and a widefield three-dimensional (3D) structured illumination microscope. Results showed that the method outperformed commonly-used modal-based sensorless AO methods. We also showed that our ML-based method was robust in a range of challenging imaging conditions, such as 3D sample structures, specimen motion, low signal to noise ratio and activity-induced fluorescence fluctuations. Moreover, as the bespoke architecture encapsulated physical understanding of the imaging process, the internal NN configuration was no-longer a "black box", but provided physical insights on internal workings, which could influence future designs.

Dual convolutional neural network for aberration pre-correction and image quality enhancement in integral imaging display

This paper proposes a method that utilizes a dual neural network model to address the challenges posed by aberration in the integral imaging microlens array (MLA) and the degradation of 3D image quality. The approach involves a cascaded dual convolutional neural network (CNN) model designed to handle aberration pre-correction and image quality restoration tasks. By training these models end-to-end, the MLA aberration is corrected effectively and the image quality of integral imaging is enhanced. The feasibility of the proposed method is validated through simulations and optical experiments, using an optimized, high-quality pre-corrected element image array (EIA) as the image source for 3D display. The proposed method achieves high-quality integral imaging 3D display by alleviating the contradiction between MLA aberration and 3D image resolution reduction caused by system noise without introducing additional complexity to the display system.

Adaptive optics for optical microscopy [Invited]

Optical microscopy is widely used to visualize fine structures. When applied to bioimaging, its performance is often degraded by sample-induced aberrations. In recent years, adaptive optics (AO), originally developed to correct for atmosphere-associated aberrations, has been applied to a wide range of microscopy modalities, enabling high- or super-resolution imaging of biological structure and function in complex tissues. Here, we review classic and recently developed AO techniques and their applications in optical microscopy.

Prediction of electrical properties of FDSOI devices based on deep learning

Fully depleted Silicon on insulator technology (FDSOI) is proposed to solve the various non-ideal effects when the process size of integrated circuits is reduced to 45 nm. The research of traditional FDSOI devices is mostly based on simulation software, which requires a lot of calculation and takes a long time. In this paper, a deep learning (DL) based electrical characteristic prediction method for FDSOI devices is proposed. DL algorithm is used to train the simulation data and establish the relationship between the physical parameters and electrical characteristics of the device. The network structure used in the experiment has high prediction accuracy. The mean square error of electrical parameters and transfer characteristic curve is only 4.34 × 10^-4and 2.44 × 10^-3respectively. This method can quickly and accurately predict the electrical characteristics of FDSOI devices without microelectronic expertise. In addition, this method can be extended to study the effects of various physical variables on device performance, which provides a new research method for the field of microelectronics.

Correction of uneven illumination in color microscopic image based on fully convolutional network

The correction of uneven illumination in microscopic image is a basic task in medical imaging. Most of the existing methods are designed for monochrome images. An effective fully convolutional network (FCN) is proposed to directly process color microscopic image in this paper. The proposed method estimates the distribution of illumination information in input image, and then carry out the correction of the corresponding uneven illumination through a feature encoder module, a feature decoder module, and a detail supplement module. In this process, overlapping residual blocks are designed to better transfer the illumination information, and in particular a well-designed weighted loss function ensures that the network can not only correct the illumination but also preserve image details. The proposed method is compared with some related methods on real pathological cell images qualitatively and quantitatively. Experimental results show that our method achieves the excellent performance. The proposed method is also applied to the preprocessing of whole slide imaging (WSI) tiles, which greatly improves the effect of image mosaicking.

Differentiable model-based adaptive optics for two-photon microscopy

Aberrations limit scanning fluorescence microscopy when imaging in scattering materials such as biological tissue. Model-based approaches for adaptive optics take advantage of a computational model of the optical setup. Such models can be combined with the optimization techniques of machine learning frameworks to find aberration corrections, as was demonstrated for focusing a laser beam through aberrations onto a camera [Opt. Express2826436 (26436)10.1364/OE.403487]. Here, we extend this approach to two-photon scanning microscopy. The developed sensorless technique finds corrections for aberrations in scattering samples and will be useful for a range of imaging application, for example in brain tissue.

Practical sensorless aberration estimation for 3D microscopy with deep learning

Estimation of optical aberrations from volumetric intensity images is a key step in sensorless adaptive optics for 3D microscopy. Recent approaches based on deep learning promise accurate results at fast processing speeds. However, collecting ground truth microscopy data for training the network is typically very difficult or even impossible thereby limiting this approach in practice. Here, we demonstrate that neural networks trained only on simulated data yield accurate predictions for real experimental images. We validate our approach on simulated and experimental datasets acquired with two different microscopy modalities and also compare the results to non-learned methods. Additionally, we study the predictability of individual aberrations with respect to their data requirements and find that the symmetry of the wavefront plays a crucial role. Finally, we make our implementation freely available as open source software in Python.

Differentiable model-based adaptive optics with transmitted and reflected light

Aberrations limit optical systems in many situations, for example when imaging in biological tissue. Machine learning offers novel ways to improve imaging under such conditions by learning inverse models of aberrations. Learning requires datasets that cover a wide range of possible aberrations, which however becomes limiting for more strongly scattering samples, and does not take advantage of prior information about the imaging process. Here, we show that combining model-based adaptive optics with the optimization techniques of machine learning frameworks can find aberration corrections with a small number of measurements. Corrections are determined in a transmission configuration through a single aberrating layer and in a reflection configuration through two different layers at the same time. Additionally, corrections are not limited by a predetermined model of aberrations (such as combinations of Zernike modes). Focusing in transmission can be achieved based only on reflected light, compatible with an epidetection imaging configuration.