Object-independent image-based wavefront sensing approach using phase diversity images and deep learning

Experimental demonstration of wavefront reconstruction and correction techniques for variable targets based on distorted grating and deep learning

This research presents a practical approach for wavefront reconstruction and correction adaptable to variable targets, with the aim of constructing a high-precision, general extended target adaptive optical system. Firstly, we delve into the detailed design of a crucial component, the distorted grating, simplifying the optical system implementation while circumventing potential issues in traditional phase difference-based collection methods. Subsequently, normalized fine features (NFFs) and structure focus features (SFFs) which both are independent of the imaging target but corresponded precisely to the wavefront aberration are proposed. The two features provide a more accurate and robust characterization of the wavefront aberrations. Then, a Noise-to-Denoised Generative Adversarial Network (N2D-GAN) is employed for denoising real images. And a lightweight network, Attention Mechanism-based Efficient Network (AM-EffNet), is applied to achieve efficient and high-precision mapping between features and wavefronts. A prototype of object-independent adaptive optics system is demonstrated by experimental setup, and the effectiveness of this method in wavefront reconstruction for different imaging targets has been verified. This research holds significant relevance for engineering applications of adaptive optics, providing robust support for addressing challenges within practical systems.

Phase diversity-based wavefront sensing for fluorescence microscopy

Fluorescence microscopy is an invaluable tool in biology, yet its performance is compromised when the wavefront of light is distorted due to optical imperfections or the refractile nature of the sample. Such optical aberrations can dramatically lower the information content of images by degrading image contrast, resolution, and signal. Adaptive optics (AO) methods can sense and subsequently cancel the aberrated wavefront, but are too complex, inefficient, slow, or expensive for routine adoption by most labs. Here we introduce a rapid, sensitive, and robust wavefront sensing scheme based on phase diversity, a method successfully deployed in astronomy but underused in microscopy. Our method enables accurate wavefront sensing to less than λ/35 root mean square (RMS) error with few measurements, and AO with no additional hardware besides a corrective element. After validating the method with simulations, we demonstrate calibration of a deformable mirror > 100-fold faster than comparable methods (corresponding to wavefront sensing on the ~100 ms scale), and sensing and subsequent correction of severe aberrations (RMS wavefront distortion exceeding λ/2), restoring diffraction-limited imaging on extended biological samples.

Universal adaptive optics for microscopy through embedded neural network control

The resolution and contrast of microscope imaging is often affected by aberrations introduced by imperfect optical systems and inhomogeneous refractive structures in specimens. Adaptive optics (AO) compensates these aberrations and restores diffraction limited performance. A wide range of AO solutions have been introduced, often tailored to a specific microscope type or application. Until now, a universal AO solution - one that can be readily transferred between microscope modalities - has not been deployed. We propose versatile and fast aberration correction using a physics-based machine learning assisted wavefront-sensorless AO control (MLAO) method. Unlike previous ML methods, we used a specially constructed neural network (NN) architecture, designed using physical understanding of the general microscope image formation, that was embedded in the control loop of different microscope systems. The approach means that not only is the resulting NN orders of magnitude simpler than previous NN methods, but the concept is translatable across microscope modalities. We demonstrated the method on a two-photon, a three-photon and a widefield three-dimensional (3D) structured illumination microscope. Results showed that the method outperformed commonly-used modal-based sensorless AO methods. We also showed that our ML-based method was robust in a range of challenging imaging conditions, such as 3D sample structures, specimen motion, low signal to noise ratio and activity-induced fluorescence fluctuations. Moreover, as the bespoke architecture encapsulated physical understanding of the imaging process, the internal NN configuration was no-longer a "black box", but provided physical insights on internal workings, which could influence future designs.

Object-independent wavefront sensing method based on an unsupervised learning model for overcoming aberrations in optical systems

This Letter introduces the idea of unsupervised learning into object-independent wavefront sensing for the first time, to the best of our knowledge, which can achieve fast phase recovery of arbitrary objects without labels. First, a fine feature extraction method which only depends on the wavefront aberrations is proposed. Then, a lightweight neural network and an optical feature system are combined to form an unsupervised learning model, and the neural network is promoted to be well trained by reversely outputting fine features. Simulation results prove that the proposed method can effectively overcome the aberrations (static or variable) existing in the optical system and achieve wavefront sensing of different objects with high precision and efficiency.

Deep learning-based adaptive optics for light sheet fluorescence microscopy

Light sheet fluorescence microscopy (LSFM) is a high-speed imaging technique that is often used to image intact tissue-cleared specimens with cellular or subcellular resolution. Like other optical imaging systems, LSFM suffers from sample-induced optical aberrations that decrement imaging quality. Optical aberrations become more severe when imaging a few millimeters deep into tissue-cleared specimens, complicating subsequent analyses. Adaptive optics are commonly used to correct sample-induced aberrations using a deformable mirror. However, routinely used sensorless adaptive optics techniques are slow, as they require multiple images of the same region of interest to iteratively estimate the aberrations. In addition to the fading of fluorescent signal, this is a major limitation as thousands of images are required to image a single intact organ even without adaptive optics. Thus, a fast and accurate aberration estimation method is needed. Here, we used deep-learning techniques to estimate sample-induced aberrations from only two images of the same region of interest in cleared tissues. We show that the application of correction using a deformable mirror greatly improves image quality. We also introduce a sampling technique that requires a minimum number of images to train the network. Two conceptually different network architectures are compared; one that shares convolutional features and another that estimates each aberration independently. Overall, we have presented an efficient way to correct aberrations in LSFM and to improve image quality.

Adaptive Optical Closed-Loop Control Based on the Single-Dimensional Perturbation Descent Algorithm

Modal-free optimization algorithms do not require specific mathematical models, and they, along with their other benefits, have great application potential in adaptive optics. In this study, two different algorithms, the single-dimensional perturbation descent algorithm (SDPD) and the second-order stochastic parallel gradient descent algorithm (2SPGD), are proposed for wavefront sensorless adaptive optics, and a theoretical analysis of the algorithms' convergence rates is presented. The results demonstrate that the single-dimensional perturbation descent algorithm outperforms the stochastic parallel gradient descent (SPGD) and 2SPGD algorithms in terms of convergence speed. Then, a 32-unit deformable mirror is constructed as the wavefront corrector, and the SPGD, single-dimensional perturbation descent, and 2SPSA algorithms are used in an adaptive optics numerical simulation model of the wavefront controller. Similarly, a 39-unit deformable mirror is constructed as the wavefront controller, and the SPGD and single-dimensional perturbation descent algorithms are used in an adaptive optics experimental verification device of the wavefront controller. The outcomes demonstrate that the convergence speed of the algorithm developed in this paper is more than twice as fast as that of the SPGD and 2SPGD algorithms, and the convergence accuracy of the algorithm is 4% better than that of the SPGD algorithm.

Adaptive optics for optical microscopy [Invited]

Optical microscopy is widely used to visualize fine structures. When applied to bioimaging, its performance is often degraded by sample-induced aberrations. In recent years, adaptive optics (AO), originally developed to correct for atmosphere-associated aberrations, has been applied to a wide range of microscopy modalities, enabling high- or super-resolution imaging of biological structure and function in complex tissues. Here, we review classic and recently developed AO techniques and their applications in optical microscopy.

Generalization of learned Fourier-based phase-diversity wavefront sensing

Proper initialization of the nonlinear optimization is important to avoid local minima in phase diversity wavefront sensing (PDWS). An effective neural network based on low-frequency coefficients in the Fourier domain has proved effective to determine a better estimate of the unknown aberrations. However, the network relies significantly on the training settings, such as imaging object and optical system parameters, resulting in a weak generalization ability. Here we propose a generalized Fourier-based PDWS method by combining an object-independent network with a system-independent image processing procedure. We demonstrate that a network trained with a specific setting can be applied to any image regardless of the actual settings. Experimental results show that a network trained with one setting can be applied to images with four other settings. For 1000 aberrations with RMS wavefront errors bounded within [0.2 λ, 0.4 λ], the mean RMS residual errors are 0.032 λ, 0.039 λ, 0.035 λ, and 0.037 λ, respectively, and 98.9% of the RMS residual errors are less than 0.05 λ.

Predictive correction method based on deep learning for a phase compensation system with frozen flow turbulence

We propose a deep learning method that includes convolution neural network (CNN) and convolutional long short-term memory (ConvLSTM) models to realize atmospheric turbulence compensation and correction of distorted beams. The trained CNN model can automatically obtain the equivalent turbulent compensation phase screen based on the Gaussian beams affected by turbulence and without turbulence. To solve the time delay problem, we use the ConvLSTM model to predict the atmospheric turbulence evolution and acquire a more accurate compensation phase under the Taylor frozen hypothesis. The experimental results show that the distorted Gaussian and vortex beams are effectively and accurately compensated.

Region-correlation algorithm with improved dynamic range and reconstruction accuracy for extended object wavefront sensing

The correlation Shack-Hartmann wavefront sensor (SHWFS) is widely used in many fields in addition to solar adaptive optics. The requirement for the SHWFS dynamic range increases with the diameter of the telescope, which means a larger detector array is needed. However, the size of the detector would be restricted by the high frame rate needed for the solar observation. To solve this problem, a new, to the best of our knowledge, method called the region-correlation algorithm (RCA) is proposed. In this method, the sub-image array is divided into several regions, and the slopes of sub-apertures are calculated by referring to a selected sub-image in each region. Note that the final slope over a sub-aperture is obtained by combining the relative slopes between the selected sub-image in different regions. The dynamic range in each region is similar to the conventional correlation algorithm, and the final dynamic range of the RCA would be accumulated from those of the regions. The reconstruction accuracy under large aberration would also be improved due to the extended dynamic range. Meanwhile, the RCA does not require any extra device and the increase in calculation time resulting from the RCA is acceptable. The results of numerical simulation and experiment, compared with conventional correlation algorithm, confirm the advantages in the performance of the RCA as well.

Phase-diversity wavefront sensing enhanced by a Fourier-based neural network

Phase diversity wavefront sensing (PDWS) has been a successful approach to quantifying wavefront aberrations with only a few intensity measurements and nonlinear optimization. However, the inherent non-convexity of the inverse problem may lead to stagnation at a local minimum far from the true solution. Proper initialization of the nonlinear optimization is important to avoid local minima and improve wavefront retrieval accuracy. In this paper, we propose an effective neural network based on low-frequency coefficients in the Fourier domain to determine a better estimate of the unknown aberrations. By virtue of the proposed network, only a small amount of simulation data suffice for a robust training, two orders of magnitude less than those in existing work. Experimental results show that, when compared with some existing methods, our method achieves the highest accuracy while drastically reducing the training time to 1.4 min. The minimum, maximum, and mean values of the root mean square (RMS) residual errors for 800 aberrations are 0.017λ, 0.056λ, and 0.039λ, respectively, and 95% of the RMS residual errors are less than 0.05λ.

Jitter-Robust Phase Retrieval Wavefront Sensing Algorithms

Phase retrieval wavefront sensing methods are now of importance for imaging quality maintenance of space telescopes. However, their accuracy is susceptible to line-of-sight jitter due to the micro-vibration of the platform, which changes the intensity distribution of the image. The effect of the jitter shows some stochastic properties and it is hard to present an analytic solution to this problem. This paper establishes a framework for jitter-robust image-based wavefront sensing algorithm, which utilizes two-dimensional Gaussian convolution to describe the effect of jitter on an image. On this basis, two classes of jitter-robust phase retrieval algorithms are proposed, which can be categorized into iterative-transform algorithms and parametric algorithms, respectively. Further discussions are presented for the cases where the magnitude of jitter is unknown to us. Detailed simulations and a real experiment are performed to demonstrate the effectiveness and practicality of the proposed approaches. This work improves the accuracy and practicality of the phase retrieval wavefront sensing methods in the space condition with non-ignorable micro-vibration.

A Target Detection Algorithm for Remote Sensing Images Based on Deep Learning

In order to improve the accuracy of remote sensing image target detection, this paper proposes a remote sensing image target detection algorithm DFS based on deep learning. Firstly, dimension clustering module, loss function, and sliding window segmentation detection are designed. The data set used in the experiment comes from GoogleEarth, and there are 6 types of objects: airplanes, boats, warehouses, large ships, bridges, and ports. Training set, verification set, and test set contain 73490 images, 22722 images, and 2138 images, respectively. It is assumed that the number of detected positive samples and negative samples is A and B, respectively, and the number of undetected positive samples and negative samples is C and D, respectively. The experimental results show that the precision-recall curve of DFS for six types of targets shows that DFS has the best detection effect for bridges and the worst detection effect for boats. The main reason is that the size of the bridge is relatively large, and it is clearly distinguished from the background in the image, so the detection difficulty is low. However, the target of the boat is very small, and it is easy to be mixed with the background, so it is difficult to detect. The MAP of DFS is improved by 12.82%, the detection accuracy is improved by 13%, and the recall rate is slightly decreased by 1% compared with YOLOv2. According to the number of detection targets, the number of false positives (FPs) of DFS is much less than that of YOLOv2. The false positive rate is greatly reduced. In addition, the average IOU of DFS is 11.84% higher than that of YOLOv2. For small target detection efficiency and large remote sensing image detection, the DFS algorithm has obvious advantages.

Joint object classification and turbulence strength estimation using convolutional neural networks

In a recent paper, Kee et al. [Appl. Opt.59, 9434 (2020)APOPAI0003-693510.1364/AO.405663] use a multilayer perceptron neural network to classify objects in imagery after degradation through atmospheric turbulence. They also estimate turbulence strength when prior knowledge of the object is available. In this work, we significantly increase the realism of the turbulence simulation used to train and evaluate the Kee et al. neural network. Second, we develop a new convolutional neural network for joint character classification and turbulence strength estimation, thereby eliminating the prior knowledge constraint. This joint classifier-estimator expands applicability to a broad range of remote sensing problems, where the observer cannot access the object of interest directly.

Cross-iteration multi-step optimization strategy for three-dimensional intensity position correction in phase diverse phase retrieval

Parameters mismatching between the real optical system and phase retrieval model undermines wavefront reconstruction accuracy. The three-dimensional intensity position is corrected in phase retrieval, which is traditionally separated from lateral position correction and axial position correction. In this paper, we propose a three-dimensional intensity position correction method for phase diverse phase retrieval with the cross-iteration nonlinear optimization strategy. The intensity position is optimized via the coarse optimization method at first, then the intensity position is cross-optimized in the iterative wavefront reconstruction process with the exact optimization method. The analytic gradients about the three-dimensional intensity position are derived. The cross-iteration optimization strategy avoids the interference between the incomplete position correction and wavefront reconstruction during the iterative process. The accuracy and robustness of the proposed method are verified both numerically and experimentally. The proposed method achieves robust and accurate intensity position correction and wavefront reconstruction, which is available for wavefront measurement and phase imaging.

Deep learning wavefront sensing for fine phasing of segmented mirrors

Segmented primary mirror provides many crucial important advantages for the construction of extra-large space telescopes. The imaging quality of this class of telescope is susceptible to phasing error between primary mirror segments. Deep learning has been widely applied in the field of optical imaging and wavefront sensing, including phasing segmented mirrors. Compared to other image-based phasing techniques, such as phase retrieval and phase diversity, deep learning has the advantage of high efficiency and free of stagnation problem. However, at present deep learning methods are mainly applied to coarse phasing and used to estimate piston error between segments. In this paper, deep Bi-GRU neural work is introduced to fine phasing of segmented mirrors, which not only has a much simpler structure than CNN or LSTM network, but also can effectively solve the gradient vanishing problem in training due to long term dependencies. By incorporating phasing errors (piston and tip-tilt errors), some low-order aberrations as well as other practical considerations, Bi-GRU neural work can effectively be used for fine phasing of segmented mirrors. Simulations and real experiments are used to demonstrate the accuracy and effectiveness of the proposed methods.

Non-propagation fast phase diverse phase retrieval for wavefront measurement with generalized FFT-based basis function

Phase retrieval is an attractive optical testing method with a simple experimental arrangement. The sampling grids wave propagation computation based on the FFT operations is usually involved in each iterative process for the classical phase retrieval model. In this paper, a novel non-propagation optimization phase retrieval technique with the FFT-based basis function is proposed to accelerate wavefront measurement. The sampling grids wave diffraction propagation computation is converted to matrix-vector products that have small dimensions to reduce the computational burden. The diffraction basis function based on generalized numerical orthogonal polynomial and two-step Fresnel propagation is deduced, which is suitable for the generally shaped pupil. This paper provides a universal non-propagation framework to accelerate phase retrieval which is applicable to the arbitrarily shaped wavefront measurement.

Breadth-first piston diagnosing approach for segmented mirrors through supervised learning of multiple-wavelength images

Piston diagnosing approaches for segmented mirrors via machine-learning have shown great success. However, they are inevitably challenged with 2π ambiguity, and the accuracy is usually influenced by the location and number of submirrors. A piston diagnosing approach for segmented mirrors, which employs the breadth-first search (BFS) algorithm and supervised learning strategies of multi-wavelength images, is investigated. An original kind of object-independent and normalized dataset is generated by the in-focal and defocused images at different wavelengths. Additionally, the segmented mirrors are divided into several sub-models of binary tree and are traversed through the BFS algorithm. Furthermore, two deep image-based convolutional neural networks are constructed for predicting the ranges and values of piston aberrations. Finally, simulations are performed, and the accuracy is independent of the location and number of submirrors. The Pearson correlation coefficients for test sets are above 0.99, and the average root mean square error of segmented mirrors is approximately 0.01λ. This technique allows the piston error between segmented mirrors to be measured without 2π ambiguity. Moreover, it can be used for data collected by a real setup. Furthermore, it can be applied to segmented mirrors with different numbers of submirrors based on the sub-model of a binary tree.

Assisting target recognition through strong turbulence with the help of neural networks

Imaging and target recognition through strong turbulence is regarded as one of the most challenging problems in modern turbulence research. As the aggregated turbulence distortion inevitably degrades remote targets and makes them less recognizable, both adaptive optics approaches and image correction methods will become less effective in retrieving correct attributes of the target. Meanwhile, machine learning (ML)-based algorithms have been proposed and studied using both hardware and software approaches to alleviate turbulence effects. In this work, we propose a straightforward approach that treats images with turbulence distortion as a data augmentation in the training set, and investigate the effectiveness of the ML-assisted recognition outcomes under different turbulence strengths. Retrospectively, we also apply the recognition outcomes to evaluate the turbulence strength through regression techniques. As a result, our study helps to build a deep connection between turbulence distortion and imaging effects through a standard perceptron neural network (NN), where mutual inference between turbulence levels and target recognition rates can be achieved.

Deep residual learning for low-order wavefront sensing in high-contrast imaging systems

Sensing and correction of low-order wavefront aberrations is critical for high-contrast astronomical imaging. State of the art coronagraph systems typically use image-based sensing methods that exploit the rejected on-axis light, such as Lyot-based low order wavefront sensors (LLOWFS); these methods rely on linear least-squares fitting to recover Zernike basis coefficients from intensity data. However, the dynamic range of linear recovery is limited. We propose the use of deep neural networks with residual learning techniques for non-linear wavefront sensing. The deep residual learning approach extends the usable range of the LLOWFS sensor by more than an order of magnitude compared to the conventional methods, and can improve closed-loop control of systems with large initial wavefront error. We demonstrate that the deep learning approach performs well even in low-photon regimes common to coronagraphic imaging of exoplanets.

Sub-Millisecond Phase Retrieval for Phase-Diversity Wavefront Sensor

We propose a convolutional neural network (CNN) based method, namely phase diversity convolutional neural network (PD-CNN) for the speed acceleration of phase-diversity wavefront sensing. The PD-CNN has achieved a state-of-the-art result, with the inference speed about 0.5 ms, while fusing the information of the focal and defocused intensity images. When compared to the traditional phase diversity (PD) algorithms, the PD-CNN is a light-weight model without complicated iterative transformation and optimization process. Experiments have been done to demonstrate the accuracy and speed of the proposed approach.

Image based aberration retrieval using helical point spread functions

A practical method for determining wavefront aberrations in optical systems based on the acquisition of an extended, unknown object is presented. The approach utilizes a conventional phase diversity approach in combination with a pupil-engineered, helical point spread function (PSF) to discriminate the aberrated PSF from the object features. The analysis of the image's power cepstrum enables an efficient retrieval of the aberration coefficients by solving a simple linear system of equations. An extensive Monte Carlo simulation is performed to demonstrate that the approach makes it possible to measure low-order Zernike modes including defocus, primary astigmatism, coma, and trefoil. The presented approach is tested experimentally by retrieving the two-dimensional aberration distribution of a test setup by imaging an extended, unknown scene.

Direct determination of aberration functions in microscopy by an artificial neural network

Adaptive optics relies on the fast and accurate determination of aberrations but is often hindered by wavefront sensor limitations or lengthy optimization algorithms. Deep learning by artificial neural networks has recently been shown to provide determination of aberration coefficients from various microscope metrics. Here we numerically investigate the direct determination of aberration functions in the pupil plane of a high numerical aperture microscope using an artificial neural network. We show that an aberration function can be determined from fluorescent guide stars and used to improve the Strehl ratio without the need for reconstruction from Zernike polynomial coefficients.

Does deep learning always outperform simple linear regression in optical imaging?

Deep learning has been extensively applied in many optical imaging problems in recent years. Despite the success, the limitations and drawbacks of deep learning in optical imaging have been seldom investigated. In this work, we show that conventional linear-regression-based methods can outperform the previously proposed deep learning approaches for two black-box optical imaging problems in some extent. Deep learning demonstrates its weakness especially when the number of training samples is small. The advantages and disadvantages of linear-regression-based methods and deep learning are analyzed and compared. Since many optical systems are essentially linear, a deep learning network containing many nonlinearity functions sometimes may not be the most suitable option.

Object-independent piston diagnosing approach for segmented optical mirrors via deep convolutional neural network

Piston diagnosing approaches based on neural networks have shown great success, while a few methods are heavily dependent on the imaging target of the optical system. In addition, they are inevitably faced with the interference of submirrors. Therefore, a unique object-independent feature image is used to form an original kind of data set. Besides, an extremely deep image-based convolutional neural network (CNN) of 18 layers is constructed. Furthermore, 9600 images are generated as a data set for each submirror with a special measure of sensitive area extracting. The diversity of results among all the submirrors is also analyzed to ensure generalization ability. Finally, the average root mean square error of six submirrors between the real piston values and the predicted values is approximately 0.0622λ. Our approach has the following characteristics: (1) the data sets are object-independent and contain more effective details, which behave comparatively better in CNN training; (2) the complex network is deep enough and only a limited number of images are required; (3) the method can be applied to the piston diagnosing of segmented mirror to overcome the difficulty brought by the interference of submirrors. Our method does not require special hardware, and is fast to be used at any time, which may be widely applied in piston diagnosing of segmented mirrors.