Fast and robust misalignment correction of Fourier ptychographic microscopy for full field of view reconstruction

Self-calibrating Fourier ptychographic microscopy using automatic differentiation

Fourier ptychographic microscopy (FPM) can provide high-throughput imaging by computationally combining low-resolution images at different spatial frequencies within the Fourier domain. The core algorithm for FPM reconstruction draws upon phase retrieval techniques, including methods such as the ptychographic iterative engine (PIE), regularized PIE (rPIE), and embedded pupil function FPM (EPRY-FPM). The calibration of the physical setup plays a crucial role in the quality of the reconstructed high space-bandwidth product (SPB) image. Despite advances, many methods, incorporating either machine learning or calibration algorithm, face challenges. These include the need for extensive parameter tuning and extra optical system information, hindering their practical use. To address these limitations, we introduce a novel, to the best of our knowledge, self-calibrating FPM reconstruction approach that utilizes automatic differentiation. This method diverges from traditional iterative phase and amplitude updates, opting instead to simultaneously recover a complex 2D image and refine the optical system's physical parameters. Our approach matches the effectiveness of existing recovery techniques while significantly reducing the calibration burden. In this report, we will demonstrate our method is capable of self-calibrating without needing extra system information. We validate our algorithm's performance through numerical simulations and then show its practicality by reconstructing a full field of view of cervical cell slides using ultraviolet Fourier ptychographic microscopy (UV-FPM).

Transformed pupil-function misalignment calibration strategy for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is an enabling quantitative phase imaging technique with both high-resolution (HR) and wide field-of-view (FOV), which can surpass the diffraction limit of the objective lens by employing an LED array to provide angular-varying illumination. The precise illumination angles are critical to ensure exact reconstruction, while it's difficult to separate actual positional parameters in conventional algorithmic self-calibration approaches due to the mixing of multiple systematic error sources. In this paper, we report a pupil-function-based strategy for independently calibrating the position of LED array. We first deduce the relationship between positional deviation and pupil function in the Fourier domain through a common iterative route. Then, we propose a judgment criterion to determine the misalignment situations, which is based on the arrangement of LED array in the spatial domain. By combining the mapping of complex domains, we can accurately solve the spatial positional parameters concerning the LED array through a boundary-finding scheme. Relevant simulations and experiments demonstrate the proposed method is accessible to precisely correct the positional misalignment of LED array. The approach based on the pupil function is expected to provide valuable insights for precise position correction in the field of microscopy.

Fast and robust Fourier ptychographic microscopy with position misalignment correction

Significance

Fourier ptychographic microscopy (FPM) is a new, developing computational imaging technology. It can realize the quantitative phase imaging of a wide field of view and high-resolution (HR) simultaneously by means of multi-angle illumination via a light emitting diode (LED) array, combined with a phase recovery algorithm and the synthetic aperture principle. However, in the FPM reconstruction process, LED position misalignment affects the quality of the reconstructed image, and the reconstruction efficiency of the existing LED position correction algorithms needs to be improved.

Aim

This study aims to improve the FPM correction method based on simulated annealing (SA) and proposes a position misalignment correction method (AA-C algorithm) using an improved phase recovery strategy.

Approach

The spectrum function update strategy was optimized by adding an adaptive control factor, and the reconstruction efficiency of the algorithm was improved.

Results

The experimental results show that the proposed method is effective and robust for position misalignment correction of LED arrays in FPM, and the convergence speed can be improved by 21.2% and 54.9% compared with SC-FPM and PC-FPM, respectively.

Conclusions

These results can reduce the requirement of the FPM system for LED array accuracy and improve robustness.

LED array microscopy system correction method with comprehensive error parameters optimized by phase smoothing criterion

LED array microscopy is a novel computational imaging technique that can achieve two-dimensional (2D) phase imaging and three-dimensional (3D) refractive index imaging with both high resolution and a large field of view. Although its experimental setup is simple, the errors caused by LED array position and light source central wavelength obviously decrease the quality of reconstructed results. To solve this problem, comprehensive error parameters optimized by the phase smoothing criterion are put forward in this paper. The central wavelength error and 3D misalignment model with six freedom degree errors of LED array are considered as the comprehensive error parameters when the spatial positional and optical features of arbitrarily placed LED array are unknown. Phase smoothing criterion is also introduced to the cost function for optimizing comprehensive error parameters to improve the convergence results. Compared with current system correction methods, the simulation and experimental results show that the proposed method in this paper has the best reconstruction accuracy, which can be well applied to an LED array microscope system with unknown positional and optical features of the LED array.

Hybrid full-pose parameter calibration of a freeform illuminator for Fourier ptychographic microscopy

As a typical computational method, Fourier ptychographic microscopy (FPM) can realize high spatial resolution and quantitative phase imaging while preserving the large field of view with a low numerical aperture (NA) objective. A programmable light-emitting diode (LED) array is used as a typical illuminator in an FPM system, and the illumination parameters of each LED element are crucial to the success of the FPM reconstruction algorithm. Compared with LED arrays arranged in rectangular arrays, LED arrays with special structures such as domes or rings can effectively improve FPM imaging results and imaging efficiency. As a trade-off, their calibration difficulty is greatly increased due to the lack of geometric constraints of rectangular arrays. In this paper, we propose an effective hybrid full-pose parameter calibration method for freeform LED array illuminators, combining stereoscopic 3D imaging techniques and the geometric constraints of the microscopic platform. First, a stereovision system is used to obtain the accurate 3D position of each LED element of the freeform illuminator and to construct a rigid 3D coordinate LED array system. Then, calibration between the coordinate system of the LED array and that of the optical imaging component is realized according to the geometric features of the brightfield-to-darkfield edges. Finally, we verify the feasibility and effectiveness of the proposed method through full-pose parameter calibration of LED arrays with different arrangement rules.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

Robust full-pose-parameter estimation for the LED array in Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) can achieve quantitative phase imaging with a large space-bandwidth product by synthesizing a set of low-resolution intensity images captured under angularly varying illuminations. Determining accurate illumination angles is critical because the consistency between actual systematic parameters and those used in the recovery algorithm is essential for high-quality imaging. This paper presents a full-pose-parameter and physics-based method for calibrating illumination angles. Using a physics-based model constructed with general knowledge of the employed microscope and the brightfield-to-darkfield boundaries inside captured images, we can solve for the full-pose parameters of misplaced LED array, which consist of the distance between the sample and the LED array, two orthogonal lateral shifts, one in-plane rotation angle, and two tilt angles, to correct illumination angles precisely. The feasibility and effectiveness of the proposed method for recovering random or remarkable pose parameters have been demonstrated by both qualitative and quantitative experiments. Due to the completeness of the pose parameters, the clarity of the physical model, and the high robustness for arbitrary misalignments, our method can significantly facilitate the design, implementation, and application of concise and robust FPM platforms.

Accurate and stable two-step LED position calibration method for Fourier ptychographic microscopy

SIGNIFICANCE

Fourier ptychography microscopy (FPM) is a computational optical imaging technology that employs angularly varying illuminations and a phase retrieval algorithm to achieve a wide field of view and high-resolution imaging simultaneously. In the FPM, LED position error will reduce the quality of the reconstructed high-resolution image. To correct the LED positions, current methods consider each of the LED positions as independent and use an optimization algorithm to get each of the positions. When the positional misalignment is large or the search position falls into a local optimal value, the current methods may lack stability and accuracy.

AIM

We improve the model of the LED position and propose an accurate and stable two-step correction scheme (tcFPM) to calibrate the LED position error.

APPROACH

The improved LED positions model combines the overall offset, which represents the relative deviation of the LED array and the optical axis, with the slight deviation of each LED's independent position. In the tcFPM, the overall offset of the LED array is corrected at first, which obtains an approximate value of the overall offset of the LED array. Then the position of each LED is precisely adjusted, which obtains the slight offset of each LED.

RESULTS

This LED position error model is more in line with the actual situation. The simulation and experimental results show that the method has high accuracy in correcting the LED position. Furthermore, the reconstruction process of tcFPM is more stable and significantly improves the quality of the reconstruction results, which is compared with some LED position error correction methods.

CONCLUSIONS

An LED position error correction technology is proposed, which has a stable iterative process and improves the reconstruction accuracy of complex amplitude.

Double-flow convolutional neural network for rapid large field of view Fourier ptychographic reconstruction

Fourier ptychographic microscopy is a promising imaging technique which can circumvent the space-bandwidth product of the system and achieve a reconstruction result with wide field-of-view (FOV), high-resolution and quantitative phase information. However, traditional iterative-based methods typically require multiple times to get convergence, and due to the wave vector deviation in different areas, the millimeter-level full-FOV cannot be well reconstructed once and typically required to be separated into several portions with sufficient overlaps and reconstructed separately, which makes traditional methods suffer from long reconstruction time for a large-FOV (of the order of minutes) and limits the application in real-time large-FOV monitoring of live sample in vitro. Here we propose a novel deep-learning based method called DFNN which can be used in place of traditional iterative-based methods to increase the quality of single large-FOV reconstruction and reducing the processing time from 167.5 to 0.1125 second. In addition, we demonstrate that by training based on the simulation dataset with high-entropy property (Opt. Express 28, 24 152 [2020]), DFNN could has fine generalizability and little dependence on the morphological features of samples. The superior robustness of DFNN against noise is also demonstrated in both simulation and experiment. Furthermore, our model shows more robustness against the wave vector deviation. Therefore, we could achieve better results at the edge areas of a single large-FOV reconstruction. Our method demonstrates a promising way to perform real-time single large-FOV reconstructions and provides further possibilities for real-time large-FOV monitoring of live samples with sub-cellular resolution.

Single full-FOV reconstruction Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique that has high-resolution and wide field-of-view (FOV). FPM bypasses the NA limit of the system by stitching a number of variable-illuminated measured images in Fourier space. On the basis of the wide FOV of the low NA objective, the high-resolution image with a wide FOV can be reconstructed through the phase recovery algorithm. However, the high-resolution reconstruction images are affected by the LED array point light source. The results are: (1) the intensities collected by the sample are severely declined when edge LEDs illuminate the sample; (2) the multiple reconstructions are caused by wavevectors inconsistency for the full FOV images. Here, we propose a new lighting scheme termed full FOV Fourier ptychographic microscopy (F³PM). By combining the LED array and telecentric lens, the method can provide plane waves with different angles while maintaining uniform intensity. Benefiting from the telecentric performance and f‒θ property of the telecentric lens, the system stability is improved and the relationship between the position of LED and its illumination angle is simplified. The excellent plane wave provided by the telecentric lens guarantees the same wavevector in the full FOV, and we use this wavevector to reconstruct the full FOV during one time. The area and diameter of the single reconstruction FOV reached 14.6mm ² and 5.4 mm, respectively, and the diameter is very close to the field number (5.5 mm) of the 4× objective. Compared with the traditional FPM, we have increased the diameter of FOV in a single reconstruction by ∼ 10 times, eliminating the complicated steps of computational redundancy and image stitching.

Forward imaging neural network with correction of positional misalignment for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a computational imaging technology used to achieve high-resolution imaging with a wide field-of-view. The existing methods of FPM suffer from the positional misalignment in the system, by which the quality of the recovered high-resolution image is determined. In this paper, a forward neural network method with correction of the positional misalignment (FNN-CP) is proposed based on TensorFlow, which consists of two models. Both the spectrum of the sample and four global position factors, which are introduced to describe the positions of the LED elements, are treated as the learnable weights in layers in the first model. By minimizing the loss function in the training process, the positional error can be corrected based on the trained position factors. In order to fit the wavefront aberrations caused by optical components in the FPM system for better recovery results, the second model is designed, in which the spectrum of the sample and coefficients of different Zernike modes are treated as the learnable weights in layers. After the training process of the second model, the wavefront aberration can be fit according to the coefficients of different Zernike modes and the high-resolution complex image can be obtained based on the trained spectrum of the sample. Both the simulation and experiment have been performed to verify the effectiveness of our proposed method. Compared with the state-of-art FPM methods based on forward neural network, FNN-CP can achieve the best reconstruction results.

Phase imaging with an untrained neural network

Most of the neural networks proposed so far for computational imaging (CI) in optics employ a supervised training strategy, and thus need a large training set to optimize their weights and biases. Setting aside the requirements of environmental and system stability during many hours of data acquisition, in many practical applications, it is unlikely to be possible to obtain sufficient numbers of ground-truth images for training. Here, we propose to overcome this limitation by incorporating into a conventional deep neural network a complete physical model that represents the process of image formation. The most significant advantage of the resulting physics-enhanced deep neural network (PhysenNet) is that it can be used without training beforehand, thus eliminating the need for tens of thousands of labeled data. We take single-beam phase imaging as an example for demonstration. We experimentally show that one needs only to feed PhysenNet a single diffraction pattern of a phase object, and it can automatically optimize the network and eventually produce the object phase through the interplay between the neural network and the physical model. This opens up a new paradigm of neural network design, in which the concept of incorporating a physical model into a neural network can be generalized to solve many other CI problems.

Reflective Fourier ptychographic microscopy using a parabolic mirror

Fourier ptychography uses a phase retrieval algorithm to reconstruct a high-resolution image with a wide field-of-view. Reflective-type Fourier ptychographic microscopy (FPM) is expected to be very useful for surface inspection, but the reported methods have several limitations. We propose a darkfield illuminator for reflective FPM consisting of a parabolic mirror and a flat LED panel. This increases the signal-to-noise ratio of the acquired images because the normal beam of each LED is directed toward the object. Furthermore, the LEDs do not have to be far from the object because they are collimated by the parabolic surface before illumination. Based on this, a reflective FPM with a synthesized numerical aperture (NA) of 1.06 was achieved, which is the highest value by reflective FPM as far as we know. To validate this experimentally, we measured a USAF reflective resolution target and reconstructed a high-resolution image. This resolved up to the period of 488 nm, which corresponds to the synthesized NA. Additionally, an integrated circuit was measured to demonstrate the effectiveness of surface inspection of the proposed system.

A Simply Equipped Fourier Ptychography Platform Based on an Industrial Camera and Telecentric Objective

Fourier ptychography microscopy (FPM) is a recently emerged computational imaging method, which combines the advantages of synthetic aperture and phase retrieval to achieve super-resolution microscopic imaging. FPM can bypass the diffraction limit of the numerical aperture (NA) system and achieve complex images with wide field of view and high resolution (HR) on the basis of the existing microscopic platform, which has low resolution and wide field of view. Conventional FPM platforms are constructed based on basic microscopic platform and a scientific complementary metal-oxide-semiconductor (sCMOS) camera, which has ultrahigh dynamic range. However, sCMOS, or even the microscopic platform, is too expensive to afford for some researchers. Furthermore, the fixed microscopic platform limits the space for function expansion and system modification. In this work, we present a simply equipped FPM platform based on an industrial camera and telecentric objective, which is much cheaper than sCMOS camera and microscopic platform and has accurate optical calibration. A corresponding algorithm was embedded into a conventional FP framework to overcome the low dynamic range of industrial cameras. Simulation and experimental results showed the feasibility and good performance of the designed FPM platform and algorithms.

Neural network model combined with pupil recovery for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed imaging approach aiming at circumventing the limitation of the space-bandwidth product (SBP) and acquiring a complex image with both wide field and high resolution. So far, in many algorithms that have been proposed to solve the FPM reconstruction problem, the pupil function is set to be a fixed value such as the coherent transfer function (CTF) of the system. However, the pupil aberration of the optical components in an FPM imaging system can significantly degrade the quality of the reconstruction results. In this paper, we build a trainable network (FINN-P) which combines the pupil recovery with the forward imaging process of FPM based on TensorFlow. Both the spectrum of the sample and pupil function are treated as the two-dimensional (2D) learnable weights of layers. Therefore, the complex object information and pupil function can be obtained simultaneously by minimizing the loss function in the training process. Simulated datasets are used to verify the effectiveness of pupil recovery, and experiments on the open source measured dataset demonstrate that our method can achieve better reconstruction results even in the presence of a large aberration. In addition, the recovered pupil function can be used as a good estimate before further analysis of the system optical transmission capability.

3D Imaging Based on Depth Measurement Technologies

Three-dimensional (3D) imaging has attracted more and more interest because of its widespread applications, especially in information and life science. These techniques can be broadly divided into two types: ray-based and wavefront-based 3D imaging. Issues such as imaging quality and system complexity of these techniques limit the applications significantly, and therefore many investigations have focused on 3D imaging from depth measurements. This paper presents an overview of 3D imaging from depth measurements, and provides a summary of the connection between the ray-based and wavefront-based 3D imaging techniques.