High-resolution and large field-of-view Fourier ptychographic microscopy and its applications in biomedicine

Observing multi-frequency structured illumination patterns based on an evanescent field in a millimeter-scale polymer slide

Millimeter-scale slide optical waveguides (OWGs) show the potential to break the barrier of easy-to-use and versatility for total internal reflection (TIR) fluorescence technology. In this paper, multi-frequency structured illumination (SI) patterns resulting from the evanescent field (EF) on the surface of a millimeter-scale polymer slide OWG are observed by measuring the fluorescence intensity distribution of fluorescent dyes deposited on the top of the OWG. The frequency, intensity, and stability of the SI patterns show a strong dependence on the coupling angle of the incident light (changing with the incident position). The distribution of multi-frequency SI patterns in the frequency space is demonstrated for different numerical aperture (NA) imaging systems (NA = 0.3, 0.6, and 0.8), indicating the potential for enhanced resolution for low NA systems with a simple and cheap polymer slide.

Enhanced tissue slide imaging in the complex domain via cross-explainable GAN for Fourier ptychographic microscopy

Achieving microscopy with large space-bandwidth products plays a key role in diagnostic imaging and is widely significant in the overall field of clinical practice. Among quantitative microscopy techniques, Fourier Ptychography (FP) provides a wide field of view and high-resolution images, suitable to the histopathological field, but onerous in computational terms. Artificial intelligence can be a solution in this sense. In particular, this research delves into the application of Generative Adversarial Networks (GAN) for the dual-channel complex FP image enhancement of human kidney samples. The study underscores the GANs' efficacy in promoting biological architectures in FP domain, thereby still guaranteeing high resolution and visibility of detailed microscopic structures. We demonstrate successful GAN-based enhanced reconstruction through two strategies: cross-explainability and expert survey. The cross-explainability is evaluated through the comparison of explanation maps for both real and imaginary components underlining its robustness. This comparison further shows that their interplay is pivotal for accurate reconstruction without hallucinations. Secondly, the enhanced reconstruction accuracy and effectiveness in a clinical workflow are confirmed through a two-step survey conducted with nephrologists.

Development and Assessment of Multiple Illumination Color Fourier Ptychographic Microscopy for High Throughput Sample Digitization

In this study, we proposed a multiplexed color illumination strategy to improve the data acquisition efficiency of Fourier ptychography microscopy (FPM). Instead of sequentially lighting up one single channel LED, our method turns on multiple white light LEDs for each image acquisition via a color camera. Thus, each raw image contains multiplexed spectral information. An FPM prototype was developed, which was equipped with a 4×/0.13 NA objective lens to achieve a spatial resolution equivalent to that of a 20×/0.4 NA objective lens. Both two- and four-LED illumination patterns were designed and applied during the experiments. A USAF 1951 resolution target was first imaged under these illumination conditions, based on which MTF curves were generated to assess the corresponding imaging performance. Next, H&E tissue samples and analyzable metaphase chromosome cells were used to evaluate the clinical utility of our strategy. The results show that the single and multiplexed (two- or four-LED) illumination results achieved comparable imaging performance on all the three channels of the MTF curves. Meanwhile, the reconstructed tissue or cell images successfully retain the definition of cell nuclei and cytoplasm and can better preserve the cell edges as compared to the results from the conventional microscopes. This study initially validates the feasibility of multiplexed color illumination for the future development of high-throughput FPM scanning systems.

Fourier Ptychographic Microscopy Reconstruction Method Based on Residual Local Mixture Network

Fourier Ptychographic Microscopy (FPM) is a microscopy imaging technique based on optical principles. It employs Fourier optics to separate and combine different optical information from a sample. However, noise introduced during the imaging process often results in poor resolution of the reconstructed image. This article has designed an approach based on a residual local mixture network to improve the quality of Fourier ptychographic reconstruction images. By incorporating channel attention and spatial attention into the FPM reconstruction process, the network enhances the efficiency of the network reconstruction and reduces the reconstruction time. Additionally, the introduction of the Gaussian diffusion model further reduces coherent artifacts and improves image reconstruction quality. Comparative experimental results indicate that this network achieves better reconstruction quality, and outperforming existing methods in both subjective observation and objective quantitative evaluation.

High-precision spatiotemporal three-dimensional ultrashort pulse synchronization with optical Kerr effect

In studying the interaction of multiple ultrashort pulses with matter, high requirements are put forward for spatiotemporal synchronization accuracy. Limited by the response time and bandwidth of existing devices, the synchronization of multiple ultrashort pulses still faces significant difficulties. By observing the transient phenomena of the optical Kerr effect, high-precision, three-dimensional (x, y, t) synchronization of ultrashort pulses at different angles was achieved. In the optical Kerr effect, the polarization state of the signal pulse changes only when it coincides with the pump pulse, at which point the signal pulse passes through the analyzer. The changes in the intensity and phase of the signal pulse is positively correlated with the degree of spatiotemporal coincidence. In this study, 10-ps pulses were used in the experiments. By observing the intensity and phase distribution of the signal pulses, a time synchronization accuracy between two pulses of less than 1 ps and spatial synchronization accuracy of ±125 µm and ±3 µm in the x and y directions, respectively, were achieved. Moreover, the synchronization of two pulses at an angle of 90 ° was measured, further proving that the method can achieve the spatiotemporal synchronization of pulses with large angles. Therefore, this method has important application prospects in the study of multi-beam interactions with matter and other ultrafast physical phenomena.

Depth-of-field extended Fourier ptychographic microscopy without defocus distance priori

Fourier ptychographic microscopy (FPM) provides a solution of high-throughput phase imaging. Thanks to its coherent imaging model, FPM has the capacity of depth-of-field (DOF) extension by simultaneously recovering the sample's transmittance function and pupil aberration, which contains a defocus term. However, existing phase retrieval algorithms (PRs) often struggle in the presence of a significant defocus. In this Letter, different PRs with embedded pupil recovery are compared, and the one based on the alternating direction multiplier method (ADMM-FPM) demonstrates promising potential for reconstructing highly defocused FPM images. Besides, we present a plug-and-play framework that integrates ADMM-FPM and total variation or Hessian denoiser for pupil function enhancement. Both simulations and experiments demonstrate that this framework enables robust reconstruction of defocused FPM images without any prior knowledge of defocus distance or sample characteristics. In experiments involving USAF 1951 targets and pathologic slides, ADMM-FPM combined with the Hessian denoiser successfully corrected the defocus up to approximately 200 µm, i.e., extending the DOF to 400 µm.

Spatially-coded Fourier ptychography: flexible and detachable coded thin films for quantitative phase imaging with uniform phase transfer characteristics

Fourier ptychography (FP) is an enabling imaging technique that produces high-resolution complex-valued images with extended field coverages. However, when FP images a phase object with any specific spatial frequency, the captured images contain only constant values, rendering the recovery of the corresponding linear phase ramp impossible. This challenge is not unique to FP but also affects other common microscopy techniques -- a rather counterintuitive outcome given their widespread use in phase imaging. The underlying issue originates from the non-uniform phase transfer characteristic inherent in microscope systems, which impedes the conversion of object wavefields into discernible intensity variations. To address this challenge, we present spatially-coded Fourier ptychography (scFP), a new method that synergizes FP with spatial-domain coded detection for true quantitative phase imaging. In scFP, a flexible and detachable coded thin film is attached atop the image sensor in a regular FP setup. The spatial modulation of this thin film ensures a uniform phase response across the entire synthetic bandwidth. It improves reconstruction quality and corrects refractive index underestimation issues prevalent in conventional FP and related tomographic implementations. The inclusion of the coded thin film further adds a new dimension of measurement diversity in the spatial domain. The development of scFP is expected to catalyse new research directions and applications for phase imaging, emphasizing the need for true quantitative accuracy with uniform frequency response.

Mixed-state ptychography for quantitative optical properties measurement of vector beam

Recent advances in ptychography have extended to anisotropic specimens, but vectorial reconstruction of probes owing to polarization aliasing remains a challenge. A polarization-sensitive ptychography that enables full optical property measurement of vector light is proposed. An optimized reconstruction strategy, first calibrating the propagation direction and then performing faithful retrieval, is established. This method avoids multiple image acquisitions with various polarizer configurations and significantly improves the measurement accuracy by correlating the intensity and position of different polarization components. The capability of the proposed method to quantify anisotropic parameters of optical materials and polarization properties of vector probe is demonstrated by experiment.

Fourier Ptychographic Microscopy 10 Years on: A Review

Fourier ptychographic microscopy (FPM) emerged as a prominent imaging technique in 2013, attracting significant interest due to its remarkable features such as precise phase retrieval, expansive field of view (FOV), and superior resolution. Over the past decade, FPM has become an essential tool in microscopy, with applications in metrology, scientific research, biomedicine, and inspection. This achievement arises from its ability to effectively address the persistent challenge of achieving a trade-off between FOV and resolution in imaging systems. It has a wide range of applications, including label-free imaging, drug screening, and digital pathology. In this comprehensive review, we present a concise overview of the fundamental principles of FPM and compare it with similar imaging techniques. In addition, we present a study on achieving colorization of restored photographs and enhancing the speed of FPM. Subsequently, we showcase several FPM applications utilizing the previously described technologies, with a specific focus on digital pathology, drug screening, and three-dimensional imaging. We thoroughly examine the benefits and challenges associated with integrating deep learning and FPM. To summarize, we express our own viewpoints on the technological progress of FPM and explore prospective avenues for its future developments.

Fast Fourier ptychographic quantitative phase microscopy for in vitro label-free imaging

Quantitative phase microscopy (QPM) is indispensable in biomedical research due to its advantages in unlabeled transparent sample thickness quantification and obtaining refractive index information. Fourier ptychographic microscopy (FPM) is among the most promising QPM methods, incorporating multi-angle illumination and iterative phase recovery for high-resolution quantitative phase imaging (QPI) of large cell populations over a wide field of-view (FOV) in a single pass. However, FPM is limited by data redundancy and sequential acquisition strategies, resulting in low imaging efficiency, which in turn limits its real-time application in in vitro label-free imaging. Here, we report a fast QPM based on Fourier ptychography (FQP-FPM), which uses an optimized annular downsampling and parallel acquisition strategy to minimize the amount of data required in the front end and reduce the iteration time of the back-end algorithm (3.3% and 4.4% of conventional FPM, respectively). Theoretical and data redundancy analyses show that FQP-FPM can realize high-throughput quantitative phase reconstruction at thrice the resolution of the coherent diffraction limit by acquiring only ten raw images, providing a precondition for in vitro label-free real-time imaging. The FQP-FPM application was validated for various in vitro label-free live-cell imaging. Cell morphology and subcellular phenomena in different periods were observed with a synthetic aperture of 0.75 NA at a 10× FOV, demonstrating its advantages and application potential for fast high-throughput QPI.

Redundant information model for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a computational optical imaging technique that overcomes the traditional trade-off between resolution and field of view (FOV) by exploiting abundant redundant information in both spatial and frequency domains for high-quality image reconstruction. However, the redundant information in FPM remains ambiguous or abstract, which presents challenges to further enhance imaging capabilities and deepen our understanding of the FPM technique. Inspired by Shannon's information theory and extensive experimental experience in FPM, we defined the specimen complexity and reconstruction algorithm utilization rate and reported a model of redundant information for FPM to predict reconstruction results and guide the optimization of imaging parameters. The model has been validated through extensive simulations and experiments. In addition, it provides a useful tool to evaluate different algorithms, revealing a utilization rate of 24%±1% for the Gauss-Newton algorithm, LED Multiplexing, Wavelength Multiplexing, EPRY-FPM, and GS. In contrast, mPIE exhibits a lower utilization rate of 19%±1%.

Improving the resolution of Fourier ptychographic imaging using an a priori neural network

In this paper, we propose a dual-structured prior neural network model that independently restores both the amplitude and phase image using a random latent code for Fourier ptychography (FP). We demonstrate that the inherent prior information within the neural network can generate super-resolution images with a resolution that exceeds the combined numerical aperture of the FP system. This method circumvents the need for a large labeled dataset. The training process is guided by an appropriate forward physical model. We validate the effectiveness of our approach through simulations and experimental data. The results suggest that integrating image prior information with system-collected data is a potentially effective approach for improving the resolution of FP systems.

Mathematical uniqueness of multimode ptychographic imaging

By writing diffracted intensities as a set of linear equations with the self-correlation of sample's Fourier components as unknown terms and the self-correlation of illumination's Fourier components as coefficients, it was found that the number of unknown terms to be determined is much larger in partially coherent PIE than that in purely coherent PIE. When a partially coherent illumination composed of N modes was applied a unique reconstruction can be determined by scanning the sample to at least 4N positions and recording 4N frames of diffraction patterns. While mathematically illustrating the physical mechanism of multimode ptychography and numerically demonstrating its capability in generating unique reconstruction under partially coherent illumination, this study showed for the first time that multimode ptychography could be an analytic imaging method.

Fourier ptychographic and deep learning using breast cancer histopathological image classification

Automated, as well as accurate classification with breast cancer histological images, was crucial for medical applications because of detecting malignant tumors via histopathological images. In this work create a Fourier ptychographic (FP) and deep learning using breast cancer histopathological image classification. Here the FP method used in the process begins with such a random guess that builds a high-resolution complex hologram, subsequently uses iterative retrieval using FP constraints to stitch around each other low-resolution multi-view means of production owned from either the hologram's high-resolution hologram's elemental images captured via integral imaging. Next, the feature extraction process includes entropy, geometrical features, and textural features. The entropy-based normalization is used to optimize the features. Finally, it attains the classification process of the proposed ENDNN classifies the breast cancer images into normal or abnormal. The experimental outcomes demonstrate that our presented technique overtakes the traditional techniques.

Design of Fourier ptychographic illuminator for single full-FOV reconstruction

Fourier ptychographic microscopy (FPM) is a spatial-temporal-modulation high-throughput imaging technique via a sequential angle-varied LED illumination. Therefore, the illuminator is one of the key components and the design of this illuminator is significant. However, because of the property of spherical wave, partial coherence, and aperture-induced vignetting, the acquired images must be processed in blocks first, and rely on parallel reconstruction via a graphics processing unit (GPU). The high cost makes it unappealing compared with commercial whole slide imaging system via a low-cost central processing unit (CPU). Especially, the vignetting severely destroys the space-invariant model and induces obvious artifacts in FPM, which is the most difficult problem. The conventional method is to divide the field of view (FOV) into many tiles and omit those imperfect images, which is crude and may discards low frequency information. In this paper, we reevaluated the conditions of vignetting in FPM. Through our analysis, the maximum side length of FOV is 0.759 mm for a single full-FOV reconstruction via a 4×/0.1 NA objective and a 4 mm spacing LED array in theory, while almost 1.0 mm can be achieved in practice due to the tolerance of algorithm. We found that FPM system can treat the vignetting coefficient Vf below 0.1 as brightfield images and Vf lager than 0.9 as darkfield images, respectively. We reported an optimized distribution for designing an illuminator without vignetting effect according to the off-the-shelf commercial products, which can reconstruct full FOV in one time via a CPU. By adjusting the distribution of LED units, the system could retrieve the object with the side length of FOV up to 3.8 mm for a single full-FOV reconstruction, which achieves the largest FOV that a typical 4×/0.1 NA objective with the field number of 22 mm can afford.

A Physics-Inspired Deep Learning Framework for an Efficient Fourier Ptychographic Microscopy Reconstruction under Low Overlap Conditions

Two-dimensional observation of biological samples at hundreds of nanometers resolution or even below is of high interest for many sensitive medical applications. Recent advances have been obtained over the last ten years with computational imaging. Among them, Fourier Ptychographic Microscopy is of particular interest because of its important super-resolution factor. In complement to traditional intensity images, phase images are also produced. A large set of N raw images (with typically N = 225) is, however, required because of the reconstruction process that is involved. In this paper, we address the problem of FPM image reconstruction using a few raw images only (here, N = 37) as is highly desirable to increase microscope throughput. In contrast to previous approaches, we develop an algorithmic approach based on a physics-informed optimization deep neural network and statistical reconstruction learning. We demonstrate its efficiency with the help of simulations. The forward microscope image formation model is explicitly introduced in the deep neural network model to optimize its weights starting from an initialization that is based on statistical learning. The simulation results that are presented demonstrate the conceptual benefits of the approach. We show that high-quality images are effectively reconstructed without any appreciable resolution degradation. The learning step is also shown to be mandatory.

Image reconstruction for low cost spatial light interference microscopy with fixed and arbitrary phase modulation

During the past decade, spatial light interference microscopy (SLIM) has undergone rapid development, evidenced by its broadening applications in biology and medicine. However, the need for an expensive spatial light modulator (SLM) may limit its adoption, and the requirement for multiple images per plane limits its speed in volumetric imaging. Here we propose to address these issues by replacing the SLM with a mask fabricated from a low cost optical density (OD) filter, and recover high contrast images computationally rather than through phase-shifting. This is done using a specially constructed Wiener filter to recover the object scattering potential. A crucial part of the Wiener filter is estimating the arbitrary phase introduced by the OD filter. Our results demonstrate that not only were we able to estimate the OD filter's phase modulation in situ, but also the contrast of the reconstructed images is greatly improved. Comparisons with other related methods are also performed, with the conclusion that the combination of an inexpensive OD mask and modified Wiener filtering leads to results that are closest to the traditional SLIM setup. Thus, we have demonstrated the feasibility of a low cost, high speed SLIM system utilizing computational phase reconstruction, paving the way for wider adoption of high resolution phase microscopy.

Reflectional quantitative differential phase microscopy using polarized wavefront phase modulation

Quantitative phase microscopy (QPM), as a label-free and nondestructive technique, has been playing an indispensable tool in biomedical imaging and industrial inspection. Herein, we introduce a reflectional quantitative differential phase microscopy (termed RQDPM) based on polarized wavefront phase modulation and partially coherent full-aperture illumination, which has high spatial resolution and spatio-temporal phase sensitivity and is applicable to opaque surfaces and turbid biological specimens. RQDPM does not require additional polarized devices and can be easily switched from reflectional mode to transmission mode. In addition, RQDPM inherits the characteristic of high axial resolution of differential interference contrast microscope, thereby providing topography for opaque surfaces. We experimentally demonstrate the reflectional phase imaging ability of RQDPM with several samples: semiconductor wafer, thick biological tissues, red blood cells, and Hela cells. Furthermore, we dynamically monitor the flow state of microspheres in a self-built microfluidic channel by using RQDPM converted into the transmission mode.

Spatial- and Fourier-domain ptychography for high-throughput bio-imaging

First envisioned for determining crystalline structures, ptychography has become a useful imaging tool for microscopists. However, ptychography remains underused by biomedical researchers due to its limited resolution and throughput in the visible light regime. Recent developments of spatial- and Fourier-domain ptychography have successfully addressed these issues and now offer the potential for high-resolution, high-throughput optical imaging with minimal hardware modifications to existing microscopy setups, often providing an excellent trade-off between resolution and field of view inherent to conventional imaging systems, giving biomedical researchers the best of both worlds. Here, we provide extensive information to enable the implementation of ptychography by biomedical researchers in the visible light regime. We first discuss the intrinsic connections between spatial-domain coded ptychography and Fourier ptychography. A step-by-step guide then provides the user instructions for developing both systems with practical examples. In the spatial-domain implementation, we explain how a large-scale, high-performance blood-cell lens can be made at negligible expense. In the Fourier-domain implementation, we explain how adding a low-cost light source to a regular microscope can improve the resolution beyond the limit of the objective lens. The turnkey operation of these setups is suitable for use by professional research laboratories, as well as citizen scientists. Users with basic experience in optics and programming can build the setups within a week. The do-it-yourself nature of the setups also allows these procedures to be implemented in laboratory courses related to Fourier optics, biomedical instrumentation, digital image processing, robotics and capstone projects.

High-precision Fourier ptychographic microscopy based on Gaussian apodization coherent transfer function constraints

Fourier ptychographic microscopy (FPM) combines the concepts of phase retrieval algorithms and synthetic apertures and can solve the problem in which it is difficult to combine a large field of view with high resolution. However, the use of the coherent transfer function in conventional calculations to describe the linear transfer process of an imaging system can lead to ringing artifacts. In addition, the Gerchberg-Saxton iterative algorithm can cause the phase retrieval part of the FPM algorithm to fall into a local optimum. In this paper, Gaussian apodization coherent transfer function is proposed to describe the imaging process and is combined with an iterative method based on amplitude weighting and phase gradient descent to reduce the presence of ringing artifacts while ensuring the accuracy of the reconstructed results. In simulated experiments, the proposed algorithm is shown to give a smaller mean square error and higher structural similarity, both in the presence and absence of noise. Finally, the proposed algorithm is validated in terms of giving reconstruction results with high accuracy and high resolution, using images acquired with a new microscope system and open-source images.

Using symmetric illumination and color camera to achieve high throughput Fourier ptychographic microscopy

This study aims to develop a high throughput Fourier ptychographic microscopy (FPM) technique based on symmetric illumination and a color detector, which is able to accelerate image acquisition by up to 12 times. As an emerging technology, the efficiency of FPM is limited by its data acquisition process, especially for color microscope image reconstruction. To overcome this, we built an FPM prototype equipped with a color camera and a 4×/0.13 NA objective lens. During the image acquisition, two symmetric LEDs illuminate the sample simultaneously using white light, which doubles the light intensity and reduces the total captured raw patterns by half. A standard USAF 1951 resolution target was used to measure the system's modulation transfer function (MTF) curve, and the H&E-stained ovarian cancer samples were then imaged to assess the feature qualities depicted on the reconstructed images. The results showed that the measured MTF curves of red, green, and blue channels are generally comparable to the corresponding curves generated by conventional FPM, while symmetric illumination FPM preserves more tissue details, which is superior to the results captured by conventional 20×/0.4 NA objective lens. This investigation initially verified the feasibility of symmetric illumination based color FPM.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

Parallel Fourier ptychographic microscopy reconstruction method based on FPGA

Fourier ptychographic microscopy (FPM) can bypass the limitation of spatial bandwidth product to get images with large field-of-view and high resolution. The complicated sequential iterative calculation in the FPM reconstruction process reduces the reconstruction efficiency of the FPM. Therefore, we propose a parallel FPM reconstruction method based on field programmable gate array (FPGA) to accelerate the FPM reconstruction process. Using this method, multiple sub-regions in the Fourier domain can be computed in parallel and we customize a dedicated high-performance computational architecture for this approach. We deploy 4 FPM reconstruct computing architectures with a parallelism of 4 in a FPGA to compute the FPM reconstruction process, achieving the speed nearly 180 times faster than traditional methods. The proposed method provides a new perspective of parallel computing for FPM reconstruction.

Computational Portable Microscopes for Point-of-Care-Test and Tele-Diagnosis

In bio-medical mobile workstations, e.g., the prevention of epidemic viruses/bacteria, outdoor field medical treatment and bio-chemical pollution monitoring, the conventional bench-top microscopic imaging equipment is limited. The comprehensive multi-mode (bright/dark field imaging, fluorescence excitation imaging, polarized light imaging, and differential interference microscopy imaging, etc.) biomedical microscopy imaging systems are generally large in size and expensive. They also require professional operation, which means high labor-cost, money-cost and time-cost. These characteristics prevent them from being applied in bio-medical mobile workstations. The bio-medical mobile workstations need microscopy systems which are inexpensive and able to handle fast, timely and large-scale deployment. The development of lightweight, low-cost and portable microscopic imaging devices can meet these demands. Presently, for the increasing needs of point-of-care-test and tele-diagnosis, high-performance computational portable microscopes are widely developed. Bluetooth modules, WLAN modules and 3G/4G/5G modules generally feature very small sizes and low prices. And industrial imaging lens, microscopy objective lens, and CMOS/CCD photoelectric image sensors are also available in small sizes and at low prices. Here we review and discuss these typical computational, portable and low-cost microscopes by refined specifications and schematics, from the aspect of optics, electronic, algorithms principle and typical bio-medical applications.

Embedded parallel Fourier ptychographic microscopy reconstruction system

Fourier ptychographic microscopy (FPM) has attracted a wide range of focus for its ability of large space-bandwidth product and quantitative phase imaging. It is a typical computational imaging technique that jointly optimizes imaging hardware and reconstruction algorithms. The data redundancy and inverse problem algorithms are the sources of FPM's excellent performance. But at the same time, this large amount of data processing and complex algorithms also evidently reduce the imaging speed. To accelerate the FPM reconstruction speed, we proposed a fast FPM reconstruction framework consisting of three levels of parallel computation and implemented it with an embedded computing module. In the conventional FPM framework, the sample image is divided into multiple sub-regions to process separately because the illumination angles and defocus distances for different sub-regions may also be different. Our parallel framework first performs digital refocusing and high-resolution reconstruction for each sub-region separately and then stitches the complex sub-regions together to obtain the final high-resolution complex image. The feasibility of the proposed parallel FPM reconstruction framework is verified with different experimental results acquired with the system we built.

Fourier ptychographic microscopy with untrained deep neural network priors

We propose a physics-assisted deep neural network scheme in Fourier ptychographic microscopy (FPM) using untrained deep neural network priors (FPMUP) to achieve a high-resolution image reconstruction from multiple low-resolution images. Unlike the traditional training type of deep neural network that requires a large labelled dataset, this proposed scheme does not require training and instead outputs the high-resolution image by optimizing the parameters of neural networks to fit the experimentally measured low-resolution images. Besides the amplitude and phase of the sample function, another two parallel neural networks that generate the general pupil function and illumination intensity factors are incorporated into the carefully designed neural networks, which effectively improves the image quality and robustness when both the aberration and illumination intensity fluctuation are present in FPM. Reconstructions using simulated and experimental datasets are demonstrated, showing that the FPMUP scheme has better image quality than the traditional iterative algorithms, especially for the phase recovery, but at the expense of increasing computational cost. Most importantly, it is found that the FPMUP scheme can predict the Fourier spectrum of the sample outside synthetic aperture of FPM and thus eliminate the ringing effect of the recovered images due to the spectral truncation. Inspired by deep image prior in the field of image processing, we may impute the expansion of Fourier spectrums to the deep prior rooted in the architecture of the careful designed four parallel deep neural networks. We envisage that the resolution of FPM will be further enhanced if the Fourier spectrum of the sample outside the synthetic aperture of FPM is accurately predicted.

Stress and wavefront measurement of large-aperture optical components with a ptychographical iterative engine

The clamping stress of large-aperture optical elements has a significant influence on the optical quality of the system. In this study, a comprehensive measurement system combined with ptychographical iterative engine (PIE) wavefront sensors and polarization components is developed to determine the stress distribution of the optical elements and its effect on the transmitted and reflected wavefronts. This system avoids the use of multiple measuring instruments and has low cost and strong anti-interference ability. The experimental results demonstrate that the stress distributions measured at different resolutions are consistent with the finite element analysis, and the wavefront measurement accuracy is 0.1λ. This test configuration is very flexible and provides a useful means for online installation and quality control of large-aperture optical systems.

Integration of Fourier ptychography with machine learning: an alternative scheme

As the core task of the reconstruction in conventional ptychography (CP) and Fourier ptychographic microscopy (FPM), the meticulous design of ptychographical iterative engine (PIE) largely affects the performance of reconstruction algorithms. Compared to traditional PIE algorithms, the paradigm of combining with machine learning to cross a local optimum has recently achieved significant progress. Nevertheless, existing designed engines still suffer drawbacks such as excessive hyper-parameters, heavy tuning work and lack of compatibility, which greatly limit their practical applications. In this work, we present a complete set of alternative schemes comprised of a kind of new perspective, a uniform design template, and a fusion framework, to naturally integrate Fourier ptychography (FP) with machine learning concepts. The new perspective, Dynamic Physics, is taken as the preferred tool to analyze a path (algorithm) at the physical level; the uniform design template, T-FP, clarifies the physical significance and optimization part in a path; the fusion framework follows two workable guidelines that are specially designed to keep convergence and make later localized modification for a new path, and further establishes a link between FP iterations and the gradient update in machine learning. Our scheme is compatible with both traditional FP paths and machine learning concepts. By combining ideas in both fields, we offer two design examples, MaFP and AdamFP. Results for both simulations and experiments show that designed algorithms following our scheme obtain better, faster (converge at the early stage after a few iterations) and more stable recovery with only minimal tuning hyper-parameters, demonstrating the effectiveness and superiority of our scheme.

High-performance lensless diffraction imaging from diverse holograms by three-dimensional scanning

For lensless diffraction imaging, it is a challenging dilemma to achieve a large field of view (FOV) and high resolution with a small amount of data at the same time. Ptychography can reconstruct the high-resolution image and illumination light simultaneously. But the illumination is limited to a small size by a probe in typical ptychography. For large samples, it takes much time to collect abundant patterns and has strict requirements for the computing power of computers. Another widely applied method, multi-height measurement, can realize a wide FOV with several holograms. But, the recovered image is easily destroyed by the background noise. In this Letter, a lensless diffraction imaging method by three-dimensional scanning is proposed. All positions of the object are different in three directions instead of scanning schemes only on a plane or along the optic axis, so more diversity of diffraction information is obtained. We apply the illumination without the limit of a confined aperture, which means that the imaging FOV of a pattern is equal to the size of the utilized image sensor. In comparison with the multi-height method, our method can separate the illumination background noise from the retrieved object. Consequently, the proposed method realized high resolution and contrast, large FOV, and the removal of background noise simultaneously. Experimental validations and comparisons with other methods are presented.

Pixel Super-Resolution Phase Retrieval for Lensless On-Chip Microscopy via Accelerated Wirtinger Flow

Empowered by pixel super-resolution (PSR) and phase retrieval techniques, lensless on-chip microscopy opens up new possibilities for high-throughput biomedical imaging. However, the current PSR phase retrieval approaches are time consuming in terms of both the measurement and reconstruction procedures. In this work, we present a novel computational framework for PSR phase retrieval to address these concerns. Specifically, a sparsity-promoting regularizer is introduced to enhance the well posedness of the nonconvex problem under limited measurements, and Nesterov's momentum is used to accelerate the iterations. The resulting algorithm, termed accelerated Wirtinger flow (AWF), achieves at least an order of magnitude faster rate of convergence and allows a twofold reduction in the measurement number while maintaining competitive reconstruction quality. Furthermore, we provide general guidance for step size selection based on theoretical analyses, facilitating simple implementation without the need for complicated parameter tuning. The proposed AWF algorithm is compatible with most of the existing lensless on-chip microscopes and could help achieve label-free rapid whole slide imaging of dynamic biological activities at subpixel resolution.

X-ray Ptychographic Imaging and Spectroscopic Studies of Plasma-Treated Plastic Films

Polyethylene terephthalate (PET) is a thermoplastic polyester with numerous applications in industry. However, it requires surface modification on an industrial scale for printing and coating processes and plasma treatment is one of the most commonly used techniques to increase the hydrophilicity of the PET films. Systematic improvement of the surface modification by adaption of the plasma process can be aided by a comprehensive understanding of the surface morphology and chemistry. However, imaging large surface areas (tens of microns) with a resolution that allows understanding the surface quality and modification is challenging. As a proof-of-principle, plasma-treated PET films were used to demonstrate the capabilities of X-ray ptychography, currently under development at the soft X-ray free-electron laser FLASH at DESY, for imaging macroscopic samples. In combination with scanning electron microscopy (SEM), this new technique was used to study the effects of different plasma treatment processes on PET plastic films. The studies on the surface morphology were complemented by investigations of the surface chemistry using X-ray photoelectron spectroscopy (XPS) and Fourier transform infrared spectroscopy (FT-IR). While both imaging techniques consistently showed an increase in roughness and change in morphology of the PET films after plasma treatment, X-ray ptychography can provide additional information on the three-dimensional morphology of the surface. At the same time, the chemical analysis shows an increase in the oxygen content and polarity of the surface without significant damage to the polymer, which is important for printing and coating processes.

Deep distributed optimization for blind diffuser-modulation ptychography

Blind diffuser-modulation ptychography has emerged as a low-cost technique for micro-nano holographic imaging, which enables breaking the resolution limit of optical systems. However, the existing reconstruction method requires thousands of measurements to recover object and diffuser profile simultaneously, which makes the data acquisition time-consuming and cumbersome. In this Letter, we report a novel, to the best of our knowledge, blind ptychography technique with deep distributed optimization, termed BPD²O. It decomposes the complicated optimization task into subproblems, then introduces extended ptychographical iterative engine and enhanced network solver to optimize each in a distributed strategy. In this way, BPD²O combines the advantages of both model-driven and data-driven strategies, realizing high-fidelity robust ptychography imaging. Extensive experiments validate that BPD²O can realize better resolution and lead to a reduction of more than one order of magnitude in the number of measurements.

Fourier Ptychographic Microscopy via Alternating Direction Method of Multipliers

Fourier ptychographic microscopy (FPM) has risen as a promising computational imaging technique that breaks the trade-off between high resolution and large field of view (FOV). Its reconstruction is normally formulated as a blind phase retrieval problem, where both the object and probe have to be recovered from phaseless measured data. However, the stability and reconstruction quality may dramatically deteriorate in the presence of noise interference. Herein, we utilized the concept of alternating direction method of multipliers (ADMM) to solve this problem (termed ADMM-FPM) by breaking it into multiple subproblems, each of which may be easier to deal with. We compared its performance against existing algorithms in both simulated and practical FPM platform. It is found that ADMM-FPM method belongs to a global optimization algorithm with a high degree of parallelism and thus results in a more stable and robust phase recovery under noisy conditions. We anticipate that ADMM will rekindle interest in FPM as more modifications and innovations are implemented in the future.

A guide into the world of high-resolution 3D imaging: the case of soft X-ray tomography for the life sciences

In the world of bioimaging, every choice made determines the quality and content of the data collected. The choice of imaging techniques for a study could showcase or dampen expected outcomes. Synchrotron radiation is indispensable for biomedical research, driven by the need to see into biological materials and capture intricate biochemical and biophysical details at controlled environments. The same need drives correlative approaches that enable the capture of heterologous but complementary information when studying any one single target subject. Recently, the applicability of one such synchrotron technique in bioimaging, soft X-ray tomography (SXT), facilitates exploratory and basic research and is actively progressing towards filling medical and industrial needs for the rapid screening of biomaterials, reagents and processes of immediate medical significance. Soft X-ray tomography at cryogenic temperatures (cryoSXT) fills the imaging resolution gap between fluorescence microscopy (in the hundreds of nanometers but relatively accessible) and electron microscopy (few nanometers but requires extensive effort and can be difficult to access). CryoSXT currently is accessible, fully documented, can deliver 3D imaging to 25 nm resolution in a high throughput fashion, does not require laborious sample preparation procedures and can be correlated with other imaging techniques. Here, we present the current state of SXT and outline its place within the bioimaging world alongside a guided matrix that aids decision making with regards to the applicability of any given imaging technique to a particular project. Case studies where cryoSXT has facilitated a better understanding of biological processes are highlighted and future directions are discussed.

Label-free imaging of intracellular organelle dynamics using flat-fielding quantitative phase contrast microscopy (FF-QPCM)

Panoramic and long-term observation of nanosized organelle dynamics and interactions with high spatiotemporal resolution still hold great challenge for current imaging platforms. In this study, we propose a live-organelle imaging platform, where a flat-fielding quantitative phase contrast microscope (FF-QPCM) visualizes all the membrane-bound subcellular organelles, and an intermittent fluorescence channel assists in specific organelle identification. FF-QPCM features a high spatiotemporal resolution of 245 nm and 250 Hz and strong immunity against external disturbance. Thus, we could investigate several important dynamic processes of intracellular organelles from direct perspectives, including chromosome duplication in mitosis, mitochondrial fusion and fission, filaments, and vesicles' morphologies in apoptosis. Of note, we have captured, for the first time, a new type of mitochondrial fission (entitled mitochondrial disintegration), the generation and fusion process of vesicle-like organelles, as well as the mitochondrial vacuolization during necrosis. All these results bring us new insights into spatiotemporal dynamics and interactions among organelles, and hence aid us in understanding the real behaviors and functional implications of the organelles in cellular activities.

Robust Fourier ptychographic microscopy via a physics-based defocusing strategy for calibrating angle-varied LED illumination

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique for wide-field, high-resolution microscopy with a high space-bandwidth product. It integrates the concepts of synthetic aperture and phase retrieval to surpass the resolution limit imposed by the employed objective lens. In the FPM framework, the position of each sub-spectrum needs to be accurately known to ensure the success of the phase retrieval process. Different from the conventional methods with mechanical adjustment or data-driven optimization strategies, here we report a physics-based defocusing strategy for correcting large-scale positional deviation of the LED illumination in FPM. Based on a subpixel image registration process with a defocused object, we can directly infer the illumination parameters including the lateral offsets of the light source, the in-plane rotation angle of the LED array, and the distance between the sample and the LED board. The feasibility and effectiveness of our method are validated with both simulations and experiments. We show that the reported strategy can obtain high-quality reconstructions of both the complex object and pupil function even the LED array is randomly placed under the sample with both unknown lateral offsets and rotations. As such, it enables the development of robust FPM systems by reducing the requirements on fine mechanical adjustment and data-driven correction in the construction process.

High-performance heterogeneous FPGA data-flow architecture for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique that can achieve both high-resolution and a wide field-of-view via a sequence of low-resolution images. FPM is a complex iterative process, and it is difficult to meet the needs of rapid reconstruction imaging with the conventional FPM deployed on general purpose processors. In this paper, we propose a high-performance heterogeneous field-programmable gate array (FPGA) architecture based on the principle of full pipeline and the data-flow structure for the iterative reconstruction procedure of FPM. By optimizing the architecture network at gate-level logic circuits, the running time of the FPGA-based FPM reconstruction procedure is nearly 20 times faster than conventional methods. Our proposed architecture can be used to develop FPM imaging equipment that meets resource and performance requirements.

Synthetic aperture interference light (SAIL) microscopy for high-throughput label-free imaging

Quantitative phase imaging (QPI) is a valuable label-free modality that has gained significant interest due to its wide potentials, from basic biology to clinical applications. Most existing QPI systems measure microscopic objects via interferometry or nonlinear iterative phase reconstructions from intensity measurements. However, all imaging systems compromise spatial resolution for the field of view and vice versa, i.e., suffer from a limited space bandwidth product. Current solutions to this problem involve computational phase retrieval algorithms, which are time-consuming and often suffer from convergence problems. In this article, we presented synthetic aperture interference light (SAIL) microscopy as a solution for high-resolution, wide field of view QPI. The proposed approach employs low-coherence interferometry to directly measure the optical phase delay under different illumination angles and produces large space-bandwidth product label-free imaging. We validate the performance of SAIL on standard samples and illustrate the biomedical applications on various specimens: pathology slides, entire insects, and dynamic live cells in large cultures. The reconstructed images have a synthetic numeric aperture of 0.45 and a field of view of 2.6 × 2.6 mm2. Due to its direct measurement of the phase information, SAIL microscopy does not require long computational time, eliminates data redundancy, and always converges.

Quantitative multi-height phase retrieval via a coded image sensor

Multi-height phase retrieval introduces different object-to-detector distances for obtaining phase diversity measurements. In the acquisition process, the slow-varying phase information, however, cannot be converted to intensity variations for detection. Therefore, the low-frequency contents of the phase profile are lost during acquisition and cannot be properly restored via phase retrieval. Here, we demonstrate the use of a coded image sensor for addressing this challenge in multi-height phase retrieval. In our scheme, we add a coded layer on top of the image sensor for encoding the slow-varying complex wavefronts into intensity variations of the modulated patterns. Inspired by the concept of blind ptychography, we report a reconstruction scheme to jointly recover the complex object and the unknown coded layer using multi-height measurements. With both simulation and experimental results, we show that the recovered phase is quantitative and the slow-varying phase profiles can be properly restored using lensless multi-height measurements. We also show that the image quality using the coded sensor is better than that of a regular image sensor. For demonstrations, we validate the reported scheme with various biospecimens and compare the results to those of regular lensless multi-height phase retrieval. The use of a coded image sensor may enable true quantitative phase imaging for the lensless multi-height, multi-wavelength, and transport-of-intensity equation approaches.

Accurate and stable two-step LED position calibration method for Fourier ptychographic microscopy

SIGNIFICANCE

Fourier ptychography microscopy (FPM) is a computational optical imaging technology that employs angularly varying illuminations and a phase retrieval algorithm to achieve a wide field of view and high-resolution imaging simultaneously. In the FPM, LED position error will reduce the quality of the reconstructed high-resolution image. To correct the LED positions, current methods consider each of the LED positions as independent and use an optimization algorithm to get each of the positions. When the positional misalignment is large or the search position falls into a local optimal value, the current methods may lack stability and accuracy.

AIM

We improve the model of the LED position and propose an accurate and stable two-step correction scheme (tcFPM) to calibrate the LED position error.

APPROACH

The improved LED positions model combines the overall offset, which represents the relative deviation of the LED array and the optical axis, with the slight deviation of each LED's independent position. In the tcFPM, the overall offset of the LED array is corrected at first, which obtains an approximate value of the overall offset of the LED array. Then the position of each LED is precisely adjusted, which obtains the slight offset of each LED.

RESULTS

This LED position error model is more in line with the actual situation. The simulation and experimental results show that the method has high accuracy in correcting the LED position. Furthermore, the reconstruction process of tcFPM is more stable and significantly improves the quality of the reconstruction results, which is compared with some LED position error correction methods.

CONCLUSIONS

An LED position error correction technology is proposed, which has a stable iterative process and improves the reconstruction accuracy of complex amplitude.

Virtual depth-scan multi-slice ptychography for improved three-dimensional imaging

Multi-slice ptychography (MSP) is a fast three-dimensional ptychography technology developed on the basis of conventional ptychography. With this method, three-dimensional imaging can be achieved without rotating the sample. The prototype multi-slice algorithm can only reconstruct three-dimensional samples with a limited number of slices, which greatly limits the depth range and resolution of sample imaging. Here we reported a virtual depth-scan scheme of MSP in which a thick sample is scanned virtually in the depth direction across its whole thickness range within the reconstruction process, thereby eliminating the restriction on slice number and potentially improving the depth resolution of MSP. This new approach also improves the flexibility of multi-slice ptychography. Both the simulation and experimental results validate the feasibility of our new approach.

Microscopic phase reconstruction of cervical exfoliated cell under partially coherent illumination

Basic coherent diffraction imaging methods strongly rely on having a highly coherent illumination in order to reconstruct the phase accurately. However, regardless of considering the turbulent transport medium, the instability of the system or the generation mechanism of the light source, partially coherent illumination is more common in real case. In this paper, we proposed an efficient microscopic phase imaging method to study normal and abnormal cervical exfoliated cells. By applying three phase modulations in a single point of the sample's transmitted field, the phase can be retrieved with correspoding three intensities under partially coherent illumination. Compared with intensity map, we can efficiently and clearly judge the proportion of high density shrinking abnormal cells from the phase distributions, which provides a confident analysis and evaluation basis for early medical diagnosis of cervical cancer. This study also has potential applications in noninvasive optical imaging of dynamic biological tissues.