High-resolution, wide-field object reconstruction with synthetic aperture Fourier holographic optical microscopy

Lens-free on-chip 3D microscopy based on wavelength-scanning Fourier ptychographic diffraction tomography

Lens-free on-chip microscopy is a powerful and promising high-throughput computational microscopy technique due to its unique advantage of creating high-resolution images across the full field-of-view (FOV) of the imaging sensor. Nevertheless, most current lens-free microscopy methods have been designed for imaging only two-dimensional thin samples. Lens-free on-chip tomography (LFOCT) with a uniform resolution across the entire FOV and at a subpixel level remains a critical challenge. In this paper, we demonstrated a new LFOCT technique and associated imaging platform based on wavelength scanning Fourier ptychographic diffraction tomography (wsFPDT). Instead of using angularly-variable illuminations, in wsFPDT, the sample is illuminated by on-axis wavelength-variable illuminations, ranging from 430 to 1200 nm. The corresponding under-sampled diffraction patterns are recorded, and then an iterative ptychographic reconstruction procedure is applied to fill the spectrum of the three-dimensional (3D) scattering potential to recover the sample's 3D refractive index (RI) distribution. The wavelength-scanning scheme not only eliminates the need for mechanical motion during image acquisition and precise registration of the raw images but secures a quasi-uniform, pixel-super-resolved imaging resolution across the entire imaging FOV. With wsFPDT, we demonstrate the high-throughput, billion-voxel 3D tomographic imaging results with a half-pitch lateral resolution of 775 nm and an axial resolution of 5.43 μm across a large FOV of 29.85 mm2 and an imaging depth of >200 μm. The effectiveness of the proposed method was demonstrated by imaging various types of samples, including micro-polystyrene beads, diatoms, and mouse mononuclear macrophage cells. The unique capability to reveal quantitative morphological properties, such as area, volume, and sphericity index of single cell over large cell populations makes wsFPDT a powerful quantitative and label-free tool for high-throughput biological applications.

Fourier Ptychographic Microscopy 10 Years on: A Review

Fourier ptychographic microscopy (FPM) emerged as a prominent imaging technique in 2013, attracting significant interest due to its remarkable features such as precise phase retrieval, expansive field of view (FOV), and superior resolution. Over the past decade, FPM has become an essential tool in microscopy, with applications in metrology, scientific research, biomedicine, and inspection. This achievement arises from its ability to effectively address the persistent challenge of achieving a trade-off between FOV and resolution in imaging systems. It has a wide range of applications, including label-free imaging, drug screening, and digital pathology. In this comprehensive review, we present a concise overview of the fundamental principles of FPM and compare it with similar imaging techniques. In addition, we present a study on achieving colorization of restored photographs and enhancing the speed of FPM. Subsequently, we showcase several FPM applications utilizing the previously described technologies, with a specific focus on digital pathology, drug screening, and three-dimensional imaging. We thoroughly examine the benefits and challenges associated with integrating deep learning and FPM. To summarize, we express our own viewpoints on the technological progress of FPM and explore prospective avenues for its future developments.

Aberration Estimation for Synthetic Aperture Digital Holographic Microscope Using Deep Neural Network

Digital holographic microscopy (DHM) is a valuable technique for investigating the optical properties of samples through the measurement of intensity and phase of diffracted beams. However, DHMs are constrained by Lagrange invariance, compromising the spatial bandwidth product (SBP) which relates resolution and field of view. Synthetic aperture DHM (SA-DHM) was introduced to overcome this limitation, but it faces significant challenges such as aberrations in synthesizing the optical information corresponding to the steering angle of incident wave. This paper proposes a novel approach utilizing deep neural networks (DNNs) for compensating aberrations in SA-DHM, extending the compensation scope beyond the numerical aperture (NA) of the objective lens. The method involves training a DNN from diffraction patterns and Zernike coefficients through a circular aperture, enabling effective aberration compensation in the illumination beam. This method makes it possible to estimate aberration coefficients from the only part of the diffracted beam cutoff by the circular aperture mask. With the proposed technique, the simulation results present improved resolution and quality of sample images. The integration of deep neural networks with SA-DHM holds promise for advancing microscopy capabilities and overcoming existing limitations.

Ptychographic lens-less birefringence microscopy using a mask-modulated polarization image sensor

Birefringence, an inherent characteristic of optically anisotropic materials, is widely utilized in various imaging applications ranging from material characterizations to clinical diagnosis. Polarized light microscopy enables high-resolution, high-contrast imaging of optically anisotropic specimens, but it is associated with mechanical rotations of polarizer/analyzer and relatively complex optical designs. Here, we present a form of lens-less polarization-sensitive microscopy capable of complex and birefringence imaging of transparent objects without an optical lens and any moving parts. Our method exploits an optical mask-modulated polarization image sensor and single-input-state LED illumination design to obtain complex and birefringence images of the object via ptychographic phase retrieval. Using a camera with a pixel size of 3.45 μm, the method achieves birefringence imaging with a half-pitch resolution of 2.46 μm over a 59.74 mm2 field-of-view, which corresponds to a space-bandwidth product of 9.9 megapixels. We demonstrate the high-resolution, large-area, phase and birefringence imaging capability of our method by presenting the phase and birefringence images of various anisotropic objects, including a monosodium urate crystal, and excised mouse eye and heart tissues.

High-precision Fourier ptychographic microscopy based on Gaussian apodization coherent transfer function constraints

Fourier ptychographic microscopy (FPM) combines the concepts of phase retrieval algorithms and synthetic apertures and can solve the problem in which it is difficult to combine a large field of view with high resolution. However, the use of the coherent transfer function in conventional calculations to describe the linear transfer process of an imaging system can lead to ringing artifacts. In addition, the Gerchberg-Saxton iterative algorithm can cause the phase retrieval part of the FPM algorithm to fall into a local optimum. In this paper, Gaussian apodization coherent transfer function is proposed to describe the imaging process and is combined with an iterative method based on amplitude weighting and phase gradient descent to reduce the presence of ringing artifacts while ensuring the accuracy of the reconstructed results. In simulated experiments, the proposed algorithm is shown to give a smaller mean square error and higher structural similarity, both in the presence and absence of noise. Finally, the proposed algorithm is validated in terms of giving reconstruction results with high accuracy and high resolution, using images acquired with a new microscope system and open-source images.

Pose correction scheme for camera-scanning Fourier ptychography based on camera calibration and homography transform

Fourier ptychography (FP), as a computational imaging method, is a powerful tool to improve imaging resolution. Camera-scanning Fourier ptychography extends the application of FP from micro to macro creatively. Due to the non-ideal scanning of the camera driven by the mechanical translation stage, the pose error of the camera occurs, greatly degrading the reconstruction quality, while a precise translation stage is expensive and not suitable for wide-range imaging. Here, to improve the imaging performance of camera-scanning Fourier ptychography, we propose a pose correction scheme based on camera calibration and homography transform approaches. The scheme realizes the accurate alignment of data set and location error correction in the frequency domain. Simulation and experimental results demonstrate this method can optimize the reconstruction results and realize high-quality imaging effectively. Combined with the feature recognition algorithm, the scheme provides the possibility for applying FP in remote sensing imaging and space imaging.

Roadmap on Digital Holography-Based Quantitative Phase Imaging

Quantitative Phase Imaging (QPI) provides unique means for the imaging of biological or technical microstructures, merging beneficial features identified with microscopy, interferometry, holography, and numerical computations. This roadmap article reviews several digital holography-based QPI approaches developed by prominent research groups. It also briefly discusses the present and future perspectives of 2D and 3D QPI research based on digital holographic microscopy, holographic tomography, and their applications.

Accessing depth-resolved high spatial frequency content from the optical coherence tomography signal

Optical coherence tomography (OCT) is a rapidly evolving technology with a broad range of applications, including biomedical imaging and diagnosis. Conventional intensity-based OCT provides depth-resolved imaging with a typical resolution and sensitivity to structural alterations of about 5–10 microns. It would be desirable for functional biological imaging to detect smaller features in tissues due to the nature of pathological processes. In this article, we perform the analysis of the spatial frequency content of the OCT signal based on scattering theory. We demonstrate that the OCT signal, even at limited spectral bandwidth, contains information about high spatial frequencies present in the object which relates to the small, sub-wavelength size structures. Experimental single frame imaging of phantoms with well-known sub-micron internal structures confirms the theory. Examples of visualization of the nanoscale structural changes within mesenchymal stem cells (MSC), which are invisible using conventional OCT, are also shown. Presented results provide a theoretical and experimental basis for the extraction of high spatial frequency information to substantially improve the sensitivity of OCT to structural alterations at clinically relevant depths.

Spatial light interference microscopy: principle and applications to biomedicine

In this paper, we review spatial light interference microscopy (SLIM), a common-path, phase-shifting interferometer, built onto a phase-contrast microscope, with white-light illumination. As one of the most sensitive quantitative phase imaging (QPI) methods, SLIM allows for speckle-free phase reconstruction with sub-nanometer path-length stability. We first review image formation in QPI, scattering, and full-field methods. Then, we outline SLIM imaging from theory and instrumentation to diffraction tomography. Zernike's phase-contrast microscopy, phase retrieval in SLIM, and halo removal algorithms are discussed. Next, we discuss the requirements for operation, with a focus on software developed in-house for SLIM that enables high-throughput acquisition, whole slide scanning, mosaic tile registration, and imaging with a color camera. We introduce two methods for solving the inverse problem using SLIM, white-light tomography, and Wolf phase tomography. Lastly, we review the applications of SLIM in basic science and clinical studies. SLIM can study cell dynamics, cell growth and proliferation, cell migration, mass transport, etc. In clinical settings, SLIM can assist with cancer studies, reproductive technology, blood testing, etc. Finally, we review an emerging trend, where SLIM imaging in conjunction with artificial intelligence brings computational specificity and, in turn, offers new solutions to outstanding challenges in cell biology and pathology.

Precise Nanosizing with High Dynamic Range Holography

Optical sensing is one of the key enablers of modern diagnostics. Especially label-free imaging modalities hold great promise as they eliminate labeling procedures prior to analysis. However, scattering signals of nanometric particles scale with their volume square. This unfavorable scaling makes it extremely difficult to quantitatively characterize intrinsically heterogeneous clinical samples, such as extracellular vesicles, as their signal variation easily exceeds the dynamic range of currently available cameras. Here, we introduce off-axis k-space holography that circumvents this limitation. By imaging the back-focal plane of our microscope, we project the scattering signal of all particles onto all camera pixels, thus dramatically boosting the achievable dynamic range to up to 110 dB. We validate our platform by detecting and quantitatively sizing metallic and dielectric particles over a 200 × 200 μm field of view and demonstrate that independently performed signal calibrations allow correctly sizing particles made from different materials. Finally, we present quantitative size distributions of extracellular vesicle samples.

Single full-FOV reconstruction Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique that has high-resolution and wide field-of-view (FOV). FPM bypasses the NA limit of the system by stitching a number of variable-illuminated measured images in Fourier space. On the basis of the wide FOV of the low NA objective, the high-resolution image with a wide FOV can be reconstructed through the phase recovery algorithm. However, the high-resolution reconstruction images are affected by the LED array point light source. The results are: (1) the intensities collected by the sample are severely declined when edge LEDs illuminate the sample; (2) the multiple reconstructions are caused by wavevectors inconsistency for the full FOV images. Here, we propose a new lighting scheme termed full FOV Fourier ptychographic microscopy (F³PM). By combining the LED array and telecentric lens, the method can provide plane waves with different angles while maintaining uniform intensity. Benefiting from the telecentric performance and f‒θ property of the telecentric lens, the system stability is improved and the relationship between the position of LED and its illumination angle is simplified. The excellent plane wave provided by the telecentric lens guarantees the same wavevector in the full FOV, and we use this wavevector to reconstruct the full FOV during one time. The area and diameter of the single reconstruction FOV reached 14.6mm ² and 5.4 mm, respectively, and the diameter is very close to the field number (5.5 mm) of the 4× objective. Compared with the traditional FPM, we have increased the diameter of FOV in a single reconstruction by ∼ 10 times, eliminating the complicated steps of computational redundancy and image stitching.

High spatial and temporal resolution synthetic aperture phase microscopy

A new optical microscopy technique, termed high spatial and temporal resolution synthetic aperture phase microscopy (HISTR-SAPM), is proposed to improve the lateral resolution of wide-field coherent imaging. Under plane wave illumination, the resolution is increased by twofold to around 260 nm, while achieving millisecond-level temporal resolution. In HISTR-SAPM, digital micromirror devices are used to actively change the sample illumination beam angle at high speed with high stability. An off-axis interferometer is used to measure the sample scattered complex fields, which are then processed to reconstruct high-resolution phase images. Using HISTR-SAPM, we are able to map the height profiles of subwavelength photonic structures and resolve the period structures that have 198 nm linewidth and 132 nm gap (i.e., a full pitch of 330 nm). As the reconstruction averages out laser speckle noise while maintaining high temporal resolution, HISTR-SAPM further enables imaging and quantification of nanoscale dynamics of live cells, such as red blood cell membrane fluctuations and subcellular structure dynamics within nucleated cells. We envision that HISTR-SAPM will broadly benefit research in material science and biology.

Long-Distance Sub-Diffraction High-Resolution Imaging Using Sparse Sampling

How to perform imaging beyond the diffraction limit has always been an essential subject for the research of optical systems. One effective way to achieve this purpose is Fourier ptychography, which has been widely used in microscopic imaging. However, microscopic imaging measurement technology cannot be directly extended to imaging macro objects at long distances. In this paper, a reconstruction algorithm is proposed to solve the need for oversampling low-resolution images, and it is successfully applied to macroscopic imaging. Compared with the traditional FP technology, the proposed sub-sampling method can significantly reduce the number of iterations in reconstruction. Experiments prove that the proposed method can reconstruct low-resolution images captured by the camera and achieve high-resolution imaging of long-range macroscopic objects.

Fourier ptychography: current applications and future promises

Traditional imaging systems exhibit a well-known trade-off between the resolution and the field of view of their captured images. Typical cameras and microscopes can either "zoom in" and image at high-resolution, or they can "zoom out" to see a larger area at lower resolution, but can rarely achieve both effects simultaneously. In this review, we present details about a relatively new procedure termed Fourier ptychography (FP), which addresses the above trade-off to produce gigapixel-scale images without requiring any moving parts. To accomplish this, FP captures multiple low-resolution, large field-of-view images and computationally combines them in the Fourier domain into a high-resolution, large field-of-view result. Here, we present details about the various implementations of FP and highlight its demonstrated advantages to date, such as aberration recovery, phase imaging, and 3D tomographic reconstruction, to name a few. After providing some basics about FP, we list important details for successful experimental implementation, discuss its relationship with other computational imaging techniques, and point to the latest advances in the field while highlighting persisting challenges.

Coherent imager module with a large field of view for synthetic aperture interferometry applications

Optical areal profilometry of large precision-engineered surfaces require high-resolution measurements over large fields of view. Synthetic Aperture Interferometry (SAI) offers an alternative to the conventional approach of stitching small fields of view (FOV) obtained with Coherent Scanning Interferometry (CSI) using high-NA objectives. In SAI, low-resolution digital holograms are recorded for different illumination and observation directions and they are added coherently to produce a high-resolution reconstruction over a large FOV. This paper describes the design, fabrication and characterization of a large FOV, compact and low-cost coherent imager (CI) as a building block of a coherent sensor array for a SAI system. The CI consists of a CMOS photodetector array with 1.12 µm pixel pitch, a square entrance pupil and a highly divergent reference beam that emerges from a pinhole milled with a focused ion beam on the cylindrical cladding at the tip of an optical fibre. In order to accurately reconstruct the digital holograms, the wavefront of the reference beam is estimated by localizing the reference source relative to the photodetector array. This is done using an optimization approach that simultaneously reconstructs plane waves that reach the aperture from 121 different illumination directions and guarantees a phase root-mean-squared (RMS) error of less than a fifth of the wavelength across the CI entrance pupil at a boundary of the FOV. The CI performance is demonstrated with a holographic reconstruction of a 0.110 m wide object placed at a distance of 0.085 m, i.e. a FOV = ±0.57 rad, the highest reported to date with a holographic camera.

Reflective Fourier ptychographic microscopy using a parabolic mirror

Fourier ptychography uses a phase retrieval algorithm to reconstruct a high-resolution image with a wide field-of-view. Reflective-type Fourier ptychographic microscopy (FPM) is expected to be very useful for surface inspection, but the reported methods have several limitations. We propose a darkfield illuminator for reflective FPM consisting of a parabolic mirror and a flat LED panel. This increases the signal-to-noise ratio of the acquired images because the normal beam of each LED is directed toward the object. Furthermore, the LEDs do not have to be far from the object because they are collimated by the parabolic surface before illumination. Based on this, a reflective FPM with a synthesized numerical aperture (NA) of 1.06 was achieved, which is the highest value by reflective FPM as far as we know. To validate this experimentally, we measured a USAF reflective resolution target and reconstructed a high-resolution image. This resolved up to the period of 488 nm, which corresponds to the synthesized NA. Additionally, an integrated circuit was measured to demonstrate the effectiveness of surface inspection of the proposed system.

Computational aberration compensation by coded-aperture-based correction of aberration obtained from optical Fourier coding and blur estimation

We report a novel generalized optical measurement system and computational approach to determine and correct aberrations in optical systems. The system consists of a computational imaging method capable of reconstructing an optical system's pupil function by adapting overlapped Fourier coding to an incoherent imaging modality. It recovers the high-resolution image latent in an aberrated image via deconvolution. The deconvolution is made robust to noise by using coded apertures to capture images. We term this method coded-aperture-based correction of aberration obtained from overlapped Fourier coding and blur estimation (CACAO-FB). It is well-suited for various imaging scenarios where aberration is present and where providing a spatially coherent illumination is very challenging or impossible. We report the demonstration of CACAO-FB with a variety of samples including an in vivo imaging experiment on the eye of a rhesus macaque to correct for its inherent aberration in the rendered retinal images. CACAO-FB ultimately allows for an aberrated imaging system to achieve diffraction-limited performance over a wide field of view by casting optical design complexity to computational algorithms in post-processing.

Reliable deep-learning-based phase imaging with uncertainty quantification

Emerging deep-learning (DL)-based techniques have significant potential to revolutionize biomedical imaging. However, one outstanding challenge is the lack of reliability assessment in the DL predictions, whose errors are commonly revealed only in hindsight. Here, we propose a new Bayesian convolutional neural network (BNN)-based framework that overcomes this issue by quantifying the uncertainty of DL predictions. Foremost, we show that BNN-predicted uncertainty maps provide surrogate estimates of the true error from the network model and measurement itself. The uncertainty maps characterize imperfections often unknown in real-world applications, such as noise, model error, incomplete training data, and out-of-distribution testing data. Quantifying this uncertainty provides a per-pixel estimate of the confidence level of the DL prediction as well as the quality of the model and data set. We demonstrate this framework in the application of large space-bandwidth product phase imaging using a physics-guided coded illumination scheme. From only five multiplexed illumination measurements, our BNN predicts gigapixel phase images in both static and dynamic biological samples with quantitative credibility assessment. Furthermore, we show that low-certainty regions can identify spatially and temporally rare biological phenomena. We believe our uncertainty learning framework is widely applicable to many DL-based biomedical imaging techniques for assessing the reliability of DL predictions.

Low-coherent optical diffraction tomography by angle-scanning illumination

Temporally low-coherent optical diffraction tomography (ODT) is proposed and demonstrated based on angle-scanning Mach-Zehnder interferometry. Using a digital micromirror device based on diffractive tilting, the full-field interference of incoherent light is successfully maintained during every angle-scanning sequences. Further, current ODT reconstruction principles for temporally incoherent illuminations are thoroughly reviewed and developed. Several limitations of incoherent illumination are also discussed, such as the nondispersive assumption, optical sectioning capacity and illumination angle limitation. Using the proposed setup and reconstruction algorithms, low-coherent ODT imaging of plastic microspheres, human red blood cells and rat pheochromocytoma cells is experimentally demonstrated.

Fourier ptychographic microscopy reconstruction with multiscale deep residual network

Fourier ptychographic microscopy (FPM) is a newly developed microscopic technique for large field of view, high-resolution and quantitative phase imaging by combining the techniques from ptychographic imaging, aperture synthesizing and phase retrieval. In FPM, an LED array is utilized to illuminate the specimen from different angles and the corresponding intensity images are synthesized to reconstruct a high-resolution complex field. As a flexible and low-cost approach to achieve high-resolution, wide-field and quantitative phase imaging, FPM is of enormous potential in biomedical applications such as hematology and pathology. Conventionally, the FPM reconstruction problem is solved by using a phase retrieval method, termed Alternate Projection. By iteratively updating the Fourier spectrum with low-resolution-intensity images, the result converges to a high-resolution complex field. Here we propose a new FPM reconstruction framework with deep learning methods and design a multiscale, deep residual neural network for FPM reconstruction. We employ the widely used open-source deep learning library PyTorch to train and test our model and carefully choose the hyperparameters of our model. To train and analyze our model, we build a large-scale simulation dataset with an FPM imaging model and an actual dataset captured with an FPM system. The simulation dataset and actual dataset are separated as training datasets and test datasets, respectively. Our model is trained with the simulation training dataset and fine tuned with the fine-tune dataset, which contains actual training data. Both our model and the conventional method are tested on the simulation test dataset and the actual test dataset to evaluate the performances. We also show the reconstruction result of another neural network-based method for comparison. The experiments demonstrate that our model achieves better reconstruction results and consumes much less time than conventional methods. The results also point out that our model is more robust under system aberrations such as noise and blurring (fewer intensity images) compared with conventional methods.

High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network

We combine a generative adversarial network (GAN) with light microscopy to achieve deep learning super-resolution under a large field of view (FOV). By appropriately adopting prior microscopy data in an adversarial training, the neural network can recover a high-resolution, accurate image of new specimen from its single low-resolution measurement. Its capacity has been broadly demonstrated via imaging various types of samples, such as USAF resolution target, human pathological slides, fluorescence-labelled fibroblast cells, and deep tissues in transgenic mouse brain, by both wide-field and light-sheet microscopes. The gigapixel, multi-color reconstruction of these samples verifies a successful GAN-based single image super-resolution procedure. We also propose an image degrading model to generate low resolution images for training, making our approach free from the complex image registration during training data set preparation. After a well-trained network has been created, this deep learning-based imaging approach is capable of recovering a large FOV (~95 mm2) enhanced resolution of ~1.7 μm at high speed (within 1 second), while not necessarily introducing any changes to the setup of existing microscopes.

Scanning diffracted-light photography using white-light and thermal radiation sources

A 4-f imaging arrangement of lenses with a camera and a rotating slit placed at the Fourier plane of the system was used to obtain the optical disturbance produced by a macroscopic sample. The sample was illuminated by collimated beams from white-light and thermal radiation sources. The agreement between simulated and experimental results, obtained by processing the captured images using a Fourier ptychographic algorithm, demonstrates that scanning with the slit the direction of the light diffracted by the sample permits achieving the image diversity necessary for successful implementation of the scanning diffracted-light imaging technique.

Accelerated and high-quality Fourier ptychographic method using a double truncated Wirtinger criteria

Fourier ptychographic microscopy (FPM) is a recently developed computational microscopy technique for wide-field-of-view and super-resolution complex imaging. Wirtinger-flow-based methods can effectively suppress noise and reduce data acquisition time, but they are time-consuming during the phase reconstruction. In this paper, we present a Wirtinger-flow-based reconstruction method for FPM, which combines the Poisson maximum likelihood objective function, improved truncated Wirtinger criteria, and improved adaptive momentum method. Both the simulation and experimental results demonstrate that the proposed method runs faster and the reconstruction quality is similar to or better than other state-of-the-art Wirtinger-flow-based methods.

Label-free ultra-sensitive visualization of structure below the diffraction resolution limit

For both fundamental study of biological processes and early diagnosis of diseases, information about nanoscale changes in tissue and cell structure is crucial. Nowadays, almost all currently known nanoscopy methods rely upon the contrast created by fluorescent stains attached to the object or molecule of interest. This causes limitations due to the impact of the label on the object and its environment, as well as its applicability in vivo, particularly in humans. In this paper, a new label-free approach to visualize small structure with nano-sensitivity to structural alterations is introduced. Numerically synthesized profiles of the axial spatial frequencies are used to probe the structure within areas whose size can be beyond the diffraction resolution limit. Thereafter, nanoscale structural alterations within such areas can be visualized and objects, including biological ones, can be investigated with sub-wavelength resolution, in vivo, in their natural environment. Some preliminary results, including numerical simulations and experiments, which demonstrate the nano-sensitivity and super-resolution ability of our approach, are presented.

Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow

Fourier ptychography is a recently developed imaging approach for large field-of-view and high-resolution microscopy. Here we model the Fourier ptychographic forward imaging process using a convolutional neural network (CNN) and recover the complex object information in a network training process. In this approach, the input of the network is the point spread function in the spatial domain or the coherent transfer function in the Fourier domain. The object is treated as 2D learnable weights of a convolutional or a multiplication layer. The output of the network is modeled as the loss function we aim to minimize. The batch size of the network corresponds to the number of captured low-resolution images in one forward/backward pass. We use a popular open-source machine learning library, TensorFlow, for setting up the network and conducting the optimization process. We analyze the performance of different learning rates, different solvers, and different batch sizes. It is shown that a large batch size with the Adam optimizer achieves the best performance in general. To accelerate the phase retrieval process, we also discuss a strategy to implement Fourier-magnitude projection using a multiplication neural network model. Since convolution and multiplication are the two most-common operations in imaging modeling, the reported approach may provide a new perspective to examine many coherent and incoherent systems. As a demonstration, we discuss the extensions of the reported networks for modeling single-pixel imaging and structured illumination microscopy (SIM). 4-frame resolution doubling is demonstrated using a neural network for SIM. The link between imaging systems and neural network modeling may enable the use of machine-learning hardware such as neural engine and tensor processing unit for accelerating the image reconstruction process. We have made our implementation code open-source for researchers.

Optical imaging featuring both long working distance and high spatial resolution by correcting the aberration of a large aperture lens

High-resolution optical imaging within thick objects has been a challenging task due to the short working distance of conventional high numerical aperture (NA) objective lenses. Lenses with a large physical diameter and thus a large aperture, such as microscope condenser lenses, can feature both a large NA and a long working distance. However, such lenses suffer from strong aberrations. To overcome this problem, we present a method to correct the aberrations of a transmission-mode imaging system that is composed of two condensers. The proposed method separately identifies and corrects aberrations of illumination and collection lenses of up to 1.2 NA by iteratively optimizing the total intensity of the synthetic aperture images in the forward and phase-conjugation processes. At a source wavelength of 785 nm, we demonstrated a spatial resolution of 372 nm at extremely long working distances of up to 1.6 mm, an order of magnitude improvement in comparison to conventional objective lenses. Our method of converting microscope condensers to high-quality objectives may facilitate increases in the imaging depths of super-resolution and expansion microscopes.

High-speed Fourier ptychographic microscopy based on programmable annular illuminations

High-throughput quantitative phase imaging (QPI) is essential to cellular phenotypes characterization as it allows high-content cell analysis and avoids adverse effects of staining reagents on cellular viability and cell signaling. Among different approaches, Fourier ptychographic microscopy (FPM) is probably the most promising technique to realize high-throughput QPI by synthesizing a wide-field, high-resolution complex image from multiple angle-variably illuminated, low-resolution images. However, the large dataset requirement in conventional FPM significantly limits its imaging speed, resulting in low temporal throughput. Moreover, the underlying theoretical mechanism as well as optimum illumination scheme for high-accuracy phase imaging in FPM remains unclear. Herein, we report a high-speed FPM technique based on programmable annular illuminations (AIFPM). The optical-transfer-function (OTF) analysis of FPM reveals that the low-frequency phase information can only be correctly recovered if the LEDs are precisely located at the edge of the objective numerical aperture (NA) in the frequency space. By using only 4 low-resolution images corresponding to 4 tilted illuminations matching a 10×, 0.4 NA objective, we present the high-speed imaging results of in vitro Hela cells mitosis and apoptosis at a frame rate of 25 Hz with a full-pitch resolution of 655 nm at a wavelength of 525 nm (effective NA = 0.8) across a wide field-of-view (FOV) of 1.77 mm², corresponding to a space-bandwidth-time product of 411 megapixels per second. Our work reveals an important capability of FPM towards high-speed high-throughput imaging of in vitro live cells, achieving video-rate QPI performance across a wide range of scales, both spatial and temporal.

Characterization of the reference wave in a compact digital holographic camera

A hologram is a recording of the interference between an unknown object wave and a coherent reference wave. Providing the object and reference waves are sufficiently separated in some region of space and the reference beam is known, a high-fidelity reconstruction of the object wave is possible. In traditional optical holography, high-quality reconstruction is achieved by careful reillumination of the holographic plate with the exact same reference wave that was used at the recording stage. To reconstruct high-quality digital holograms the exact parameters of the reference wave must be known mathematically. This paper discusses a technique that obtains the mathematical parameters that characterize a strongly divergent reference wave that originates from a fiber source in a new compact digital holographic camera. This is a lensless design that is similar in principle to a Fourier hologram, but because of the large numerical aperture, the usual paraxial approximations cannot be applied and the Fourier relationship is inexact. To characterize the reference wave, recordings of quasi-planar object waves are made at various angles of incidence using a Dammann grating. An optimization process is then used to find the reference wave that reconstructs a stigmatic image of the object wave regardless of the angle of incidence.

Adaptive pixel-super-resolved lensfree in-line digital holography for wide-field on-chip microscopy

High-resolution wide field-of-view (FOV) microscopic imaging plays an essential role in various fields of biomedicine, engineering, and physical sciences. As an alternative to conventional lens-based scanning techniques, lensfree holography provides a new way to effectively bypass the intrinsical trade-off between the spatial resolution and FOV of conventional microscopes. Unfortunately, due to the limited sensor pixel-size, unpredictable disturbance during image acquisition, and sub-optimum solution to the phase retrieval problem, typical lensfree microscopes only produce compromised imaging quality in terms of lateral resolution and signal-to-noise ratio (SNR). Here, we propose an adaptive pixel-super-resolved lensfree imaging (APLI) method which can solve, or at least partially alleviate these limitations. Our approach addresses the pixel aliasing problem by Z-scanning only, without resorting to subpixel shifting or beam-angle manipulation. Automatic positional error correction algorithm and adaptive relaxation strategy are introduced to enhance the robustness and SNR of reconstruction significantly. Based on APLI, we perform full-FOV reconstruction of a USAF resolution target (~29.85 mm²) and achieve half-pitch lateral resolution of 770 nm, surpassing 2.17 times of the theoretical Nyquist–Shannon sampling resolution limit imposed by the sensor pixel-size (1.67µm). Full-FOV imaging result of a typical dicot root is also provided to demonstrate its promising potential applications in biologic imaging.

Resolution-enhanced Fourier ptychographic microscopy based on high-numerical-aperture illuminations

High-resolution and wide field-of-view (FOV) microscopic imaging plays a central role in diverse applications such as high-throughput screening and digital pathology. However, conventional microscopes face inherent trade-offs between the spatial resolution and FOV, which are fundamental limited by the space-bandwidth product (SBP) of the optical system. The resolution-FOV tradeoff can be effectively decoupled in Fourier ptychography microscopy (FPM), however, to date, the effective imaging NA achievable with a typical FPM system is still limited to the range of 0.4–0.7. Herein, we report, for the first time, a high-NA illumination based resolution-enhanced FPM (REFPM) platform, in which a LED-array-based digital oil-immersion condenser is used to create high-angle programmable plane-wave illuminations, endowing a 10×, 0.4 NA objective lens with final effective imaging performance of 1.6 NA. With REFPM, we present the highest-resolution results with a unprecedented half-pitch resolution of 154 nm at a wavelength of 435 nm across a wide FOV of 2.34 mm², corresponding to an SBP of 98.5 megapixels (~50 times higher than that of the conventional incoherent microscope with the same resolution). Our work provides an important step of FPM towards high-resolution large-NA imaging applications, generating comparable resolution performance but significantly broadening the FOV of conventional oil-immersion microscopes.

SAVI: Synthetic apertures for long-range, subdiffraction-limited visible imaging using Fourier ptychography

Synthetic aperture radar is a well-known technique for improving resolution in radio imaging. Extending these synthetic aperture techniques to the visible light domain is not straightforward because optical receivers cannot measure phase information. We propose to use macroscopic Fourier ptychography (FP) as a practical means of creating a synthetic aperture for visible imaging to achieve subdiffraction-limited resolution. We demonstrate the first working prototype for macroscopic FP in a reflection imaging geometry that is capable of imaging optically rough objects. In addition, a novel image space denoising regularization is introduced during phase retrieval to reduce the effects of speckle and improve perceptual quality of the recovered high-resolution image. Our approach is validated experimentally where the resolution of various diffuse objects is improved sixfold.

3D-printed eagle eye: Compound microlens system for foveated imaging

We present a highly miniaturized camera, mimicking the natural vision of predators, by 3D-printing different multilens objectives directly onto a complementary metal-oxide semiconductor (CMOS) image sensor. Our system combines four printed doublet lenses with different focal lengths (equivalent to f = 31 to 123 mm for a 35-mm film) in a 2 × 2 arrangement to achieve a full field of view of 70° with an increasing angular resolution of up to 2 cycles/deg field of view in the center of the image. The footprint of the optics on the chip is below 300 μm × 300 μm, whereas their height is <200 μm. Because the four lenses are printed in one single step without the necessity for any further assembling or alignment, this approach allows for fast design iterations and can lead to a plethora of different miniaturized multiaperture imaging systems with applications in fields such as endoscopy, optical metrology, optical sensing, surveillance drones, or security.

Fourier ptychographic microscopy using a generalized Anscombe transform approximation of the mixed Poisson-Gaussian likelihood

Fourier ptychographic microscopy (FPM) is a novel computational microscopy technique that provides intensity images with both wide field-of-view (FOV) and high-resolution (HR). By combining ideas from synthetic aperture and phase retrieval, FPM iteratively stitches together a number of variably illuminated, low-resolution (LR) intensity images in Fourier space to reconstruct an HR complex sample image. In practice, however, the reconstruction of FPM is sensitive to the input noise, including Gaussian noise, Poisson shot noise or mixed Poisson-Gaussian noise. To efficiently address these noises, we developed a novel FPM reconstruction method termed generalized Anscombe transform approximation Fourier ptychographic (GATFP) reconstruction. The method utilizes the generalized Anscombe transform (GAT) approximation for the noise model, and a maximum likelihood theory is employed for formulating the FPM optimization problem. We validated the proposed method with both simulated data for quantitative evaluation and real experimental data captured using FPM setup. The results show that the proposed method achieves state-of-the-art performance in comparison with other approaches.

Surface wave illumination Fourier ptychographic microscopy

We propose a novel microscopy method, combining surface wave illumination and the Fourier ptychographic microscopy (FPM) algorithm to achieve super-resolution (SR) imaging. In our system, an oil-immersion objective lens is used to excite both the total internal reflection (TIR) evanescent waves and the surface plasmon waves (SPWs), which illuminate the sample with large wave vectors. Through the FPM algorithm, a resolution approximately twice that of conventional wide-field microscopy is obtained. Meanwhile, we could retrieve the sample's quantitative phase map in order to obtain its surface profile. Importantly, the field enhancement from a SPW has improved the contrast of the reconstructed images, which is critical for revealing the finer structural details of the specimen. In our experiments, we have imaged metallic gratings with a 120 or 150 nm wide line and trench features. We accurately retrieved their axial dimensions with a lateral resolution better than 240 nm that is close to the theoretical resolution of 215 nm, thus demonstrating the quantitative phase imaging capability of our technique. As this approach provides a label-free solution for intensity and phase imaging of samples with lateral resolution exceeding the limit introduced by the optical system, it can be potentially used in a wide range of noninvasive biological imaging applications.

Simulation study of dual-space microscopy

We explore the convergence of the dual-space microscopy (DSM) phase-recovery algorithm. DSM is an optical microscopy technique based on simultaneous observation of an object in the position and momentum spaces. We present one-dimensional (1D) simulations of this technique, demonstrating that the DSM technique is capable to resolve periodic and nonperiodic structures with a resolution well below the Rayleigh resolution limit. Using a simple and faster 1D version of the full 2D DSM algorithm, we simulated the DSM technique for thousands of different samples. Our results demonstrate that the DSM algorithm always converges rapidly to the correct optical disturbance.

Unconventional methods of imaging: computational microscopy and compact implementations

In the past two decades or so, there has been a renaissance of optical microscopy research and development. Much work has been done in an effort to improve the resolution and sensitivity of microscopes, while at the same time to introduce new imaging modalities, and make existing imaging systems more efficient and more accessible. In this review, we look at two particular aspects of this renaissance: computational imaging techniques and compact imaging platforms. In many cases, these aspects go hand-in-hand because the use of computational techniques can simplify the demands placed on optical hardware in obtaining a desired imaging performance. In the first main section, we cover lens-based computational imaging, in particular, light-field microscopy, structured illumination, synthetic aperture, Fourier ptychography, and compressive imaging. In the second main section, we review lensfree holographic on-chip imaging, including how images are reconstructed, phase recovery techniques, and integration with smart substrates for more advanced imaging tasks. In the third main section we describe how these and other microscopy modalities have been implemented in compact and field-portable devices, often based around smartphones. Finally, we conclude with some comments about opportunities and demand for better results, and where we believe the field is heading.

Functional imaging for regenerative medicine

In vivo imaging is a platform technology with the power to put function in its natural structural context. With the drive to translate stem cell therapies into pre-clinical and clinical trials, early selection of the right imaging techniques is paramount to success. There are many instances in regenerative medicine where the biological, biochemical, and biomechanical mechanisms behind the proposed function of stem cell therapies can be elucidated by appropriate imaging. Imaging techniques can be divided according to whether labels are used and as to whether the imaging can be done in vivo. In vivo human imaging places additional restrictions on the imaging tools that can be used. Microscopies and nanoscopies, especially those requiring fluorescent markers, have made an extraordinary impact on discovery at the molecular and cellular level, but due to their very limited ability to focus in the scattering tissues encountered for in vivo applications they are largely confined to superficial imaging applications in research laboratories. Nanoscopy, which has tremendous benefits in resolution, is limited to the near-field (e.g. near-field scanning optical microscope (NSNOM)) or to very high light intensity (e.g. stimulated emission depletion (STED)) or to slow stochastic events (photo-activated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM)). In all cases, nanoscopy is limited to very superficial applications. Imaging depth may be increased using multiphoton or coherence gating tricks. Scattering dominates the limitation on imaging depth in most tissues and this can be mitigated by the application of optical clearing techniques that can impose mild (e.g. topical application of glycerol) or severe (e.g. CLARITY) changes to the tissue to be imaged. Progression of therapies through to clinical trials requires some thought as to the imaging and sensing modalities that should be used. Smoother progression is facilitated by the use of comparable imaging modalities throughout the discovery and trial phases, giving label-free techniques an advantage wherever they can be used, although this is seldom considered in the early stages. In this paper, we will explore the techniques that have found success in aiding discovery in stem cell therapies and try to predict the likely technologies best suited to translation and future directions.

Multi-aperture foveated imaging

Foveated imaging, such as that evolved by biological systems to provide high angular resolution with a reduced space-bandwidth product, also offers advantages for man-made task-specific imaging. Foveated imaging systems using exclusively optical distortion are complex, bulky, and high cost, however. We demonstrate foveated imaging using a planar array of identical cameras combined with a prism array and superresolution reconstruction of a mosaicked image with a foveal variation in angular resolution of 5.9:1 and a quadrupling of the field of view. The combination of low-cost, mass-produced cameras and optics with computational image recovery offers enhanced capability of achieving large foveal ratios from compact, low-cost imaging systems.

Reflective Fourier ptychography

The Fourier ptychography technique in reflection mode has great potential applications in tissue imaging and optical inspection, but the current configuration either has a limitation on cut-off frequency or is not practical. By placing the imaging aperture stop outside the illumination path, the illumination numerical aperture (NA) can be greater than the imaging NA of the objective lens. Thus, the cut-off frequency achieved in the proposed optical system is greater than twice the objective's NA divided by the wavelength (2NAobj/λ ), which is the diffraction limit for the cut-off frequency in an incoherent epi-illumination configuration. We experimentally demonstrated that the synthesized NA is increased by a factor of 4.5 using the proposed optical concept. The key advantage of the proposed system is that it can achieve high-resolution imaging over a large field of view with a simple objective. It will have a great potential for applications in endoscopy, biomedical imaging, surface metrology, and industrial inspection.

Transfer function analysis in epi-illumination Fourier ptychography

This Letter explores Fourier ptychography (FP) using epi-illumination. The approach effectively modifies the FP transfer function to be coherent-like out to the incoherent limit of twice the numerical aperture over the wavelength 2NA/λ. Images reconstructed using this approach are shown to have higher contrast at finer details compared with images using incoherent illumination, indicating that the FP transfer function is superior in high spatial frequency regions.

Plane wave analysis of coherent holographic image reconstruction by phase transfer (CHIRPT)

Fluorescent imaging plays a critical role in a myriad of scientific endeavors, particularly in the biological sciences. Three-dimensional imaging of fluorescent intensity often requires serial data acquisition, that is, voxel-by-voxel collection of fluorescent light emitted throughout the specimen with a nonimaging single-element detector. While nonimaging fluorescence detection offers some measure of scattering robustness, the rate at which dynamic specimens can be imaged is severely limited. Other fluorescent imaging techniques utilize imaging detection to enhance collection rates. A notable example is light-sheet fluorescence microscopy, also known as selective-plane illumination microscopy, which illuminates a large region within the specimen and collects emitted fluorescent light at an angle either perpendicular or oblique to the illumination light sheet. Unfortunately, scattering of the emitted fluorescent light can cause blurring of the collected images in highly turbid biological media. We recently introduced an imaging technique called coherent holographic image reconstruction by phase transfer (CHIRPT) that combines light-sheet-like illumination with nonimaging fluorescent light detection. By combining the speed of light-sheet illumination with the scattering robustness of nonimaging detection, CHIRPT is poised to have a dramatic impact on biological imaging, particularly for in vivo preparations. Here we present the mathematical formalism for CHIRPT imaging under spatially coherent illumination and present experimental data that verifies the theoretical model.

Digital micromirror device-based laser-illumination Fourier ptychographic microscopy

We report a novel approach to Fourier ptychographic microscopy (FPM) by using a digital micromirror device (DMD) and a coherent laser source (532 nm) for generating spatially modulated sample illumination. Previously demonstrated FPM systems are all based on partially-coherent illumination, which offers limited throughput due to insufficient brightness. Our FPM employs a high power coherent laser source to enable shot-noise limited high-speed imaging. For the first time, a digital micromirror device (DMD), imaged onto the back focal plane of the illumination objective, is used to generate spatially modulated sample illumination field for ptychography. By coding the on/off states of the micromirrors, the illumination plane wave angle can be varied at speeds more than 4 kHz. A set of intensity images, resulting from different oblique illuminations, are used to numerically reconstruct one high-resolution image without obvious laser speckle. Experiments were conducted using a USAF resolution target and a fiber sample, demonstrating high-resolution imaging capability of our system. We envision that our approach, if combined with a coded-aperture compressive-sensing algorithm, will further improve the imaging speed in DMD-based FPM systems.

Electromagnetically induced holographic imaging in hybrid artificial molecule

We propose two schemes of holographic imaging with an object that has no any macro structure itself. The tunable electromagnetically induced grating (EIG) is such a kind of object. We obtain an EIG based on the periodically modulated strong susceptibility in a three-level ladder-type hybrid artificial molecule, which is comprised of a semiconductor quantum dot and a metal nanoparticle coupled via the Coulomb interaction. The holographic interference pattern is detected either directly in the way of classical holographic imaging with a coherent field being the imaging light, or indirectly and nonlocally in the way of two-photon coincidence measurement with a pair of entangled photons playing the role of imaging light. This work provides a practical prototype of electromagnetically induced transparency-based holographic solid-state devices for all-optical classical and quantum information processing.

Novel approach for label free super-resolution imaging in far field

Progress in the emerging areas of science and technology, such as bio- and nano-technologies, depends on development of corresponding techniques for imaging and probing the structures with high resolution. Recently, the far field diffraction resolution limit in the optical range has been circumvented and different methods of super-resolution optical microscopy have been developed. The importance of this breakthrough achievement has been recognized by Nobel Prize for Chemistry in 2014. However, the fluorescence based super-resolution techniques only function with fluorescent molecules (most of which are toxic and can destroy or lead to artificial results in living biological objects) and suffer from photobleaching. Here we show a new way to break the diffraction resolution limit, which is based on nano-sensitivity to internal structure. Instead of conventional image formation as 2D intensity distribution, in our approach images are formed as a result of comparison of the axial spatial frequency profiles, reconstructed for each image point. The proposed approach dramatically increases the lateral resolution even in presence of noise and allows objects to be imaged in their natural state, without any labels.

High numerical aperture Fourier ptychography: principle, implementation and characterization

Fourier ptychography (FP) utilizes illumination control and computational post-processing to increase the resolution of bright-field microscopes. In effect, FP extends the fixed numerical aperture (NA) of an objective lens to form a larger synthetic system NA. Here, we build an FP microscope (FPM) using a 40X 0.75NA objective lens to synthesize a system NA of 1.45. This system achieved a two-slit resolution of 335 nm at a wavelength of 632 nm. This resolution closely adheres to theoretical prediction and is comparable to the measured resolution (315 nm) associated with a standard, commercially available 1.25 NA oil immersion microscope. Our work indicates that Fourier ptychography is an attractive method to improve the resolution-versus-NA performance, increase the working distance, and enlarge the field-of-view of high-resolution bright-field microscopes by employing lower NA objectives.

Resolution enhancement of spectrum normalization in synthetic aperture digital holographic microscopy

This study describes the overlapping of spatial frequency bands for synthetic aperture in digital holography using spectrum normalization to effectively enhance the spatial resolutions of image reconstruction. Synthesized spectrum swelling induced by excessive frequency overlaps can be normalized through the inverse apodization of an adjustable window function, which is similar to the effects of suppressing low-frequency expansion and strengthening high-frequency components of the reconstructed images. The results indicated that using the normalized spectrum synthesis that requires only a few frequency bands effectively enhances the spatial resolution and phase sensitivity of reconstructed images in digital holographic microscopy.

An image cytometer based on angular spatial frequency processing and its validation for rapid detection and quantification of waterborne microorganisms

Image cytometer based on angular spatial frequency processing for the early detection of waterborne bacteria.

Digital pathology with Fourier ptychography

Fourier ptychographic microscopy (FPM) is a recently introduced method of acquiring high-resolution, wide field of view (FOV) giga-pixel histology images. The FPM procedure first acquires a sequence of low-resolution images of a sample under variable-angle illumination. It then combines these images using a novel phase retrieval algorithm to improve the employed microscope's resolution beyond its conventional limit. Here, we first describe how FPM's resolution improvement can enhance wide FOV histology imaging. Second, we show that FPM also records a thin sample's optical phase, which can help pathologists digitally extract as much information as possible from a given histology slide.