Fourier ptychographic microscopy for filtration-based circulating tumor cell enumeration and analysis

Deep-ultraviolet Fourier ptychography (DUV-FP) for label-free biochemical imaging via feature-domain optimization

We present deep-ultraviolet Fourier ptychography (DUV-FP) for high-resolution chemical imaging of biological specimens in their native state without exogenous stains. This approach uses a customized 265-nm DUV LED array for angle-varied illumination, leveraging the unique DUV absorption properties of biomolecules at this wavelength region. We implemented a robust feature-domain optimization framework to overcome common challenges in Fourier ptychographic reconstruction, including vignetting, pupil aberrations, stray light problems, intensity variations, and other systematic errors. By using a 0.12 numerical aperture low-resolution objective lens, our DUV-FP prototype can resolve the 345-nm linewidth on a resolution target, demonstrating at least a four-fold resolution gain compared to the captured raw images. Testing on various biospecimens demonstrates that DUV-FP significantly enhances absorption-based chemical contrast and reveals detailed structural and molecular information. To further address the limitations of conventional FP in quantitative phase imaging, we developed a spatially coded DUV-FP system. This platform enables true quantitative phase imaging of biospecimens with DUV light, overcoming the non-uniform phase response inherent in traditional microscopy techniques. The demonstrated advancements in high-resolution, label-free chemical imaging may accelerate developments in digital pathology, potentially enabling rapid, on-site analysis of biopsy samples in clinical settings.

Fourier Ptychographic Microscopy 10 Years on: A Review

Fourier ptychographic microscopy (FPM) emerged as a prominent imaging technique in 2013, attracting significant interest due to its remarkable features such as precise phase retrieval, expansive field of view (FOV), and superior resolution. Over the past decade, FPM has become an essential tool in microscopy, with applications in metrology, scientific research, biomedicine, and inspection. This achievement arises from its ability to effectively address the persistent challenge of achieving a trade-off between FOV and resolution in imaging systems. It has a wide range of applications, including label-free imaging, drug screening, and digital pathology. In this comprehensive review, we present a concise overview of the fundamental principles of FPM and compare it with similar imaging techniques. In addition, we present a study on achieving colorization of restored photographs and enhancing the speed of FPM. Subsequently, we showcase several FPM applications utilizing the previously described technologies, with a specific focus on digital pathology, drug screening, and three-dimensional imaging. We thoroughly examine the benefits and challenges associated with integrating deep learning and FPM. To summarize, we express our own viewpoints on the technological progress of FPM and explore prospective avenues for its future developments.

Fast Fourier ptychographic quantitative phase microscopy for in vitro label-free imaging

Quantitative phase microscopy (QPM) is indispensable in biomedical research due to its advantages in unlabeled transparent sample thickness quantification and obtaining refractive index information. Fourier ptychographic microscopy (FPM) is among the most promising QPM methods, incorporating multi-angle illumination and iterative phase recovery for high-resolution quantitative phase imaging (QPI) of large cell populations over a wide field of-view (FOV) in a single pass. However, FPM is limited by data redundancy and sequential acquisition strategies, resulting in low imaging efficiency, which in turn limits its real-time application in in vitro label-free imaging. Here, we report a fast QPM based on Fourier ptychography (FQP-FPM), which uses an optimized annular downsampling and parallel acquisition strategy to minimize the amount of data required in the front end and reduce the iteration time of the back-end algorithm (3.3% and 4.4% of conventional FPM, respectively). Theoretical and data redundancy analyses show that FQP-FPM can realize high-throughput quantitative phase reconstruction at thrice the resolution of the coherent diffraction limit by acquiring only ten raw images, providing a precondition for in vitro label-free real-time imaging. The FQP-FPM application was validated for various in vitro label-free live-cell imaging. Cell morphology and subcellular phenomena in different periods were observed with a synthetic aperture of 0.75 NA at a 10× FOV, demonstrating its advantages and application potential for fast high-throughput QPI.

Long-range Fourier ptychographic imaging of the dynamic object with a single camera

Fourier ptychographic imaging technology is a new imaging method proposed in recent years. This technology captures multiple low-resolution images, and synthesizes them into a high-resolution image in the Fourier domain by a phase retrieval algorithm, breaking through the diffraction limit of the lens. In the field of macroscopic Fourier ptychographic imaging, most of the existing research generally focus on high-resolution imaging of static objects, and applying Fourier ptychographic imaging technology to dynamic objects is a hot research area now. At present, most of the researches are to use camera arrays combined with multiplexed lighting, deep learning or other algorithms, but the implementation of these methods is complicated or costly. Based on the diffraction theory of Fourier optics, this paper proposes that by expanding and focusing the illumination area, we can apply Fourier ptychographic imaging technology with a single camera to moving objects within a certain range. Theoretical analysis and experiments prove the feasibility of the proposed method. We successfully achieve high-resolution imaging of the dynamic object, increasing the resolution by about 2.5 times. This paper also researches the impact of speckles in the illuminated area on imaging results and proposes a processing method to reduce the impact of speckles.

Design of Fourier ptychographic illuminator for single full-FOV reconstruction

Fourier ptychographic microscopy (FPM) is a spatial-temporal-modulation high-throughput imaging technique via a sequential angle-varied LED illumination. Therefore, the illuminator is one of the key components and the design of this illuminator is significant. However, because of the property of spherical wave, partial coherence, and aperture-induced vignetting, the acquired images must be processed in blocks first, and rely on parallel reconstruction via a graphics processing unit (GPU). The high cost makes it unappealing compared with commercial whole slide imaging system via a low-cost central processing unit (CPU). Especially, the vignetting severely destroys the space-invariant model and induces obvious artifacts in FPM, which is the most difficult problem. The conventional method is to divide the field of view (FOV) into many tiles and omit those imperfect images, which is crude and may discards low frequency information. In this paper, we reevaluated the conditions of vignetting in FPM. Through our analysis, the maximum side length of FOV is 0.759 mm for a single full-FOV reconstruction via a 4×/0.1 NA objective and a 4 mm spacing LED array in theory, while almost 1.0 mm can be achieved in practice due to the tolerance of algorithm. We found that FPM system can treat the vignetting coefficient Vf below 0.1 as brightfield images and Vf lager than 0.9 as darkfield images, respectively. We reported an optimized distribution for designing an illuminator without vignetting effect according to the off-the-shelf commercial products, which can reconstruct full FOV in one time via a CPU. By adjusting the distribution of LED units, the system could retrieve the object with the side length of FOV up to 3.8 mm for a single full-FOV reconstruction, which achieves the largest FOV that a typical 4×/0.1 NA objective with the field number of 22 mm can afford.

Spatial- and Fourier-domain ptychography for high-throughput bio-imaging

First envisioned for determining crystalline structures, ptychography has become a useful imaging tool for microscopists. However, ptychography remains underused by biomedical researchers due to its limited resolution and throughput in the visible light regime. Recent developments of spatial- and Fourier-domain ptychography have successfully addressed these issues and now offer the potential for high-resolution, high-throughput optical imaging with minimal hardware modifications to existing microscopy setups, often providing an excellent trade-off between resolution and field of view inherent to conventional imaging systems, giving biomedical researchers the best of both worlds. Here, we provide extensive information to enable the implementation of ptychography by biomedical researchers in the visible light regime. We first discuss the intrinsic connections between spatial-domain coded ptychography and Fourier ptychography. A step-by-step guide then provides the user instructions for developing both systems with practical examples. In the spatial-domain implementation, we explain how a large-scale, high-performance blood-cell lens can be made at negligible expense. In the Fourier-domain implementation, we explain how adding a low-cost light source to a regular microscope can improve the resolution beyond the limit of the objective lens. The turnkey operation of these setups is suitable for use by professional research laboratories, as well as citizen scientists. Users with basic experience in optics and programming can build the setups within a week. The do-it-yourself nature of the setups also allows these procedures to be implemented in laboratory courses related to Fourier optics, biomedical instrumentation, digital image processing, robotics and capstone projects.

Automatic detection of circulating tumor cells and cancer associated fibroblasts using deep learning

Circulating tumor cells (CTCs) and cancer-associated fibroblasts (CAFs) from whole blood are emerging as important biomarkers that potentially aid in cancer diagnosis and prognosis. The microfilter technology provides an efficient capture platform for them but is confounded by two challenges. First, uneven microfilter surfaces makes it hard for commercial scanners to obtain images with all cells in-focus. Second, current analysis is labor-intensive with long turnaround time and user-to-user variability. Here we addressed the first challenge through developing a customized imaging system and data pre-processing algorithms. Utilizing cultured cancer and CAF cells captured by microfilters, we showed that images from our custom system are 99.3% in-focus compared to 89.9% from a top-of-the-line commercial scanner. Then we developed a deep-learning-based method to automatically identify tumor cells serving to mimic CTC (mCTC) and CAFs. Our deep learning method achieved precision and recall of 94% (± 0.2%) and 96% (± 0.2%) for mCTC detection, and 93% (± 1.7%) and 84% (± 3.1%) for CAF detection, significantly better than a conventional computer vision method, whose numbers are 92% (± 0.2%) and 78% (± 0.3%) for mCTC and 58% (± 3.9%) and 56% (± 3.5%) for CAF. Our custom imaging system combined with deep learning cell identification method represents an important advance on CTC and CAF analysis.

Depth-multiplexed ptychographic microscopy for high-throughput imaging of stacked bio-specimens on a chip

Imaging a large number of bio-specimens at high speed is essential for many biomedical applications. The common strategy is to place specimens at different lateral positions and image them sequentially. Here we report a new on-chip imaging strategy, termed depth-multiplexed ptychographic microscopy (DPM), for parallel imaging and sensing at high speed. Different from the common strategy, DPM stacks multiple specimens in the axial direction and images the entire z-stack all at once. In our prototype platform, we modify a low-cost car mirror for programmable steering of the incident laser beam. A blood-coated image sensor is then placed underneath the stacked sample for acquiring the resulting diffraction patterns. With the captured images, we perform blind recovery of the incident beam angle and model different layers of the stacked sample as different coded surfaces for object reconstruction. For in vitro experiment, we demonstrate time-lapse cell culture monitoring by imaging 3 stacked microfluidic channels on the coded sensor. For high-throughput cytometric analysis, we image 5 stacked brain sections with a 205-mm² field of view in ∼50 s. Cytometric analysis is also performed to quantify the cellular proliferation biomarkers on the slides. The DPM approach adds a new degree of freedom for data multiplexing in microscopy, enabling parallel imaging of multiple specimens using a single detector. The demonstrated 6-mm depth of field is among the longest ones in microscopy imaging. The novel depth-multiplexed configuration also complements the miniaturization provided by microfluidics devices, offering a solution for on-chip sensing and imaging with efficient sample handling.

Beyond conventional microscopy: Observing kidney tissues by means of fourier ptychography

Kidney microscopy is a mainstay in studying the morphological structure, physiology and pathology of kidney tissues, as histology provides important results for a reliable diagnosis. A microscopy modality providing at same time high-resolution images and a wide field of view could be very useful for analyzing the whole architecture and the functioning of the renal tissue. Recently, Fourier Ptychography (FP) has been proofed to yield images of biology samples such as tissues and in vitro cells while providing high resolution and large field of view, thus making it a unique and attractive opportunity for histopathology. Moreover, FP offers tissue imaging with high contrast assuring visualization of small desirable features, although with a stain-free mode that avoids any chemical process in histopathology. Here we report an experimental measuring campaign for creating the first comprehensive and extensive collection of images of kidney tissues captured by this FP microscope. We show that FP microscopy unlocks a new opportunity for the physicians to observe and judge renal tissue slides through the novel FP quantitative phase-contrast microscopy. Phase-contrast images of kidney tissue are analyzed by comparing them with the corresponding renal images taken under a conventional bright-field microscope both for stained and unstained tissue samples of different thicknesses. In depth discussion on the advantages and limitations of this new stain-free microscopy modality is reported, showing its usefulness over the classical light microscopy and opening a potential route for using FP in clinical practice for histopathology of kidney.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

Parallel Fourier ptychographic microscopy reconstruction method based on FPGA

Fourier ptychographic microscopy (FPM) can bypass the limitation of spatial bandwidth product to get images with large field-of-view and high resolution. The complicated sequential iterative calculation in the FPM reconstruction process reduces the reconstruction efficiency of the FPM. Therefore, we propose a parallel FPM reconstruction method based on field programmable gate array (FPGA) to accelerate the FPM reconstruction process. Using this method, multiple sub-regions in the Fourier domain can be computed in parallel and we customize a dedicated high-performance computational architecture for this approach. We deploy 4 FPM reconstruct computing architectures with a parallelism of 4 in a FPGA to compute the FPM reconstruction process, achieving the speed nearly 180 times faster than traditional methods. The proposed method provides a new perspective of parallel computing for FPM reconstruction.

Fourier ptychographic microscopy with untrained deep neural network priors

We propose a physics-assisted deep neural network scheme in Fourier ptychographic microscopy (FPM) using untrained deep neural network priors (FPMUP) to achieve a high-resolution image reconstruction from multiple low-resolution images. Unlike the traditional training type of deep neural network that requires a large labelled dataset, this proposed scheme does not require training and instead outputs the high-resolution image by optimizing the parameters of neural networks to fit the experimentally measured low-resolution images. Besides the amplitude and phase of the sample function, another two parallel neural networks that generate the general pupil function and illumination intensity factors are incorporated into the carefully designed neural networks, which effectively improves the image quality and robustness when both the aberration and illumination intensity fluctuation are present in FPM. Reconstructions using simulated and experimental datasets are demonstrated, showing that the FPMUP scheme has better image quality than the traditional iterative algorithms, especially for the phase recovery, but at the expense of increasing computational cost. Most importantly, it is found that the FPMUP scheme can predict the Fourier spectrum of the sample outside synthetic aperture of FPM and thus eliminate the ringing effect of the recovered images due to the spectral truncation. Inspired by deep image prior in the field of image processing, we may impute the expansion of Fourier spectrums to the deep prior rooted in the architecture of the careful designed four parallel deep neural networks. We envisage that the resolution of FPM will be further enhanced if the Fourier spectrum of the sample outside the synthetic aperture of FPM is accurately predicted.

Batch-based alternating direction methods of multipliers for Fourier ptychography

Fourier ptychography (FP) has been developed as a general imaging tool for various applications. However, the redundancy data has to be enforced to get a stable recovery, leading to a large dataset and a high computational cost. Based on the additive property of the optical pupils in FP recovery, we report batch-based alternating direction methods of multipliers (ADMM) for FP reconstruction. The reported scheme is performed by implementing partial updates in sub-problems of the standard ADMM. We validate the reconstruction performance using both simulated and experimental measurements. Compared with the embedded pupil function recovery (EPRY) algorithm, the proposed algorithms can converge faster and produce higher-quality images.

Fourier Ptychographic Microscopy via Alternating Direction Method of Multipliers

Fourier ptychographic microscopy (FPM) has risen as a promising computational imaging technique that breaks the trade-off between high resolution and large field of view (FOV). Its reconstruction is normally formulated as a blind phase retrieval problem, where both the object and probe have to be recovered from phaseless measured data. However, the stability and reconstruction quality may dramatically deteriorate in the presence of noise interference. Herein, we utilized the concept of alternating direction method of multipliers (ADMM) to solve this problem (termed ADMM-FPM) by breaking it into multiple subproblems, each of which may be easier to deal with. We compared its performance against existing algorithms in both simulated and practical FPM platform. It is found that ADMM-FPM method belongs to a global optimization algorithm with a high degree of parallelism and thus results in a more stable and robust phase recovery under noisy conditions. We anticipate that ADMM will rekindle interest in FPM as more modifications and innovations are implemented in the future.

Robust Fourier ptychographic microscopy via a physics-based defocusing strategy for calibrating angle-varied LED illumination

Fourier ptychographic microscopy (FPM) is a recently developed computational imaging technique for wide-field, high-resolution microscopy with a high space-bandwidth product. It integrates the concepts of synthetic aperture and phase retrieval to surpass the resolution limit imposed by the employed objective lens. In the FPM framework, the position of each sub-spectrum needs to be accurately known to ensure the success of the phase retrieval process. Different from the conventional methods with mechanical adjustment or data-driven optimization strategies, here we report a physics-based defocusing strategy for correcting large-scale positional deviation of the LED illumination in FPM. Based on a subpixel image registration process with a defocused object, we can directly infer the illumination parameters including the lateral offsets of the light source, the in-plane rotation angle of the LED array, and the distance between the sample and the LED board. The feasibility and effectiveness of our method are validated with both simulations and experiments. We show that the reported strategy can obtain high-quality reconstructions of both the complex object and pupil function even the LED array is randomly placed under the sample with both unknown lateral offsets and rotations. As such, it enables the development of robust FPM systems by reducing the requirements on fine mechanical adjustment and data-driven correction in the construction process.

Ptychographic sensor for large-scale lensless microbial monitoring with high spatiotemporal resolution

Traditional microbial detection methods often rely on the overall property of microbial cultures and cannot resolve individual growth event at high spatiotemporal resolution. As a result, they require bacteria to grow to confluence and then interpret the results. Here, we demonstrate the application of an integrated ptychographic sensor for lensless cytometric analysis of microbial cultures over a large scale and with high spatiotemporal resolution. The reported device can be placed within a regular incubator or used as a standalone incubating unit for long-term microbial monitoring. For longitudinal study where massive data are acquired at sequential time points, we report a new temporal-similarity constraint to increase the temporal resolution of ptychographic reconstruction by 7-fold. With this strategy, the reported device achieves a centimeter-scale field of view, a half-pitch spatial resolution of 488 nm, and a temporal resolution of 15-s intervals. For the first time, we report the direct observation of bacterial growth in a 15-s interval by tracking the phase wraps of the recovered images, with high phase sensitivity like that in interferometric measurements. We also characterize cell growth via longitudinal dry mass measurement and perform rapid bacterial detection at low concentrations. For drug-screening application, we demonstrate proof-of-concept antibiotic susceptibility testing and perform single-cell analysis of antibiotic-induced filamentation. The combination of high phase sensitivity, high spatiotemporal resolution, and large field of view is unique among existing microscopy techniques. As a quantitative and miniaturized platform, it can improve studies with microorganisms and other biospecimens at resource-limited settings.

Review of bio-optical imaging systems with a high space-bandwidth product

Optical imaging has served as a primary method to collect information about biosystems across scales-from functionalities of tissues to morphological structures of cells and even at biomolecular levels. However, to adequately characterize a complex biosystem, an imaging system with a number of resolvable points, referred to as a space-bandwidth product (SBP), in excess of one billion is typically needed. Since a gigapixel-scale far exceeds the capacity of current optical imagers, compromises must be made to obtain either a low spatial resolution or a narrow field-of-view (FOV). The problem originates from constituent refractive optics-the larger the aperture, the more challenging the correction of lens aberrations. Therefore, it is impractical for a conventional optical imaging system to achieve an SBP over hundreds of millions. To address this unmet need, a variety of high-SBP imagers have emerged over the past decade, enabling an unprecedented resolution and FOV beyond the limit of conventional optics. We provide a comprehensive survey of high-SBP imaging techniques, exploring their underlying principles and applications in bioimaging.

Autofocusing technologies for whole slide imaging and automated microscopy

Whole slide imaging (WSI) has moved digital pathology closer to diagnostic practice in recent years. Due to the inherent tissue topography variability, accurate autofocusing remains a critical challenge for WSI and automated microscopy systems. The traditional focus map surveying method is limited in its ability to acquire a high degree of focus points while still maintaining high throughput. Real-time approaches decouple image acquisition from focusing, thus allowing for rapid scanning while maintaining continuous accurate focus. This work reviews the traditional focus map approach and discusses the choice of focus measure for focal plane determination. It also discusses various real-time autofocusing approaches including reflective-based triangulation, confocal pinhole detection, low-coherence interferometry, tilted sensor approach, independent dual sensor scanning, beam splitter array, phase detection, dual-LED illumination and deep-learning approaches. The technical concepts, merits and limitations of these methods are explained and compared to those of a traditional WSI system. This review may provide new insights for the development of high-throughput automated microscopy imaging systems that can be made broadly available and utilizable without loss of capacity.

Virtual brightfield and fluorescence staining for Fourier ptychography via unsupervised deep learning

Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. Acquiring color images of histology slides often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learning approach for performing unsupervised image-to-image translation of FPM reconstructions. A cycle-consistent adversarial network with multiscale structure similarity loss is trained to perform virtual brightfield and fluorescence staining of the recovered FPM images. In the training stage, we feed the network with two sets of unpaired images: (1) monochromatic FPM recovery and (2) color or fluorescence images captured using a regular microscope. In the inference stage, the network takes the FPM input and outputs a virtually stained image with reduced coherent artifacts and improved image quality. We test the approach on various samples with different staining protocols. High-quality color and fluorescence reconstructions validate its effectiveness.

High-resolution and large field-of-view Fourier ptychographic microscopy and its applications in biomedicine

Fourier ptychographic microscopy (FPM) is a promising and fast-growing computational imaging technique with high resolution, wide field-of-view (FOV) and quantitative phase recovery, which effectively tackles the problems of phase loss, aberration-introduced artifacts, narrow depth-of-field and the trade-off between resolution and FOV in conventional microscopy simultaneously. In this review, we provide a comprehensive roadmap of microscopy, the fundamental principles, advantages, and drawbacks of existing imaging techniques, and the significant roles that FPM plays in the development of science. Since FPM is an optimization problem in nature, we discuss the framework and related work. We also reveal the connection of Euler's formula between FPM and structured illumination microscopy. We review recent advances in FPM, including the implementation of high-precision quantitative phase imaging, high-throughput imaging, high-speed imaging, three-dimensional imaging, mixed-state decoupling, and introduce the prosperous biomedical applications. We conclude by discussing the challenging problems and future applications. FPM can be extended to a kind of framework to tackle the phase loss and system limits in the imaging system. This insight can be used easily in speckle imaging, incoherent imaging for retina imaging, large-FOV fluorescence imaging, etc.

An efficient Fourier ptychographic microscopy method based on optimized pattern of LED angle illumination

Considering angle diversity and synthetic aperture, Fourier ptychographic microscopy (FPM) could address contradiction of high resolution and wide field of view. However, in the conventional FPM method, large capture quantity leads to poor efficiency. So, an efficient FPM method based on optimized pattern of LED angle illumination is proposed. Firstly, from position relationship between the LED, aperture and sample, we obtain the theoretical expandable spectrum range and the spectrum distribution in Fourier space of all LEDs. Secondly, we u se image quality assessment methods to extract differential expressions between an arbitrary LED illumination and full LEDs illumination. Thirdly, the optimized angle illumination strategy is achieved according to the analysis of different expressions. Based on this method, we design a rhombus-based illumination method for the LED array to accelerate the FPM efficiency. Finally, we validate the effectiveness and efficiency of our method with both simulated and real experiments. The results indicate that our method can effectively improve the efficiency for FPM without sacrificing image reconstruction quality.

Fourier ptychography: current applications and future promises

Traditional imaging systems exhibit a well-known trade-off between the resolution and the field of view of their captured images. Typical cameras and microscopes can either "zoom in" and image at high-resolution, or they can "zoom out" to see a larger area at lower resolution, but can rarely achieve both effects simultaneously. In this review, we present details about a relatively new procedure termed Fourier ptychography (FP), which addresses the above trade-off to produce gigapixel-scale images without requiring any moving parts. To accomplish this, FP captures multiple low-resolution, large field-of-view images and computationally combines them in the Fourier domain into a high-resolution, large field-of-view result. Here, we present details about the various implementations of FP and highlight its demonstrated advantages to date, such as aberration recovery, phase imaging, and 3D tomographic reconstruction, to name a few. After providing some basics about FP, we list important details for successful experimental implementation, discuss its relationship with other computational imaging techniques, and point to the latest advances in the field while highlighting persisting challenges.

Wide-field, high-resolution lensless on-chip microscopy via near-field blind ptychographic modulation

We report a novel lensless on-chip microscopy platform based on near-field blind ptychographic modulation. In this platform, we place a thin diffuser in between the object and the image sensor for light wave modulation. By blindly scanning the unknown diffuser to different x-y positions, we acquire a sequence of modulated intensity images for quantitative object recovery. Different from previous ptychographic implementations, we employ a unit magnification configuration with a Fresnel number of ∼50 000, which is orders of magnitude higher than those of previous ptychographic setups. The unit magnification configuration allows us to have the entire sensor area, 6.4 mm by 4.6 mm, as the imaging field of view. The ultra-high Fresnel number enables us to directly recover the positional shift of the diffuser in the phase retrieval process, addressing the positioning accuracy issue plaguing regular ptychographic experiments. In our implementation, we use a low-cost, DIY scanning stage to perform blind diffuser modulation. Precise mechanical scanning that is critical in conventional ptychography experiments is no longer needed in our setup. We further employ an up-sampling phase retrieval scheme to bypass the resolution limit set by the imager pixel size and demonstrate a half-pitch resolution of 0.78 μm. We validate the imaging performance via in vitro cell cultures, transparent and stained tissue sections, and a thick biological sample. We show that the recovered quantitative phase map can be used to perform effective cell segmentation of a dense yeast culture. We also demonstrate 3D digital refocusing of the thick biological sample based on the recovered wavefront. The reported platform provides a cost-effective and turnkey solution for large field-of-view, high-resolution, and quantitative on-chip microscopy. It is adaptable for a wide range of point-of-care-, global-health-, and telemedicine-related applications.

Whole slide imaging of circulating tumor cells captured on a capillary microchannel device

Liquid biopsy with circulating tumor cells (CTCs) can aid in cancer detection at early stages and determine whether a certain treatment is effective or not. However, existing CTC techniques focus on one or two aspects of CTC management including sampling, enrichment, enumeration, and treatment selection. This paper reports an integrated capillary microchannel device that allows efficient capturing of CTCs with a wide microchannel, rapid enumeration of captured CTCs with whole slide cell imaging, and in situ drug testing with captured CTCs. Blood is drawn into the microchannel whose height is appropriate to the diameter of cancer cells, while its width is a thousand times larger than the diameter of cancer cells. The inner bottom surface of the microchannel is modified with long chain polymers that have cell adhesive ends to efficiently capture CTCs from blood. With this design, cells including CTCs are forced to move through the polymer coated microchannel, and the chance of cell adhesive ends interacting with specific antigens overexpressed on surfaces of cancer cells is significantly enhanced without a channel blockage issue. Captured CTCs are enumerated with a whole slide imaging platform via dual LED autofocusing technology then exposed to anti-cancer drugs, followed by live/dead assay and fluorescence imaging. Given its straightforward, easy and powerless operation, this device with whole slide imaging will be useful for cancer diagnosis, prognosis and point-of-care treatments.

Apodized coherent transfer function constraint for partially coherent Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a recently developed computational microscopy approach that produces both wide field-of-view (FOV) and high resolution (HR) intensity and a phase image of the sample. Inspired by the ideas of synthetic aperture and phase retrieval, FPM iteratively stitches multiple low-resolution (LR) images with variable illumination angles in Fourier space to reconstruct an HR complex image. Typically, FPM illuminating the sample with an LED array is approximated as a coherent imaging process, and the coherent transfer function (CTF) is imposed as a support constraint in Fourier space. However, a millimeter-scale LED is inapposite to be treated as a coherent light source. As a result, the quality of reconstructed image is degraded by the inappropriate approximation. In this paper, we analyze the coherence of an FPM system and propose a novel constraint approach termed Apodized CTF (AC) constraint in Fourier space. Results on both simulated data and actual captured data show that this new constraint is more stable and robust than CTF. This approach can also relax the coherence requirement of illumination. In addition, it is simple, does not require additional computations, and is easy to be embedded in almost all the reconstruction algorithms proposed so far.

Computational aberration compensation by coded-aperture-based correction of aberration obtained from optical Fourier coding and blur estimation

We report a novel generalized optical measurement system and computational approach to determine and correct aberrations in optical systems. The system consists of a computational imaging method capable of reconstructing an optical system's pupil function by adapting overlapped Fourier coding to an incoherent imaging modality. It recovers the high-resolution image latent in an aberrated image via deconvolution. The deconvolution is made robust to noise by using coded apertures to capture images. We term this method coded-aperture-based correction of aberration obtained from overlapped Fourier coding and blur estimation (CACAO-FB). It is well-suited for various imaging scenarios where aberration is present and where providing a spatially coherent illumination is very challenging or impossible. We report the demonstration of CACAO-FB with a variety of samples including an in vivo imaging experiment on the eye of a rhesus macaque to correct for its inherent aberration in the rendered retinal images. CACAO-FB ultimately allows for an aberrated imaging system to achieve diffraction-limited performance over a wide field of view by casting optical design complexity to computational algorithms in post-processing.

Circulating Tumor Cells: Strategies for Capture, Analyses, and Propagation

Circulating tumor cells (CTCs) play a central role in tumor dissemination and metastases, which are ultimately responsible for most cancer deaths. Technologies that allow for identification and enumeration of rare CTC from cancer patients' blood have already established CTC as an important clinical biomarker for cancer diagnosis and prognosis. Indeed, current efforts to robustly characterize CTC as well as the associated cells of the tumor microenvironment such as circulating cancer associated fibroblasts (cCAF), are poised to unmask key insights into the metastatic process. Ultimately, the clinical utility of CTC will be fully realized once CTC can be reliably cultured and proliferated as a biospecimen for precision management of cancer patients, and for discovery of novel therapeutics. In this review, we highlight the latest CTC capture and analyses technologies, and discuss in vitro strategies for culturing and propagating CTC.

Fourier ptychographic microscopy reconstruction with multiscale deep residual network

Fourier ptychographic microscopy (FPM) is a newly developed microscopic technique for large field of view, high-resolution and quantitative phase imaging by combining the techniques from ptychographic imaging, aperture synthesizing and phase retrieval. In FPM, an LED array is utilized to illuminate the specimen from different angles and the corresponding intensity images are synthesized to reconstruct a high-resolution complex field. As a flexible and low-cost approach to achieve high-resolution, wide-field and quantitative phase imaging, FPM is of enormous potential in biomedical applications such as hematology and pathology. Conventionally, the FPM reconstruction problem is solved by using a phase retrieval method, termed Alternate Projection. By iteratively updating the Fourier spectrum with low-resolution-intensity images, the result converges to a high-resolution complex field. Here we propose a new FPM reconstruction framework with deep learning methods and design a multiscale, deep residual neural network for FPM reconstruction. We employ the widely used open-source deep learning library PyTorch to train and test our model and carefully choose the hyperparameters of our model. To train and analyze our model, we build a large-scale simulation dataset with an FPM imaging model and an actual dataset captured with an FPM system. The simulation dataset and actual dataset are separated as training datasets and test datasets, respectively. Our model is trained with the simulation training dataset and fine tuned with the fine-tune dataset, which contains actual training data. Both our model and the conventional method are tested on the simulation test dataset and the actual test dataset to evaluate the performances. We also show the reconstruction result of another neural network-based method for comparison. The experiments demonstrate that our model achieves better reconstruction results and consumes much less time than conventional methods. The results also point out that our model is more robust under system aberrations such as noise and blurring (fewer intensity images) compared with conventional methods.

Fast and robust misalignment correction of Fourier ptychographic microscopy for full field of view reconstruction

Fourier ptychographic microscopy (FPM) is a newly developed computational imaging technique that can provide gigapixel images with both high resolution (HR) and wide field of view (FOV). However, there are two possible reasons for position misalignment, which induce a degradation of the reconstructed image. The first one is the position misalignment of the LED array, which can largely be eliminated during the experimental system building process. The more important one is the segment-dependent position misalignment. Note that, this segment-dependent positional misalignment still exists, even after we correct the central coordinates of every small segment. In this paper, we carefully analyze this segment-dependent misalignment and find that this global shift matters more, compared with the rotational misalignments. According to this fact, we propose a robust and fast method to correct the two factors of position misalignment of the FPM, termed as misalignment correction for the FPM misalignment correction (mcFPM). Although different regions in the FOV have different sensitivities to the position misalignment, the experimental results show that the mcFPM is robust with respect to the elimination of each region. Compared with the state-of-the-art methods, the mcFPM is much faster.

Fourier Ptychographic Microscopy for Rapid, High-Resolution Imaging of Circulating Tumor Cells Enriched by Microfiltration

Examining the hematogenous compartment for evidence of metastasis has increased significantly within the oncology research community in recent years, due to the development of technologies aimed at the enrichment of circulating tumor cells (CTCs), the subpopulation of primary tumor cells that gain access to the circulatory system and are responsible for colonization at distant sites. In contrast to other technologies, filtration-based CTC enrichment, which exploits differences in size between larger tumor cells and surrounding smaller, non-tumor blood cells, has the potential to improve CTC characterization through isolation of tumor cell populations with greater molecular heterogeneity. However, microscopic analysis of uneven filtration surfaces containing CTCs is laborious, time-consuming, and inconsistent, preventing widespread use of filtration-based enrichment technologies. Here, integrated with a microfiltration-based CTC and rare cell enrichment device we have previously described, we present a protocol for Fourier Ptychographic Microscopy (FPM), a method that, unlike many automated imaging platforms, produces high-speed, high-resolution images that can be digitally refocused, allowing users to observe objects of interest present on multiple focal planes within the same image frame. The development of a cost-effective and high-throughput CTC analysis system for filtration-based enrichment technologies could have profound clinical implications for improved CTC detection and analysis.

Enrichment and Detection of Circulating Tumor Cells and Other Rare Cell Populations by Microfluidic Filtration

The current standard methods for isolating circulating tumor cells (CTCs) from blood involve EPCAM-based immunomagnetic approaches. A major disadvantage of these strategies is that CTCs with low EPCAM expression will be missed. Isolation by size using filter membranes circumvents the reliance on this cell surface marker, and can facilitate the capture not only of EPCAM-negative CTCs but other rare cells as well. These cells that are trapped on the filter membrane can be characterized by immunocytochemistry (ICC) , enumerated and profiled to elucidate their clinical significance. In this chapter, we discuss advances in filtration systems to capture rare cells as well as downstream ICC methods to detect and identify these cells. We highlight our recent clinical study demonstrating the feasibility of using a novel method consisting of automated microfluidic filtration and sequential ICC for detection and enumeration of CTCs, as well as circulating mesenchymal cells (CMCs), circulating endothelial cells (CECs), and putative circulating stem cells (CSCs). We hypothesize that simultaneous analysis of circulating rare cells in blood of cancer patients may lead to a better understanding of disease progression and development of resistance to therapy.

Wide-field Fourier ptychographic microscopy using laser illumination source

Fourier ptychographic (FP) microscopy is a coherent imaging method that can synthesize an image with a higher bandwidth using multiple low-bandwidth images captured at different spatial frequency regions. The method's demand for multiple images drives the need for a brighter illumination scheme and a high-frame-rate camera for a faster acquisition. We report the use of a guided laser beam as an illumination source for an FP microscope. It uses a mirror array and a 2-dimensional scanning Galvo mirror system to provide a sample with plane-wave illuminations at diverse incidence angles. The use of a laser presents speckles in the image capturing process due to reflections between glass surfaces in the system. They appear as slowly varying background fluctuations in the final reconstructed image. We are able to mitigate these artifacts by including a phase image obtained by differential phase contrast (DPC) deconvolution in the FP algorithm. We use a 1-Watt laser configured to provide a collimated beam with 150 mW of power and beam diameter of 1 cm to allow for the total capturing time of 0.96 seconds for 96 raw FPM input images in our system, with the camera sensor's frame rate being the bottleneck for speed. We demonstrate a factor of 4 resolution improvement using a 0.1 NA objective lens over the full camera field-of-view of 2.7 mm by 1.5 mm.

Incubator embedded cell culture imaging system (EmSight) based on Fourier ptychographic microscopy

Multi-day tracking of cells in culture systems can provide valuable information in bioscience experiments. We report the development of a cell culture imaging system, named EmSight, which incorporates multiple compact Fourier ptychographic microscopes with a standard multiwell imaging plate. The system is housed in an incubator and presently incorporates six microscopes. By using the same low magnification objective lenses as the objective and the tube lens, the EmSight is configured as a 1:1 imaging system that, providing large field-of-view (FOV) imaging onto a low-cost CMOS imaging sensor. The EmSight improves the image resolution by capturing a series of images of the sample at varying illumination angles; the instrument reconstructs a higher-resolution image by using the iterative Fourier ptychographic algorithm. In addition to providing high-resolution brightfield and phase imaging, the EmSight is also capable of fluorescence imaging at the native resolution of the objectives. We characterized the system using a phase Siemens star target, and show four-fold improved coherent resolution (synthetic NA of 0.42) and a depth of field of 0.2 mm. To conduct live, long-term dopaminergic neuron imaging, we cultured ventral midbrain from mice driving eGFP from the tyrosine hydroxylase promoter. The EmSight system tracks movements of dopaminergic neurons over a 21 day period.

Wide field-of-view fluorescence image deconvolution with aberration-estimation from Fourier ptychography

This paper presents a method to simultaneously acquire an aberration-corrected, wide field-of-view fluorescence image and a high-resolution coherent bright-field image using a computational microscopy method. First, the procedure applies Fourier ptychographic microscopy (FPM) to retrieve the amplitude and phase of a sample, at a resolution that significantly exceeds the cutoff spatial frequency of the microscope objective lens. At the same time, redundancy within the set of acquired FPM bright-field images offers a means to estimate microscope aberrations. Second, the procedure acquires an aberrated fluorescence image, and computationally improves its resolution through deconvolution with the estimated aberration map. An experimental demonstration successfully improves the bright-field resolution of fixed, stained and fluorescently tagged HeLa cells by a factor of 4.9, and reduces the error caused by aberrations in a fluorescence image by up to 31%, over a field of view of 6.2 mm by 9.3 mm. For optimal deconvolution, we show the fluorescence image needs to have a signal-to-noise ratio of at least ~18.

Experimental robustness of Fourier ptychography phase retrieval algorithms

Fourier ptychography is a new computational microscopy technique that provides gigapixel-scale intensity and phase images with both wide field-of-view and high resolution. By capturing a stack of low-resolution images under different illumination angles, an inverse algorithm can be used to computationally reconstruct the high-resolution complex field. Here, we compare and classify multiple proposed inverse algorithms in terms of experimental robustness. We find that the main sources of error are noise, aberrations and mis-calibration (i.e. model mis-match). Using simulations and experiments, we demonstrate that the choice of cost function plays a critical role, with amplitude-based cost functions performing better than intensity-based ones. The reason for this is that Fourier ptychography datasets consist of images from both brightfield and darkfield illumination, representing a large range of measured intensities. Both noise (e.g. Poisson noise) and model mis-match errors are shown to scale with intensity. Hence, algorithms that use an appropriate cost function will be more tolerant to both noise and model mis-match. Given these insights, we propose a global Newton's method algorithm which is robust and accurate. Finally, we discuss the impact of procedures for algorithmic correction of aberrations and mis-calibration.

A Novel Strategy for Detection and Enumeration of Circulating Rare Cell Populations in Metastatic Cancer Patients Using Automated Microfluidic Filtration and Multiplex Immunoassay

Size selection via filtration offers an antigen-independent approach for the enrichment of rare cell populations in blood of cancer patients. We evaluated the performance of a novel approach for multiplex rare cell detection in blood samples from metastatic breast (n = 19) and lung cancer patients (n = 21), and healthy controls (n = 30) using an automated microfluidic filtration and multiplex immunoassay strategy. Captured cells were enumerated after sequential staining for specific markers to identify circulating tumor cells (CTCs), circulating mesenchymal cells (CMCs), putative circulating stem cells (CSCs), and circulating endothelial cells (CECs). Preclinical validation experiments using cancer cells spiked into healthy blood demonstrated high recovery rate (mean = 85%) and reproducibility of the assay. In clinical studies, CTCs and CMCs were detected in 35% and 58% of cancer patients, respectively, and were largely absent from healthy controls (3%, p = 0.001). Mean levels of CTCs were significantly higher in breast than in lung cancer patients (p = 0.03). Fifty-three percent (53%) of cancer patients harbored putative CSCs, while none were detectable in healthy controls (p<0.0001). In contrast, CECs were observed in both cancer and control groups. Direct comparison of CellSearch^® vs. our microfluidic filter method revealed moderate correlation (R² = 0.46, kappa = 0.47). Serial blood analysis in breast cancer patients demonstrated the feasibility of monitoring circulating rare cell populations over time. Simultaneous assessment of CTCs, CMCs, CSCs and CECs may provide new tools to study mechanisms of disease progression and treatment response/resistance.

Digital micromirror device-based laser-illumination Fourier ptychographic microscopy

We report a novel approach to Fourier ptychographic microscopy (FPM) by using a digital micromirror device (DMD) and a coherent laser source (532 nm) for generating spatially modulated sample illumination. Previously demonstrated FPM systems are all based on partially-coherent illumination, which offers limited throughput due to insufficient brightness. Our FPM employs a high power coherent laser source to enable shot-noise limited high-speed imaging. For the first time, a digital micromirror device (DMD), imaged onto the back focal plane of the illumination objective, is used to generate spatially modulated sample illumination field for ptychography. By coding the on/off states of the micromirrors, the illumination plane wave angle can be varied at speeds more than 4 kHz. A set of intensity images, resulting from different oblique illuminations, are used to numerically reconstruct one high-resolution image without obvious laser speckle. Experiments were conducted using a USAF resolution target and a fiber sample, demonstrating high-resolution imaging capability of our system. We envision that our approach, if combined with a coded-aperture compressive-sensing algorithm, will further improve the imaging speed in DMD-based FPM systems.

Optimization of sampling pattern and the design of Fourier ptychographic illuminator

Fourier ptychography (FP) is a recently developed imaging approach that facilitates high-resolution imaging beyond the cutoff frequency of the employed optics. In the original FP approach, a periodic LED array is used for sample illumination, and therefore, the scanning pattern is a uniform grid in the Fourier space. Such a uniform sampling scheme leads to 3 major problems for FP, namely: 1) it requires a large number of raw images, 2) it introduces the raster grid artefacts in the reconstruction process, and 3) it requires a high-dynamic-range detector. Here, we investigate scanning sequences and sampling patterns to optimize the FP approach. For most biological samples, signal energy is concentrated at low-frequency region, and as such, we can perform non-uniform Fourier sampling in FP by considering the signal structure. In contrast, conventional ptychography perform uniform sampling over the entire real space. To implement the non-uniform Fourier sampling scheme in FP, we have designed and built an illuminator using LEDs mounted on a 3D-printed plastic case. The advantages of this illuminator are threefold in that: 1) it reduces the number of image acquisitions by at least 50% (68 raw images versus 137 in the original FP setup), 2) it departs from the translational symmetry of sampling to solve the raster grid artifact problem, and 3) it reduces the dynamic range of the captured images 6 fold. The results reported in this paper significantly shortened acquisition time and improved quality of FP reconstructions. It may provide new insights for developing Fourier ptychographic imaging platforms and find important applications in digital pathology.

Microscopy illumination engineering using a low-cost liquid crystal display

Illumination engineering is critical for obtaining high-resolution, high-quality images in microscope settings. In a typical microscope, the condenser lens provides sample illumination that is uniform and free from glare. The associated condenser diaphragm can be manually adjusted to obtain the optimal illumination numerical aperture. In this paper, we report a programmable condenser lens for active illumination control. In our prototype setup, we used a $15 liquid crystal display as a transparent spatial light modulator and placed it at the back focal plane of the condenser lens. By setting different binary patterns on the display, we can actively control the illumination and the spatial coherence of the microscope platform. We demonstrated the use of such a simple scheme for multimodal imaging, including bright-field microscopy, darkfield microscopy, phase-contrast microscopy, polarization microscopy, 3D tomographic imaging, and super-resolution Fourier ptychographic imaging. The reported illumination engineering scheme is cost-effective and compatible with most existing platforms. It enables a turnkey solution with high flexibility for researchers in various communities. From the engineering point-of-view, the reported illumination scheme may also provide new insights for the development of multimodal microscopy and Fourier ptychographic imaging.

High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography

Fluorescence microscopy plays a vital role in modern biological research and clinical diagnosis. Here, we report an imaging approach, termed pattern-illuminated Fourier ptychography (FP), for fluorescence imaging beyond the diffraction limit of the employed optics. This approach iteratively recovers a high-resolution fluorescence image from many pattern-illuminated low-resolution intensity measurements. The recovery process starts with one low-resolution measurement as the initial guess. This initial guess is then sequentially updated by other measurements, both in the spatial and Fourier domains. In the spatial domain, we use the pattern-illuminated low-resolution images as intensity constraints for the sample estimate. In the Fourier domain, we use the incoherent optical-transfer-function of the objective lens as the object support constraint for the solution. The sequential updating process is then repeated until the sample estimate converges, typically for 5-20 times. Different from the conventional structured illumination microscopy, any unknown pattern can be used for sample illumination in the reported framework. In particular, we are able to recover both the high-resolution sample image and the unknown illumination pattern at the same time. As a demonstration, we improved the resolution of a conventional fluorescence microscope beyond the diffraction limit of the employed optics. The reported approach may provide an alternative solution for structure illumination microscopy and find applications in wide-field, high-resolution fluorescence imaging.