Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Classifying breast cancer and fibroadenoma tissue biopsies from paraffined stain-free slides by fractal biomarkers in Fourier Ptychographic Microscopy

Breast cancer is one of the most spread and monitored pathologies in high-income countries. After breast biopsy, histological tissue is stored in paraffin, sectioned and mounted. Conventional inspection of tissue slides under benchtop light microscopes involves paraffin removal and staining, typically with H&E. Then, expert pathologists are called to judge the stained slides. However, paraffin removal and staining are operator-dependent, time and resources consuming processes that can generate ambiguities due to non-uniform staining. Here we propose a novel method that can work directly on paraffined stain-free slides. We use Fourier Ptychography as a quantitative phase-contrast microscopy method, which allows accessing a very wide field of view (i.e., mm2) in one single image while guaranteeing high lateral resolution (i.e., 0.5 µm). This imaging method is multi-scale, since it enables looking at the big picture, i.e. the complex tissue structure and connections, with the possibility to zoom-in up to the single-cell level. To handle this informative image content, we introduce elements of fractal geometry as multi-scale analysis method. We show the effectiveness of fractal features in describing and classifying fibroadenoma and breast cancer tissue slides from ten patients with very high accuracy. We reach 94.0 ± 4.2% test accuracy in classifying single images. Above all, we show that combining the decisions of the single images, each patient's slide can be classified with no error. Besides, fractal geometry returns a guide map to help pathologist to judge the different tissue portions based on the likelihood these can be associated to a breast cancer or fibroadenoma biomarker. The proposed automatic method could significantly simplify the steps of tissue analysis and make it independent from the sample preparation, the skills of the lab operator and the pathologist.

Efficient, gigapixel-scale, aberration-free whole slide scanner using angular ptychographic imaging with closed-form solution

Whole slide imaging provides a wide field-of-view (FOV) across cross-sections of biopsy or surgery samples, significantly facilitating pathological analysis and clinical diagnosis. Such high-quality images that enable detailed visualization of cellular and tissue structures are essential for effective patient care and treatment planning. To obtain such high-quality images for pathology applications, there is a need for scanners with high spatial bandwidth products, free from aberrations, and without the requirement for z-scanning. Here we report a whole slide imaging system based on angular ptychographic imaging with a closed-form solution (WSI-APIC), which offers efficient, tens-of-gigapixels, large-FOV, aberration-free imaging. WSI-APIC utilizes oblique incoherent illumination for initial high-level segmentation, thereby bypassing unnecessary scanning of the background regions and enhancing image acquisition efficiency. A GPU-accelerated APIC algorithm analytically reconstructs phase images with effective digital aberration corrections and improved optical resolutions. Moreover, an auto-stitching technique based on scale-invariant feature transform ensures the seamless concatenation of whole slide phase images. In our experiment, WSI-APIC achieved an optical resolution of 772 nm using a 10×/0.25 NA objective lens and captures 80-gigapixel aberration-free phase images for a standard 76.2 mm × 25.4 mm microscopic slide.

Enhanced tissue slide imaging in the complex domain via cross-explainable GAN for Fourier ptychographic microscopy

Achieving microscopy with large space-bandwidth products plays a key role in diagnostic imaging and is widely significant in the overall field of clinical practice. Among quantitative microscopy techniques, Fourier Ptychography (FP) provides a wide field of view and high-resolution images, suitable to the histopathological field, but onerous in computational terms. Artificial intelligence can be a solution in this sense. In particular, this research delves into the application of Generative Adversarial Networks (GAN) for the dual-channel complex FP image enhancement of human kidney samples. The study underscores the GANs' efficacy in promoting biological architectures in FP domain, thereby still guaranteeing high resolution and visibility of detailed microscopic structures. We demonstrate successful GAN-based enhanced reconstruction through two strategies: cross-explainability and expert survey. The cross-explainability is evaluated through the comparison of explanation maps for both real and imaginary components underlining its robustness. This comparison further shows that their interplay is pivotal for accurate reconstruction without hallucinations. Secondly, the enhanced reconstruction accuracy and effectiveness in a clinical workflow are confirmed through a two-step survey conducted with nephrologists.

Deep-ultraviolet Fourier ptychography (DUV-FP) for label-free biochemical imaging via feature-domain optimization

We present deep-ultraviolet Fourier ptychography (DUV-FP) for high-resolution chemical imaging of biological specimens in their native state without exogenous stains. This approach uses a customized 265-nm DUV LED array for angle-varied illumination, leveraging the unique DUV absorption properties of biomolecules at this wavelength region. We implemented a robust feature-domain optimization framework to overcome common challenges in Fourier ptychographic reconstruction, including vignetting, pupil aberrations, stray light problems, intensity variations, and other systematic errors. By using a 0.12 numerical aperture low-resolution objective lens, our DUV-FP prototype can resolve the 345-nm linewidth on a resolution target, demonstrating at least a four-fold resolution gain compared to the captured raw images. Testing on various biospecimens demonstrates that DUV-FP significantly enhances absorption-based chemical contrast and reveals detailed structural and molecular information. To further address the limitations of conventional FP in quantitative phase imaging, we developed a spatially coded DUV-FP system. This platform enables true quantitative phase imaging of biospecimens with DUV light, overcoming the non-uniform phase response inherent in traditional microscopy techniques. The demonstrated advancements in high-resolution, label-free chemical imaging may accelerate developments in digital pathology, potentially enabling rapid, on-site analysis of biopsy samples in clinical settings.

Computational optical sectioning via near-field multi-slice ptychography

We introduce a method for the computational sectioning of optically thick samples based on a combination of near-field and multi-slice ptychography. The method enables a large field-of-view 3D phase imaging of samples that is an order of magnitude thicker than the depth of field of bright-field microscopy. An axial resolution for these thick samples is maintained in the presence of multiple scattering, revealing a complex structure beyond the depth of the field limit. In this Letter, we describe the new, to the best of our knowledge, approach and demonstrate its effectiveness using a range of samples with diverse thicknesses and optical properties.

Ptycho-endoscopy on a lensless ultrathin fiber bundle tip

Synthetic aperture radar (SAR) utilizes an aircraft-carried antenna to emit electromagnetic pulses and detect the returning echoes. As the aircraft travels across a designated area, it synthesizes a large virtual aperture to improve image resolution. Inspired by SAR, we introduce synthetic aperture ptycho-endoscopy (SAPE) for micro-endoscopic imaging beyond the diffraction limit. SAPE operates by hand-holding a lensless fiber bundle tip to record coherent diffraction patterns from specimens. The fiber cores at the distal tip modulate the diffracted wavefield within a confined area, emulating the role of the 'airborne antenna' in SAR. The handheld operation introduces positional shifts to the tip, analogous to the aircraft's movement. These shifts facilitate the acquisition of a ptychogram and synthesize a large virtual aperture extending beyond the bundle's physical limit. We mitigate the influences of hand motion and fiber bending through a low-rank spatiotemporal decomposition of the bundle's modulation profile. Our tests demonstrate the ability to resolve a 548-nm linewidth on a resolution target. The achieved space-bandwidth product is ~1.1 million effective pixels, representing a 36-fold increase compared to that of the original fiber bundle. Furthermore, SAPE's refocusing capability enables imaging over an extended depth of field exceeding 2 cm. The aperture synthesizing process in SAPE surpasses the diffraction limit set by the probe's maximum collection angle, opening new opportunities for both fiber-based and distal-chip endoscopy in applications such as medical diagnostics and industrial inspection.

Label-free, high-throughput holographic imaging to evaluate mammalian gametes and embryos†

Assisted reproduction is one of the significant tools to treat human infertility. Morphological assessment is the primary method to determine sperm and embryo viability during in vitro fertilization cycles. It has the advantage of being a quick, convenient, and inexpensive means of assessment. However, visual observation is of limited predictive value for early embryo morphology. It has led many to search for other imaging tools to assess the reproductive potential of a given embryo. The limitations of visual assessment apply to both humans and animals. One recent innovation in assisted reproduction technology imaging is interferometric phase microscopy, also known as holographic microscopy. Interferometric phase microscopy/quantitative phase imaging is the next likely progression of analytical microscopes for the assisted reproduction laboratory. The interferometric phase microscopy system analyzes waves produced by the light as it passes through the specimen observed. The microscope collects the light waves produced and uses the algorithm to create a hologram of the specimen. Recently, interferometric phase microscopy has been combined with quantitative phase imaging, which joins phase contrast microscopy with holographic microscopy. These microscopes collect light waves produced and use the algorithm to create a hologram of the specimen. Unlike other systems, interferometric phase microscopy can provide a quantitative digital image, and it can make 2D and 3D images of the samples. This review summarizes some newer and more promising quantitative phase imaging microscopy systems for evaluating gametes and embryos. Studies clearly show that quantitative phase imaging is superior to bright field microscopy-based evaluation methods when evaluating sperm and oocytes prior to IVF and embryos prior to transfer. However, further assessment of these systems for efficacy, reproducibility, cost-effectiveness, and embryo/gamete safety must take place before they are widely adopted.

Analytical solution for single-pixel ptychography through linear modeling

Amplitude-modulated single-pixel ptychography (SPP) enables non-interferometric complex-field imaging of objects. However, the conventional iterative and nondeterministic reconstruction methods, based on the ptychography algorithm, pose challenges in fully understanding the role of critical optical parameters. In response, this paper introduces an innovative analytical approach that establishes a theoretical foundation for the uniqueness of SPP reconstruction results. The proposed method conceptualizes SPP as a system of linear equations in the frequency domain, involving both object and modulated illumination. Solving this equation system reveals a determined solution for the complex object, providing an alternative to iterative and nondeterministic techniques. Through a series of simulations, this approach not only validates the uniqueness of SPP reconstruction, but also explores key properties influencing accuracy.

Spatially-coded Fourier ptychography: flexible and detachable coded thin films for quantitative phase imaging with uniform phase transfer characteristics

Fourier ptychography (FP) is an enabling imaging technique that produces high-resolution complex-valued images with extended field coverages. However, when FP images a phase object with any specific spatial frequency, the captured images contain only constant values, rendering the recovery of the corresponding linear phase ramp impossible. This challenge is not unique to FP but also affects other common microscopy techniques -- a rather counterintuitive outcome given their widespread use in phase imaging. The underlying issue originates from the non-uniform phase transfer characteristic inherent in microscope systems, which impedes the conversion of object wavefields into discernible intensity variations. To address this challenge, we present spatially-coded Fourier ptychography (scFP), a new method that synergizes FP with spatial-domain coded detection for true quantitative phase imaging. In scFP, a flexible and detachable coded thin film is attached atop the image sensor in a regular FP setup. The spatial modulation of this thin film ensures a uniform phase response across the entire synthetic bandwidth. It improves reconstruction quality and corrects refractive index underestimation issues prevalent in conventional FP and related tomographic implementations. The inclusion of the coded thin film further adds a new dimension of measurement diversity in the spatial domain. The development of scFP is expected to catalyse new research directions and applications for phase imaging, emphasizing the need for true quantitative accuracy with uniform frequency response.

Fly-scan high-throughput coded ptychographic microscopy via active micro-vibration and rolling-shutter distortion correction

Recent advancements in ptychography have demonstrated the potential of coded ptychography (CP) for high-resolution optical imaging in a lensless configuration. However, CP suffers imaging throughput limitations due to scanning inefficiencies. To address this, we propose what we believe is a novel 'fly-scan' scanning strategy utilizing two eccentric rotating mass (ERM) vibration motors for high-throughput coded ptychographic microscopy. The intrinsic continuity of the 'fly-scan' technique effectively eliminates the scanning overhead typically encountered during data acquisition. Additionally, its randomized scanning trajectory considerably reduces periodic artifacts in image reconstruction. We also developed what we believe to be a novel rolling-shutter distortion correction algorithm to fix the rolling-shutter effects. We built up a low-cost, DIY-made prototype platform and validated our approach with various samples including a resolution target, a quantitative phase target, a thick potato sample and biospecimens. The reported platform may offer a cost-effective and turnkey solution for high-throughput bio-imaging.

Mixed-state ptychography for quantitative optical properties measurement of vector beam

Recent advances in ptychography have extended to anisotropic specimens, but vectorial reconstruction of probes owing to polarization aliasing remains a challenge. A polarization-sensitive ptychography that enables full optical property measurement of vector light is proposed. An optimized reconstruction strategy, first calibrating the propagation direction and then performing faithful retrieval, is established. This method avoids multiple image acquisitions with various polarizer configurations and significantly improves the measurement accuracy by correlating the intensity and position of different polarization components. The capability of the proposed method to quantify anisotropic parameters of optical materials and polarization properties of vector probe is demonstrated by experiment.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

Noise-robust latent vector reconstruction in ptychography using deep generative models

Computational imaging is increasingly vital for a broad spectrum of applications, ranging from biological to material sciences. This includes applications where the object is known and sufficiently sparse, allowing it to be described with a reduced number of parameters. When no explicit parameterization is available, a deep generative model can be trained to represent an object in a low-dimensional latent space. In this paper, we harness this dimensionality reduction capability of autoencoders to search for the object solution within the latent space rather than the object space. We demonstrate what we believe to be a novel approach to ptychographic image reconstruction by integrating a deep generative model obtained from a pre-trained autoencoder within an automatic differentiation ptychography (ADP) framework. This approach enables the retrieval of objects from highly ill-posed diffraction patterns, offering an effective method for noise-robust latent vector reconstruction in ptychography. Moreover, the mapping into a low-dimensional latent space allows us to visualize the optimization landscape, which provides insight into the convexity and convergence behavior of the inverse problem. With this work, we aim to facilitate new applications for sparse computational imaging such as when low radiation doses or rapid reconstructions are essential.

Improving the resolution of Fourier ptychographic imaging using an a priori neural network

In this paper, we propose a dual-structured prior neural network model that independently restores both the amplitude and phase image using a random latent code for Fourier ptychography (FP). We demonstrate that the inherent prior information within the neural network can generate super-resolution images with a resolution that exceeds the combined numerical aperture of the FP system. This method circumvents the need for a large labeled dataset. The training process is guided by an appropriate forward physical model. We validate the effectiveness of our approach through simulations and experimental data. The results suggest that integrating image prior information with system-collected data is a potentially effective approach for improving the resolution of FP systems.

Ptychographic lens-less birefringence microscopy using a mask-modulated polarization image sensor

Birefringence, an inherent characteristic of optically anisotropic materials, is widely utilized in various imaging applications ranging from material characterizations to clinical diagnosis. Polarized light microscopy enables high-resolution, high-contrast imaging of optically anisotropic specimens, but it is associated with mechanical rotations of polarizer/analyzer and relatively complex optical designs. Here, we present a form of lens-less polarization-sensitive microscopy capable of complex and birefringence imaging of transparent objects without an optical lens and any moving parts. Our method exploits an optical mask-modulated polarization image sensor and single-input-state LED illumination design to obtain complex and birefringence images of the object via ptychographic phase retrieval. Using a camera with a pixel size of 3.45 μm, the method achieves birefringence imaging with a half-pitch resolution of 2.46 μm over a 59.74 mm2 field-of-view, which corresponds to a space-bandwidth product of 9.9 megapixels. We demonstrate the high-resolution, large-area, phase and birefringence imaging capability of our method by presenting the phase and birefringence images of various anisotropic objects, including a monosodium urate crystal, and excised mouse eye and heart tissues.

Structured modulation multi-height microscopy for high-resolution imaging

Conventional multi-height microscopy techniques introduce different object-to-detector distances to obtain multiple measurements for phase retrieval. However, surpassing the diffraction limit imposed by the numerical aperture (NA) of the objective lens remains a challenging task. Here, we report a novel structured modulation multi-height microscopy technique for quantitative high-resolution imaging. In our platform, a thin diffuser is placed in between the sample and the objective lens. By translating the diffuser to different axial positions, a sequence of modulated intensity images is captured for reconstruction. The otherwise inaccessible high-resolution object information can thus be encoded into the optical system for detection. In the construction process, we report a ptychographic phase retrieval algorithm to recover the existing wavefront of the complex object. We validate our approach using a resolution target, a phase target, and various biological samples. We demonstrate a ∼4-fold resolution gain over the diffraction limit. We also demonstrate our approach to achieve a 6.5 mm by 4.3 mm field of view and a half-pitch resolution of 1.2 µm. The reported methodology provides a portable, turnkey solution for quantitative high-resolution imaging with potential applications in disease diagnosis, sample screening, and other fields.

Extending the capabilities of vectorial ptychography to circular-polarizing materials such as cholesteric liquid crystals

The problem of imaging materials with circular-polarization properties is discussed within the framework of vectorial ptychography. We demonstrate, both theoretically and numerically, that using linear polarizations to investigate such materials compromises the unicity of the solution provided by this computational method. To overcome this limitation, an improved measurement approach is proposed, which involves specific combinations of elliptical polarizations. The effectiveness of this strategy is demonstrated by numerical simulations and experimental measurements on cholesteric liquid crystal films, which possess unique polarization properties. With the help of Pauli matrices algebra, our results highlight the technique's ability to discern between the different types of circular polarizers, uniform vs. non-uniform, and determine their handedness.

Broadband ptychography using curved wavefront illumination

We examine the interplay between spectral bandwidth and illumination curvature in ptychography. By tailoring the divergence of the illumination, broader spectral bandwidths can be tolerated without requiring algorithmic modifications to the forward model. In particular, a strong wavefront curvature transitions a far-field diffraction geometry to an effectively near-field one, which is less affected by temporal coherence effects. The relaxed temporal coherence requirements allow for leveraging wider spectral bandwidths and larger illumination spots. Our findings open up new avenues towards utilizing pink and broadband beams for increased flux and throughput at both synchrotron facilities and lab-scale beamlines.

Spatial- and Fourier-domain ptychography for high-throughput bio-imaging

First envisioned for determining crystalline structures, ptychography has become a useful imaging tool for microscopists. However, ptychography remains underused by biomedical researchers due to its limited resolution and throughput in the visible light regime. Recent developments of spatial- and Fourier-domain ptychography have successfully addressed these issues and now offer the potential for high-resolution, high-throughput optical imaging with minimal hardware modifications to existing microscopy setups, often providing an excellent trade-off between resolution and field of view inherent to conventional imaging systems, giving biomedical researchers the best of both worlds. Here, we provide extensive information to enable the implementation of ptychography by biomedical researchers in the visible light regime. We first discuss the intrinsic connections between spatial-domain coded ptychography and Fourier ptychography. A step-by-step guide then provides the user instructions for developing both systems with practical examples. In the spatial-domain implementation, we explain how a large-scale, high-performance blood-cell lens can be made at negligible expense. In the Fourier-domain implementation, we explain how adding a low-cost light source to a regular microscope can improve the resolution beyond the limit of the objective lens. The turnkey operation of these setups is suitable for use by professional research laboratories, as well as citizen scientists. Users with basic experience in optics and programming can build the setups within a week. The do-it-yourself nature of the setups also allows these procedures to be implemented in laboratory courses related to Fourier optics, biomedical instrumentation, digital image processing, robotics and capstone projects.

Near-field multi-slice ptychography: quantitative phase imaging of optically thick samples with visible light and X-rays

Ptychography is a form of lens-free coherent diffractive imaging now used extensively in electron and synchrotron-based X-ray microscopy. In its near-field implementation, it offers a route to quantitative phase imaging at an accuracy and resolution competitive with holography, with the added advantages of extended field of view and blind deconvolution of the illumination beam profile from the sample image. In this paper we show how near-field ptychography can be combined with a multi-slice model, adding to this list of advantages the unique ability to recover high-resolution phase images of larger samples, whose thickness places them beyond the depth of field of alternative methods.

PtyLab.m/py/jl: a cross-platform, open-source inverse modeling toolbox for conventional and Fourier ptychography

Conventional (CP) and Fourier (FP) ptychography have emerged as versatile quantitative phase imaging techniques. While the main application cases for each technique are different, namely lens-less short wavelength imaging for CP and lens-based visible light imaging for FP, both methods share a common algorithmic ground. CP and FP have in part independently evolved to include experimentally robust forward models and inversion techniques. This separation has resulted in a plethora of algorithmic extensions, some of which have not crossed the boundary from one modality to the other. Here, we present an open source, cross-platform software, called PtyLab, enabling both CP and FP data analysis in a unified framework. With this framework, we aim to facilitate and accelerate cross-pollination between the two techniques. Moreover, the availability in Matlab, Python, and Julia will set a low barrier to enter each field.