Super-resolved multispectral lensless microscopy via angle-tilted, wavelength-multiplexed ptychographic modulation

Super-resolution lensless on-chip microscopy based on array illumination and sub-pixel shift search

The resolution of a lensless on-chip microscopy system is constrained by the pixel size of image sensors. This Letter introduces a super-resolution on-chip microscopy system based on a compact array light source illumination and sub-pixel shift search. The system utilizes a closely spaced array light source composed by four RGB LED modules, sequentially illuminating the sample. A sub-pixel shift search algorithm is proposed, which determines the sub-pixel shift by comparing the frequency of captured low-resolution holograms. Leveraging this sub-pixel shift, a super-resolution reconstruction algorithm is introduced, building upon a multi-wavelength phase retrieval method, enabling the rapid super-resolution reconstruction of holograms with the region-of-interest. The system and algorithms presented herein obviate the need for a displacement control platform and calibration of the illumination angles of the light source, facilitating a super-resolution phase reconstruction under partially coherent illumination.

Fly-scan high-throughput coded ptychographic microscopy via active micro-vibration and rolling-shutter distortion correction

Recent advancements in ptychography have demonstrated the potential of coded ptychography (CP) for high-resolution optical imaging in a lensless configuration. However, CP suffers imaging throughput limitations due to scanning inefficiencies. To address this, we propose what we believe is a novel 'fly-scan' scanning strategy utilizing two eccentric rotating mass (ERM) vibration motors for high-throughput coded ptychographic microscopy. The intrinsic continuity of the 'fly-scan' technique effectively eliminates the scanning overhead typically encountered during data acquisition. Additionally, its randomized scanning trajectory considerably reduces periodic artifacts in image reconstruction. We also developed what we believe to be a novel rolling-shutter distortion correction algorithm to fix the rolling-shutter effects. We built up a low-cost, DIY-made prototype platform and validated our approach with various samples including a resolution target, a quantitative phase target, a thick potato sample and biospecimens. The reported platform may offer a cost-effective and turnkey solution for high-throughput bio-imaging.

Broadband ptychography using curved wavefront illumination

We examine the interplay between spectral bandwidth and illumination curvature in ptychography. By tailoring the divergence of the illumination, broader spectral bandwidths can be tolerated without requiring algorithmic modifications to the forward model. In particular, a strong wavefront curvature transitions a far-field diffraction geometry to an effectively near-field one, which is less affected by temporal coherence effects. The relaxed temporal coherence requirements allow for leveraging wider spectral bandwidths and larger illumination spots. Our findings open up new avenues towards utilizing pink and broadband beams for increased flux and throughput at both synchrotron facilities and lab-scale beamlines.

Spatial- and Fourier-domain ptychography for high-throughput bio-imaging

First envisioned for determining crystalline structures, ptychography has become a useful imaging tool for microscopists. However, ptychography remains underused by biomedical researchers due to its limited resolution and throughput in the visible light regime. Recent developments of spatial- and Fourier-domain ptychography have successfully addressed these issues and now offer the potential for high-resolution, high-throughput optical imaging with minimal hardware modifications to existing microscopy setups, often providing an excellent trade-off between resolution and field of view inherent to conventional imaging systems, giving biomedical researchers the best of both worlds. Here, we provide extensive information to enable the implementation of ptychography by biomedical researchers in the visible light regime. We first discuss the intrinsic connections between spatial-domain coded ptychography and Fourier ptychography. A step-by-step guide then provides the user instructions for developing both systems with practical examples. In the spatial-domain implementation, we explain how a large-scale, high-performance blood-cell lens can be made at negligible expense. In the Fourier-domain implementation, we explain how adding a low-cost light source to a regular microscope can improve the resolution beyond the limit of the objective lens. The turnkey operation of these setups is suitable for use by professional research laboratories, as well as citizen scientists. Users with basic experience in optics and programming can build the setups within a week. The do-it-yourself nature of the setups also allows these procedures to be implemented in laboratory courses related to Fourier optics, biomedical instrumentation, digital image processing, robotics and capstone projects.

PtyLab.m/py/jl: a cross-platform, open-source inverse modeling toolbox for conventional and Fourier ptychography

Conventional (CP) and Fourier (FP) ptychography have emerged as versatile quantitative phase imaging techniques. While the main application cases for each technique are different, namely lens-less short wavelength imaging for CP and lens-based visible light imaging for FP, both methods share a common algorithmic ground. CP and FP have in part independently evolved to include experimentally robust forward models and inversion techniques. This separation has resulted in a plethora of algorithmic extensions, some of which have not crossed the boundary from one modality to the other. Here, we present an open source, cross-platform software, called PtyLab, enabling both CP and FP data analysis in a unified framework. With this framework, we aim to facilitate and accelerate cross-pollination between the two techniques. Moreover, the availability in Matlab, Python, and Julia will set a low barrier to enter each field.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

High-throughput digital pathology via a handheld, multiplexed, and AI-powered ptychographic whole slide scanner

The recent advent of whole slide imaging (WSI) systems has moved digital pathology closer to diagnostic applications and clinical practices. Integrating WSI with machine learning promises the growth of this field in upcoming years. Here we report the design and implementation of a handheld, colour-multiplexed, and AI-powered ptychographic whole slide scanner for digital pathology applications. This handheld scanner is built using low-cost and off-the-shelf components, including red, green, and blue laser diodes for sample illumination, a modified stage for programmable sample positioning, and a synchronized image sensor pair for data acquisition. We smear a monolayer of goat blood cells on the main sensor for high-resolution lensless coded ptychographic imaging. The synchronized secondary sensor acts as a non-contact encoder for precisely tracking the absolute object position for ptychographic reconstruction. For WSI, we introduce a new phase-contrast-based focus metric for post-acquisition autofocusing of both stained and unstained specimens. We show that the scanner can resolve the 388-nm linewidth on the resolution target and acquire gigapixel images with a 14 mm × 11 mm area in ∼70 seconds. The imaging performance is validated with regular stained pathology slides, unstained thyroid smears, and malaria-infected blood smears. The deep neural network developed in this study further enables high-throughput cytometric analysis using the recovered complex amplitude. The reported do-it-yourself scanner offers a portable solution to transform the high-end WSI system into one that can be made widely available at a low cost. The capability of high-throughput quantitative phase imaging may also find applications in rapid on-site evaluations.

Pixel Super-Resolution Phase Retrieval for Lensless On-Chip Microscopy via Accelerated Wirtinger Flow

Empowered by pixel super-resolution (PSR) and phase retrieval techniques, lensless on-chip microscopy opens up new possibilities for high-throughput biomedical imaging. However, the current PSR phase retrieval approaches are time consuming in terms of both the measurement and reconstruction procedures. In this work, we present a novel computational framework for PSR phase retrieval to address these concerns. Specifically, a sparsity-promoting regularizer is introduced to enhance the well posedness of the nonconvex problem under limited measurements, and Nesterov’s momentum is used to accelerate the iterations. The resulting algorithm, termed accelerated Wirtinger flow (AWF), achieves at least an order of magnitude faster rate of convergence and allows a twofold reduction in the measurement number while maintaining competitive reconstruction quality. Furthermore, we provide general guidance for step size selection based on theoretical analyses, facilitating simple implementation without the need for complicated parameter tuning. The proposed AWF algorithm is compatible with most of the existing lensless on-chip microscopes and could help achieve label-free rapid whole slide imaging of dynamic biological activities at subpixel resolution.

Lensfree on-chip microscopy based on single-plane phase retrieval

We propose a novel single-plane phase retrieval method to realize high-quality sample reconstruction for lensfree on-chip microscopy. In our method, complex wavefield reconstruction is modeled as a quadratic minimization problem, where total variation and joint denoising regularization are designed to keep a balance of artifact removal and resolution enhancement. In experiment, we built a 3D-printed field-portable platform to validate the imaging performance of our method, where resolution chart, dynamic target, transparent cell, polystyrene beads, and stained tissue sections are employed for the imaging test. Compared to state-of-the-art methods, our method eliminates image degradation and obtains a higher imaging resolution. Different from multi-wavelength or multi-height phase retrieval methods, our method only utilizes a single-frame intensity data record to accomplish high-fidelity reconstruction of different samples, which contributes a simple, robust, and data-efficient solution to design a resource-limited lensfree on-chip microscope. We believe that it will become a useful tool for telemedicine and point-of-care application.

Optofluidic ptychography on a chip

We report the implementation of a fully on-chip, lensless microscopy technique termed optofluidic ptychography. This imaging modality complements the miniaturization provided by microfluidics and allows the integration of ptychographic microscopy into various lab-on-a-chip devices. In our prototype, we place a microfluidic channel on the top surface of a coverslip and coat the bottom surface with a scattering layer. The channel and the coated coverslip substrate are then placed on top of an image sensor for diffraction data acquisition. Similar to the operation of a flow cytometer, the device utilizes microfluidic flow to deliver specimens across the channel. The diffracted light from the flowing objects is modulated by the scattering layer and recorded by the image sensor for ptychographic reconstruction, where high-resolution quantitative complex images are recovered from the diffraction measurements. By using an image sensor with a 1.85 μm pixel size, our device can resolve the 550 nm linewidth on the resolution target. We validate the device by imaging different types of biospecimens, including C. elegans, yeast cells, paramecium, and closterium sp. We also demonstrate a high-resolution ptychographic reconstruction at a video framerate of 30 frames per second. The reported technique can address a wide range of biomedical needs and engenders new ptychographic imaging innovations in a flow cytometer configuration.

Deep learning-enabled whole slide imaging (DeepWSI): oil-immersion quality using dry objectives, longer depth of field, higher system throughput, and better functionality

Whole slide imaging (WSI) has moved the traditional manual slide inspection process to the era of digital pathology. A typical WSI system translates the sample to different positions and captures images using a high numerical aperture (NA) objective lens. Performing oil-immersion microscopy is a major obstacle for WSI as it requires careful liquid handling during the scanning process. Switching between dry objective and oil-immersion lens is often impossible as it disrupts the acquisition process. For a high-NA objective lens, the sub-micron depth of field also poses a challenge to acquiring in-focus images of samples with uneven topography. Additionally, it implies a small field of view for each tile, thus limiting the system throughput and resulting in a long acquisition time. Here we report a deep learning-enabled WSI platform, termed DeepWSI, to substantially improve the system performance and imaging throughput. With this platform, we show that images captured with a regular dry objective lens can be transformed into images comparable to that of a 1.4-NA oil immersion lens. Blurred images with defocus distance from -5 µm to +5 µm can be virtually refocused to the in-focus plane post measurement. We demonstrate an equivalent data throughput of >2 gigapixels per second, the highest among existing WSI systems. Using the same deep neural network, we also report a high-resolution virtual staining strategy and demonstrate it for Fourier ptychographic WSI. The DeepWSI platform may provide a turnkey solution for developing high-performance diagnostic tools for digital pathology.

Ptychography-based high-throughput lensless on-chip microscopy via incremental proximal algorithms

Ptychography-based lensless on-chip microscopy enables high-throughput imaging by retrieving the missing phase information from intensity measurements. Numerous reconstruction algorithms for ptychography have been proposed, yet only a few incremental algorithms can be extended to lensless on-chip microscopy because of large-scale datasets but limited computational efficiency. In this paper, we propose the use of accelerated proximal gradient methods for blind ptychographic phase retrieval in lensless on-chip microscopy. Incremental gradient approaches are adopted in the reconstruction routine. Our algorithms divide the phase retrieval problem into sub-problems involving the evaluation of proximal operator, stochastic gradient descent, and Wirtinger derivatives. We benchmark the performances of accelerated proximal gradient, extended ptychographic iterative engine, and alternating direction method of multipliers, and discuss their convergence and accuracy in both noisy and noiseless cases. We also validate our algorithms using experimental datasets, where full field of view measurements are captured to recover the high-resolution complex samples. Among these algorithms, accelerated proximal gradient presents the overall best performance regarding accuracy and convergence rate. The proposed methods may find applications in ptychographic reconstruction, especially for cases where a wide field of view and high resolution are desired at the same time.

Quantitative multi-height phase retrieval via a coded image sensor

Multi-height phase retrieval introduces different object-to-detector distances for obtaining phase diversity measurements. In the acquisition process, the slow-varying phase information, however, cannot be converted to intensity variations for detection. Therefore, the low-frequency contents of the phase profile are lost during acquisition and cannot be properly restored via phase retrieval. Here, we demonstrate the use of a coded image sensor for addressing this challenge in multi-height phase retrieval. In our scheme, we add a coded layer on top of the image sensor for encoding the slow-varying complex wavefronts into intensity variations of the modulated patterns. Inspired by the concept of blind ptychography, we report a reconstruction scheme to jointly recover the complex object and the unknown coded layer using multi-height measurements. With both simulation and experimental results, we show that the recovered phase is quantitative and the slow-varying phase profiles can be properly restored using lensless multi-height measurements. We also show that the image quality using the coded sensor is better than that of a regular image sensor. For demonstrations, we validate the reported scheme with various biospecimens and compare the results to those of regular lensless multi-height phase retrieval. The use of a coded image sensor may enable true quantitative phase imaging for the lensless multi-height, multi-wavelength, and transport-of-intensity equation approaches.

High-throughput lensless whole slide imaging via continuous height-varying modulation of a tilted sensor

We report a new, to the best of our knowledge, lensless microscopy configuration by integrating the concepts of transverse translational ptychography and defocus multi-height phase retrieval. In this approach, we place a tilted image sensor under the specimen for introducing linearly increasing phase modulation along one lateral direction. Similar to the operation of ptychography, we laterally translate the specimen and acquire the diffraction images for reconstruction. Since the axial distance between the specimen and the sensor varies at different lateral positions, laterally translating the specimen effectively introduces defocus multi-height measurements while eliminating axial scanning. Lateral translation further introduces sub-pixel shift for pixel super-resolution imaging and naturally expands the field of view for rapid whole slide imaging. We show that the equivalent height variation can be precisely estimated from the lateral shift of the specimen, thereby addressing the challenge of precise axial positioning in conventional multi-height phase retrieval. Using a sensor with 1.67 µm pixel size, our low-cost and field-portable prototype can resolve the 690 nm linewidth on the resolution target. We show that a whole slide image of a blood smear with a 120mm² field of view can be acquired in 18 s. We also demonstrate accurate automatic white blood cell counting from the recovered image. The reported approach may provide a turnkey solution for addressing point-of-care and telemedicine-related challenges.

Adaptive super-resolution enabled on-chip contact microscopy

We demonstrate an adaptive super-resolution based contact imaging on a CMOS chip to achieve subcellular spatial resolution over a large field of view of ∼24 mm². By using regular LED illumination, we acquire the single lower-resolution image of the objects placed approximate to the sensor with unit magnification. For the raw contact-mode lens-free image, the pixel size of the sensor chip limits the spatial resolution. We develop a hybrid supervised-unsupervised strategy to train a super-resolution network, circumventing the missing of in-situ ground truth, effectively recovering a much higher resolution image of the objects, permitting sub-micron spatial resolution to be achieved across the entire sensor chip active area. We demonstrate the success of this approach by imaging the proliferation dynamics of cells directly cultured on the chip.

Virtual brightfield and fluorescence staining for Fourier ptychography via unsupervised deep learning

Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. Acquiring color images of histology slides often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learning approach for performing unsupervised image-to-image translation of FPM reconstructions. A cycle-consistent adversarial network with multiscale structure similarity loss is trained to perform virtual brightfield and fluorescence staining of the recovered FPM images. In the training stage, we feed the network with two sets of unpaired images: (1) monochromatic FPM recovery and (2) color or fluorescence images captured using a regular microscope. In the inference stage, the network takes the FPM input and outputs a virtually stained image with reduced coherent artifacts and improved image quality. We test the approach on various samples with different staining protocols. High-quality color and fluorescence reconstructions validate its effectiveness.