Wide-field, high-resolution lensless on-chip microscopy via near-field blind ptychographic modulation

Fly-scan high-throughput coded ptychographic microscopy via active micro-vibration and rolling-shutter distortion correction

Recent advancements in ptychography have demonstrated the potential of coded ptychography (CP) for high-resolution optical imaging in a lensless configuration. However, CP suffers imaging throughput limitations due to scanning inefficiencies. To address this, we propose what we believe is a novel 'fly-scan' scanning strategy utilizing two eccentric rotating mass (ERM) vibration motors for high-throughput coded ptychographic microscopy. The intrinsic continuity of the 'fly-scan' technique effectively eliminates the scanning overhead typically encountered during data acquisition. Additionally, its randomized scanning trajectory considerably reduces periodic artifacts in image reconstruction. We also developed what we believe to be a novel rolling-shutter distortion correction algorithm to fix the rolling-shutter effects. We built up a low-cost, DIY-made prototype platform and validated our approach with various samples including a resolution target, a quantitative phase target, a thick potato sample and biospecimens. The reported platform may offer a cost-effective and turnkey solution for high-throughput bio-imaging.

Dual-constrained physics-enhanced untrained neural network for lensless imaging

An untrained neural network (UNN) paves a new way to realize lensless imaging from single-frame intensity data. Based on the physics engine, such methods utilize the smoothness property of a convolutional kernel and provide an iterative self-supervised learning framework to release the needs of an end-to-end training scheme with a large dataset. However, the intrinsic overfitting problem of UNN is a challenging issue for stable and robust reconstruction. To address it, we model the phase retrieval problem into a dual-constrained untrained network, in which a phase-amplitude alternating optimization framework is designed to split the intensity-to-phase problem into two tasks: phase and amplitude optimization. In the process of phase optimization, we combine a deep image prior with a total variation prior to retrain the loss function for the phase update. In the process of amplitude optimization, a total variation denoising-based Wirtinger gradient descent method is constructed to form an amplitude constraint. Alternative iterations of the two tasks result in high-performance wavefield reconstruction. Experimental results demonstrate the superiority of our method.

Ptychographic lens-less birefringence microscopy using a mask-modulated polarization image sensor

Birefringence, an inherent characteristic of optically anisotropic materials, is widely utilized in various imaging applications ranging from material characterizations to clinical diagnosis. Polarized light microscopy enables high-resolution, high-contrast imaging of optically anisotropic specimens, but it is associated with mechanical rotations of polarizer/analyzer and relatively complex optical designs. Here, we present a form of lens-less polarization-sensitive microscopy capable of complex and birefringence imaging of transparent objects without an optical lens and any moving parts. Our method exploits an optical mask-modulated polarization image sensor and single-input-state LED illumination design to obtain complex and birefringence images of the object via ptychographic phase retrieval. Using a camera with a pixel size of 3.45 μm, the method achieves birefringence imaging with a half-pitch resolution of 2.46 μm over a 59.74 mm2 field-of-view, which corresponds to a space-bandwidth product of 9.9 megapixels. We demonstrate the high-resolution, large-area, phase and birefringence imaging capability of our method by presenting the phase and birefringence images of various anisotropic objects, including a monosodium urate crystal, and excised mouse eye and heart tissues.

Structured modulation multi-height microscopy for high-resolution imaging

Conventional multi-height microscopy techniques introduce different object-to-detector distances to obtain multiple measurements for phase retrieval. However, surpassing the diffraction limit imposed by the numerical aperture (NA) of the objective lens remains a challenging task. Here, we report a novel structured modulation multi-height microscopy technique for quantitative high-resolution imaging. In our platform, a thin diffuser is placed in between the sample and the objective lens. By translating the diffuser to different axial positions, a sequence of modulated intensity images is captured for reconstruction. The otherwise inaccessible high-resolution object information can thus be encoded into the optical system for detection. In the construction process, we report a ptychographic phase retrieval algorithm to recover the existing wavefront of the complex object. We validate our approach using a resolution target, a phase target, and various biological samples. We demonstrate a ∼4-fold resolution gain over the diffraction limit. We also demonstrate our approach to achieve a 6.5 mm by 4.3 mm field of view and a half-pitch resolution of 1.2 µm. The reported methodology provides a portable, turnkey solution for quantitative high-resolution imaging with potential applications in disease diagnosis, sample screening, and other fields.

Lensless masked imaging with self-calibrated phase retrieval

Lensless imaging with a mask is an attractive topic as it enables a compact configuration to acquire wavefront information of a sample with computational approaches. Most existing methods choose a customized phase mask for wavefront modulation and then decode the sample's wave field from modulated diffraction patterns. Different from phase masks, lensless imaging with a binary amplitude mask facilitates a cheaper fabrication cost, but high-quality mask calibration and image reconstruction have not been well resolved. Here we propose a self-calibrated phase retrieval (SCPR) method to realize a joint recovery of a binary mask and sample's wave field for a lensless masked imaging system. Compared with conventional methods, our method shows a high-performance and flexible image recovery without the help of an extra calibration device. Experimental results of different samples demonstrate the superiority of our method.

Spatial- and Fourier-domain ptychography for high-throughput bio-imaging

First envisioned for determining crystalline structures, ptychography has become a useful imaging tool for microscopists. However, ptychography remains underused by biomedical researchers due to its limited resolution and throughput in the visible light regime. Recent developments of spatial- and Fourier-domain ptychography have successfully addressed these issues and now offer the potential for high-resolution, high-throughput optical imaging with minimal hardware modifications to existing microscopy setups, often providing an excellent trade-off between resolution and field of view inherent to conventional imaging systems, giving biomedical researchers the best of both worlds. Here, we provide extensive information to enable the implementation of ptychography by biomedical researchers in the visible light regime. We first discuss the intrinsic connections between spatial-domain coded ptychography and Fourier ptychography. A step-by-step guide then provides the user instructions for developing both systems with practical examples. In the spatial-domain implementation, we explain how a large-scale, high-performance blood-cell lens can be made at negligible expense. In the Fourier-domain implementation, we explain how adding a low-cost light source to a regular microscope can improve the resolution beyond the limit of the objective lens. The turnkey operation of these setups is suitable for use by professional research laboratories, as well as citizen scientists. Users with basic experience in optics and programming can build the setups within a week. The do-it-yourself nature of the setups also allows these procedures to be implemented in laboratory courses related to Fourier optics, biomedical instrumentation, digital image processing, robotics and capstone projects.

Polarization-sensitive intensity diffraction tomography

Optical anisotropy, which is an intrinsic property of many materials, originates from the structural arrangement of molecular structures, and to date, various polarization-sensitive imaging (PSI) methods have been developed to investigate the nature of anisotropic materials. In particular, the recently developed tomographic PSI technologies enable the investigation of anisotropic materials through volumetric mappings of the anisotropy distribution of these materials. However, these reported methods mostly operate on a single scattering model, and are thus not suitable for three-dimensional (3D) PSI imaging of multiple scattering samples. Here, we present a novel reference-free 3D polarization-sensitive computational imaging technique-polarization-sensitive intensity diffraction tomography (PS-IDT)-that enables the reconstruction of 3D anisotropy distribution of both weakly and multiple scattering specimens from multiple intensity-only measurements. A 3D anisotropic object is illuminated by circularly polarized plane waves at various illumination angles to encode the isotropic and anisotropic structural information into 2D intensity information. These information are then recorded separately through two orthogonal analyzer states, and a 3D Jones matrix is iteratively reconstructed based on the vectorial multi-slice beam propagation model and gradient descent method. We demonstrate the 3D anisotropy imaging capabilities of PS-IDT by presenting 3D anisotropy maps of various samples, including potato starch granules and tardigrade.

Multi-angle lensless ptychographic imaging via adaptive correction and the Nesterov method

Lensless systems based on ptychographic imaging can simultaneously achieve a large field of view and high resolution while having the advantages of small size, portability, and low cost compared to traditional lensed imaging. However, lensless imaging systems are susceptible to environmental noise and have a lower resolution of individual images than lens-based imaging systems, which means that they require a longer time to obtain a good result. Therefore, in this paper, to improve the convergence rate and robustness of noise in lensless ptychographic imaging, we propose an adaptive correction method, in which we add an adaptive error term and noise correction term in lensless ptychographic algorithms to reach convergence faster and create a better suppression effect on both Gaussian noise and Poisson noise. The Wirtinger flow and the Nesterov algorithms are used in our method to reduce computational complexity and improve the convergence rate. We applied the method to phase reconstruction for lensless imaging and demonstrated the effectiveness of the method by simulation and experiment. The method can be easily applied to other ptychographic iterative algorithms.

Depth-multiplexed ptychographic microscopy for high-throughput imaging of stacked bio-specimens on a chip

Imaging a large number of bio-specimens at high speed is essential for many biomedical applications. The common strategy is to place specimens at different lateral positions and image them sequentially. Here we report a new on-chip imaging strategy, termed depth-multiplexed ptychographic microscopy (DPM), for parallel imaging and sensing at high speed. Different from the common strategy, DPM stacks multiple specimens in the axial direction and images the entire z-stack all at once. In our prototype platform, we modify a low-cost car mirror for programmable steering of the incident laser beam. A blood-coated image sensor is then placed underneath the stacked sample for acquiring the resulting diffraction patterns. With the captured images, we perform blind recovery of the incident beam angle and model different layers of the stacked sample as different coded surfaces for object reconstruction. For in vitro experiment, we demonstrate time-lapse cell culture monitoring by imaging 3 stacked microfluidic channels on the coded sensor. For high-throughput cytometric analysis, we image 5 stacked brain sections with a 205-mm² field of view in ∼50 s. Cytometric analysis is also performed to quantify the cellular proliferation biomarkers on the slides. The DPM approach adds a new degree of freedom for data multiplexing in microscopy, enabling parallel imaging of multiple specimens using a single detector. The demonstrated 6-mm depth of field is among the longest ones in microscopy imaging. The novel depth-multiplexed configuration also complements the miniaturization provided by microfluidics devices, offering a solution for on-chip sensing and imaging with efficient sample handling.

sPhaseStation: a whole slide quantitative phase imaging system based on dual-view transport of intensity phase microscopy

Whole slide imaging scans a microscope slide into a high-resolution digital image, and it paves the way from pathology to digital diagnostics. However, most of them rely on bright-field and fluorescence imaging with sample labels. In this work, we designed sPhaseStation, which is a dual-view transport of intensity phase microscopy-based whole slide quantitative phase imaging system for label-free samples. sPhaseStation relies on a compact microscopic system with two imaging recorders that can capture both under and over-focus images. Combined with the field of view (FoV) scan, a series of these defocus images in different FoVs can be captured and stitched into two FoV-extended under and over-focus ones, which are used for phase retrieval via solving the transport of intensity equation. Using a 10× micro-objective, sPhaseStation reaches the spatial resolution of 2.19 µm and obtains the phase with high accuracy. Additionally, it acquires a whole slide image of a 3m m×3m m region in 2 min. The reported sPhaseStation could be a prototype of the whole slide quantitative phase imaging device, which may provide a new perspective for digital pathology.

Optofluidic imaging meets deep learning: from merging to emerging

Propelled by the striking advances in optical microscopy and deep learning (DL), the role of imaging in lab-on-a-chip has dramatically been transformed from a silo inspection tool to a quantitative "smart" engine. A suite of advanced optical microscopes now enables imaging over a range of spatial scales (from molecules to organisms) and temporal window (from microseconds to hours). On the other hand, the staggering diversity of DL algorithms has revolutionized image processing and analysis at the scale and complexity that were once inconceivable. Recognizing these exciting but overwhelming developments, we provide a timely review of their latest trends in the context of lab-on-a-chip imaging, or coined optofluidic imaging. More importantly, here we discuss the strengths and caveats of how to adopt, reinvent, and integrate these imaging techniques and DL algorithms in order to tailor different lab-on-a-chip applications. In particular, we highlight three areas where the latest advances in lab-on-a-chip imaging and DL can form unique synergisms: image formation, image analytics and intelligent image-guided autonomous lab-on-a-chip. Despite the on-going challenges, we anticipate that they will represent the next frontiers in lab-on-a-chip imaging that will spearhead new capabilities in advancing analytical chemistry research, accelerating biological discovery, and empowering new intelligent clinical applications.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

Recent progress of second near-infrared (NIR-II) fluorescence microscopy in bioimaging

Visualizing biological tissues in vivo at a cellular or subcellular resolution to explore molecular signaling and cell behaviors is a crucial direction for research into biological processes. In vivo imaging can provide quantitative and dynamic visualization/mapping in biology and immunology. New microscopy techniques combined with near-infrared region fluorophores provide additional avenues for further progress in vivo bioimaging. Based on the development of chemical materials and physical optoelectronics, new NIR-II microscopy techniques are emerging, such as confocal and multiphoton microscopy, light-sheet fluorescence microscopy (LSFM), and wide-field microscopy. In this review, we introduce the characteristics of in vivo imaging using NIR-II fluorescence microscopy. We also cover the recent advances in NIR-II fluorescence microscopy techniques in bioimaging and the potential for overcoming current challenges.

Automatic whole blood cell analysis from blood smear using label-free multi-modal imaging with deep neural networks

Whole blood cell analysis is widely used in medical applications since its results are indicators for diagnosing a series of diseases. In this work, we report automatic whole blood cell analysis from blood smear using label-free multi-modal imaging with deep neural networks. First, a commercial microscope equipped with our developed Phase Real-time Microscope Camera (PhaseRMiC) obtains both bright-field and quantitative phase images. Then, these images are automatically processed by our designed blood smear recognition networks (BSRNet) that recognize erythrocytes, leukocytes and platelets. Finally, blood cell parameters such as counts, shapes and volumes can be extracted according to both quantitative phase images and automatic recognition results. The proposed whole blood cell analysis technique provides high-quality blood cell images and supports accurate blood cell recognition and analysis. Moreover, this approach requires rather simple and cost-effective setups as well as easy and rapid sample preparations. Therefore, this proposed method has great potential application in blood testing aiming at disease diagnostics.

Miniaturized optimal incident light angle-fitted dark field system for contrast-enhanced real-time monitoring of 2D/3D-projected cell motions

In the field of biology, dark field microscopy provides superior insight into cells and subcellular structures. However, most dark field microscopes are equipped with a dark field filter and a light source on a 2D-based specimen, so only a flat sample can be observed in a limited space. We propose a compact cell monitoring system with built-in dark field filter with an optimized incident angle of the light source to provide real-time cell imaging and spatial cell monitoring for long-term free from phototoxicity. 2D projection imaging was implemented using a modular condenser lens to acquire high-contrast images. This enabled the long-term monitoring of cells, and the real-time monitoring of cell division and death. This system was able to image, by 2D projection, cells on the surface thinly coated with multiwalled carbon nanotubes, as well as living cells that migrated along the surface of glass beads and hydrogel droplets with a diameter of about 160 μm. The optimal incident light angle-fitted dark field system combines high-contrast imaging sensitivity and high spatial resolution to even image cells on 3D surfaces.

Lensless computational imaging with a hybrid framework of holographic propagation and deep learning

Lensless imaging has attracted attention as it avoids the bulky optical lens. Lensless holographic imaging is a type of a lensless imaging technique. Recently, deep learning has also shown tremendous potential in lensless holographic imaging. A labeled complex field including real and imaginary components of the samples is usually used as a training dataset. However, obtaining such a holographic dataset is challenging. In this Letter, we propose a lensless computational imaging technique with a hybrid framework of holographic propagation and deep learning. The proposed framework takes recorded holograms as input instead of complex fields, and compares the input and regenerated holograms. Compared to previous supervised learning schemes with a labeled complex field, our method does not require this supervision. Furthermore, we use the generative adversarial network to constrain the proposed framework and tackle the trivial solution. We demonstrate high-quality reconstruction with the proposed framework compared to previous deep learning methods.

Disposable Paper-Based Microfluidics for Fertility Testing

Fifteen percent of couples of reproductive age suffer from infertility globally and the burden of infertility disproportionately impacts residents of developing countries. Assisted reproductive technologies (ARTs), including in vitro fertilization (IVF) and intracytoplasmic sperm injection (ICSI), have been successful in overcoming various reasons for infertility including borderline and severe male factor infertility which consists of 20%–30% of all infertile cases. Approximately half of male infertility cases stem from suboptimal sperm parameters. Therefore, healthy/normal sperm enrichment and sorting remains crucial in advancing reproductive medicine. Microfluidic technologies have emerged as promising tools to develop in-home rapid fertility tests and point-of-care (POC) diagnostic tools. Here, we review advancements in fabrication methods for paper-based microfluidic devices and their emerging fertility testing applications assessing sperm concentration, sperm motility, sperm DNA analysis, and other sperm functionalities, and provide a glimpse into future directions for paper-based fertility microfluidic systems.

•

Paper-based technologies are emerging to develop in-home rapid fertility tests

•

Fabrication methods for paper-based microfluidic devices are presented

•

Emerging disposable paper-based fertility testing applications are reviewed

High-throughput digital pathology via a handheld, multiplexed, and AI-powered ptychographic whole slide scanner

The recent advent of whole slide imaging (WSI) systems has moved digital pathology closer to diagnostic applications and clinical practices. Integrating WSI with machine learning promises the growth of this field in upcoming years. Here we report the design and implementation of a handheld, colour-multiplexed, and AI-powered ptychographic whole slide scanner for digital pathology applications. This handheld scanner is built using low-cost and off-the-shelf components, including red, green, and blue laser diodes for sample illumination, a modified stage for programmable sample positioning, and a synchronized image sensor pair for data acquisition. We smear a monolayer of goat blood cells on the main sensor for high-resolution lensless coded ptychographic imaging. The synchronized secondary sensor acts as a non-contact encoder for precisely tracking the absolute object position for ptychographic reconstruction. For WSI, we introduce a new phase-contrast-based focus metric for post-acquisition autofocusing of both stained and unstained specimens. We show that the scanner can resolve the 388-nm linewidth on the resolution target and acquire gigapixel images with a 14 mm × 11 mm area in ∼70 seconds. The imaging performance is validated with regular stained pathology slides, unstained thyroid smears, and malaria-infected blood smears. The deep neural network developed in this study further enables high-throughput cytometric analysis using the recovered complex amplitude. The reported do-it-yourself scanner offers a portable solution to transform the high-end WSI system into one that can be made widely available at a low cost. The capability of high-throughput quantitative phase imaging may also find applications in rapid on-site evaluations.

Pixel Super-Resolution Phase Retrieval for Lensless On-Chip Microscopy via Accelerated Wirtinger Flow

Empowered by pixel super-resolution (PSR) and phase retrieval techniques, lensless on-chip microscopy opens up new possibilities for high-throughput biomedical imaging. However, the current PSR phase retrieval approaches are time consuming in terms of both the measurement and reconstruction procedures. In this work, we present a novel computational framework for PSR phase retrieval to address these concerns. Specifically, a sparsity-promoting regularizer is introduced to enhance the well posedness of the nonconvex problem under limited measurements, and Nesterov’s momentum is used to accelerate the iterations. The resulting algorithm, termed accelerated Wirtinger flow (AWF), achieves at least an order of magnitude faster rate of convergence and allows a twofold reduction in the measurement number while maintaining competitive reconstruction quality. Furthermore, we provide general guidance for step size selection based on theoretical analyses, facilitating simple implementation without the need for complicated parameter tuning. The proposed AWF algorithm is compatible with most of the existing lensless on-chip microscopes and could help achieve label-free rapid whole slide imaging of dynamic biological activities at subpixel resolution.

Deep distributed optimization for blind diffuser-modulation ptychography

Blind diffuser-modulation ptychography has emerged as a low-cost technique for micro-nano holographic imaging, which enables breaking the resolution limit of optical systems. However, the existing reconstruction method requires thousands of measurements to recover object and diffuser profile simultaneously, which makes the data acquisition time-consuming and cumbersome. In this Letter, we report a novel, to the best of our knowledge, blind ptychography technique with deep distributed optimization, termed BPD²O. It decomposes the complicated optimization task into subproblems, then introduces extended ptychographical iterative engine and enhanced network solver to optimize each in a distributed strategy. In this way, BPD²O combines the advantages of both model-driven and data-driven strategies, realizing high-fidelity robust ptychography imaging. Extensive experiments validate that BPD²O can realize better resolution and lead to a reduction of more than one order of magnitude in the number of measurements.

Lensfree on-chip microscopy based on single-plane phase retrieval

We propose a novel single-plane phase retrieval method to realize high-quality sample reconstruction for lensfree on-chip microscopy. In our method, complex wavefield reconstruction is modeled as a quadratic minimization problem, where total variation and joint denoising regularization are designed to keep a balance of artifact removal and resolution enhancement. In experiment, we built a 3D-printed field-portable platform to validate the imaging performance of our method, where resolution chart, dynamic target, transparent cell, polystyrene beads, and stained tissue sections are employed for the imaging test. Compared to state-of-the-art methods, our method eliminates image degradation and obtains a higher imaging resolution. Different from multi-wavelength or multi-height phase retrieval methods, our method only utilizes a single-frame intensity data record to accomplish high-fidelity reconstruction of different samples, which contributes a simple, robust, and data-efficient solution to design a resource-limited lensfree on-chip microscope. We believe that it will become a useful tool for telemedicine and point-of-care application.

Blood-Coated Sensor for High-Throughput Ptychographic Cytometry on a Blu-ray Disc

The Blu-ray drive is an engineering masterpiece that integrates disc rotation, pickup head translation, and three lasers in a compact and portable format. Here, we integrate a blood-coated image sensor with a modified Blu-ray drive for high-throughput cytometric analysis of various biospecimens. In this device, samples are mounted on the rotating Blu-ray disc and illuminated by the built-in lasers from the pickup head. The resulting coherent diffraction patterns are then recorded by the blood-coated image sensor. The rich spatial features of the blood-cell monolayer help down-modulate the object information for sensor detection, thus forming a high-resolution computational biolens with a theoretically unlimited field of view. With the acquired data, we develop a lensless coherent diffraction imaging modality termed rotational ptychography for image reconstruction. We show that our device can resolve the 435 nm line width on the resolution target and has a field of view only limited by the size of the Blu-ray disc. To demonstrate its applications, we perform high-throughput urinalysis by locating disease-related calcium oxalate crystals over the entire microscope slide. We also quantify different types of cells on a blood smear with an acquisition speed of ∼10,000 cells per second. For in vitro experiments, we monitor live bacterial cultures over the entire Petri dish with single-cell resolution. Using biological cells as a computational lens could enable new intriguing imaging devices for point-of-care diagnostics. Modifying a Blu-ray drive with the blood-coated sensor further allows the spread of high-throughput optical microscopy from well-equipped laboratories to citizen scientists worldwide.

Recent Advances in Lensless Imaging

Lensless imaging provides opportunities to design imaging systems free from the constraints imposed by traditional camera architectures. Thanks to advances in imaging hardware, fabrication techniques, and new algorithms, researchers have recently developed lensless imaging systems that are extremely compact, lightweight or able to image higher-dimensional quantities. Here we review these recent advances and describe the design principles and their effects that one should consider when developing and using lensless imaging systems.

Ptychographic sensor for large-scale lensless microbial monitoring with high spatiotemporal resolution

Traditional microbial detection methods often rely on the overall property of microbial cultures and cannot resolve individual growth event at high spatiotemporal resolution. As a result, they require bacteria to grow to confluence and then interpret the results. Here, we demonstrate the application of an integrated ptychographic sensor for lensless cytometric analysis of microbial cultures over a large scale and with high spatiotemporal resolution. The reported device can be placed within a regular incubator or used as a standalone incubating unit for long-term microbial monitoring. For longitudinal study where massive data are acquired at sequential time points, we report a new temporal-similarity constraint to increase the temporal resolution of ptychographic reconstruction by 7-fold. With this strategy, the reported device achieves a centimeter-scale field of view, a half-pitch spatial resolution of 488 nm, and a temporal resolution of 15-s intervals. For the first time, we report the direct observation of bacterial growth in a 15-s interval by tracking the phase wraps of the recovered images, with high phase sensitivity like that in interferometric measurements. We also characterize cell growth via longitudinal dry mass measurement and perform rapid bacterial detection at low concentrations. For drug-screening application, we demonstrate proof-of-concept antibiotic susceptibility testing and perform single-cell analysis of antibiotic-induced filamentation. The combination of high phase sensitivity, high spatiotemporal resolution, and large field of view is unique among existing microscopy techniques. As a quantitative and miniaturized platform, it can improve studies with microorganisms and other biospecimens at resource-limited settings.

Optofluidic ptychography on a chip

We report the implementation of a fully on-chip, lensless microscopy technique termed optofluidic ptychography. This imaging modality complements the miniaturization provided by microfluidics and allows the integration of ptychographic microscopy into various lab-on-a-chip devices. In our prototype, we place a microfluidic channel on the top surface of a coverslip and coat the bottom surface with a scattering layer. The channel and the coated coverslip substrate are then placed on top of an image sensor for diffraction data acquisition. Similar to the operation of a flow cytometer, the device utilizes microfluidic flow to deliver specimens across the channel. The diffracted light from the flowing objects is modulated by the scattering layer and recorded by the image sensor for ptychographic reconstruction, where high-resolution quantitative complex images are recovered from the diffraction measurements. By using an image sensor with a 1.85 μm pixel size, our device can resolve the 550 nm linewidth on the resolution target. We validate the device by imaging different types of biospecimens, including C. elegans, yeast cells, paramecium, and closterium sp. We also demonstrate a high-resolution ptychographic reconstruction at a video framerate of 30 frames per second. The reported technique can address a wide range of biomedical needs and engenders new ptychographic imaging innovations in a flow cytometer configuration.

Deep learning-enabled whole slide imaging (DeepWSI): oil-immersion quality using dry objectives, longer depth of field, higher system throughput, and better functionality

Whole slide imaging (WSI) has moved the traditional manual slide inspection process to the era of digital pathology. A typical WSI system translates the sample to different positions and captures images using a high numerical aperture (NA) objective lens. Performing oil-immersion microscopy is a major obstacle for WSI as it requires careful liquid handling during the scanning process. Switching between dry objective and oil-immersion lens is often impossible as it disrupts the acquisition process. For a high-NA objective lens, the sub-micron depth of field also poses a challenge to acquiring in-focus images of samples with uneven topography. Additionally, it implies a small field of view for each tile, thus limiting the system throughput and resulting in a long acquisition time. Here we report a deep learning-enabled WSI platform, termed DeepWSI, to substantially improve the system performance and imaging throughput. With this platform, we show that images captured with a regular dry objective lens can be transformed into images comparable to that of a 1.4-NA oil immersion lens. Blurred images with defocus distance from -5 µm to +5 µm can be virtually refocused to the in-focus plane post measurement. We demonstrate an equivalent data throughput of >2 gigapixels per second, the highest among existing WSI systems. Using the same deep neural network, we also report a high-resolution virtual staining strategy and demonstrate it for Fourier ptychographic WSI. The DeepWSI platform may provide a turnkey solution for developing high-performance diagnostic tools for digital pathology.

Ptychography-based high-throughput lensless on-chip microscopy via incremental proximal algorithms

Ptychography-based lensless on-chip microscopy enables high-throughput imaging by retrieving the missing phase information from intensity measurements. Numerous reconstruction algorithms for ptychography have been proposed, yet only a few incremental algorithms can be extended to lensless on-chip microscopy because of large-scale datasets but limited computational efficiency. In this paper, we propose the use of accelerated proximal gradient methods for blind ptychographic phase retrieval in lensless on-chip microscopy. Incremental gradient approaches are adopted in the reconstruction routine. Our algorithms divide the phase retrieval problem into sub-problems involving the evaluation of proximal operator, stochastic gradient descent, and Wirtinger derivatives. We benchmark the performances of accelerated proximal gradient, extended ptychographic iterative engine, and alternating direction method of multipliers, and discuss their convergence and accuracy in both noisy and noiseless cases. We also validate our algorithms using experimental datasets, where full field of view measurements are captured to recover the high-resolution complex samples. Among these algorithms, accelerated proximal gradient presents the overall best performance regarding accuracy and convergence rate. The proposed methods may find applications in ptychographic reconstruction, especially for cases where a wide field of view and high resolution are desired at the same time.

Quantitative multi-height phase retrieval via a coded image sensor

Multi-height phase retrieval introduces different object-to-detector distances for obtaining phase diversity measurements. In the acquisition process, the slow-varying phase information, however, cannot be converted to intensity variations for detection. Therefore, the low-frequency contents of the phase profile are lost during acquisition and cannot be properly restored via phase retrieval. Here, we demonstrate the use of a coded image sensor for addressing this challenge in multi-height phase retrieval. In our scheme, we add a coded layer on top of the image sensor for encoding the slow-varying complex wavefronts into intensity variations of the modulated patterns. Inspired by the concept of blind ptychography, we report a reconstruction scheme to jointly recover the complex object and the unknown coded layer using multi-height measurements. With both simulation and experimental results, we show that the recovered phase is quantitative and the slow-varying phase profiles can be properly restored using lensless multi-height measurements. We also show that the image quality using the coded sensor is better than that of a regular image sensor. For demonstrations, we validate the reported scheme with various biospecimens and compare the results to those of regular lensless multi-height phase retrieval. The use of a coded image sensor may enable true quantitative phase imaging for the lensless multi-height, multi-wavelength, and transport-of-intensity equation approaches.

High-throughput lensless whole slide imaging via continuous height-varying modulation of a tilted sensor

We report a new, to the best of our knowledge, lensless microscopy configuration by integrating the concepts of transverse translational ptychography and defocus multi-height phase retrieval. In this approach, we place a tilted image sensor under the specimen for introducing linearly increasing phase modulation along one lateral direction. Similar to the operation of ptychography, we laterally translate the specimen and acquire the diffraction images for reconstruction. Since the axial distance between the specimen and the sensor varies at different lateral positions, laterally translating the specimen effectively introduces defocus multi-height measurements while eliminating axial scanning. Lateral translation further introduces sub-pixel shift for pixel super-resolution imaging and naturally expands the field of view for rapid whole slide imaging. We show that the equivalent height variation can be precisely estimated from the lateral shift of the specimen, thereby addressing the challenge of precise axial positioning in conventional multi-height phase retrieval. Using a sensor with 1.67 µm pixel size, our low-cost and field-portable prototype can resolve the 690 nm linewidth on the resolution target. We show that a whole slide image of a blood smear with a 120mm² field of view can be acquired in 18 s. We also demonstrate accurate automatic white blood cell counting from the recovered image. The reported approach may provide a turnkey solution for addressing point-of-care and telemedicine-related challenges.

Adaptive super-resolution enabled on-chip contact microscopy

We demonstrate an adaptive super-resolution based contact imaging on a CMOS chip to achieve subcellular spatial resolution over a large field of view of ∼24 mm². By using regular LED illumination, we acquire the single lower-resolution image of the objects placed approximate to the sensor with unit magnification. For the raw contact-mode lens-free image, the pixel size of the sensor chip limits the spatial resolution. We develop a hybrid supervised-unsupervised strategy to train a super-resolution network, circumventing the missing of in-situ ground truth, effectively recovering a much higher resolution image of the objects, permitting sub-micron spatial resolution to be achieved across the entire sensor chip active area. We demonstrate the success of this approach by imaging the proliferation dynamics of cells directly cultured on the chip.

A Compact Raster Lensless Microscope Based on a Microdisplay

Lensless microscopy requires the simplest possible configuration, as it uses only a light source, the sample and an image sensor. The smallest practical microscope is demonstrated here. In contrast to standard lensless microscopy, the object is located near the lighting source. Raster optical microscopy is applied by using a single-pixel detector and a microdisplay. Maximum resolution relies on reduced LED size and the position of the sample respect the microdisplay. Contrarily to other sort of digital lensless holographic microscopes, light backpropagation is not required to reconstruct the images of the sample. In a mm-high microscope, resolutions down to 800 nm have been demonstrated even when measuring with detectors as large as 138 μm × 138 μm, with field of view given by the display size. Dedicated technology would shorten measuring time.

PhaseRMiC: phase real-time microscope camera for live cell imaging

We design a novel phase real-time microscope camera (PhaseRMiC) for live cell phase imaging. PhaseRMiC has a simple and cost-effective configuration only consisting of a beam splitter and a board-level camera with two CMOS imaging chips. Moreover, integrated with 3-D printed structures, PhaseRMiC has a compact size of 136×91×60 mm³, comparable to many commercial microscope cameras, and can be directly connected to the microscope side port. Additionally, PhaseRMiC can be well adopted in real-time phase imaging proved with satisfied accuracy, good stability and large field of view. Considering its compact and cost-effective device design as well as real-time phase imaging capability, PhaseRMiC is a preferred solution for live cell imaging.

A Novel Approach for a Chip-Sized Scanning Optical Microscope

The recent advances in chip-size microscopy based on optical scanning with spatially resolved nano-illumination light sources are presented. This new straightforward technique takes advantage of the currently achieved miniaturization of LEDs in fully addressable arrays. These nano-LEDs are used to scan the sample with a resolution comparable to the LED sizes, giving rise to chip-sized scanning optical microscopes without mechanical parts or optical accessories. The operation principle and the potential of this new kind of microscope are analyzed through three different implementations of decreasing LED dimensions from 20 µm down to 200 nm.

Mask-modulated lensless imaging via translated structured illumination

Lensless microscopy technique enables high-resolution image recovery over a large field of view. By integrating the concept of phase retrieval, it can also retrieve the lost phase information from intensity-only measurements. Here we report a mask-modulated lensless imaging platform based on translated structured illumination. In the reported platform, we sandwich the object in-between a coded mask and a naked image sensor for lensless data acquisition. An LED array is used to provide angle-varied illumination for projecting a translated structured pattern without involving mechanical scanning. For different LED elements, we acquire the lensless intensity data for recovering the complex-valued object. In the reconstruction process, we employ the regularized ptychographic iterative engine and implement an up-sampling process in the reciprocal space. As demonstrated by experimental results, the reported platform is able to recover complex-valued object images with higher resolution and better quality than previous implementations. Our approach may provide a cost-effective solution for high-resolution and wide field-of-view ptychographic imaging without involving mechanical scanning.

Autofocusing technologies for whole slide imaging and automated microscopy

Whole slide imaging (WSI) has moved digital pathology closer to diagnostic practice in recent years. Due to the inherent tissue topography variability, accurate autofocusing remains a critical challenge for WSI and automated microscopy systems. The traditional focus map surveying method is limited in its ability to acquire a high degree of focus points while still maintaining high throughput. Real-time approaches decouple image acquisition from focusing, thus allowing for rapid scanning while maintaining continuous accurate focus. This work reviews the traditional focus map approach and discusses the choice of focus measure for focal plane determination. It also discusses various real-time autofocusing approaches including reflective-based triangulation, confocal pinhole detection, low-coherence interferometry, tilted sensor approach, independent dual sensor scanning, beam splitter array, phase detection, dual-LED illumination and deep-learning approaches. The technical concepts, merits and limitations of these methods are explained and compared to those of a traditional WSI system. This review may provide new insights for the development of high-throughput automated microscopy imaging systems that can be made broadly available and utilizable without loss of capacity.

Lensless imaging of plant samples using the cross-polarized light

Lensless imaging has recently become an alternative and cost-effective choice for many macro and micro applications, like wave-front sensing, fluorescence imaging, holographic microscopy, and so on. However, the polarized imaging, especially the cross-polarized light, has rarely been explored and integrated in lensless imaging methods. In this paper, we introduce the cross-polarized illumination into the lensless system for high-contrast and background-free imaging of plant samples. We capture a snapshot measurement and apply the blind deconvolution for reconstruction, obtaining the depolarized imaging of plant samples. Experiments exhibit the specific and sparse structures of the root system and vessel distribution of samples. We also build a corresponding lens-based system for performance comparison. This proposed lensless system is believed to have the potential in studies on the root development and water transport mechanism of plants in the future.

Virtual brightfield and fluorescence staining for Fourier ptychography via unsupervised deep learning

Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. Acquiring color images of histology slides often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learning approach for performing unsupervised image-to-image translation of FPM reconstructions. A cycle-consistent adversarial network with multiscale structure similarity loss is trained to perform virtual brightfield and fluorescence staining of the recovered FPM images. In the training stage, we feed the network with two sets of unpaired images: (1) monochromatic FPM recovery and (2) color or fluorescence images captured using a regular microscope. In the inference stage, the network takes the FPM input and outputs a virtually stained image with reduced coherent artifacts and improved image quality. We test the approach on various samples with different staining protocols. High-quality color and fluorescence reconstructions validate its effectiveness.

Single-shot ptychography with highly tilted illuminations

A single-shot ptychographic iterative engine (PIE) using highly tilted illumination is proposed to realize accurate phase retrieval from a single frame of multiple and non-overlapping sub-diffraction patterns generated by a bunch of laser beams propagating at greater angles with respect to the optical axis. A non-paraxial reconstruction algorithm is developed to numerically propagate these highly tilted laser beams in back and forth iterative computations. Faster data acquisition and higher reconstruction quality are achieved in the proposed method by recording non-overlapping sub-diffraction patterns in a single frame and eliminating usual reconstruction errors arising from paraxial approximations.

Super-resolved multispectral lensless microscopy via angle-tilted, wavelength-multiplexed ptychographic modulation

We report an angle-tilted, wavelength-multiplexed ptychographic modulation approach for multispectral lensless on-chip microscopy. In this approach, we illuminate the specimen with lights at five wavelengths simultaneously. A prism is added at the illumination path for spectral dispersion. Thus, lightwaves at different wavelengths hit the specimen at slightly different incident angles, breaking the ambiguities in mixed-state ptychographic reconstruction. At the detection path, we place a thin diffuser between the specimen and the monochromatic image sensor for encoding the spectral information into 2D intensity measurements. By scanning the sample to different x-y positions, we acquire a sequence of monochromatic images for reconstructing the five complex object profiles at the five wavelengths. An up-sampling procedure is integrated into the recovery process to bypass the resolution limit imposed by the imager pixel size. We demonstrate a half-pitch resolution of 0.55 µm using an image sensor with 1.85 µm pixel size. We also demonstrate quantitative and high-quality multispectral reconstructions of stained tissue sections for digital pathology applications.

A super-resolution scanning algorithm for lensless microfluidic imaging using the dual-line array image sensor

The lensless optical fluid microscopy is of great significance to the miniaturization, portability and low cost development of cell detection instruments. However, the resolution of the cell image collected directly is low, because the physical pixel size of the image sensor is the same order of magnitude as the cell size. To solve this problem, this paper proposes a super-resolution scanning algorithm using a dual-line array sensor and a microfluidic chip. For dual-line array sensor images, the multi-group velocity and acceleration of cells flowing through the line array sensor are calculated. Then the reconstruction model of the super-resolution image is constructed with variable acceleration. By changing the angle between the line array image sensor and the direction of cell flow, the super-resolution image scanning and reconstruction are achieved in both horizontal and vertical directions. In addition, it is necessary to study the row by row extraction algorithm for cell foreground image. In this paper, the dual-line array sensor is implemented by adjusting the acquisition window of the image sensor with a pixel size of 2.2μm. When the tilt angle is 21 degrees, the equivalent pixel size is 0.79μm, improved 2.8 times, and after de-diffraction its average size error was 3.249%. As the angle decreases, the image resolution is higher, but the amount of information is less. This super-resolution scanning algorithm can be integrated on the chip and used with a microfluidic chip to realize on-chip instrument.

Pixel super-resolved lens-free on-chip microscopy based on dual laterally shifting modulation

Achieving high spatial resolution over a wide field of view (FOV) is the goal of many imaging systems. In a traditional lens-based microscope, designing a complex objective with high numerical aperture (NA) to achieve this goal is a tough and challenging task. The lens-free wide-field imaging method based on phase retrieval provides a new way to bypass the trade-off between the spatial resolution and FOV of conventional microscopy. However, the typical lens-free microscopy usually requires mechanical devices with high precision and repeatability. In this paper, we report a robust and cost-effective pixel super-resolved lens-free imaging method based on dual laterally shifting modulation. A thin diffuser is inserted between the object and the image sensor to be used as the modulator. The diffuser and the object are transversely scanned at the same time to add diversities for phase retrieval and pixel super-resolution, respectively. In this way, the positional shifts of the diffuser and the object can be directly recovered with the registration algorithm, thus addressing the low stability and inaccuracy issues of translation stages. We also propose a pixel super-resolution phase-retrieval algorithm to recover the object and the unknown diffuser. We first use numerical simulations to evaluate the proposed scheme. Then, we validate this approach by imaging a resolution target and a pollen sample, thus achieving an FOV of ${\sim}{30}\;{\text{mm}^2}$∼30mm² and a half-pitch resolution of 0.78 µm, which surpasses 2.14 times the theoretical Nyquist-Shannon sampling resolution limit. Finally, the 3D refocusing ability is also verified by imaging a thick mosquito sample.