Computational aberration compensation by coded-aperture-based correction of aberration obtained from optical Fourier coding and blur estimation

Computational single fundus image restoration techniques: a review

Fundus cameras are widely used by ophthalmologists for monitoring and diagnosing retinal pathologies. Unfortunately, no optical system is perfect, and the visibility of retinal images can be greatly degraded due to the presence of problematic illumination, intraocular scattering, or blurriness caused by sudden movements. To improve image quality, different retinal image restoration/enhancement techniques have been developed, which play an important role in improving the performance of various clinical and computer-assisted applications. This paper gives a comprehensive review of these restoration/enhancement techniques, discusses their underlying mathematical models, and shows how they may be effectively applied in real-life practice to increase the visual quality of retinal images for potential clinical applications including diagnosis and retinal structure recognition. All three main topics of retinal image restoration/enhancement techniques, i.e., illumination correction, dehazing, and deblurring, are addressed. Finally, some considerations about challenges and the future scope of retinal image restoration/enhancement techniques will be discussed.

Dynamic opto-mechanical eye model with peripheral refractions

Many myopia control methods based on the peripheral defocus theory have emerged towards applications in recent years. However, peripheral aberration is a critical issue, which is still not well-addressed. To validate the aberrometer for peripheral aberration measurement, a dynamic opto-mechanical eye model with a wide visual field is developed in this study. This model consists of a plano-convex lens representing cornea (f' = 30 mm), a double-convex lens representing crystalline lens (f' = 100 mm), and a spherical retinal screen with a radius of 12 mm. To optimize the quality of spot-field images from the Hartman-Shack sensor, the materials and surface topography for the retina are studied. The model has an adjustable retina to achieve Zernike 4th item (Z4 focus) ranging from -6.28 µm to +6.84 µm. As for mean sphere equivalent, it can achieve -10.52 D to +9.16 D at 0° visual field and -6.97 D to +5.88 D at 30° visual field with a pupil size of 3 mm. To realize a changing pupil size, a slot at the back of the cornea mount and a series of thin metal sheets with 2, 3, 4, and 6 mm holes are generated. Both on-axis aberrations and peripheral aberrations of the eye model are verified by a well-used aberrometer and the eye model to mimic a human eye in a peripheral aberration measurement system is illustrated.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

Extended depth-of-field infrared imaging with deeply learned wavefront coding

Wavefront coding (WFC) techniques, including optical coding and digital image processing stages, enable significant capabilities for extending the depth of field of imaging systems. In this study, we demonstrated a deeply learned far-infrared WFC camera with an extended depth of field. We designed and optimized a high-order polynomial phase mask by a genetic algorithm, exhibiting a higher defocus consistency of the modulated transfer functions than works published previously. Additionally, we trained a generative adversarial network based on a synthesized WFC dataset for the digital processing part, which is more effective and robust than conventional decoding methods. Furthermore, we captured real-world infrared images using the WFC camera with far, middle, and near object distances. Their results after wavefront coding/decoding showed that the model of deeply learned networks improves the image quality and signal-to-noise ratio significantly and quickly. Therefore, we construct a novel artificial intelligent method of deeply learned WFC optical imaging by applying infrared wavelengths, but not limited to, and provide good potential for its practical application in "smart" imaging and large range target detection.

An integrated imaging sensor for aberration-corrected 3D photography

Planar digital image sensors facilitate broad applications in a wide range of areas^1-5, and the number of pixels has scaled up rapidly in recent years^2,6. However, the practical performance of imaging systems is fundamentally limited by spatially nonuniform optical aberrations originating from imperfect lenses or environmental disturbances^7,8. Here we propose an integrated scanning light-field imaging sensor, termed a meta-imaging sensor, to achieve high-speed aberration-corrected three-dimensional photography for universal applications without additional hardware modifications. Instead of directly detecting a two-dimensional intensity projection, the meta-imaging sensor captures extra-fine four-dimensional light-field distributions through a vibrating coded microlens array, enabling flexible and precise synthesis of complex-field-modulated images in post-processing. Using the sensor, we achieve high-performance photography up to a gigapixel with a single spherical lens without a data prior, leading to orders-of-magnitude reductions in system capacity and costs for optical imaging. Even in the presence of dynamic atmosphere turbulence, the meta-imaging sensor enables multisite aberration correction across 1,000 arcseconds on an 80-centimetre ground-based telescope without reducing the acquisition speed, paving the way for high-resolution synoptic sky surveys. Moreover, high-density accurate depth maps can be retrieved simultaneously, facilitating diverse applications from autonomous driving to industrial inspections.

Calibration of transverse ray and pupil aberrations for light field cameras

The accuracy of reconstructing depth maps or performing digital refocusing in light field cameras depends largely on how well the spatial and angular samples of light rays can be obtained. Ray sample errors induced by optical aberrations in a light field camera may be digitally corrected using the ray tracing data when its nominal lens design is available. However, the most commonly nominal lens prescription is not accessible to end users. Additionally, even if available, due to tolerances in optomechanical design, the ray tracing data can be inaccurate. We propose a calibration method based on measurements of fiducial markers on a checkerboard for modeling the imaging properties of light field cameras. The calibration accounts for vignetting, transverse ray errors, as well as pupil aberration, and can be applied to light field camera modeling of arbitrary pupil sampling systems. We further demonstrate the utility of the method for calibrating a tri-aperture camera that captures simultaneous stereo views via artificially induced transverse ray errors.

Single-shot blind deconvolution with coded aperture

In this paper, we present a method for single-shot blind deconvolution incorporating a coded aperture (CA). In this method, we utilize the CA, inserted on the pupil plane, as support constraints in blind deconvolution. Not only an object is estimated, but also a point spread function of turbulence from a single captured image by a reconstruction algorithm with CA support. The proposed method is demonstrated by simulation and an experiment in which point sources are recovered under severe turbulence.

Fourier Ptychographic Microscopy via Alternating Direction Method of Multipliers

Fourier ptychographic microscopy (FPM) has risen as a promising computational imaging technique that breaks the trade-off between high resolution and large field of view (FOV). Its reconstruction is normally formulated as a blind phase retrieval problem, where both the object and probe have to be recovered from phaseless measured data. However, the stability and reconstruction quality may dramatically deteriorate in the presence of noise interference. Herein, we utilized the concept of alternating direction method of multipliers (ADMM) to solve this problem (termed ADMM-FPM) by breaking it into multiple subproblems, each of which may be easier to deal with. We compared its performance against existing algorithms in both simulated and practical FPM platform. It is found that ADMM-FPM method belongs to a global optimization algorithm with a high degree of parallelism and thus results in a more stable and robust phase recovery under noisy conditions. We anticipate that ADMM will rekindle interest in FPM as more modifications and innovations are implemented in the future.

SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-View Stereopsis

Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. As the observations become sparser, the significant 3D information loss makes the MVS problem more challenging. Instead of only focusing on densely sampled conditions, we investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. By investigating various observation sparsities, we show that the classical depth-fusion pipeline becomes powerless for the case with a larger baseline angle that worsens the photo-consistency check. As another line of the solution, we present SurfaceNet+, a volumetric method to handle the 'incompleteness' and the 'inaccuracy' problems induced by a very sparse MVS setup. Specifically, the former problem is handled by a novel volume-wise view selection approach. It owns superiority in selecting valid views while discarding invalid occluded views by considering the geometric prior. Furthermore, the latter problem is handled via a multi-scale strategy that consequently refines the recovered geometry around the region with the repeating pattern. The experiments demonstrate the tremendous performance gap between SurfaceNet+ and state-of-the-art methods in terms of precision and recall. Under the extreme sparse-MVS settings in two datasets, where existing methods can only return very few points, SurfaceNet+ still works as well as in the dense MVS setting.

Review of bio-optical imaging systems with a high space-bandwidth product

Optical imaging has served as a primary method to collect information about biosystems across scales-from functionalities of tissues to morphological structures of cells and even at biomolecular levels. However, to adequately characterize a complex biosystem, an imaging system with a number of resolvable points, referred to as a space-bandwidth product (SBP), in excess of one billion is typically needed. Since a gigapixel-scale far exceeds the capacity of current optical imagers, compromises must be made to obtain either a low spatial resolution or a narrow field-of-view (FOV). The problem originates from constituent refractive optics-the larger the aperture, the more challenging the correction of lens aberrations. Therefore, it is impractical for a conventional optical imaging system to achieve an SBP over hundreds of millions. To address this unmet need, a variety of high-SBP imagers have emerged over the past decade, enabling an unprecedented resolution and FOV beyond the limit of conventional optics. We provide a comprehensive survey of high-SBP imaging techniques, exploring their underlying principles and applications in bioimaging.

Optical Aberration Calibration and Correction of Photographic System Based on Wavefront Coding

The image deconvolution technique can recover potential sharp images from blurred images affected by aberrations. Obtaining the point spread function (PSF) of the imaging system accurately is a prerequisite for robust deconvolution. In this paper, a computational imaging method based on wavefront coding is proposed to reconstruct the wavefront aberration of a photographic system. Firstly, a group of images affected by local aberration is obtained by applying wavefront coding on the optical system’s spectral plane. Then, the PSF is recovered accurately by pupil function synthesis, and finally, the aberration-affected images are recovered by image deconvolution. After aberration correction, the image’s coefficient of variation and mean relative deviation are improved by 60% and 30%, respectively, and the image can reach the limit of resolution of the sensor, as proved by the resolution test board. Meanwhile, the method’s robust anti-noise capability is confirmed through simulation experiments. Through the conversion of the complexity of optical design to a post-processing algorithm, this method offers an economical and efficient strategy for obtaining high-resolution and high-quality images using a simple large-field lens.

Coherent synthetic aperture imaging for visible remote sensing via reflective Fourier ptychography

Synthetic aperture radar can measure the phase of a microwave with an antenna, which cannot be directly extended to visible light imaging due to phase lost. In this Letter, we report an active remote sensing with visible light via reflective Fourier ptychography, termed coherent synthetic aperture imaging (CSAI), achieving high resolution, a wide field-of-view (FOV), and phase recovery. A proof-of-concept experiment is reported with laser scanning and a collimator for the infinite object. Both smooth and rough objects are tested, and the spatial resolution increased from 15.6 to 3.48 µm with a factor of 4.5. The speckle noise can be suppressed obviously, which is important for coherent imaging. Meanwhile, the CSAI method can tackle the aberration induced from the optical system by one-step deconvolution and shows the potential to replace the adaptive optics for aberration removal of atmospheric turbulence.

Punching holes in light: recent progress in single-shot coded-aperture optical imaging

Single-shot coded-aperture optical imaging physically captures a code-aperture-modulated optical signal in one exposure and then recovers the scene via computational image reconstruction. Recent years have witnessed dazzling advances in various modalities in this hybrid imaging scheme in concomitant technical improvement and widespread applications in physical, chemical and biological sciences. This review comprehensively surveys state-of-the-art single-shot coded-aperture optical imaging. Based on the detected photon tags, this field is divided into six categories: planar imaging, depth imaging, light-field imaging, temporal imaging, spectral imaging, and polarization imaging. In each category, we start with a general description of the available techniques and design principles, then provide two representative examples of active-encoding and passive-encoding approaches, with a particular emphasis on their methodology and applications as well as their advantages and challenges. Finally, we envision prospects for further technical advancement in this field.

High-resolution and large field-of-view Fourier ptychographic microscopy and its applications in biomedicine

Fourier ptychographic microscopy (FPM) is a promising and fast-growing computational imaging technique with high resolution, wide field-of-view (FOV) and quantitative phase recovery, which effectively tackles the problems of phase loss, aberration-introduced artifacts, narrow depth-of-field and the trade-off between resolution and FOV in conventional microscopy simultaneously. In this review, we provide a comprehensive roadmap of microscopy, the fundamental principles, advantages, and drawbacks of existing imaging techniques, and the significant roles that FPM plays in the development of science. Since FPM is an optimization problem in nature, we discuss the framework and related work. We also reveal the connection of Euler's formula between FPM and structured illumination microscopy. We review recent advances in FPM, including the implementation of high-precision quantitative phase imaging, high-throughput imaging, high-speed imaging, three-dimensional imaging, mixed-state decoupling, and introduce the prosperous biomedical applications. We conclude by discussing the challenging problems and future applications. FPM can be extended to a kind of framework to tackle the phase loss and system limits in the imaging system. This insight can be used easily in speckle imaging, incoherent imaging for retina imaging, large-FOV fluorescence imaging, etc.

Fourier ptychography: current applications and future promises

Traditional imaging systems exhibit a well-known trade-off between the resolution and the field of view of their captured images. Typical cameras and microscopes can either "zoom in" and image at high-resolution, or they can "zoom out" to see a larger area at lower resolution, but can rarely achieve both effects simultaneously. In this review, we present details about a relatively new procedure termed Fourier ptychography (FP), which addresses the above trade-off to produce gigapixel-scale images without requiring any moving parts. To accomplish this, FP captures multiple low-resolution, large field-of-view images and computationally combines them in the Fourier domain into a high-resolution, large field-of-view result. Here, we present details about the various implementations of FP and highlight its demonstrated advantages to date, such as aberration recovery, phase imaging, and 3D tomographic reconstruction, to name a few. After providing some basics about FP, we list important details for successful experimental implementation, discuss its relationship with other computational imaging techniques, and point to the latest advances in the field while highlighting persisting challenges.

Raster adaptive optics for video rate aberration correction and large FOV multiphoton imaging

Removal of complex aberrations at millisecond time scales over millimeters in distance in multiphoton laser scanning microscopy limits the total spatiotemporal imaging throughput for deep tissue imaging. Using a single low resolution deformable mirror and time multiplexing (TM) adaptive optics, we demonstrate video rate aberration correction (5 ms update rate for a single wavefront mask) for a complex heterogeneous distribution of refractive index differences through a depth of up to 1.1 mm and an extended imaging FOV of up to 0.8 mm, with up to 167% recovery of fluorescence intensity 335 µm from the center of the FOV. The proposed approach, termed raster adaptive optics (RAO), integrates image-based aberration retrieval and video rate removal of arbitrarily defined regions of dominant, spatially varied wavefronts. The extended FOV was achieved by demonstrating rapid recovery of up to 50 distinct wavefront masks at 500 ms update rates that increased imaging throughput by 2.3-fold. Because RAO only requires a single deformable mirror with image-based aberration retrieval, it can be directly implemented on a standard laser scanning multiphoton microscope.

Reflective Fourier ptychographic microscopy using a parabolic mirror

Fourier ptychography uses a phase retrieval algorithm to reconstruct a high-resolution image with a wide field-of-view. Reflective-type Fourier ptychographic microscopy (FPM) is expected to be very useful for surface inspection, but the reported methods have several limitations. We propose a darkfield illuminator for reflective FPM consisting of a parabolic mirror and a flat LED panel. This increases the signal-to-noise ratio of the acquired images because the normal beam of each LED is directed toward the object. Furthermore, the LEDs do not have to be far from the object because they are collimated by the parabolic surface before illumination. Based on this, a reflective FPM with a synthesized numerical aperture (NA) of 1.06 was achieved, which is the highest value by reflective FPM as far as we know. To validate this experimentally, we measured a USAF reflective resolution target and reconstructed a high-resolution image. This resolved up to the period of 488 nm, which corresponds to the synthesized NA. Additionally, an integrated circuit was measured to demonstrate the effectiveness of surface inspection of the proposed system.

Computational aberration correction of VIS-NIR multispectral imaging microscopy based on Fourier ptychography

Due to the chromatic dispersion properties inherent in all optical materials, even the best-designed multispectral objective will exhibit residual chromatic aberration. Here, we demonstrate a multispectral microscope with a computational scheme based on the Fourier ptychographic microscopy (FPM) to correct these effects in order to render undistorted, in-focus images. The microscope consists of 4 spectral channels ranging from 405 nm to 1552 nm. After the computational aberration correction, it can achieve isotropic resolution enhancement as verified with the Siemens star sample. We image a flip-chip to show the promise of our system to conduct fault detection on silicon chips. This computational approach provides a cost-efficient strategy for high quality multispectral imaging over a broad spectral range.