Co-designed annular binary phase masks for depth-of-field extension in single-molecule localization microscopy

Highly sensitive volumetric single-molecule imaging

Volumetric subcellular imaging has long been essential for studying structures and dynamics in cells and tissues. However, due to limited imaging speed and depth of field, it has been challenging to perform live-cell imaging and single-particle tracking. Here we report a 2.5D fluorescence microscopy combined with highly inclined illumination beams, which significantly reduce not only the image acquisition time but also the out-of-focus background by ∼2-fold compared to epi-illumination. Instead of sequential z-scanning, our method projects a certain depth of volumetric information onto a 2D plane in a single shot using multi-layered glass for incoherent wavefront splitting, enabling high photon detection efficiency. We apply our method to multi-color immunofluorescence imaging and volumetric super-resolution imaging, covering ∼3-4 µm thickness of samples without z-scanning. Additionally, we demonstrate that our approach can substantially extend the observation time of single-particle tracking in living cells.

Telephoto achromatic camera based on optical-digital co-design

Due to the difficulty of correcting chromatic aberration (CA) in telephoto cameras, recent studies have combined image algorithms with simple optical structures, such as single-spherical lenses, for high-quality photography, moving away from complex optics. However, this approach often struggles to comprehensively address compounded issues arising from optical aberrations of simple optical systems, including defocus blur and multi-channel misalignment. To tackle this challenge, this manuscript presents an approach for developing a telephoto imaging system by leveraging the distinct characteristics of axial and lateral chromatic aberrations (ACA, LCA) over the visible spectrum. The optical design is limited to a specific wavelength range to preserve high-frequency information of the green channel. A cross-channel fitting method is presented to suppress the LCA. Subsequently, the powerful capabilities of deep learning are utilized to correct ACA, defocus blur, and other residual optical aberrations. Simulation experiments demonstrate the effectiveness of the proposed approach in mitigating the CA inherent in telephoto systems, thereby delivering high-quality imaging results over the whole visible waveband.

On the equivalence of binary phase masks optimized for localization or detection in extended depth-of-field localization microscopy

Binary annular masks have recently been proposed to extend the depth of field (DoF) of single-molecule localization microscopy. A strategy for designing optimal masks has been introduced based on maximizing the emitter localization accuracy, expressed in terms of Fisher information, over a targeted DoF range. However, the complete post-processing pipeline to localize a single emitter consists of two successive steps: detection, where the regions containing emitters are determined, and localization, where the sub-pixel position of each detected emitter is estimated. Phase masks usually optimize only this second step. The presence of a phase mask also affecting detection, the purpose of this paper is to quantify and mitigate this effect. Using a rigorous framework built from a detection-oriented information theoretical criterion (Bhattacharyya distance), we demonstrate that in most cases of practical significance, annular binary phase masks maximizing Fisher information also maximize the detection probability. This result supports the common design practice consisting of optimizing a phase mask by maximizing Fisher information only.

Miniature color camera via flat hybrid meta-optics

The race for miniature color cameras using flat meta-optics has rapidly developed the end-to-end design framework using neural networks. Although a large body of work has shown the potential of this methodology, the reported performance is still limited due to fundamental limitations coming from meta-optics, mismatch between simulated and resultant experimental point spread functions, and calibration errors. Here, we use a HIL optics design methodology to solve these limitations and demonstrate a miniature color camera via flat hybrid meta-optics (refractive + meta-mask). The resulting camera achieves high-quality full-color imaging for a 5-mm aperture optics with a focal length of 5 mm. We observed a superior quality of the images captured by the hybrid meta-optical camera compared to a compound multi-lens optics of a mirrorless commercial camera.

Hybrid diffractive optics design via hardware-in-the-loop methodology for achromatic extended-depth-of-field imaging

End-to-end optimization of diffractive optical elements (DOEs) profile through a digital differentiable model combined with computational imaging have gained an increasing attention in emerging applications due to the compactness of resultant physical setups. Despite recent works have shown the potential of this methodology to design optics, its performance in physical setups is still limited and affected by manufacturing artefacts of DOE, mismatch between simulated and resultant experimental point spread functions, and calibration errors. Additionally, the computational burden of the digital differentiable model to effectively design the DOE is increasing, thus limiting the size of the DOE that can be designed. To overcome the above mentioned limitations, a co-design of hybrid optics and image reconstruction algorithm is produced following the end-to-end hardware-in-the-loop strategy, using for optimization a convolutional neural network equipped with quantitative and qualitative loss functions. The optics of the imaging system consists on the phase-only spatial light modulator (SLM) as DOE and refractive lens. SLM phase-pattern is optimized by applying the Hardware-in-the-loop technique, which helps to eliminate the mismatch between numerical modelling and physical reality of image formation as light propagation is not numerically modelled but is physically done. Comparison with compound multi-lens optics of a last generation smartphone and a mirrorless commercial cameras show that the proposed system is advanced in all-in-focus sharp imaging for a depth range 0.4-1.9 m.

Optimizing sampling for surface localization in 3D-scanning microscopy

3D-scanning fluorescence imaging of living tissue is in demand for less phototoxic acquisition process. For the imaging of biological surfaces, adaptive and sparse scanning schemes have been proven to efficiently reduce the light dose by concentrating acquisitions around the surface. In this paper, we focus on optimizing the scanning scheme at a constant photon budget, when the problem is to estimate the position of a biological surface whose intensity profile is modeled as a Gaussian shape. We propose an approach based on the Cramér-Rao bound to optimize the positions and number of scanning points, assuming signal-dependant Gaussian noise. We show that, in the case of regular sampling, the optimization problem can be reduced to a few parameters, allowing us to define quasi-optimal acquisition strategies, first when no prior knowledge of the surface location is available and then when the user has a prior on this location.

A Drop-in, Focus-Extending Phase Mask Simplifies Microscopic and Microfluidic Imaging Systems for Cost-Effective Point-of-Care Diagnostics

Microscopic imaging and imaging flow cytometry have wide potential in point-of-care assays; however, their narrow depth of focus necessitates precise mechanical or fluidic focus control of a sample in order to acquire high-quality images that can be used for downstream analysis, increasing the cost and complexity of the imaging system. This complexity represents a barrier to miniaturization and translation of point-of-care assays based on microscopic imaging or imaging flow cytometry. To address this challenge, we present a simple drop-in phase mask with a physics-informed, circularly symmetric asphere phase profile that extends the depth of focus by >5-fold while largely preserving the image quality compared to other depth extending methods. We show that such a focus-extended system overcomes manufacturing tolerances in low-cost sample chambers, enlarges the useable field-of-view of low-cost objectives, and permits increased throughput and precision in flow imaging systems without the need for complex flow-focusing. As the image quality is preserved without the need for postacquisition image restoration, our solution is also highly appropriate for on-line applications such as cell sorting.

On the validity domain of maximum likelihood estimators for depth-of-field extension in single-molecule localization microscopy

Localization microscopy approaches with enhanced depth-of-field (EDoF) are commonly optimized using the Cramér-Rao bound (CRB) as a criterion. It is widely believed that the CRB can be attained in practice by using the maximum-likelihood estimator (MLE). This is, however, an approximation, of which we define in this paper the precise domain of validity. Exploring a wide range of settings and noise levels, we show that the MLE is efficient when the signal-to-noise ratio (SNR) is such that the localization standard deviation of a single molecule is less than 20 nm. Thus, our results provide an explicit and quantitative validity boundary for the use of the MLE in EDoF localization microscopy setups optimized with the CRB.

Power-balanced hybrid optics boosted design for achromatic extended depth-of-field imaging via optimized mixed OTF

A power-balanced hybrid optical imaging system has a diffractive computational camera, introduced in this paper, with image formation by a refractive lens and multilevel phase mask (MPM). This system provides a long focal depth with low chromatic aberrations thanks to MPM and a high energy light concentration due to the refractive lens. We introduce the concept of optical power balance between the lens and MPM, which controls the contribution of each element to modulate the incoming light. Additional features of our MPM design are the inclusion of the quantization of the MPM's shape on the number of levels and the Fresnel order (thickness) using a smoothing function. To optimize the optical power balance as well as the MPM, we built a fully differentiable image formation model for joint optimization of optical and imaging parameters for the proposed camera using neural network techniques. We also optimized a single Wiener-like optical transfer function (OTF) invariant to depth to reconstruct a sharp image. We numerically and experimentally compare the designed system with its counterparts, lensless and just-lens optical systems, for the visible wavelength interval (400-700) nm and the depth-of-field range (0.5-∞ m for numerical and 0.5-2 m for experimental). We believe the attained results demonstrate that the proposed system equipped with the optimal OTF overcomes its counterparts--even when they are used with optimized OTF--in terms of the reconstruction quality for off-focus distances. The simulation results also reveal that optimizing the optical power balance, Fresnel order, and the number of levels parameters are essential for system performance attaining an improvement of up to 5 dB of PSNR using the optimized OTF compared to its counterpart lensless setup.

End-to-end sensor and neural network design using differential ray tracing

In this paper we propose a new method to jointly design a sensor and its neural-network based processing. Using a differential ray tracing (DRT) model, we simulate the sensor point-spread function (PSF) and its partial derivative with respect to any of the sensor lens parameters. The proposed ray tracing model makes no thin lens nor paraxial approximation, and is valid for any field of view and point source position. Using the gradient backpropagation framework for neural network optimization, any of the lens parameter can then be jointly optimized along with the neural network parameters. We validate our method for image restoration applications using three proves of concept of focus setting optimization of a given sensor. We provide here interpretations of the joint optical and processing optimization results obtained with the proposed method in these simple cases. Our method paves the way to end-to-end design of a neural network and lens using the complete set of optical parameters within the full sensor field of view.

Influence of high numerical aperture on depth-of-field enhancing phase mask optimization in localization microscopy

The depth-of-field (DoF) of localization microscopes can be extended by placing a phase mask in the aperture stop of the objective. To optimize these masks and characterize their performance, defocus is in general modeled by a simple quadratic pupil phase term. However, this model does not take into account two essential characteristics of localization microscopy setups: an extremely high numerical aperture (NA) and a mismatch between the refractive indices of the immersion liquid and sample. Using the more realistic high NA image formation model of Gibson & Lanni (GL), we show that the DoF extension is simply reduced by a NA-dependent scaling factor. We also show that, provided this scaled DoF extension factor is taken into account, masks optimized with the approximate quadratic model are still nearly optimal in the framework of the GL model. This result is important since it establishes that generic optimized masks can be used in setups with different NA and immersion indices.

Learning Optimal Wavefront Shaping for Multi-Channel Imaging

Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.