Deconvolution approach for 3D scanning microscopy with helical phase engineering

Image restoration model for microscopic defocused images based on blurring kernel guidance

Defocus blurring imaging seriously affects the observation accuracy and application range of optical microscopes, and the blurring kernel function is a key parameter for high-resolution image restoration. However, its solving process is complicated and high in computational cost. Image restoration based on most neural networks has high requirements on data sets and the image resolution after restoration is limited because of the lack of quantitative estimation of blurring kernels. In this study, an image restoration method guided by blurring kernel estimation for microscopic defocused images is proposed. First, to reduce the blurring kernel estimation error caused by the positive and negative difference in microscopic defocused imaging, a defocused image classification network is designed to classify the input defocused images with different defocus distances and directions, and its output images are input into the blurring kernel extraction network composed of the feature extraction, correlation, and blurring kernel reconstruction layers. Second, a non-blind defocused image restoration model to restore the high-resolution images is proposed by introducing the blurring kernel extraction module into the restoration network based on U-Net, and the blurring kernel estimation and image restoration losses are jointly trained to realize image restoration guided by blurring kernel estimation. Finally, the experimental results of our proposed method demonstrate significant improvements in both the peak signal-to-noise ratio and structural similarity index measure when compared to other methods.

Three-dimensional modelling of blur property for conventional optical microscopes

Intensity diffusion caused by optical diffraction limits the imaging resolution of conventional optical microscopes, therefore modelling and measuring the intensity transmission and distribution property of the light sources is a significant research topic in system development and pattern recognition. However, the complicated wave propagation process in optical imaging makes it difficult to provide a direct, analytical and simple mathematical model to measure the relationship between the blur degree and various camera parameters. In this study, an improved intensity transmission and distribution calculation method for conventional optical microscopes was proposed; furthermore, a simple mathematical relation between the blur degree and camera parameters was achieved based on the proposed method. First, the light intensity distribution and propagation characteristics of a conventional optical microscope were modeled based on the property of the Fresnel circular hole diffraction combined with the practical optical parameters. Second, by analyzing the property of intensity distribution and blurring imaging, a quantitative simplified mathematical relationship between the blur degree and camera parameters in optical microscope imaging was obtained, and the three-dimensional (3D) blur property in the optical imaging process was analyzed under different conditions. Third, the connection between diffractive optics and geometric optics was obtained by summarizing and generalizing the 3D blur property curve of each monochromatic light source. Finally, the proposed method was verified through a series of simulations and experiments.

Global deblurring for continuous out-of-focus images using a depth-varying diffusion model

The phenomenon of continuous out-of-focus imaging often occurs in high-magnification optical microscopy when observing large-scale targets. Lacking of accurate depth-varying point spread functions (DVPSFs) for blurred regions at different depths, it is difficult to locally reconstruct the clear images of these blurred regions using traditional deblurring methods, making it unreasonable to globally observe the optical features of large-scale targets in high-magnification optical microscopy. This paper proposes a global deblurring method for continuous out-of-focus images of large-scale sphere samples. In this study, first we analyze the energy diffusion characteristics of the optical imaging process, integrating the relationship between high-frequency energy parameters, optical range distance, and depth of field, and we propose a three-dimensional continuous energy diffusion model for optical imaging. Next, we propose an adaptive weight depth calculation method for a continuously changing surface based on the depth varying diffusion model by introducing the sample surface curvature variation and light direction. Finally, we propose a universal method for deblurring continuous out-of-focus images of large-scale sphere samples for the purpose of observing the global optical features in high-magnification optical microscopy. Moreover, we use dynamic microspheres of different sizes to verify the effectiveness of our proposed method. The results prove that our proposed method can accurately calculate the depth of the sample surface and the energy diffusion parameters at each depth, and it can achieve the image deblurring of a continuously changing surface and the global deblurring of multiple samples in a wide field of view.

Hybrid multifocal structured illumination microscopy with enhanced lateral resolution and axial localization capability

Super-resolution (SR) fluorescence microscopy that breaks through the diffraction barrier has drawn great interest in biomedical research. However, obtaining a high precision three-dimensional distribution of the specimen in a short time still remains a challenging task for existing techniques. In this paper, we propose a super-resolution fluorescence microscopy with axial localization capability by combining multifocal structured illumination microscopy with a hybrid detection PSF composed of a Gaussian PSF and a double-helix PSF. A modified reconstruction scheme is presented to accommodate the new hybrid PSF. This method can not only recover the lateral super-resolution image of the specimen but also retain the specimen's depth map within a range of 600 nm with an axial localization precision of 20.8 nm. The performance of this approach is verified by testing fluorescent beads and tubulin in 293-cells. The developed microscope is well suited for observing the precise 3D distribution of thin specimens.

Spectral image scanning microscopy

For decades, the confocal microscope has represented one of the dominant imaging systems in biomedical imaging at sub-cellular lengthscales. Recently, however, it has increasingly been replaced by a related, but more powerful successor technique termed image scanning microscopy (ISM). In this article, we present ISM capable of measuring spectroscopic information such as that contained in fluorescence or Raman images. Compared to established confocal spectroscopic imaging systems, our implementation offers similar spectral resolution, but higher spatial resolution and detection efficiency. Color sensitivity is achieved by a grating placed in the detection path in conjunction with a camera collecting both spatial and spectral information. The multidimensional data is processed using multi-view maximum likelihood image reconstruction. Our findings are supported by numerical simulations and experiments on micro beads and double-stained HeLa cells.

Two-photon PSF-engineered image scanning microscopy

We present two-photon fluorescence image scanning microscopy (ISM) with engineered excitation and detection point-spread-functions enabling 3D imaging in a single 2D scan. This demonstration combines excitation using a holographic multispot array of focused femtosecond pulses with a high-efficiency single-helix PSF phase mask detection. Camera detection along with a multiview reconstruction algorithm allows volumetric imaging of biological samples over a depth of field spanning more than 1500 nm with an axial resolution of better than 400 nm. The nonlinear two-photon process improves sectioning and the inherent longer wavelengths increase the penetration depth in scattering samples. Our method extends the performance of 3D ISM towards thicker biological samples.

Rapid 3D image scanning microscopy with multi-spot excitation and double-helix point spread function detection

Refocusing after Scanning using Helical phase engineering (RESCH) microscopy has previously been demonstrated to provide volumetric information from a single 2D scan. However, the practical application of this method is challenging due to its limited image acquisition speed and spatial resolution. Here, we report on a combination of RESCH and multifocal structured illumination microscopy (MSIM) to improve the image acquisition speed and spatial resolution. A phase mask is introduced to modulate the conventional point spread function (PSF) to the double-helix PSF (DH-PSF), which provides volumetric information, and meanwhile, sparse multifocal illumination patterns are generated by a digital micromirror device (DMD) for parallel 3D subdiffractive imaging information acquisition. We also present a strategy for processing the collected raw data with a Richardson-Lucy deconvolution and pixel reassignment algorithm to improve the spatial resolution of the depth estimation and imaging performance. The proposed 3D image scanning microscopy can record 3D specimen information and the corresponding depth information from a single multi-spot 2D planar scan, which ensures faster data acquisition, larger field of view, and higher spatial resolution than RESCH. Finally, we demonstrate the capability of our system with actual experiments.

Measure and model a 3-D space-variant PSF for fluorescence microscopy image deblurring

Conventional deconvolution methods assume that the microscopy system is spatially invariant, introducing considerable errors. We developed a method to more precisely estimate space-variant point-spread functions from sparse measurements. To this end, a space-variant version of deblurring algorithm was developed and combined with a total-variation regularization. Validation with both simulation and real data showed that our PSF model is more accurate than the piecewise-invariant model and the blending model. Comparing with the orthogonal basis decomposition based PSF model, our proposed model also performed with a considerable improvement. We also evaluated the proposed deblurring algorithm. Our new deblurring algorithm showed a significantly better signal-to-noise ratio and higher image quality than those of the conventional space-invariant algorithm.

High-order-helix point spread functions for monocular three-dimensional imaging with superior aberration robustness

An approach for designing purely refractive optical elements that generate engineered, multi-order-helix point spread functions (PSFs) with large peak separation for passive, optical depth measurement is presented. The influence of aberrations on the PSF's rotation angle, which limits the depth retrieval accuracy, is studied numerically and analytically. It appears that only Zernike modes with an azimuthal index that is an integer multiple of the number of PSF peaks introduce PSF rotation, and hence depth estimation errors. This implies that high-order-helix designs have superior robustness with respect to aberrations. This is experimentally demonstrated by imaging an extended scene in the presence of severe system aberrations using novel, cost-efficient phase elements based on UV-replication on the wafer-scale.

Resolution-enhanced SOFI via structured illumination

By analyzing the statistics of the temporal fluctuations from the blinking emitters, super-resolution fluctuation imaging (SOFI) achieves super-resolution while imposing fewer constraints on the blinking behavior of the probes and is more suitable for low signal-to-noise ratio acquisition than localization methods. However, determined by the square root of cumulation orders, the resolution improvement of SOFI highly restricts its promotion into high-resolution observations. In this Letter, abandoning the default flat illumination in stochastic imaging methods, we introduce structured illumination (SI) (e.g., Gaussian or sinusoidal pattern) into SOFI (SI-SOFI) to render greatly enhanced resolution. Through simulation with parameters of both real acquisition procedures and microscope properties, we examine the feasibility of SI-SOFI and obtain a resolution improvement of four-six folds at just second-order cumulation compared to wide-field imaging. In addition, a practical pathway for the SI-SOFI reconstruction is offered.

Image scanning microscopy: an overview

For almost a century, the resolution of optical microscopy was thought to be limited by Abbé's law describing the diffraction limit of light. At the turn of the millennium, aided by new technologies and fluorophores, the field of optical microscopy finally surpassed the diffraction barrier: a milestone achievement that has been recognized by the 2014 Nobel Prize in Chemistry. Many super-resolution methods rely on the unique photophysical properties of the fluorophores to improve resolution, posing significant limitations on biological imaging, such as multicoloured staining, live-cell imaging and imaging thick specimens. Structured Illumination Microscopy (SIM) is one branch of super-resolution microscopy that requires no such special properties of the applied fluorophores, making it more versatile than other techniques. Since its introduction in biological imaging, SIM has proven to be a popular tool in the biologist's arsenal for following biological interaction and probing structures of nanometre scale. SIM continues to see much advancement in design and implementation, including the development of Image Scanning Microscopy (ISM), which uses patterned excitation via either predefined arrays or raster-scanned single point-spread functions (PSF). This review aims to give a brief overview of the SIM and ISM processes and subsequent developments in the image reconstruction process. Drawing from this, and incorporating more recent achievements in light shaping (i.e. pattern scanning and super-resolution beam shaping), this study also intends to suggest potential future directions for this ever-expanding field.