Image scanning microscopy with a quadrant detector

Signal strength and integrated intensity in confocal and image scanning microscopy

The properties of signal strength and integrated intensity in a scanned imaging system are reviewed. These properties are especially applied to confocal imaging systems, including image scanning microscopy. The integrated intensity, equal to the image of a uniform planar (sheet) object, rather than the peak of the point spread function, is a measure of the flux in an image. Analytic expressions are presented for the intensity in the detector plane for a uniform volume object, and for the resulting background. The variation in the integrated intensity with defocus for an offset point detector is presented. This axial fingerprint is independent of any pixel reassignment. The intensity in the detector plane is shown to contain the defocus information, and simple processing of the recorded data can improve optical sectioning and background rejection.

Focus image scanning microscopy for sharp and gentle super-resolved microscopy

To date, the feasibility of super-resolution microscopy for imaging live and thick samples is still limited. Stimulated emission depletion (STED) microscopy requires high-intensity illumination to achieve sub-diffraction resolution, potentially introducing photodamage to live specimens. Moreover, the out-of-focus background may degrade the signal stemming from the focal plane. Here, we propose a new method to mitigate these limitations without drawbacks. First, we enhance a STED microscope with a detector array, enabling image scanning microscopy (ISM). Therefore, we implement STED-ISM, a method that exploits the working principle of ISM to reduce the depletion intensity and achieve a target resolution. Later, we develop Focus-ISM, a strategy to improve the optical sectioning and remove the background of any ISM-based imaging technique, with or without a STED beam. The proposed approach requires minimal architectural changes to a conventional microscope but provides substantial advantages for live and thick sample imaging.

Deep learning enables confocal laser-scanning microscopy with enhanced resolution

Theoretical resolution enhancement of confocal laser-scanning microscopy (CLSM) is sacrificed for the best compromise between optical sectioning and the signal-to-noise ratio (SNR). The pixel reassignment reconstruction algorithm can improve the effective spatial resolution of CLSM to its theoretical limit. However, current implementations are not versatile and are time-consuming or technically complex. Here we present a parameter-free post-processing strategy for laser-scanning microscopy based on deep learning, which enables a spatial resolution enhancement by a factor of ∼1.3, compared to conventional CLSM. To speed up the training process for experimental data, transfer learning, combined with a hybrid dataset consisting of simulated synthetic and experimental images, is employed. The overall resolution and SNR improvement, validated by quantitative evaluation metrics, allowed us to correctly infer the fine structures of real experimental images.

The Development of Microscopy for Super-Resolution: Confocal Microscopy, and Image Scanning Microscopy

Optical methods of super-resolution microscopy, such as confocal microscopy, structured illumination, nonlinear microscopy, and image scanning microscopy are reviewed. These methods avoid strong invasive interaction with a sample, allowing the observation of delicate biological samples. The meaning of resolution and the basic principles and different approaches to superresolution are discussed.

Advances in the study of organelle interactions and their role in neurodegenerative diseases enabled by super-resolution microscopy

From the first illustrations of neuronal morphology by Ramón y Cajal to the recent three-dimensional reconstruction of synaptic connections, the development of modern neuroscience has greatly benefited from breakthroughs in imaging technology. This also applies specifically to the study of neurodegenerative diseases. Much of the research into these diseases relies on the direct visualisation of intracellular structures and their dynamics in degenerating neural cells, which cannot be fully resolved by diffraction-limited microscopes. Progress in the field has therefore been closely linked to the development of super-resolution imaging methods. Their application has greatly advanced our understanding of disease mechanisms, ranging from the structural progression of protein aggregates to defects in organelle morphology. Recent super-resolution studies have specifically implicated the disruption of inter-organelle interactions in multiple neurodegenerative diseases. In this article, we describe some of the key super-resolution techniques that have contributed to this field. We then discuss work to visualise changes in the structure and dynamics of organelles and associated dysfunctions. Finally, we consider what future developments in imaging technology may further our knowledge of these processes.

Pixel reassignment in image scanning microscopy with a doughnut beam: example of maximum likelihood restoration

In image scanning microscopy, the pinhole of a confocal microscope is replaced by a detector array. The point spread function for each detector element can be interpreted as the probability density function of the signal, the peak giving the most likely origin. This thus allows a form of maximum likelihood restoration, and compensation for aberrations, with similarities to adaptive optics. As an example of an aberration, we investigate theoretically and experimentally illumination with a vortex doughnut beam. After reassignment and summation over the detector array, the point spread function is compact, and the resolution and signal level higher than in a conventional microscope.

Structured illumination microscopy and image scanning microscopy: a review and comparison of imaging properties

Structured illumination microscopy and image scanning microscopy are two microscopical tech- niques, rapidly increasing in practical application, that can result in improvement in transverse spatial resolution, and/or improvement in axial imaging performance. The history and principles of these techniques are reviewed, and the imaging properties of the two methods compared. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Image scanning microscopy with a long depth of focus generated by an annular radially polarized beam

Image scanning microscopy (ISM) is a promising tool for bioimaging owing to its integration of signal to noise ratio (SNR) and super resolution superior to that obtained in confocal scanning microscopy. In this paper, we introduce the annular radially polarized beam to the ISM, which yields an axially extended excitation focus and enhanced resolution, providing a new possibility to obtain the whole information of thick specimen with a single scan. We present the basic principle and a rigorous theoretical model for ISM with annular radially polarized beam (ISM-aRP). Results show that the resolution of ISM-aRP can be enhanced by 4% compared with that in conventional ISM, and the axial extent of the focus is longer than 6λ. The projected view of the simulated fluorescent beads suspension specimen demonstrates the validity of ISM-aRP to obtain the whole information of volume sample. Moreover, this simple method can be easily integrated into the commercial laser scanning microscopy systems.

Image scanning microscopy with multiphoton excitation or Bessel beam illumination

Image scanning microscopy is a technique of confocal microscopy in which the confocal pinhole is replaced by a detector array, and the image is reconstructed most straightforwardly by pixel reassignment. In the fluorescence mode, the detector array collects most of the fluorescent light, so the signal-to-noise ratio is much improved compared with confocal microscopy with a small pinhole, while the resolution is improved compared with conventional fluorescence microscopy. Here we consider two cases in which the illumination and detection point spread functions are dissimilar: illumination with a Bessel beam and multiphoton microscopy. It has been shown previously that for Bessel beam illumination in image scanning microscopy with a large array, the imaging performance is degraded. On the other hand, it is also known that the resolution of confocal microscopy is improved by Bessel beam illumination. Here we analyze image scanning microscopy with Bessel beam illumination together with a small array and show that an improvement in transverse resolution (width of the point spread function) by a factor of 1.78 compared with a conventional fluorescence microscope can be obtained. We also examine the behavior of image scanning microscopy in two- or three-photon fluorescence and for two-photon excitation also with Bessel beam illumination. The combination of the optical sectioning effect of image scanning microscopy with multiphoton microscopy reduces background from the sample surface, which can increase penetration depth. For a detector array size of two Airy units, the resolution of two-photon image scanning microscopy is a factor 1.85 better and the peak of the point spread function 2.84 times higher than in nonconfocal two-photon fluorescence. The resolution of three-photon image scanning microscopy is a factor 2.10 better, and the peak of the point spread function is 3.77 times higher than in nonconfocal three-photon fluorescence. The resolution of two-photon image scanning microscopy with Bessel beam illumination is a factor 2.13 better than in standard two-photon fluorescence. Axial resolution and optical sectioning in two-photon or three-photon fluorescence are also improved by using the image scanning modality.

Two-photon image-scanning microscopy with SPAD array and blind image reconstruction

Two-photon excitation (2PE) laser scanning microscopy is the imaging modality of choice when one desires to work with thick biological samples. However, its spatial resolution is poor, below confocal laser scanning microscopy. Here, we propose a straightforward implementation of 2PE image scanning microscopy (2PE-ISM) that, by leveraging our recently introduced single-photon avalanche diode (SPAD) array detector and a novel blind image reconstruction method, is shown to enhance the effective resolution, as well as the overall image quality of 2PE microscopy. With our adaptive pixel reassignment procedure ∼1.6 times resolution increase is maintained deep into thick semi-transparent samples. The integration of Fourier ring correlation based semi-blind deconvolution is shown to further enhance the effective resolution by a factor of ∼2 - and automatic background correction is shown to boost the image quality especially in noisy images. Most importantly, our 2PE-ISM implementation requires no calibration measurements or other input from the user, which is an important aspect in terms of day-to-day usability of the technique.

Two-photon image-scanning microscopy with SPAD array and blind image reconstruction

Two-photon excitation (2PE) laser scanning microscopy is the imaging modality of choice when one desires to work with thick biological samples. However, its spatial resolution is poor, below confocal laser scanning microscopy. Here, we propose a straightforward implementation of 2PE image scanning microscopy (2PE-ISM) that, by leveraging our recently introduced single-photon avalanche diode (SPAD) array detector and a novel blind image reconstruction method, is shown to enhance the effective resolution, as well as the overall image quality of 2PE microscopy. With our adaptive pixel reassignment procedure ∼1.6 times resolution increase is maintained deep into thick semi-transparent samples. The integration of Fourier ring correlation based semi-blind deconvolution is shown to further enhance the effective resolution by a factor of ∼2 - and automatic background correction is shown to boost the image quality especially in noisy images. Most importantly, our 2PE-ISM implementation requires no calibration measurements or other input from the user, which is an important aspect in terms of day-to-day usability of the technique.

Analyzing the super-resolution characteristics of focused-spot illumination approaches

SIGNIFICANCE

It is commonly assumed that using the objective lens to create a tightly focused light spot for illumination provides a twofold resolution improvement over the Rayleigh resolution limit and that resolution improvement is independent of object properties. Nevertheless, such an assumption has not been carefully examined. We examine this assumption by analyzing the performance of two super-resolution methods, known as image scanning microscopy (ISM) and illumination-enhanced sparsity (IES).

AIM

We aim to identify the fundamental differences between the two methods, and to provide examples that help researchers determine which method to utilize for different imaging conditions.

APPROACH

We input the same image datasets into the two methods and analyze their restorations. In numerical simulations, we design objects of distinct brightness and sparsity levels for imaging. We use biological imaging experiments to verify the simulation results.

RESULTS

The resolution of IES often exceeds twice the Rayleigh resolution limit when imaging sparse objects. A decrease in object sparsity negatively affects the resolution improvement in both methods.

CONCLUSIONS

The IES method is superior for imaging sparse objects with its main features being bright and small against a dark, large background. For objects that are largely bright with small dark features, the ISM method is favorable.

Pixel reassignment in image scanning microscopy: a re-evaluation

Image scanning microscopy is a technique based on confocal microscopy, in which the confocal pinhole is replaced by a detector array, and the resulting image is reconstructed, usually by the process of pixel reassignment. The detector array collects most of the fluorescent light, so the signal-to-noise ratio is much improved compared with confocal microscopy with a small pinhole, while the resolution is improved compared with conventional (wide-field) microscopy. In previous studies, it has usually been assumed that pixels should be reassigned by a constant factor, to a point midway between the illumination and detection spots. Here it is shown that the peak intensity of the effective point spread function (PSF) can be further increased by 4% by a new choice of the pixel reassignment factor. For an array of two Airy units, the peak of the effective PSF is 1.90 times that of a conventional microscope, and the transverse resolution is 1.53 times better. It is confirmed that image scanning microscopy gives optical sectioning strength identical to that of a confocal microscope with a pinhole equal to the size of the detector array. However, it is shown that image scanning microscopy exhibits axial resolution superior to a confocal microscope with a pinhole the same size as the detector array. For a two-Airy-unit array, the axial resolution is 1.34 times better than in a conventional microscope for the standard reassignment factor, and 1.28 times better for the new reassignment factor. The axial resolution of a confocal microscope with a two-Airy-unit pinhole is only 1.04 times better than conventional microscopy. We also examine the signal-to-noise ratio of a point object in a uniform background (called the detectability), and show that it is 1.6 times higher than in a confocal microscope.

Quantum correlation measurement with single photon avalanche diode arrays

Temporal photon correlation measurement, instrumental to probing the quantum properties of light, typically requires multiple single photon detectors. Progress in single photon avalanche diode (SPAD) array technology highlights their potential as high-performance detector arrays for quantum imaging and photon number-resolving (PNR) experiments. Here, we demonstrate this potential by incorporating a novel on-chip SPAD array with 42% peak photon detection efficiency, low dark count rate and crosstalk probability of 0.14% per detection in a confocal microscope. This enables reliable measurements of second and third order photon correlations from a single quantum dot emitter. Our analysis overcomes the inter-detector optical crosstalk background even though it is over an order of magnitude larger than our faint signal. To showcase the vast application space of such an approach, we implement a recently introduced super-resolution imaging method, quantum image scanning microscopy (Q-ISM).

Improved algorithm for expanding the measurement linear range of a four-quadrant detector

A four-quadrant detector is a kind of photoelectric detector that can quickly and accurately measure the incident angle of light. However, its ability to measure in a large field of view (FOV) is limited by its hardware structure and its calculation principle. To solve these problems, this paper proposes an improved algorithm that can extend the measurement linear range without reducing its measurement accuracy. After that, through simulation and experiment, we compare it with many other location algorithms, including the most widely used classical algorithm and the logarithmic algorithm suitable for large FOVs. Finally, the following conclusions can be drawn from both theoretical data and experimental data: the improved algorithm can significantly improve the measurement accuracy over 50% in the same FOV condition, and the measurable range can be expanded over 25% in the same accuracy requirement. At the same time, the robustness of noise does not decrease; when the root mean square error of the classical algorithm fluctuates at 0.1° in different SNR conditions, the improved algorithm is also 0.1°, while the logarithmic algorithm can reach 1.7°, and other algorithms are around 0.25°. In addition, the improved algorithm is more stable in measuring a certain direction and can effectively avoid the influence from the offset of incident light in another axis.

A robust and versatile platform for image scanning microscopy enabling super-resolution FLIM

Image scanning microscopy (ISM) can improve the effective spatial resolution of confocal microscopy to its theoretical limit. However, current implementations are not robust or versatile, and are incompatible with fluorescence lifetime imaging (FLIM). We describe an implementation of ISM based on a single-photon detector array that enables super-resolution FLIM and improves multicolor, live-cell and in-depth imaging, thereby paving the way for a massive transition from confocal microscopy to ISM.

Versatile multi-detector scheme for adaptive optics scanning laser ophthalmoscopy

Adaptive optics scanning laser ophthalmoscopy (AOSLO) is a powerful tool for imaging the retina at high spatial and temporal resolution. In this paper, we present a multi-detector scheme for AOSLO which has two main configurations: pixel reassignment and offset aperture imaging. In this detection scheme, the single element detector of the standard AOSLO is replaced by a fiber bundle which couples the detected light into multiple detectors. The pixel reassignment configuration enables high resolution imaging with an increased light collection. The increase in signal-to-noise ratio (SNR) from this configuration can improve the accuracy of motion registration techniques. The offset aperture imaging configuration enhances the detection of multiply scattered light, which improves the contrast of retinal vasculature and inner retinal layers similar to methods such as nonconfocal split-detector imaging and multi-offset aperture imaging.

Pixellated circle

For applications in optical systems it is often necessary to represent a circular aperture in a pixellated form. An objective parameter is introduced that is a measure of how well an approximate circle can be generated from a small array of square pixels. Both filled circles (disks) and rings are considered. Arrays with a width given by an even number of pixels can also be used to generate quadrants of a circle. Rings with outer and inner profiles given by optimum circles or quadrants can be summed to fill a complete circle or quadrant.

Quadrant response model and error analysis of four-quadrant detectors related to the non-uniform spot and blind area

In the system of tracking and detection based on the four-quadrant detector (4-QD), the energy distribution of the incident spot and the blind area of the photosensitive surface will affect the location accuracy. The current model of the spot is based on the ideal circular Gauss spot, which makes the error caused by the spot shape easily ignored. In this paper, the model of the spot energy distribution is improved, which can adapt to the elliptical Gauss distribution. The width of the blind area is also added to the response models of the detector so that the output of each quadrant and the error of the localization algorithm can be calculated more accurately. The simulation results show that the measurement accuracy of 4-QD decreases with the increase of the blind area, the shape, and the inclination of the light spot. In the experiment, we first verify the correctness and practicability of the improved model of the spot energy distribution, and then the improved model is proved to be able to make the response and error calculation more accurate.

Dynamic range extension for photon counting arrays

Confocal microscopes use photomultiplier tubes and hybrid detectors due to their large dynamic range, which typically exceeds the one of single-photon avalanche diodes (SPADs). The latter, due to their photon counting operation, are usually limited to an output count rate to 1/Tdead. In this paper, we present a thorough analysis, which can actually be applied to any photon counting detector, on how to extend the SPAD dynamic range by exploiting the nonlinear photon response at high count rates and for different recharge mechanisms. We applied passive, active event-driven and clock-driven (i.e. clocked, following quanta image sensor response) recharge directly to the SPADs. The photon response, photon count standard deviation, signal-to-noise ratio and dynamic range were measured and compared to models. Measurements were performed with a CMOS SPAD array targeted for image scanning microscopy, featuring best-in-class 11 V excess bias, 55% peak photon detection probability at 520 nm and >40% from 440 to 640 nm. The array features an extremely low median dark count rate below 0.05 cps/μm² at 9 V of excess bias and 0°C. We show that active event-driven recharge provides ×75 dynamic range extension and offers novel ways for high dynamic range imaging. When compared to the clock-driven recharge and the quanta image sensor approach, the dynamic range is extended by a factor of ×12.7-26.4. Additionally, for the first time, we evaluate the influence of clock-driven recharge on the SPAD afterpulsing.

Response of quadrant detectors to structured beams via convolution integrals

We propose a new expression for the response of a quadrant detector using convolution integrals. This expression, exploiting the properties of the convolution, is easier to evaluate by hand. Computationally, it is also practicable to use since a large number of computer programs can evaluate convolutions right away. We use the new expression to obtain an analytical form of the response of a quadrant detector for a Gaussian beam and for Hermite-Gaussian beams in general. We compare this analytic expression for the response for the Gaussian beam with the approximations from previous studies and with the response obtained through simulations. From the response, we also obtained an analytical form for the sensitivity of the quadrant detector to a Gaussian beam. Lastly, we demonstrate the computational ease of using our new expression for the response by calculating the sensitivity of the quadrant detector to the Bessel beam.

Image scanning fluorescence emission difference microscopy based on a detector array

We propose a novel imaging method that enables the enhancement of three-dimensional resolution of confocal microscopy significantly and achieve experimentally a new fluorescence emission difference method for the first time, based on the parallel detection with a detector array. Following the principles of photon reassignment in image scanning microscopy, images captured by the detector array were arranged. And by selecting appropriate reassign patterns, the imaging result with enhanced resolution can be achieved with the method of fluorescence emission difference. Two specific methods are proposed in this paper, showing that the difference between an image scanning microscopy image and a confocal image will achieve an improvement of transverse resolution by approximately 43% compared with that in confocal microscopy, and the axial resolution can also be enhanced by at least 22% experimentally and 35% theoretically. Moreover, the methods presented in this paper can improve the lateral resolution by around 10% than fluorescence emission difference and 15% than Airyscan. The mechanism of our methods is verified by numerical simulations and experimental results, and it has significant potential in biomedical applications.