Characterizing and correcting camera noise in back-illuminated sCMOS cameras

Noise correction in differential phase contrast for improving phase sensitivity

Differential phase contrast (DPC) imaging relies on computational analysis to extract quantitative phase information from phase gradient images. However, even modest noise level can introduce errors that propagate through the computational process, degrading the quality of the final phase result and further reducing phase sensitivity. Here, we introduce the noise-corrected DPC (ncDPC) to enhance phase sensitivity. This approach is based on a theoretical DPC model that effectively considers most relevant noise sources in the camera and non-uniform illumination in DPC. In particular, the dominating shot noise and readout noise variance can be jointly estimated using frequency analysis and further corrected by block-matching 3D (BM3D) method. Finally, the denoised images are used for phase retrieval based on the common Tikhonov inversion. Our results, based on both simulated and experimental data, demonstrate that ncDPC outperforms the traditional DPC (tDPC), enabling significant improvements in both phase reconstruction quality and phase sensitivity. Besides, we have demonstrated the broad applicability of ncDPC by showing its performance in various experimental datasets.

Enhancing Weak-Signal Extraction for Single-Molecule Localization Microscopy

Single-molecule localization microscopy (SMLM) has been widely used in biological imaging due to its ultrahigh spatial resolution. However, due to the strategy of reducing photodamage to living cells, the fluorescence signals of emitters are usually weak and the detector noises become non-negligible, which leads to localization misalignments and signal losses, thus deteriorating the imaging capability of SMLM. Here, we propose an active modulation method to control the fluorescence of the probe emitters. It actually marks the emitters with artificial blinking character, which directly distinguishes weak signals from multiple detector noises. We demonstrated from simulations and experiments that this method improves the signal-to-noise ratio by about 10 dB over the non-modulated method and boosts the sensitivity of single-molecule localization down to -4 dB, which significantly reduces localization misalignments and signal losses in SMLM. This signal-noise decoupling strategy is generally applicable to the super-resolution system with versatile labeled probes to improve their imaging capability. We also showed its application to the densely labeled sample, showing its flexibility in super-resolution nanoscopy.

Deep learning-based high-speed, large-field, and high-resolution multiphoton imaging

Multiphoton microscopy is a formidable tool for the pathological analysis of tumors. The physical limitations of imaging systems and the low efficiencies inherent in nonlinear processes have prevented the simultaneous achievement of high imaging speed and high resolution. We demonstrate a self-alignment dual-attention-guided residual-in-residual generative adversarial network trained with various multiphoton images. The network enhances image contrast and spatial resolution, suppresses noise, and scanning fringe artifacts, and eliminates the mutual exclusion between field of view, image quality, and imaging speed. The network may be integrated into commercial microscopes for large-scale, high-resolution, and low photobleaching studies of tumor environments.

Uniformity Correction of CMOS Image Sensor Modules for Machine Vision Cameras

Flat-field correction (FFC) is commonly used in image signal processing (ISP) to improve the uniformity of image sensor pixels. Image sensor nonuniformity and lens system characteristics have been known to be temperature-dependent. Some machine vision applications, such as visual odometry and single-pixel airborne object tracking, are extremely sensitive to pixel-to-pixel sensitivity variations. Numerous cameras, especially in the fields of infrared imaging and staring cameras, use multiple calibration images to correct for nonuniformities. This paper characterizes the temperature and analog gain dependence of the dark signal nonuniformity (DSNU) and photoresponse nonuniformity (PRNU) of two contemporary global shutter CMOS image sensors for machine vision applications. An optimized hardware architecture is proposed to compensate for nonuniformities, with optional parametric lens shading correction (LSC). Three different performance configurations are outlined for different application areas, costs, and power requirements. For most commercial applications, the correction of LSC suffices. For both DSNU and PRNU, compensation with one or multiple calibration images, captured at different gain and temperature settings are considered. For more demanding applications, the effectiveness, external memory bandwidth, power consumption, implementation, and calibration complexity, as well as the camera manufacturability of different nonuniformity correction approaches were compared.

Pushing the colorimetry camera-based fluorescence microscopy to low light imaging by denoising and dye combination

Colorimetry camera-based fluorescence microscopy (CCFM) is a single-frame imaging method for observing multiple biological events simultaneously. Compared with the traditional multi-color fluorescence microscopy methods based on sequential excitation or spectral splitting, the CCFM method simplifies multi-color fluorescence imaging experiments, while keeping a high spatial resolution. However, when the level of the detected fluorescence signal decreases, the image quality, the demosaicking algorithm precision, and the discrimination of fluorescence channels on the colorimetry camera will also decrease. Thus, CCFM has a poor color resolution under a low signal level. For example, the crosstalk will be higher than 10% when the signal is less than 100 photons/pixel. To solve this problem, we developed a new algorithm that combines sCMOS noise correction with demosaicking, and a dye selection method based on the spectral response characteristics of the colorimetry camera. By combining the above two strategies, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and low light single-frame four-color fluorescence imaging was successfully performed on fixed cos-7 cells. This study expands the power of the CCFM method, and provides a simple and efficient way for various bioimaging applications in low-light conditions.

Photon-free (s)CMOS camera characterization for artifact reduction in high- and super-resolution microscopy

Modern implementations of widefield fluorescence microscopy often rely on sCMOS cameras, but this camera architecture inherently features pixel-to-pixel variations. Such variations lead to image artifacts and render quantitative image interpretation difficult. Although a variety of algorithmic corrections exists, they require a thorough characterization of the camera, which typically is not easy to access or perform. Here, we developed a fully automated pipeline for camera characterization based solely on thermally generated signal, and implemented it in the popular open-source software Micro-Manager and ImageJ/Fiji. Besides supplying the conventional camera maps of noise, offset and gain, our pipeline also gives access to dark current and thermal noise as functions of the exposure time. This allowed us to avoid structural bias in single-molecule localization microscopy (SMLM), which without correction is substantial even for scientific-grade, cooled cameras. In addition, our approach enables high-quality 3D super-resolution as well as live-cell time-lapse microscopy with cheap, industry-grade cameras. As our approach for camera characterization does not require any user interventions or additional hardware implementations, numerous correction algorithms that rely on camera characterization become easily applicable.

sCMOS Noise-Corrected Superresolution Reconstruction Algorithm for Structured Illumination Microscopy

Structured illumination microscopy (SIM) is widely applied due to its high temporal and spatial resolution imaging ability. sCMOS cameras are often used in SIM due to their superior sensitivity, resolution, field of view, and frame rates. However, the unique single-pixel-dependent readout noise of sCMOS cameras may lead to SIM reconstruction artefacts and affect the accuracy of subsequent statistical analysis. We first established a nonuniform sCMOS noise model to address this issue, which incorporates the single-pixel-dependent offset, gain, and variance based on the SIM imaging process. The simulation indicates that the sCMOS pixel-dependent readout noise causes artefacts in the reconstructed SIM superresolution (SR) image. Thus, we propose a novel sCMOS noise-corrected SIM reconstruction algorithm derived from the imaging model, which can effectively suppress the sCMOS noise-related reconstruction artefacts and improve the signal-to-noise ratio (SNR).

Raw Data to Results: A Hands-On Introduction and Overview of Computational Analysis for Single-Molecule Localization Microscopy

Single-molecule localization microscopy (SMLM) is an advanced microscopy method that uses the blinking of fluorescent molecules to determine the position of these molecules with a resolution below the diffraction limit (∼5-40 nm). While SMLM imaging itself is becoming more popular, the computational analysis surrounding the technique is still a specialized area and often remains a "black box" for experimental researchers. Here, we provide an introduction to the required computational analysis of SMLM imaging, post-processing and typical data analysis. Importantly, user-friendly, ready-to-use and well-documented code in Python and MATLAB with exemplary data is provided as an interactive experience for the reader, as well as a starting point for further analysis. Our code is supplemented by descriptions of the computational problems and their implementation. We discuss the state of the art in computational methods and software suites used in SMLM imaging and data analysis. Finally, we give an outlook into further computational challenges in the field.

Diffraction-Limited Molecular Cluster Quantification with Bayesian Nonparametrics

Life's fundamental processes involve multiple molecules operating in close proximity within cells. To probe the composition and kinetics of molecular clusters confined within small (diffraction-limited) regions, experiments often report on the total fluorescence intensity simultaneously emitted from labeled molecules confined to such regions. Methods exist to enumerate total fluorophore numbers (e.g., step counting by photobleaching). However, methods aimed at step counting by photobleaching cannot treat photophysical dynamics in counting nor learn their associated kinetic rates. Here we propose a method to simultaneously enumerate fluorophores and determine their individual photophysical state trajectories. As the number of active (fluorescent) molecules at any given time is unknown, we rely on Bayesian nonparametrics and use specialized Monte Carlo algorithms to derive our estimates. Our formulation is benchmarked on synthetic and real data sets. While our focus here is on photophysical dynamics (in which labels transition between active and inactive states), such dynamics can also serve as a proxy for other types of dynamics such as assembly and disassembly kinetics of clusters. Similarly, while we focus on the case where all labels are initially fluorescent, other regimes, more appropriate to photoactivated localization microscopy, where fluorophores are instantiated in a non-fluorescent state, fall within the scope of the framework. As such, we provide a complete and versatile framework for the interpretation of complex time traces arising from the simultaneous activity of up to 100 fluorophores.

Contemporary Whole Slide Imaging Devices and Their Applications within the Modern Pathology Department: A Selected Hardware Review

Digital pathology (DP) has disrupted the practice of traditional pathology, including applications in education, research, and clinical practice. Contemporary whole slide imaging (WSI) devices include technological advances that help address some of the challenges facing modern pathology, such as increasing workloads with fewer subspecialized pathologists, expanding integrated delivery networks with global reach, and greater customization when working up cases for precision medicine. This review focuses on integral hardware components of 43 market available and soon-to-be released digital WSI devices utilized throughout the world. Components such as objective lens type and magnification, scanning camera, illumination, and slide capacity were evaluated with respect to scan time, throughput, accuracy of scanning, and image quality. This analysis of assorted modern WSI devices offers essential, valuable information for successfully selecting and implementing a digital WSI solution for any given pathology practice.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.