Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy

Astigmatism-based active focus stabilisation with universal objective lens compatibility, extended operating range and nanometer precision

Focus stabilisation is vital for long-term fluorescence imaging, particularly in the case of high-resolution imaging techniques. Current stabilisation solutions either rely on fiducial markers that can be perturbative, or on beam reflection monitoring that is limited to high-numerical aperture objective lenses, making multimodal and large-scale imaging challenging. We introduce a beam-based method that relies on astigmatism, which offers advantages in terms of precision and the range over which focus stabilisation is effective. This approach is shown to be compatible with a wide range of objective lenses (10x-100x), typically achieving <10 nm precision with >10 μm operating range. Notably, our technique is largely unaffected by pointing stability errors, which in combination with implementation through a standalone Raspberry Pi architecture, offers a versatile focus stabilisation unit that can be added onto most existing microscope setups.

Robust autofocus method based on patterned active illumination and image cross-correlation analysis

For the effectiveness of a computer-aided diagnosis system, the quality of whole-slide image (WSI) is the foundation, and a useful autofocus method is an important part of ensuring the quality of WSI. The existing autofocus methods need to balance focusing speed and focusing accuracy, and need to be optimized separately for different samples or scenes. In this paper, a robust autofocus method based on fiber bundle illumination and image normalization analysis is proposed. For various application scenes, it meets the requirements of autofocusing through active illumination, such as bright field imaging and fluorescence imaging. For different structures on samples, it ensures the autofocusing accuracy through image analysis. The experimental results imply that the autofocusing method in this paper can effectively track the change of the distance from the sample to the focal plane and significantly improve the WSI quality.

Raman signal optimization based on residual network adaptive focusing

Due to its high sensitivity and specificity, Micro-Raman spectroscopy has emerged as a vital technique for molecular recognition and identification. As a weakly scattered signal, ensuring the accurate focus of the sample is essential for acquiring high quality Raman spectral signal and its analysis, especially in some complex microenvironments such as intracellular settings. Traditional autofocus methods are often time consuming or necessitate additional hardware, limiting real-time sample observation and device compatibility. Here, we propose an adaptive focusing method based on residual network to realize rapid and accurate focusing on Micro-Raman measurements. Using only a bright field image of the sample acquired on any image plane, we can predict the defocus distance with a residual network trained by Resnet50, in which the focus position is determined by combining the gradient and discrete cosine transform. Further, detailed regional division of the bright field map used for characterizing the height variation of actual sample surface is performed. As a result, a focus prediction map with 1μm accuracy is obtained from a bright field image in 120 ms. Based on this method, we successfully realize Raman signal optimization and the necessary correction of spectral information. This adaptive focusing method based on residual network is beneficial to further enhance the sensitivity and accuracy of Micro-Raman spectroscopy technology, which is of great significance in promoting the wide application of Raman spectroscopy.

openFrame: A modular, sustainable, open microscopy platform with single-shot, dual-axis optical autofocus module providing high precision and long range of operation

'openFrame' is a modular, low-cost, open-hardware microscopy platform that can be configured or adapted to most light microscopy techniques and is easily upgradeable or expandable to multiple modalities. The ability to freely mix and interchange both open-source and proprietary hardware components or software enables low-cost, yet research-grade instruments to be assembled and maintained. It also enables rapid prototyping of advanced or novel microscope systems. For long-term time-lapse image data acquisition, slide-scanning or high content analysis, we have developed a novel optical autofocus incorporating orthogonal cylindrical optics to provide robust single-shot closed-loop focus lock, which we have demonstrated to accommodate defocus up to ±37 μm with <200 nm accuracy, and a two-step autofocus mode which we have shown can operate with defocus up to ±68 μm. We have used this to implement automated single molecule localisation microscopy (SMLM) in a relatively low-cost openFrame-based instrument using multimode diode lasers for excitation and cooled CMOS cameras.

A method for obtaining object defocus information in the RC observation mode

The micro-operation robot is widely used in micro-manipulations of biological cells in biological and medical experiments. It plans and controls micro-effector movement based on image feedback information to achieve micro-operations. However, the displacement information of the micro-effector on the x-y plane can be obtained from the image, but not the position information of the micro-effector in the z-axis direction. This makes the micro-effector movement in the z-axis direction discontinuous, which is time-consuming and reduces operational efficiency. In this study, starting from the optical imaging principle of Robert Hoffman modulation contrast method (RC), we propose a defocus detection method for the RC observation mode of an optical microscope. Our method can determine the direction of defocus, which is not available in previous defocusing detection methods. Utilizing this method, we achieve rapid focus for the micro-effector while it is moving along the z-axis direction.

Deep learning in single-molecule imaging and analysis: recent advances and prospects

Single-molecule microscopy is advantageous in characterizing heterogeneous dynamics at the molecular level. However, there are several challenges that currently hinder the wide application of single molecule imaging in bio-chemical studies, including how to perform single-molecule measurements efficiently with minimal run-to-run variations, how to analyze weak single-molecule signals efficiently and accurately without the influence of human bias, and how to extract complete information about dynamics of interest from single-molecule data. As a new class of computer algorithms that simulate the human brain to extract data features, deep learning networks excel in task parallelism and model generalization, and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. In this perspective, we will highlight recent advances in the application of deep learning to single-molecule studies, discuss how deep learning has been used to address the challenges in the field as well as the pitfalls of existing applications, and outline the directions for future development.

Real-time image resolution measurement for single molecule localization microscopy

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.