Deep learning autofluorescence-harmonic microscopy

Data-driven coordinated attention deep learning for high-fidelity brain imaging denoising and inpainting

Deep learning offers promise in enhancing low-quality images by addressing weak fluorescence signals, especially in deep in vivo mouse brain imaging. However, current methods struggle with photon scarcity and noise within in vivo deep mouse brains, and often neglecting tissue preservation. In this study, we propose an innovative in vivo cortical fluorescence image restoration approach, combining signal enhancement, denoising, and inpainting. We curated a deep brain cortical image dataset and developed a novel deep brain coordinate attention restoration network (DeepCAR), integrating coordinate attention with optimized residual networks. Our method swiftly and accurately restores deep cortex images exceeding 800 μm, preserving small-scale tissue structures. It boosts the peak signal-to-noise ratio (PSNR) by 6.94 dB for weak signals and 11.22 dB for large noisy images. Crucially, we validate the effectiveness on external datasets with diverse noise distributions, structural features compared to those in our training data, showcasing real-time high-performance image restoration capabilities.

Noninvasive Nonlinear Optical Computational Histology

Cancer remains a global health challenge, demanding early detection and accurate diagnosis for improved patient outcomes. An intelligent paradigm is introduced that elevates label-free nonlinear optical imaging with contrastive patch-wise learning, yielding stain-free nonlinear optical computational histology (NOCH). NOCH enables swift, precise diagnostic analysis of fresh tissues, reducing patient anxiety and healthcare costs. Nonlinear modalities are evaluated, including stimulated Raman scattering and multiphoton imaging, for their ability to enhance tumor microenvironment sensitivity, pathological analysis, and cancer examination. Quantitative analysis confirmed that NOCH images accurately reproduce nuclear morphometric features across different cancer stages. Key diagnostic features, such as nuclear morphology, size, and nuclear-cytoplasmic contrast, are well preserved. NOCH models also demonstrate promising generalization when applied to other pathological tissues. The study unites label-free nonlinear optical imaging with histopathology using contrastive learning to establish stain-free computational histology. NOCH provides a rapid, non-invasive, and precise approach to surgical pathology, holding immense potential for revolutionizing cancer diagnosis and surgical interventions.

Restoration of metabolic functional metrics from label-free, two-photon human tissue images using multiscale deep-learning-based denoising algorithms

Significance

Label-free, two-photon excited fluorescence (TPEF) imaging captures morphological and functional metabolic tissue changes and enables enhanced understanding of numerous diseases. However, noise and other artifacts present in these images severely complicate the extraction of biologically useful information.

Aim

We aim to employ deep neural architectures in the synthesis of a multiscale denoising algorithm optimized for restoring metrics of metabolic activity from low-signal-to-noise ratio (SNR), TPEF images.

Approach

TPEF images of reduced nicotinamide adenine dinucleotide (phosphate) (NAD(P)H) and flavoproteins (FAD) from freshly excised human cervical tissues are used to assess the impact of various denoising models, preprocessing methods, and data on metrics of image quality and the recovery of six metrics of metabolic function from the images relative to ground truth images.

Results

Optimized recovery of the redox ratio and mitochondrial organization is achieved using a novel algorithm based on deep denoising in the wavelet transform domain. This algorithm also leads to significant improvements in peak-SNR (PSNR) and structural similarity index measure (SSIM) for all images. Interestingly, other models yield even higher PSNR and SSIM improvements, but they are not optimal for recovery of metabolic function metrics.

Conclusions

Denoising algorithms can recover diagnostically useful information from low SNR label-free TPEF images and will be useful for the clinical translation of such imaging.

Dependence of ghost on the incident light angle into dichroic mirror

The dichroic mirror (DM) is a key component in microscope. We found a ghost in the reflection channel of a dual-channel fluorescence microscope and studied the relationship between the ghost and the incidence angle θ into the DM. The DM emission surface reflection generated ghost if the θ is not45 ° . We analyzed the distance and intensity relationship between the ghost and the primary image, which is θ -dependent and was demonstrated by imaging live cells and a stage micrometer. The ghost can be eliminated by placing the DM between objective and tube lens, but not between tube lens and detector, ensuring that the incident light into the DM is approximately parallel. Furthermore, the transmitted light of the DM is shifted towards a longer wavelength with increasing θ . Collectively, microscopists must carefully optimize the θ when designing a microscope to avoid the ghost.

Dual-modality imaging of immunofluorescence and imaging mass cytometry for whole-slide imaging and accurate segmentation

Imaging mass cytometry (IMC) is a powerful technique capable of detecting over 30 markers on a single slide. It has been increasingly used for single-cell-based spatial phenotyping in a wide range of samples. However, it only acquires a rectangle field of view (FOV) with a relatively small size and low image resolution, which hinders downstream analysis. Here, we reported a highly practical dual-modality imaging method that combines high-resolution immunofluorescence (IF) and high-dimensional IMC on the same tissue slide. Our computational pipeline uses the whole-slide image (WSI) of IF as a spatial reference and integrates small-FOV IMC into a WSI of IMC. The high-resolution IF images enable accurate single-cell segmentation to extract robust high-dimensional IMC features for downstream analysis. We applied this method in esophageal adenocarcinoma of different stages, identified the single-cell pathology landscape via reconstruction of WSI IMC images, and demonstrated the advantage of the dual-modality imaging strategy.

A deep learning-based stripe self-correction method for stitched microscopic images

Stitched fluorescence microscope images inevitably exist in various types of stripes or artifacts caused by uncertain factors such as optical devices or specimens, which severely affects the image quality and downstream quantitative analysis. Here, we present a deep learning-based Stripe Self-Correction method, so-called SSCOR. Specifically, we propose a proximity sampling scheme and adversarial reciprocal self-training paradigm that enable SSCOR to utilize stripe-free patches sampled from the stitched microscope image itself to correct their adjacent stripe patches. Comparing to off-the-shelf approaches, SSCOR can not only adaptively correct non-uniform, oblique, and grid stripes, but also remove scanning, bubble, and out-of-focus artifacts, achieving the state-of-the-art performance across different imaging conditions and modalities. Moreover, SSCOR does not require any physical parameter estimation, patch-wise manual annotation, or raw stitched information in the correction process. This provides an intelligent prior-free image restoration solution for microscopists or even microscope companies, thus ensuring more precise biomedical applications for researchers.

Enhancing Total Optical Throughput of Microscopy with Deep Learning for Intravital Observation

The significance of performing large-depth dynamic microscopic imaging in vivo for life science research cannot be overstated. However, the optical throughput of the microscope limits the available information per unit of time, i.e., it is difficult to obtain both high spatial and temporal resolution at once. Here, a method is proposed to construct a kind of intravital microscopy with high optical throughput, by making near-infrared-II (NIR-II, 900-1880 nm) wide-field fluorescence microscopy learn from two-photon fluorescence microscopy based on a scale-recurrent network. Using this upgraded NIR-II fluorescence microscope, vessels in the opaque brain of a rodent are reconstructed three-dimensionally. Five-fold axial and thirteen-fold lateral resolution improvements are achieved without sacrificing temporal resolution and light utilization. Also, tiny cerebral vessel dilatations in early acute respiratory failure mice are observed, with this high optical throughput NIR-II microscope at an imaging speed of 30 fps.

Restoration of metabolic functional metrics from label-free, two-photon cervical tissue images using multiscale deep-learning-based denoising algorithms

Label-free, two-photon imaging captures morphological and functional metabolic tissue changes and enables enhanced understanding of numerous diseases. However, this modality suffers from low signal arising from limitations imposed by the maximum permissible dose of illumination and the need for rapid image acquisition to avoid motion artifacts. Recently, deep learning methods have been developed to facilitate the extraction of quantitative information from such images. Here, we employ deep neural architectures in the synthesis of a multiscale denoising algorithm optimized for restoring metrics of metabolic activity from low-SNR, two-photon images. Two-photon excited fluorescence (TPEF) images of reduced nicotinamide adenine dinucleotide (phosphate) (NAD(P)H) and flavoproteins (FAD) from freshly excised human cervical tissues are used. We assess the impact of the specific denoising model, loss function, data transformation, and training dataset on established metrics of image restoration when comparing denoised single frame images with corresponding six frame averages, considered as the ground truth. We further assess the restoration accuracy of six metrics of metabolic function from the denoised images relative to ground truth images. Using a novel algorithm based on deep denoising in the wavelet transform domain, we demonstrate optimal recovery of metabolic function metrics. Our results highlight the promise of denoising algorithms to recover diagnostically useful information from low SNR label-free two-photon images and their potential importance in the clinical translation of such imaging.

Label-Free Optical Metabolic Imaging in Cells and Tissues

Over the last half century, the autofluorescence of the metabolic cofactors NADH (reduced nicotinamide adenine dinucleotide) and FAD (flavin adenine dinucleotide) has been quantified in a variety of cell types and disease states. With the spread of nonlinear optical microscopy techniques in biomedical research, NADH and FAD imaging has offered an attractive solution to noninvasively monitor cell and tissue status and elucidate dynamic changes in cell or tissue metabolism. Various tools and methods to measure the temporal, spectral, and spatial properties of NADH and FAD autofluorescence have been developed. Specifically, an optical redox ratio of cofactor fluorescence intensities and NADH fluorescence lifetime parameters have been used in numerous applications, but significant work remains to mature this technology for understanding dynamic changes in metabolism. This article describes the current understanding of our optical sensitivity to different metabolic pathways and highlights current challenges in the field. Recent progress in addressing these challenges and acquiring more quantitative information in faster and more metabolically relevant formats is also discussed.

Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning

According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.

Nonlinear microscopy and deep learning classification for mammary gland microenvironment studies

Tumors, their microenvironment, and the mechanisms by which collagen morphology changes throughout cancer progression have recently been a topic of interest. Second harmonic generation (SHG) and polarization second harmonic (P-SHG) microscopy are label-free, hallmark methods that can highlight this alteration in the extracellular matrix (ECM). This article uses automated sample scanning SHG and P-SHG microscopy to investigate ECM deposition associated with tumors residing in the mammary gland. We show two different analysis approaches using the acquired images to distinguish collagen fibrillar orientation changes in the ECM. Lastly, we apply a supervised deep-learning model to classify naïve and tumor-bearing mammary gland SHG images. We benchmark the trained model using transfer learning with the well-known MobileNetV2 architecture. By fine-tuning the different parameters of these models, we show a trained deep-learning model that suits such a small dataset with 73% accuracy.

Detection of pathological response of axillary lymph node metastasis after neoadjuvant chemotherapy in breast cancer using multiphoton microscopy

Neoadjuvant treatment is often considered in breast cancer patients with axillary lymph node involvement, but most of patients do not have a pathologic complete response to therapy. The detection of residual nodal disease has a significant impact on adjuvant therapy recommendations which may improve survival. Here, we investigate whether multiphoton microscopy (MPM) could identify the pathological changes of axillary lymphatic metastasis after neoadjuvant chemotherapy in breast cancer. And furthermore, we find that there are obvious differences in seven collagen morphological features between normal node and residual axillary disease by combining with a semi-automatic image processing method, and also find that there are significant differences in four collagen features between the effective and no-response treatment groups. These research results indicate that MPM may help estimate axillary treatment response in the neoadjuvant setting and thereby tailor more appropriate and personalized adjuvant treatments for breast cancer patients.

Hardware-software co-design of an open-source automatic multimodal whole slide histopathology imaging system

Significance

Advanced digital control of microscopes and programmable data acquisition workflows have become increasingly important for improving the throughput and reproducibility of optical imaging experiments. Combinations of imaging modalities have enabled a more comprehensive understanding of tissue biology and tumor microenvironments in histopathological studies. However, insufficient imaging throughput and complicated workflows still limit the scalability of multimodal histopathology imaging.

Aim

We present a hardware-software co-design of a whole slide scanning system for high-throughput multimodal tissue imaging, including brightfield (BF) and laser scanning microscopy.

Approach

The system can automatically detect regions of interest using deep neural networks in a low-magnification rapid BF scan of the tissue slide and then conduct high-resolution BF scanning and laser scanning imaging on targeted regions with deep learning-based run-time denoising and resolution enhancement. The acquisition workflow is built using Pycro-Manager, a Python package that bridges hardware control libraries of the Java-based open-source microscopy software Micro-Manager in a Python environment.

Results

The system can achieve optimized imaging settings for both modalities with minimized human intervention and speed up the laser scanning by an order of magnitude with run-time image processing.

Conclusions

The system integrates the acquisition pipeline and data analysis pipeline into a single workflow that improves the throughput and reproducibility of multimodal histopathological imaging.

Deep learning-based high-speed, large-field, and high-resolution multiphoton imaging

Multiphoton microscopy is a formidable tool for the pathological analysis of tumors. The physical limitations of imaging systems and the low efficiencies inherent in nonlinear processes have prevented the simultaneous achievement of high imaging speed and high resolution. We demonstrate a self-alignment dual-attention-guided residual-in-residual generative adversarial network trained with various multiphoton images. The network enhances image contrast and spatial resolution, suppresses noise, and scanning fringe artifacts, and eliminates the mutual exclusion between field of view, image quality, and imaging speed. The network may be integrated into commercial microscopes for large-scale, high-resolution, and low photobleaching studies of tumor environments.

Multidimensional quantitative characterization of the tumor microenvironment by multicontrast nonlinear microscopy

Characterization of the microenvironment features of tumors, such as its microstructures, biomolecular metabolism, and functional dynamics, may provide essential pathologic information about the tumor, tumor margin, and adjacent normal tissue for early and intraoperative diagnosis. However, it can be particularly challenging to obtain faithful and comprehensive pathological information simultaneously from unperturbed tissues due to the complexity of the microenvironment in organisms. Super-multiplex nonlinear optical imaging system emerged and matured as an attractive tool for acquisition and elucidation of the nonlinear properties correlated with tumor microenvironment. Here, we introduced a nonlinear effects-based multidimensional optical imaging platform and methodology to simultaneously and efficiently capture contrasting and complementary nonlinear optical signatures of freshly excised human skin tissues. The qualitative and quantitative analysis of autofluorescence (FAD), collagen fiber, and intracellular components (lipids and proteins) illustrated the differences about morphological changes and biomolecular metabolic processes of the epidermis and dermis in different skin carcinogenic types. Interpretation of multi-parameter stain-free histological findings complements conventional H&E-stained slides for investigating basal cell carcinoma and pigmented nevus, validates the platform's versatility and efficiency for classifying subtypes of skin carcinoma, and provides the potential to translate endogenous molecule into biomarker for assisting in rapid cancer screening and diagnosis.

Deep learning augmented microscopy: a faster, wider view, higher resolution autofluorescence-harmonic microscopy

Deep learning enables bypassing the tradeoffs between imaging speed, field of view, and spatial resolution in autofluorescence-harmonic microscopy.