Deep-learning-based motion-correction algorithm in optical resolution photoacoustic microscopy

Unsupervised deep learning enables real-time image registration of fast-scanning optical-resolution photoacoustic microscopy

A fast scanner of optical-resolution photoacoustic microscopy is inherently vulnerable to perturbation, leading to severe image distortion and significant misalignment among multiple 2D or 3D images. Restoration and registration of these images is critical for accurately quantifying dynamic information in long-term imaging. However, traditional registration algorithms face a great challenge in computational throughput. Here, we develop an unsupervised deep learning based registration network to achieve real-time image restoration and registration. This method can correct artifacts from B-scan distortion and remove misalignment among adjacent and repetitive images in real time. Compared with conventional intensity based registration algorithms, the throughput of the developed algorithm is improved by 50 times. After training, the new deep learning method performs better than conventional feature based image registration algorithms. The results show that the proposed method can accurately restore and register the images of fast-scanning photoacoustic microscopy in real time, offering a powerful tool to extract dynamic vascular structural and functional information.

Implicit neural representations in light microscopy

Three-dimensional stacks acquired with confocal or two-photon microscopy are crucial for studying neuroanatomy. However, high-resolution image stacks acquired at multiple depths are time-consuming and susceptible to photobleaching. In vivo microscopy is further prone to motion artifacts. In this work, we suggest that deep neural networks with sine activation functions encoding implicit neural representations (SIRENs) are suitable for predicting intermediate planes and correcting motion artifacts, addressing the aforementioned shortcomings. We show that we can accurately estimate intermediate planes across multiple micrometers and fully automatically and unsupervised estimate a motion-corrected denoised picture. We show that noise statistics can be affected by SIRENs, however, rescued by a downstream denoising neural network, shown exemplarily with the recovery of dendritic spines. We believe that the application of these technologies will facilitate more efficient acquisition and superior post-processing in the future.

Noise-insensitive defocused signal and resolution enhancement for optical-resolution photoacoustic microscopy via deep learning

Optical-resolution photoacoustic microscopy suffers from narrow depth of field and a significant deterioration in defocused signal intensity and spatial resolution. Here, a method based on deep learning was proposed to enhance the defocused resolution and signal-to-noise ratio. A virtual optical-resolution photoacoustic microscopy based on k-wave was used to obtain the datasets of deep learning with different noise levels. A fully dense U-Net was trained with randomly distributed sources to improve the quality of photoacoustic images. The results show that the PSNR of defocused signal was enhanced by more than 1.2 times. An over 2.6-fold enhancement in lateral resolution and an over 3.4-fold enhancement in axial resolution of defocused regions were achieved. The large volumetric and high-resolution imaging of blood vessels further verified that the proposed method can effectively overcome the deterioration of the signal and the spatial resolution due to the narrow depth of field of optical-resolution photoacoustic microscopy.

Niche preclinical and clinical applications of photoacoustic imaging with endogenous contrast

In the past decade, photoacoustic (PA) imaging has attracted a great deal of popularity as an emergent diagnostic technology owing to its successful demonstration in both preclinical and clinical arenas by various academic and industrial research groups. Such steady growth of PA imaging can mainly be attributed to its salient features, including being non-ionizing, cost-effective, easily deployable, and having sufficient axial, lateral, and temporal resolutions for resolving various tissue characteristics and assessing the therapeutic efficacy. In addition, PA imaging can easily be integrated with the ultrasound imaging systems, the combination of which confers the ability to co-register and cross-reference various features in the structural, functional, and molecular imaging regimes. PA imaging relies on either an endogenous source of contrast (e.g., hemoglobin) or those of an exogenous nature such as nano-sized tunable optical absorbers or dyes that may boost imaging contrast beyond that provided by the endogenous sources. In this review, we discuss the applications of PA imaging with endogenous contrast as they pertain to clinically relevant niches, including tissue characterization, cancer diagnostics/therapies (termed as theranostics), cardiovascular applications, and surgical applications. We believe that PA imaging's role as a facile indicator of several disease-relevant states will continue to expand and evolve as it is adopted by an increasing number of research laboratories and clinics worldwide.

Deep learning-assisted frequency-domain photoacoustic microscopy

Frequency-domain photoacoustic microscopy (FD-PAM) constitutes a powerful cost-efficient imaging method integrating intensity-modulated laser beams for the excitation of single-frequency photoacoustic waves. Nevertheless, FD-PAM provides an extremely small signal-to-noise ratio (SNR), which can be up to two orders of magnitude lower than the conventional time-domain (TD) systems. To overcome this inherent SNR limitation of FD-PAM, we utilize a U-Net neural network aiming at image augmentation without the need for excessive averaging or the application of high optical power. In this context, we improve the accessibility of PAM as the system's cost is dramatically reduced, and we expand its applicability to demanding observations while retaining sufficiently high image quality standards.

A Deep Learning Method for Motion Artifact Correction in Intravascular Photoacoustic Image Sequence

In vivo application of intravascular photoacoustic (IVPA) imaging for coronary arteries is hampered by motion artifacts associated with the cardiac cycle. Gating is a common strategy to mitigate motion artifacts. However, a large amount of diagnostically valuable information might be lost due to one frame per cycle. In this work, we present a deep learning-based method for directly correcting motion artifacts in non-gated IVPA pullback sequences. The raw signal frames are classified into dynamic and static frames by clustering. Then, a neural network named Motion Artifact Correction (MAC)-Net is designed to correct motion in dynamic frames. Given the lack of the ground truth information on the underlying dynamics of coronary arteries, we trained and tested the network using a computer-generated dataset. Based on the results, it has been observed that the trained network can directly correct motion in successive frames while preserving the original structures without discarding any frames. The improvement in the visual effect of the longitudinal view has been demonstrated based on quantitative evaluation of the inter-frame dissimilarity. The comparison results validated the motion-suppression ability of our method comparable to gating and image registration-based non-learning methods, while maintaining the integrity of the pullbacks without image preprocessing. Experimental results from in vivo intravascular ultrasound and optical coherence tomography pullbacks validated the feasibility of our method in the in vivo intracoronary imaging scenario.

Deep Learning-Powered Bessel-Beam Multiparametric Photoacoustic Microscopy

Enabling simultaneous and high-resolution quantification of the total concentration of hemoglobin ( [Formula: see text]), oxygen saturation of hemoglobin (sO2), and cerebral blood flow (CBF), multi-parametric photoacoustic microscopy (PAM) has emerged as a promising tool for functional and metabolic imaging of the live mouse brain. However, due to the limited depth of focus imposed by the Gaussian-beam excitation, the quantitative measurements become inaccurate when the imaging object is out of focus. To address this problem, we have developed a hardware-software combined approach by integrating Bessel-beam excitation and conditional generative adversarial network (cGAN)-based deep learning. Side-by-side comparison of the new cGAN-powered Bessel-beam multi-parametric PAM against the conventional Gaussian-beam multi-parametric PAM shows that the new system enables high-resolution, quantitative imaging of [Formula: see text], sO2, and CBF over a depth range of [Formula: see text] in the live mouse brain, with errors 13-58 times lower than those of the conventional system. Better fulfilling the rigid requirement of light focusing for accurate hemodynamic measurements, the deep learning-powered Bessel-beam multi-parametric PAM may find applications in large-field functional recording across the uneven brain surface and beyond (e.g., tumor imaging).

Photoacoustic imaging for microcirculation

Microcirculation facilitates the blood-tissue exchange of nutrients and regulates blood perfusion. It is, therefore, essential in maintaining tissue health. Aberrations in microcirculation are potentially indicative of underlying cardiovascular and metabolic pathologies. Thus, quantitative information about it is of great clinical relevance. Photoacoustic imaging (PAI) is a capable technique that relies on the generation of imaging contrast via the absorption of light and can image at micron-scale resolution. PAI is especially desirable to map microvasculature as hemoglobin strongly absorbs light and can generate a photoacoustic signal. This paper reviews the current state of the art for imaging microvascular networks using photoacoustic imaging. We further describe how quantitative information about blood dynamics such as the total hemoglobin concentration, oxygen saturation, and blood flow rate is obtained using PAI. We also discuss its importance in understanding key pathophysiological processes in neurovascular, cardiovascular, ophthalmic, and cancer research fields. We then discuss the current challenges and limitations of PAI and the approaches that can help overcome these limitations. Finally, we provide the reader with an overview of future trends in the field of PAI for imaging microcirculation.

Deep-Learning-Based Algorithm for the Removal of Electromagnetic Interference Noise in Photoacoustic Endoscopic Image Processing

Despite all the expectations for photoacoustic endoscopy (PAE), there are still several technical issues that must be resolved before the technique can be successfully translated into clinics. Among these, electromagnetic interference (EMI) noise, in addition to the limited signal-to-noise ratio (SNR), have hindered the rapid development of related technologies. Unlike endoscopic ultrasound, in which the SNR can be increased by simply applying a higher pulsing voltage, there is a fundamental limitation in leveraging the SNR of PAE signals because they are mostly determined by the optical pulse energy applied, which must be within the safety limits. Moreover, a typical PAE hardware situation requires a wide separation between the ultrasonic sensor and the amplifier, meaning that it is not easy to build an ideal PAE system that would be unaffected by EMI noise. With the intention of expediting the progress of related research, in this study, we investigated the feasibility of deep-learning-based EMI noise removal involved in PAE image processing. In particular, we selected four fully convolutional neural network architectures, U-Net, Segnet, FCN-16s, and FCN-8s, and observed that a modified U-Net architecture outperformed the other architectures in the EMI noise removal. Classical filter methods were also compared to confirm the superiority of the deep-learning-based approach. Still, it was by the U-Net architecture that we were able to successfully produce a denoised 3D vasculature map that could even depict the mesh-like capillary networks distributed in the wall of a rat colorectum. As the development of a low-cost laser diode or LED-based photoacoustic tomography (PAT) system is now emerging as one of the important topics in PAT, we expect that the presented AI strategy for the removal of EMI noise could be broadly applicable to many areas of PAT, in which the ability to apply a hardware-based prevention method is limited and thus EMI noise appears more prominently due to poor SNR.

Virtual optical-resolution photoacoustic microscopy using the k-Wave method

Deep learning has been widely used in image processing, quantitative analysis, and other applications in optical-resolution photoacoustic microscopy (OR-PAM). It requires a large amount of photoacoustic data for training and testing. However, due to the complex structure, high cost, slow imaging speed, and other factors of OR-PAM, it is difficult to obtain enough data required by deep learning, which limits the research of deep learning in OR-PAM to a certain extent. To solve this problem, a virtual OR-PAM based on k-Wave is proposed. The virtual photoacoustic microscopy mainly includes the setting of excitation light source and ultrasonic probe, scanning and signal processing, which can realize the common Gaussian-beam and Bessel-beam OR-PAMs. The system performance (lateral resolution, axial resolution, and depth of field) was tested by imaging a vertically tilted fiber, and the effectiveness and feasibility of the virtual simulation platform were verified by 3D imaging of the virtual vascular network. The ability to the generation of the dataset for deep learning was also verified. The construction of the virtual OR-PAM can promote the research of OR-PAM and the application of deep learning in OR-PAM.

Suppression of motion artifacts in intravascular photoacoustic image sequences

Intravascular photoacoustic (IVPA) imaging is an image-based imaging modality for the assessment of atherosclerotic plaques. Successful application of IVPA for in vivo coronary arterial imaging requires one overcomes the challenge of motion artifacts associated with the cardiac cycle. We propose a method for correcting artifacts owing to cardiac motion, which are observed in sequential IVPA images acquired by the continuous pullback of the imaging catheter. This method groups raw photoacoustic signals into subsets corresponding to similar phases in the cardiac cycles. Thereafter, the sequential images are reconstructed, by representing the initial pressure distribution on the vascular cross-sections based on the clustered frames of signals by time reversal. Results of simulation data demonstrate the efficacy of this method in suppressing motion artifacts. Qualitative and quantitative evaluations of the method indicate an enhancement of the image quality. Comparison results reveal that this method is computationally efficient in motion correction compared with the image-based gating.

Low-consumption photoacoustic method to measure liquid viscosity

Viscosity measurement is important in many areas of biomedicine and industry. Traditional viscometers are usually time-consuming and require huge sample volumes. Microfluidic viscometry may overcome the challenge of large sample consumption but suffers from a long process time and a complicated structure design and interaction. Here, we present a photoacoustic method that measures the liquid viscosity in a simple microfluidic-based tube. This new viscosity measurement method embraces fast detection speed and low fluid consumption, offering a new tool for efficient and convenient liquid viscosity measurement in a broad range of applications.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Sounding out the hidden data: A concise review of deep learning in photoacoustic imaging

The rapidly evolving field of photoacoustic tomography utilizes endogenous chromophores to extract both functional and structural information from deep within tissues. It is this power to perform precise quantitative measurements in vivo-with endogenous or exogenous contrast-that makes photoacoustic tomography highly promising for clinical translation in functional brain imaging, early cancer detection, real-time surgical guidance, and the visualization of dynamic drug responses. Considering photoacoustic tomography has benefited from numerous engineering innovations, it is of no surprise that many of photoacoustic tomography's current cutting-edge developments incorporate advances from the equally novel field of artificial intelligence. More specifically, alongside the growth and prevalence of graphical processing unit capabilities within recent years has emerged an offshoot of artificial intelligence known as deep learning. Rooted in the solid foundation of signal processing, deep learning typically utilizes a method of optimization known as gradient descent to minimize a loss function and update model parameters. There are already a number of innovative efforts in photoacoustic tomography utilizing deep learning techniques for a variety of purposes, including resolution enhancement, reconstruction artifact removal, undersampling correction, and improved quantification. Most of these efforts have proven to be highly promising in addressing long-standing technical obstacles where traditional solutions either completely fail or make only incremental progress. This concise review focuses on the history of applied artificial intelligence in photoacoustic tomography, presents recent advances at this multifaceted intersection of fields, and outlines the most exciting advances that will likely propagate into promising future innovations.

Deep learning for biomedical photoacoustic imaging: A review

Photoacoustic imaging (PAI) is a promising emerging imaging modality that enables spatially resolved imaging of optical tissue properties up to several centimeters deep in tissue, creating the potential for numerous exciting clinical applications. However, extraction of relevant tissue parameters from the raw data requires the solving of inverse image reconstruction problems, which have proven extremely difficult to solve. The application of deep learning methods has recently exploded in popularity, leading to impressive successes in the context of medical imaging and also finding first use in the field of PAI. Deep learning methods possess unique advantages that can facilitate the clinical translation of PAI, such as extremely fast computation times and the fact that they can be adapted to any given problem. In this review, we examine the current state of the art regarding deep learning in PAI and identify potential directions of research that will help to reach the goal of clinical applicability.

[Progress of motion artifact correction in photoacoustic microscopy and photoacoustic tomography]

Photoacoustic imaging (PAI) is a rapidly developing hybrid biomedical imaging technology, which is capable of providing structural and functional information of biological tissues. Due to inevitable motion of the imaging object, such as respiration, heartbeat or eye rotation, motion artifacts are observed in the reconstructed images, which reduce the imaging resolution and increase the difficulty of obtaining high-quality images. This paper summarizes current methods for correcting and compensating motion artifacts in photoacoustic microscopy (PAM) and photoacoustic tomography (PAT), discusses their advantages and limits and forecasts possible future work.

Deep learning in photoacoustic imaging: a review

SIGNIFICANCE

Photoacoustic (PA) imaging can provide structural, functional, and molecular information for preclinical and clinical studies. For PA imaging (PAI), non-ideal signal detection deteriorates image quality, and quantitative PAI (QPAI) remains challenging due to the unknown light fluence spectra in deep tissue. In recent years, deep learning (DL) has shown outstanding performance when implemented in PAI, with applications in image reconstruction, quantification, and understanding.

AIM

We provide (i) a comprehensive overview of the DL techniques that have been applied in PAI, (ii) references for designing DL models for various PAI tasks, and (iii) a summary of the future challenges and opportunities.

APPROACH

Papers published before November 2020 in the area of applying DL in PAI were reviewed. We categorized them into three types: image understanding, reconstruction of the initial pressure distribution, and QPAI.

RESULTS

When applied in PAI, DL can effectively process images, improve reconstruction quality, fuse information, and assist quantitative analysis.

CONCLUSION

DL has become a powerful tool in PAI. With the development of DL theory and technology, it will continue to boost the performance and facilitate the clinical translation of PAI.

Review of deep learning for photoacoustic imaging

Machine learning has been developed dramatically and witnessed a lot of applications in various fields over the past few years. This boom originated in 2009, when a new model emerged, that is, the deep artificial neural network, which began to surpass other established mature models on some important benchmarks. Later, it was widely used in academia and industry. Ranging from image analysis to natural language processing, it fully exerted its magic and now become the state-of-the-art machine learning models. Deep neural networks have great potential in medical imaging technology, medical data analysis, medical diagnosis and other healthcare issues, and is promoted in both pre-clinical and even clinical stages. In this review, we performed an overview of some new developments and challenges in the application of machine learning to medical image analysis, with a special focus on deep learning in photoacoustic imaging. The aim of this review is threefold: (i) introducing deep learning with some important basics, (ii) reviewing recent works that apply deep learning in the entire ecological chain of photoacoustic imaging, from image reconstruction to disease diagnosis, (iii) providing some open source materials and other resources for researchers interested in applying deep learning to photoacoustic imaging.

Reconstructing Undersampled Photoacoustic Microscopy Images Using Deep Learning

One primary technical challenge in photoacoustic microscopy (PAM) is the necessary compromise between spatial resolution and imaging speed. In this study, we propose a novel application of deep learning principles to reconstruct undersampled PAM images and transcend the trade-off between spatial resolution and imaging speed. We compared various convolutional neural network (CNN) architectures, and selected a Fully Dense U-net (FD U-net) model that produced the best results. To mimic various undersampling conditions in practice, we artificially downsampled fully-sampled PAM images of mouse brain vasculature at different ratios. This allowed us to not only definitively establish the ground truth, but also train and test our deep learning model at various imaging conditions. Our results and numerical analysis have collectively demonstrated the robust performance of our model to reconstruct PAM images with as few as 2% of the original pixels, which can effectively shorten the imaging time without substantially sacrificing the image quality.

Another decade of photoacoustic imaging

Photoacoustic imaging - a hybrid biomedical imaging modality finding its way to clinical practices. Although the photoacoustic phenomenon was known more than a century back, only in the last two decades it has been widely researched and used for biomedical imaging applications. In this review we focus on the development and progress of the technology in the last decade (2010-2020). From becoming more and more user friendly, cheaper in cost, portable in size, photoacoustic imaging promises a wide range of applications, if translated to clinic. The growth of photoacoustic community is steady, and with several new directions researchers are exploring, it is inevitable that photoacoustic imaging will one day establish itself as a regular imaging system in the clinical practices.

Convolutional neural network for resolution enhancement and noise reduction in acoustic resolution photoacoustic microscopy

In acoustic resolution photoacoustic microscopy (AR-PAM), a high numerical aperture focused ultrasound transducer (UST) is used for deep tissue high resolution photoacoustic imaging. There is a significant degradation of lateral resolution in the out-of-focus region. Improvement in out-of-focus resolution without degrading the image quality remains a challenge. In this work, we propose a deep learning-based method to improve the resolution of AR-PAM images, especially at the out of focus plane. A modified fully dense U-Net based architecture was trained on simulated AR-PAM images. Applying the trained model on experimental images showed that the variation in resolution is ∼10% across the entire imaging depth (∼4 mm) in the deep learning-based method, compared to ∼180% variation in the original PAM images. Performance of the trained network on in vivo rat vasculature imaging further validated that noise-free, high resolution images can be obtained using this method.