Deep-Learning Image Reconstruction for Real-Time Photoacoustic System

Joint segmentation and image reconstruction with error prediction in photoacoustic imaging using deep learning

Deep learning has been used to improve photoacoustic (PA) image reconstruction. One major challenge is that errors cannot be quantified to validate predictions when ground truth is unknown. Validation is key to quantitative applications, especially using limited-bandwidth ultrasonic linear detector arrays. Here, we propose a hybrid Bayesian convolutional neural network (Hybrid-BCNN) to jointly predict PA image and segmentation with error (uncertainty) predictions. Each output pixel represents a probability distribution where error can be quantified. The Hybrid-BCNN was trained with simulated PA data and applied to both simulations and experiments. Due to the sparsity of PA images, segmentation focuses Hybrid-BCNN on minimizing the loss function in regions with PA signals for better predictions. The results show that accurate PA segmentations and images are obtained, and error predictions are highly statistically correlated to actual errors. To leverage error predictions, confidence processing created PA images above a specific confidence level.

Masked cross-domain self-supervised deep learning framework for photoacoustic computed tomography reconstruction

Accurate image reconstruction is crucial for photoacoustic (PA) computed tomography (PACT). Recently, deep learning has been used to reconstruct PA images with a supervised scheme, which requires high-quality images as ground truth labels. However, practical implementations encounter inevitable trade-offs between cost and performance due to the expensive nature of employing additional channels for accessing more measurements. Here, we propose a masked cross-domain self-supervised (CDSS) reconstruction strategy to overcome the lack of ground truth labels from limited PA measurements. We implement the self-supervised reconstruction in a model-based form. Simultaneously, we take advantage of self-supervision to enforce the consistency of measurements and images across three partitions of the measured PA data, achieved by randomly masking different channels. Our findings indicate that dynamically masking a substantial proportion of channels, such as 80%, yields meaningful self-supervisors in both the image and signal domains. Consequently, this approach reduces the multiplicity of pseudo solutions and enables efficient image reconstruction using fewer PA measurements, ultimately minimizing reconstruction error. Experimental results on in-vivo PACT dataset of mice demonstrate the potential of our self-supervised framework. Moreover, our method exhibits impressive performance, achieving a structural similarity index (SSIM) of 0.87 in an extreme sparse case utilizing only 13 channels, which outperforms the performance of the supervised scheme with 16 channels (0.77 SSIM). Adding to its advantages, our method can be deployed on different trainable models in an end-to-end manner, further enhancing its versatility and applicability.

Removing Artifacts in Transcranial Photoacoustic Imaging With Polarized Self-Attention Dense-UNet

OBJECTIVE

Photoacoustic imaging (PAI) is a promising transcranial imaging technique. However, the distortion of photoacoustic signals induced by the skull significantly influences its imaging quality. We aimed to use deep learning for removing artifacts in PAI.

METHODS

In this study, we propose a polarized self-attention dense U-Net, termed PSAD-UNet, to correct the distortion and accurately recover imaged objects beneath bone plates. To evaluate the performance of the proposed method, a series of experiments was performed using a custom-built PAI system.

RESULTS

The experimental results showed that the proposed PSAD-UNet method could effectively implement transcranial PAI through a one- or two-layer bone plate. Compared with the conventional delay-and-sum and classical U-Net methods, PSAD-UNet can diminish the influence of bone plates and provide high-quality PAI results in terms of structural similarity and peak signal-to-noise ratio. The 3-D experimental results further confirm the feasibility of PSAD-UNet in 3-D transcranial imaging.

CONCLUSION

PSAD-UNet paves the way for implementing transcranial PAI with high imaging accuracy, which reveals broad application prospects in preclinical and clinical fields.

Unsupervised disentanglement strategy for mitigating artifact in photoacoustic tomography under extremely sparse view

Traditional methods under sparse view for reconstruction of photoacoustic tomography (PAT) often result in significant artifacts. Here, a novel image to image transformation method based on unsupervised learning artifact disentanglement network (ADN), named PAT-ADN, was proposed to address the issue. This network is equipped with specialized encoders and decoders that are responsible for encoding and decoding the artifacts and content components of unpaired images, respectively. The performance of the proposed PAT-ADN was evaluated using circular phantom data and the animal in vivo experimental data. The results demonstrate that PAT-ADN exhibits excellent performance in effectively removing artifacts. In particular, under extremely sparse view (e.g., 16 projections), structural similarity index and peak signal-to-noise ratio are improved by ∼188 % and ∼85 % in in vivo experimental data using the proposed method compared to traditional reconstruction methods. PAT-ADN improves the imaging performance of PAT, opening up possibilities for its application in multiple domains.

Compensating unknown speed of sound in learned fast 3D limited-view photoacoustic tomography

Real-time applications in three-dimensional photoacoustic tomography from planar sensors rely on fast reconstruction algorithms that assume the speed of sound (SoS) in the tissue is homogeneous. Moreover, the reconstruction quality depends on the correct choice for the constant SoS. In this study, we discuss the possibility of ameliorating the problem of unknown or heterogeneous SoS distributions by using learned reconstruction methods. This can be done by modelling the uncertainties in the training data. In addition, a correction term can be included in the learned reconstruction method. We investigate the influence of both and while a learned correction component can improve reconstruction quality further, we show that a careful choice of uncertainties in the training data is the primary factor to overcome unknown SoS. We support our findings with simulated and in vivo measurements in 3D.

Dual-modal Photoacoustic and Ultrasound Imaging: from preclinical to clinical applications

Photoacoustic imaging is a novel biomedical imaging modality that has emerged over the recent decades. Due to the conversion of optical energy into the acoustic wave, photoacoustic imaging offers high-resolution imaging in depth beyond the optical diffusion limit. Photoacoustic imaging is frequently used in conjunction with ultrasound as a hybrid modality. The combination enables the acquisition of both optical and acoustic contrasts of tissue, providing functional, structural, molecular, and vascular information within the same field of view. In this review, we first described the principles of various photoacoustic and ultrasound imaging techniques and then classified the dual-modal imaging systems based on their preclinical and clinical imaging applications. The advantages of dual-modal imaging were thoroughly analyzed. Finally, the review ends with a critical discussion of existing developments and a look toward the future.

Deep learning framework for three-dimensional surface reconstruction of object of interest in photoacoustic tomography

Photoacoustic tomography (PAT) is a non-ionizing hybrid imaging technology of clinical importance that combines the high contrast of optical imaging with the high penetration of ultrasonic imaging. Two-dimensional (2D) tomographic images can only provide the cross-sectional structure of the imaging target rather than its overall spatial morphology. This work proposes a deep learning framework for reconstructing three-dimensional (3D) surface of an object of interest from a series of 2D images. It achieves end-to-end mapping from a series of 2D images to a 3D image, visually displaying the overall morphology of the object. The framework consists of four modules: segmentation module, point cloud generation module, point cloud completion module, and mesh conversion module, which respectively implement the tasks of segmenting a region of interest, generating a sparse point cloud, completing sparse point cloud and reconstructing 3D surface. The network model is trained on simulation data sets and verified on simulation, phantom, and in vivo data sets. The results showed superior 3D reconstruction performance both visually and on the basis of quantitative evaluation metrics compared to the state-of-the-art non-learning and learning approaches. This method potentially enables high-precision 3D surface reconstruction from the tomographic images output by the preclinical PAT system without changing the imaging system. It provides a general deep learning scheme for 3D reconstruction from tomographic scanning data.

Review of Deep Learning Approaches for Interleaved Photoacoustic and Ultrasound (PAUS) Imaging

Photoacoustic (PA) imaging provides optical contrast at relatively large depths within the human body, compared to other optical methods, at ultrasound (US) spatial resolution. By integrating real-time PA and US (PAUS) modalities, PAUS imaging has the potential to become a routine clinical modality bringing the molecular sensitivity of optics to medical US imaging. For applications where the full capabilities of clinical US scanners must be maintained in PAUS, conventional limited view and bandwidth transducers must be used. This approach, however, cannot provide high-quality maps of PA sources, especially vascular structures. Deep learning (DL) using data-driven modeling with minimal human design has been very effective in medical imaging, medical data analysis, and disease diagnosis, and has the potential to overcome many of the technical limitations of current PAUS imaging systems. The primary purpose of this article is to summarize the background and current status of DL applications in PAUS imaging. It also looks beyond current approaches to identify remaining challenges and opportunities for robust translation of PAUS technologies to the clinic.

Illuminating the Brain: Advances and Perspectives in Optoelectronics for Neural Activity Monitoring and Modulation

Optogenetic modulation of brain neural activity that combines optical and electrical modes in a unitary neural system has recently gained robust momentum. Controlling illumination spatial coverage, designing light-activated modulators, and developing wireless light delivery and data transmission are crucial for maximizing the use of optical neuromodulation. To this end, biocompatible electrodes with enhanced optoelectrical performance, device integration for multiplexed addressing, wireless transmission, and multimodal operation in soft systems have been developed. This review provides an outlook for uniformly illuminating large brain areas while spatiotemporally imaging the neural responses upon optoelectrical stimulation with little artifacts. Representative concepts and important breakthroughs, such as head-mounted illumination, multiple implanted optical fibers, and micro-light-delivery devices, are discussed. Examples of techniques that incorporate electrophysiological monitoring and optoelectrical stimulation are presented. Challenges and perspectives are posed for further research efforts toward high-density optoelectrical neural interface modulation, with the potential for nonpharmacological neurological disease treatments and wireless optoelectrical stimulation.

Sparse-view reconstruction for photoacoustic tomography combining diffusion model with model-based iteration

As a non-invasive hybrid biomedical imaging technology, photoacoustic tomography combines high contrast of optical imaging and high penetration of acoustic imaging. However, the conventional standard reconstruction under sparse view could result in low-quality image in photoacoustic tomography. Here, a novel model-based sparse reconstruction method for photoacoustic tomography via diffusion model was proposed. A score-based diffusion model is designed for learning the prior information of the data distribution. The learned prior information is utilized as a constraint for the data consistency term of an optimization problem based on the least-square method in the model-based iterative reconstruction, aiming to achieve the optimal solution. Blood vessels simulation data and the animal in vivo experimental data were used to evaluate the performance of the proposed method. The results demonstrate that the proposed method achieves higher-quality sparse reconstruction compared with conventional reconstruction methods and U-Net. In particular, under the extreme sparse projection (e.g., 32 projections), the proposed method achieves an improvement of ∼ 260 % in structural similarity and ∼ 30 % in peak signal-to-noise ratio for in vivo data, compared with the conventional delay-and-sum method. This method has the potential to reduce the acquisition time and cost of photoacoustic tomography, which will further expand the application range.

Approximating the uncertainty of deep learning reconstruction predictions in single-pixel imaging

Single-pixel imaging (SPI) has the advantages of high-speed acquisition over a broad wavelength range and system compactness. Deep learning (DL) is a powerful tool that can achieve higher image quality than conventional reconstruction approaches. Here, we propose a Bayesian convolutional neural network (BCNN) to approximate the uncertainty of the DL predictions in SPI. Each pixel in the predicted image represents a probability distribution rather than an image intensity value, indicating the uncertainty of the prediction. We show that the BCNN uncertainty predictions are correlated to the reconstruction errors. When the BCNN is trained and used in practical applications where the ground truths are unknown, the level of the predicted uncertainty can help to determine whether system, data, or network adjustments are needed. Overall, the proposed BCNN can provide a reliable tool to indicate the confidence levels of DL predictions as well as the quality of the model and dataset for many applications of SPI.

Niche preclinical and clinical applications of photoacoustic imaging with endogenous contrast

In the past decade, photoacoustic (PA) imaging has attracted a great deal of popularity as an emergent diagnostic technology owing to its successful demonstration in both preclinical and clinical arenas by various academic and industrial research groups. Such steady growth of PA imaging can mainly be attributed to its salient features, including being non-ionizing, cost-effective, easily deployable, and having sufficient axial, lateral, and temporal resolutions for resolving various tissue characteristics and assessing the therapeutic efficacy. In addition, PA imaging can easily be integrated with the ultrasound imaging systems, the combination of which confers the ability to co-register and cross-reference various features in the structural, functional, and molecular imaging regimes. PA imaging relies on either an endogenous source of contrast (e.g., hemoglobin) or those of an exogenous nature such as nano-sized tunable optical absorbers or dyes that may boost imaging contrast beyond that provided by the endogenous sources. In this review, we discuss the applications of PA imaging with endogenous contrast as they pertain to clinically relevant niches, including tissue characterization, cancer diagnostics/therapies (termed as theranostics), cardiovascular applications, and surgical applications. We believe that PA imaging's role as a facile indicator of several disease-relevant states will continue to expand and evolve as it is adopted by an increasing number of research laboratories and clinics worldwide.

Review on ultrasound-guided photoacoustic imaging for complementary analyses of biological systems in vivo

Photoacoustic imaging has been developed as a new biomedical molecular imaging modality. Due to its similarity to conventional ultrasound imaging in terms of signal detection and image generation, dual-modal photoacoustic and ultrasound imaging has been applied to visualize physiological and morphological information in biological systems in vivo. By complementing each other, dual-modal photoacoustic and ultrasound imaging showed synergistic advances in photoacoustic imaging with the guidance of ultrasound images. In this review, we introduce our recent progresses in dual-modal photoacoustic and ultrasound imaging systems at various scales of study, from preclinical small animals to clinical humans. A summary of the works reveals various strategies for combining the structural information of ultrasound images with the molecular information of photoacoustic images.

Laser diode photoacoustic point source detection: machine learning-based denoising and reconstruction

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

Adaptive enhancement of acoustic resolution photoacoustic microscopy imaging via deep CNN prior

Acoustic resolution photoacoustic microscopy (AR-PAM) is a promising medical imaging modality that can be employed for deep bio-tissue imaging. However, its relatively low imaging resolution has greatly hindered its wide applications. Previous model-based or learning-based PAM enhancement algorithms either require design of complex handcrafted prior to achieve good performance or lack the interpretability and flexibility that can adapt to different degradation models. However, the degradation model of AR-PAM imaging is subject to both imaging depth and center frequency of ultrasound transducer, which varies in different imaging conditions and cannot be handled by a single neural network model. To address this limitation, an algorithm integrating both learning-based and model-based method is proposed here so that a single framework can deal with various distortion functions adaptively. The vasculature image statistics is implicitly learned by a deep convolutional neural network, which served as plug and play (PnP) prior. The trained network can be directly plugged into the model-based optimization framework for iterative AR-PAM image enhancement, which fitted for different degradation mechanisms. Based on physical model, the point spread function (PSF) kernels for various AR-PAM imaging situations are derived and used for the enhancement of simulation and in vivo AR-PAM images, which collectively proved the effectiveness of proposed method. Quantitatively, the PSNR and SSIM values have all achieve best performance with the proposed algorithm in all three simulation scenarios; The SNR and CNR values have also significantly raised from 6.34 and 5.79 to 35.37 and 29.66 respectively in an in vivo testing result with the proposed algorithm.

A jointed feature fusion framework for photoacoustic image reconstruction

The standard reconstruction of Photoacoustic (PA) computed tomography (PACT) image could cause the artifacts due to interferences or ill-posed setup. Recently, deep learning has been used to reconstruct the PA image with ill-posed conditions. In this paper, we propose a jointed feature fusion framework (JEFF-Net) based on deep learning to reconstruct the PA image using limited-view data. The cross-domain features from limited-view position-wise data and the reconstructed image are fused by a backtracked supervision. A quarter position-wise data (32 channels) is fed into model, which outputs another 3-quarters-view data (96 channels). Moreover, two novel losses are designed to restrain the artifacts by sufficiently manipulating superposed data. The experimental results have demonstrated the superior performance and quantitative evaluations show that our proposed method outperformed the ground-truth in some metrics by 135% (SSIM for simulation) and 40% (gCNR for in-vivo) improvement.

Fast iterative reconstruction for photoacoustic tomography using learned physical model: Theoretical validation

Iterative reconstruction has demonstrated superior performance in medical imaging under compressed, sparse, and limited-view sensing scenarios. However, iterative reconstruction algorithms are slow to converge and rely heavily on hand-crafted parameters to achieve good performance. Many iterations are usually required to reconstruct a high-quality image, which is computationally expensive due to repeated evaluations of the physical model. While learned iterative reconstruction approaches such as model-based learning (MBLr) can reduce the number of iterations through convolutional neural networks, it still requires repeated evaluations of the physical models at each iteration. Therefore, the goal of this study is to develop a Fast Iterative Reconstruction (FIRe) algorithm that incorporates a learned physical model into the learned iterative reconstruction scheme to further reduce the reconstruction time while maintaining robust reconstruction performance. We also propose an efficient training scheme for FIRe, which releases the enormous memory footprint required by learned iterative reconstruction methods through the concept of recursive training. The results of our proposed method demonstrate comparable reconstruction performance to learned iterative reconstruction methods with a 9x reduction in computation time and a 620x reduction in computation time compared to variational reconstruction.

Deep and Domain Transfer Learning Aided Photoacoustic Microscopy: Acoustic Resolution to Optical Resolution

Acoustic resolution photoacoustic micros- copy (AR-PAM) can achieve deeper imaging depth in biological tissue, with the sacrifice of imaging resolution compared with optical resolution photoacoustic microscopy (OR-PAM). Here we aim to enhance the AR-PAM image quality towards OR-PAM image, which specifically includes the enhancement of imaging resolution, restoration of micro-vasculatures, and reduction of artifacts. To address this issue, a network (MultiResU-Net) is first trained as generative model with simulated AR-OR image pairs, which are synthesized with physical transducer model. Moderate enhancement results can already be obtained when applying this model to in vivo AR imaging data. Nevertheless, the perceptual quality is unsatisfactory due to domain shift. Further, domain transfer learning technique under generative adversarial network (GAN) framework is proposed to drive the enhanced image's manifold towards that of real OR image. In this way, perceptually convincing AR to OR enhancement result is obtained, which can also be supported by quantitative analysis. Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) values are significantly increased from 14.74 dB to 19.01 dB and from 0.1974 to 0.2937, respectively, validating the improvement of reconstruction correctness and overall perceptual quality. The proposed algorithm has also been validated across different imaging depths with experiments conducted in both shallow and deep tissue. The above AR to OR domain transfer learning with GAN (AODTL-GAN) framework has enabled the enhancement target with limited amount of matched in vivo AR-OR imaging data.

Deep Learning Enhances Multiparametric Dynamic Volumetric Photoacoustic Computed Tomography In Vivo (DL-PACT)

Photoacoustic computed tomography (PACT) has become a premier preclinical and clinical imaging modality. Although PACT's image quality can be dramatically improved with a large number of ultrasound (US) transducer elements and associated multiplexed data acquisition systems, the associated high system cost and/or slow temporal resolution are significant problems. Here, a deep learning-based approach is demonstrated that qualitatively and quantitively diminishes the limited-view artifacts that reduce image quality and improves the slow temporal resolution. This deep learning-enhanced multiparametric dynamic volumetric PACT approach, called DL-PACT, requires only a clustered subset of many US transducer elements on the conventional multiparametric PACT. Using DL-PACT, high-quality static structural and dynamic contrast-enhanced whole-body images as well as dynamic functional brain images of live animals and humans are successfully acquired, all in a relatively fast and cost-effective manner. It is believed that the strategy can significantly advance the use of PACT technology for preclinical and clinical applications such as neurology, cardiology, pharmacology, endocrinology, and oncology.

Image Quality Improvement Techniques and Assessment Adequacy in Clinical Optoacoustic Imaging: A Systematic Review

Optoacoustic imaging relies on the detection of optically induced acoustic waves to offer new possibilities in morphological and functional imaging. As the modality matures towards clinical application, research efforts aim to address multifactorial limitations that negatively impact the resulting image quality. In an endeavor to obtain a clear view on the limitations and their effects, as well as the status of this progressive refinement process, we conduct an extensive search for optoacoustic image quality improvement approaches that have been evaluated with humans in vivo, thus focusing on clinically relevant outcomes. We query six databases (PubMed, Scopus, Web of Science, IEEE Xplore, ACM Digital Library, and Google Scholar) for articles published from 1 January 2010 to 31 October 2021, and identify 45 relevant research works through a systematic screening process. We review the identified approaches, describing their primary objectives, targeted limitations, and key technical implementation details. Moreover, considering comprehensive and objective quality assessment as an essential prerequisite for the adoption of such approaches in clinical practice, we subject 36 of the 45 papers to a further in-depth analysis of the reported quality evaluation procedures, and elicit a set of criteria with the intent to capture key evaluation aspects. Through a comparative criteria-wise rating process, we seek research efforts that exhibit excellence in quality assessment of their proposed methods, and discuss features that distinguish them from works with similar objectives. Additionally, informed by the rating results, we highlight areas with improvement potential, and extract recommendations for designing quality assessment pipelines capable of providing rich evidence.

A Review of Optical Ultrasound Imaging Modalities for Intravascular Imaging

Recent advances in medical imaging include integrating photoacoustic and optoacoustic techniques with conventional imaging modalities. The developments in the latter have led to the use of optics combined with the conventional ultrasound technique for imaging intravascular tissues and applied to different areas of the human body. Conventional ultrasound is a skin contact-based method used for imaging. It does not expose patients to harmful radiation compared to other techniques such as Computerised Tomography (CT) and Magnetic Resonance Imaging (MRI) scans. On the other hand, optical Ultrasound (OpUS) provides a new way of viewing internal organs of the human body by using skin and an eye-safe laser range. OpUS is mostly used for binary measurements since they do not require to be resolved at a much higher resolution but can be used to check for intravascular imaging. Various signal processing techniques and reconstruction methodologies exist for Photo-Acoustic Imaging, and their applicability in bioimaging is explored in this paper.

Deep learning methods hold promise for light fluence compensation in three-dimensional optoacoustic imaging

SIGNIFICANCE

Quantitative optoacoustic imaging (QOAI) continues to be a challenge due to the influence of nonlinear optical fluence distribution, which distorts the optoacoustic image representation. Nonlinear optical fluence correction in OA imaging is highly ill-posed, leading to the inaccurate recovery of optical absorption maps. This work aims to recover the optical absorption maps using deep learning (DL) approach by correcting for the fluence effect.

AIM

Different DL models were compared and investigated to enable optical absorption coefficient recovery at a particular wavelength in a nonhomogeneous foreground and background medium.

APPROACH

Data-driven models were trained with two-dimensional (2D) Blood vessel and three-dimensional (3D) numerical breast phantom with highly heterogeneous/realistic structures to correct for the nonlinear optical fluence distribution. The trained DL models such as U-Net, Fully Dense (FD) U-Net, Y-Net, FD Y-Net, Deep residual U-Net (Deep ResU-Net), and generative adversarial network (GAN) were tested to evaluate the performance of optical absorption coefficient recovery (or fluence compensation) with in-silico and in-vivo datasets.

RESULTS

The results indicated that FD U-Net-based deconvolution improves by about 10% over reconstructed optoacoustic images in terms of peak-signal-to-noise ratio. Further, it was observed that DL models can indeed highlight deep-seated structures with higher contrast due to fluence compensation. Importantly, the DL models were found to be about 17 times faster than solving diffusion equation for fluence correction.

CONCLUSIONS

The DL methods were able to compensate for nonlinear optical fluence distribution more effectively and improve the optoacoustic image quality.

Approaching closed spherical, full-view detection for photoacoustic tomography

SIGNIFICANCE

Photoacoustic tomography (PAT) is a widely explored imaging modality and has excellent potential for clinical applications. On the acoustic detection side, limited-view angle and limited-bandwidth are common key issues in PAT systems that result in unwanted artifacts. While analytical and simulation studies of limited-view artifacts are fairly extensive, experimental setups capable of comparing limited-view to an ideal full-view case are lacking.

AIMS

A custom ring-shaped detector array was assembled and mounted to a 6-axis robot, then rotated and translated to achieve up to 3.8π steradian view angle coverage of an imaged object.

APPROACH

Minimization of negativity artifacts and phantom imaging were used to optimize the system, followed by demonstrative imaging of a star contrast phantom, a synthetic breast tumor specimen phantom, and a vascular phantom.

RESULTS

Optimization of the angular/rotation scans found ≈212 effective detectors were needed for high-quality images, while 15-mm steps were used to increase the field of view as required depending on the size of the imaged object. Example phantoms were clearly imaged with all discerning features visible and minimal artifacts.

CONCLUSIONS

A near full-view closed spherical system has been developed, paving the way for future work demonstrating experimentally the significant advantages of using a full-view PAT setup.

Deep Learning-Based Photoacoustic Imaging of Vascular Network Through Thick Porous Media

Photoacoustic imaging is a promising approach used to realize in vivo transcranial cerebral vascular imaging. However, the strong attenuation and distortion of the photoacoustic wave caused by the thick porous skull greatly affect the imaging quality. In this study, we developed a convolutional neural network based on U-Net to extract the effective photoacoustic information hidden in the speckle patterns obtained from vascular network images datasets under porous media. Our simulation and experimental results show that the proposed neural network can learn the mapping relationship between the speckle pattern and the target, and extract the photoacoustic signals of the vessels submerged in noise to reconstruct high-quality images of the vessels with a sharp outline and a clean background. Compared with the traditional photoacoustic reconstruction methods, the proposed deep learning-based reconstruction algorithm has a better performance with a lower mean absolute error, higher structural similarity, and higher peak signal-to-noise ratio of reconstructed images. In conclusion, the proposed neural network can effectively extract valid information from highly blurred speckle patterns for the rapid reconstruction of target images, which offers promising applications in transcranial photoacoustic imaging.

Looking deep inside tissue with photoacoustic molecular probes: a review

Significance

Deep tissue noninvasive high-resolution imaging with light is challenging due to the high degree of light absorption and scattering in biological tissue. Photoacoustic imaging (PAI) can overcome some of the challenges of pure optical or ultrasound imaging to provide high-resolution deep tissue imaging. However, label-free PAI signals from light absorbing chromophores within the tissue are nonspecific. The use of exogeneous contrast agents (probes) not only enhances the imaging contrast (and imaging depth) but also increases the specificity of PAI by binding only to targeted molecules and often providing signals distinct from the background.

Aim

We aim to review the current development and future progression of photoacoustic molecular probes/contrast agents.

Approach

First, PAI and the need for using contrast agents are briefly introduced. Then, the recent development of contrast agents in terms of materials used to construct them is discussed. Then, various probes are discussed based on targeting mechanisms, in vivo molecular imaging applications, multimodal uses, and use in theranostic applications.

Results

Material combinations are being used to develop highly specific contrast agents. In addition to passive accumulation, probes utilizing activation mechanisms show promise for greater controllability. Several probes also enable concurrent multimodal use with fluorescence, ultrasound, Raman, magnetic resonance imaging, and computed tomography. Finally, targeted probes are also shown to aid localized and molecularly specific photo-induced therapy.

Conclusions

The development of contrast agents provides a promising prospect for increased contrast, higher imaging depth, and molecularly specific information. Of note are agents that allow for controlled activation, explore other optical windows, and enable multimodal use to overcome some of the shortcomings of label-free PAI.

Spectral crosstalk in photoacoustic computed tomography

Multispectral photoacoustic (PA) imaging faces two major challenges: the spectral coloring effect, which has been studied extensively as an optical inversion problem, and the spectral crosstalk, which is basically a result of non-ideal acoustic inversion. So far, there is no systematic work to analyze the spectral crosstalk because acoustic inversion and spectroscopic measurement are always treated as decoupled. In this work, we theorize and demonstrate through a series of simulations and experiments how imperfect acoustic inversion induces inaccurate PA spectrum measurement. We provide detailed analysis to elucidate how different factors, including limited bandwidth, limited view, light attenuation, out-of-plane signal, and image reconstruction schemes, conspire to render the measured PA spectrum inaccurate. We found that the model-based reconstruction outperforms universal back-projection in suppressing the spectral crosstalk in some cases.

Dermoscopic Image Classification of Pigmented Nevus under Deep Learning and the Correlation with Pathological Features

The objective of this study was to explore the image classification and case characteristics of pigmented nevus (PN) diagnosed by dermoscopy under deep learning. 268 patients were included as the research objects and they were randomly divided into observation group (n = 134) and control group (n = 134). Image recognition algorithm was used for feature extraction, segmentation, and classification of dermoscopic images, and the image recognition and classification algorithm were studied as the performance and accuracy of fusion classifier were compared. The results showed that the classifier was optimized, and the linear kernel accuracy was 85.82%. The PN studied mainly included mixed nevus, junctional nevus, intradermal nevus, and acral nevus. The sensitivity under collaborative training was higher than that under feature training and fusion feature training, and the differences among three trainings were significant (P < 0.05). The sensitivity of the observation group was 88.65%, and the specificity was 90.26%, while the sensitivity and the specificity of the control group were 85.65% and 84.03%, respectively; there were significant differences between the two groups (P < 0.05). In conclusion, dermoscopy under deep learning could be applied as a diagnostic way of PN, which helped improve the accuracy of diagnosis. The dermoscopic manifestations of PN showed a certain corresponding relationship with the type of cases and could provide auxiliary diagnosis in clinical practice. It could be applied clinically.

Deep-E: A Fully-Dense Neural Network for Improving the Elevation Resolution in Linear-Array-Based Photoacoustic Tomography

Linear-array-based photoacoustic tomography has shown broad applications in biomedical research and preclinical imaging. However, the elevational resolution of a linear array is fundamentally limited due to the weak cylindrical focus of the transducer element. While several methods have been proposed to address this issue, they have all handled the problem in a less time-efficient way. In this work, we propose to improve the elevational resolution of a linear array through Deep-E, a fully dense neural network based on U-net. Deep-E exhibits high computational efficiency by converting the three-dimensional problem into a two-dimension problem: it focused on training a model to enhance the resolution along elevational direction by only using the 2D slices in the axial and elevational plane and thereby reducing the computational burden in simulation and training. We demonstrated the efficacy of Deep-E using various datasets, including simulation, phantom, and human subject results. We found that Deep-E could improve elevational resolution by at least four times and recover the object's true size. We envision that Deep-E will have a significant impact in linear-array-based photoacoustic imaging studies by providing high-speed and high-resolution image enhancement.

Deep learning Mueller matrix feature retrieval from a snapshot Stokes image

A Mueller matrix (MM) provides a comprehensive representation of the polarization properties of a complex medium and encodes very rich information on the macro- and microstructural features. Histopathological features can be characterized by polarization parameters derived from MM. However, a MM must be derived from at least four Stokes vectors corresponding to four different incident polarization states, which makes the qualities of MM very sensitive to small changes in the imaging system or the sample during the exposures, such as fluctuations in illumination light and co-registration of polarization component images. In this work, we use a deep learning approach to retrieve MM-based specific polarimetry basis parameters (PBPs) from a snapshot Stokes vector. This data post-processing method is capable of eliminating errors introduced by multi-exposure, as well as reducing the imaging time and hardware complexity. It shows the potential for accurate MM imaging on dynamic samples or in unstable environments. The translation model is designed based on generative adversarial network with customized loss functions. The effectiveness of the approach was demonstrated on liver and breast tissue slices and blood smears. Finally, we evaluated the performance by quantitative similarity assessment methods in both pixel and image levels.

Deep Learning for Instrumented Ultrasonic Tracking: From Synthetic Training Data to In Vivo Application

Instrumented ultrasonic tracking is used to improve needle localization during ultrasound guidance of minimally invasive percutaneous procedures. Here, it is implemented with transmitted ultrasound pulses from a clinical ultrasound imaging probe, which is detected by a fiber-optic hydrophone integrated into a needle. The detected transmissions are then reconstructed to form the tracking image. Two challenges are considered with the current implementation of ultrasonic tracking. First, tracking transmissions are interleaved with the acquisition of B-mode images, and thus, the effective B-mode frame rate is reduced. Second, it is challenging to achieve an accurate localization of the needle tip when the signal-to-noise ratio is low. To address these challenges, we present a framework based on a convolutional neural network (CNN) to maintain spatial resolution with fewer tracking transmissions and enhance signal quality. A major component of the framework included the generation of realistic synthetic training data. The trained network was applied to unseen synthetic data and experimental in vivo tracking data. The performance of needle localization was investigated when reconstruction was performed with fewer (up to eightfold) tracking transmissions. CNN-based processing of conventional reconstructions showed that the axial and lateral spatial resolutions could be improved even with an eightfold reduction in tracking transmissions. The framework presented in this study will significantly improve the performance of ultrasonic tracking, leading to faster image acquisition rates and increased localization accuracy.

Recovery of Blood Flow From Undersampled Photoacoustic Microscopy Data Using Sparse Modeling

Photoacoustic microscopy (PAM) leverages the optical absorption contrast of blood hemoglobin for high-resolution, multi-parametric imaging of the microvasculature in vivo. However, to quantify the blood flow speed, dense spatial sampling is required to assess blood flow-induced loss of correlation of sequentially acquired A-line signals, resulting in increased laser pulse repetition rate and consequently optical fluence. To address this issue, we have developed a sparse modeling approach for blood flow quantification based on downsampled PAM data. Evaluation of its performance both in vitro and in vivo shows that this sparse modeling method can accurately recover the substantially downsampled data (up to 8 times) for correlation-based blood flow analysis, with a relative error of 12.7 ± 6.1 % across 10 datasets in vitro and 12.7 ± 12.1 % in vivo for data downsampled 8 times. Reconstruction with the proposed method is on par with recovery using compressive sensing, which exhibits an error of 12.0 ± 7.9 % in vitro and 33.86 ± 26.18 % in vivo for data downsampled 8 times. Both methods outperform bicubic interpolation, which shows an error of 15.95 ± 9.85 % in vitro and 110.7 ± 87.1 % in vivo for data downsampled 8 times.

Preparation of a gold@europium-based coordination polymer nanocomposite with excellent photothermal properties and its potential for four-mode imaging

A nanocomposite was synthesized by replacing the toxic CTAB on the surface of GNRs with a europium-based hyaluronic acid coordination polymer. The nanocomposite exhibits excellent photothermal performance and also has potential for four-mode imaging.

Advanced Ultrasound and Photoacoustic Imaging in Cardiology

Cardiovascular diseases (CVDs) remain the leading cause of death worldwide. An effective management and treatment of CVDs highly relies on accurate diagnosis of the disease. As the most common imaging technique for clinical diagnosis of the CVDs, US imaging has been intensively explored. Especially with the introduction of deep learning (DL) techniques, US imaging has advanced tremendously in recent years. Photoacoustic imaging (PAI) is one of the most promising new imaging methods in addition to the existing clinical imaging methods. It can characterize different tissue compositions based on optical absorption contrast and thus can assess the functionality of the tissue. This paper reviews some major technological developments in both US (combined with deep learning techniques) and PA imaging in the application of diagnosis of CVDs.

Improving Minimum Variance Beamforming with Sub-Aperture Processing for Photoacoustic Imaging

Minimum variance (MV) beamforming improves resolution and reduces sidelobes when compared to delay-and-sum (DAS) beamforming for photoacoustic imaging (PAI). However, some level of sidelobe signal and incoherent clutter persist degrading MV PAI quality. Here, an adaptive beamforming algorithm (PSAPMV) combining MV formulation and sub-aperture processing is proposed. In PSAPMV, the received channel data are split into two complementary nonoverlapping sub-apertures and beamformed using MV. A weighting matrix based on similarity between sub-aperture beamformed images was derived and multiplied with the full aperture MV image resulting in suppression of sidelobe and incoherent clutter in the PA image. Numerical simulation experiments with point targets, diffuse inclusions and microvasculature networks are used to validate PSAPMV. Quantitative evaluation was done in terms of main-lobe-to-side-lobe ratio, full width at half maximum (FWHM), contrast ratio (CR) and generalized contrast-to-noise ratio (gCNR). PSAPMV demonstrated improved beamforming performance both qualitatively and quantitatively. PSAPMV had higher resolution (FWHM =0.19 mm) than MV (0.21 mm) and DAS (0.22mm) in point target simulations, better target detectability (gCNR =0.99) than MV (0.89) and DAS (0.84) for diffuse inclusions and improved contrast (CR in microvasculature simulation, DAS = 15.38, MV = 22.42, PSAPMV = 51.74 dB).

Artifact removal in photoacoustic tomography with an unsupervised method

Photoacoustic tomography (PAT) is an emerging biomedical imaging technology that can realize high contrast imaging with a penetration depth of the acoustic. Recently, deep learning (DL) methods have also been successfully applied to PAT for improving the image reconstruction quality. However, the current DL-based PAT methods are implemented by the supervised learning strategy, and the imaging performance is dependent on the available ground-truth data. To overcome the limitation, this work introduces a new image domain transformation method based on cyclic generative adversarial network (CycleGAN), termed as PA-GAN, which is used to remove artifacts in PAT images caused by the use of the limited-view measurement data in an unsupervised learning way. A series of data from phantom and in vivo experiments are used to evaluate the performance of the proposed PA-GAN. The experimental results show that PA-GAN provides a good performance in removing artifacts existing in photoacoustic tomographic images. In particular, when dealing with extremely sparse measurement data (e.g., 8 projections in circle phantom experiments), higher imaging performance is achieved by the proposed unsupervised PA-GAN, with an improvement of ∼14% in structural similarity (SSIM) and ∼66% in peak signal to noise ratio (PSNR), compared with the supervised-learning U-Net method. With an increasing number of projections (e.g., 128 projections), U-Net, especially FD U-Net, shows a slight improvement in artifact removal capability, in terms of SSIM and PSNR. Furthermore, the computational time obtained by PA-GAN and U-Net is similar (∼60 ms/frame), once the network is trained. More importantly, PA-GAN is more flexible than U-Net that allows the model to be effectively trained with unpaired data. As a result, PA-GAN makes it possible to implement PAT with higher flexibility without compromising imaging performance.

Real-Time 3D Reconstruction Method Based on Monocular Vision

Real-time 3D reconstruction is one of the current popular research directions of computer vision, and it has become the core technology in the fields of virtual reality, industrialized automatic systems, and mobile robot path planning. Currently, there are three main problems in the real-time 3D reconstruction field. Firstly, it is expensive. It requires more varied sensors, so it is less convenient. Secondly, the reconstruction speed is slow, and the 3D model cannot be established accurately in real time. Thirdly, the reconstruction error is large, which cannot meet the requirements of scenes with accuracy. For this reason, we propose a real-time 3D reconstruction method based on monocular vision in this paper. Firstly, a single RGB-D camera is used to collect visual information in real time, and the YOLACT++ network is used to identify and segment the visual information to extract part of the important visual information. Secondly, we combine the three stages of depth recovery, depth optimization, and deep fusion to propose a three-dimensional position estimation method based on deep learning for joint coding of visual information. It can reduce the depth error caused by the depth measurement process, and the accurate 3D point values of the segmented image can be obtained directly. Finally, we propose a method based on the limited outlier adjustment of the cluster center distance to optimize the three-dimensional point values obtained above. It improves the real-time reconstruction accuracy and obtains the three-dimensional model of the object in real time. Experimental results show that this method only needs a single RGB-D camera, which is not only low cost and convenient to use, but also significantly improves the speed and accuracy of 3D reconstruction.

Photoacoustic imaging for monitoring of stroke diseases: A review

Stroke is the leading cause of death and disability after ischemic heart disease. However, there is lacking a non-invasive long-time monitoring technique for stroke diagnosis and therapy. The photoacoustic imaging approach reconstructs images of an object based on the energy excitation by optical absorption and its conversion to acoustic waves, due to corresponding thermoelastic expansion, which has optical resolution and acoustic propagation. This emerging functional imaging method is a non-invasive technique. Due to its precision, this method is particularly attractive for stroke monitoring purpose. In this paper, we review the achievements of this technology and its applications on stroke, as well as the development status in both animal and human applications. Also, various photoacoustic systems and multi-modality photoacoustic imaging are introduced as for potential clinical applications. Finally, the challenges of photoacoustic imaging for monitoring stroke are discussed.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Subaperture Processing-Based Adaptive Beamforming for Photoacoustic Imaging

Delay-and-sum (DAS) beamformers, when applied to photoacoustic (PA) image reconstruction, produce strong sidelobes due to the absence of transmit focusing. Consequently, DAS PA images are often severely degraded by strong off-axis clutter. For preclinical in vivo cardiac PA imaging, the presence of these noise artifacts hampers the detectability and interpretation of PA signals from the myocardial wall, crucial for studying blood-dominated cardiac pathological information and to complement functional information derived from ultrasound imaging. In this article, we present PA subaperture processing (PSAP), an adaptive beamforming method, to mitigate these image degrading effects. In PSAP, a pair of DAS reconstructed images is formed by splitting the received channel data into two complementary nonoverlapping subapertures. Then, a weighting matrix is derived by analyzing the correlation between subaperture beamformed images and multiplied with the full-aperture DAS PA image to reduce sidelobes and incoherent clutter. We validated PSAP using numerical simulation studies using point target, diffuse inclusion and microvasculature imaging, and in vivo feasibility studies on five healthy murine models. Qualitative and quantitative analysis demonstrate improvements in PAI image quality with PSAP compared to DAS and coherence factor weighted DAS (DAS _CF ). PSAP demonstrated improved target detectability with a higher generalized contrast-to-noise (gCNR) ratio in vasculature simulations where PSAP produces 19.61% and 19.53% higher gCNRs than DAS and DAS _CF , respectively. Furthermore, PSAP provided higher image contrast quantified using contrast ratio (CR) (e.g., PSAP produces 89.26% and 11.90% higher CR than DAS and DAS _CF in vasculature simulations) and improved clutter suppression.

Deep learning of image- and time-domain data enhances the visibility of structures in optoacoustic tomography

Images rendered with common optoacoustic system implementations are often afflicted with distortions and poor visibility of structures, hindering reliable image interpretation and quantification of bio-chrome distribution. Among the practical limitations contributing to artifactual reconstructions are insufficient tomographic detection coverage and suboptimal illumination geometry, as well as inability to accurately account for acoustic reflections and speed of sound heterogeneities in the imaged tissues. Here we developed a convolutional neural network (CNN) approach for enhancement of optoacoustic image quality which combines training on both time-resolved signals and tomographic reconstructions. Reference human finger data for training the CNN were recorded using a full-ring array system that provides optimal tomographic coverage around the imaged object. The reconstructions were further refined with a dedicated algorithm that minimizes acoustic reflection artifacts induced by acoustically mismatch structures, such as bones. The combined methodology is shown to outperform other learning-based methods solely operating on image-domain data.

Photoacoustic computed tomography for functional human brain imaging [Invited]

The successes of magnetic resonance imaging and modern optical imaging of human brain function have stimulated the development of complementary modalities that offer molecular specificity, fine spatiotemporal resolution, and sufficient penetration simultaneously. By virtue of its rich optical contrast, acoustic resolution, and imaging depth far beyond the optical transport mean free path (∼1 mm in biological tissues), photoacoustic computed tomography (PACT) offers a promising complementary modality. In this article, PACT for functional human brain imaging is reviewed in its hardware, reconstruction algorithms, in vivo demonstration, and potential roadmap.

Sounding out the hidden data: A concise review of deep learning in photoacoustic imaging

The rapidly evolving field of photoacoustic tomography utilizes endogenous chromophores to extract both functional and structural information from deep within tissues. It is this power to perform precise quantitative measurements in vivo-with endogenous or exogenous contrast-that makes photoacoustic tomography highly promising for clinical translation in functional brain imaging, early cancer detection, real-time surgical guidance, and the visualization of dynamic drug responses. Considering photoacoustic tomography has benefited from numerous engineering innovations, it is of no surprise that many of photoacoustic tomography's current cutting-edge developments incorporate advances from the equally novel field of artificial intelligence. More specifically, alongside the growth and prevalence of graphical processing unit capabilities within recent years has emerged an offshoot of artificial intelligence known as deep learning. Rooted in the solid foundation of signal processing, deep learning typically utilizes a method of optimization known as gradient descent to minimize a loss function and update model parameters. There are already a number of innovative efforts in photoacoustic tomography utilizing deep learning techniques for a variety of purposes, including resolution enhancement, reconstruction artifact removal, undersampling correction, and improved quantification. Most of these efforts have proven to be highly promising in addressing long-standing technical obstacles where traditional solutions either completely fail or make only incremental progress. This concise review focuses on the history of applied artificial intelligence in photoacoustic tomography, presents recent advances at this multifaceted intersection of fields, and outlines the most exciting advances that will likely propagate into promising future innovations.

Deep learning for biomedical photoacoustic imaging: A review

Photoacoustic imaging (PAI) is a promising emerging imaging modality that enables spatially resolved imaging of optical tissue properties up to several centimeters deep in tissue, creating the potential for numerous exciting clinical applications. However, extraction of relevant tissue parameters from the raw data requires the solving of inverse image reconstruction problems, which have proven extremely difficult to solve. The application of deep learning methods has recently exploded in popularity, leading to impressive successes in the context of medical imaging and also finding first use in the field of PAI. Deep learning methods possess unique advantages that can facilitate the clinical translation of PAI, such as extremely fast computation times and the fact that they can be adapted to any given problem. In this review, we examine the current state of the art regarding deep learning in PAI and identify potential directions of research that will help to reach the goal of clinical applicability.

Enhancement of in vivo cardiac photoacoustic signal specificity using spatiotemporal singular value decomposition

SIGNIFICANCE

Photoacoustic imaging (PAI) can be used to infer molecular information about myocardial health non-invasively in vivo using optical excitation at ultrasonic spatial resolution. For clinical and preclinical linear array imaging systems, conventional delay-and-sum (DAS) beamforming is typically used. However, DAS cardiac PA images are prone to artifacts such as diffuse quasi-static clutter with temporally varying noise-reducing myocardial signal specificity. Typically, multiple frame averaging schemes are utilized to improve the quality of cardiac PAI, which affects the spatial and temporal resolution and reduces sensitivity to subtle PA signal variation. Furthermore, frame averaging might corrupt myocardial oxygen saturation quantification due to the presence of natural cardiac wall motion. In this paper, a spatiotemporal singular value decomposition (SVD) processing algorithm is proposed to reduce DAS PAI artifacts and subsequent enhancement of myocardial signal specificity.

AIM

Demonstrate enhancement of PA signals from myocardial tissue compared to surrounding tissues and blood inside the left-ventricular (LV) chamber using spatiotemporal SVD processing with electrocardiogram (ECG) and respiratory signal (ECG-R) gated in vivo murine cardiac PAI.

APPROACH

In vivo murine cardiac PAI was performed by collecting single wavelength (850 nm) photoacoustic channel data on eight healthy mice. A three-dimensional (3D) volume of complex PAI data over a cardiac cycle was reconstructed using a custom ECG-R gating algorithm and DAS beamforming. Spatiotemporal SVD was applied on a two-dimensional Casorati matrix generated using the 3D volume of PAI data. The singular value spectrum (SVS) was then filtered to remove contributions from diffuse quasi-static clutter and random noise. Finally, SVD processed beamformed images were derived using filtered SVS and inverse SVD computations.

RESULTS

Qualitative comparison with DAS and minimum variance (MV) beamforming shows that SVD processed images had better myocardial signal specificity, contrast, and target detectability. DAS, MV, and SVD images were quantitatively evaluated by calculating contrast ratio (CR), generalized contrast-to-noise ratio (gCNR), and signal-to-noise ratio (SNR). Quantitative evaluations were done at three cardiac time points (during systole, at end-systole (ES), and during diastole) identified from co-registered ultrasound M-Mode image. Mean CR, gCNR, and SNR values of SVD images at ES were 245, 115.15, and 258.17 times higher than DAS images with statistical significance evaluated with one-way analysis of variance.

CONCLUSIONS

Our results suggest that significantly better-quality images can be realized using spatiotemporal SVD processing for in vivo murine cardiac PAI.

Deep learning in photoacoustic imaging: a review

SIGNIFICANCE

Photoacoustic (PA) imaging can provide structural, functional, and molecular information for preclinical and clinical studies. For PA imaging (PAI), non-ideal signal detection deteriorates image quality, and quantitative PAI (QPAI) remains challenging due to the unknown light fluence spectra in deep tissue. In recent years, deep learning (DL) has shown outstanding performance when implemented in PAI, with applications in image reconstruction, quantification, and understanding.

AIM

We provide (i) a comprehensive overview of the DL techniques that have been applied in PAI, (ii) references for designing DL models for various PAI tasks, and (iii) a summary of the future challenges and opportunities.

APPROACH

Papers published before November 2020 in the area of applying DL in PAI were reviewed. We categorized them into three types: image understanding, reconstruction of the initial pressure distribution, and QPAI.

RESULTS

When applied in PAI, DL can effectively process images, improve reconstruction quality, fuse information, and assist quantitative analysis.

CONCLUSION

DL has become a powerful tool in PAI. With the development of DL theory and technology, it will continue to boost the performance and facilitate the clinical translation of PAI.

Review of deep learning for photoacoustic imaging

Machine learning has been developed dramatically and witnessed a lot of applications in various fields over the past few years. This boom originated in 2009, when a new model emerged, that is, the deep artificial neural network, which began to surpass other established mature models on some important benchmarks. Later, it was widely used in academia and industry. Ranging from image analysis to natural language processing, it fully exerted its magic and now become the state-of-the-art machine learning models. Deep neural networks have great potential in medical imaging technology, medical data analysis, medical diagnosis and other healthcare issues, and is promoted in both pre-clinical and even clinical stages. In this review, we performed an overview of some new developments and challenges in the application of machine learning to medical image analysis, with a special focus on deep learning in photoacoustic imaging. The aim of this review is threefold: (i) introducing deep learning with some important basics, (ii) reviewing recent works that apply deep learning in the entire ecological chain of photoacoustic imaging, from image reconstruction to disease diagnosis, (iii) providing some open source materials and other resources for researchers interested in applying deep learning to photoacoustic imaging.

Compensating for visibility artefacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties

Conventional photoacoustic imaging may suffer from the limited view and bandwidth of ultrasound transducers. A deep learning approach is proposed to handle these problems and is demonstrated both in simulations and in experiments on a multi-scale model of leaf skeleton. We employed an experimental approach to build the training and the test sets using photographs of the samples as ground truth images. Reconstructions produced by the neural network show a greatly improved image quality as compared to conventional approaches. In addition, this work aimed at quantifying the reliability of the neural network predictions. To achieve this, the dropout Monte-Carlo procedure is applied to estimate a pixel-wise degree of confidence on each predicted picture. Last, we address the possibility to use transfer learning with simulated data in order to drastically limit the size of the experimental dataset.

Real-time interleaved spectroscopic photoacoustic and ultrasound (PAUS) scanning with simultaneous fluence compensation and motion correction

For over two decades photoacoustic imaging has been tested clinically, but successful human trials have been limited. To enable quantitative clinical spectroscopy, the fundamental issues of wavelength-dependent fluence variations and inter-wavelength motion must be overcome. Here we propose a real-time, spectroscopic photoacoustic/ultrasound (PAUS) imaging approach using a compact, 1-kHz rate wavelength-tunable laser. Instead of illuminating tissue over a large area, the fiber-optic delivery system surrounding an US array sequentially scans a narrow laser beam, with partial PA image reconstruction for each laser pulse. The final image is then formed by coherently summing partial images. This scheme enables (i) automatic compensation for wavelength-dependent fluence variations in spectroscopic PA imaging and (ii) motion correction of spectroscopic PA frames using US speckle tracking in real-time systems. The 50-Hz video rate PAUS system is demonstrated in vivo using a murine model of labelled drug delivery.