Toward accurate quantitative photoacoustic imaging: learning vascular blood oxygen saturation in three dimensions

Current trends in the characterization and monitoring of vascular response to cancer therapy

Tumor vascular physiology is an important determinant of disease progression as well as the therapeutic outcome of cancer treatment. Angiogenesis or the lack of it provides crucial information about the tumor's blood supply and therefore can be used as an index for cancer growth and progression. While standalone anti-angiogenic therapy demonstrated limited therapeutic benefits, its combination with chemotherapeutic agents improved the overall survival of cancer patients. This could be attributed to the effect of vascular normalization, a dynamic process that temporarily reverts abnormal vasculature to the normal phenotype maximizing the delivery and intratumor distribution of chemotherapeutic agents. Longitudinal monitoring of vascular changes following antiangiogenic therapy can indicate an optimal window for drug administration and estimate the potential outcome of treatment. This review primarily focuses on the status of various imaging modalities used for the longitudinal characterization of vascular changes before and after anti-angiogenic therapies and their clinical prospects.

Extractor-attention-predictor network for quantitative photoacoustic tomography

Quantitative photoacoustic tomography (qPAT) holds great potential in estimating chromophore concentrations, whereas the involved optical inverse problem, aiming to recover absorption coefficient distributions from photoacoustic images, remains challenging. To address this problem, we propose an extractor-attention-predictor network architecture (EAPNet), which employs a contracting-expanding structure to capture contextual information alongside a multilayer perceptron to enhance nonlinear modeling capability. A spatial attention module is introduced to facilitate the utilization of important information. We also use a balanced loss function to prevent network parameter updates from being biased towards specific regions. Our method obtains satisfactory quantitative metrics in simulated and real-world validations. Moreover, it demonstrates superior robustness to target properties and yields reliable results for targets with small size, deep location, or relatively low absorption intensity, indicating its broader applicability. The EAPNet, compared to the conventional UNet, exhibits improved efficiency, which significantly enhances performance while maintaining similar network size and computational complexity.

Compensating unknown speed of sound in learned fast 3D limited-view photoacoustic tomography

Real-time applications in three-dimensional photoacoustic tomography from planar sensors rely on fast reconstruction algorithms that assume the speed of sound (SoS) in the tissue is homogeneous. Moreover, the reconstruction quality depends on the correct choice for the constant SoS. In this study, we discuss the possibility of ameliorating the problem of unknown or heterogeneous SoS distributions by using learned reconstruction methods. This can be done by modelling the uncertainties in the training data. In addition, a correction term can be included in the learned reconstruction method. We investigate the influence of both and while a learned correction component can improve reconstruction quality further, we show that a careful choice of uncertainties in the training data is the primary factor to overcome unknown SoS. We support our findings with simulated and in vivo measurements in 3D.

Oxygenation heterogeneity facilitates spatiotemporal flow pattern visualization inside human blood vessels using photoacoustic computed tomography

Hemodynamics can be explored through various biomedical imaging techniques. However, observing transient spatiotemporal variations in the saturation of oxygen (sO2) within human blood vessels proves challenging with conventional methods. In this study, we employed photoacoustic computed tomography (PACT) to reconstruct the evolving spatiotemporal patterns in a human vein. Through analysis of the multi-wavelength photoacoustic (PA) spectrum, we illustrated the dynamic distribution within blood vessels. Additionally, we computationally rendered the dynamic process of venous blood flowing into the major vein and entering a branching vessel. Notably, we successfully recovered, in real time, the parabolic wavefront profile of laminar flow inside a deep vein in vivo-a first-time achievement. While the study is preliminary, the demonstrated capability of dynamic sO2 imaging holds promise for new applications in biology and medicine.

Simultaneous photoacoustic and ultrasound imaging: A review

Photoacoustic imaging (PAI) is an emerging biomedical imaging technique that combines the advantages of optical and ultrasound imaging, enabling the generation of images with both optical resolution and acoustic penetration depth. By leveraging similar signal acquisition and processing methods, the integration of photoacoustic and ultrasound imaging has introduced a novel hybrid imaging modality suitable for clinical applications. Photoacoustic-ultrasound imaging allows for non-invasive, high-resolution, and deep-penetrating imaging, providing a wealth of image information. In recent years, with the deepening research and the expanding biomedical application scenarios of photoacoustic-ultrasound bimodal systems, the immense potential of photoacoustic-ultrasound bimodal imaging in basic research and clinical applications has been demonstrated, with some research achievements already commercialized. In this review, we introduce the principles, technical advantages, and biomedical applications of photoacoustic-ultrasound bimodal imaging techniques, specifically focusing on tomographic, microscopic, and endoscopic imaging modalities. Furthermore, we discuss the future directions of photoacoustic-ultrasound bimodal imaging technology.

Photoacoustic imaging of squirrel monkey cortical responses induced by peripheral mechanical stimulation

Non-human primates (NHPs) are crucial models for studies of neuronal activity. Emerging photoacoustic imaging modalities offer excellent tools for studying NHP brains with high sensitivity and high spatial resolution. In this research, a photoacoustic microscopy (PAM) device was used to provide a label-free quantitative characterization of cerebral hemodynamic changes due to peripheral mechanical stimulation. A 5 × 5 mm area within the somatosensory cortex region of an adult squirrel monkey was imaged. A deep, fully connected neural network was characterized and applied to the PAM images of the cortex to enhance the vessel structures after mechanical stimulation on the forelimb digits. The quality of the PAM images was improved significantly with a neural network while preserving the hemodynamic responses. The functional responses to the mechanical stimulation were characterized based on the improved PAM images. This study demonstrates capability of PAM combined with machine learning for functional imaging of the NHP brain.

Dual-modal Photoacoustic and Ultrasound Imaging: from preclinical to clinical applications

Photoacoustic imaging is a novel biomedical imaging modality that has emerged over the recent decades. Due to the conversion of optical energy into the acoustic wave, photoacoustic imaging offers high-resolution imaging in depth beyond the optical diffusion limit. Photoacoustic imaging is frequently used in conjunction with ultrasound as a hybrid modality. The combination enables the acquisition of both optical and acoustic contrasts of tissue, providing functional, structural, molecular, and vascular information within the same field of view. In this review, we first described the principles of various photoacoustic and ultrasound imaging techniques and then classified the dual-modal imaging systems based on their preclinical and clinical imaging applications. The advantages of dual-modal imaging were thoroughly analyzed. Finally, the review ends with a critical discussion of existing developments and a look toward the future.

Quantitative photoacoustic tomography: modeling and inverse problems

Significance

Quantitative photoacoustic tomography (QPAT) exploits the photoacoustic effect with the aim of estimating images of clinically relevant quantities related to the tissue's optical absorption. The technique has two aspects: an acoustic part, where the initial acoustic pressure distribution is estimated from measured photoacoustic time-series, and an optical part, where the distributions of the optical parameters are estimated from the initial pressure.

Aim

Our study is focused on the optical part. In particular, computational modeling of light propagation (forward problem) and numerical solution methodologies of the image reconstruction (inverse problem) are discussed.

Approach

The commonly used mathematical models of how light and sound propagate in biological tissue are reviewed. A short overview of how the acoustic inverse problem is usually treated is given. The optical inverse problem and methods for its solution are reviewed. In addition, some limitations of real-life measurements and their effect on the inverse problems are discussed.

Results

An overview of QPAT with a focus on the optical part was given. Computational modeling and inverse problems of QPAT were addressed, and some key challenges were discussed. Furthermore, the developments for tackling these problems were reviewed. Although modeling of light transport is well-understood and there is a well-developed framework of inverse mathematics for approaching the inverse problem of QPAT, there are still challenges in taking these methodologies to practice.

Conclusions

Modeling and inverse problems of QPAT together were discussed. The scope was limited to the optical part, and the acoustic aspects were discussed only to the extent that they relate to the optical aspect.

Review of Deep Learning Approaches for Interleaved Photoacoustic and Ultrasound (PAUS) Imaging

Photoacoustic (PA) imaging provides optical contrast at relatively large depths within the human body, compared to other optical methods, at ultrasound (US) spatial resolution. By integrating real-time PA and US (PAUS) modalities, PAUS imaging has the potential to become a routine clinical modality bringing the molecular sensitivity of optics to medical US imaging. For applications where the full capabilities of clinical US scanners must be maintained in PAUS, conventional limited view and bandwidth transducers must be used. This approach, however, cannot provide high-quality maps of PA sources, especially vascular structures. Deep learning (DL) using data-driven modeling with minimal human design has been very effective in medical imaging, medical data analysis, and disease diagnosis, and has the potential to overcome many of the technical limitations of current PAUS imaging systems. The primary purpose of this article is to summarize the background and current status of DL applications in PAUS imaging. It also looks beyond current approaches to identify remaining challenges and opportunities for robust translation of PAUS technologies to the clinic.

Recurrent and convolutional neural networks for sequential multispectral optoacoustic tomography (MSOT) imaging

Multispectral optoacoustic tomography (MSOT) is a beneficial technique for diagnosing and analyzing biological samples since it provides meticulous details in anatomy and physiology. However, acquiring high through-plane resolution volumetric MSOT is time-consuming. Here, we propose a deep learning model based on hybrid recurrent and convolutional neural networks to generate sequential cross-sectional images for an MSOT system. This system provides three modalities (MSOT, ultrasound, and optoacoustic imaging of a specific exogenous contrast agent) in a single scan. This study used ICG-conjugated nanoworms particles (NWs-ICG) as the contrast agent. Instead of acquiring seven images with a step size of 0.1 mm, we can receive two images with a step size of 0.6 mm as input for the proposed deep learning model. The deep learning model can generate five other images with a step size of 0.1 mm between these two input images meaning we can reduce acquisition time by approximately 71%.

A jointed feature fusion framework for photoacoustic image reconstruction

The standard reconstruction of Photoacoustic (PA) computed tomography (PACT) image could cause the artifacts due to interferences or ill-posed setup. Recently, deep learning has been used to reconstruct the PA image with ill-posed conditions. In this paper, we propose a jointed feature fusion framework (JEFF-Net) based on deep learning to reconstruct the PA image using limited-view data. The cross-domain features from limited-view position-wise data and the reconstructed image are fused by a backtracked supervision. A quarter position-wise data (32 channels) is fed into model, which outputs another 3-quarters-view data (96 channels). Moreover, two novel losses are designed to restrain the artifacts by sufficiently manipulating superposed data. The experimental results have demonstrated the superior performance and quantitative evaluations show that our proposed method outperformed the ground-truth in some metrics by 135% (SSIM for simulation) and 40% (gCNR for in-vivo) improvement.

Photoacoustic image synthesis with generative adversarial networks

Photoacoustic tomography (PAT) has the potential to recover morphological and functional tissue properties with high spatial resolution. However, previous attempts to solve the optical inverse problem with supervised machine learning were hampered by the absence of labeled reference data. While this bottleneck has been tackled by simulating training data, the domain gap between real and simulated images remains an unsolved challenge. We propose a novel approach to PAT image synthesis that involves subdividing the challenge of generating plausible simulations into two disjoint problems: (1) Probabilistic generation of realistic tissue morphology, and (2) pixel-wise assignment of corresponding optical and acoustic properties. The former is achieved with Generative Adversarial Networks (GANs) trained on semantically annotated medical imaging data. According to a validation study on a downstream task our approach yields more realistic synthetic images than the traditional model-based approach and could therefore become a fundamental step for deep learning-based quantitative PAT (qPAT).

Ultrasound-enhanced Unet model for quantitative photoacoustic tomography of ovarian lesions

Quantitative photoacoustic tomography (QPAT) is a valuable tool in characterizing ovarian lesions for accurate diagnosis. However, accurately reconstructing a lesion's optical absorption distributions from photoacoustic signals measured with multiple wavelengths is challenging because it involves an ill-posed inverse problem with three unknowns: the Grüneisen parameter ( Γ ) , the absorption distribution, and the optical fluence ( ϕ ) . Here, we propose a novel ultrasound-enhanced Unet model (US-Unet) that reconstructs optical absorption distribution from PAT data. A pre-trained ResNet-18 extracts the US features typically identified as morphologies of suspicious ovarian lesions, and a Unet is implemented to reconstruct optical absorption coefficient maps, using the initial pressure and US features extracted by ResNet-18. To test this US-Unet model, we calculated the blood oxygenation saturation values and total hemoglobin concentrations from 655 regions of interest (ROIs) (421 benign, 200 malignant, and 34 borderline ROIs) obtained from clinical images of 35 patients with ovarian/adnexal lesions. A logistic regression model was used to compute the ROC, the area under the ROC curve (AUC) was 0.94, and the accuracy was 0.89. To the best of our knowledge, this is the first study to reconstruct quantitative PAT with PA signals and US-based structural features.

Modeling photoacoustic pressure generation in colloidal suspensions at different volume fractions based on a multi-scale approach

Further development of quantitative photoacoustic tomography requires understanding the photoacoustic pressure generation by modeling the generation process. This study modeled the initial photoacoustic pressure in colloidal suspensions, used as tissue phantoms, at different volume fractions on a multi-scale approach. We modeled the thermodynamic and light scattering properties on a microscopic scale with/without treating the hard-sphere interaction between colloidal particles. Meanwhile, we did the light energy density on a macroscopic scale. We showed that the hard-sphere interaction significantly influences the initial pressure and related quantities at a high volume fraction except for the thermodynamic properties. We also showed the initial pressure at the absorber inside the medium logarithmically decreases with increasing the volume fractions. This result is mainly due to the decay of the light energy density with light scattering. Our numerical results suggest that modeling light scattering and propagation is crucial over modeling thermal expansion.

Spectral crosstalk in photoacoustic computed tomography

Multispectral photoacoustic (PA) imaging faces two major challenges: the spectral coloring effect, which has been studied extensively as an optical inversion problem, and the spectral crosstalk, which is basically a result of non-ideal acoustic inversion. So far, there is no systematic work to analyze the spectral crosstalk because acoustic inversion and spectroscopic measurement are always treated as decoupled. In this work, we theorize and demonstrate through a series of simulations and experiments how imperfect acoustic inversion induces inaccurate PA spectrum measurement. We provide detailed analysis to elucidate how different factors, including limited bandwidth, limited view, light attenuation, out-of-plane signal, and image reconstruction schemes, conspire to render the measured PA spectrum inaccurate. We found that the model-based reconstruction outperforms universal back-projection in suppressing the spectral crosstalk in some cases.

Machine learning enabled multiple illumination quantitative optoacoustic oximetry imaging in humans

Optoacoustic (OA) imaging is a promising modality for quantifying blood oxygen saturation (sO2) in various biomedical applications - in diagnosis, monitoring of organ function, or even tumor treatment planning. We present an accurate and practically feasible real-time capable method for quantitative imaging of sO2 based on combining multispectral (MS) and multiple illumination (MI) OA imaging with learned spectral decoloring (LSD). For this purpose we developed a hybrid real-time MI MS OA imaging setup with ultrasound (US) imaging capability; we trained gradient boosting machines on MI spectrally colored absorbed energy spectra generated by generic Monte Carlo simulations and used the trained models to estimate sO2 on real OA measurements. We validated MI-LSD in silico and on in vivo image sequences of radial arteries and accompanying veins of five healthy human volunteers. We compared the performance of the method to prior LSD work and conventional linear unmixing. MI-LSD provided highly accurate results in silico and consistently plausible results in vivo. This preliminary study shows a potentially high applicability of quantitative OA oximetry imaging, using our method.

Quantitative endoscopic photoacoustic tomography using a convolutional neural network

Endoscopic photoacoustic tomography (EPAT) is a catheter-based hybrid imaging modality capable of providing structural and functional information of biological luminal structures, such as coronary arterial vessels and the digestive tract. The recovery of the optical properties of the imaged tissue from acoustic measurements achieved by optical inversion is essential for implementing quantitative EPAT (qEPAT). In this paper, a convolutional neural network (CNN) based on deep gradient descent is developed for qEPAT. The network enables the reconstruction of images representing the spatially varying absorption coefficient in cross-sections of the tubular structures from limited measurement data. The forward operator reflecting the mapping from the absorption coefficient to the optical deposition due to pulsed irradiation is embedded into the network training. The network parameters are optimized layer by layer through the deep gradient descent mechanism using the numerically simulated data. The operation processes of the forward operator and its adjoint operator are separated from the network training. The trained network outputs an image representing the distribution of absorption coefficients by inputting an image that represents the optical deposition. The method has been tested with computer-generated phantoms mimicking coronary arterial vessels containing various tissue types. Results suggest that the structural similarity of the images reconstructed by our method is increased by about 10% in comparison with the non-learning method based on error minimization in the case of the same measuring view.

SIMPA: an open-source toolkit for simulation and image processing for photonics and acoustics

SIGNIFICANCE

Optical and acoustic imaging techniques enable noninvasive visualisation of structural and functional properties of tissue. The quantification of measurements, however, remains challenging due to the inverse problems that must be solved. Emerging data-driven approaches are promising, but they rely heavily on the presence of high-quality simulations across a range of wavelengths due to the lack of ground truth knowledge of tissue acoustical and optical properties in realistic settings.

AIM

To facilitate this process, we present the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit. SIMPA is being developed according to modern software design standards.

APPROACH

SIMPA enables the use of computational forward models, data processing algorithms, and digital device twins to simulate realistic images within a single pipeline. SIMPA's module implementations can be seamlessly exchanged as SIMPA abstracts from the concrete implementation of each forward model and builds the simulation pipeline in a modular fashion. Furthermore, SIMPA provides comprehensive libraries of biological structures, such as vessels, as well as optical and acoustic properties and other functionalities for the generation of realistic tissue models.

RESULTS

To showcase the capabilities of SIMPA, we show examples in the context of photoacoustic imaging: the diversity of creatable tissue models, the customisability of a simulation pipeline, and the degree of realism of the simulations.

CONCLUSIONS

SIMPA is an open-source toolkit that can be used to simulate optical and acoustic imaging modalities. The code is available at: https://github.com/IMSY-DKFZ/simpa, and all of the examples and experiments in this paper can be reproduced using the code available at: https://github.com/IMSY-DKFZ/simpa_paper_experiments.

Normalization of optical fluence distribution for three-dimensional functional optoacoustic tomography of the breast

SIGNIFICANCE

In three-dimensional (3D) functional optoacoustic tomography (OAT), wavelength-dependent optical attenuation and nonuniform incident optical fluence limit imaging depth and field of view and can hinder accurate estimation of functional quantities, such as the vascular blood oxygenation. These limitations hinder OAT of large objects, such as a human female breast.

AIM

We aim to develop a measurement-data-driven method for normalization of the optical fluence distribution and to investigate blood vasculature detectability and accuracy for estimating vascular blood oxygenation.

APPROACH

The proposed method is based on reasonable assumptions regarding breast anatomy and optical properties. The nonuniform incident optical fluence is estimated based on the illumination geometry in the OAT system, and the depth-dependent optical attenuation is approximated using Beer-Lambert law.

RESULTS

Numerical studies demonstrated that the proposed method significantly enhanced blood vessel detectability and improved estimation accuracy of the vascular blood oxygenation from multiwavelength OAT measurements, compared with direct application of spectral linear unmixing without optical fluence compensation. Experimental results showed that the proposed method revealed previously invisible structures in regions deeper than 15 mm and/or near the chest wall.

CONCLUSIONS

The proposed method provides a straightforward and computationally inexpensive approximation of wavelength-dependent effective optical attenuation and, thus, enables mitigation of the spectral coloring effect in functional 3D OAT imaging.

Design of Metaheuristic Optimization-Based Vascular Segmentation Techniques for Photoacoustic Images

Biomedical imaging technologies are designed to offer functional, anatomical, and molecular details related to the internal organs. Photoacoustic imaging (PAI) is becoming familiar among researchers and industrialists. The PAI is found useful in several applications of brain and cancer imaging such as prostate cancer, breast cancer, and ovarian cancer. At the same time, the vessel images hold important medical details which offer strategies for a qualified diagnosis. Recently developed image processing techniques can be employed to segment vessels. Since vessel segmentation on PAI is a difficult process, this paper employs metaheuristic optimization-based vascular segmentation techniques for PAI. The proposed model involves two distinct kinds of vessel segmentation approaches such as Shannon’s entropy function (SEF) and multilevel Otsu thresholding (MLOT). Moreover, the threshold value and entropy function in the segmentation process are optimized using three metaheuristics such as the cuckoo search (CS), equilibrium optimizer (EO), and harmony search (HS) algorithms. A detailed experimental analysis is made on benchmark PAI dataset, and the results are inspected under varying aspects. The obtained results pointed out the supremacy of the presented model with a higher accuracy of 98.71%.

Multiple illumination learned spectral decoloring for quantitative optoacoustic oximetry imaging

SIGNIFICANCE

Quantitative measurement of blood oxygen saturation (sO2) with optoacoustic (OA) imaging is one of the most sought after goals of quantitative OA imaging research due to its wide range of biomedical applications.

AIM

A method for accurate and applicable real-time quantification of local sO2 with OA imaging.

APPROACH

We combine multiple illumination (MI) sensing with learned spectral decoloring (LSD). We train LSD feedforward neural networks and random forests on Monte Carlo simulations of spectrally colored absorbed energy spectra, to apply the trained models to real OA measurements. We validate our combined MI-LSD method on a highly reliable, reproducible, and easily scalable phantom model, based on copper and nickel sulfate solutions.

RESULTS

With this sulfate model, we see a consistently high estimation accuracy using MI-LSD, with median absolute estimation errors of 2.5 to 4.5 percentage points. We further find fewer outliers in MI-LSD estimates compared with LSD. Random forest regressors outperform previously reported neural network approaches.

CONCLUSIONS

Random forest-based MI-LSD is a promising method for accurate quantitative OA oximetry imaging.

Recent progress in near-infrared photoacoustic imaging

The emergence of the photoacoustic imaging (PAI) expands the application of biomolecules bioimaging in cells, various tissues, and living body to monitor multiple physiological processes in complex internal environments. The PAI possesses intriguing properties such as non-invasive, highly selective excitation, and weak signal attenuation. Especially, the near-infrared (NIR) PAI displays low optical absorption and scattering, good temporal or spatial resolution and deep penetration, holds great potential in biomedical applications. We briefly compare different imaging modalities to provide a comprehensive understanding of their characteristics and related applications, highlighting the feature of the PAI. The principle of PAI is then delineated and the emerging NIR-PAI is discussed. We then focus on elaboration of the recent achievement of typical NIR-PAI contrast and their biomedical applications, especially the strategies used to improve contrast rational design and PAI performance are summarized. The PAI-related multimodal imaging approaches for improving imaging accuracy are also covered in the review. Finally, the challenges and prospective are pointed out for attracting more researchers to accelerate the development of PAI.

Sounding out the hidden data: A concise review of deep learning in photoacoustic imaging

The rapidly evolving field of photoacoustic tomography utilizes endogenous chromophores to extract both functional and structural information from deep within tissues. It is this power to perform precise quantitative measurements in vivo-with endogenous or exogenous contrast-that makes photoacoustic tomography highly promising for clinical translation in functional brain imaging, early cancer detection, real-time surgical guidance, and the visualization of dynamic drug responses. Considering photoacoustic tomography has benefited from numerous engineering innovations, it is of no surprise that many of photoacoustic tomography's current cutting-edge developments incorporate advances from the equally novel field of artificial intelligence. More specifically, alongside the growth and prevalence of graphical processing unit capabilities within recent years has emerged an offshoot of artificial intelligence known as deep learning. Rooted in the solid foundation of signal processing, deep learning typically utilizes a method of optimization known as gradient descent to minimize a loss function and update model parameters. There are already a number of innovative efforts in photoacoustic tomography utilizing deep learning techniques for a variety of purposes, including resolution enhancement, reconstruction artifact removal, undersampling correction, and improved quantification. Most of these efforts have proven to be highly promising in addressing long-standing technical obstacles where traditional solutions either completely fail or make only incremental progress. This concise review focuses on the history of applied artificial intelligence in photoacoustic tomography, presents recent advances at this multifaceted intersection of fields, and outlines the most exciting advances that will likely propagate into promising future innovations.

Perspective on fast-evolving photoacoustic tomography

SIGNIFICANCE

Acoustically detecting the rich optical absorption contrast in biological tissues, photoacoustic tomography (PAT) seamlessly bridges the functional and molecular sensitivity of optical excitation with the deep penetration and high scalability of ultrasound detection. As a result of continuous technological innovations and commercial development, PAT has been playing an increasingly important role in life sciences and patient care, including functional brain imaging, smart drug delivery, early cancer diagnosis, and interventional therapy guidance.

AIM

Built on our 2016 tutorial article that focused on the principles and implementations of PAT, this perspective aims to provide an update on the exciting technical advances in PAT.

APPROACH

This perspective focuses on the recent PAT innovations in volumetric deep-tissue imaging, high-speed wide-field microscopic imaging, high-sensitivity optical ultrasound detection, and machine-learning enhanced image reconstruction and data processing. Representative applications are introduced to demonstrate these enabling technical breakthroughs in biomedical research.

CONCLUSIONS

We conclude the perspective by discussing the future development of PAT technologies.

Deep learning for biomedical photoacoustic imaging: A review

Photoacoustic imaging (PAI) is a promising emerging imaging modality that enables spatially resolved imaging of optical tissue properties up to several centimeters deep in tissue, creating the potential for numerous exciting clinical applications. However, extraction of relevant tissue parameters from the raw data requires the solving of inverse image reconstruction problems, which have proven extremely difficult to solve. The application of deep learning methods has recently exploded in popularity, leading to impressive successes in the context of medical imaging and also finding first use in the field of PAI. Deep learning methods possess unique advantages that can facilitate the clinical translation of PAI, such as extremely fast computation times and the fact that they can be adapted to any given problem. In this review, we examine the current state of the art regarding deep learning in PAI and identify potential directions of research that will help to reach the goal of clinical applicability.

Role of blood oxygenation saturation in ovarian cancer diagnosis using multi-spectral photoacoustic tomography

In photoacoustic tomography (PAT), a tunable laser typically illuminates the tissue at multiple wavelengths, and the received photoacoustic waves are used to form functional images of relative total haemoglobin (rHbT) and blood oxygenation saturation (%sO2 ). Due to measurement errors, the estimation of these parameters can be challenging, especially in clinical studies. In this study, we use a multi-pixel method to smooth the measurements before calculating rHbT and %sO2 . We first perform phantom studies using blood tubes of calibrated %sO2 to evaluate the accuracy of our %sO2 estimation. We conclude by presenting diagnostic results from PAT of 33 patients with 51 ovarian masses imaged by our co-registered PAT and ultrasound system. The ovarian masses were divided into malignant and benign/normal groups. Functional maps of rHbT and %sO2 and their histograms as well as spectral features were calculated using the PAT data from all ovaries in these two groups. Support vector machine models were trained on different combinations of the significant features. The area under ROC (AUC) of 0.93 (0.95%CI: 0.90-0.96) on the testing data set was achieved by combining mean %sO2 , a spectral feature, and the score of the study radiologist.

Deep learning in photoacoustic imaging: a review

SIGNIFICANCE

Photoacoustic (PA) imaging can provide structural, functional, and molecular information for preclinical and clinical studies. For PA imaging (PAI), non-ideal signal detection deteriorates image quality, and quantitative PAI (QPAI) remains challenging due to the unknown light fluence spectra in deep tissue. In recent years, deep learning (DL) has shown outstanding performance when implemented in PAI, with applications in image reconstruction, quantification, and understanding.

AIM

We provide (i) a comprehensive overview of the DL techniques that have been applied in PAI, (ii) references for designing DL models for various PAI tasks, and (iii) a summary of the future challenges and opportunities.

APPROACH

Papers published before November 2020 in the area of applying DL in PAI were reviewed. We categorized them into three types: image understanding, reconstruction of the initial pressure distribution, and QPAI.

RESULTS

When applied in PAI, DL can effectively process images, improve reconstruction quality, fuse information, and assist quantitative analysis.

CONCLUSION

DL has become a powerful tool in PAI. With the development of DL theory and technology, it will continue to boost the performance and facilitate the clinical translation of PAI.

A three-dimensional modeling method for quantitative photoacoustic breast imaging with handheld probe

By providing complementary functional information, photoacoustic (PA) breast imaging based on the handheld ultrasound (US) probe has demonstrated promising potential for breast cancer diagnosis. However, the quantitative PA imaging primarily relies on the knowledge of the optical fluence distribution in the three-dimensional (3D) heterogeneous breast tissue. Previous studies based on the handheld system generally provided two-dimensional (2D) B-scan results, which contains limited anatomical information of the tissue and the lesion. This study proposed a method to perform 3D modeling of the photon transportation for dual-modality PA/US system based on the local 3D breast anatomical information by scanning US probe. Then the calculated optical fluence distribution can be used for PA imaging. Our phantom and clinical pilot study results demonstrated that this method has potential to improve the accuracy of the quantitative PA breast imaging, and it can also be used in other clinical implementations.

Review of deep learning for photoacoustic imaging

Machine learning has been developed dramatically and witnessed a lot of applications in various fields over the past few years. This boom originated in 2009, when a new model emerged, that is, the deep artificial neural network, which began to surpass other established mature models on some important benchmarks. Later, it was widely used in academia and industry. Ranging from image analysis to natural language processing, it fully exerted its magic and now become the state-of-the-art machine learning models. Deep neural networks have great potential in medical imaging technology, medical data analysis, medical diagnosis and other healthcare issues, and is promoted in both pre-clinical and even clinical stages. In this review, we performed an overview of some new developments and challenges in the application of machine learning to medical image analysis, with a special focus on deep learning in photoacoustic imaging. The aim of this review is threefold: (i) introducing deep learning with some important basics, (ii) reviewing recent works that apply deep learning in the entire ecological chain of photoacoustic imaging, from image reconstruction to disease diagnosis, (iii) providing some open source materials and other resources for researchers interested in applying deep learning to photoacoustic imaging.

Compensating for visibility artefacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties

Conventional photoacoustic imaging may suffer from the limited view and bandwidth of ultrasound transducers. A deep learning approach is proposed to handle these problems and is demonstrated both in simulations and in experiments on a multi-scale model of leaf skeleton. We employed an experimental approach to build the training and the test sets using photographs of the samples as ground truth images. Reconstructions produced by the neural network show a greatly improved image quality as compared to conventional approaches. In addition, this work aimed at quantifying the reliability of the neural network predictions. To achieve this, the dropout Monte-Carlo procedure is applied to estimate a pixel-wise degree of confidence on each predicted picture. Last, we address the possibility to use transfer learning with simulated data in order to drastically limit the size of the experimental dataset.

Oxygen Saturation Imaging Using LED-Based Photoacoustic System

Oxygen saturation imaging has potential in several preclinical and clinical applications. Dual-wavelength LED array-based photoacoustic oxygen saturation imaging can be an affordable solution in this case. For the translation of this technology, there is a need to improve its accuracy and validate it against ground truth methods. We propose a fluence compensated oxygen saturation imaging method, utilizing structural information from the ultrasound image, and prior knowledge of the optical properties of the tissue with a Monte-Carlo based light propagation model for the dual-wavelength LED array configuration. We then validate the proposed method with oximeter measurements in tissue-mimicking phantoms. Further, we demonstrate in vivo imaging on small animal and a human subject. We conclude that the proposed oxygen saturation imaging can be used to image tissue at a depth of 6–8 mm in both preclinical and clinical applications.

Sound Out the Deep Colors: Photoacoustic Molecular Imaging at New Depths

Photoacoustic tomography (PAT) has become increasingly popular for molecular imaging due to its unique optical absorption contrast, high spatial resolution, deep imaging depth, and high imaging speed. Yet, the strong optical attenuation of biological tissues has traditionally prevented PAT from penetrating more than a few centimeters and limited its application for studying deeply seated targets. A variety of PAT technologies have been developed to extend the imaging depth, including employing deep-penetrating microwaves and X-ray photons as excitation sources, delivering the light to the inside of the organ, reshaping the light wavefront to better focus into scattering medium, as well as improving the sensitivity of ultrasonic transducers. At the same time, novel optical fluence mapping algorithms and image reconstruction methods have been developed to improve the quantitative accuracy of PAT, which is crucial to recover weak molecular signals at larger depths. The development of highly-absorbing near-infrared PA molecular probes has also flourished to provide high sensitivity and specificity in studying cellular processes. This review aims to introduce the recent developments in deep PA molecular imaging, including novel imaging systems, image processing methods and molecular probes, as well as their representative biomedical applications. Existing challenges and future directions are also discussed.