Modeling combined ultrasound and photoacoustic imaging: Simulations aiding device development and artificial intelligence

Review of Deep Learning Approaches for Interleaved Photoacoustic and Ultrasound (PAUS) Imaging

Photoacoustic (PA) imaging provides optical contrast at relatively large depths within the human body, compared to other optical methods, at ultrasound (US) spatial resolution. By integrating real-time PA and US (PAUS) modalities, PAUS imaging has the potential to become a routine clinical modality bringing the molecular sensitivity of optics to medical US imaging. For applications where the full capabilities of clinical US scanners must be maintained in PAUS, conventional limited view and bandwidth transducers must be used. This approach, however, cannot provide high-quality maps of PA sources, especially vascular structures. Deep learning (DL) using data-driven modeling with minimal human design has been very effective in medical imaging, medical data analysis, and disease diagnosis, and has the potential to overcome many of the technical limitations of current PAUS imaging systems. The primary purpose of this article is to summarize the background and current status of DL applications in PAUS imaging. It also looks beyond current approaches to identify remaining challenges and opportunities for robust translation of PAUS technologies to the clinic.

Artificial Intelligence-Aided Optical Imaging for Cancer Theranostics

The use of artificial intelligence (AI) to assist biomedical imaging have demonstrated its high accuracy and high efficiency in medical decision-making for individualized cancer medicine. In particular, optical imaging methods are able to visualize both the structural and functional information of tumors tissues with high contrast, low cost, and noninvasive property. However, no systematic work has been performed to inspect the recent advances on AI-aided optical imaging for cancer theranostics. In this review, we demonstrated how AI can guide optical imaging methods to improve the accuracy on tumor detection, automated analysis and prediction of its histopathological section, its monitoring during treatment, and its prognosis by using computer vision, deep learning and natural language processing. By contrast, the optical imaging techniques involved mainly consisted of various tomography and microscopy imaging methods such as optical endoscopy imaging, optical coherence tomography, photoacoustic imaging, diffuse optical tomography, optical microscopy imaging, Raman imaging, and fluorescent imaging. Meanwhile, existing problems, possible challenges and future prospects for AI-aided optical imaging protocol for cancer theranostics were also discussed. It is expected that the present work can open a new avenue for precision oncology by using AI and optical imaging tools.

Evaluation of Cholesterol Thickness of Blood Vessels Using Photoacoustic Technology

One of the primary indicators of plaque vulnerability is the lipid composition of atherosclerotic plaques. Therefore, the medical industry requires a method to evaluate necrotic nuclei in atherosclerosis imaging with sensitivity. In this regard, photoacoustic imaging is a plaque detection method that provides chemical information on lipids and cholesterol thickness in the arterial walls of the patient. This aspect aims to increase the low-frequency axial resolution by developing a new photoacoustic-based system. A photoacoustic system has been developed to detect the cholesterol thickness of the blood vessels to observe the progression of plaque in the heart's blood vessels. The application of the coherent photoacoustic discontinuous correlation tomography technique, which is based on a novel signal processing, significantly increased the cholesterol oleate's sensitivity to plaque necrosis. By enhancing the quality of thickness detection, the system for measuring the thickness of cholesterol in blood vessels has been reduced to approximately 23 microns. The results show that the phase spectrum peaked at 100 Hz at 58.66 degrees, and at 400 Hz, the phase spectrum was 46.37 degrees. The minimum amplitude is 1.95 at 100 Hz and 17.67 at 400 Hz. In conclusion, it can be stated that photoacoustic imaging as a method based on new technologies is of great importance in medical research, which is based on the use of nonionizing radiation to perform diagnostic processes and measure different types of body tissues.

Photoacoustic imaging with limited sampling: a review of machine learning approaches

Photoacoustic imaging combines high optical absorption contrast and deep acoustic penetration, and can reveal structural, molecular, and functional information about biological tissue non-invasively. Due to practical restrictions, photoacoustic imaging systems often face various challenges, such as complex system configuration, long imaging time, and/or less-than-ideal image quality, which collectively hinder their clinical application. Machine learning has been applied to improve photoacoustic imaging and mitigate the otherwise strict requirements in system setup and data acquisition. In contrast to the previous reviews of learned methods in photoacoustic computed tomography (PACT), this review focuses on the application of machine learning approaches to address the limited spatial sampling problems in photoacoustic imaging, specifically the limited view and undersampling issues. We summarize the relevant PACT works based on their training data, workflow, and model architecture. Notably, we also introduce the recent limited sampling works on the other major implementation of photoacoustic imaging, i.e., photoacoustic microscopy (PAM). With machine learning-based processing, photoacoustic imaging can achieve improved image quality with modest spatial sampling, presenting great potential for low-cost and user-friendly clinical applications.

Multiperspective Photoacoustic Imaging Using Spatially Diverse CMUTs

Photoacoustic imaging (PAI) is a promising technique to assess different constituents in tissue. In PAI, the propagating waves are low-amplitude, isotropic, and broadband. A common approach in PAI is the use of a single linear or curved piezoelectric transducer array to perform both PA and ultrasound imaging. These systems provide freedom, agility, and versatility for performing imaging, but have limited field of view (FOV) and directivity that degrade the final image quality. Capacitive micromachined ultrasonic transducers (CMUTs) have a great potential to be used for PAI since they provide larger bandwidth and better cost efficiency. In this study, to improve the FOV, resolution, and contrast, we propose a multiperspective PAI (MP-PAI) approach using multiple CMUTs on a flexible array with shared channels. The designed array was used to perform MP-PAI in an in vitro experiment using a plaque mimicking phantom where the images were compounded both incoherently and coherently. The MP-PAI approach showed a significant improvement in overall image quality. Using only three CMUTs led to about 20% increase in generalized-contrast-to-noise ratio (gCNR), 2-dB improvement in peak signal-to-noise ratio (PSNR), and double the structural coverage in comparison to a single CMUT setup. In numerical studies, the MP-PAI was thoroughly evaluated for both the coherent and incoherent compounding methods. The assessments showed that the image quality further improved for increased number of transducers and angular coverage. For 15 transducers, the improvement for resolution and contrast could be up to three times the amount in a single-perspective image. Nonetheless, the most prominent improvement of MP-PAI was its ability to resolve the structural information of the phantoms.

A Comprehensive Review on Photoacoustic-Based Devices for Biomedical Applications

The photoacoustic effect is an emerging technology that has sparked significant interest in the research field since an acoustic wave can be produced simply by the incidence of light on a material or tissue. This phenomenon has been extensively investigated, not only to perform photoacoustic imaging but also to develop highly miniaturized ultrasound probes that can provide biologically meaningful information. Therefore, this review aims to outline the materials and their fabrication process that can be employed as photoacoustic targets, both biological and non-biological, and report the main components' features to achieve a certain performance. When designing a device, it is of utmost importance to model it at an early stage for a deeper understanding and to ease the optimization process. As such, throughout this article, the different methods already implemented to model the photoacoustic effect are introduced, as well as the advantages and drawbacks inherent in each approach. However, some remaining challenges are still faced when developing such a system regarding its fabrication, modeling, and characterization, which are also discussed.

Biomedical Application of Photoacoustics: A Plethora of Opportunities

The photoacoustic (PA) technique is a non-invasive, non-ionizing hybrid technique that exploits laser irradiation for sample excitation and acquires an ultrasound signal generated due to thermoelastic expansion of the sample. Being a hybrid technique, PA possesses the inherent advantages of conventional optical (high resolution) and ultrasonic (high depth of penetration in biological tissue) techniques and eliminates some of the major limitations of these conventional techniques. Hence, PA has been employed for different biomedical applications. In this review, we first discuss the basic physics of PA. Then, we discuss different aspects of PA techniques, which includes PA imaging and also PA frequency spectral analysis. The theory of PA signal generation, detection and analysis is also detailed in this work. Later, we also discuss the major biomedical application area of PA technique.

Clinical photoacoustic/ultrasound dual-modal imaging: Current status and future trends

Photoacoustic tomography (PAT) is an emerging biomedical imaging modality that combines optical and ultrasonic imaging, providing overlapping fields of view. This hybrid approach allows for a natural integration of PAT and ultrasound (US) imaging in a single platform. Due to the similarities in signal acquisition and processing, the combination of PAT and US imaging creates a new hybrid imaging for novel clinical applications. Over the recent years, particular attention is paid to the development of PAT/US dual-modal systems highlighting mutual benefits in clinical cases, with an aim of substantially improving the specificity and sensitivity for diagnosis of diseases. The demonstrated feasibility and accuracy in these efforts open an avenue of translating PAT/US imaging to practical clinical applications. In this review, the current PAT/US dual-modal imaging systems are discussed in detail, and their promising clinical applications are presented and compared systematically. Finally, this review describes the potential impacts of these combined systems in the coming future.

Photoacoustic imaging for microcirculation

Microcirculation facilitates the blood-tissue exchange of nutrients and regulates blood perfusion. It is, therefore, essential in maintaining tissue health. Aberrations in microcirculation are potentially indicative of underlying cardiovascular and metabolic pathologies. Thus, quantitative information about it is of great clinical relevance. Photoacoustic imaging (PAI) is a capable technique that relies on the generation of imaging contrast via the absorption of light and can image at micron-scale resolution. PAI is especially desirable to map microvasculature as hemoglobin strongly absorbs light and can generate a photoacoustic signal. This paper reviews the current state of the art for imaging microvascular networks using photoacoustic imaging. We further describe how quantitative information about blood dynamics such as the total hemoglobin concentration, oxygen saturation, and blood flow rate is obtained using PAI. We also discuss its importance in understanding key pathophysiological processes in neurovascular, cardiovascular, ophthalmic, and cancer research fields. We then discuss the current challenges and limitations of PAI and the approaches that can help overcome these limitations. Finally, we provide the reader with an overview of future trends in the field of PAI for imaging microcirculation.

Modeling toolchain for realistic simulation of photoacoustic data acquisition

SIGNIFICANCE

Physics-based simulations of photoacoustic (PA) signals are used to validate new methods, to characterize PA setups and to generate training datasets for machine learning. However, a thoroughly validated PA simulation toolchain that can simulate realistic images is still lacking.

AIM

A quantitative toolchain was developed to model PA image acquisition in complex tissues, by simulating both the optical fluence and the acoustic wave propagation.

APPROACH

Sampling techniques were developed to decrease artifacts in acoustic simulations. The performance of the simulations was analyzed by measuring the point spread function (PSF) and using a rotatable three-channel phantom, filled with cholesterol, a human carotid plaque sample, and porcine blood. Ex vivo human plaque samples were simulated to validate the methods in more complex tissues.

RESULTS

The sampling techniques could enhance the quality of the simulated PA images effectively. The resolution and intensity of the PSF in the turbid medium matched the experimental data well. Overall, the appearance, signal-to-noise ratio and speckle of the images could be simulated accurately.

CONCLUSIONS

A PA toolchain was developed and validated, and the results indicate a great potential of PA simulations in more complex and heterogeneous media.

Video-rate full-ring ultrasound and photoacoustic computed tomography with real-time sound speed optimization

Full-ring dual-modal ultrasound and photoacoustic imaging provide complementary contrasts, high spatial resolution, full view angle and are more desirable in pre-clinical and clinical applications. However, two long-standing challenges exist in achieving high-quality video-rate dual-modal imaging. One is the increased data processing burden from the dense acquisition. Another one is the object-dependent speed of sound variation, which may cause blurry, splitting artifacts, and low imaging contrast. Here, we develop a video-rate full-ring ultrasound and photoacoustic computed tomography (VF-USPACT) with real-time optimization of the speed of sound. We improve the imaging speed by selective and parallel image reconstruction. We determine the optimal sound speed via co-registered ultrasound imaging. Equipped with a 256-channel ultrasound array, the dual-modal system can optimize the sound speed and reconstruct dual-modal images at 10 Hz in real-time. The optimized sound speed can effectively enhance the imaging quality under various sample sizes, types, or physiological states. In animal and human imaging, the system shows co-registered dual contrasts, high spatial resolution (140 µm), single-pulse photoacoustic imaging (< 50 µs), deep penetration (> 20 mm), full view, and adaptive sound speed correction. We believe VF-USPACT can advance many real-time biomedical imaging applications, such as vascular disease diagnosing, cancer screening, or neuroimaging.

SIMPA: an open-source toolkit for simulation and image processing for photonics and acoustics

SIGNIFICANCE

Optical and acoustic imaging techniques enable noninvasive visualisation of structural and functional properties of tissue. The quantification of measurements, however, remains challenging due to the inverse problems that must be solved. Emerging data-driven approaches are promising, but they rely heavily on the presence of high-quality simulations across a range of wavelengths due to the lack of ground truth knowledge of tissue acoustical and optical properties in realistic settings.

AIM

To facilitate this process, we present the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit. SIMPA is being developed according to modern software design standards.

APPROACH

SIMPA enables the use of computational forward models, data processing algorithms, and digital device twins to simulate realistic images within a single pipeline. SIMPA's module implementations can be seamlessly exchanged as SIMPA abstracts from the concrete implementation of each forward model and builds the simulation pipeline in a modular fashion. Furthermore, SIMPA provides comprehensive libraries of biological structures, such as vessels, as well as optical and acoustic properties and other functionalities for the generation of realistic tissue models.

RESULTS

To showcase the capabilities of SIMPA, we show examples in the context of photoacoustic imaging: the diversity of creatable tissue models, the customisability of a simulation pipeline, and the degree of realism of the simulations.

CONCLUSIONS

SIMPA is an open-source toolkit that can be used to simulate optical and acoustic imaging modalities. The code is available at: https://github.com/IMSY-DKFZ/simpa, and all of the examples and experiments in this paper can be reproduced using the code available at: https://github.com/IMSY-DKFZ/simpa_paper_experiments.

Simultaneous Denoising and Localization Network for Photoacoustic Target Localization

A significant research problem of recent interest is the localization of targets like vessels, surgical needles, and tumors in photoacoustic (PA) images.To achieve accurate localization, a high photoacoustic signal-to-noise ratio (SNR) is required. However, this is not guaranteed for deep targets, as optical scattering causes an exponential decay in optical fluence with respect to tissue depth. To address this, we develop a novel deep learning method designed to explicitly exhibit robustness to noise present in photoacoustic radio-frequency (RF) data. More precisely, we describe and evaluate a deep neural network architecture consisting of a shared encoder and two parallel decoders. One decoder extracts the target coordinates from the input RF data while the other boosts the SNR and estimates clean RF data. The joint optimization of the shared encoder and dual decoders lends significant noise robustness to the features extracted by the encoder, which in turn enables the network to contain detailed information about deep targets that may be obscured by noise. Additional custom layers and newly proposed regularizers in the training loss function (designed based on observed RF data signal and noise behavior) serve to increase the SNR in the cleaned RF output and improve model performance. To account for depth-dependent strong optical scattering, our network was trained with simulated photoacoustic datasets of targets embedded at different depths inside tissue media of different scattering levels. The network trained on this novel dataset accurately locates targets in experimental PA data that is clinically relevant with respect to the localization of vessels, needles, or brachytherapy seeds. We verify the merits of the proposed architecture by outperforming the state of the art on both simulated and experimental datasets.