1
|
Schellenberg M, Gröhl J, Dreher KK, Nölke JH, Holzwarth N, Tizabi MD, Seitel A, Maier-Hein L. Photoacoustic image synthesis with generative adversarial networks. PHOTOACOUSTICS 2022; 28:100402. [PMID: 36281320 PMCID: PMC9587371 DOI: 10.1016/j.pacs.2022.100402] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Revised: 09/03/2022] [Accepted: 09/08/2022] [Indexed: 06/16/2023]
Abstract
Photoacoustic tomography (PAT) has the potential to recover morphological and functional tissue properties with high spatial resolution. However, previous attempts to solve the optical inverse problem with supervised machine learning were hampered by the absence of labeled reference data. While this bottleneck has been tackled by simulating training data, the domain gap between real and simulated images remains an unsolved challenge. We propose a novel approach to PAT image synthesis that involves subdividing the challenge of generating plausible simulations into two disjoint problems: (1) Probabilistic generation of realistic tissue morphology, and (2) pixel-wise assignment of corresponding optical and acoustic properties. The former is achieved with Generative Adversarial Networks (GANs) trained on semantically annotated medical imaging data. According to a validation study on a downstream task our approach yields more realistic synthetic images than the traditional model-based approach and could therefore become a fundamental step for deep learning-based quantitative PAT (qPAT).
Collapse
Affiliation(s)
- Melanie Schellenberg
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- HIDSS4Health - Helmholtz Information and Data Science School for Health, Heidelberg, Germany
| | - Janek Gröhl
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Cancer Research UK Cambridge Institute, University of Cambridge, Robinson Way, Cambridge, CB2 0RE, UK
| | - Kris K. Dreher
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
| | - Jan-Hinrich Nölke
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Niklas Holzwarth
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Minu D. Tizabi
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Seitel
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Lena Maier-Hein
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- HIDSS4Health - Helmholtz Information and Data Science School for Health, Heidelberg, Germany
- Medical Faculty, Heidelberg University, Heidelberg, Germany
- HIP Applied Computer Vision Lab, Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
2
|
Madasamy A, Gujrati V, Ntziachristos V, Prakash J. Deep learning methods hold promise for light fluence compensation in three-dimensional optoacoustic imaging. JOURNAL OF BIOMEDICAL OPTICS 2022; 27:106004. [PMID: 36209354 PMCID: PMC9547608 DOI: 10.1117/1.jbo.27.10.106004] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 08/30/2022] [Indexed: 06/16/2023]
Abstract
SIGNIFICANCE Quantitative optoacoustic imaging (QOAI) continues to be a challenge due to the influence of nonlinear optical fluence distribution, which distorts the optoacoustic image representation. Nonlinear optical fluence correction in OA imaging is highly ill-posed, leading to the inaccurate recovery of optical absorption maps. This work aims to recover the optical absorption maps using deep learning (DL) approach by correcting for the fluence effect. AIM Different DL models were compared and investigated to enable optical absorption coefficient recovery at a particular wavelength in a nonhomogeneous foreground and background medium. APPROACH Data-driven models were trained with two-dimensional (2D) Blood vessel and three-dimensional (3D) numerical breast phantom with highly heterogeneous/realistic structures to correct for the nonlinear optical fluence distribution. The trained DL models such as U-Net, Fully Dense (FD) U-Net, Y-Net, FD Y-Net, Deep residual U-Net (Deep ResU-Net), and generative adversarial network (GAN) were tested to evaluate the performance of optical absorption coefficient recovery (or fluence compensation) with in-silico and in-vivo datasets. RESULTS The results indicated that FD U-Net-based deconvolution improves by about 10% over reconstructed optoacoustic images in terms of peak-signal-to-noise ratio. Further, it was observed that DL models can indeed highlight deep-seated structures with higher contrast due to fluence compensation. Importantly, the DL models were found to be about 17 times faster than solving diffusion equation for fluence correction. CONCLUSIONS The DL methods were able to compensate for nonlinear optical fluence distribution more effectively and improve the optoacoustic image quality.
Collapse
Affiliation(s)
- Arumugaraj Madasamy
- Indian Institute of Science, Department of Instrumentation and Applied Physics, Bengaluru, Karnataka, India
| | - Vipul Gujrati
- Institute of Biological and Medical Imaging, Helmholtz Zentrum München (GmbH), Neuherberg, Germany
- Technical University of Munich, School of Medicine, Chair of Biological Imaging, Munich, Germany
| | - Vasilis Ntziachristos
- Institute of Biological and Medical Imaging, Helmholtz Zentrum München (GmbH), Neuherberg, Germany
- Technical University of Munich, School of Medicine, Chair of Biological Imaging, Munich, Germany
- Technical University of Munich, Munich Institute of Robotics and Machine Intelligence (MIRMI), Munich, Germany
| | - Jaya Prakash
- Indian Institute of Science, Department of Instrumentation and Applied Physics, Bengaluru, Karnataka, India
| |
Collapse
|
3
|
Zuo H, Cui M, Wang X, Ma C. Spectral crosstalk in photoacoustic computed tomography. PHOTOACOUSTICS 2022; 26:100356. [PMID: 35574185 PMCID: PMC9095891 DOI: 10.1016/j.pacs.2022.100356] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Revised: 04/04/2022] [Accepted: 04/11/2022] [Indexed: 06/15/2023]
Abstract
Multispectral photoacoustic (PA) imaging faces two major challenges: the spectral coloring effect, which has been studied extensively as an optical inversion problem, and the spectral crosstalk, which is basically a result of non-ideal acoustic inversion. So far, there is no systematic work to analyze the spectral crosstalk because acoustic inversion and spectroscopic measurement are always treated as decoupled. In this work, we theorize and demonstrate through a series of simulations and experiments how imperfect acoustic inversion induces inaccurate PA spectrum measurement. We provide detailed analysis to elucidate how different factors, including limited bandwidth, limited view, light attenuation, out-of-plane signal, and image reconstruction schemes, conspire to render the measured PA spectrum inaccurate. We found that the model-based reconstruction outperforms universal back-projection in suppressing the spectral crosstalk in some cases.
Collapse
Affiliation(s)
- Hongzhi Zuo
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
| | - Manxiu Cui
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
| | - Xuanhao Wang
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
| | - Cheng Ma
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
- Center for Clinical Big Data Research, Institute of Precision Medicine, Tsinghua University, Beijing 100084, China
- Photomedicine Laboratory, Institute of Precision Medicine, Tsinghua University, Beijing 100084, China
| |
Collapse
|
4
|
Gröhl J, Hacker L, Cox BT, Dreher KK, Morscher S, Rakotondrainibe A, Varray F, Yip LC, Vogt WC, Bohndiek SE. The IPASC data format: A consensus data format for photoacoustic imaging. PHOTOACOUSTICS 2022; 26:100339. [PMID: 35287304 PMCID: PMC8917284 DOI: 10.1016/j.pacs.2022.100339] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Revised: 02/15/2022] [Accepted: 02/17/2022] [Indexed: 06/14/2023]
Abstract
Photoacoustic imaging (PAI) is an emerging modality that has shown promise for improving patient management in a range of applications. Unfortunately, the current lack of uniformity in PAI data formats compromises inter-user data exchange and comparison, which impedes: technological progress; effective research collaboration; and efforts to deliver multi-centre clinical trials. To overcome this challenge, the International Photoacoustic Standardisation Consortium (IPASC) has established a data format with a defined consensus metadata structure and developed an open-source software application programming interface (API) to enable conversion from proprietary file formats into the IPASC format. The format is based on Hierarchical Data Format 5 (HDF5) and designed to store photoacoustic raw time series data. Internal quality control mechanisms are included to ensure completeness and consistency of the converted data. By unifying the variety of proprietary data and metadata definitions into a consensus format, IPASC hopes to facilitate the exchange and comparison of PAI data.
Collapse
Affiliation(s)
- Janek Gröhl
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, United Kingdom
- Department of Physics, University of Cambridge, Cambridge, United Kingdom
| | - Lina Hacker
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, United Kingdom
- Department of Physics, University of Cambridge, Cambridge, United Kingdom
| | - Ben T. Cox
- Department of Medical Physics and Biomedical Engineering, University College London, London, United Kingdom
| | - Kris K. Dreher
- German Cancer Research Center, Division of Computer Assisted Medical Interventions, Heidelberg, Germany
- Heidelberg University, Faculty of Physics and Astronomy, Heidelberg, Germany
| | | | | | - François Varray
- Univ Lyon, INSA-Lyon, Université Claude Bernard Lyon 1, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621, Lyon, France
| | - Lawrence C.M. Yip
- Department of Medical Biophysics, Schulich School of Medicine and Dentistry, Western University, London, Canada
- Imaging Program, Lawson Health Research Institute, London, Canada
| | - William C. Vogt
- Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, MD, United States
| | - Sarah E. Bohndiek
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, United Kingdom
- Department of Physics, University of Cambridge, Cambridge, United Kingdom
| | | |
Collapse
|
5
|
Kirchner T, Jaeger M, Frenz M. Machine learning enabled multiple illumination quantitative optoacoustic oximetry imaging in humans. BIOMEDICAL OPTICS EXPRESS 2022; 13:2655-2667. [PMID: 35774340 PMCID: PMC9203099 DOI: 10.1364/boe.455514] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Revised: 03/25/2022] [Accepted: 03/26/2022] [Indexed: 06/15/2023]
Abstract
Optoacoustic (OA) imaging is a promising modality for quantifying blood oxygen saturation (sO2) in various biomedical applications - in diagnosis, monitoring of organ function, or even tumor treatment planning. We present an accurate and practically feasible real-time capable method for quantitative imaging of sO2 based on combining multispectral (MS) and multiple illumination (MI) OA imaging with learned spectral decoloring (LSD). For this purpose we developed a hybrid real-time MI MS OA imaging setup with ultrasound (US) imaging capability; we trained gradient boosting machines on MI spectrally colored absorbed energy spectra generated by generic Monte Carlo simulations and used the trained models to estimate sO2 on real OA measurements. We validated MI-LSD in silico and on in vivo image sequences of radial arteries and accompanying veins of five healthy human volunteers. We compared the performance of the method to prior LSD work and conventional linear unmixing. MI-LSD provided highly accurate results in silico and consistently plausible results in vivo. This preliminary study shows a potentially high applicability of quantitative OA oximetry imaging, using our method.
Collapse
Affiliation(s)
- Thomas Kirchner
- Institut für Physik, Martin-Luther-Universität Halle-Wittenberg, Halle (Saale), Germany
- Biomedical Photonics, Institute of Applied Physics, University of Bern, Bern, Switzerland
| | - Michael Jaeger
- Biomedical Photonics, Institute of Applied Physics, University of Bern, Bern, Switzerland
| | - Martin Frenz
- Biomedical Photonics, Institute of Applied Physics, University of Bern, Bern, Switzerland
| |
Collapse
|
6
|
Zheng S, Meng Q, Wang XY. Quantitative endoscopic photoacoustic tomography using a convolutional neural network. APPLIED OPTICS 2022; 61:2574-2581. [PMID: 35471325 DOI: 10.1364/ao.441250] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/24/2021] [Accepted: 02/23/2022] [Indexed: 06/14/2023]
Abstract
Endoscopic photoacoustic tomography (EPAT) is a catheter-based hybrid imaging modality capable of providing structural and functional information of biological luminal structures, such as coronary arterial vessels and the digestive tract. The recovery of the optical properties of the imaged tissue from acoustic measurements achieved by optical inversion is essential for implementing quantitative EPAT (qEPAT). In this paper, a convolutional neural network (CNN) based on deep gradient descent is developed for qEPAT. The network enables the reconstruction of images representing the spatially varying absorption coefficient in cross-sections of the tubular structures from limited measurement data. The forward operator reflecting the mapping from the absorption coefficient to the optical deposition due to pulsed irradiation is embedded into the network training. The network parameters are optimized layer by layer through the deep gradient descent mechanism using the numerically simulated data. The operation processes of the forward operator and its adjoint operator are separated from the network training. The trained network outputs an image representing the distribution of absorption coefficients by inputting an image that represents the optical deposition. The method has been tested with computer-generated phantoms mimicking coronary arterial vessels containing various tissue types. Results suggest that the structural similarity of the images reconstructed by our method is increased by about 10% in comparison with the non-learning method based on error minimization in the case of the same measuring view.
Collapse
|
7
|
Kulkarni PH, Merchant S, Awate SP. Mixed-Dictionary Models and Variational Inference in Task fMRI for Shorter Scans and Better Image Quality. Med Image Anal 2022; 78:102392. [DOI: 10.1016/j.media.2022.102392] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2021] [Revised: 10/31/2021] [Accepted: 02/10/2022] [Indexed: 11/28/2022]
|
8
|
Photoacoustic imaging aided with deep learning: a review. Biomed Eng Lett 2021; 12:155-173. [DOI: 10.1007/s13534-021-00210-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2021] [Revised: 10/19/2021] [Accepted: 11/07/2021] [Indexed: 12/21/2022] Open
|
9
|
Wang Y, Xu M, Gao F, Kang F, Zhu S. Nonlinear iterative perturbation scheme with simplified spherical harmonics (SP 3 ) light propagation model for quantitative photoacoustic tomography. JOURNAL OF BIOPHOTONICS 2021; 14:e202000446. [PMID: 33576563 DOI: 10.1002/jbio.202000446] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/06/2020] [Revised: 01/31/2021] [Accepted: 02/06/2021] [Indexed: 06/12/2023]
Abstract
When using quantitative photoacoustic tomography (q-PAT) reconstruction to recover the optical absorption coefficients of tissue, the commonly used diffusion equation has several limitations in the case of the objects that have small geometries and high-absorption or low-scattering areas. Furthermore, the conventional perturbation reconstruction strategy is unsatisfactory when the target tissue containing large heterogeneous features. We herein present a modified q-PAT implementation that employs the higher-order photon migration model achieving the tradeoff between mathematical rigidity and computational efficiency. Besides, a nonlinear iterative method is proposed to obtain the perturbations of optical absorption considering the updating of the sensitivity matrix in calculating the fluence perturbations. Consequently, the distribution of tissue optical properties can be recovered in a robust way even if the targets with high absorption are included. The proposed approach has been validated by simulation, phantom and in vivo experiments, exhibiting promising performances in image fidelity and quantitative feasibility for practical applications.
Collapse
Affiliation(s)
- Yihan Wang
- School of Life Science and Technology, Xidian University, Xi'an, China
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, Xi'an, China
| | - Menglu Xu
- School of Life Science and Technology, Xidian University, Xi'an, China
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, Xi'an, China
| | - Feng Gao
- College of Precision Instrument and Optoelectronics Engineering, Tianjin University, Tianjin, China
- Tianjin Key Laboratory of Biomedical Detecting Techniques and Instruments, Tianjin University, Tianjin, China
| | - Fei Kang
- Department of Nuclear Medicine, Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Shouping Zhu
- School of Life Science and Technology, Xidian University, Xi'an, China
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, Xi'an, China
| |
Collapse
|
10
|
Gröhl J, Schellenberg M, Dreher K, Maier-Hein L. Deep learning for biomedical photoacoustic imaging: A review. PHOTOACOUSTICS 2021; 22:100241. [PMID: 33717977 PMCID: PMC7932894 DOI: 10.1016/j.pacs.2021.100241] [Citation(s) in RCA: 86] [Impact Index Per Article: 28.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2020] [Revised: 01/18/2021] [Accepted: 01/20/2021] [Indexed: 05/04/2023]
Abstract
Photoacoustic imaging (PAI) is a promising emerging imaging modality that enables spatially resolved imaging of optical tissue properties up to several centimeters deep in tissue, creating the potential for numerous exciting clinical applications. However, extraction of relevant tissue parameters from the raw data requires the solving of inverse image reconstruction problems, which have proven extremely difficult to solve. The application of deep learning methods has recently exploded in popularity, leading to impressive successes in the context of medical imaging and also finding first use in the field of PAI. Deep learning methods possess unique advantages that can facilitate the clinical translation of PAI, such as extremely fast computation times and the fact that they can be adapted to any given problem. In this review, we examine the current state of the art regarding deep learning in PAI and identify potential directions of research that will help to reach the goal of clinical applicability.
Collapse
Affiliation(s)
- Janek Gröhl
- German Cancer Research Center, Computer Assisted Medical Interventions, Heidelberg, Germany
- Heidelberg University, Medical Faculty, Heidelberg, Germany
| | - Melanie Schellenberg
- German Cancer Research Center, Computer Assisted Medical Interventions, Heidelberg, Germany
| | - Kris Dreher
- German Cancer Research Center, Computer Assisted Medical Interventions, Heidelberg, Germany
- Heidelberg University, Faculty of Physics and Astronomy, Heidelberg, Germany
| | - Lena Maier-Hein
- German Cancer Research Center, Computer Assisted Medical Interventions, Heidelberg, Germany
- Heidelberg University, Medical Faculty, Heidelberg, Germany
- Heidelberg University, Faculty of Mathematics and Computer Science, Heidelberg, Germany
| |
Collapse
|
11
|
Deng H, Qiao H, Dai Q, Ma C. Deep learning in photoacoustic imaging: a review. JOURNAL OF BIOMEDICAL OPTICS 2021; 26:JBO-200374VRR. [PMID: 33837678 PMCID: PMC8033250 DOI: 10.1117/1.jbo.26.4.040901] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/18/2020] [Accepted: 03/18/2021] [Indexed: 05/18/2023]
Abstract
SIGNIFICANCE Photoacoustic (PA) imaging can provide structural, functional, and molecular information for preclinical and clinical studies. For PA imaging (PAI), non-ideal signal detection deteriorates image quality, and quantitative PAI (QPAI) remains challenging due to the unknown light fluence spectra in deep tissue. In recent years, deep learning (DL) has shown outstanding performance when implemented in PAI, with applications in image reconstruction, quantification, and understanding. AIM We provide (i) a comprehensive overview of the DL techniques that have been applied in PAI, (ii) references for designing DL models for various PAI tasks, and (iii) a summary of the future challenges and opportunities. APPROACH Papers published before November 2020 in the area of applying DL in PAI were reviewed. We categorized them into three types: image understanding, reconstruction of the initial pressure distribution, and QPAI. RESULTS When applied in PAI, DL can effectively process images, improve reconstruction quality, fuse information, and assist quantitative analysis. CONCLUSION DL has become a powerful tool in PAI. With the development of DL theory and technology, it will continue to boost the performance and facilitate the clinical translation of PAI.
Collapse
Affiliation(s)
- Handi Deng
- Tsinghua University, Department of Electronic Engineering, Haidian, Beijing, China
| | - Hui Qiao
- Tsinghua University, Department of Automation, Haidian, Beijing, China
- Tsinghua University, Institute for Brain and Cognitive Science, Beijing, China
- Tsinghua University, Beijing Laboratory of Brain and Cognitive Intelligence, Beijing, China
- Tsinghua University, Beijing Key Laboratory of Multi-Dimension and Multi-Scale Computational Photography, Beijing, China
| | - Qionghai Dai
- Tsinghua University, Department of Automation, Haidian, Beijing, China
- Tsinghua University, Institute for Brain and Cognitive Science, Beijing, China
- Tsinghua University, Beijing Laboratory of Brain and Cognitive Intelligence, Beijing, China
- Tsinghua University, Beijing Key Laboratory of Multi-Dimension and Multi-Scale Computational Photography, Beijing, China
| | - Cheng Ma
- Tsinghua University, Department of Electronic Engineering, Haidian, Beijing, China
- Beijing Innovation Center for Future Chip, Beijing, China
| |
Collapse
|
12
|
Gröhl J, Kirchner T, Adler TJ, Hacker L, Holzwarth N, Hernández-Aguilera A, Herrera MA, Santos E, Bohndiek SE, Maier-Hein L. Learned spectral decoloring enables photoacoustic oximetry. Sci Rep 2021; 11:6565. [PMID: 33753769 PMCID: PMC7985523 DOI: 10.1038/s41598-021-83405-8] [Citation(s) in RCA: 26] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2020] [Accepted: 01/27/2021] [Indexed: 01/15/2023] Open
Abstract
The ability of photoacoustic imaging to measure functional tissue properties, such as blood oxygenation sO[Formula: see text], enables a wide variety of possible applications. sO[Formula: see text] can be computed from the ratio of oxyhemoglobin HbO[Formula: see text] and deoxyhemoglobin Hb, which can be distuinguished by multispectral photoacoustic imaging due to their distinct wavelength-dependent absorption. However, current methods for estimating sO[Formula: see text] yield inaccurate results in realistic settings, due to the unknown and wavelength-dependent influence of the light fluence on the signal. In this work, we propose learned spectral decoloring to enable blood oxygenation measurements to be inferred from multispectral photoacoustic imaging. The method computes sO[Formula: see text] pixel-wise, directly from initial pressure spectra [Formula: see text], which represent initial pressure values at a fixed spatial location [Formula: see text] over all recorded wavelengths [Formula: see text]. The method is compared to linear unmixing approaches, as well as pO[Formula: see text] and blood gas analysis reference measurements. Experimental results suggest that the proposed method is able to obtain sO[Formula: see text] estimates from multispectral photoacoustic measurements in silico, in vitro, and in vivo.
Collapse
Affiliation(s)
- Janek Gröhl
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany.
- Medical Faculty, Heidelberg University, Heidelberg, Germany.
| | - Thomas Kirchner
- Institute of Applied Physics, Biomedical Photonics, Bern University, Bern, Switzerland
| | - Tim J Adler
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Lina Hacker
- Department of Physics, University of Cambridge, JJ Thomson Avenue, Cambridge, CB3 0HE, UK
| | - Niklas Holzwarth
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany
- Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
| | | | - Mildred A Herrera
- Department of Neurosurgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Edgar Santos
- Department of Neurosurgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Sarah E Bohndiek
- Department of Physics, University of Cambridge, JJ Thomson Avenue, Cambridge, CB3 0HE, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Robinson Way, Cambridge, CB2 0RE, UK
| | - Lena Maier-Hein
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany.
- Medical Faculty, Heidelberg University, Heidelberg, Germany.
| |
Collapse
|
13
|
Godefroy G, Arnal B, Bossy E. Compensating for visibility artefacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties. PHOTOACOUSTICS 2021; 21:100218. [PMID: 33364161 PMCID: PMC7750172 DOI: 10.1016/j.pacs.2020.100218] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/24/2020] [Revised: 10/15/2020] [Accepted: 10/17/2020] [Indexed: 05/04/2023]
Abstract
Conventional photoacoustic imaging may suffer from the limited view and bandwidth of ultrasound transducers. A deep learning approach is proposed to handle these problems and is demonstrated both in simulations and in experiments on a multi-scale model of leaf skeleton. We employed an experimental approach to build the training and the test sets using photographs of the samples as ground truth images. Reconstructions produced by the neural network show a greatly improved image quality as compared to conventional approaches. In addition, this work aimed at quantifying the reliability of the neural network predictions. To achieve this, the dropout Monte-Carlo procedure is applied to estimate a pixel-wise degree of confidence on each predicted picture. Last, we address the possibility to use transfer learning with simulated data in order to drastically limit the size of the experimental dataset.
Collapse
Affiliation(s)
| | - Bastien Arnal
- Univ. Grenoble Alpes, CNRS, LIPhy, 38000 Grenoble, France
| | - Emmanuel Bossy
- Univ. Grenoble Alpes, CNRS, LIPhy, 38000 Grenoble, France
| |
Collapse
|
14
|
Olefir I, Tzoumas S, Restivo C, Mohajerani P, Xing L, Ntziachristos V. Deep Learning-Based Spectral Unmixing for Optoacoustic Imaging of Tissue Oxygen Saturation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:3643-3654. [PMID: 32746111 PMCID: PMC7671861 DOI: 10.1109/tmi.2020.3001750] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/18/2023]
Abstract
Label free imaging of oxygenation distribution in tissues is highly desired in numerous biomedical applications, but is still elusive, in particular in sub-epidermal measurements. Eigenspectra multispectral optoacoustic tomography (eMSOT) and its Bayesian-based implementation have been introduced to offer accurate label-free blood oxygen saturation (sO2) maps in tissues. The method uses the eigenspectra model of light fluence in tissue to account for the spectral changes due to the wavelength dependent attenuation of light with tissue depth. eMSOT relies on the solution of an inverse problem bounded by a number of ad hoc hand-engineered constraints. Despite the quantitative advantage offered by eMSOT, both the non-convex nature of the optimization problem and the possible sub-optimality of the constraints may lead to reduced accuracy. We present herein a neural network architecture that is able to learn how to solve the inverse problem of eMSOT by directly regressing from a set of input spectra to the desired fluence values. The architecture is composed of a combination of recurrent and convolutional layers and uses both spectral and spatial features for inference. We train an ensemble of such networks using solely simulated data and demonstrate how this approach can improve the accuracy of sO2 computation over the original eMSOT, not only in simulations but also in experimental datasets obtained from blood phantoms and small animals (mice) in vivo. The use of a deep-learning approach in optoacoustic sO2 imaging is confirmed herein for the first time on ground truth sO2 values experimentally obtained in vivo and ex vivo.
Collapse
|
15
|
Feng J, Deng J, Li Z, Sun Z, Dou H, Jia K. End-to-end Res-Unet based reconstruction algorithm for photoacoustic imaging. BIOMEDICAL OPTICS EXPRESS 2020; 11:5321-5340. [PMID: 33014617 PMCID: PMC7510873 DOI: 10.1364/boe.396598] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/01/2020] [Revised: 07/30/2020] [Accepted: 08/20/2020] [Indexed: 05/03/2023]
Abstract
Recently, deep neural networks have attracted great attention in photoacoustic imaging (PAI). In PAI, reconstructing the initial pressure distribution from acquired photoacoustic (PA) signals is a typically inverse problem. In this paper, an end-to-end Unet with residual blocks (Res-Unet) is designed and trained to solve the inverse problem in PAI. The performance of the proposed algorithm is explored and analyzed by comparing a recent model-resolution-based regularization algorithm (MRR) with numerical and physical phantom experiments. The improvement obtained in the reconstructed images was more than 95% in pearson correlation and 39% in peak signal-to-noise ratio in comparison to the MRR. The Res-Unet also achieved superior performance over the state-of-the-art Unet++ architecture by more than 18% in PSNR in simulation experiments.
Collapse
Affiliation(s)
- Jinchao Feng
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Jianguang Deng
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Zhe Li
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Zhonghua Sun
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| | - Huijing Dou
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Kebin Jia
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
- Beijing Laboratory of Advanced Information Networks, Beijing 100124, China
| |
Collapse
|
16
|
Bench C, Hauptmann A, Cox B. Toward accurate quantitative photoacoustic imaging: learning vascular blood oxygen saturation in three dimensions. JOURNAL OF BIOMEDICAL OPTICS 2020; 25:jbo-200119R. [PMID: 32840068 PMCID: PMC7443711 DOI: 10.1117/1.jbo.25.8.085003] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/23/2020] [Accepted: 07/23/2020] [Indexed: 05/02/2023]
Abstract
SIGNIFICANCE Two-dimensional (2-D) fully convolutional neural networks have been shown capable of producing maps of sO2 from 2-D simulated images of simple tissue models. However, their potential to produce accurate estimates in vivo is uncertain as they are limited by the 2-D nature of the training data when the problem is inherently three-dimensional (3-D), and they have not been tested with realistic images. AIM To demonstrate the capability of deep neural networks to process whole 3-D images and output 3-D maps of vascular sO2 from realistic tissue models/images. APPROACH Two separate fully convolutional neural networks were trained to produce 3-D maps of vascular blood oxygen saturation and vessel positions from multiwavelength simulated images of tissue models. RESULTS The mean of the absolute difference between the true mean vessel sO2 and the network output for 40 examples was 4.4% and the standard deviation was 4.5%. CONCLUSIONS 3-D fully convolutional networks were shown capable of producing accurate sO2 maps using the full extent of spatial information contained within 3-D images generated under conditions mimicking real imaging scenarios. We demonstrate that networks can cope with some of the confounding effects present in real images such as limited-view artifacts and have the potential to produce accurate estimates in vivo.
Collapse
Affiliation(s)
- Ciaran Bench
- University College London, Department of Medical Physics and Biomedical Engineering, Gower Street, London, United Kingdom
- Address all correspondence to Ciaran Bench, E-mail:
| | - Andreas Hauptmann
- University of Oulu, Research Unit of Mathematical Sciences, Oulu, Finland
- University College London, Department of Computer Science, Gower Street, London, United Kingdom
| | - Ben Cox
- University College London, Department of Medical Physics and Biomedical Engineering, Gower Street, London, United Kingdom
| |
Collapse
|
17
|
Kulkarni PH, Merchant SN, Awate SP. R-fMRI reconstruction from k-t undersampled data using a subject-invariant dictionary model and VB-EM with nested minorization. Med Image Anal 2020; 65:101752. [PMID: 32623273 DOI: 10.1016/j.media.2020.101752] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2019] [Revised: 06/01/2020] [Accepted: 06/04/2020] [Indexed: 11/30/2022]
Abstract
Higher spatial resolution in resting-state functional magnetic resonance imaging (R-fMRI) can give reliable information about the functional networks in the cerebral cortex. Typical methods can achieve higher spatial or temporal resolution by speeding up scans using either (i) complex pulse-sequence designs or (ii) k-space undersampling coupled with priors on the signal. We propose to undersample the R-fMRI acquisition in k-space and time to speedup scans in order to improve spatial resolution. We propose a novel model-based R-fMRI reconstruction framework using a robust, subject-invariant, spatially regularized dictionary prior on the signal. Furthermore, we propose a novel inference framework based on variational Bayesian expectation maximization with nested minorization (VB-EM-NM). Our inference framework allows us to provide an estimate of uncertainty of the reconstruction, unlike typical reconstruction methods. Empirical evaluation of (i) simulated R-fMRI reconstruction and (ii) functional-network estimates from brain R-fMRI reconstructions demonstrate that our framework improves over the state of the art, and, additionally, enables significantly higher spatial resolution.
Collapse
Affiliation(s)
- Prachi H Kulkarni
- Electrical Engineering (EE) Department, Indian Institute of Technology (IIT) Bombay, Mumbai, India.
| | - S N Merchant
- Electrical Engineering (EE) Department, Indian Institute of Technology (IIT) Bombay, Mumbai, India
| | - Suyash P Awate
- Computer Science and Engineering (CSE) Department, Indian Institute of Technology (IIT) Bombay, Mumbai, India
| |
Collapse
|
18
|
Ulrich L, Held KG, Jaeger M, Frenz M, Akarçay HG. Reliability assessment for blood oxygen saturation levels measured with optoacoustic imaging. JOURNAL OF BIOMEDICAL OPTICS 2020; 25:1-15. [PMID: 32323509 PMCID: PMC7175414 DOI: 10.1117/1.jbo.25.4.046005] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/08/2020] [Accepted: 03/25/2020] [Indexed: 06/11/2023]
Abstract
SIGNIFICANCE Quantitative optoacoustic (OA) imaging has the potential to provide blood oxygen saturation (SO2) estimates due to the proportionality between the measured signal and the blood's absorption coefficient. However, due to the wavelength-dependent attenuation of light in tissue, a spectral correction of the OA signals is required, and a prime challenge is the validation of both the optical characterization of the tissue and the SO2. AIM We propose to assess the reliability of SO2 levels retrieved from spectral fitting by measuring the similarity of OA spectra to the fitted blood absorption spectra. APPROACH We introduce a metric that quantifies the trends of blood spectra by assigning a pair of spectral slopes to each spectrum. The applicability of the metric is illustrated with in vivo measurements on a human forearm. RESULTS We show that physiologically sound SO2 values do not necessarily imply a successful spectral correction and demonstrate how the metric can be used to distinguish SO2 values that are trustworthy from unreliable ones. CONCLUSIONS The metric is independent of the methods used for the OA data acquisition, image reconstruction, and spectral correction, thus it can be readily combined with existing approaches, in order to monitor the accuracy of quantitative OA imaging.
Collapse
Affiliation(s)
- Leonie Ulrich
- University of Bern, Institute of Applied Physics, Biomedical Photonics, Bern, Switzerland
| | - Kai Gerrit Held
- University of Bern, Institute of Applied Physics, Biomedical Photonics, Bern, Switzerland
- ABB Switzerland, Corporate Research, Baden-Daettwil, Switzerland
| | - Michael Jaeger
- University of Bern, Institute of Applied Physics, Biomedical Photonics, Bern, Switzerland
| | - Martin Frenz
- University of Bern, Institute of Applied Physics, Biomedical Photonics, Bern, Switzerland
| | - Hidayet Günhan Akarçay
- University of Bern, Institute of Applied Physics, Biomedical Photonics, Bern, Switzerland
| |
Collapse
|