1
|
Chang S, Marsh JF, Koons EK, Gong H, McCollough CH, Leng S. Improved noise reduction in photon-counting detector CT using prior knowledge-aware iterative denoising neural network. J Med Imaging (Bellingham) 2024; 11:S12804. [PMID: 38799270 PMCID: PMC11124219 DOI: 10.1117/1.jmi.11.s1.s12804] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2024] [Revised: 04/10/2024] [Accepted: 05/14/2024] [Indexed: 05/29/2024] Open
Abstract
Purpose We aim to reduce image noise in high-resolution (HR) virtual monoenergetic images (VMIs) from photon-counting detector (PCD) CT scans by developing a prior knowledge-aware iterative denoising neural network (PKAID-Net) that efficiently exploits the unique noise characteristics of VMIs at different energy (keV) levels. Approach PKAID-Net offers two major features: first, it utilizes a lower-noise VMI (e.g., 70 keV) as a prior input; second, it iteratively constructs a refined training dataset to improve the neural network's denoising performance. In each iteration, the denoised image from the previous module serves as an updated target image, which is included in the dataset for the subsequent training iteration. Our study includes 10 patient coronary CT angiography exams acquired on a clinical dual-source PCD-CT (NAEOTOM Alpha, Siemens Healthineers). The HR VMIs were reconstructed at 50, 70, and 100 keV, using a sharp vascular kernel (Bv68) and thin (0.6 mm) slice thickness (0.3 mm increment). PKAID-Net's performance was evaluated in terms of image noise, spatial detail preservation, and quantitative accuracy. Results PKAID-Net achieved a noise reduction of 96% compared to filtered back projection and 65% relative to iterative reconstruction, all while preserving spatial and spectral fidelity and maintaining a natural noise texture. The iterative refinement of PCD-CT data during the training process substantially enhanced the robustness of deep learning-based denoising compared to the original method, which resulted in some spatial detail loss. Conclusions The PKAID-Net provides substantial noise reduction while maintaining spatial and spectral fidelity of the HR VMIs from PCD-CT.
Collapse
Affiliation(s)
- Shaojie Chang
- Mayo Clinic, Department of Radiology, Rochester, Minnesota, United States
| | - Jeffrey F. Marsh
- Mayo Clinic, Department of Radiology, Rochester, Minnesota, United States
| | - Emily K. Koons
- Mayo Clinic, Department of Radiology, Rochester, Minnesota, United States
| | - Hao Gong
- Mayo Clinic, Department of Radiology, Rochester, Minnesota, United States
| | | | - Shuai Leng
- Mayo Clinic, Department of Radiology, Rochester, Minnesota, United States
| |
Collapse
|
2
|
Lan H, Huang L, Wei X, Li Z, Lv J, Ma C, Nie L, Luo J. Masked cross-domain self-supervised deep learning framework for photoacoustic computed tomography reconstruction. Neural Netw 2024; 179:106515. [PMID: 39032393 DOI: 10.1016/j.neunet.2024.106515] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2023] [Revised: 06/24/2024] [Accepted: 07/05/2024] [Indexed: 07/23/2024]
Abstract
Accurate image reconstruction is crucial for photoacoustic (PA) computed tomography (PACT). Recently, deep learning has been used to reconstruct PA images with a supervised scheme, which requires high-quality images as ground truth labels. However, practical implementations encounter inevitable trade-offs between cost and performance due to the expensive nature of employing additional channels for accessing more measurements. Here, we propose a masked cross-domain self-supervised (CDSS) reconstruction strategy to overcome the lack of ground truth labels from limited PA measurements. We implement the self-supervised reconstruction in a model-based form. Simultaneously, we take advantage of self-supervision to enforce the consistency of measurements and images across three partitions of the measured PA data, achieved by randomly masking different channels. Our findings indicate that dynamically masking a substantial proportion of channels, such as 80%, yields meaningful self-supervisors in both the image and signal domains. Consequently, this approach reduces the multiplicity of pseudo solutions and enables efficient image reconstruction using fewer PA measurements, ultimately minimizing reconstruction error. Experimental results on in-vivo PACT dataset of mice demonstrate the potential of our self-supervised framework. Moreover, our method exhibits impressive performance, achieving a structural similarity index (SSIM) of 0.87 in an extreme sparse case utilizing only 13 channels, which outperforms the performance of the supervised scheme with 16 channels (0.77 SSIM). Adding to its advantages, our method can be deployed on different trainable models in an end-to-end manner, further enhancing its versatility and applicability.
Collapse
Affiliation(s)
- Hengrong Lan
- School of Biomedical Engineering, Tsinghua University, Beijing 100084, China
| | - Lijie Huang
- School of Biomedical Engineering, Tsinghua University, Beijing 100084, China
| | - Xingyue Wei
- School of Biomedical Engineering, Tsinghua University, Beijing 100084, China
| | - Zhiqiang Li
- School of Biomedical Engineering, Tsinghua University, Beijing 100084, China
| | - Jing Lv
- Medical Research Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou 510080, China
| | - Cheng Ma
- Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
| | - Liming Nie
- Medical Research Institute, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, Southern Medical University, Guangzhou 510080, China
| | - Jianwen Luo
- School of Biomedical Engineering, Tsinghua University, Beijing 100084, China.
| |
Collapse
|
3
|
Jia X, Carter BW, Duffton A, Harris E, Hobbs R, Li H. Advancing the Collaboration Between Imaging and Radiation Oncology. Semin Radiat Oncol 2024; 34:402-417. [PMID: 39271275 PMCID: PMC11407744 DOI: 10.1016/j.semradonc.2024.07.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/15/2024]
Abstract
The fusion of cutting-edge imaging technologies with radiation therapy (RT) has catalyzed transformative breakthroughs in cancer treatment in recent decades. It is critical for us to review our achievements and preview into the next phase for future synergy between imaging and RT. This paper serves as a review and preview for fostering collaboration between these two domains in the forthcoming decade. Firstly, it delineates ten prospective directions ranging from technological innovations to leveraging imaging data in RT planning, execution, and preclinical research. Secondly, it presents major directions for infrastructure and team development in facilitating interdisciplinary synergy and clinical translation. We envision a future where seamless integration of imaging technologies into RT will not only meet the demands of RT but also unlock novel functionalities, enhancing accuracy, efficiency, safety, and ultimately, the standard of care for patients worldwide.
Collapse
Affiliation(s)
- Xun Jia
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD..
| | - Brett W Carter
- Department of Thoracic Imaging, Division of Diagnostic Imaging, University of Texas MD Anderson Cancer Center, Houston, TX
| | - Aileen Duffton
- Beatson West of Scotland Cancer Centre, Glasgow, UK.; Institute of Cancer Science, University of Glasgow, UK
| | - Emma Harris
- Division of Radiotherapy and Imaging, Institute of Cancer Research, London, UK
| | - Robert Hobbs
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD
| | - Heng Li
- Department of Radiation Oncology and Molecular Radiation Sciences, Johns Hopkins University, Baltimore, MD
| |
Collapse
|
4
|
Hein D, Holmin S, Szczykutowicz T, Maltz JS, Danielsson M, Wang G, Persson M. Noise suppression in photon-counting computed tomography using unsupervised Poisson flow generative models. Vis Comput Ind Biomed Art 2024; 7:24. [PMID: 39311990 PMCID: PMC11420411 DOI: 10.1186/s42492-024-00175-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 09/01/2024] [Indexed: 09/26/2024] Open
Abstract
Deep learning (DL) has proven to be important for computed tomography (CT) image denoising. However, such models are usually trained under supervision, requiring paired data that may be difficult to obtain in practice. Diffusion models offer unsupervised means of solving a wide range of inverse problems via posterior sampling. In particular, using the estimated unconditional score function of the prior distribution, obtained via unsupervised learning, one can sample from the desired posterior via hijacking and regularization. However, due to the iterative solvers used, the number of function evaluations (NFE) required may be orders of magnitudes larger than for single-step samplers. In this paper, we present a novel image denoising technique for photon-counting CT by extending the unsupervised approach to inverse problem solving to the case of Poisson flow generative models (PFGM)++. By hijacking and regularizing the sampling process we obtain a single-step sampler, that is NFE = 1. Our proposed method incorporates posterior sampling using diffusion models as a special case. We demonstrate that the added robustness afforded by the PFGM++ framework yields significant performance gains. Our results indicate competitive performance compared to popular supervised, including state-of-the-art diffusion-style models with NFE = 1 (consistency models), unsupervised, and non-DL-based image denoising techniques, on clinical low-dose CT data and clinical images from a prototype photon-counting CT system developed by GE HealthCare.
Collapse
Affiliation(s)
- Dennis Hein
- Department of Physics, KTH Royal Institute of Technology, Stockholm, 1142, Sweden.
- MedTechLabs, Karolinska University Hospital, Stockholm, 17164, Sweden.
| | - Staffan Holmin
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, 17164, Sweden
- Department of Neuroradiology, Karolinska University Hospital, Stockholm, 17164, Sweden
| | - Timothy Szczykutowicz
- Department of Radiology, School of Medicine and Public Health, University of Wisconsin, Madison, WI, 53705, United States
| | | | - Mats Danielsson
- Department of Physics, KTH Royal Institute of Technology, Stockholm, 1142, Sweden
- MedTechLabs, Karolinska University Hospital, Stockholm, 17164, Sweden
| | - Ge Wang
- Department of Biomedical Engineering, School of Engineering, Biomedical Imaging Center, Center for Biotechnology and Interdisciplinary Studies, Rensselaer Polytechnic Institute, Troy, NY, 12180, United States
| | - Mats Persson
- Department of Physics, KTH Royal Institute of Technology, Stockholm, 1142, Sweden
- MedTechLabs, Karolinska University Hospital, Stockholm, 17164, Sweden
| |
Collapse
|
5
|
Huang J, Yang L, Wang F, Wu Y, Nan Y, Wu W, Wang C, Shi K, Aviles-Rivero AI, Schönlieb CB, Zhang D, Yang G. Enhancing global sensitivity and uncertainty quantification in medical image reconstruction with Monte Carlo arbitrary-masked mamba. Med Image Anal 2024; 99:103334. [PMID: 39255733 DOI: 10.1016/j.media.2024.103334] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2024] [Revised: 08/05/2024] [Accepted: 09/01/2024] [Indexed: 09/12/2024]
Abstract
Deep learning has been extensively applied in medical image reconstruction, where Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) represent the predominant paradigms, each possessing distinct advantages and inherent limitations: CNNs exhibit linear complexity with local sensitivity, whereas ViTs demonstrate quadratic complexity with global sensitivity. The emerging Mamba has shown superiority in learning visual representation, which combines the advantages of linear scalability and global sensitivity. In this study, we introduce MambaMIR, an Arbitrary-Masked Mamba-based model with wavelet decomposition for joint medical image reconstruction and uncertainty estimation. A novel Arbitrary Scan Masking (ASM) mechanism "masks out" redundant information to introduce randomness for further uncertainty estimation. Compared to the commonly used Monte Carlo (MC) dropout, our proposed MC-ASM provides an uncertainty map without the need for hyperparameter tuning and mitigates the performance drop typically observed when applying dropout to low-level tasks. For further texture preservation and better perceptual quality, we employ the wavelet transformation into MambaMIR and explore its variant based on the Generative Adversarial Network, namely MambaMIR-GAN. Comprehensive experiments have been conducted for multiple representative medical image reconstruction tasks, demonstrating that the proposed MambaMIR and MambaMIR-GAN outperform other baseline and state-of-the-art methods in different reconstruction tasks, where MambaMIR achieves the best reconstruction fidelity and MambaMIR-GAN has the best perceptual quality. In addition, our MC-ASM provides uncertainty maps as an additional tool for clinicians, while mitigating the typical performance drop caused by the commonly used dropout.
Collapse
Affiliation(s)
- Jiahao Huang
- Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, United Kingdom; National Heart and Lung Institute, Imperial College London, London SW7 2AZ, United Kingdom; Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, United Kingdom.
| | - Liutao Yang
- College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Fanwen Wang
- Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, United Kingdom; National Heart and Lung Institute, Imperial College London, London SW7 2AZ, United Kingdom; Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, United Kingdom
| | - Yinzhe Wu
- Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, United Kingdom; National Heart and Lung Institute, Imperial College London, London SW7 2AZ, United Kingdom; Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, United Kingdom.
| | - Yang Nan
- Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, United Kingdom; National Heart and Lung Institute, Imperial College London, London SW7 2AZ, United Kingdom
| | - Weiwen Wu
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, Guangdong, China
| | - Chengyan Wang
- Human Phenome Institute, Fudan University, Shanghai, China
| | - Kuangyu Shi
- Department of Nuclear Medicine, Inselspital, University of Bern, Bern, Switzerland; Department of Informatics, Technical University of Munich, Munich, Germany
| | - Angelica I Aviles-Rivero
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom
| | - Carola-Bibiane Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom
| | - Daoqiang Zhang
- College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Guang Yang
- Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, United Kingdom; National Heart and Lung Institute, Imperial College London, London SW7 2AZ, United Kingdom; Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, United Kingdom; School of Biomedical Engineering & Imaging Sciences, King's College London, London WC2R 2LS, United Kingdom.
| |
Collapse
|
6
|
Hu Y, Gan W, Ying C, Wang T, Eldeniz C, Liu J, Chen Y, An H, Kamilov US. SPICER: Self-supervised learning for MRI with automatic coil sensitivity estimation and reconstruction. Magn Reson Med 2024; 92:1048-1063. [PMID: 38725383 DOI: 10.1002/mrm.30121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Revised: 02/28/2024] [Accepted: 04/02/2024] [Indexed: 06/27/2024]
Abstract
PURPOSE To introduce a novel deep model-based architecture (DMBA), SPICER, that uses pairs of noisy and undersampled k-space measurements of the same object to jointly train a model for MRI reconstruction and automatic coil sensitivity estimation. METHODS SPICER consists of two modules to simultaneously reconstructs accurate MR images and estimates high-quality coil sensitivity maps (CSMs). The first module, CSM estimation module, uses a convolutional neural network (CNN) to estimate CSMs from the raw measurements. The second module, DMBA-based MRI reconstruction module, forms reconstructed images from the input measurements and the estimated CSMs using both the physical measurement model and learned CNN prior. With the benefit of our self-supervised learning strategy, SPICER can be efficiently trained without any fully sampled reference data. RESULTS We validate SPICER on both open-access datasets and experimentally collected data, showing that it can achieve state-of-the-art performance in highly accelerated data acquisition settings (up to10 × $$ 10\times $$ ). Our results also highlight the importance of different modules of SPICER-including the DMBA, the CSM estimation, and the SPICER training loss-on the final performance of the method. Moreover, SPICER can estimate better CSMs than pre-estimation methods especially when the ACS data is limited. CONCLUSION Despite being trained on noisy undersampled data, SPICER can reconstruct high-quality images and CSMs in highly undersampled settings, which outperforms other self-supervised learning methods and matches the performance of the well-known E2E-VarNet trained on fully sampled ground-truth data.
Collapse
Affiliation(s)
- Yuyang Hu
- Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, Missouri
| | - Weijie Gan
- Department of Computer Science and Engineering, Washington University in St. Louis, St. Louis, Missouri
| | - Chunwei Ying
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, Missouri
| | - Tongyao Wang
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, Missouri
| | - Cihat Eldeniz
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, Missouri
| | - Jiaming Liu
- Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, Missouri
| | - Yasheng Chen
- Department of Neurology, Washington University in St. Louis, St. Louis, Missouri
| | - Hongyu An
- Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, Missouri
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, Missouri
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, Missouri
- Department of Neurology, Washington University in St. Louis, St. Louis, Missouri
| | - Ulugbek S Kamilov
- Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, Missouri
- Department of Computer Science and Engineering, Washington University in St. Louis, St. Louis, Missouri
| |
Collapse
|
7
|
Krüppel S, Khani MH, Schreyer HM, Sridhar S, Ramakrishna V, Zapp SJ, Mietsch M, Karamanlis D, Gollisch T. Applying Super-Resolution and Tomography Concepts to Identify Receptive Field Subunits in the Retina. PLoS Comput Biol 2024; 20:e1012370. [PMID: 39226328 PMCID: PMC11398665 DOI: 10.1371/journal.pcbi.1012370] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Revised: 09/13/2024] [Accepted: 07/28/2024] [Indexed: 09/05/2024] Open
Abstract
Spatially nonlinear stimulus integration by retinal ganglion cells lies at the heart of various computations performed by the retina. It arises from the nonlinear transmission of signals that ganglion cells receive from bipolar cells, which thereby constitute functional subunits within a ganglion cell's receptive field. Inferring these subunits from recorded ganglion cell activity promises a new avenue for studying the functional architecture of the retina. This calls for efficient methods, which leave sufficient experimental time to leverage the acquired knowledge for further investigating identified subunits. Here, we combine concepts from super-resolution microscopy and computed tomography and introduce super-resolved tomographic reconstruction (STR) as a technique to efficiently stimulate and locate receptive field subunits. Simulations demonstrate that this approach can reliably identify subunits across a wide range of model variations, and application in recordings of primate parasol ganglion cells validates the experimental feasibility. STR can potentially reveal comprehensive subunit layouts within only a few tens of minutes of recording time, making it ideal for online analysis and closed-loop investigations of receptive field substructure in retina recordings.
Collapse
Affiliation(s)
- Steffen Krüppel
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
- Cluster of Excellence "Multiscale Bioimaging: from Molecular Machines to Networks of Excitable Cells" (MBExC), University of Göttingen, Göttingen, Germany
| | - Mohammad H Khani
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
| | - Helene M Schreyer
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
| | - Shashwat Sridhar
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
| | - Varsha Ramakrishna
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
- International Max Planck Research School for Neurosciences, Göttingen, Germany
| | - Sören J Zapp
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
| | - Matthias Mietsch
- German Primate Center, Laboratory Animal Science Unit, Göttingen, Germany
- German Center for Cardiovascular Research, Partner Site Göttingen, Göttingen, Germany
| | - Dimokratis Karamanlis
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
| | - Tim Gollisch
- University Medical Center Göttingen, Department of Ophthalmology, Göttingen, Germany
- Bernstein Center for Computational Neuroscience Göttingen, Göttingen, Germany
- Cluster of Excellence "Multiscale Bioimaging: from Molecular Machines to Networks of Excitable Cells" (MBExC), University of Göttingen, Göttingen, Germany
- Else Kröner Fresenius Center for Optogenetic Therapies, University Medical Center Göttingen, Göttingen, Germany
| |
Collapse
|
8
|
Fu M, Fang M, Khan RA, Liao B, Hu Z, Wu FX. SG-Fusion: A swin-transformer and graph convolution-based multi-modal deep neural network for glioma prognosis. Artif Intell Med 2024; 157:102972. [PMID: 39232270 DOI: 10.1016/j.artmed.2024.102972] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 07/22/2024] [Accepted: 08/29/2024] [Indexed: 09/06/2024]
Abstract
The integration of morphological attributes extracted from histopathological images and genomic data holds significant importance in advancing tumor diagnosis, prognosis, and grading. Histopathological images are acquired through microscopic examination of tissue slices, providing valuable insights into cellular structures and pathological features. On the other hand, genomic data provides information about tumor gene expression and functionality. The fusion of these two distinct data types is crucial for gaining a more comprehensive understanding of tumor characteristics and progression. In the past, many studies relied on single-modal approaches for tumor diagnosis. However, these approaches had limitations as they were unable to fully harness the information from multiple data sources. To address these limitations, researchers have turned to multi-modal methods that concurrently leverage both histopathological images and genomic data. These methods better capture the multifaceted nature of tumors and enhance diagnostic accuracy. Nonetheless, existing multi-modal methods have, to some extent, oversimplified the extraction processes for both modalities and the fusion process. In this study, we presented a dual-branch neural network, namely SG-Fusion. Specifically, for the histopathological modality, we utilize the Swin-Transformer structure to capture both local and global features and incorporate contrastive learning to encourage the model to discern commonalities and differences in the representation space. For the genomic modality, we developed a graph convolutional network based on gene functional and expression level similarities. Additionally, our model integrates a cross-attention module to enhance information interaction and employs divergence-based regularization to enhance the model's generalization performance. Validation conducted on glioma datasets from the Cancer Genome Atlas unequivocally demonstrates that our SG-Fusion model outperforms both single-modal methods and existing multi-modal approaches in both survival analysis and tumor grading.
Collapse
Affiliation(s)
- Minghan Fu
- Department of Mechanical Engineering, University of Saskatchewan, Saskatoon, S7N 5A9, SK, Canada
| | - Ming Fang
- Division of Biomedical Engineering, University of Saskatchewan, Saskatoon, S7N 5A9, SK, Canada
| | - Rayyan Azam Khan
- Division of Biomedical Engineering, University of Saskatchewan, Saskatoon, S7N 5A9, SK, Canada
| | - Bo Liao
- School of Mathematics and Statistics, Hainan Normal University, Haikou, 571158, Hainan, China
| | - Zhanli Hu
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, Guangdong, China
| | - Fang-Xiang Wu
- Department of Mechanical Engineering, University of Saskatchewan, Saskatoon, S7N 5A9, SK, Canada; Division of Biomedical Engineering, University of Saskatchewan, Saskatoon, S7N 5A9, SK, Canada; Department of Computer Science, University of Saskatchewan, Saskatoon, S7N 5A9, SK, Canada.
| |
Collapse
|
9
|
Vashistha R, Vegh V, Moradi H, Hammond A, O'Brien K, Reutens D. Modular GAN: positron emission tomography image reconstruction using two generative adversarial networks. FRONTIERS IN RADIOLOGY 2024; 4:1466498. [PMID: 39328298 PMCID: PMC11425657 DOI: 10.3389/fradi.2024.1466498] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/18/2024] [Accepted: 08/08/2024] [Indexed: 09/28/2024]
Abstract
Introduction The reconstruction of PET images involves converting sinograms, which represent the measured counts of radioactive emissions using detector rings encircling the patient, into meaningful images. However, the quality of PET data acquisition is impacted by physical factors, photon count statistics and detector characteristics, which affect the signal-to-noise ratio, resolution and quantitative accuracy of the resulting images. To address these influences, correction methods have been developed to mitigate each of these issues separately. Recently, generative adversarial networks (GANs) based on machine learning have shown promise in learning the complex mapping between acquired PET data and reconstructed tomographic images. This study aims to investigate the properties of training images that contribute to GAN performance when non-clinical images are used for training. Additionally, we describe a method to correct common PET imaging artefacts without relying on patient-specific anatomical images. Methods The modular GAN framework includes two GANs. Module 1, resembling Pix2pix architecture, is trained on non-clinical sinogram-image pairs. Training data are optimised by considering image properties defined by metrics. The second module utilises adaptive instance normalisation and style embedding to enhance the quality of images from Module 1. Additional perceptual and patch-based loss functions are employed in training both modules. The performance of the new framework was compared with that of existing methods, (filtered backprojection (FBP) and ordered subset expectation maximisation (OSEM) without and with point spread function (OSEM-PSF)) with respect to correction for attenuation, patient motion and noise in simulated, NEMA phantom and human imaging data. Evaluation metrics included structural similarity (SSIM), peak-signal-to-noise ratio (PSNR), relative root mean squared error (rRMSE) for simulated data, and contrast-to-noise ratio (CNR) for NEMA phantom and human data. Results For simulated test data, the performance of the proposed framework was both qualitatively and quantitatively superior to that of FBP and OSEM. In the presence of noise, Module 1 generated images with a SSIM of 0.48 and higher. These images exhibited coarse structures that were subsequently refined by Module 2, yielding images with an SSIM higher than 0.71 (at least 22% higher than OSEM). The proposed method was robust against noise and motion. For NEMA phantoms, it achieved higher CNR values than OSEM. For human images, the CNR in brain regions was significantly higher than that of FBP and OSEM (p < 0.05, paired t-test). The CNR of images reconstructed with OSEM-PSF was similar to those reconstructed using the proposed method. Conclusion The proposed image reconstruction method can produce PET images with artefact correction.
Collapse
Affiliation(s)
- Rajat Vashistha
- Centre for Advanced Imaging, University of Queensland, Brisbane, QLD, Australia
- ARC Training Centre for Innovation in Biomedical Imaging Technology, University of Queensland, Brisbane, QLD, Australia
| | - Viktor Vegh
- Centre for Advanced Imaging, University of Queensland, Brisbane, QLD, Australia
- ARC Training Centre for Innovation in Biomedical Imaging Technology, University of Queensland, Brisbane, QLD, Australia
| | - Hamed Moradi
- Centre for Advanced Imaging, University of Queensland, Brisbane, QLD, Australia
- ARC Training Centre for Innovation in Biomedical Imaging Technology, University of Queensland, Brisbane, QLD, Australia
- Diagnostic Imaging, Siemens Healthcare Pty Ltd., Melbourne, QLD, Australia
| | - Amanda Hammond
- Diagnostic Imaging, Siemens Healthcare Pty Ltd., Melbourne, QLD, Australia
| | - Kieran O'Brien
- Diagnostic Imaging, Siemens Healthcare Pty Ltd., Melbourne, QLD, Australia
| | - David Reutens
- Centre for Advanced Imaging, University of Queensland, Brisbane, QLD, Australia
- ARC Training Centre for Innovation in Biomedical Imaging Technology, University of Queensland, Brisbane, QLD, Australia
| |
Collapse
|
10
|
Liu T, Huang S, Li R, Gao P, Li W, Lu H, Song Y, Rong J. Dual and Multi-Target Cone-Beam X-ray Luminescence Computed Tomography Based on the DeepCB-XLCT Network. Bioengineering (Basel) 2024; 11:874. [PMID: 39329616 PMCID: PMC11428951 DOI: 10.3390/bioengineering11090874] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2024] [Revised: 08/24/2024] [Accepted: 08/27/2024] [Indexed: 09/28/2024] Open
Abstract
BACKGROUND AND OBJECTIVE Emerging as a hybrid imaging modality, cone-beam X-ray luminescence computed tomography (CB-XLCT) has been developed using X-ray-excitable nanoparticles. In contrast to conventional bio-optical imaging techniques like bioluminescence tomography (BLT) and fluorescence molecular tomography (FMT), CB-XLCT offers the advantage of greater imaging depth while significantly reducing interference from autofluorescence and background fluorescence, owing to its utilization of X-ray-excited nanoparticles. However, due to the intricate excitation process and extensive light scattering within biological tissues, the inverse problem of CB-XLCT is fundamentally ill-conditioned. METHODS An end-to-end three-dimensional deep encoder-decoder network, termed DeepCB-XLCT, is introduced to improve the quality of CB-XLCT reconstructions. This network directly establishes a nonlinear mapping between the distribution of internal X-ray-excitable nanoparticles and the corresponding boundary fluorescent signals. To improve the fidelity of target shape restoration, the structural similarity loss (SSIM) was incorporated into the objective function of the DeepCB-XLCT network. Additionally, a loss term specifically for target regions was introduced to improve the network's emphasis on the areas of interest. As a result, the inaccuracies in reconstruction caused by the simplified linear model used in conventional methods can be effectively minimized by the proposed DeepCB-XLCT method. RESULTS AND CONCLUSIONS Numerical simulations, phantom experiments, and in vivo experiments with two targets were performed, revealing that the DeepCB-XLCT network enhances reconstruction accuracy regarding contrast-to-noise ratio and shape similarity when compared to traditional methods. In addition, the findings from the XLCT tomographic images involving three targets demonstrate its potential for multi-target CB-XLCT imaging.
Collapse
Affiliation(s)
- Tianshuai Liu
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi’an 710032, China
| | - Shien Huang
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China
| | - Ruijing Li
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi’an 710032, China
| | - Peng Gao
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi’an 710032, China
| | - Wangyang Li
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi’an 710032, China
| | - Hongbing Lu
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi’an 710032, China
| | - Yonghong Song
- School of Software Engineering, Xi’an Jiaotong University, Xi’an 710049, China
| | - Junyan Rong
- Biomedical Engineering Department, Fourth Military Medical University, Xi’an 710032, China; (T.L.); (S.H.); (R.L.); (P.G.); (W.L.)
- Shaanxi Provincial Key Laboratory of Bioelectromagnetic Detection and Intelligent Perception, Xi’an 710032, China
| |
Collapse
|
11
|
Cao W, Gou L, Li B, Jiang L, Shen J. Whole-cell visualization of plant organelles by electron tomography. TRENDS IN PLANT SCIENCE 2024; 29:937-938. [PMID: 38987058 DOI: 10.1016/j.tplants.2024.06.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/17/2024] [Revised: 06/15/2024] [Accepted: 06/17/2024] [Indexed: 07/12/2024]
Affiliation(s)
- Wenhan Cao
- State Key Laboratory of Subtropical Silviculture, Zhejiang A&F University, 311300 Hangzhou, China
| | - Liangpeng Gou
- State Key Laboratory of Subtropical Silviculture, Zhejiang A&F University, 311300 Hangzhou, China
| | - Baiying Li
- Department of Biology, Hong Kong Baptist University, Hong Kong, SAR, China
| | - Liwen Jiang
- School of Life Sciences, Centre for Cell and Developmental Biology and State Key Laboratory of Agrobiotechnology, The Chinese University of Hong Kong, Shatin, Hong Kong, China
| | - Jinbo Shen
- State Key Laboratory of Subtropical Silviculture, Zhejiang A&F University, 311300 Hangzhou, China.
| |
Collapse
|
12
|
Cam RM, Villa U, Anastasio MA. Learning a stable approximation of an existing but unknown inverse mapping: application to the half-time circular Radon transform. INVERSE PROBLEMS 2024; 40:085002. [PMID: 38933410 PMCID: PMC11197394 DOI: 10.1088/1361-6420/ad4f0a] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 04/05/2024] [Accepted: 05/22/2024] [Indexed: 06/28/2024]
Abstract
Supervised deep learning-based methods have inspired a new wave of image reconstruction methods that implicitly learn effective regularization strategies from a set of training data. While they hold potential for improving image quality, they have also raised concerns regarding their robustness. Instabilities can manifest when learned methods are applied to find approximate solutions to ill-posed image reconstruction problems for which a unique and stable inverse mapping does not exist, which is a typical use case. In this study, we investigate the performance of supervised deep learning-based image reconstruction in an alternate use case in which a stable inverse mapping is known to exist but is not yet analytically available in closed form. For such problems, a deep learning-based method can learn a stable approximation of the unknown inverse mapping that generalizes well to data that differ significantly from the training set. The learned approximation of the inverse mapping eliminates the need to employ an implicit (optimization-based) reconstruction method and can potentially yield insights into the unknown analytic inverse formula. The specific problem addressed is image reconstruction from a particular case of radially truncated circular Radon transform (CRT) data, referred to as 'half-time' measurement data. For the half-time image reconstruction problem, we develop and investigate a learned filtered backprojection method that employs a convolutional neural network to approximate the unknown filtering operation. We demonstrate that this method behaves stably and readily generalizes to data that differ significantly from training data. The developed method may find application to wave-based imaging modalities that include photoacoustic computed tomography.
Collapse
Affiliation(s)
- Refik Mert Cam
- Department of Electrical and Computer Engineering, University of Illinois Urbana–Champaign, Urbana, IL 61801, United States of America
| | - Umberto Villa
- Oden Institute for Computational Engineering & Sciences, The University of Texas at Austin, Austin, TX 78712, United States of America
| | - Mark A Anastasio
- Department of Electrical and Computer Engineering, University of Illinois Urbana–Champaign, Urbana, IL 61801, United States of America
- Department of Bioengineering, University of Illinois Urbana–Champaign, Urbana, IL 61801, United States of America
| |
Collapse
|
13
|
Żydowicz WM, Skokowski J, Marano L, Polom K. Navigating the Metaverse: A New Virtual Tool with Promising Real Benefits for Breast Cancer Patients. J Clin Med 2024; 13:4337. [PMID: 39124604 PMCID: PMC11313674 DOI: 10.3390/jcm13154337] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2024] [Revised: 05/22/2024] [Accepted: 07/22/2024] [Indexed: 08/12/2024] Open
Abstract
BC, affecting both women and men, is a complex disease where early diagnosis plays a crucial role in successful treatment and enhances patient survival rates. The Metaverse, a virtual world, may offer new, personalized approaches to diagnosing and treating BC. Although Artificial Intelligence (AI) is still in its early stages, its rapid advancement indicates potential applications within the healthcare sector, including consolidating patient information in one accessible location. This could provide physicians with more comprehensive insights into disease details. Leveraging the Metaverse could facilitate clinical data analysis and improve the precision of diagnosis, potentially allowing for more tailored treatments for BC patients. However, while this article highlights the possible transformative impacts of virtual technologies on BC treatment, it is important to approach these developments with cautious optimism, recognizing the need for further research and validation to ensure enhanced patient care with greater accuracy and efficiency.
Collapse
Affiliation(s)
- Weronika Magdalena Żydowicz
- Department of General Surgery and Surgical Oncology, “Saint Wojciech” Hospital, “Nicolaus Copernicus” Health Center, Jana Pawła II 50, 80-462 Gdańsk, Poland; (W.M.Ż.); (J.S.)
| | - Jaroslaw Skokowski
- Department of General Surgery and Surgical Oncology, “Saint Wojciech” Hospital, “Nicolaus Copernicus” Health Center, Jana Pawła II 50, 80-462 Gdańsk, Poland; (W.M.Ż.); (J.S.)
- Department of Medicine, Academy of Applied Medical and Social Sciences, Akademia Medycznych I Spolecznych Nauk Stosowanych (AMiSNS), 2 Lotnicza Street, 82-300 Elbląg, Poland;
| | - Luigi Marano
- Department of General Surgery and Surgical Oncology, “Saint Wojciech” Hospital, “Nicolaus Copernicus” Health Center, Jana Pawła II 50, 80-462 Gdańsk, Poland; (W.M.Ż.); (J.S.)
- Department of Medicine, Academy of Applied Medical and Social Sciences, Akademia Medycznych I Spolecznych Nauk Stosowanych (AMiSNS), 2 Lotnicza Street, 82-300 Elbląg, Poland;
| | - Karol Polom
- Department of Medicine, Academy of Applied Medical and Social Sciences, Akademia Medycznych I Spolecznych Nauk Stosowanych (AMiSNS), 2 Lotnicza Street, 82-300 Elbląg, Poland;
- Department of Gastrointestinal Surgical Oncology, Greater Poland Cancer Centre, Garbary 15, 61-866 Poznan, Poland
| |
Collapse
|
14
|
Du W, Cui H, He L, Chen H, Zhang Y, Yang H. Structure-aware diffusion for low-dose CT imaging. Phys Med Biol 2024; 69:155008. [PMID: 38942004 DOI: 10.1088/1361-6560/ad5d47] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2024] [Accepted: 06/28/2024] [Indexed: 06/30/2024]
Abstract
Reducing the radiation dose leads to the x-ray computed tomography (CT) images suffering from heavy noise and artifacts, which inevitably interferes with the subsequent clinic diagnostic and analysis. Leading works have explored diffusion models for low-dose CT imaging to avoid the structure degeneration and blurring effects of previous deep denoising models. However, most of them always begin their generative processes with Gaussian noise, which has little or no structure priors of the clean data distribution, thereby leading to long-time inference and unpleasant reconstruction quality. To alleviate these problems, this paper presents a Structure-Aware Diffusion model (SAD), an end-to-end self-guided learning framework for high-fidelity CT image reconstruction. First, SAD builds a nonlinear diffusion bridge between clean and degraded data distributions, which could directly learn the implicit physical degradation prior from observed measurements. Second, SAD integrates the prompt learning mechanism and implicit neural representation into the diffusion process, where rich and diverse structure representations extracted by degraded inputs are exploited as prompts, which provides global and local structure priors, to guide CT image reconstruction. Finally, we devise an efficient self-guided diffusion architecture using an iterative updated strategy, which further refines structural prompts during each generative step to drive finer image reconstruction. Extensive experiments on AAPM-Mayo and LoDoPaB-CT datasets demonstrate that our SAD could achieve superior performance in terms of noise removal, structure preservation, and blind-dose generalization, with few generative steps, even one step only.
Collapse
Affiliation(s)
- Wenchao Du
- College of Computer Science, Sichuan University, Chengdu 610065, People's Republic of China
| | - HuanHuan Cui
- West China Hospital of Sichuan University, Chengdu 610041, People's Republic of China
| | - LinChao He
- College of Computer Science, Sichuan University, Chengdu 610065, People's Republic of China
| | - Hu Chen
- College of Computer Science, Sichuan University, Chengdu 610065, People's Republic of China
| | - Yi Zhang
- College of Computer Science, Sichuan University, Chengdu 610065, People's Republic of China
| | - Hongyu Yang
- College of Computer Science, Sichuan University, Chengdu 610065, People's Republic of China
| |
Collapse
|
15
|
Li S, Zhu Y, Spencer BA, Wang G. Single-Subject Deep-Learning Image Reconstruction With a Neural Optimization Transfer Algorithm for PET-Enabled Dual-Energy CT Imaging. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:4075-4089. [PMID: 38941203 DOI: 10.1109/tip.2024.3418347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/30/2024]
Abstract
Combining dual-energy computed tomography (DECT) with positron emission tomography (PET) offers many potential clinical applications but typically requires expensive hardware upgrades or increases radiation doses on PET/CT scanners due to an extra X-ray CT scan. The recent PET-enabled DECT method allows DECT imaging on PET/CT without requiring a second X-ray CT scan. It combines the already existing X-ray CT image with a 511 keV γ -ray CT (gCT) image reconstructed from time-of-flight PET emission data. A kernelized framework has been developed for reconstructing gCT image but this method has not fully exploited the potential of prior knowledge. Use of deep neural networks may explore the power of deep learning in this application. However, common approaches require a large database for training, which is impractical for a new imaging method like PET-enabled DECT. Here, we propose a single-subject method by using neural-network representation as a deep coefficient prior to improving gCT image reconstruction without population-based pre-training. The resulting optimization problem becomes the tomographic estimation of nonlinear neural-network parameters from gCT projection data. This complicated problem can be efficiently solved by utilizing the optimization transfer strategy with quadratic surrogates. Each iteration of the proposed neural optimization transfer algorithm includes: PET activity image update; gCT image update; and least-square neural-network learning in the gCT image domain. This algorithm is guaranteed to monotonically increase the data likelihood. Results from computer simulation, real phantom data and real patient data have demonstrated that the proposed method can significantly improve gCT image quality and consequent multi-material decomposition as compared to other methods.
Collapse
|
16
|
Song Q, Gong C. Image reconstruction method for incomplete CT projection based on self-guided image filtering. Med Biol Eng Comput 2024; 62:2101-2116. [PMID: 38457068 DOI: 10.1007/s11517-024-03044-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Accepted: 02/03/2024] [Indexed: 03/09/2024]
Abstract
In some fields of medical diagnosis or industrial nondestructive testing, it is difficult to obtain complete computed tomography (CT) data due to the limitation of radiation dose or other factors. Therefore, image reconstruction of incomplete projection data is the focus of this paper. In this paper, a new image reconstruction model based on self-guided image filtering (SGIF) term is proposed for few-view and segmental limited-angle (SLA) CT reconstruction. Then the alternating direction method (ADM) is used to solve this model. For simplicity, we call it ADM-SGIF method. The key idea of ADM-SGIF method is to use the reconstructed image itself as a reference and utilize its structural features to guide CT reconstruction. This method can effectively preserve image structures and remove shading artifacts. To validate the effectiveness of the proposed reconstruction method, we conduct digital phantom and real CT data experiments. The results indicate that ADM-SGIF method outperforms competing methods, including total variation (TV), relative total variation (RTV), and L0-norm minimization solved by ADM (ADM-L0) methods, in both subjective and objective evaluations.
Collapse
Affiliation(s)
- Qiang Song
- School of Mathematics and Statistics, Chongqing Technology and Business University, Chongqing, 400067, China
| | - Changcheng Gong
- School of Mathematics and Statistics, Chongqing Technology and Business University, Chongqing, 400067, China.
- Chongqing Key Laboratory of Statistical Intelligent Computing and Monitoring, Chongqing Technology and Business University, Chongqing, 400067, China.
| |
Collapse
|
17
|
Hamidpour P, Araee A, Baniassadi M, Garmestani H. Multiphase Reconstruction of Heterogeneous Materials Using Machine Learning and Quality of Connection Function. MATERIALS (BASEL, SWITZERLAND) 2024; 17:3049. [PMID: 38998131 PMCID: PMC11242835 DOI: 10.3390/ma17133049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/30/2024] [Revised: 06/17/2024] [Accepted: 06/18/2024] [Indexed: 07/14/2024]
Abstract
Establishing accurate structure-property linkages and precise phase volume accuracy in 3D microstructure reconstruction of materials remains challenging, particularly with limited samples. This paper presents an optimized method for reconstructing 3D microstructures of various materials, including isotropic and anisotropic types with two and three phases, using convolutional occupancy networks and point clouds from inner layers of the microstructure. The method emphasizes precise phase representation and compatibility with point cloud data. A stage within the Quality of Connection Function (QCF) repetition loop optimizes the weights of the convolutional occupancy networks model to minimize error between the microstructure's statistical properties and the reconstructive model. This model successfully reconstructs 3D representations from initial 2D serial images. Comparisons with screened Poisson surface reconstruction and local implicit grid methods demonstrate the model's efficacy. The developed model proves suitable for high-quality 3D microstructure reconstruction, aiding in structure-property linkages and finite element analysis.
Collapse
Affiliation(s)
- Pouria Hamidpour
- School of Mechanical Engineering, College of Engineering, University of Tehran, Tehran 14155-6619, Iran
| | - Alireza Araee
- School of Mechanical Engineering, College of Engineering, University of Tehran, Tehran 14155-6619, Iran
| | - Majid Baniassadi
- School of Mechanical Engineering, College of Engineering, University of Tehran, Tehran 14155-6619, Iran
- University of Strasbourg, CNRS, ICube UMR 7357, 67081 Strasbourg, France
| | - Hamid Garmestani
- University of Strasbourg, CNRS, ICube UMR 7357, 67081 Strasbourg, France
- School of Materials Science and Engineering, Georgia Institute of Technology, 771 Ferst Drive NW, Atlanta, GA 30332, USA
| |
Collapse
|
18
|
Qiao Z, Liu P, Fang C, Redler G, Epel B, Halpern H. Directional TV algorithm for image reconstruction from sparse-view projections in EPR imaging. Phys Med Biol 2024; 69:115051. [PMID: 38729205 DOI: 10.1088/1361-6560/ad4a1b] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Accepted: 05/10/2024] [Indexed: 05/12/2024]
Abstract
Objective.Electron paramagnetic resonance (EPR) imaging is an advanced in vivo oxygen imaging modality. The main drawback of EPR imaging is the long scanning time. Sparse-view projections collection is an effective fast scanning pattern. However, the commonly-used filtered back projection (FBP) algorithm is not competent to accurately reconstruct images from sparse-view projections because of the severe streak artifacts. The aim of this work is to develop an advanced algorithm for sparse reconstruction of 3D EPR imaging.Methods.The optimization based algorithms including the total variation (TV) algorithm have proven to be effective in sparse reconstruction in EPR imaging. To further improve the reconstruction accuracy, we propose the directional TV (DTV) model and derive its Chambolle-Pock solving algorithm.Results.After the algorithm correctness validation on simulation data, we explore the sparse reconstruction capability of the DTV algorithm via a simulated six-sphere phantom and two real bottle phantoms filled with OX063 trityl solution and scanned by an EPR imager with a magnetic field strength of 250 G.Conclusion.Both the simulated and real data experiments show that the DTV algorithm is superior to the existing FBP and TV-type algorithms and a deep learning based method according to visual inspection and quantitative evaluations in sparse reconstruction of EPR imaging.Significance.These insights gained in this work may be used in the development of fast EPR imaging workflow of practical significance.
Collapse
Affiliation(s)
- Zhiwei Qiao
- School of Computer and Information Technology, Shanxi University, Taiyuan, Shanxi, People's Republic of China
| | - Peng Liu
- School of Computer and Information Technology, Shanxi University, Taiyuan, Shanxi, People's Republic of China
- Department of Big Data and Intelligent Engineering, Shanxi Institute of Technology, Yangquan, Shanxi, People's Republic of China
| | - Chenyun Fang
- School of Computer and Information Technology, Shanxi University, Taiyuan, Shanxi, People's Republic of China
| | - Gage Redler
- Department of Radiation Oncology, Moffitt Cancer Center, Tampa, FL, United States of America
| | - Boris Epel
- Department of Radiation and Cellular Oncology, University of Chicago, Chicago, IL, United States of America
| | - Howard Halpern
- Department of Radiation and Cellular Oncology, University of Chicago, Chicago, IL, United States of America
| |
Collapse
|
19
|
Nguyen AH, Wang Z. Single-Shot 3D Reconstruction via Nonlinear Fringe Transformation: Supervised and Unsupervised Learning Approaches. SENSORS (BASEL, SWITZERLAND) 2024; 24:3246. [PMID: 38794100 PMCID: PMC11125235 DOI: 10.3390/s24103246] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/27/2024] [Revised: 04/22/2024] [Accepted: 05/16/2024] [Indexed: 05/26/2024]
Abstract
The field of computer vision has been focusing on achieving accurate three-dimensional (3D) object representations from a single two-dimensional (2D) image through deep artificial neural networks. Recent advancements in 3D shape reconstruction techniques that combine structured light and deep learning show promise in acquiring high-quality geometric information about object surfaces. This paper introduces a new single-shot 3D shape reconstruction method that uses a nonlinear fringe transformation approach through both supervised and unsupervised learning networks. In this method, a deep learning network learns to convert a grayscale fringe input into multiple phase-shifted fringe outputs with different frequencies, which act as an intermediate result for the subsequent 3D reconstruction process using the structured-light fringe projection profilometry technique. Experiments have been conducted to validate the practicality and robustness of the proposed technique. The experimental results demonstrate that the unsupervised learning approach using a deep convolutional generative adversarial network (DCGAN) is superior to the supervised learning approach using UNet in image-to-image generation. The proposed technique's ability to accurately reconstruct 3D shapes of objects using only a single fringe image opens up vast opportunities for its application across diverse real-world scenarios.
Collapse
Affiliation(s)
- Andrew-Hieu Nguyen
- Neuroimaging Research Branch, National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD 21224, USA;
| | - Zhaoyang Wang
- Department of Mechanical Engineering, School of Engineering, The Catholic University of America, Washington, DC 20064, USA
| |
Collapse
|
20
|
Chen Z, Niu C, Gao Q, Wang G, Shan H. LIT-Former: Linking In-Plane and Through-Plane Transformers for Simultaneous CT Image Denoising and Deblurring. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1880-1894. [PMID: 38194396 DOI: 10.1109/tmi.2024.3351723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
This paper studies 3D low-dose computed tomography (CT) imaging. Although various deep learning methods were developed in this context, typically they focus on 2D images and perform denoising due to low-dose and deblurring for super-resolution separately. Up to date, little work was done for simultaneous in-plane denoising and through-plane deblurring, which is important to obtain high-quality 3D CT images with lower radiation and faster imaging speed. For this task, a straightforward method is to directly train an end-to-end 3D network. However, it demands much more training data and expensive computational costs. Here, we propose to link in-plane and through-plane transformers for simultaneous in-plane denoising and through-plane deblurring, termed as LIT-Former, which can efficiently synergize in-plane and through-plane sub-tasks for 3D CT imaging and enjoy the advantages of both convolution and transformer networks. LIT-Former has two novel designs: efficient multi-head self-attention modules (eMSM) and efficient convolutional feed-forward networks (eCFN). First, eMSM integrates in-plane 2D self-attention and through-plane 1D self-attention to efficiently capture global interactions of 3D self-attention, the core unit of transformer networks. Second, eCFN integrates 2D convolution and 1D convolution to extract local information of 3D convolution in the same fashion. As a result, the proposed LIT-Former synergizes these two sub-tasks, significantly reducing the computational complexity as compared to 3D counterparts and enabling rapid convergence. Extensive experimental results on simulated and clinical datasets demonstrate superior performance over state-of-the-art models. The source code is made available at https://github.com/hao1635/LIT-Former.
Collapse
|
21
|
Shin M, Seo M, Lee K, Yoon K. Super-resolution techniques for biomedical applications and challenges. Biomed Eng Lett 2024; 14:465-496. [PMID: 38645589 PMCID: PMC11026337 DOI: 10.1007/s13534-024-00365-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2023] [Revised: 02/12/2024] [Accepted: 02/18/2024] [Indexed: 04/23/2024] Open
Abstract
Super-resolution (SR) techniques have revolutionized the field of biomedical applications by detailing the structures at resolutions beyond the limits of imaging or measuring tools. These techniques have been applied in various biomedical applications, including microscopy, magnetic resonance imaging (MRI), computed tomography (CT), X-ray, electroencephalogram (EEG), ultrasound, etc. SR methods are categorized into two main types: traditional non-learning-based methods and modern learning-based approaches. In both applications, SR methodologies have been effectively utilized on biomedical images, enhancing the visualization of complex biological structures. Additionally, these methods have been employed on biomedical data, leading to improvements in computational precision and efficiency for biomedical simulations. The use of SR techniques has resulted in more detailed and accurate analyses in diagnostics and research, essential for early disease detection and treatment planning. However, challenges such as computational demands, data interpretation complexities, and the lack of unified high-quality data persist. The article emphasizes these issues, underscoring the need for ongoing development in SR technologies to further improve biomedical research and patient care outcomes.
Collapse
Affiliation(s)
- Minwoo Shin
- School of Mathematics and Computing (Computational Science and Engineering), Yonsei University, 50 Yonsei-Ro, Seodaemun-Gu, Seoul, 03722 Republic of Korea
| | - Minjee Seo
- School of Mathematics and Computing (Computational Science and Engineering), Yonsei University, 50 Yonsei-Ro, Seodaemun-Gu, Seoul, 03722 Republic of Korea
| | - Kyunghyun Lee
- School of Mathematics and Computing (Computational Science and Engineering), Yonsei University, 50 Yonsei-Ro, Seodaemun-Gu, Seoul, 03722 Republic of Korea
| | - Kyungho Yoon
- School of Mathematics and Computing (Computational Science and Engineering), Yonsei University, 50 Yonsei-Ro, Seodaemun-Gu, Seoul, 03722 Republic of Korea
| |
Collapse
|
22
|
Zhu M, Fu Q, Liu B, Zhang M, Li B, Luo X, Zhou F. RT-SRTS: Angle-agnostic real-time simultaneous 3D reconstruction and tumor segmentation from single X-ray projection. Comput Biol Med 2024; 173:108390. [PMID: 38569234 DOI: 10.1016/j.compbiomed.2024.108390] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 03/24/2024] [Accepted: 03/26/2024] [Indexed: 04/05/2024]
Abstract
Radiotherapy is one of the primary treatment methods for tumors, but the organ movement caused by respiration limits its accuracy. Recently, 3D imaging from a single X-ray projection has received extensive attention as a promising approach to address this issue. However, current methods can only reconstruct 3D images without directly locating the tumor and are only validated for fixed-angle imaging, which fails to fully meet the requirements of motion control in radiotherapy. In this study, a novel imaging method RT-SRTS is proposed which integrates 3D imaging and tumor segmentation into one network based on multi-task learning (MTL) and achieves real-time simultaneous 3D reconstruction and tumor segmentation from a single X-ray projection at any angle. Furthermore, the attention enhanced calibrator (AEC) and uncertain-region elaboration (URE) modules have been proposed to aid feature extraction and improve segmentation accuracy. The proposed method was evaluated on fifteen patient cases and compared with three state-of-the-art methods. It not only delivers superior 3D reconstruction but also demonstrates commendable tumor segmentation results. Simultaneous reconstruction and segmentation can be completed in approximately 70 ms, significantly faster than the required time threshold for real-time tumor tracking. The efficacies of both AEC and URE have also been validated in ablation studies. The code of work is available at https://github.com/ZywooSimple/RT-SRTS.
Collapse
Affiliation(s)
- Miao Zhu
- Image Processing Center, Beihang University, Beijing, 100191, PR China
| | - Qiming Fu
- Image Processing Center, Beihang University, Beijing, 100191, PR China
| | - Bo Liu
- Image Processing Center, Beihang University, Beijing, 100191, PR China.
| | - Mengxi Zhang
- Image Processing Center, Beihang University, Beijing, 100191, PR China
| | - Bojian Li
- Image Processing Center, Beihang University, Beijing, 100191, PR China
| | - Xiaoyan Luo
- Image Processing Center, Beihang University, Beijing, 100191, PR China.
| | - Fugen Zhou
- Image Processing Center, Beihang University, Beijing, 100191, PR China
| |
Collapse
|
23
|
Li Z, Gao Q, Wu Y, Niu C, Zhang J, Wang M, Wang G, Shan H. Quad-Net: Quad-Domain Network for CT Metal Artifact Reduction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1866-1879. [PMID: 38194399 DOI: 10.1109/tmi.2024.3351722] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Abstract
Metal implants and other high-density objects in patients introduce severe streaking artifacts in CT images, compromising image quality and diagnostic performance. Although various methods were developed for CT metal artifact reduction over the past decades, including the latest dual-domain deep networks, remaining metal artifacts are still clinically challenging in many cases. Here we extend the state-of-the-art dual-domain deep network approach into a quad-domain counterpart so that all the features in the sinogram, image, and their corresponding Fourier domains are synergized to eliminate metal artifacts optimally without compromising structural subtleties. Our proposed quad-domain network for MAR, referred to as Quad-Net, takes little additional computational cost since the Fourier transform is highly efficient, and works across the four receptive fields to learn both global and local features as well as their relations. Specifically, we first design a Sinogram-Fourier Restoration Network (SFR-Net) in the sinogram domain and its Fourier space to faithfully inpaint metal-corrupted traces. Then, we couple SFR-Net with an Image-Fourier Refinement Network (IFR-Net) which takes both an image and its Fourier spectrum to improve a CT image reconstructed from the SFR-Net output using cross-domain contextual information. Quad-Net is trained on clinical datasets to minimize a composite loss function. Quad-Net does not require precise metal masks, which is of great importance in clinical practice. Our experimental results demonstrate the superiority of Quad-Net over the state-of-the-art MAR methods quantitatively, visually, and statistically. The Quad-Net code is publicly available at https://github.com/longzilicart/Quad-Net.
Collapse
|
24
|
Zhang J, Wang Z, Cao T, Cao G, Ren W, Jiang J. Robust residual-guided iterative reconstruction for sparse-view CT in small animal imaging. Phys Med Biol 2024; 69:105010. [PMID: 38507796 DOI: 10.1088/1361-6560/ad360a] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Accepted: 03/20/2024] [Indexed: 03/22/2024]
Abstract
Objective. We introduce a robust image reconstruction algorithm named residual-guided Golub-Kahan iterative reconstruction technique (RGIRT) designed for sparse-view computed tomography (CT), which aims at high-fidelity image reconstruction from a limited number of projection views.Approach. RGIRT utilizes an inner-outer dual iteration framework, with a flexible least square QR (FLSQR) algorithm implemented in the inner iteration and a restarted iterative scheme applied in the outer iteration. The inner FLSQR employs a flexible Golub-Kahan bidiagonalization method to reduce the size of the inverse problem, and a weighted generalized cross-validation method to adaptively estimate the regularization hyper-parameter. The inner iteration efficiently yields the intermediate reconstruction result, while the outer iteration minimizes the residual and refines the solution by using the result obtained from the inner iteration.Main results. The reconstruction performance of RGIRT is evaluated and compared to other reference methods (FBPConvNet, SART-TV, and FLSQR) using projection data from both numerical phantoms and real experimental Micro-CT data. The experimental findings, from testing various numbers of projection views and different noise levels, underscore the robustness of RGIRT. Meanwhile, theoretical analysis confirms the convergence of residual for our approach.Significance. We propose a robust iterative reconstruction algorithm for x-ray CT scans with sparse views, thereby shortening scanning time and mitigating excessive ionizing radiation exposure to small animals.
Collapse
Affiliation(s)
- Jianru Zhang
- School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, People's Republic of China
- School of Mathematics, University of Birmingham, Edgbaston, B15 2TT, United Kingdom
| | - Zhe Wang
- School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, 201210, People's Republic of China
| | - Tuoyu Cao
- United Imaging Healthcare Co., Ltd, Shanghai, 201807, People's Republic of China
| | - Guohua Cao
- School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, 201210, People's Republic of China
| | - Wuwei Ren
- School of Information Science and Technology, ShanghaiTech University, Shanghai, 201210, People's Republic of China
| | - Jiahua Jiang
- Institute of Mathematical Science, ShanghaiTech University, Shanghai, 201210, People's Republic of China
- School of Mathematics, University of Birmingham, Edgbaston, B15 2TT, United Kingdom
| |
Collapse
|
25
|
Pham M, Lu X, Rana A, Osher S, Miao J. Real space iterative reconstruction for vector tomography (RESIRE-V). Sci Rep 2024; 14:9541. [PMID: 38664487 PMCID: PMC11045750 DOI: 10.1038/s41598-024-59140-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 04/08/2024] [Indexed: 04/28/2024] Open
Abstract
Tomography has had an important impact on the physical, biological, and medical sciences. To date, most tomographic applications have been focused on 3D scalar reconstructions. However, in some crucial applications, vector tomography is required to reconstruct 3D vector fields such as the electric and magnetic fields. Over the years, several vector tomography methods have been developed. Here, we present the mathematical foundation and algorithmic implementation of REal Space Iterative REconstruction for Vector tomography, termed RESIRE-V. RESIRE-V uses multiple tilt series of projections and iterates between the projections and a 3D reconstruction. Each iteration consists of a forward step using the Radon transform and a backward step using its transpose, then updates the object via gradient descent. Incorporating with a 3D support constraint, the algorithm iteratively minimizes an error metric, defined as the difference between the measured and calculated projections. The algorithm can also be used to refine the tilt angles and further improve the 3D reconstruction. To validate RESIRE-V, we first apply it to a simulated data set of the 3D magnetization vector field, consisting of two orthogonal tilt series, each with a missing wedge. Our quantitative analysis shows that the three components of the reconstructed magnetization vector field agree well with the ground-truth counterparts. We then use RESIRE-V to reconstruct the 3D magnetization vector field of a ferromagnetic meta-lattice consisting of three tilt series. Our 3D vector reconstruction reveals the existence of topological magnetic defects with positive and negative charges. We expect that RESIRE-V can be incorporated into different imaging modalities as a general vector tomography method. To make the algorithm accessible to a broad user community, we have made our RESIRE-V MATLAB source codes and the data freely available at https://github.com/minhpham0309/RESIRE-V .
Collapse
Affiliation(s)
- Minh Pham
- Department of Physics and Astronomy and California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA.
- Department of Mathematics, University of California, Los Angeles, CA, 90095, USA.
- Institute of Pure and Applied Mathematics, University of California, Los Angeles, CA, 90095, USA.
| | - Xingyuan Lu
- Department of Physics and Astronomy and California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
- School of Physical Science and Technology, Soochow University, Suzhou, 215006, China
| | - Arjun Rana
- Department of Physics and Astronomy and California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA
| | - Stanley Osher
- Department of Mathematics, University of California, Los Angeles, CA, 90095, USA
- Institute of Pure and Applied Mathematics, University of California, Los Angeles, CA, 90095, USA
| | - Jianwei Miao
- Department of Physics and Astronomy and California NanoSystems Institute, University of California, Los Angeles, CA, 90095, USA.
| |
Collapse
|
26
|
Xiong S, Yang X. Optical color routing enabled by deep learning. NANOSCALE 2024. [PMID: 38592716 DOI: 10.1039/d4nr00105b] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/10/2024]
Abstract
Nano-color routing has emerged as an immensely popular and widely discussed subject in the realms of light field manipulation, image sensing, and the integration of deep learning. The conventional dye filters employed in commercial applications have long been hampered by several limitations, including subpar signal-to-noise ratio, restricted upper bounds on optical efficiency, and challenges associated with miniaturization. Nonetheless, the advent of bandpass-free color routing has opened up unprecedented avenues for achieving remarkable optical spectral efficiency and operation at sub-wavelength scales within the area of image sensing applications. This has brought about a paradigm shift, fundamentally transforming the field by offering a promising solution to surmount the constraints encountered with traditional dye filters. This review presents a comprehensive exploration of representative deep learning-driven nano-color routing structure designs, encompassing forward simulation algorithms, photonic neural networks, and various global and local topology optimization methods. A thorough comparison is drawn between the exceptional light-splitting capabilities exhibited by these methods and those of traditional design approaches. Additionally, the existing research on color routing is summarized, highlighting a promising direction for forthcoming development, delivering valuable insights to advance the field of color routing and serving as a powerful reference for future endeavors.
Collapse
Affiliation(s)
- Shijie Xiong
- Guangdong Provincial Key Laboratory of Nanophotonic Manipulation, Institute of Nanophotonics, College of Physics & Optoelectronic Engineering, Jinan University, Guangzhou 511443, China.
| | - Xianguang Yang
- Guangdong Provincial Key Laboratory of Nanophotonic Manipulation, Institute of Nanophotonics, College of Physics & Optoelectronic Engineering, Jinan University, Guangzhou 511443, China.
| |
Collapse
|
27
|
Li W, Juanes R. Dynamic imaging of force chains in 3D granular media. Proc Natl Acad Sci U S A 2024; 121:e2319160121. [PMID: 38527198 PMCID: PMC10998587 DOI: 10.1073/pnas.2319160121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Accepted: 02/25/2024] [Indexed: 03/27/2024] Open
Abstract
Granular media constitute the most abundant form of solid matter on Earth and beyond. When external forces are applied to a granular medium, the forces are transmitted through it via chains of contacts among grains-force chains. Understanding the spatial structure and temporal evolution of force chains constitutes a fundamental goal of granular mechanics. Here, we introduce an experimental technique, interference optical projection tomography, to study force chains in three-dimensional (3D) granular packs under triaxial shear loads and illustrate the technique with random assemblies of spheres and icosahedra. We find that, in response to an increasing vertical load, the pack of spheres forms intensifying vertical force chains, while the pack of icosahedra forms more interconnected force-chain networks. This provides microscopic insights into why particles with more angularity are more resistant to shear failure-the interconnected force-chain network is stronger (that is, more resilient to topological collapse) than the isolated force chains in round particles. The longer force chains with less branching in the pack of round particles are more likely to buckle, which leads to the macroscopic failure of the pack. This work paves the way for understanding the grain-scale underpinning of localized failure of 3D granular media, such as shear localization in landslides and stick-slip frictional motion in tectonic and induced earthquakes.
Collapse
Affiliation(s)
- Wei Li
- Department of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA02139
- Department of Civil Engineering, Stony Brook University, Stony Brook, NY11794
| | - Ruben Juanes
- Department of Civil and Environmental Engineering, Massachusetts Institute of Technology, Cambridge, MA02139
| |
Collapse
|
28
|
Li G, Togo R, Ogawa T, Haseyama M. Importance-aware adaptive dataset distillation. Neural Netw 2024; 172:106154. [PMID: 38309137 DOI: 10.1016/j.neunet.2024.106154] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Revised: 01/04/2024] [Accepted: 01/28/2024] [Indexed: 02/05/2024]
Abstract
Herein, we propose a novel dataset distillation method for constructing small informative datasets that preserve the information of the large original datasets. The development of deep learning models is enabled by the availability of large-scale datasets. Despite unprecedented success, large-scale datasets considerably increase the storage and transmission costs, resulting in a cumbersome model training process. Moreover, using raw data for training raises privacy and copyright concerns. To address these issues, a new task named dataset distillation has been introduced, aiming to synthesize a compact dataset that retains the essential information from the large original dataset. State-of-the-art (SOTA) dataset distillation methods have been proposed by matching gradients or network parameters obtained during training on real and synthetic datasets. The contribution of different network parameters to the distillation process varies, and uniformly treating them leads to degraded distillation performance. Based on this observation, we propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance by automatically assigning importance weights to different network parameters during distillation, thereby synthesizing more robust distilled datasets. IADD demonstrates superior performance over other SOTA dataset distillation methods based on parameter matching on multiple benchmark datasets and outperforms them in terms of cross-architecture generalization. In addition, the analysis of self-adaptive weights demonstrates the effectiveness of IADD. Furthermore, the effectiveness of IADD is validated in a real-world medical application such as COVID-19 detection.
Collapse
Affiliation(s)
- Guang Li
- Education and Research Center for Mathematical and Data Science, Hokkaido University, N-12, W-7, Kita-Ku, Sapporo, 060-0812, Japan.
| | - Ren Togo
- Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-Ku, Sapporo, 060-0814, Japan.
| | - Takahiro Ogawa
- Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-Ku, Sapporo, 060-0814, Japan.
| | - Miki Haseyama
- Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-Ku, Sapporo, 060-0814, Japan.
| |
Collapse
|
29
|
Lu B, Fu L, Pan Y, Dong Y. SWISTA-Nets: Subband-adaptive wavelet iterative shrinkage thresholding networks for image reconstruction. Comput Med Imaging Graph 2024; 113:102345. [PMID: 38330636 DOI: 10.1016/j.compmedimag.2024.102345] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2023] [Revised: 01/29/2024] [Accepted: 01/29/2024] [Indexed: 02/10/2024]
Abstract
Robust and interpretable image reconstruction is central to imageology applications in clinical practice. Prevalent deep networks, with strong learning ability to extract implicit information from data manifold, are still lack of prior knowledge introduced from mathematics or physics, leading to instability, poor structure interpretability and high computation cost. As to this issue, we propose two prior knowledge-driven networks to combine the good interpretability of mathematical methods and the powerful learnability of deep learning methods. Incorporating different kinds of prior knowledge, we propose subband-adaptive wavelet iterative shrinkage thresholding networks (SWISTA-Nets), where almost every network module is in one-to-one correspondence with each step involved in the iterative algorithm. By end-to-end training of proposed SWISTA-Nets, implicit information can be extracted from training data and guide the tuning process of key parameters that possess mathematical definition. The inverse problems associated with two medical imaging modalities, i.e., electromagnetic tomography and X-ray computational tomography are applied to validate the proposed networks. Both visual and quantitative results indicate that the SWISTA-Nets outperform mathematical methods and state-of-the-art prior knowledge-driven networks, especially with fewer training parameters, interpretable network structures and well robustness. We assume that our analysis will support further investigation of prior knowledge-driven networks in the field of ill-posed image reconstruction.
Collapse
Affiliation(s)
- Binchun Lu
- Department of Precision Instrument, Tsinghua University, Beijing 100084, China.
| | - Lidan Fu
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China.
| | - Yixuan Pan
- Department of Precision Instrument, Tsinghua University, Beijing 100084, China.
| | - Yonggui Dong
- Department of Precision Instrument, Tsinghua University, Beijing 100084, China.
| |
Collapse
|
30
|
He B, Sun C, Li H, Wang Y, She Y, Zhao M, Fang M, Zhu Y, Wang K, Liu Z, Wei Z, Mu W, Wang S, Tang Z, Wei J, Shao L, Tong L, Huang F, Tang M, Guo Y, Zhang H, Dong D, Chen C, Ma J, Tian J. Breaking boundaries in radiology: redefining AI diagnostics via raw data ahead of reconstruction. Phys Med Biol 2024; 69:075015. [PMID: 38224617 DOI: 10.1088/1361-6560/ad1e7c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Accepted: 01/15/2024] [Indexed: 01/17/2024]
Abstract
Objective.In the realm of utilizing artificial intelligence (AI) for medical image analysis, the paradigm of 'signal-image-knowledge' has remained unchanged. However, the process of 'signal to image' inevitably introduces information distortion, ultimately leading to irrecoverable biases in the 'image to knowledge' process. Our goal is to skip reconstruction and build a diagnostic model directly from the raw data (signal).Approach. This study focuses on computed tomography (CT) and its raw data (sinogram) as the research subjects. We simulate the real-world process of 'human-signal-image' using the workflow 'CT-simulated data- reconstructed CT,' and we develop a novel AI predictive model directly targeting raw data (RCTM). This model comprises orientation, spatial, and global analysis modules, embodying the fusion of local to global information extraction from raw data. We selected 1994 patients with retrospective cases of solid lung nodules and modeled different types of data.Main results. We employed predefined radiomic features to assess the diagnostic feature differences caused by reconstruction. The results indicated that approximately 14% of the features had Spearman correlation coefficients below 0.8. These findings suggest that despite the increasing maturity of CT reconstruction algorithms, they still introduce perturbations to diagnostic features. Moreover, our proposed RCTM achieved an area under the curve (AUC) of 0.863 in the diagnosis task, showcasing a comprehensive superiority over models constructed from secondary reconstructed CTs (0.840, 0.822, and 0.825). Additionally, the performance of RCTM closely resembled that of models constructed from original CT scans (0.868, 0.878, and 0.866).Significance. The diagnostic and therapeutic approach directly based on CT raw data can enhance the precision of AI models and the concept of 'signal-to-image' can be extended to other types of imaging. AI diagnostic models tailored to raw data offer the potential to disrupt the traditional paradigm of 'signal-image-knowledge', opening up new avenues for more accurate medical diagnostics.
Collapse
Affiliation(s)
- Bingxi He
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Caixia Sun
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Hailin Li
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Yongbo Wang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, People's Republic of China
| | - Yunlang She
- Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, People's Republic of China
| | - Mengmeng Zhao
- Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, People's Republic of China
| | - Mengjie Fang
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Yongbei Zhu
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Kun Wang
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Zhenyu Liu
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Ziqi Wei
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Wei Mu
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Shuo Wang
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Zhenchao Tang
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Jingwei Wei
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Lizhi Shao
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Lixia Tong
- Neusoft Medical Systems Co. Ltd, Shenyang, People's Republic of China
| | - Feng Huang
- Neusoft Medical Systems Co. Ltd, Shenyang, People's Republic of China
| | - Mingze Tang
- School of Mechanical and Materials Engineering, North China University of Technology, Beijing, People's Republic of China
| | - Yu Guo
- Department of Radiology, The First Hospital of Jilin University, Changchun, Jilin, People's Republic of China
| | - Huimao Zhang
- Department of Radiology, The First Hospital of Jilin University, Changchun, Jilin, People's Republic of China
| | - Di Dong
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| | - Chang Chen
- Department of Thoracic Surgery, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, People's Republic of China
| | - Jianhua Ma
- School of Biomedical Engineering, Southern Medical University, Guangzhou, People's Republic of China
| | - Jie Tian
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Engineering Medicine, Beihang University, Beijing, People's Republic of China
- Key Laboratory of Big Data-Based Precision Medicine, Beihang University, Ministry of Industry and Information Technology, Beijing, People's Republic of China
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, People's Republic of China
| |
Collapse
|
31
|
Li H, Song Y. Sparse-view X-ray CT based on a box-constrained nonlinear weighted anisotropic TV regularization. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2024; 21:5047-5067. [PMID: 38872526 DOI: 10.3934/mbe.2024223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2024]
Abstract
Sparse-view computed tomography (CT) is an important way to reduce the negative effect of radiation exposure in medical imaging by skipping some X-ray projections. However, due to violating the Nyquist/Shannon sampling criterion, there are severe streaking artifacts in the reconstructed CT images that could mislead diagnosis. Noting the ill-posedness nature of the corresponding inverse problem in a sparse-view CT, minimizing an energy functional composed by an image fidelity term together with properly chosen regularization terms is widely used to reconstruct a medical meaningful attenuation image. In this paper, we propose a regularization, called the box-constrained nonlinear weighted anisotropic total variation (box-constrained NWATV), and minimize the regularization term accompanying the least square fitting using an alternative direction method of multipliers (ADMM) type method. The proposed method is validated through the Shepp-Logan phantom model, alongisde the actual walnut X-ray projections provided by Finnish Inverse Problems Society and the human lung images. The experimental results show that the reconstruction speed of the proposed method is significantly accelerated compared to the existing $ L_1/L_2 $ regularization method. Precisely, the central processing unit (CPU) time is reduced more than 8 times.
Collapse
Affiliation(s)
- Huiying Li
- School of Mathematics and Statistics, Shandong Normal University, Jinan 250014, China
| | - Yizhuang Song
- School of Mathematics and Statistics, Shandong Normal University, Jinan 250014, China
| |
Collapse
|
32
|
Li S, Wang B, Yu J, He X, Guo H, He X. FSMN-Net: a free space matching network based on manifold convolution for optical molecular tomography. OPTICS LETTERS 2024; 49:1161-1164. [PMID: 38426963 DOI: 10.1364/ol.512235] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Accepted: 01/22/2024] [Indexed: 03/02/2024]
Abstract
Optical molecular tomography (OMT) can monitor glioblastomas in small animals non-invasively. Although deep learning (DL) methods have made remarkable achievements in this field, improving its generalization against diverse reconstruction systems remains a formidable challenge. In this Letter, a free space matching network (FSMN-Net) was presented to overcome the parameter mismatch problem in different reconstruction systems. Specifically, a novel, to the best of our knowledge, manifold convolution operator was designed by considering the mathematical model of OMT as a space matching process. Based on the dynamic domain expansion concept, an end-to-end fully convolutional codec further integrates this operator to realize robust reconstruction with voxel-level accuracy. The results of numerical simulations and in vivo experiments demonstrate that the FSMN-Net can stably generate high-resolution reconstruction volumetric images under different reconstruction systems.
Collapse
|
33
|
Nigam S, Gjelaj E, Wang R, Wei GW, Wang P. Machine Learning and Deep Learning Applications in Magnetic Particle Imaging. J Magn Reson Imaging 2024:10.1002/jmri.29294. [PMID: 38358090 PMCID: PMC11324856 DOI: 10.1002/jmri.29294] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 01/29/2024] [Accepted: 01/31/2024] [Indexed: 02/16/2024] Open
Abstract
In recent years, magnetic particle imaging (MPI) has emerged as a promising imaging technique depicting high sensitivity and spatial resolution. It originated in the early 2000s where it proposed a new approach to challenge the low spatial resolution achieved by using relaxometry in order to measure the magnetic fields. MPI presents 2D and 3D images with high temporal resolution, non-ionizing radiation, and optimal visual contrast due to its lack of background tissue signal. Traditionally, the images were reconstructed by the conversion of signal from the induced voltage by generating system matrix and X-space based methods. Because image reconstruction and analyses play an integral role in obtaining precise information from MPI signals, newer artificial intelligence-based methods are continuously being researched and developed upon. In this work, we summarize and review the significance and employment of machine learning and deep learning models for applications with MPI and the potential they hold for the future. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 1.
Collapse
Affiliation(s)
- Saumya Nigam
- Precision Health Program, Michigan State University, East Lansing, Michigan 48824, United States
- Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan 48824, United States
| | - Elvira Gjelaj
- Precision Health Program, Michigan State University, East Lansing, Michigan 48824, United States
- Lyman Briggs College, Michigan State University, East Lansing, Michigan 48824, United States
| | - Rui Wang
- Department of Mathematics, College of Natural Science, Michigan State University, East Lansing, Michigan, 48824, United States
| | - Guo-Wei Wei
- Department of Mathematics, College of Natural Science, Michigan State University, East Lansing, Michigan, 48824, United States
- Department of Electrical and Computer Engineering, College of Engineering, Michigan State University, East Lansing, Michigan, 48824, United States
- Department of Biochemistry and Molecular Biology, College of Natural Science, Michigan State University, East Lansing, Michigan, 48824, United States
| | - Ping Wang
- Precision Health Program, Michigan State University, East Lansing, Michigan 48824, United States
- Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan 48824, United States
| |
Collapse
|
34
|
Sun T, Yu M, Yu L, Deng D, Chen M, Lin H, Chen S, Chang C, Chen X. Iterative Reconstruction Algorithms in Magneto-Acousto-Electrical Computed Tomography (MAE-CT) for Image Quality Improvement. IEEE Trans Biomed Eng 2024; 71:669-678. [PMID: 37698962 DOI: 10.1109/tbme.2023.3314617] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/14/2023]
Abstract
Magneto-acousto-electrical computed tomography (MAE-CT) is a recently developed rotational magneto-acousto-electrical tomography (MAET) method, which can map the conductivity parameter of tissues with high spatial resolution. Since the imaging mode of MAE-CT is similar to that of CT, the reconstruction algorithms for CT are possible to be adopted for MAE-CT. Previous studies have demonstrated that the filtered back-projection (FBP) algorithm, which is one of the most common CT reconstruction algorithms, can be used for MAE-CT reconstruction. However, FBP has some inherent shortcomings of being sensitive to noise and non-uniform distribution of views. In this study, we introduced iterative reconstruction (IR) method in MAE-CT reconstruction and compared its performance with that of the FBP. The numerical simulation, the phantom, and in vitro experiments were performed, and several IR algorithms (ART, SART, SIRT) were used for reconstruction. The results show that the images reconstructed by the FBP and IR are similar when the data is noise-free in the simulation. As the noise level increases, the images reconstructed by SART and SIRT are more robust to the noise than FBP. In the phantom experiment, noise and some stripe artifacts caused by the FBP are removed by SART and SIRT algorithms. In conclusion, the IR method used in CT is applicable in MAE-CT, and it performs better than FBP, which indicates that the state-of-the-art achievements in the CT algorithm can also be adopted for the MAE-CT reconstruction in the future.
Collapse
|
35
|
Zhang C, Chen GH. Deep-Interior: A new pathway to interior tomographic image reconstruction via a weighted backprojection and deep learning. Med Phys 2024; 51:946-963. [PMID: 38063251 PMCID: PMC10993302 DOI: 10.1002/mp.16880] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2023] [Revised: 11/21/2023] [Accepted: 11/22/2023] [Indexed: 02/10/2024] Open
Abstract
BACKGROUND In recent years, deep learning strategies have been combined with either the filtered backprojection or iterative methods or the direct projection-to-image by deep learning only to reconstruct images. Some of these methods can be applied to address the interior reconstruction problems for centered regions of interest (ROIs) with fixed sizes. Developing a method to enable interior tomography with arbitrarily located ROIs with nearly arbitrary ROI sizes inside a scanning field of view (FOV) remains an open question. PURPOSE To develop a new pathway to enable interior tomographic reconstruction for arbitrarily located ROIs with arbitrary sizes using a single trained deep neural network model. METHODS The method consists of two steps. First, an analytical weighted backprojection reconstruction algorithm was developed to perform domain transform from divergent fan-beam projection data to an intermediate image feature space,B ( x ⃗ ) $B(\vec{x})$ , for an arbitrary size ROI at an arbitrary location inside the FOV. Second, a supervised learning technique was developed to train a deep neural network architecture to perform deconvolution to obtain the true imagef ( x ⃗ ) $f(\vec{x})$ from the new feature spaceB ( x ⃗ ) $B(\vec{x})$ . This two-step method is referred to as Deep-Interior for convenience. Both numerical simulations and experimental studies were performed to validate the proposed Deep-Interior method. RESULTS The results showed that ROIs as small as a diameter of 5 cm could be accurately reconstructed (similarity index 0.985 ± 0.018 on internal testing data and 0.940 ± 0.025 on external testing data) at arbitrary locations within an imaging object covering a wide variety of anatomical structures of different body parts. Besides, ROIs of arbitrary size can be reconstructed by stitching small ROIs without additional training. CONCLUSION The developed Deep-Interior framework can enable interior tomographic reconstruction from divergent fan-beam projections for short-scan and super-short-scan acquisitions for small ROIs (with a diameter larger than 5 cm) at an arbitrary location inside the scanning FOV with high quantitative reconstruction accuracy.
Collapse
Affiliation(s)
- Chengzhu Zhang
- Department of Medical Physics, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, USA
| | - Guang-Hong Chen
- Department of Medical Physics, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, USA
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, Wisconsin, USA
| |
Collapse
|
36
|
Chang H, Kobzarenko V, Mitra D. Inverse radon transform with deep learning: an application in cardiac motion correction. Phys Med Biol 2024; 69:035010. [PMID: 37988757 DOI: 10.1088/1361-6560/ad0eb5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Accepted: 11/21/2023] [Indexed: 11/23/2023]
Abstract
Objective. This paper addresses performing inverse radon transform (IRT) with artificial neural network (ANN) or deep learning, simultaneously with cardiac motion correction (MC). The suggested application domain is cardiac image reconstruction in emission or transmission tomography where IRT is relevant. Our main contribution is in proposing an ANN architecture that is particularly suitable for this purpose.Approach. We validate our approach with two types of datasets. First, we use an abstract object that looks like a heart to simulate motion-blurred radon transform. With the known ground truth in hand, we then train our proposed ANN architecture and validate its effectiveness in MC. Second, we used human cardiac gated datasets for training and validation of our approach. The gating mechanism bins data over time using the electro-cardiogram (ECG) signals for cardiac motion correction.Main results. We have shown that trained ANNs can perform motion-corrected image reconstruction directly from a motion-corrupted sinogram. We have compared our model against two other known ANN-based approaches.Significance. Our method paves the way for eliminating any need for hardware gating in medical imaging.
Collapse
Affiliation(s)
- Haoran Chang
- Department of Electrical Engineering and Computer Science, Florida Institute of Technology, Melbourne, FL 32901, United States of America
| | - Valerie Kobzarenko
- Department of Electrical Engineering and Computer Science, Florida Institute of Technology, Melbourne, FL 32901, United States of America
| | - Debasis Mitra
- Department of Electrical Engineering and Computer Science, Florida Institute of Technology, Melbourne, FL 32901, United States of America
| |
Collapse
|
37
|
Lozenski L, Wang H, Li F, Anastasio M, Wohlberg B, Lin Y, Villa U. Learned Full Waveform Inversion Incorporating Task Information for Ultrasound Computed Tomography. IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING 2024; 10:69-82. [PMID: 39184532 PMCID: PMC11343509 DOI: 10.1109/tci.2024.3351529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/27/2024]
Abstract
Ultrasound computed tomography (USCT) is an emerging imaging modality that holds great promise for breast imaging. Full-waveform inversion (FWI)-based image reconstruction methods incorporate accurate wave physics to produce high spatial resolution quantitative images of speed of sound or other acoustic properties of the breast tissues from USCT measurement data. However, the high computational cost of FWI reconstruction represents a significant burden for its widespread application in a clinical setting. The research reported here investigates the use of a convolutional neural network (CNN) to learn a mapping from USCT waveform data to speed of sound estimates. The CNN was trained using a supervised approach with a task-informed loss function aiming at preserving features of the image that are relevant to the detection of lesions. A large set of anatomically and physiologically realistic numerical breast phantoms (NBPs) and corresponding simulated USCT measurements was employed during training. Once trained, the CNN can perform real-time FWI image reconstruction from USCT waveform data. The performance of the proposed method was assessed and compared against FWI using a hold-out sample of 41 NBPs and corresponding USCT data. Accuracy was measured using relative mean square error (RMSE), structural self-similarity index measure (SSIM), and lesion detection performance (DICE score). This numerical experiment demonstrates that a supervised learning model can achieve accuracy comparable to FWI in terms of RMSE and SSIM, and better performance in terms of task performance, while significantly reducing computational time.
Collapse
Affiliation(s)
- Luke Lozenski
- Department of Electrical and Systems Engineering, Washington University in St. Louis, St. Louis, MO 63130, USA and the Energy and Natural Resources Security Group, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
| | - Hanchen Wang
- Energy and Natural Resources Security Group, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
| | - Fu Li
- Department of Bioengineering, University of Illinois Urbana-Champaign, Urbana, IL, 61801 USA
| | - Mark Anastasio
- Department of Bioengineering, University of Illinois Urbana-Champaign, Urbana, IL, 61801 USA
| | - Brendt Wohlberg
- Applied Mathematics and Plasma Physics Group, Los Alamos National Laboratory, Los Alamos, NM 87545, USA
| | - Youzuo Lin
- School of Data Science and Society, the University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA, and the Energy and Natural Resources Security Group Los Alamos National Laboratory, Los Alamos, NM 87545, USA
| | - Umberto Villa
- Oden Institute for Computational Engineering and Sciences, University of Texas at Austin, Austin, TX 78712
| |
Collapse
|
38
|
Schramm G, Thielemans K. PARALLELPROJ-an open-source framework for fast calculation of projections in tomography. FRONTIERS IN NUCLEAR MEDICINE (LAUSANNE, SWITZERLAND) 2024; 3:1324562. [PMID: 39355030 PMCID: PMC11440996 DOI: 10.3389/fnume.2023.1324562] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/19/2023] [Accepted: 12/07/2023] [Indexed: 10/03/2024]
Abstract
In this article, we introduce parallelproj, a novel open-source framework designed for efficient parallel computation of projections in tomography leveraging either multiple CPU cores or GPUs. This framework efficiently implements forward and back projection functions for both sinogram and listmode data, utilizing Joseph's method, which is further extended to encompass time-of-flight (TOF) PET projections. Our evaluation involves a series of tests focusing on PET image reconstruction using data sourced from a state-of-the-art clinical PET/CT system. We thoroughly benchmark the performance of the projectors in non-TOF and TOF, sinogram, and listmode employing multi CPU-cores, hybrid CPU/GPU, and exclusive GPU mode. Moreover, we also investigate the timing of non-TOF sinogram projections calculated in STIR (Software for Tomographic Image Reconstruction) which recently integrated parallelproj as one of its projection backends. Our results indicate that the exclusive GPU mode provides acceleration factors between 25 and 68 relative to the multi-CPU-core mode. Furthermore, we demonstrate that OSEM listmode reconstruction of state-of-the-art real-world PET data sets is achievable within a few seconds using a single consumer GPU.
Collapse
Affiliation(s)
- Georg Schramm
- Department of Imaging and Pathology, Division of Nuclear Medicine, KU Leuven, Leuven, Belgium
| | - Kris Thielemans
- Institute of Nuclear Medicine, University College London, London, United Kingdom
- Centre for Medical Image Computing, University College London, London, United Kingdom
| |
Collapse
|
39
|
Mvuh FL, Ebode Ko'a COV, Bodo B. Multichannel high noise level ECG denoising based on adversarial deep learning. Sci Rep 2024; 14:801. [PMID: 38191583 PMCID: PMC10774433 DOI: 10.1038/s41598-023-50334-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Accepted: 12/19/2023] [Indexed: 01/10/2024] Open
Abstract
This paper proposes a denoising method based on an adversarial deep learning approach for the post-processing of multi-channel fetal electrocardiogram (ECG) signals. As it's well known, noise leads to misinterpretations of fetal ECG signals and thus limits the use of fetal electrocardiography for healthcare applications. Therefore, denoising algorithms are essential for the exploitation of non-invasive fetal ECG. The proposed method is based on the combination of three end-to-end trained sub-networks to convert noisy fetal ECG signals into clean signals. The first two sub-networks are linked by skip connections and form a deep convolutional network that downsamples the noisy signals into a latent representation and subsequently upsamples this latent representation to recover clean signals. The third sub-network aims to boost the decoder sub-network to generate realistic clean signals. Experiments carried out on synthetic and real data showed that the proposed method improved by the signal-to-noise (SNR) of fetal ECG signals with input SNR ranging from [Formula: see text] to 0 dB by an average of 20 dB, and improve fetal signal quality by significantly increasing the number of true detected QRS complexes and halving QRS complex detection errors.
Collapse
Affiliation(s)
- Franck Lino Mvuh
- Departement of Physics, University of Yaoundé 1, PO.BOX 812, Yaoundé, Cameroon
| | | | - Bertrand Bodo
- Departement of Physics, University of Yaoundé 1, PO.BOX 812, Yaoundé, Cameroon.
| |
Collapse
|
40
|
Wang D, Jiang C, He J, Teng Y, Qin H, Liu J, Yang X. M 3S-Net: multi-modality multi-branch multi-self-attention network with structure-promoting loss for low-dose PET/CT enhancement. Phys Med Biol 2024; 69:025001. [PMID: 38086073 DOI: 10.1088/1361-6560/ad14c5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2023] [Accepted: 12/12/2023] [Indexed: 01/05/2024]
Abstract
Objective.PET (Positron Emission Tomography) inherently involves radiotracer injections and long scanning time, which raises concerns about the risk of radiation exposure and patient comfort. Reductions in radiotracer dosage and acquisition time can lower the potential risk and improve patient comfort, respectively, but both will also reduce photon counts and hence degrade the image quality. Therefore, it is of interest to improve the quality of low-dose PET images.Approach.A supervised multi-modality deep learning model, named M3S-Net, was proposed to generate standard-dose PET images (60 s per bed position) from low-dose ones (10 s per bed position) and the corresponding CT images. Specifically, we designed a multi-branch convolutional neural network with multi-self-attention mechanisms, which first extracted features from PET and CT images in two separate branches and then fused the features to generate the final generated PET images. Moreover, a novel multi-modality structure-promoting term was proposed in the loss function to learn the anatomical information contained in CT images.Main results.We conducted extensive numerical experiments on real clinical data collected from local hospitals. Compared with state-of-the-art methods, the proposed M3S-Net not only achieved higher objective metrics and better generated tumors, but also performed better in preserving edges and suppressing noise and artifacts.Significance.The experimental results of quantitative metrics and qualitative displays demonstrate that the proposed M3S-Net can generate high-quality PET images from low-dose ones, which are competable to standard-dose PET images. This is valuable in reducing PET acquisition time and has potential applications in dynamic PET imaging.
Collapse
Affiliation(s)
- Dong Wang
- School of Mathematics/S.T.Yau Center of Southeast University, Southeast University, 210096, People's Republic of China
- Nanjing Center of Applied Mathematics, Nanjing, 211135, People's Republic of China
| | - Chong Jiang
- Department of Nuclear Medicine, West China Hospital of Sichuan University, Sichuan University, Chengdu, 610041, People's Republic of China
| | - Jian He
- Department of Nuclear Medicine, Nanjing Drum Tower Hospital, the Affiliated Hospital of Nanjing University Medical School, Nanjing, 210008, People's Republic of China
| | - Yue Teng
- Department of Nuclear Medicine, Nanjing Drum Tower Hospital, the Affiliated Hospital of Nanjing University Medical School, Nanjing, 210008, People's Republic of China
| | - Hourong Qin
- Department of Mathematics, Nanjing University, Nanjing, 210093, People's Republic of China
| | - Jijun Liu
- School of Mathematics/S.T.Yau Center of Southeast University, Southeast University, 210096, People's Republic of China
- Nanjing Center of Applied Mathematics, Nanjing, 211135, People's Republic of China
| | - Xiaoping Yang
- Department of Mathematics, Nanjing University, Nanjing, 210093, People's Republic of China
| |
Collapse
|
41
|
Chao L, Wang Y, Zhang T, Shan W, Zhang H, Wang Z, Li Q. Joint denoising and interpolating network for low-dose cone-beam CT reconstruction under hybrid dose-reduction strategy. Comput Biol Med 2024; 168:107830. [PMID: 38086140 DOI: 10.1016/j.compbiomed.2023.107830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 11/12/2023] [Accepted: 12/04/2023] [Indexed: 01/10/2024]
Abstract
Cone-beam computed tomography (CBCT) is generally reconstructed with hundreds of two-dimensional X-Ray projections through the FDK algorithm, and its excessive ionizing radiation of X-Ray may impair patients' health. Two common dose-reduction strategies are to either lower the intensity of X-Ray, i.e., low-intensity CBCT, or reduce the number of projections, i.e., sparse-view CBCT. Existing efforts improve the low-dose CBCT images only under a single dose-reduction strategy. In this paper, we argue that applying the two strategies simultaneously can reduce dose in a gentle manner and avoid the extreme degradation of the projection data in a single dose-reduction strategy, especially under ultra-low-dose situations. Therefore, we develop a Joint Denoising and Interpolating Network (JDINet) in projection domain to improve the CBCT quality with the hybrid low-intensity and sparse-view projections. Specifically, JDINet mainly includes two important components, i.e., denoising module and interpolating module, to respectively suppress the noise caused by the low-intensity strategy and interpolate the missing projections caused by the sparse-view strategy. Because FDK actually utilizes the projection information after ramp-filtering, we develop a filtered structural similarity constraint to help JDINet focus on the reconstruction-required information. Afterward, we employ a Postprocessing Network (PostNet) in the reconstruction domain to refine the CBCT images that are reconstructed with denoised and interpolated projections. In general, a complete CBCT reconstruction framework is built with JDINet, FDK, and PostNet. Experiments demonstrate that our framework decreases RMSE by approximately 8 %, 15 %, and 17 %, respectively, on the 1/8, 1/16, and 1/32 dose data, compared to the latest methods. In conclusion, our learning-based framework can be deeply imbedded into the CBCT systems to promote the development of CBCT. Source code is available at https://github.com/LianyingChao/FusionLowDoseCBCT.
Collapse
Affiliation(s)
- Lianying Chao
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China
| | - Yanli Wang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China
| | - TaoTao Zhang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China; Northern Jiangsu People's Hospital, Yangzhou, Jiangsu, China
| | - Wenqi Shan
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China
| | - Haobo Zhang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China
| | - Zhiwei Wang
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China
| | - Qiang Li
- Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, Hubei, China; MoE Key Laboratory for Biomedical Photonics, Collaborative Innovation Center for Biomedical Engineering, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, Hubei, China.
| |
Collapse
|
42
|
Gao M, Fessler JA, Chan HP. Model-based deep CNN-regularized reconstruction for digital breast tomosynthesis with a task-based CNN image assessment approach. Phys Med Biol 2023; 68:245024. [PMID: 37988758 PMCID: PMC10719554 DOI: 10.1088/1361-6560/ad0eb4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 11/02/2023] [Accepted: 11/21/2023] [Indexed: 11/23/2023]
Abstract
Objective. Digital breast tomosynthesis (DBT) is a quasi-three-dimensional breast imaging modality that improves breast cancer screening and diagnosis because it reduces fibroglandular tissue overlap compared with 2D mammography. However, DBT suffers from noise and blur problems that can lower the detectability of subtle signs of cancers such as microcalcifications (MCs). Our goal is to improve the image quality of DBT in terms of image noise and MC conspicuity.Approach. We proposed a model-based deep convolutional neural network (deep CNN or DCNN) regularized reconstruction (MDR) for DBT. It combined a model-based iterative reconstruction (MBIR) method that models the detector blur and correlated noise of the DBT system and the learning-based DCNN denoiser using the regularization-by-denoising framework. To facilitate the task-based image quality assessment, we also proposed two DCNN tools for image evaluation: a noise estimator (CNN-NE) trained to estimate the root-mean-square (RMS) noise of the images, and an MC classifier (CNN-MC) as a DCNN model observer to evaluate the detectability of clustered MCs in human subject DBTs.Main results. We demonstrated the efficacies of CNN-NE and CNN-MC on a set of physical phantom DBTs. The MDR method achieved low RMS noise and the highest detection area under the receiver operating characteristic curve (AUC) rankings evaluated by CNN-NE and CNN-MC among the reconstruction methods studied on an independent test set of human subject DBTs.Significance. The CNN-NE and CNN-MC may serve as a cost-effective surrogate for human observers to provide task-specific metrics for image quality comparisons. The proposed reconstruction method shows the promise of combining physics-based MBIR and learning-based DCNNs for DBT image reconstruction, which may potentially lead to lower dose and higher sensitivity and specificity for MC detection in breast cancer screening and diagnosis.
Collapse
Affiliation(s)
- Mingjie Gao
- Department of Radiology, University of Michigan, Ann Arbor, MI 48109, United States of America
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Jeffrey A Fessler
- Department of Radiology, University of Michigan, Ann Arbor, MI 48109, United States of America
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Heang-Ping Chan
- Department of Radiology, University of Michigan, Ann Arbor, MI 48109, United States of America
| |
Collapse
|
43
|
Rahman H, Khan AR, Sadiq T, Farooqi AH, Khan IU, Lim WH. A Systematic Literature Review of 3D Deep Learning Techniques in Computed Tomography Reconstruction. Tomography 2023; 9:2158-2189. [PMID: 38133073 PMCID: PMC10748093 DOI: 10.3390/tomography9060169] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 11/27/2023] [Accepted: 12/01/2023] [Indexed: 12/23/2023] Open
Abstract
Computed tomography (CT) is used in a wide range of medical imaging diagnoses. However, the reconstruction of CT images from raw projection data is inherently complex and is subject to artifacts and noise, which compromises image quality and accuracy. In order to address these challenges, deep learning developments have the potential to improve the reconstruction of computed tomography images. In this regard, our research aim is to determine the techniques that are used for 3D deep learning in CT reconstruction and to identify the training and validation datasets that are accessible. This research was performed on five databases. After a careful assessment of each record based on the objective and scope of the study, we selected 60 research articles for this review. This systematic literature review revealed that convolutional neural networks (CNNs), 3D convolutional neural networks (3D CNNs), and deep learning reconstruction (DLR) were the most suitable deep learning algorithms for CT reconstruction. Additionally, two major datasets appropriate for training and developing deep learning systems were identified: 2016 NIH-AAPM-Mayo and MSCT. These datasets are important resources for the creation and assessment of CT reconstruction models. According to the results, 3D deep learning may increase the effectiveness of CT image reconstruction, boost image quality, and lower radiation exposure. By using these deep learning approaches, CT image reconstruction may be made more precise and effective, improving patient outcomes, diagnostic accuracy, and healthcare system productivity.
Collapse
Affiliation(s)
- Hameedur Rahman
- Department of Computer Games Development, Faculty of Computing & AI, Air University, E9, Islamabad 44000, Pakistan;
| | - Abdur Rehman Khan
- Department of Creative Technologies, Faculty of Computing & AI, Air University, E9, Islamabad 44000, Pakistan;
| | - Touseef Sadiq
- Centre for Artificial Intelligence Research, Department of Information and Communication Technology, University of Agder, Jon Lilletuns vei 9, 4879 Grimstad, Norway
| | - Ashfaq Hussain Farooqi
- Department of Computer Science, Faculty of Computing AI, Air University, Islamabad 44000, Pakistan;
| | - Inam Ullah Khan
- Department of Electronic Engineering, School of Engineering & Applied Sciences (SEAS), Isra University, Islamabad Campus, Islamabad 44000, Pakistan;
| | - Wei Hong Lim
- Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia;
| |
Collapse
|
44
|
Ikuta M, Zhang J. A Deep Convolutional Gated Recurrent Unit for CT Image Reconstruction. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023; 34:10612-10625. [PMID: 35522637 DOI: 10.1109/tnnls.2022.3169569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Computed tomography (CT) is one of the most important medical imaging technologies in use today. Most commercial CT products use a technique known as the filtered backprojection (FBP) that is fast and can produce decent image quality when an X-ray dose is high. However, the FBP is not good enough on low-dose X-ray CT imaging because the CT image reconstruction problem becomes more stochastic. A more effective reconstruction technique proposed recently and implemented in a limited number of CT commercial products is an iterative reconstruction (IR). The IR technique is based on a Bayesian formulation of the CT image reconstruction problem with an explicit model of the CT scanning, including its stochastic nature, and a prior model that incorporates our knowledge about what a good CT image should look like. However, constructing such prior knowledge is more complicated than it seems. In this article, we propose a novel neural network for CT image reconstruction. The network is based on the IR formulation and constructed with a recurrent neural network (RNN). Specifically, we transform the gated recurrent unit (GRU) into a neural network performing CT image reconstruction. We call it "GRU reconstruction." This neural network conducts concurrent dual-domain learning. Many deep learning (DL)-based methods in medical imaging are single-domain learning, but dual-domain learning performs better because it learns from both the sinogram and the image domain. In addition, we propose backpropagation through stage (BPTS) as a new RNN backpropagation algorithm. It is similar to the backpropagation through time (BPTT) of an RNN; however, it is tailored for iterative optimization. Results from extensive experiments indicate that our proposed method outperforms conventional model-based methods, single-domain DL methods, and state-of-the-art DL techniques in terms of the root mean squared error (RMSE), the peak signal-to-noise ratio (PSNR), and the structure similarity (SSIM) and in terms of visual appearance.
Collapse
|
45
|
Deng Z, Zhang W, Chen K, Zhou Y, Tian J, Quan G, Zhao J. TT U-Net: Temporal Transformer U-Net for Motion Artifact Reduction Using PAD (Pseudo All-Phase Clinical-Dataset) in Cardiac CT. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3805-3816. [PMID: 37651491 DOI: 10.1109/tmi.2023.3310933] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/02/2023]
Abstract
Involuntary motion of the heart remains a challenge for cardiac computed tomography (CT) imaging. Although the electrocardiogram (ECG) gating strategy is widely adopted to perform CT scans at the quasi-quiescent cardiac phase, motion-induced artifacts are still unavoidable for patients with high heart rates or irregular rhythms. Dynamic cardiac CT, which provides functional information of the heart, suffers even more severe motion artifacts. In this paper, we develop a deep learning based framework for motion artifact reduction in dynamic cardiac CT. First, we build a PAD (Pseudo All-phase clinical-Dataset) based on a whole-heart motion model and single-phase cardiac CT images. This dataset provides dynamic CT images with realistic-looking motion artifacts that help to develop data-driven approaches. Second, we formulate the problem of motion artifact reduction as a video deblurring task according to its dynamic nature. A novel TT U-Net (Temporal Transformer U-Net) is proposed to excavate the spatiotemporal features for better motion artifact reduction. The self-attention mechanism along the temporal dimension effectively encodes motion information and thus aids image recovery. Experiments show that the TT U-Net trained on the proposed PAD performs well on clinical CT scans, which substantiates the effectiveness and fine generalization ability of our method. The source code, trained models, and dynamic demo will be available at https://github.com/ivy9092111111/TT-U-Net.
Collapse
|
46
|
Shao HC, Li Y, Wang J, Jiang S, Zhang Y. Real-time liver motion estimation via deep learning-based angle-agnostic X-ray imaging. Med Phys 2023; 50:6649-6662. [PMID: 37922461 PMCID: PMC10629841 DOI: 10.1002/mp.16691] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 07/17/2023] [Accepted: 08/06/2023] [Indexed: 11/05/2023] Open
Abstract
BACKGROUND Real-time liver imaging is challenged by the short imaging time (within hundreds of milliseconds) to meet the temporal constraint posted by rapid patient breathing, resulting in extreme under-sampling for desired 3D imaging. Deep learning (DL)-based real-time imaging/motion estimation techniques are emerging as promising solutions, which can use a single X-ray projection to estimate 3D moving liver volumes by solved deformable motion. However, such techniques were mostly developed for a specific, fixed X-ray projection angle, thereby impractical to verify and guide arc-based radiotherapy with continuous gantry rotation. PURPOSE To enable deformable motion estimation and 3D liver imaging from individual X-ray projections acquired at arbitrary X-ray scan angles, and to further improve the accuracy of single X-ray-driven motion estimation. METHODS We developed a DL-based method, X360, to estimate the deformable motion of the liver boundary using an X-ray projection acquired at an arbitrary gantry angle (angle-agnostic). X360 incorporated patient-specific prior information from planning 4D-CTs to address the under-sampling issue, and adopted a deformation-driven approach to deform a prior liver surface mesh to new meshes that reflect real-time motion. The liver mesh motion is solved via motion-related image features encoded in the arbitrary-angle X-ray projection, and through a sequential combination of rigid and deformable registration modules. To achieve the angle agnosticism, a geometry-informed X-ray feature pooling layer was developed to allow X360 to extract angle-dependent image features for motion estimation. As a liver boundary motion solver, X360 was also combined with priorly-developed, DL-based optical surface imaging and biomechanical modeling techniques for intra-liver motion estimation and tumor localization. RESULTS With geometry-aware feature pooling, X360 can solve the liver boundary motion from an arbitrary-angle X-ray projection. Evaluated on a set of 10 liver patient cases, the mean (± s.d.) 95-percentile Hausdorff distance between the solved liver boundary and the "ground-truth" decreased from 10.9 (±4.5) mm (before motion estimation) to 5.5 (±1.9) mm (X360). When X360 was further integrated with surface imaging and biomechanical modeling for liver tumor localization, the mean (± s.d.) center-of-mass localization error of the liver tumors decreased from 9.4 (± 5.1) mm to 2.2 (± 1.7) mm. CONCLUSION X360 can achieve fast and robust liver boundary motion estimation from arbitrary-angle X-ray projections for real-time imaging guidance. Serving as a surface motion solver, X360 can be integrated into a combined framework to achieve accurate, real-time, and marker-less liver tumor localization.
Collapse
Affiliation(s)
- Hua-Chieh Shao
- The Advanced Imaging and Informatics for Radiation Therapy (AIRT) Laboratory, Dallas, Texas, USA
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Dallas, Texas, USA
- Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Yunxiang Li
- The Advanced Imaging and Informatics for Radiation Therapy (AIRT) Laboratory, Dallas, Texas, USA
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Dallas, Texas, USA
- Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Jing Wang
- The Advanced Imaging and Informatics for Radiation Therapy (AIRT) Laboratory, Dallas, Texas, USA
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Dallas, Texas, USA
- Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Steve Jiang
- The Advanced Imaging and Informatics for Radiation Therapy (AIRT) Laboratory, Dallas, Texas, USA
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Dallas, Texas, USA
- Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - You Zhang
- The Advanced Imaging and Informatics for Radiation Therapy (AIRT) Laboratory, Dallas, Texas, USA
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Dallas, Texas, USA
- Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| |
Collapse
|
47
|
Liu Y, Chen G, Pang S, Zeng D, Ding Y, Xie G, Ma J, He J. Cross-Domain Unpaired Learning for Low-Dose CT Imaging. IEEE J Biomed Health Inform 2023; 27:5471-5482. [PMID: 37676796 DOI: 10.1109/jbhi.2023.3312748] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/09/2023]
Abstract
Supervised deep-learning techniques with paired training datasets have been widely studied for low-dose computed tomography (LDCT) imaging with excellent performance. However, the paired training datasets are usually difficult to obtain in clinical routine, which restricts the wide adoption of supervised deep-learning techniques in clinical practices. To address this issue, a general idea is to construct a pseudo paired training dataset based on the widely available unpaired data, after which, supervised deep-learning techniques can be adopted for improving the LDCT imaging performance by training on the pseudo paired training dataset. However, due to the complexity of noise properties in CT imaging, the LDCT data are difficult to generate in order to construct the pseudo paired training dataset. In this article, we propose a simple yet effective cross-domain unpaired learning framework for pseudo LDCT data generation and LDCT image reconstruction, which is denoted as CrossDuL. Specifically, a dedicated pseudo LDCT sinogram generative module is constructed based on a data-dependent noise model in the sinogram domain, and then instead of in the sinogram domain, a pseudo paired dataset is constructed in the image domain to train an LDCT image restoration module. To validate the effectiveness of the proposed framework, clinical datasets are adopted. Experimental results demonstrate that the CrossDuL framework can obtain promising LDCT imaging performance in both quantitative and qualitative measurements.
Collapse
|
48
|
Omori NE, Bobitan AD, Vamvakeros A, Beale AM, Jacques SDM. Recent developments in X-ray diffraction/scattering computed tomography for materials science. PHILOSOPHICAL TRANSACTIONS. SERIES A, MATHEMATICAL, PHYSICAL, AND ENGINEERING SCIENCES 2023; 381:20220350. [PMID: 37691470 PMCID: PMC10493554 DOI: 10.1098/rsta.2022.0350] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Accepted: 07/17/2023] [Indexed: 09/12/2023]
Abstract
X-ray diffraction/scattering computed tomography (XDS-CT) methods are a non-destructive class of chemical imaging techniques that have the capacity to provide reconstructions of sample cross-sections with spatially resolved chemical information. While X-ray diffraction CT (XRD-CT) is the most well-established method, recent advances in instrumentation and data reconstruction have seen greater use of related techniques like small angle X-ray scattering CT and pair distribution function CT. Additionally, the adoption of machine learning techniques for tomographic reconstruction and data analysis are fundamentally disrupting how XDS-CT data is processed. The following narrative review highlights recent developments and applications of XDS-CT with a focus on studies in the last five years. This article is part of the theme issue 'Exploring the length scales, timescales and chemistry of challenging materials (Part 2)'.
Collapse
Affiliation(s)
- Naomi E. Omori
- Finden Limited, Merchant House, 5 East St Helens Street,Abingdon OX14 5EG, UK
| | - Antonia D. Bobitan
- Finden Limited, Merchant House, 5 East St Helens Street,Abingdon OX14 5EG, UK
- Department of Chemistry, University College London, 20 Gordon Street, London WC1H 0AJ, UK
- Research Complex at Harwell, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus, Didcot, Oxon OX11 0FA, UK
| | - Antonis Vamvakeros
- Finden Limited, Merchant House, 5 East St Helens Street,Abingdon OX14 5EG, UK
- Dyson School of Design Engineering, Imperial College London, London SW7 2DB, UK
| | - Andrew M. Beale
- Finden Limited, Merchant House, 5 East St Helens Street,Abingdon OX14 5EG, UK
- Department of Chemistry, University College London, 20 Gordon Street, London WC1H 0AJ, UK
- Research Complex at Harwell, Rutherford Appleton Laboratory, Harwell Science and Innovation Campus, Didcot, Oxon OX11 0FA, UK
| | - Simon D. M. Jacques
- Finden Limited, Merchant House, 5 East St Helens Street,Abingdon OX14 5EG, UK
| |
Collapse
|
49
|
Shimizu K. Near-Infrared Transillumination for Macroscopic Functional Imaging of Animal Bodies. BIOLOGY 2023; 12:1362. [PMID: 37997961 PMCID: PMC10668962 DOI: 10.3390/biology12111362] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Revised: 10/17/2023] [Accepted: 10/18/2023] [Indexed: 11/25/2023]
Abstract
The classical transillumination technique has been revitalized through recent advancements in optical technology, enhancing its applicability in the realm of biomedical research. With a new perspective on near-axis scattered light, we have harnessed near-infrared (NIR) light to visualize intricate internal light-absorbing structures within animal bodies. By leveraging the principle of differentiation, we have extended the applicability of the Beer-Lambert law even in cases of scattering-dominant media, such as animal body tissues. This approach facilitates the visualization of dynamic physiological changes occurring within animal bodies, thereby enabling noninvasive, real-time imaging of macroscopic functionality in vivo. An important challenge inherent to transillumination imaging lies in the image blur caused by pronounced light scattering within body tissues. By extracting near-axis scattered components from the predominant diffusely scattered light, we have achieved cross-sectional imaging of animal bodies. Furthermore, we have introduced software-based techniques encompassing deconvolution using the point spread function and the application of deep learning principles to counteract the scattering effect. Finally, transillumination imaging has been elevated from two-dimensional to three-dimensional imaging. The effectiveness and applicability of these proposed techniques have been validated through comprehensive simulations and experiments involving human and animal subjects. As demonstrated through these studies, transillumination imaging coupled with emerging technologies offers a promising avenue for future biomedical applications.
Collapse
Affiliation(s)
- Koichi Shimizu
- School of Optoelectronic Engineering, Xidian University, Xi’an 710071, China;
- IPS Research Center, Waseda University, Kitakyushu 808-0135, Japan
| |
Collapse
|
50
|
Sahiner B, Chen W, Samala RK, Petrick N. Data drift in medical machine learning: implications and potential remedies. Br J Radiol 2023; 96:20220878. [PMID: 36971405 PMCID: PMC10546450 DOI: 10.1259/bjr.20220878] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Revised: 02/16/2023] [Accepted: 02/20/2023] [Indexed: 03/29/2023] Open
Abstract
Data drift refers to differences between the data used in training a machine learning (ML) model and that applied to the model in real-world operation. Medical ML systems can be exposed to various forms of data drift, including differences between the data sampled for training and used in clinical operation, differences between medical practices or context of use between training and clinical use, and time-related changes in patient populations, disease patterns, and data acquisition, to name a few. In this article, we first review the terminology used in ML literature related to data drift, define distinct types of drift, and discuss in detail potential causes within the context of medical applications with an emphasis on medical imaging. We then review the recent literature regarding the effects of data drift on medical ML systems, which overwhelmingly show that data drift can be a major cause for performance deterioration. We then discuss methods for monitoring data drift and mitigating its effects with an emphasis on pre- and post-deployment techniques. Some of the potential methods for drift detection and issues around model retraining when drift is detected are included. Based on our review, we find that data drift is a major concern in medical ML deployment and that more research is needed so that ML models can identify drift early, incorporate effective mitigation strategies and resist performance decay.
Collapse
Affiliation(s)
- Berkman Sahiner
- Center for Devices and Radiological Health, U.S. Food and Drug Administration 10903 New Hampshire Avenue, Silver Spring, MD 20993-0002
| | - Weijie Chen
- Center for Devices and Radiological Health, U.S. Food and Drug Administration 10903 New Hampshire Avenue, Silver Spring, MD 20993-0002
| | - Ravi K. Samala
- Center for Devices and Radiological Health, U.S. Food and Drug Administration 10903 New Hampshire Avenue, Silver Spring, MD 20993-0002
| | - Nicholas Petrick
- Center for Devices and Radiological Health, U.S. Food and Drug Administration 10903 New Hampshire Avenue, Silver Spring, MD 20993-0002
| |
Collapse
|