1
|
Noda M, Yoshimura H, Okubo T, Koshu R, Uchiyama Y, Nomura A, Ito M, Takumi Y. Feasibility of Multimodal Artificial Intelligence Using GPT-4 Vision for the Classification of Middle Ear Disease: Qualitative Study and Validation. JMIR AI 2024; 3:e58342. [PMID: 38875669 PMCID: PMC11179042 DOI: 10.2196/58342] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/13/2024] [Revised: 04/23/2024] [Accepted: 05/07/2024] [Indexed: 06/16/2024]
Abstract
BACKGROUND The integration of artificial intelligence (AI), particularly deep learning models, has transformed the landscape of medical technology, especially in the field of diagnosis using imaging and physiological data. In otolaryngology, AI has shown promise in image classification for middle ear diseases. However, existing models often lack patient-specific data and clinical context, limiting their universal applicability. The emergence of GPT-4 Vision (GPT-4V) has enabled a multimodal diagnostic approach, integrating language processing with image analysis. OBJECTIVE In this study, we investigated the effectiveness of GPT-4V in diagnosing middle ear diseases by integrating patient-specific data with otoscopic images of the tympanic membrane. METHODS The design of this study was divided into two phases: (1) establishing a model with appropriate prompts and (2) validating the ability of the optimal prompt model to classify images. In total, 305 otoscopic images of 4 middle ear diseases (acute otitis media, middle ear cholesteatoma, chronic otitis media, and otitis media with effusion) were obtained from patients who visited Shinshu University or Jichi Medical University between April 2010 and December 2023. The optimized GPT-4V settings were established using prompts and patients' data, and the model created with the optimal prompt was used to verify the diagnostic accuracy of GPT-4V on 190 images. To compare the diagnostic accuracy of GPT-4V with that of physicians, 30 clinicians completed a web-based questionnaire consisting of 190 images. RESULTS The multimodal AI approach achieved an accuracy of 82.1%, which is superior to that of certified pediatricians at 70.6%, but trailing behind that of otolaryngologists at more than 95%. The model's disease-specific accuracy rates were 89.2% for acute otitis media, 76.5% for chronic otitis media, 79.3% for middle ear cholesteatoma, and 85.7% for otitis media with effusion, which highlights the need for disease-specific optimization. Comparisons with physicians revealed promising results, suggesting the potential of GPT-4V to augment clinical decision-making. CONCLUSIONS Despite its advantages, challenges such as data privacy and ethical considerations must be addressed. Overall, this study underscores the potential of multimodal AI for enhancing diagnostic accuracy and improving patient care in otolaryngology. Further research is warranted to optimize and validate this approach in diverse clinical settings.
Collapse
Affiliation(s)
- Masao Noda
- Department of Otolaryngology, Head and Neck Surgery, Jichi Medical University, Shimotsuke, Japan
| | - Hidekane Yoshimura
- Department of Otolaryngology - Head and Neck Surgery, Shinshu University, Matsumoto, Japan
| | - Takuya Okubo
- Department of Otolaryngology - Head and Neck Surgery, Shinshu University, Matsumoto, Japan
| | - Ryota Koshu
- Department of Otolaryngology, Head and Neck Surgery, Jichi Medical University, Shimotsuke, Japan
| | - Yuki Uchiyama
- Department of Otolaryngology - Head and Neck Surgery, Shinshu University, Matsumoto, Japan
| | - Akihiro Nomura
- College of Transdisciplinary Sciences for Innovation, Kanazawa University, Kanazawa, Japan
| | - Makoto Ito
- Department of Otolaryngology, Head and Neck Surgery, Jichi Medical University, Shimotsuke, Japan
| | - Yutaka Takumi
- Department of Otolaryngology - Head and Neck Surgery, Shinshu University, Matsumoto, Japan
| |
Collapse
|
2
|
Mirshahvalad SA, Farag A, Thiessen J, Wong R, Veit-Haibach P. Current Applications of PET/MR: Part I: Technical Basics and Preclinical/Clinical Applications. Can Assoc Radiol J 2024:8465371241255903. [PMID: 38813998 DOI: 10.1177/08465371241255903] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/31/2024] Open
Abstract
Positron emission tomography/magnetic resonance (PET/MR) imaging has gone through major hardware improvements in recent years, making it a reliable state-of-the-art hybrid modality in clinical practice. At the same time, image reconstruction, attenuation correction, and motion correction algorithms have significantly evolved to provide high-quality images. Part I of the current review discusses technical basics, pre-clinical applications, and clinical applications of PET/MR in radiation oncology and head and neck imaging. PET/MR offers a broad range of advantages in preclinical and clinical imaging. In the preclinic, small and large animal-dedicated devices were developed, making PET/MR capable of delivering new insight into animal models in diseases and facilitating the development of methods that inform clinical PET/MR. Regarding PET/MR's clinical applications in radiation medicine, PET and MR already play crucial roles in the radiotherapy process. Their combination is particularly significant as it can provide molecular and morphological characteristics that are not achievable with other modalities. In addition, the integration of PET/MR information for therapy planning with linear accelerators is expected to provide potentially unique biomarkers for treatment guidance. Furthermore, in clinical applications in the head and neck region, it has been shown that PET/MR can be an accurate modality in head and neck malignancies for staging and resectability assessment. Also, it can play a crucial role in diagnosing residual or recurrent diseases, reliably distinguishing from oedema and fibrosis. PET/MR can furthermore help with tumour characterization and patient prognostication. Lastly, in head and neck carcinoma of unknown origin, PET/MR, with its diagnostic potential, may obviate multiple imaging sessions in the near future.
Collapse
Affiliation(s)
- Seyed Ali Mirshahvalad
- University Medical Imaging Toronto, Toronto Joint Department Medical Imaging, University Health Network, Sinai Health System, Women's College Hospital, University of Toronto, Toronto, ON, Canada
- Department of Medical Imaging, University of Toronto, Toronto, ON, Canada
| | - Adam Farag
- University Medical Imaging Toronto, Toronto Joint Department Medical Imaging, University Health Network, Sinai Health System, Women's College Hospital, University of Toronto, Toronto, ON, Canada
| | - Jonathan Thiessen
- Imaging Program, Lawson Health Research Institute, London, ON, Canada
- Medical Biophysics, Medical Imaging, Western University, London, ON, Canada
| | - Rebecca Wong
- Department of Radiation Oncology, Princess Margaret Cancer Centre, University Health Network, University of Toronto, Toronto, ON, Canada
| | - Patrick Veit-Haibach
- University Medical Imaging Toronto, Toronto Joint Department Medical Imaging, University Health Network, Sinai Health System, Women's College Hospital, University of Toronto, Toronto, ON, Canada
- Department of Medical Imaging, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
3
|
Ferreira A, Li J, Pomykala KL, Kleesiek J, Alves V, Egger J. GAN-based generation of realistic 3D volumetric data: A systematic review and taxonomy. Med Image Anal 2024; 93:103100. [PMID: 38340545 DOI: 10.1016/j.media.2024.103100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2022] [Revised: 11/20/2023] [Accepted: 01/30/2024] [Indexed: 02/12/2024]
Abstract
With the massive proliferation of data-driven algorithms, such as deep learning-based approaches, the availability of high-quality data is of great interest. Volumetric data is very important in medicine, as it ranges from disease diagnoses to therapy monitoring. When the dataset is sufficient, models can be trained to help doctors with these tasks. Unfortunately, there are scenarios where large amounts of data is unavailable. For example, rare diseases and privacy issues can lead to restricted data availability. In non-medical fields, the high cost of obtaining enough high-quality data can also be a concern. A solution to these problems can be the generation of realistic synthetic data using Generative Adversarial Networks (GANs). The existence of these mechanisms is a good asset, especially in healthcare, as the data must be of good quality, realistic, and without privacy issues. Therefore, most of the publications on volumetric GANs are within the medical domain. In this review, we provide a summary of works that generate realistic volumetric synthetic data using GANs. We therefore outline GAN-based methods in these areas with common architectures, loss functions and evaluation metrics, including their advantages and disadvantages. We present a novel taxonomy, evaluations, challenges, and research opportunities to provide a holistic overview of the current state of volumetric GANs.
Collapse
Affiliation(s)
- André Ferreira
- Center Algoritmi/LASI, University of Minho, Braga, 4710-057, Portugal; Computer Algorithms for Medicine Laboratory, Graz, Austria; Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, 52074 Aachen, Germany; Institute of Medical Informatics, University Hospital RWTH Aachen, 52074 Aachen, Germany.
| | - Jianning Li
- Computer Algorithms for Medicine Laboratory, Graz, Austria; Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, Essen, 45147, Germany.
| | - Kelsey L Pomykala
- Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany.
| | - Jens Kleesiek
- Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, Essen, 45147, Germany; German Cancer Consortium (DKTK), Partner Site Essen, Hufelandstraße 55, Essen, 45147, Germany; TU Dortmund University, Department of Physics, Otto-Hahn-Straße 4, 44227 Dortmund, Germany.
| | - Victor Alves
- Center Algoritmi/LASI, University of Minho, Braga, 4710-057, Portugal.
| | - Jan Egger
- Computer Algorithms for Medicine Laboratory, Graz, Austria; Institute for AI in Medicine (IKIM), University Medicine Essen, Girardetstraße 2, Essen, 45131, Germany; Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, Essen, 45147, Germany; Institute of Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16, Graz, 801, Austria.
| |
Collapse
|
4
|
Hou J, Nesaragi N, Tronstad C. Electrical bioimpedance in the era of artificial intelligence. JOURNAL OF ELECTRICAL BIOIMPEDANCE 2024; 15:1-3. [PMID: 38304720 PMCID: PMC10830329 DOI: 10.2478/joeb-2024-0001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Indexed: 02/03/2024]
Affiliation(s)
- Jie Hou
- Department of Physics, University of Oslo, 0316Oslo, Norway
| | | | - Christian Tronstad
- Department of Clinical and Biomedical Engineering, Oslo University Hospital, 0372Oslo, Norway
| |
Collapse
|
5
|
Zama S, Fujioka T, Yamaga E, Kubota K, Mori M, Katsuta L, Yashima Y, Sato A, Kawauchi M, Higuchi S, Kawanishi M, Ishiba T, Oda G, Nakagawa T, Tateishi U. Clinical Utility of Breast Ultrasound Images Synthesized by a Generative Adversarial Network. MEDICINA (KAUNAS, LITHUANIA) 2023; 60:14. [PMID: 38276048 PMCID: PMC10817540 DOI: 10.3390/medicina60010014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 12/10/2023] [Accepted: 12/18/2023] [Indexed: 01/27/2024]
Abstract
BACKGROUND AND OBJECTIVES This study compares the clinical properties of original breast ultrasound images and those synthesized by a generative adversarial network (GAN) to assess the clinical usefulness of GAN-synthesized images. MATERIALS AND METHODS We retrospectively collected approximately 200 breast ultrasound images for each of five representative histological tissue types (cyst, fibroadenoma, scirrhous, solid, and tubule-forming invasive ductal carcinomas) as training images. A deep convolutional GAN (DCGAN) image-generation model synthesized images of the five histological types. Two diagnostic radiologists (reader 1 with 13 years of experience and reader 2 with 7 years of experience) were given a reading test consisting of 50 synthesized and 50 original images (≥1-month interval between sets) to assign the perceived histological tissue type. The percentages of correct diagnoses were calculated, and the reader agreement was assessed using the kappa coefficient. RESULTS The synthetic and original images were indistinguishable. The correct diagnostic rates from the synthetic images for readers 1 and 2 were 86.0% and 78.0% and from the original images were 88.0% and 78.0%, respectively. The kappa values were 0.625 and 0.650 for the synthetic and original images, respectively. The diagnoses made from the DCGAN synthetic images and original images were similar. CONCLUSION The DCGAN-synthesized images closely resemble the original ultrasound images in clinical characteristics, suggesting their potential utility in clinical education and training, particularly for enhancing diagnostic skills in breast ultrasound imaging.
Collapse
Affiliation(s)
- Shu Zama
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Tomoyuki Fujioka
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Emi Yamaga
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Kazunori Kubota
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
- Department of Radiology, Dokkyo Medical University Saitama Medical Center, 2-1-50 Minami-koshigaya, Koshigaya 343-8555, Japan
| | - Mio Mori
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Leona Katsuta
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Yuka Yashima
- Department of Radiology, Dokkyo Medical University Saitama Medical Center, 2-1-50 Minami-koshigaya, Koshigaya 343-8555, Japan
| | - Arisa Sato
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Miho Kawauchi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Subaru Higuchi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Masaaki Kawanishi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Toshiyuki Ishiba
- Department of Surgery, Breast Surgery, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Goshi Oda
- Department of Surgery, Breast Surgery, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Tsuyoshi Nakagawa
- Department of Surgery, Breast Surgery, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Ukihide Tateishi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| |
Collapse
|
6
|
Krokos G, MacKewn J, Dunn J, Marsden P. A review of PET attenuation correction methods for PET-MR. EJNMMI Phys 2023; 10:52. [PMID: 37695384 PMCID: PMC10495310 DOI: 10.1186/s40658-023-00569-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 08/07/2023] [Indexed: 09/12/2023] Open
Abstract
Despite being thirteen years since the installation of the first PET-MR system, the scanners constitute a very small proportion of the total hybrid PET systems installed. This is in stark contrast to the rapid expansion of the PET-CT scanner, which quickly established its importance in patient diagnosis within a similar timeframe. One of the main hurdles is the development of an accurate, reproducible and easy-to-use method for attenuation correction. Quantitative discrepancies in PET images between the manufacturer-provided MR methods and the more established CT- or transmission-based attenuation correction methods have led the scientific community in a continuous effort to develop a robust and accurate alternative. These can be divided into four broad categories: (i) MR-based, (ii) emission-based, (iii) atlas-based and the (iv) machine learning-based attenuation correction, which is rapidly gaining momentum. The first is based on segmenting the MR images in various tissues and allocating a predefined attenuation coefficient for each tissue. Emission-based attenuation correction methods aim in utilising the PET emission data by simultaneously reconstructing the radioactivity distribution and the attenuation image. Atlas-based attenuation correction methods aim to predict a CT or transmission image given an MR image of a new patient, by using databases containing CT or transmission images from the general population. Finally, in machine learning methods, a model that could predict the required image given the acquired MR or non-attenuation-corrected PET image is developed by exploiting the underlying features of the images. Deep learning methods are the dominant approach in this category. Compared to the more traditional machine learning, which uses structured data for building a model, deep learning makes direct use of the acquired images to identify underlying features. This up-to-date review goes through the literature of attenuation correction approaches in PET-MR after categorising them. The various approaches in each category are described and discussed. After exploring each category separately, a general overview is given of the current status and potential future approaches along with a comparison of the four outlined categories.
Collapse
Affiliation(s)
- Georgios Krokos
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK.
| | - Jane MacKewn
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK
| | - Joel Dunn
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK
| | - Paul Marsden
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK
| |
Collapse
|
7
|
Du Y, Xie F, Wu G, Chen P, Yang Y, Yang L, Yin L, Wang S. A classification model for detection of ductal carcinoma in situ by Fourier transform infrared spectroscopy based on deep structured semantic model. Anal Chim Acta 2023; 1251:340991. [PMID: 36925283 DOI: 10.1016/j.aca.2023.340991] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2022] [Revised: 01/26/2023] [Accepted: 02/16/2023] [Indexed: 02/19/2023]
Abstract
At present, deep learning is widely used in spectral data processing. Deep learning requires a large amount of data for training, while the collection of biological serum spectra is limited by sample numbers and labor costs, so it is impractical to obtain a large amount of serum spectral data for disease detection. In this study, we propose a spectral classification model based on the deep structured semantic model (DSSM) and successfully apply it to Fourier Transform Infrared (FT-IR) spectroscopy for ductal carcinoma in situ (DCIS) detection. Compared with the traditional deep learning model, we match the spectral data into positive and negative pairs according to whether the spectra are from the same category. The DSSM structure is constructed by extracting features according to the spectral similarity of spectra pairs. This new construction model increases the data amount used for model training and reduces the dimension of spectral data. Firstly, the FT-IR spectra are paired. The spectra pairs are labeled as positive pairs if they come from the same category, and the spectra pairs are labeled as negative pairs if they come from different categories. Secondly, two spectra in each spectra pair are put into two deep neural networks of the DSSM structure separately. Then the spectral similarity between the output feature maps of two deep neural networks is calculated. The DSSM structure is trained by maximizing the conditional likelihood of the spectra pairs from the same category. Thirdly, after the training of DSSM is done, the training set and testing set are input into two deep neural networks separately. The output feature maps of the training set are put into the reference library. Lastly, the k-nearest neighbor (KNN) model is used for classification according to Euclidean distances between the output feature map of each unknown sample to the reference library. The category of the unknown sample is judged according to the categories of k nearest samples. We also use principal component analysis (PCA) to reduce dimension for comparison. The accuracies of the KNN model, principal component analysis-k nearest neighbor (PCA-KNN) model, and deep structured semantic model-k nearest neighbor (DSSM-KNN) model are 78.8%, 72.7%, and 97.0%, which proves that our proposed model has higher accuracy.
Collapse
Affiliation(s)
- Yu Du
- School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876, China
| | - Fei Xie
- Department of Breast Center, Peking University People's Hospital, Beijing, 100044, China
| | - Guohua Wu
- School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876, China.
| | - Peng Chen
- School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876, China
| | - Yang Yang
- Department of Breast Center, Peking University People's Hospital, Beijing, 100044, China
| | - Liu Yang
- Department of Breast Center, Peking University People's Hospital, Beijing, 100044, China
| | - Longfei Yin
- School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876, China
| | - Shu Wang
- Department of Breast Center, Peking University People's Hospital, Beijing, 100044, China.
| |
Collapse
|
8
|
Ahangari S, Beck Olin A, Kinggård Federspiel M, Jakoby B, Andersen TL, Hansen AE, Fischer BM, Littrup Andersen F. A deep learning-based whole-body solution for PET/MRI attenuation correction. EJNMMI Phys 2022; 9:55. [PMID: 35978211 PMCID: PMC9385907 DOI: 10.1186/s40658-022-00486-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Accepted: 08/08/2022] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Deep convolutional neural networks have demonstrated robust and reliable PET attenuation correction (AC) as an alternative to conventional AC methods in integrated PET/MRI systems. However, its whole-body implementation is still challenging due to anatomical variations and the limited MRI field of view. The aim of this study is to investigate a deep learning (DL) method to generate voxel-based synthetic CT (sCT) from Dixon MRI and use it as a whole-body solution for PET AC in a PET/MRI system. MATERIALS AND METHODS Fifteen patients underwent PET/CT followed by PET/MRI with whole-body coverage from skull to feet. We performed MRI truncation correction and employed co-registered MRI and CT images for training and leave-one-out cross-validation. The network was pretrained with region-specific images. The accuracy of the AC maps and reconstructed PET images were assessed by performing a voxel-wise analysis and calculating the quantification error in SUV obtained using DL-based sCT (PETsCT) and a vendor-provided atlas-based method (PETAtlas), with the CT-based reconstruction (PETCT) serving as the reference. In addition, region-specific analysis was performed to compare the performances of the methods in brain, lung, liver, spine, pelvic bone, and aorta. RESULTS Our DL-based method resulted in better estimates of AC maps with a mean absolute error of 62 HU, compared to 109 HU for the atlas-based method. We found an excellent voxel-by-voxel correlation between PETCT and PETsCT (R2 = 0.98). The absolute percentage difference in PET quantification for the entire image was 6.1% for PETsCT and 11.2% for PETAtlas. The regional analysis showed that the average errors and the variability for PETsCT were lower than PETAtlas in all regions. The largest errors were observed in the lung, while the smallest biases were observed in the brain and liver. CONCLUSIONS Experimental results demonstrated that a DL approach for whole-body PET AC in PET/MRI is feasible and allows for more accurate results compared with conventional methods. Further evaluation using a larger training cohort is required for more accurate and robust performance.
Collapse
Affiliation(s)
- Sahar Ahangari
- Department of Clinical Physiology, Nuclear Medicine, and PET, Rigshospitalet, Copenhagen, Denmark.
| | - Anders Beck Olin
- Department of Clinical Physiology, Nuclear Medicine, and PET, Rigshospitalet, Copenhagen, Denmark
| | | | | | - Thomas Lund Andersen
- Department of Clinical Physiology, Nuclear Medicine, and PET, Rigshospitalet, Copenhagen, Denmark
| | - Adam Espe Hansen
- Department of Clinical Physiology, Nuclear Medicine, and PET, Rigshospitalet, Copenhagen, Denmark.,Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark.,Department of Diagnostic Radiology, Rigshospitalet, Copenhagen, Denmark
| | - Barbara Malene Fischer
- Department of Clinical Physiology, Nuclear Medicine, and PET, Rigshospitalet, Copenhagen, Denmark.,Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Flemming Littrup Andersen
- Department of Clinical Physiology, Nuclear Medicine, and PET, Rigshospitalet, Copenhagen, Denmark.,Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
9
|
Applications of Generative Adversarial Networks (GANs) in Positron Emission Tomography (PET) imaging: A review. Eur J Nucl Med Mol Imaging 2022; 49:3717-3739. [PMID: 35451611 DOI: 10.1007/s00259-022-05805-w] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2021] [Accepted: 04/12/2022] [Indexed: 11/04/2022]
Abstract
PURPOSE This paper reviews recent applications of Generative Adversarial Networks (GANs) in Positron Emission Tomography (PET) imaging. Recent advances in Deep Learning (DL) and GANs catalysed the research of their applications in medical imaging modalities. As a result, several unique GAN topologies have emerged and been assessed in an experimental environment over the last two years. METHODS The present work extensively describes GAN architectures and their applications in PET imaging. The identification of relevant publications was performed via approved publication indexing websites and repositories. Web of Science, Scopus, and Google Scholar were the major sources of information. RESULTS The research identified a hundred articles that address PET imaging applications such as attenuation correction, de-noising, scatter correction, removal of artefacts, image fusion, high-dose image estimation, super-resolution, segmentation, and cross-modality synthesis. These applications are presented and accompanied by the corresponding research works. CONCLUSION GANs are rapidly employed in PET imaging tasks. However, specific limitations must be eliminated to reach their full potential and gain the medical community's trust in everyday clinical practice.
Collapse
|
10
|
Huang Z, Wu Y, Fu F, Meng N, Gu F, Wu Q, Zhou Y, Yang Y, Liu X, Zheng H, Liang D, Wang M, Hu Z. Parametric image generation with the uEXPLORER total-body PET/CT system through deep learning. Eur J Nucl Med Mol Imaging 2022; 49:2482-2492. [DOI: 10.1007/s00259-022-05731-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2021] [Accepted: 02/13/2022] [Indexed: 11/25/2022]
|
11
|
Lai YC, Wu KC, Tseng NC, Chen YJ, Chang CJ, Yen KY, Kao CH. Differentiation Between Malignant and Benign Pulmonary Nodules by Using Automated Three-Dimensional High-Resolution Representation Learning With Fluorodeoxyglucose Positron Emission Tomography-Computed Tomography. Front Med (Lausanne) 2022; 9:773041. [PMID: 35372415 PMCID: PMC8971840 DOI: 10.3389/fmed.2022.773041] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2021] [Accepted: 02/14/2022] [Indexed: 11/26/2022] Open
Abstract
Background The investigation of incidental pulmonary nodules has rapidly become one of the main indications for 18F-fluorodeoxyglucose (FDG) positron emission tomography (PET), currently combined with computed tomography (PET-CT). There is also a growing trend to use artificial Intelligence for optimization and interpretation of PET-CT Images. Therefore, we proposed a novel deep learning model that aided in the automatic differentiation between malignant and benign pulmonary nodules on FDG PET-CT. Methods In total, 112 participants with pulmonary nodules who underwent FDG PET-CT before surgery were enrolled retrospectively. We designed a novel deep learning three-dimensional (3D) high-resolution representation learning (HRRL) model for the automated classification of pulmonary nodules based on FDG PET-CT images without manual annotation by experts. For the images to be localized more precisely, we defined the territories of the lungs through a novel artificial intelligence-driven image-processing algorithm, instead of the conventional segmentation method, without the aid of an expert; this algorithm is based on deep HRRL, which is used to perform high-resolution classification. In addition, the 2D model was converted to a 3D model. Results All pulmonary lesions were confirmed through pathological studies (79 malignant and 33 benign). We evaluated its diagnostic performance in the differentiation of malignant and benign nodules. The area under the receiver operating characteristic curve (AUC) of the deep learning model was used to indicate classification performance in an evaluation using fivefold cross-validation. The nodule-based prediction performance of the model had an AUC, sensitivity, specificity, and accuracy of 78.1, 89.9, 54.5, and 79.4%, respectively. Conclusion Our results suggest that a deep learning algorithm using HRRL without manual annotation from experts might aid in the classification of pulmonary nodules discovered through clinical FDG PET-CT images.
Collapse
Affiliation(s)
- Yung-Chi Lai
- Department of Nuclear Medicine, PET Center, China Medical University Hospital, Taichung, Taiwan
| | - Kuo-Chen Wu
- Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei, Taiwan
- Center of Augmented Intelligence in Healthcare, China Medical University Hospital, Taichung, Taiwan
| | - Neng-Chuan Tseng
- Division of Nuclear Medicine, Tungs’ Taichung MetroHarbor Hospital, Taichung, Taiwan
| | - Yi-Jin Chen
- Center of Augmented Intelligence in Healthcare, China Medical University Hospital, Taichung, Taiwan
| | - Chao-Jen Chang
- Center of Augmented Intelligence in Healthcare, China Medical University Hospital, Taichung, Taiwan
| | - Kuo-Yang Yen
- Department of Nuclear Medicine, PET Center, China Medical University Hospital, Taichung, Taiwan
- Department of Biomedical Imaging and Radiological Science, School of Medicine, College of Medicine, China Medical University, Taichung, Taiwan
| | - Chia-Hung Kao
- Department of Nuclear Medicine, PET Center, China Medical University Hospital, Taichung, Taiwan
- Center of Augmented Intelligence in Healthcare, China Medical University Hospital, Taichung, Taiwan
- Graduate Institute of Biomedical Sciences, College of Medicine, China Medical University, Taichung, Taiwan
- Department of Bioinformatics and Medical Engineering, Asia University, Taichung, Taiwan
- *Correspondence: Chia-Hung Kao, ,
| |
Collapse
|
12
|
Hirata K, Sugimori H, Fujima N, Toyonaga T, Kudo K. Artificial intelligence for nuclear medicine in oncology. Ann Nucl Med 2022; 36:123-132. [PMID: 35028877 DOI: 10.1007/s12149-021-01693-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2021] [Accepted: 11/07/2021] [Indexed: 12/12/2022]
Abstract
As in all other medical fields, artificial intelligence (AI) is increasingly being used in nuclear medicine for oncology. There are many articles that discuss AI from the viewpoint of nuclear medicine, but few focus on nuclear medicine from the viewpoint of AI. Nuclear medicine images are characterized by their low spatial resolution and high quantitativeness. It is noted that AI has been used since before the emergence of deep learning. AI can be divided into three categories by its purpose: (1) assisted interpretation, i.e., computer-aided detection (CADe) or computer-aided diagnosis (CADx). (2) Additional insight, i.e., AI provides information beyond the radiologist's eye, such as predicting genes and prognosis from images. It is also related to the field called radiomics/radiogenomics. (3) Augmented image, i.e., image generation tasks. To apply AI to practical use, harmonization between facilities and the possibility of black box explanations need to be resolved.
Collapse
Affiliation(s)
- Kenji Hirata
- Department of Diagnostic Imaging, Hokkaido University Graduate School of Medicine, Kita 15, Nishi 7, Kita-Ku, Sapporo, Hokkaido, 060-8638, Japan. .,Department of Nuclear Medicine, Hokkaido University Hospital, Sapporo, Japan. .,Division of Medical AI Education and Research, Hokkaido University Graduate School of Medicine, Sapporo, Japan.
| | | | - Noriyuki Fujima
- Department of Diagnostic Imaging, Hokkaido University Graduate School of Medicine, Kita 15, Nishi 7, Kita-Ku, Sapporo, Hokkaido, 060-8638, Japan.,Department of Diagnostic and Interventional Radiology, Hokkaido University Hospital, Sapporo, Japan
| | - Takuya Toyonaga
- Department of Radiology and Biomedical Imaging, Yale University, New Haven, CT, USA
| | - Kohsuke Kudo
- Department of Diagnostic Imaging, Hokkaido University Graduate School of Medicine, Kita 15, Nishi 7, Kita-Ku, Sapporo, Hokkaido, 060-8638, Japan.,Division of Medical AI Education and Research, Hokkaido University Graduate School of Medicine, Sapporo, Japan.,Department of Diagnostic and Interventional Radiology, Hokkaido University Hospital, Sapporo, Japan.,Global Center for Biomedical Science and Engineering, Hokkaido University Faculty of Medicine, Sapporo, Japan
| |
Collapse
|
13
|
Wang P, Nie P, Dang Y, Wang L, Zhu K, Wang H, Wang J, Liu R, Ren J, Feng J, Fan H, Yu J, Chen B. Synthesizing the First Phase of Dynamic Sequences of Breast MRI for Enhanced Lesion Identification. Front Oncol 2021; 11:792516. [PMID: 34950593 PMCID: PMC8689139 DOI: 10.3389/fonc.2021.792516] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2021] [Accepted: 11/15/2021] [Indexed: 12/24/2022] Open
Abstract
Objective To develop a deep learning model for synthesizing the first phases of dynamic (FP-Dyn) sequences to supplement the lack of information in unenhanced breast MRI examinations. Methods In total, 97 patients with breast MRI images were collected as the training set (n = 45), the validation set (n = 31), and the test set (n = 21), respectively. An enhance border lifelike synthesize (EDLS) model was developed in the training set and used to synthesize the FP-Dyn images from the T1WI images in the validation set. The peak signal-to-noise ratio (PSNR), structural similarity (SSIM), mean square error (MSE) and mean absolute error (MAE) of the synthesized images were measured. Moreover, three radiologists subjectively assessed image quality, respectively. The diagnostic value of the synthesized FP-Dyn sequences was further evaluated in the test set. Results The image synthesis performance in the EDLS model was superior to that in conventional models from the results of PSNR, SSIM, MSE, and MAE. Subjective results displayed a remarkable visual consistency between the synthesized and original FP-Dyn images. Moreover, by using a combination of synthesized FP-Dyn sequence and an unenhanced protocol, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of MRI were 100%, 72.73%, 76.92%, and 100%, respectively, which had a similar diagnostic value to full MRI protocols. Conclusions The EDLS model could synthesize the realistic FP-Dyn sequence to supplement the lack of enhanced images. Compared with full MRI examinations, it thus provides a new approach for reducing examination time and cost, and avoids the use of contrast agents without influencing diagnostic accuracy.
Collapse
Affiliation(s)
- Pingping Wang
- Clinical Experimental Centre, Xi'an International Medical Center Hospital, Xi'an, China
| | - Pin Nie
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| | - Yanli Dang
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| | - Lifang Wang
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| | - Kaiguo Zhu
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| | - Hongyu Wang
- The School of Computer Science and Technology, Xi'an University of Posts and Telecommunications, Xi'an, China
| | - Jiawei Wang
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| | - Rumei Liu
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| | | | - Jun Feng
- The School of Information of Science and Technology, Northwest University, Xi'an, China
| | - Haiming Fan
- The School of Medicine, Northwest University, Xi'an, China
| | - Jun Yu
- Clinical Experimental Centre, Xi'an International Medical Center Hospital, Xi'an, China
| | - Baoying Chen
- Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital, Xi'an, China
| |
Collapse
|