1
|
Guo R, Wei J, Sun L, Yu B, Chang G, Liu D, Zhang S, Yao Z, Xu M, Bu L. A survey on advancements in image-text multimodal models: From general techniques to biomedical implementations. Comput Biol Med 2024; 178:108709. [PMID: 38878398 DOI: 10.1016/j.compbiomed.2024.108709] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Revised: 06/01/2024] [Accepted: 06/03/2024] [Indexed: 07/24/2024]
Abstract
With the significant advancements of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), the development of image-text multimodal models has garnered widespread attention. Current surveys on image-text multimodal models mainly focus on representative models or application domains, but lack a review on how general technical models influence the development of domain-specific models, which is crucial for domain researchers. Based on this, this paper first reviews the technological evolution of image-text multimodal models, from early explorations of feature space to visual language encoding structures, and then to the latest large model architectures. Next, from the perspective of technological evolution, we explain how the development of general image-text multimodal technologies promotes the progress of multimodal technologies in the biomedical field, as well as the importance and complexity of specific datasets in the biomedical domain. Then, centered on the tasks of image-text multimodal models, we analyze their common components and challenges. After that, we summarize the architecture, components, and data of general image-text multimodal models, and introduce the applications and improvements of image-text multimodal models in the biomedical field. Finally, we categorize the challenges faced in the development and application of general models into external factors and intrinsic factors, further refining them into 2 external factors and 5 intrinsic factors, and propose targeted solutions, providing guidance for future research directions. For more details and data, please visit our GitHub page: https://github.com/i2vec/A-survey-on-image-text-multimodal-models.
Collapse
Affiliation(s)
- Ruifeng Guo
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Jingxuan Wei
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Linzhuang Sun
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Bihui Yu
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Guiyong Chang
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Dawei Liu
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Sibo Zhang
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Zhengbing Yao
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Mingjun Xu
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Liping Bu
- Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang, 110168, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| |
Collapse
|
2
|
Li P, Hu Y. Deep magnetic resonance fingerprinting based on Local and Global Vision Transformer. Med Image Anal 2024; 95:103198. [PMID: 38759259 DOI: 10.1016/j.media.2024.103198] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 04/28/2024] [Accepted: 05/02/2024] [Indexed: 05/19/2024]
Abstract
To mitigate systematic errors in magnetic resonance fingerprinting (MRF), the precomputed dictionary is usually computed with minimal granularity across the entire range of tissue parameters. However, the dictionary grows exponentially with the number of parameters increase, posing significant challenges to the computational efficiency and matching accuracy of pattern-matching algorithms. Existing works, primarily based on convolutional neural networks (CNN), focus solely on local information to reconstruct multiple parameter maps, lacking in-depth investigations on the MRF mechanism. These methods may not exploit long-distance redundancies and the contextual information within voxel fingerprints introduced by the Bloch equation dynamics, leading to limited reconstruction speed and accuracy. To overcome these limitations, we propose a novel end-to-end neural network called the Local and Global Vision Transformer (LG-ViT) for MRF parameter reconstruction. Our proposed LG-ViT employs a multi-stage architecture that effectively reduces the computational overhead associated with the high-dimensional MRF data and the transformer model. Specifically, a local Transformer encoder is proposed to capture contextual information embedded within voxel fingerprints and local correlations introduced by the interconnected human tissues. Additionally, a global Transformer encoder is proposed to leverage long-distance dependencies arising from shared characteristics among different tissues across various spatial regions. By incorporating MRF physics-based data priors and effectively capturing local and global correlations, our proposed LG-ViT can achieve fast and accurate MRF parameter reconstruction. Experiments on both simulation and in vivo data demonstrate that the proposed method enables faster and more accurate MRF parameter reconstruction compared to state-of-the-art deep learning-based methods.
Collapse
Affiliation(s)
- Peng Li
- The School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, China
| | - Yue Hu
- The School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, China.
| |
Collapse
|
3
|
Klein DS, Karmakar S, Jonnalagadda A, Abbey CK, Eckstein MP. Greater benefits of deep learning-based computer-aided detection systems for finding small signals in 3D volumetric medical images. J Med Imaging (Bellingham) 2024; 11:045501. [PMID: 38988989 PMCID: PMC11232702 DOI: 10.1117/1.jmi.11.4.045501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2024] [Revised: 06/17/2024] [Accepted: 06/20/2024] [Indexed: 07/12/2024] Open
Abstract
Purpose Radiologists are tasked with visually scrutinizing large amounts of data produced by 3D volumetric imaging modalities. Small signals can go unnoticed during the 3D search because they are hard to detect in the visual periphery. Recent advances in machine learning and computer vision have led to effective computer-aided detection (CADe) support systems with the potential to mitigate perceptual errors. Approach Sixteen nonexpert observers searched through digital breast tomosynthesis (DBT) phantoms and single cross-sectional slices of the DBT phantoms. The 3D/2D searches occurred with and without a convolutional neural network (CNN)-based CADe support system. The model provided observers with bounding boxes superimposed on the image stimuli while they looked for a small microcalcification signal and a large mass signal. Eye gaze positions were recorded and correlated with changes in the area under the ROC curve (AUC). Results The CNN-CADe improved the 3D search for the small microcalcification signal ( Δ AUC = 0.098 , p = 0.0002 ) and the 2D search for the large mass signal ( Δ AUC = 0.076 , p = 0.002 ). The CNN-CADe benefit in 3D for the small signal was markedly greater than in 2D ( Δ Δ AUC = 0.066 , p = 0.035 ). Analysis of individual differences suggests that those who explored the least with eye movements benefited the most from the CNN-CADe ( r = - 0.528 , p = 0.036 ). However, for the large signal, the 2D benefit was not significantly greater than the 3D benefit ( Δ Δ AUC = 0.033 , p = 0.133 ). Conclusion The CNN-CADe brings unique performance benefits to the 3D (versus 2D) search of small signals by reducing errors caused by the underexploration of the volumetric data.
Collapse
Affiliation(s)
- Devi S. Klein
- University of California, Department of Psychological and Brain Sciences, Santa Barbara, California, United States
| | - Srijita Karmakar
- University of California, Department of Psychological and Brain Sciences, Santa Barbara, California, United States
| | - Aditya Jonnalagadda
- University of California, Department of Electrical and Computer Engineering, Santa Barbara, California, United States
| | - Craig K. Abbey
- University of California, Department of Psychological and Brain Sciences, Santa Barbara, California, United States
| | - Miguel P. Eckstein
- University of California, Department of Psychological and Brain Sciences, Santa Barbara, California, United States
- University of California, Department of Electrical and Computer Engineering, Santa Barbara, California, United States
- University of California, Department of Computer Science, Santa Barbara, California, United States
| |
Collapse
|
4
|
Vakli P, Weiss B, Rozmann D, Erőss G, Nárai Á, Hermann P, Vidnyánszky Z. The effect of head motion on brain age prediction using deep convolutional neural networks. Neuroimage 2024; 294:120646. [PMID: 38750907 DOI: 10.1016/j.neuroimage.2024.120646] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2024] [Revised: 05/10/2024] [Accepted: 05/12/2024] [Indexed: 05/23/2024] Open
Abstract
Deep learning can be used effectively to predict participants' age from brain magnetic resonance imaging (MRI) data, and a growing body of evidence suggests that the difference between predicted and chronological age-referred to as brain-predicted age difference (brain-PAD)-is related to various neurological and neuropsychiatric disease states. A crucial aspect of the applicability of brain-PAD as a biomarker of individual brain health is whether and how brain-predicted age is affected by MR image artifacts commonly encountered in clinical settings. To investigate this issue, we trained and validated two different 3D convolutional neural network architectures (CNNs) from scratch and tested the models on a separate dataset consisting of motion-free and motion-corrupted T1-weighted MRI scans from the same participants, the quality of which were rated by neuroradiologists from a clinical diagnostic point of view. Our results revealed a systematic increase in brain-PAD with worsening image quality for both models. This effect was also observed for images that were deemed usable from a clinical perspective, with brains appearing older in medium than in good quality images. These findings were also supported by significant associations found between the brain-PAD and standard image quality metrics indicating larger brain-PAD for lower-quality images. Our results demonstrate a spurious effect of advanced brain aging as a result of head motion and underline the importance of controlling for image quality when using brain-predicted age based on structural neuroimaging data as a proxy measure for brain health.
Collapse
Affiliation(s)
- Pál Vakli
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary.
| | - Béla Weiss
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary; Biomatics and Applied Artificial Intelligence Institute, John von Neumann Faculty of Informatics, Óbuda University, Budapest 1034, Hungary.
| | - Dorina Rozmann
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary
| | - György Erőss
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary
| | - Ádám Nárai
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary; Doctoral School of Biology and Sportbiology, Institute of Biology, Faculty of Sciences, University of Pécs, Pécs 7624, Hungary
| | - Petra Hermann
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary
| | - Zoltán Vidnyánszky
- Brain Imaging Centre, HUN-REN Research Centre for Natural Sciences, Budapest 1117, Hungary.
| |
Collapse
|
5
|
Yuan G, Cai L, Qu W, Zhou Z, Liang P, Chen J, Xu C, Zhang J, Wang S, Chu Q, Li Z. Identification of Calculous Pyonephrosis by CT-Based Radiomics and Deep Learning. Bioengineering (Basel) 2024; 11:662. [PMID: 39061744 PMCID: PMC11274102 DOI: 10.3390/bioengineering11070662] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2024] [Revised: 06/18/2024] [Accepted: 06/22/2024] [Indexed: 07/28/2024] Open
Abstract
Urgent detection of calculous pyonephrosis is crucial for surgical planning and preventing severe outcomes. This study aims to evaluate the performance of computed tomography (CT)-based radiomics and a three-dimensional convolutional neural network (3D-CNN) model, integrated with independent clinical factors, to identify patients with calculous pyonephrosis. We recruited 182 patients receiving either percutaneous nephrostomy tube placement or percutaneous nephrolithotomy for calculous hydronephrosis or pyonephrosis. The regions of interest were manually delineated on plain CT images and the CT attenuation value (HU) was measured. Radiomics analysis was performed using least absolute shrinkage and selection operator (LASSO). A 3D-CNN model was also developed. The better-performing machine-learning model was combined with independent clinical factors to build a comprehensive clinical machine-learning model. The performance of these models was assessed using receiver operating characteristic analysis and decision curve analysis. Fever, blood neutrophils, and urine leukocytes were independent risk factors for pyonephrosis. The radiomics model showed higher area under the curve (AUC) than the 3D-CNN model and HU (0.876 vs. 0.599, 0.578; p = 0.003, 0.002) in the testing cohort. The clinical machine-learning model surpassed the clinical model in both the training (0.975 vs. 0.904, p = 0.019) and testing (0.967 vs. 0.889, p = 0.045) cohorts.
Collapse
Affiliation(s)
- Guanjie Yuan
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| | - Lingli Cai
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| | - Weinuo Qu
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| | - Ziling Zhou
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| | - Ping Liang
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| | - Jun Chen
- Bayer Healthcare, Wuhan 430000, China;
| | - Chuou Xu
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| | - Jiaqiao Zhang
- Department of Urology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China;
| | - Shaogang Wang
- Department of Urology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China;
| | - Qian Chu
- Department of Oncology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China;
| | - Zhen Li
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, China; (G.Y.); (L.C.); (W.Q.); (Z.Z.); (P.L.); (Z.L.)
| |
Collapse
|
6
|
Kaczmarek E, Miguel OX, Bowie AC, Ducharme R, Dingwall-Harvey ALJ, Hawken S, Armour CM, Walker MC, Dick K. CAManim: Animating end-to-end network activation maps. PLoS One 2024; 19:e0296985. [PMID: 38889117 PMCID: PMC11185468 DOI: 10.1371/journal.pone.0296985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Accepted: 12/22/2023] [Indexed: 06/20/2024] Open
Abstract
Deep neural networks have been widely adopted in numerous domains due to their high performance and accessibility to developers and application-specific end-users. Fundamental to image-based applications is the development of Convolutional Neural Networks (CNNs), which possess the ability to automatically extract features from data. However, comprehending these complex models and their learned representations, which typically comprise millions of parameters and numerous layers, remains a challenge for both developers and end-users. This challenge arises due to the absence of interpretable and transparent tools to make sense of black-box models. There exists a growing body of Explainable Artificial Intelligence (XAI) literature, including a collection of methods denoted Class Activation Maps (CAMs), that seek to demystify what representations the model learns from the data, how it informs a given prediction, and why it, at times, performs poorly in certain tasks. We propose a novel XAI visualization method denoted CAManim that seeks to simultaneously broaden and focus end-user understanding of CNN predictions by animating the CAM-based network activation maps through all layers, effectively depicting from end-to-end how a model progressively arrives at the final layer activation. Herein, we demonstrate that CAManim works with any CAM-based method and various CNN architectures. Beyond qualitative model assessments, we additionally propose a novel quantitative assessment that expands upon the Remove and Debias (ROAD) metric, pairing the qualitative end-to-end network visual explanations assessment with our novel quantitative "yellow brick ROAD" assessment (ybROAD). This builds upon prior research to address the increasing demand for interpretable, robust, and transparent model assessment methodology, ultimately improving an end-user's trust in a given model's predictions. Examples and source code can be found at: https://omni-ml.github.io/pytorch-grad-cam-anim/.
Collapse
Affiliation(s)
- Emily Kaczmarek
- Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
| | - Olivier X. Miguel
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
| | - Alexa C. Bowie
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
| | - Robin Ducharme
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
| | - Alysha L. J. Dingwall-Harvey
- Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
| | - Steven Hawken
- Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
- ICES, Toronto, Canada
| | - Christine M. Armour
- Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
- Department of Pediatrics, University of Ottawa, Ottawa, Canada
- Prenatal Screening Ontario, Better Outcomes Registry & Network, Ottawa, Canada
| | - Mark C. Walker
- Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
- School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
- ICES, Toronto, Canada
- Department of Obstetrics and Gynecology, University of Ottawa, Ottawa, Canada
- International and Global Health Office, University of Ottawa, Ottawa, Canada
- BORN Ontario, Children’s Hospital of Eastern Ontario, Ottawa, Canada
- Department of Obstetrics, Gynecology & Newborn Care, The Ottawa Hospital, Ottawa, Canada
| | - Kevin Dick
- Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada
- Prenatal Screening Ontario, Better Outcomes Registry & Network, Ottawa, Canada
- BORN Ontario, Children’s Hospital of Eastern Ontario, Ottawa, Canada
| |
Collapse
|
7
|
Bossa MN, Nakshathri AG, Berenguer AD, Sahli H. Generative AI unlocks PET insights: brain amyloid dynamics and quantification. Front Aging Neurosci 2024; 16:1410844. [PMID: 38952479 PMCID: PMC11215072 DOI: 10.3389/fnagi.2024.1410844] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2024] [Accepted: 05/30/2024] [Indexed: 07/03/2024] Open
Abstract
Introduction Studying the spatiotemporal patterns of amyloid accumulation in the brain over time is crucial in understanding Alzheimer's disease (AD). Positron Emission Tomography (PET) imaging plays a pivotal role because it allows for the visualization and quantification of abnormal amyloid beta (Aβ) load in the living brain, providing a powerful tool for tracking disease progression and evaluating the efficacy of anti-amyloid therapies. Generative artificial intelligence (AI) can learn complex data distributions and generate realistic synthetic images. In this study, we demonstrate for the first time the potential of Generative Adversarial Networks (GANs) to build a low-dimensional representation space that effectively describes brain amyloid load and its dynamics. Methods Using a cohort of 1,259 subjects with AV45 PET images from the Alzheimer's Disease Neuroimaging Initiative (ADNI), we develop a 3D GAN model to project images into a latent representation space and generate back synthetic images. Then, we build a progression model on the representation space based on non-parametric ordinary differential equations to study brain amyloid evolution. Results We found that global SUVR can be accurately predicted with a linear regression model only from the latent representation space (RMSE = 0.08 ± 0.01). We generated synthetic PET trajectories and illustrated predicted Aβ change in four years compared with actual progression. Discussion Generative AI can generate rich representations for statistical prediction and progression modeling and simulate evolution in synthetic patients, providing an invaluable tool for understanding AD, assisting in diagnosis, and designing clinical trials. The aim of this study was to illustrate the huge potential that generative AI has in brain amyloid imaging and to encourage its advancement by providing use cases and ideas for future research tracks.
Collapse
Affiliation(s)
- Matías Nicolás Bossa
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium
| | - Akshaya Ganesh Nakshathri
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium
| | - Abel Díaz Berenguer
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium
| | - Hichem Sahli
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium
- Interuniversity Microelectronics Centre (IMEC), Leuven, Belgium
| |
Collapse
|
8
|
Wang C, Guo L, Zhu J, Zhu L, Li C, Zhu H, Song A, Lu L, Teng GJ, Navab N, Jiang Z. Review of robotic systems for thoracoabdominal puncture interventional surgery. APL Bioeng 2024; 8:021501. [PMID: 38572313 PMCID: PMC10987197 DOI: 10.1063/5.0180494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Accepted: 03/11/2024] [Indexed: 04/05/2024] Open
Abstract
Cancer, with high morbidity and high mortality, is one of the major burdens threatening human health globally. Intervention procedures via percutaneous puncture have been widely used by physicians due to its minimally invasive surgical approach. However, traditional manual puncture intervention depends on personal experience and faces challenges in terms of precisely puncture, learning-curve, safety and efficacy. The development of puncture interventional surgery robotic (PISR) systems could alleviate the aforementioned problems to a certain extent. This paper attempts to review the current status and prospective of PISR systems for thoracic and abdominal application. In this review, the key technologies related to the robotics, including spatial registration, positioning navigation, puncture guidance feedback, respiratory motion compensation, and motion control, are discussed in detail.
Collapse
Affiliation(s)
- Cheng Wang
- Hanglok-Tech Co. Ltd., Hengqin 519000, People's Republic of China
| | - Li Guo
- Hanglok-Tech Co. Ltd., Hengqin 519000, People's Republic of China
| | | | - Lifeng Zhu
- State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| | - Chichi Li
- School of Computer Science and Engineering, Macau University of Science and Technology, Macau, 999078, People's Republic of China
| | - Haidong Zhu
- Center of Interventional Radiology and Vascular Surgery, Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing 210009, People's Republic of China
| | - Aiguo Song
- State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| | | | - Gao-Jun Teng
- Center of Interventional Radiology and Vascular Surgery, Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing 210009, People's Republic of China
| | | | - Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich 80333, Germany
| |
Collapse
|
9
|
Tong B, Chen H, Wang C, Zeng W, Li D, Liu P, Liu M, Jin X, Shang S. Clinical prediction models for knee pain in patients with knee osteoarthritis: a systematic review. Skeletal Radiol 2024; 53:1045-1059. [PMID: 38265451 DOI: 10.1007/s00256-024-04590-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/14/2023] [Revised: 12/15/2023] [Accepted: 12/15/2023] [Indexed: 01/25/2024]
Abstract
OBJECTIVE To identify and describe existing models for predicting knee pain in patients with knee osteoarthritis. METHODS The electronic databases PubMed, EMBASE, CINAHL, Web of Science, and Cochrane Library were searched from their inception to May 2023 for any studies to develop and validate a prediction model for predicting knee pain in patients with knee osteoarthritis. Two reviewers independently screened titles, abstracts, and full-text qualifications, and extracted data. Risk of bias was assessed using the PROBAST. Data extraction of eligible articles was extracted by a data extraction form based on CHARMS. The quality of evidence was graded according to GRADE. The results were summarized with descriptive statistics. RESULTS The search identified 2693 records. Sixteen articles reporting on 26 prediction models were included targeting occurrence (n = 9), others (n = 7), progression (n = 5), persistent (n = 2), incident (n = 1), frequent (n = 1), and flares (n = 1) of knee pain. Most of the studies (94%) were at high risk of bias. Model discrimination was assessed by the AUROC ranging from 0.62 to 0.81. The most common predictors were age, BMI, gender, baseline pain, and joint space width. Only frequent knee pain had a moderate quality of evidence; all other types of knee pain had a low quality of evidence. CONCLUSION There are many prediction models for knee pain in patients with knee osteoarthritis that do show promise. However, the clinical extensibility, applicability, and interpretability of predictive tools should be considered during model development.
Collapse
Affiliation(s)
- Beibei Tong
- School of Nursing, Peking University, Beijing, China
| | - Hongbo Chen
- Nursing Department of Peking University Third Hospital, Beijing, China
| | - Cui Wang
- School of Nursing, Peking University, Beijing, China
| | - Wen Zeng
- School of Nursing, Peking University, Beijing, China
| | - Dan Li
- School of Nursing, Peking University, Beijing, China
| | - Peiyuan Liu
- School of Nursing, Peking University, Beijing, China
| | - Ming Liu
- Macao Polytechnic University, Macao, China
| | | | - Shaomei Shang
- School of Nursing, Peking University, Beijing, China.
| |
Collapse
|
10
|
Zhang X, Chen S, Zhang P, Wang C, Wang Q, Zhou X. Staging of Liver Fibrosis Based on Energy Valley Optimization Multiple Stacking (EVO-MS) Model. Bioengineering (Basel) 2024; 11:485. [PMID: 38790352 PMCID: PMC11117710 DOI: 10.3390/bioengineering11050485] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2024] [Revised: 05/09/2024] [Accepted: 05/10/2024] [Indexed: 05/26/2024] Open
Abstract
Currently, staging the degree of liver fibrosis predominantly relies on liver biopsy, a method fraught with potential risks, such as bleeding and infection. With the rapid development of medical imaging devices, quantification of liver fibrosis through image processing technology has become feasible. Stacking technology is one of the effective ensemble techniques for potential usage, but precise tuning to find the optimal configuration manually is challenging. Therefore, this paper proposes a novel EVO-MS model-a multiple stacking ensemble learning model optimized by the energy valley optimization (EVO) algorithm to select most informatic features for fibrosis quantification. Liver contours are profiled from 415 biopsied proven CT cases, from which 10 shape features are calculated and inputted into a Support Vector Machine (SVM) classifier to generate the accurate predictions, then the EVO algorithm is applied to find the optimal parameter combination to fuse six base models: K-Nearest Neighbors (KNNs), Decision Tree (DT), Naive Bayes (NB), Extreme Gradient Boosting (XGB), Gradient Boosting Decision Tree (GBDT), and Random Forest (RF), to create a well-performing ensemble model. Experimental results indicate that selecting 3-5 feature parameters yields satisfactory results in classification, with features such as the contour roundness non-uniformity (Rmax), maximum peak height of contour (Rp), and maximum valley depth of contour (Rm) significantly influencing classification accuracy. The improved EVO algorithm, combined with a multiple stacking model, achieves an accuracy of 0.864, a precision of 0.813, a sensitivity of 0.912, a specificity of 0.824, and an F1-score of 0.860, which demonstrates the effectiveness of our EVO-MS model in staging the degree of liver fibrosis.
Collapse
Affiliation(s)
- Xuejun Zhang
- School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China; (X.Z.); (P.Z.); (C.W.)
- Guangxi Key Laboratory of Multimedia Communications and Network Technology, Guangxi University, Nanning 530004, China
| | - Shengxiang Chen
- School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China; (X.Z.); (P.Z.); (C.W.)
| | - Pengfei Zhang
- School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China; (X.Z.); (P.Z.); (C.W.)
| | - Chun Wang
- School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China; (X.Z.); (P.Z.); (C.W.)
| | - Qibo Wang
- School of Computer, Electronics and Information, Guangxi University, Nanning 530004, China; (X.Z.); (P.Z.); (C.W.)
| | - Xiangrong Zhou
- Department of Electrical, Electronic and Computer Engineering, Gifu University, Gifu 501-1193, Japan;
| |
Collapse
|
11
|
Song AH, Williams M, Williamson DFK, Chow SSL, Jaume G, Gao G, Zhang A, Chen B, Baras AS, Serafin R, Colling R, Downes MR, Farré X, Humphrey P, Verrill C, True LD, Parwani AV, Liu JTC, Mahmood F. Analysis of 3D pathology samples using weakly supervised AI. Cell 2024; 187:2502-2520.e17. [PMID: 38729110 PMCID: PMC11168832 DOI: 10.1016/j.cell.2024.03.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Revised: 01/15/2024] [Accepted: 03/25/2024] [Indexed: 05/12/2024]
Abstract
Human tissue, which is inherently three-dimensional (3D), is traditionally examined through standard-of-care histopathology as limited two-dimensional (2D) cross-sections that can insufficiently represent the tissue due to sampling bias. To holistically characterize histomorphology, 3D imaging modalities have been developed, but clinical translation is hampered by complex manual evaluation and lack of computational platforms to distill clinical insights from large, high-resolution datasets. We present TriPath, a deep-learning platform for processing tissue volumes and efficiently predicting clinical outcomes based on 3D morphological features. Recurrence risk-stratification models were trained on prostate cancer specimens imaged with open-top light-sheet microscopy or microcomputed tomography. By comprehensively capturing 3D morphologies, 3D volume-based prognostication achieves superior performance to traditional 2D slice-based approaches, including clinical/histopathological baselines from six certified genitourinary pathologists. Incorporating greater tissue volume improves prognostic performance and mitigates risk prediction variability from sampling bias, further emphasizing the value of capturing larger extents of heterogeneous morphology.
Collapse
Affiliation(s)
- Andrew H Song
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Mane Williams
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA; Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Sarah S L Chow
- Department of Mechanical Engineering, Bioengineering, and Laboratory Medicine & Pathology, University of Washington, Seattle, WA, USA
| | - Guillaume Jaume
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Gan Gao
- Department of Mechanical Engineering, Bioengineering, and Laboratory Medicine & Pathology, University of Washington, Seattle, WA, USA
| | - Andrew Zhang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA; Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Bowen Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Alexander S Baras
- Department of Pathology, Johns Hopkins University School of Medicine, Baltimore, MD, USA; Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Robert Serafin
- Department of Mechanical Engineering, Bioengineering, and Laboratory Medicine & Pathology, University of Washington, Seattle, WA, USA
| | - Richard Colling
- Nuffield Department of Surgical Sciences, University of Oxford, UK; Department of Cellular Pathology, Oxford University Hospitals NHS Foundations Trust, John Radcliffe Hospital, Oxford, UK
| | - Michelle R Downes
- Sunnybrook Health Sciences Centre, University of Toronto, Toronto, ON, Canada
| | - Xavier Farré
- Public Health Agency of Catalonia, Lleida, Spain
| | - Peter Humphrey
- Department of Pathology, Yale School of Medicine, New Haven, CT, USA
| | - Clare Verrill
- Nuffield Department of Surgical Sciences, University of Oxford, UK; Department of Cellular Pathology, Oxford University Hospitals NHS Foundations Trust, John Radcliffe Hospital, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Lawrence D True
- Department of Laboratory Medicine & Pathology, University of Washington School of Medicine, Seattle, WA, USA
| | - Anil V Parwani
- Department of Pathology, The Ohio State University, Columbus, OH, USA
| | - Jonathan T C Liu
- Department of Mechanical Engineering, Bioengineering, and Laboratory Medicine & Pathology, University of Washington, Seattle, WA, USA.
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA; Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA; Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA.
| |
Collapse
|
12
|
Mandal S, Jammal AA, Malek D, Medeiros FA. Progression or Aging? A Deep Learning Approach for Distinguishing Glaucoma Progression From Age-Related Changes in OCT Scans. Am J Ophthalmol 2024; 266:46-55. [PMID: 38703802 DOI: 10.1016/j.ajo.2024.04.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 04/16/2024] [Accepted: 04/29/2024] [Indexed: 05/06/2024]
Abstract
PURPOSE To develop deep learning (DL) algorithm to detect glaucoma progression using optical coherence tomography (OCT) images, in the absence of a reference standard. DESIGN Retrospective cohort study. METHODS Glaucomatous and healthy eyes with ≥5 reliable peripapillary OCT (Spectralis, Heidelberg Engineering) circle scans were included. A weakly supervised time-series learning model, called noise positive-unlabeled (Noise-PU) DL was developed to classify whether sequences of OCT B-scans showed glaucoma progression. The model used 2 learning schemes, one to identify age-related changes by differentiating test sequences from glaucoma vs healthy eyes, and the other to identify test-retest variability based on scrambled OCTs of glaucoma eyes. Both models' bases were convolutional neural networks (CNN) and long short-term memory (LSTM) networks which were combined to form a CNN-LSTM model. Model features were combined and jointly trained to identify glaucoma progression, accounting for age-related loss. The DL model's outcomes were compared with ordinary least squares (OLS) regression of retinal nerve fiber layer (RNFL) thickness over time, matched for specificity. The hit ratio was used as a proxy for sensitivity. RESULTS Eight thousand seven hundred eighty-five follow-up sequences of 5 consecutive OCT tests from 3253 eyes (1859 subjects) were included in the study. The mean follow-up time was 3.5 ± 1.6 years. In the test sample, the hit ratios of the DL and OLS methods were 0.498 (95%CI: 0.470-0.526) and 0.284 (95%CI: 0.258-0.309) respectively (P < .001) when the specificities were equalized to 95%. CONCLUSION A DL model was able to identify longitudinal glaucomatous structural changes in OCT B-scans using a surrogate reference standard for progression.
Collapse
Affiliation(s)
- Sayan Mandal
- From the Department of Electrical and Computer Engineering, Pratt School of Engineering (S.M., F.A.M.), Duke University, Durham, North Carolina, USA
| | - Alessandro A Jammal
- Duke Eye Center and Department of Ophthalmology (A.A.J., F.A.M.), Duke University, Durham, North Carolina, USA; Bascom Palmer Eye Institute (A.A.J., D.M., F.A.M.), University of Miami, Miami, Florida, USA
| | - Davina Malek
- Bascom Palmer Eye Institute (A.A.J., D.M., F.A.M.), University of Miami, Miami, Florida, USA
| | - Felipe A Medeiros
- From the Department of Electrical and Computer Engineering, Pratt School of Engineering (S.M., F.A.M.), Duke University, Durham, North Carolina, USA; Duke Eye Center and Department of Ophthalmology (A.A.J., F.A.M.), Duke University, Durham, North Carolina, USA; Bascom Palmer Eye Institute (A.A.J., D.M., F.A.M.), University of Miami, Miami, Florida, USA.
| |
Collapse
|
13
|
Liu D, Hu W, Li X. Point Cloud Attacks in Graph Spectral Domain: When 3D Geometry Meets Graph Signal Processing. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:3079-3095. [PMID: 38051619 DOI: 10.1109/tpami.2023.3339130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
With the increasing attention in various 3D safety-critical applications, point cloud learning models have been shown to be vulnerable to adversarial attacks. Although existing 3D attack methods achieve high success rates, they delve into the data space with point-wise perturbation, which may neglect the geometric characteristics. Instead, we propose point cloud attacks from a new perspective-the graph spectral domain attack, aiming to perturb graph transform coefficients in the spectral domain that correspond to varying certain geometric structures. Specifically, leveraging on graph signal processing, we first adaptively transform the coordinates of points onto the spectral domain via graph Fourier transform (GFT) for compact representation. Then, we analyze the influence of different spectral bands on the geometric structure, based on which we propose to perturb the GFT coefficients via a learnable graph spectral filter. Considering the low-frequency components mainly contribute to the rough shape of the 3D object, we further introduce a low-frequency constraint to limit perturbations within imperceptible high-frequency components. Finally, the adversarial point cloud is generated by transforming the perturbed spectral representation back to the data domain via the inverse GFT. Experimental results demonstrate the effectiveness of the proposed attack in terms of both the imperceptibility and attack success rates.
Collapse
|
14
|
Zhou D, Yu C, Liu W, Liu F. Registration of multimodal bone images based on edge similarity metaheuristic. Comput Biol Med 2024; 174:108379. [PMID: 38631115 DOI: 10.1016/j.compbiomed.2024.108379] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2023] [Revised: 03/09/2024] [Accepted: 03/24/2024] [Indexed: 04/19/2024]
Abstract
OBJECTIVE Blurry medical images affect the accuracy and efficiency of multimodal image registration, whose existing methods require further improvement. METHODS We propose an edge-based similarity registration method optimised for multimodal medical images, especially bone images, by a balance optimiser. First, we use a GPU (graphics processing unit) rendering simulation to convert computed tomography data into digitally reconstructed radiographs. Second, we introduce the improved cascaded edge network (ICENet), a convolutional neural network that extracts edge information of blurry medical images. Then, the bilateral Gaussian-weighted similarity of pairs of X-ray images and digitally reconstructed radiographs is measured. The a balanced optimiser is iteratively applied to finally estimate the best pose to perform image registration. RESULTS Experimental results show that, on average, the proposed method with ICENet outperforms other edge detection networks by 20%, 12%, 18.83%, and 11.93% in the overall Dice similarity, overall intersection over union, peak signal-to-noise ratio, and structural similarity index, respectively, with a registration success rate up to 90% and average reduction of 220% in registration time. CONCLUSION The proposed method with ICENet can achieve a high registration success rate even for blurry medical images, and its efficiency and robustness are higher than those of existing methods. SIGNIFICANCE Our proposal may be suitable for supporting medical diagnosis, radiation therapy, image-guided surgery, and other clinical applications.
Collapse
Affiliation(s)
- Dibin Zhou
- School of Information Science and Technology, Hangzhou Normal University, Zhejiang, China.
| | - Chen Yu
- School of Information Science and Technology, Hangzhou Normal University, Zhejiang, China.
| | - Wenhao Liu
- School of Information Science and Technology, Hangzhou Normal University, Zhejiang, China.
| | - Fuchang Liu
- School of Information Science and Technology, Hangzhou Normal University, Zhejiang, China.
| |
Collapse
|
15
|
Saberironaghi A, Ren J. DepthCrackNet: A Deep Learning Model for Automatic Pavement Crack Detection. J Imaging 2024; 10:100. [PMID: 38786555 PMCID: PMC11122326 DOI: 10.3390/jimaging10050100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Revised: 04/18/2024] [Accepted: 04/21/2024] [Indexed: 05/25/2024] Open
Abstract
Detecting cracks in the pavement is a vital component of ensuring road safety. Since manual identification of these cracks can be time-consuming, an automated method is needed to speed up this process. However, creating such a system is challenging due to factors including crack variability, variations in pavement materials, and the occurrence of miscellaneous objects and anomalies on the pavement. Motivated by the latest progress in deep learning applied to computer vision, we propose an effective U-Net-shaped model named DepthCrackNet. Our model employs the Double Convolution Encoder (DCE), composed of a sequence of convolution layers, for robust feature extraction while keeping parameters optimally efficient. We have incorporated the TriInput Multi-Head Spatial Attention (TMSA) module into our model; in this module, each head operates independently, capturing various spatial relationships and boosting the extraction of rich contextual information. Furthermore, DepthCrackNet employs the Spatial Depth Enhancer (SDE) module, specifically designed to augment the feature extraction capabilities of our segmentation model. The performance of the DepthCrackNet was evaluated on two public crack datasets: Crack500 and DeepCrack. In our experimental studies, the network achieved mIoU scores of 77.0% and 83.9% with the Crack500 and DeepCrack datasets, respectively.
Collapse
Affiliation(s)
- Alireza Saberironaghi
- Electrical, Computer and Software Engineering, Ontario Tech University, Oshawa, ON L1G 0C5, Canada
| | - Jing Ren
- Electrical, Computer and Software Engineering, Ontario Tech University, Oshawa, ON L1G 0C5, Canada
| |
Collapse
|
16
|
Zheng Q, Gao Y, Zhou M, Li H, Lin J, Zhang W, Chen X. Semi or fully automatic tooth segmentation in CBCT images: a review. PeerJ Comput Sci 2024; 10:e1994. [PMID: 38660190 PMCID: PMC11041986 DOI: 10.7717/peerj-cs.1994] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 03/27/2024] [Indexed: 04/26/2024]
Abstract
Cone beam computed tomography (CBCT) is widely employed in modern dentistry, and tooth segmentation constitutes an integral part of the digital workflow based on these imaging data. Previous methodologies rely heavily on manual segmentation and are time-consuming and labor-intensive in clinical practice. Recently, with advancements in computer vision technology, scholars have conducted in-depth research, proposing various fast and accurate tooth segmentation methods. In this review, we review 55 articles in this field and discuss the effectiveness, advantages, and disadvantages of each approach. In addition to simple classification and discussion, this review aims to reveal how tooth segmentation methods can be improved by the application and refinement of existing image segmentation algorithms to solve problems such as irregular morphology and fuzzy boundaries of teeth. It is assumed that with the optimization of these methods, manual operation will be reduced, and greater accuracy and robustness in tooth segmentation will be achieved. Finally, we highlight the challenges that still exist in this field and provide prospects for future directions.
Collapse
Affiliation(s)
- Qianhan Zheng
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Yu Gao
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Mengqi Zhou
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Huimin Li
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Jiaqi Lin
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
| | - Weifang Zhang
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
- Social Medicine & Health Affairs Administration, Zhejiang University, Hangzhou, China
| | - Xuepeng Chen
- Stomatology Hospital, Zhejiang University School of Medicine, Hangzhou, China
- Clinical Research Center for Oral Diseases of Zhejiang Province, Cancer Center of Zhejiang University, Hangzhou, China
| |
Collapse
|
17
|
Szugye NA, Mahalingam N, Somasundaram E, Villa C, Segala J, Segala M, Zafar F, Morales DLS, Moore RA. Deep Learning for Automated Measurement of Total Cardiac Volume for Heart Transplantation Size Matching. Pediatr Cardiol 2024:10.1007/s00246-024-03470-4. [PMID: 38570368 DOI: 10.1007/s00246-024-03470-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/21/2023] [Accepted: 03/07/2024] [Indexed: 04/05/2024]
Abstract
Total Cardiac Volume (TCV)-based size matching using Computed Tomography (CT) is a novel technique to compare donor and recipient heart size in pediatric heart transplant that may increase overall utilization of available grafts. TCV requires manual segmentation, which limits its widespread use due to time and specialized software and training needed for segmentation. This study aims to determine the accuracy of a Deep Learning (DL) approach using 3-dimensional Convolutional Neural Networks (3D-CNN) to calculate TCV, with the clinical aim of enabling fast and accurate TCV use at all transplant centers. Ground truth TCV was segmented on CT scans of subjects aged 0-30 years, identified retrospectively. Ground truth segmentation masks were used to train and test a custom 3D-CNN model consisting of a DenseNet architecture in combination with residual blocks of ResNet architecture. The model was trained on a cohort of 270 subjects and a validation cohort of 44 subjects (36 normal, 8 heart disease retained for model testing). The average Dice similarity coefficient of the validation cohort was 0.94 ± 0.03 (range 0.84-0.97). The mean absolute percent error of TCV estimation was 5.5%. There is no significant association between model accuracy and subject age, weight, or height. DL-TCV was on average more accurate for normal hearts than those listed for transplant (mean absolute percent error 4.5 ± 3.9 vs. 10.5 ± 8.5, p = 0.08). A deep learning-based 3D-CNN model can provide accurate automatic measurement of TCV from CT images. This initial study is limited as a single-center study, though future multicenter studies may enable generalizable and more accurate TCV measurement by inclusion of more diverse cardiac pathology and increasing the training data.
Collapse
Affiliation(s)
- Nicholas A Szugye
- Cleveland Clinic Foundation, Pediatric Cardiology, Cleveland, OH, USA.
| | - Neeraja Mahalingam
- Cincinnati Children's Hospital Medical Center, 3333 Burnet Avenue, Cincinnati, OH, 45229, USA
| | | | - Chet Villa
- Cincinnati Children's Hospital Medical Center, 3333 Burnet Avenue, Cincinnati, OH, 45229, USA
| | | | | | - Farhan Zafar
- Cincinnati Children's Hospital Medical Center, 3333 Burnet Avenue, Cincinnati, OH, 45229, USA
| | - David L S Morales
- Cincinnati Children's Hospital Medical Center, 3333 Burnet Avenue, Cincinnati, OH, 45229, USA
| | - Ryan A Moore
- Cincinnati Children's Hospital Medical Center, 3333 Burnet Avenue, Cincinnati, OH, 45229, USA
| |
Collapse
|
18
|
Elgarba BM, Fontenele RC, Tarce M, Jacobs R. Artificial intelligence serving pre-surgical digital implant planning: A scoping review. J Dent 2024; 143:104862. [PMID: 38336018 DOI: 10.1016/j.jdent.2024.104862] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 01/22/2024] [Accepted: 01/24/2024] [Indexed: 02/12/2024] Open
Abstract
OBJECTIVES To conduct a scoping review focusing on artificial intelligence (AI) applications in presurgical dental implant planning. Additionally, to assess the automation degree of clinically available pre-surgical implant planning software. DATA AND SOURCES A systematic electronic literature search was performed in five databases (PubMed, Embase, Web of Science, Cochrane Library, and Scopus), along with exploring gray literature web-based resources until November 2023. English-language studies on AI-driven tools for digital implant planning were included based on an independent evaluation by two reviewers. An assessment of automation steps in dental implant planning software available on the market up to November 2023 was also performed. STUDY SELECTION AND RESULTS From an initial 1,732 studies, 47 met eligibility criteria. Within this subset, 39 studies focused on AI networks for anatomical landmark-based segmentation, creating virtual patients. Eight studies were dedicated to AI networks for virtual implant placement. Additionally, a total of 12 commonly available implant planning software applications were identified and assessed for their level of automation in pre-surgical digital implant workflows. Notably, only six of these featured at least one fully automated step in the planning software, with none possessing a fully automated implant planning protocol. CONCLUSIONS AI plays a crucial role in achieving accurate, time-efficient, and consistent segmentation of anatomical landmarks, serving the process of virtual patient creation. Additionally, currently available systems for virtual implant placement demonstrate different degrees of automation. It is important to highlight that, as of now, full automation of this process has not been documented nor scientifically validated. CLINICAL SIGNIFICANCE Scientific and clinical validation of AI applications for presurgical dental implant planning is currently scarce. The present review allows the clinician to identify AI-based automation in presurgical dental implant planning and assess the potential underlying scientific validation.
Collapse
Affiliation(s)
- Bahaaeldeen M Elgarba
- OMFS IMPATH Research Group, Department of Imaging and Pathology, Faculty of Medicine, KU Leuven & Department of Oral and Maxillofacial Surgery, University Hospitals, Campus Sint-Rafael, 3000 Leuven, Belgium & Department of Prosthodontics, Faculty of Dentistry, Tanta University, 31511 Tanta, Egypt.
| | - Rocharles Cavalcante Fontenele
- OMFS IMPATH Research Group, Department of Imaging and Pathology, Faculty of Medicine, KU Leuven & Department of Oral and Maxillofacial Surgery, University Hospitals, Campus Sint-Rafael, 3000 Leuven, Belgium
| | - Mihai Tarce
- Division of Periodontology & Implant Dentistry, Faculty of Dentistry, The University of Hong Kong, Hong Kong SAR, China & Periodontology and Oral Microbiology, Department of Oral Health Sciences, Faculty of Medicine, KU Leuven, Leuven, Belgium
| | - Reinhilde Jacobs
- OMFS IMPATH Research Group, Department of Imaging and Pathology, Faculty of Medicine, KU Leuven & Department of Oral and Maxillofacial Surgery, University Hospitals, Campus Sint-Rafael, 3000 Leuven, Belgium & Department of Dental Medicine, Karolinska Institute, Stockholm, Sweden
| |
Collapse
|
19
|
Granata V, Fusco R, Coluccino S, Russo C, Grassi F, Tortora F, Conforti R, Caranci F. Preliminary data on artificial intelligence tool in magnetic resonance imaging assessment of degenerative pathologies of lumbar spine. LA RADIOLOGIA MEDICA 2024; 129:623-630. [PMID: 38349415 DOI: 10.1007/s11547-024-01791-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/02/2023] [Accepted: 01/15/2024] [Indexed: 04/17/2024]
Abstract
PURPOSE To evaluate the ability of an artificial intelligence (AI) tool in magnetic resonance imaging (MRI) assessment of degenerative pathologies of lumbar spine using radiologist evaluation as a gold standard. METHODS Patients with degenerative pathologies of lumbar spine, evaluated with MRI study, were enrolled in a retrospective study approved by local ethical committee. A comprehensive software solution (CoLumbo; SmartSoft Ltd., Varna, Bulgaria) designed to label the segments of the lumbar spine and to detect a broad spectrum of degenerative pathologies based on a convolutional neural network (CNN) was employed, utilizing an automatic segmentation. The AI tool efficacy was compared to data obtained by a senior neuroradiologist that employed a semiquantitative score. Chi-square test was used to assess the differences among groups, and Spearman's rank correlation coefficient was calculated between the grading assigned by radiologist and the grading obtained by software. Moreover, agreement was assessed between the value assigned by radiologist and software. RESULTS Ninety patients (58 men; 32 women) affected with degenerative pathologies of lumbar spine and aged from 60 to 81 years (mean 66 years) were analyzed. Significant correlations were observed between grading assigned by radiologist and the grading obtained by software for each localization. However, only when the localization was L2-L3, there was a good correlation with a coefficient value of 0.72. The best agreements were obtained in case of L1-L2 and L2-L3 localizations and were, respectively, of 81.1% and 72.2%. The lowest agreement of 51.1% was detected in case of L4-L5 locations. With regard canal stenosis and compression, the highest agreement was obtained for identification of in L5-S1 localization. CONCLUSIONS AI solution represents an efficacy and useful toll degenerative pathologies of lumbar spine to improve radiologist workflow.
Collapse
Affiliation(s)
- Vincenza Granata
- Division of Radiology, "Istituto Nazionale Tumori IRCCS Fondazione Pascale - IRCCS di Napoli", Naples, Italy
| | | | - Simone Coluccino
- Division of Radiology, "Università degli Studi della Campania Luigi Vanvitelli", Naples, Italy
| | - Carmela Russo
- Unit of Neuroradiology, Department of Neurosciences, Santobono-Pausilipon Children's Hospital, Naples, Italy
| | - Francesca Grassi
- Division of Radiology, "Università degli Studi della Campania Luigi Vanvitelli", Naples, Italy
| | - Fabio Tortora
- Neuroradiology Unit, Department of Advanced Biomedical Sciences, University "Federico II", Naples, Italy
| | - Renata Conforti
- Division of Radiology, "Università degli Studi della Campania Luigi Vanvitelli", Naples, Italy
| | - Ferdinando Caranci
- Division of Radiology, "Università degli Studi della Campania Luigi Vanvitelli", Naples, Italy
- Unit of Neuroradiology, Department of Neurosciences, Santobono-Pausilipon Children's Hospital, Naples, Italy
- Neuroradiology Unit, Department of Advanced Biomedical Sciences, University "Federico II", Naples, Italy
- Italian Society of Medical and Interventional Radiology (SIRM), SIRM Foundation, Via della Signora 2, 20122, Milan, Italy
| |
Collapse
|
20
|
Morita K, Karashima S, Terao T, Yoshida K, Yamashita T, Yoroidaka T, Tanabe M, Imi T, Zaimoku Y, Yoshida A, Maruyama H, Iwaki N, Aoki G, Kotani T, Murata R, Miyamoto T, Machida Y, Matsue K, Nambo H, Takamatsu H. 3D CNN-based Deep Learning Model-based Explanatory Prognostication in Patients with Multiple Myeloma using Whole-body MRI. J Med Syst 2024; 48:30. [PMID: 38456950 DOI: 10.1007/s10916-024-02040-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Accepted: 02/05/2024] [Indexed: 03/09/2024]
Abstract
Although magnetic resonance imaging (MRI) data of patients with multiple myeloma (MM) are used to predict prognosis, few reports have applied artificial intelligence (AI) techniques for this purpose. We aimed to analyze whole-body diffusion-weighted MRI data using three-dimensional (3D) convolutional neural networks (CNNs) and Gradient-weighted Class Activation Mapping (Grad-CAM), an explainable AI, to predict prognosis and explore the factors involved in prediction. We retrospectively analyzed the MRI data of a total of 142 patients with MM obtained from two medical centers. We defined the occurrence of progressive disease after MRI evaluation within 12 months as a poor prognosis and constructed a 3D CNN-based deep learning model to predict prognosis. Images from 111 cases were used as the training and internal validation data; images from 31 cases were used as the external validation data. Internal validation of the AI model with stratified 5-fold cross-validation resulted in a significant difference in progression-free survival (PFS) between good and poor prognostic cases (2-year PFS, 91.2% versus [vs.] 61.1%, P = 0.0002). The AI model clearly stratified good and poor prognostic cases in the external validation cohort (2-year PFS, 92.9% vs. 55.6%, P = 0.004), with an area under the receiver operating characteristic curve of 0.804. According to Grad-CAM, the MRI signals of the spleen and bones of the vertebrae and pelvis contributed to prognosis prediction. This study is the first to show that image analysis of whole-body MRI using a 3D CNN without any other clinical data is effective in predicting the prognosis of patients with MM.
Collapse
Affiliation(s)
- Kento Morita
- School of Electrical, Information and Communication Engineering, Kanazawa University, Kakumamachi, Kanazawa, Ishikawa, 920-1192, Japan
| | | | - Toshiki Terao
- Department of Hematology/Oncology, Kameda Medical Center, Kamogawa, Japan
- Department of Hematology and Oncology, Okayama University Hospital, Okayama, Japan
| | - Kotaro Yoshida
- Department of Radiology, Kanazawa University, Kanazawa, Japan
| | - Takeshi Yamashita
- Division of Internal Medicine, Keiju Kanazawa Hospital, Kanazawa, Japan
| | - Takeshi Yoroidaka
- Department of Hematology, Ishikawa Central Prefectural Hospital, Kanazawa, Japan
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Mikoto Tanabe
- Department of Hematology, Ishikawa Central Prefectural Hospital, Kanazawa, Japan
| | - Tatsuya Imi
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Yoshitaka Zaimoku
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Akiyo Yoshida
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Hiroyuki Maruyama
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Noriko Iwaki
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Go Aoki
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Takeharu Kotani
- Department of Hematology, Ishikawa Central Prefectural Hospital, Kanazawa, Japan
| | - Ryoichi Murata
- Division of Internal Medicine, Keiju Kanazawa Hospital, Kanazawa, Japan
| | - Toshihiro Miyamoto
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan
| | - Youichi Machida
- Department of Radiology, Kameda Medical Center, Kamogawa, Japan
| | - Kosei Matsue
- Department of Hematology/Oncology, Kameda Medical Center, Kamogawa, Japan
| | - Hidetaka Nambo
- Faculty of Transdisciplinary Sciences for Innovation, Institute of Transdisciplinary Sciences for Innovation, Kanazawa University, Kakumamachi, Kanazawa, Ishikawa, 920-1192, Japan.
| | - Hiroyuki Takamatsu
- Department of Hematology, Faculty of Medicine, Institute of Medical, Pharmaceutical, and Health Sciences, Kanazawa University, Kanazawa, Japan.
- Faculty of Transdisciplinary Sciences for Innovation, Institute of Transdisciplinary Sciences for Innovation, Kanazawa University, Kakumamachi, Kanazawa, Ishikawa, 920-1192, Japan.
| |
Collapse
|
21
|
Lee JO, Ahn SS, Choi KS, Lee J, Jang J, Park JH, Hwang I, Park CK, Park SH, Chung JW, Choi SH. Added prognostic value of 3D deep learning-derived features from preoperative MRI for adult-type diffuse gliomas. Neuro Oncol 2024; 26:571-580. [PMID: 37855826 PMCID: PMC10912011 DOI: 10.1093/neuonc/noad202] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Indexed: 10/20/2023] Open
Abstract
BACKGROUND To investigate the prognostic value of spatial features from whole-brain MRI using a three-dimensional (3D) convolutional neural network for adult-type diffuse gliomas. METHODS In a retrospective, multicenter study, 1925 diffuse glioma patients were enrolled from 5 datasets: SNUH (n = 708), UPenn (n = 425), UCSF (n = 500), TCGA (n = 160), and Severance (n = 132). The SNUH and Severance datasets served as external test sets. Precontrast and postcontrast 3D T1-weighted, T2-weighted, and T2-FLAIR images were processed as multichannel 3D images. A 3D-adapted SE-ResNeXt model was trained to predict overall survival. The prognostic value of the deep learning-based prognostic index (DPI), a spatial feature-derived quantitative score, and established prognostic markers were evaluated using Cox regression. Model evaluation was performed using the concordance index (C-index) and Brier score. RESULTS The MRI-only median DPI survival prediction model achieved C-indices of 0.709 and 0.677 (BS = 0.142 and 0.215) and survival differences (P < 0.001 and P = 0.002; log-rank test) for the SNUH and Severance datasets, respectively. Multivariate Cox analysis revealed DPI as a significant prognostic factor, independent of clinical and molecular genetic variables: hazard ratio = 0.032 and 0.036 (P < 0.001 and P = 0.004) for the SNUH and Severance datasets, respectively. Multimodal prediction models achieved higher C-indices than models using only clinical and molecular genetic variables: 0.783 vs. 0.774, P = 0.001, SNUH; 0.766 vs. 0.748, P = 0.023, Severance. CONCLUSIONS The global morphologic feature derived from 3D CNN models using whole-brain MRI has independent prognostic value for diffuse gliomas. Combining clinical, molecular genetic, and imaging data yields the best performance.
Collapse
Affiliation(s)
- Jung Oh Lee
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Artificial Intelligence Collaborative Network, Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Sung Soo Ahn
- Department of Radiology, Yonsei University College of Medicine, Seoul, Republic of Korea
| | - Kyu Sung Choi
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Artificial Intelligence Collaborative Network, Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Junhyeok Lee
- Interdisciplinary Programs in Cancer Biology Major, Seoul National University Graduate School, Seoul, Republic of Korea
| | - Joon Jang
- Department of Biomedical Sciences, Seoul National University, Seoul, Republic of Korea
| | - Jung Hyun Park
- Department of Radiology, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, Seoul, South Korea
| | - Inpyeong Hwang
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Artificial Intelligence Collaborative Network, Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Chul-Kee Park
- Department of Neurosurgery, Seoul National University Hospital, Seoul, Republic of Korea
| | - Sung Hye Park
- Department of Pathology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Jin Wook Chung
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Artificial Intelligence Collaborative Network, Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Institute of Innovate Biomedical Technology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Seung Hong Choi
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Artificial Intelligence Collaborative Network, Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
- Center for Nanoparticle Research, Institute for Basic Science, Seoul, Republic of Korea
| |
Collapse
|
22
|
Maramraju S, Kowalczewski A, Kaza A, Liu X, Singaraju JP, Albert MV, Ma Z, Yang H. AI-organoid integrated systems for biomedical studies and applications. Bioeng Transl Med 2024; 9:e10641. [PMID: 38435826 PMCID: PMC10905559 DOI: 10.1002/btm2.10641] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 12/07/2023] [Accepted: 12/13/2023] [Indexed: 03/05/2024] Open
Abstract
In this review, we explore the growing role of artificial intelligence (AI) in advancing the biomedical applications of human pluripotent stem cell (hPSC)-derived organoids. Stem cell-derived organoids, these miniature organ replicas, have become essential tools for disease modeling, drug discovery, and regenerative medicine. However, analyzing the vast and intricate datasets generated from these organoids can be inefficient and error-prone. AI techniques offer a promising solution to efficiently extract insights and make predictions from diverse data types generated from microscopy images, transcriptomics, metabolomics, and proteomics. This review offers a brief overview of organoid characterization and fundamental concepts in AI while focusing on a comprehensive exploration of AI applications in organoid-based disease modeling and drug evaluation. It provides insights into the future possibilities of AI in enhancing the quality control of organoid fabrication, label-free organoid recognition, and three-dimensional image reconstruction of complex organoid structures. This review presents the challenges and potential solutions in AI-organoid integration, focusing on the establishment of reliable AI model decision-making processes and the standardization of organoid research.
Collapse
Affiliation(s)
- Sudhiksha Maramraju
- Department of Biomedical EngineeringUniversity of North TexasDentonTexasUSA
- Texas Academy of Mathematics and ScienceUniversity of North TexasDentonTexasUSA
| | - Andrew Kowalczewski
- Department of Biomedical & Chemical EngineeringSyracuse UniversitySyracuseNew YorkUSA
- BioInspired Institute for Material and Living SystemsSyracuse UniversitySyracuseNew YorkUSA
| | - Anirudh Kaza
- Department of Biomedical EngineeringUniversity of North TexasDentonTexasUSA
- Texas Academy of Mathematics and ScienceUniversity of North TexasDentonTexasUSA
| | - Xiyuan Liu
- Department of Mechanical & Aerospace EngineeringSyracuse UniversitySyracuseNew YorkUSA
| | - Jathin Pranav Singaraju
- Department of Biomedical EngineeringUniversity of North TexasDentonTexasUSA
- Texas Academy of Mathematics and ScienceUniversity of North TexasDentonTexasUSA
| | - Mark V. Albert
- Department of Biomedical EngineeringUniversity of North TexasDentonTexasUSA
- Department of Computer Science and EngineeringUniversity of North TexasDentonTexasUSA
| | - Zhen Ma
- Department of Biomedical & Chemical EngineeringSyracuse UniversitySyracuseNew YorkUSA
- BioInspired Institute for Material and Living SystemsSyracuse UniversitySyracuseNew YorkUSA
| | - Huaxiao Yang
- Department of Biomedical EngineeringUniversity of North TexasDentonTexasUSA
| |
Collapse
|
23
|
Alkojak Almansi A, Sugarova S, Alsanosi A, Almuhawas F, Hofmeyr L, Wagner F, Kedves E, Sriperumbudur K, Dhanasingh A, Kedves A. A novel radiological software prototype for automatically detecting the inner ear and classifying normal from malformed anatomy. Comput Biol Med 2024; 171:108168. [PMID: 38432006 DOI: 10.1016/j.compbiomed.2024.108168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2023] [Revised: 01/29/2024] [Accepted: 02/15/2024] [Indexed: 03/05/2024]
Abstract
BACKGROUND To develop an effective radiological software prototype that could read Digital Imaging and Communications in Medicine (DICOM) files, crop the inner ear automatically based on head computed tomography (CT), and classify normal and inner ear malformation (IEM). METHODS A retrospective analysis was conducted on 2053 patients from 3 hospitals. We extracted 1200 inner ear CTs for importing, cropping, and training, testing, and validating an artificial intelligence (AI) model. Automated cropping algorithms based on CTs were developed to precisely isolate the inner ear volume. Additionally, a simple graphical user interface (GUI) was implemented for user interaction. Using cropped CTs as input, a deep learning convolutional neural network (DL CNN) with 5-fold cross-validation was used to classify inner ear anatomy as normal or abnormal. Five specific IEM types (cochlear hypoplasia, ossification, incomplete partition types I and III, and common cavity) were included, with data equally distributed between classes. Both the cropping tool and the AI model were extensively validated. RESULTS The newly developed DICOM viewer/software successfully achieved its objectives: reading CT files, automatically cropping inner ear volumes, and classifying them as normal or malformed. The cropping tool demonstrated an average accuracy of 92.25%. The DL CNN model achieved an area under the curve (AUC) of 0.86 (95% confidence interval: 0.81-0.91). Performance metrics for the AI model were: accuracy (0.812), precision (0.791), recall (0.8), and F1-score (0.766). CONCLUSION This study successfully developed and validated a fully automated workflow for classifying normal versus abnormal inner ear anatomy using a combination of advanced image processing and deep learning techniques. The tool exhibited good diagnostic accuracy, suggesting its potential application in risk stratification. However, it is crucial to emphasize the need for supervision by qualified medical professionals when utilizing this tool for clinical decision-making.
Collapse
Affiliation(s)
- Abdulrahman Alkojak Almansi
- University of Pecs, Faculty of Engineering and Information Technology, Institute of Information and Electrical Technology, Pecs, Hungary
| | - Sima Sugarova
- St. Petersburg ENT and Speech Research Institute, St. Petersburg, Russia
| | - Abdulrahman Alsanosi
- King Saud University, King Abdullah Ear Specialist Center (KAESC), Department of Otolaryngology, Riyadh, Saudi Arabia
| | - Fida Almuhawas
- King Saud University, King Abdullah Ear Specialist Center (KAESC), Department of Otolaryngology, Riyadh, Saudi Arabia
| | - Louis Hofmeyr
- Dr Loius Hofmeyr's workplace to Stellenbosch University Division of Otorhinolaryngology, Stellenbosch, South Africa
| | - Franca Wagner
- University Hospital Bern, University Institute for Diagnostic and Interventional Neuroradiology, Switzerland
| | - Emerencia Kedves
- University of Sopron, Doctoral School of Wood Sciences and Technologies, Sopron, Hungary
| | - Kiran Sriperumbudur
- MED-EL Medical Electronics GmbH., Department of Research and Development, Innsbruck, Austria
| | - Anandhan Dhanasingh
- MED-EL Medical Electronics GmbH., Department of Research and Development, Innsbruck, Austria.
| | - Andras Kedves
- MED-EL Medical Electronics GmbH., Department of Research and Development, Innsbruck, Austria; University of Pecs, Faculty of Engineering and Information Technology, Institute of Information and Electrical Technology, Pecs, Hungary.
| |
Collapse
|
24
|
Sheng J, Zhang Q, Zhang Q, Wang L, Yang Z, Xin Y, Wang B. A hybrid multimodal machine learning model for Detecting Alzheimer's disease. Comput Biol Med 2024; 170:108035. [PMID: 38325214 DOI: 10.1016/j.compbiomed.2024.108035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 01/03/2024] [Accepted: 01/26/2024] [Indexed: 02/09/2024]
Abstract
Alzheimer's disease (AD) diagnosis utilizing single modality neuroimaging data has limitations. Multimodal fusion of complementary biomarkers may improve diagnostic performance. This study proposes a multimodal machine learning framework integrating magnetic resonance imaging (MRI), positron emission tomography (PET) and cerebrospinal fluid (CSF) assays for enhanced AD characterization. The model incorporates a hybrid algorithm combining enhanced Harris Hawks Optimization (HHO) algorithm referred to as ILHHO, with Kernel Extreme Learning Machine (KELM) classifier for simultaneous feature selection and classification. ILHHO enhances HHO's search efficiency by integrating iterative mapping (IM) to improve population diversity and local escaping operator (LEO) to balance exploration-exploitation. Comparative analysis with other improved HHO algorithms, classic meta-heuristic algorithms (MHAs), and state-of-the-art MHAs on IEEE CEC2014 benchmark functions indicates that ILHHO achieves superior optimization performance compared to other comparative algorithms. The synergistic ILHHO-KELM model is evaluated on 202 AD Neuroimaging Initiative (ADNI) subjects. Results demonstrate superior multimodal classification accuracy over single modalities, validating the importance of fusing heterogeneous biomarkers. MRI + PET + CSF achieves 99.2 % accuracy for AD vs. normal control (NC), outperforming conventional and proposed methods. Discriminative feature analysis provides further insights into differential AD-related neurodegeneration patterns detected by MRI and PET. The differential PET and MRI features demonstrate how the two modalities provide complementary biomarkers. The neuroanatomical relevance of selected features supports ILHHO-KELM's potential for extracting sensitive AD imaging signatures. Overall, the study showcases the advantages of capitalizing on complementary multimodal data through advanced feature learning techniques for improving AD diagnosis.
Collapse
Affiliation(s)
- Jinhua Sheng
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, 310018, China; Key Laboratory of Intelligent Image Analysis for Sensory and Cognitive Health, Ministry of Industry and Information Technology of China, Hangzhou, Zhejiang, 310018, China.
| | - Qian Zhang
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, 310018, China; Key Laboratory of Intelligent Image Analysis for Sensory and Cognitive Health, Ministry of Industry and Information Technology of China, Hangzhou, Zhejiang, 310018, China; School of Data Science and Artificial Intelligence, Wenzhou University of Technology, Wenzhou, Zhejiang, 325035, China
| | - Qiao Zhang
- Beijing Hospital, Beijing, 100730, China; National Center of Gerontology, Beijing, 100730, China; Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing, 100730, China
| | - Luyun Wang
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, 310018, China; Key Laboratory of Intelligent Image Analysis for Sensory and Cognitive Health, Ministry of Industry and Information Technology of China, Hangzhou, Zhejiang, 310018, China
| | - Ze Yang
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, 310018, China; Key Laboratory of Intelligent Image Analysis for Sensory and Cognitive Health, Ministry of Industry and Information Technology of China, Hangzhou, Zhejiang, 310018, China
| | - Yu Xin
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, 310018, China; Key Laboratory of Intelligent Image Analysis for Sensory and Cognitive Health, Ministry of Industry and Information Technology of China, Hangzhou, Zhejiang, 310018, China
| | - Binbing Wang
- School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, Zhejiang, 310018, China; Key Laboratory of Intelligent Image Analysis for Sensory and Cognitive Health, Ministry of Industry and Information Technology of China, Hangzhou, Zhejiang, 310018, China
| |
Collapse
|
25
|
Shahzadi I, Seidlitz A, Beuthien-Baumann B, Zwanenburg A, Platzek I, Kotzerke J, Baumann M, Krause M, Troost EGC, Löck S. Radiomics for residual tumour detection and prognosis in newly diagnosed glioblastoma based on postoperative [ 11C] methionine PET and T1c-w MRI. Sci Rep 2024; 14:4576. [PMID: 38403632 PMCID: PMC10894870 DOI: 10.1038/s41598-024-55092-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 02/20/2024] [Indexed: 02/27/2024] Open
Abstract
Personalized treatment strategies based on non-invasive biomarkers have potential to improve patient management in patients with newly diagnosed glioblastoma (GBM). The residual tumour burden after surgery in GBM patients is a prognostic imaging biomarker. However, in clinical patient management, its assessment is a manual and time-consuming process that is at risk of inter-rater variability. Furthermore, the prediction of patient outcome prior to radiotherapy may identify patient subgroups that could benefit from escalated radiotherapy doses. Therefore, in this study, we investigate the capabilities of traditional radiomics and 3D convolutional neural networks for automatic detection of the residual tumour status and to prognosticate time-to-recurrence (TTR) and overall survival (OS) in GBM using postoperative [11C] methionine positron emission tomography (MET-PET) and gadolinium-enhanced T1-w magnetic resonance imaging (MRI). On the independent test data, the 3D-DenseNet model based on MET-PET achieved the best performance for residual tumour detection, while the logistic regression model with conventional radiomics features performed best for T1c-w MRI (AUC: MET-PET 0.95, T1c-w MRI 0.78). For the prognosis of TTR and OS, the 3D-DenseNet model based on MET-PET integrated with age and MGMT status achieved the best performance (Concordance-Index: TTR 0.68, OS 0.65). In conclusion, we showed that both deep-learning and conventional radiomics have potential value for supporting image-based assessment and prognosis in GBM. After prospective validation, these models may be considered for treatment personalization.
Collapse
Affiliation(s)
- Iram Shahzadi
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- German Cancer Consortium (DKTK) Partner Site Dresden, Germany, and German Cancer Research Center (DKFZ), Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
| | - Annekatrin Seidlitz
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- German Cancer Consortium (DKTK) Partner Site Dresden, Germany, and German Cancer Research Center (DKFZ), Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Bettina Beuthien-Baumann
- Department of Nuclear Medicine, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Department of Radiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alex Zwanenburg
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- German Cancer Consortium (DKTK) Partner Site Dresden, Germany, and German Cancer Research Center (DKFZ), Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
| | - Ivan Platzek
- Institute of Radiology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Jörg Kotzerke
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Nuclear Medicine, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Michael Baumann
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- Division of Radiooncology/Radiobiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | - Mechthild Krause
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- German Cancer Consortium (DKTK) Partner Site Dresden, Germany, and German Cancer Research Center (DKFZ), Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden-Rossendorf, Institute of Radiooncology, Dresden, Germany
| | - Esther G C Troost
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany
- German Cancer Consortium (DKTK) Partner Site Dresden, Germany, and German Cancer Research Center (DKFZ), Heidelberg, Germany
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany
- Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
- Helmholtz-Zentrum Dresden-Rossendorf, Institute of Radiooncology, Dresden, Germany
| | - Steffen Löck
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany.
- German Cancer Consortium (DKTK) Partner Site Dresden, Germany, and German Cancer Research Center (DKFZ), Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany.
- Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
26
|
Molinski NS, Kenda M, Leithner C, Nee J, Storm C, Scheel M, Meddeb A. Deep learning-enabled detection of hypoxic-ischemic encephalopathy after cardiac arrest in CT scans: a comparative study of 2D and 3D approaches. Front Neurosci 2024; 18:1245791. [PMID: 38419661 PMCID: PMC10899383 DOI: 10.3389/fnins.2024.1245791] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Accepted: 01/31/2024] [Indexed: 03/02/2024] Open
Abstract
Objective To establish a deep learning model for the detection of hypoxic-ischemic encephalopathy (HIE) features on CT scans and to compare various networks to determine the best input data format. Methods 168 head CT scans of patients after cardiac arrest were retrospectively identified and classified into two categories: 88 (52.4%) with radiological evidence of severe HIE and 80 (47.6%) without signs of HIE. These images were randomly divided into a training and a test set, and five deep learning models based on based on Densely Connected Convolutional Networks (DenseNet121) were trained and validated using different image input formats (2D and 3D images). Results All optimized stacked 2D and 3D networks could detect signs of HIE. The networks based on the data as 2D image data stacks provided the best results (S100: AUC: 94%, ACC: 79%, S50: AUC: 93%, ACC: 79%). We provide visual explainability data for the decision making of our AI model using Gradient-weighted Class Activation Mapping. Conclusion Our proof-of-concept deep learning model can accurately identify signs of HIE on CT images. Comparing different 2D- and 3D-based approaches, most promising results were achieved by 2D image stack models. After further clinical validation, a deep learning model of HIE detection based on CT images could be implemented in clinical routine and thus aid clinicians in characterizing imaging data and predicting outcome.
Collapse
Affiliation(s)
- Noah S. Molinski
- Department for Neuroradiology, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Martin Kenda
- Department of Neurology with Experimental Neurology, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
- Berlin Institute of Health at Charité – Universitätsmedizin Berlin, BIH Biomedical Innovation Academy, Berlin, Germany
| | - Christoph Leithner
- Department of Neurology with Experimental Neurology, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Jens Nee
- Department of Nephrology and Medical Intensive Care, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Christian Storm
- Department of Nephrology and Medical Intensive Care, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Michael Scheel
- Department for Neuroradiology, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Aymen Meddeb
- Department for Neuroradiology, Charité – Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
- Berlin Institute of Health at Charité – Universitätsmedizin Berlin, BIH Biomedical Innovation Academy, Berlin, Germany
| |
Collapse
|
27
|
Zheng S, Lu L, Yingsa H, Meichen S. Deep learning framework for three-dimensional surface reconstruction of object of interest in photoacoustic tomography. OPTICS EXPRESS 2024; 32:6037-6061. [PMID: 38439316 DOI: 10.1364/oe.507476] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Accepted: 01/23/2024] [Indexed: 03/06/2024]
Abstract
Photoacoustic tomography (PAT) is a non-ionizing hybrid imaging technology of clinical importance that combines the high contrast of optical imaging with the high penetration of ultrasonic imaging. Two-dimensional (2D) tomographic images can only provide the cross-sectional structure of the imaging target rather than its overall spatial morphology. This work proposes a deep learning framework for reconstructing three-dimensional (3D) surface of an object of interest from a series of 2D images. It achieves end-to-end mapping from a series of 2D images to a 3D image, visually displaying the overall morphology of the object. The framework consists of four modules: segmentation module, point cloud generation module, point cloud completion module, and mesh conversion module, which respectively implement the tasks of segmenting a region of interest, generating a sparse point cloud, completing sparse point cloud and reconstructing 3D surface. The network model is trained on simulation data sets and verified on simulation, phantom, and in vivo data sets. The results showed superior 3D reconstruction performance both visually and on the basis of quantitative evaluation metrics compared to the state-of-the-art non-learning and learning approaches. This method potentially enables high-precision 3D surface reconstruction from the tomographic images output by the preclinical PAT system without changing the imaging system. It provides a general deep learning scheme for 3D reconstruction from tomographic scanning data.
Collapse
|
28
|
Khabaz K, Yuan K, Pugar J, Jiang D, Sankary S, Dhara S, Kim J, Kang J, Nguyen N, Cao K, Washburn N, Bohr N, Lee CJ, Kindlmann G, Milner R, Pocivavsek L. The geometric evolution of aortic dissections: Predicting surgical success using fluctuations in integrated Gaussian curvature. PLoS Comput Biol 2024; 20:e1011815. [PMID: 38306397 PMCID: PMC10866512 DOI: 10.1371/journal.pcbi.1011815] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2023] [Revised: 02/14/2024] [Accepted: 01/09/2024] [Indexed: 02/04/2024] Open
Abstract
Clinical imaging modalities are a mainstay of modern disease management, but the full utilization of imaging-based data remains elusive. Aortic disease is defined by anatomic scalars quantifying aortic size, even though aortic disease progression initiates complex shape changes. We present an imaging-based geometric descriptor, inspired by fundamental ideas from topology and soft-matter physics that captures dynamic shape evolution. The aorta is reduced to a two-dimensional mathematical surface in space whose geometry is fully characterized by the local principal curvatures. Disease causes deviation from the smooth bent cylindrical shape of normal aortas, leading to a family of highly heterogeneous surfaces of varying shapes and sizes. To deconvolute changes in shape from size, the shape is characterized using integrated Gaussian curvature or total curvature. The fluctuation in total curvature (δK) across aortic surfaces captures heterogeneous morphologic evolution by characterizing local shape changes. We discover that aortic morphology evolves with a power-law defined behavior with rapidly increasing δK forming the hallmark of aortic disease. Divergent δK is seen for highly diseased aortas indicative of impending topologic catastrophe or aortic rupture. We also show that aortic size (surface area or enclosed aortic volume) scales as a generalized cylinder for all shapes. Classification accuracy for predicting aortic disease state (normal, diseased with successful surgery, and diseased with failed surgical outcomes) is 92.8±1.7%. The analysis of δK can be applied on any three-dimensional geometric structure and thus may be extended to other clinical problems of characterizing disease through captured anatomic changes.
Collapse
Affiliation(s)
- Kameel Khabaz
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Karen Yuan
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Joseph Pugar
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
- Departments of Material Science and Engineering, Biomedical Engineering, and Chemistry, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America
| | - David Jiang
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Seth Sankary
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Sanjeev Dhara
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Junsung Kim
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Janet Kang
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Nhung Nguyen
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Kathleen Cao
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Newell Washburn
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America
| | - Nicole Bohr
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Cheong Jun Lee
- Department of Surgery, NorthShore University Health System, Evanston, Illinois, United States of America
| | - Gordon Kindlmann
- Department of Computer Science, The University of Chicago, Chicago, Illinois, United States of America
| | - Ross Milner
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| | - Luka Pocivavsek
- Department of Surgery, The University of Chicago, Chicago, Illinois, United States of America
| |
Collapse
|
29
|
Wang W, Liang H, Zhang Z, Xu C, Wei D, Li W, Qian Y, Zhang L, Liu J, Lei D. Comparing three-dimensional and two-dimensional deep-learning, radiomics, and fusion models for predicting occult lymph node metastasis in laryngeal squamous cell carcinoma based on CT imaging: a multicentre, retrospective, diagnostic study. EClinicalMedicine 2024; 67:102385. [PMID: 38261897 PMCID: PMC10796944 DOI: 10.1016/j.eclinm.2023.102385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Revised: 11/29/2023] [Accepted: 12/04/2023] [Indexed: 01/25/2024] Open
Abstract
Background The occult lymph node metastasis (LNM) of laryngeal squamous cell carcinoma (LSCC) affects the treatment and prognosis of patients. This study aimed to comprehensively compare the performance of the three-dimensional and two-dimensional deep learning models, radiomics model, and the fusion models for predicting occult LNM in LSCC. Methods In this retrospective diagnostic study, a total of 553 patients with clinical N0 stage LSCC, who underwent surgical treatment without distant metastasis and multiple primary cancers, were consecutively enrolled from four Chinese medical centres between January 01, 2016 and December 30, 2020. The participant data were manually retrieved from medical records, imaging databases, and pathology reports. The study cohort was divided into a training set (n = 300), an internal test set (n = 89), and two external test sets (n = 120 and 44, respectively). The three-dimensional deep learning (3D DL), two-dimensional deep learning (2D DL), and radiomics model were developed using CT images of the primary tumor. The clinical model was constructed based on clinical and radiological features. Two fusion strategies were utilized to develop the fusion model: the feature-based DLRad_FB model and the decision-based DLRad_DB model. The discriminative ability and correlation of 3D DL, 2D DL and radiomics features were analysed comprehensively. The performances of the predictive models were evaluated based on the pathological diagnosis. Findings The 3D DL features had superior discriminative ability and lower internal redundancy compared to 2D DL and radiomics features. The DLRad_DB model achieved the highest AUC (0.89-0.90) among all the study sets, significantly outperforming the clinical model (AUC = 0.73-0.78, P = 0.0001-0.042, Delong test). Compared to the DLRad_DB model, the AUC values for the DLRad_FB, 3D DL, 2D DL, and radiomics models were 0.82-0.84 (P = 0.025-0.46), 0.86-0.89 (P = 0.75-0.97), 0.83-0.86 (P = 0.029-0.66), and 0.79-0.82 (P = 0.0072-0.10), respectively in the study sets. Additionally, the DLRad_DB model exhibited the best sensitivity (82-88%) and specificity (79-85%) in the test sets. Interpretation The decision-based fusion model DLRad_DB, which combines 3D DL, 2D DL, radiomics, and clinical data, can be utilized to predict occult LNM in LSCC. This has the potential to minimize unnecessary lymph node dissection and prophylactic radiotherapy in patients with cN0 disease. Funding National Natural Science Foundation of China, Natural Science Foundation of Shandong Province.
Collapse
Affiliation(s)
- Wenlun Wang
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| | - Hui Liang
- Department of Otorhinolaryngology, The First Affiliated Hospital of Shandong First Medical University & Shandong Provincial Qianfoshan Hospital, Ji’nan 250014, Shandong, China
| | - Zhouyi Zhang
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| | - Chenyang Xu
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| | - Dongmin Wei
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| | - Wenming Li
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| | - Ye Qian
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| | - Lihong Zhang
- Department of Otorhinolaryngology Head & Neck Surgery, Peking University People’s Hospital, Beijing 100044, China
| | - Jun Liu
- Department of Otolaryngology-Head & Neck Surgery, West China Hospital, Sichuan University, Chengdu, China
| | - Dapeng Lei
- Department of Otorhinolaryngology, Qilu Hospital of Shandong University, Jinan, Shandong, China
- NHC Key Laboratory of Otorhinolaryngology (Shandong University), Jinan, Shandong, China
| |
Collapse
|
30
|
Hu M, Yang K, Wang J, Qiu RLJ, Roper J, Kahn S, Shu HK, Yang X. MGMT promoter methylation prediction based on multiparametric MRI via vision graph neural network. J Med Imaging (Bellingham) 2024; 11:014503. [PMID: 38370421 PMCID: PMC10869845 DOI: 10.1117/1.jmi.11.1.014503] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 12/24/2023] [Accepted: 01/29/2024] [Indexed: 02/20/2024] Open
Abstract
Purpose Glioblastoma (GBM) is aggressive and malignant. The methylation status of the O 6 -methylguanine-DNA methyltransferase (MGMT) promoter in GBM tissue is considered an important biomarker for developing the most effective treatment plan. Although the standard method for assessing the MGMT promoter methylation status is via bisulfite modification and deoxyribonucleic acid (DNA) sequencing of biopsy or surgical specimens, a secondary automated method based on medical imaging may improve the efficiency and accuracy of those tests. Approach We propose a deep vision graph neural network (ViG) using multiparametric magnetic resonance imaging (MRI) to predict the MGMT promoter methylation status noninvasively. Our model was compared to the RSNA radiogenomic classification winners. The dataset includes 583 usable patient cases. Combinations of MRI sequences were compared. Our multi-sequence fusion strategy was compared with those using single MR sequences. Results Our best model [Fluid Attenuated Inversion Recovery (FLAIR), T1-weighted pre-contrast (T1w), T2-weighted (T2)] outperformed the winning models with a test area under the curve (AUC) of 0.628, an accuracy of 0.632, a precision of 0.646, a recall of 0.677, a specificity of 0.581, and an F1 score of 0.661. Compared to the winning models with single MR sequences, our ViG utilizing fused-MRI showed a significant improvement statistically in AUC scores, which are FLAIR (p = 0.042 ), T1w (p = 0.017 ), T1wCE (p = 0.001 ), and T2 (p = 0.018 ). Conclusions Our model is superior to challenge champions. A graph representation of the medical images enabled good handling of complexity and irregularity. Our work provides an automatic secondary check pipeline to ensure the correctness of MGMT methylation status prediction.
Collapse
Affiliation(s)
- Mingzhe Hu
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
- Emory University, Department of Computer Science and Informatics, Atlanta, Georgia, United States
| | - Kailin Yang
- Cleveland Clinic, Taussig Cancer Center, Department of Radiation Oncology, Cleveland, Ohio, United States
| | - Jing Wang
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
| | - Richard L. J. Qiu
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
| | - Justin Roper
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
| | - Shannon Kahn
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
| | - Hui-Kuo Shu
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
| | - Xiaofeng Yang
- Emory University, Department of Radiation Oncology and Winship Cancer Institute, Atlanta, Georgia, United States
- Emory University, Department of Computer Science and Informatics, Atlanta, Georgia, United States
- Georgia Institute of Technology and Emory University, Department of Biomedical Engineering, Atlanta, Georgia, United States
| |
Collapse
|
31
|
Joo Y, Namgung E, Jeong H, Kang I, Kim J, Oh S, Lyoo IK, Yoon S, Hwang J. Brain age prediction using combined deep convolutional neural network and multi-layer perceptron algorithms. Sci Rep 2023; 13:22388. [PMID: 38104173 PMCID: PMC10725434 DOI: 10.1038/s41598-023-49514-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Accepted: 12/08/2023] [Indexed: 12/19/2023] Open
Abstract
The clinical applications of brain age prediction have expanded, particularly in anticipating the onset and prognosis of various neurodegenerative diseases. In the current study, we proposed a deep learning algorithm that leverages brain structural imaging data and enhances prediction accuracy by integrating biological sex information. Our model for brain age prediction, built on deep neural networks, employed a dataset of 3004 healthy subjects aged 18 and above. The T1-weighted images were minimally preprocessed and analyzed using the convolutional neural network (CNN) algorithm. The categorical sex information was then incorporated using the multi-layer perceptron (MLP) algorithm. We trained and validated both a CNN-only algorithm (utilizing only brain structural imaging data), and a combined CNN-MLP algorithm (using both structural brain imaging data and sex information) for age prediction. By integrating sex information with T1-weighted imaging data, our proposed CNN-MLP algorithm outperformed not only the CNN-only algorithm but also established algorithms, such as brainageR, in prediction accuracy. Notably, this hybrid CNN-MLP algorithm effectively distinguished between mild cognitive impairment and Alzheimer's disease groups by identifying variances in brain age gaps between them, highlighting the algorithm's potential for clinical application. Overall, these results underscore the enhanced precision of the CNN-MLP algorithm in brain age prediction, achieved through the integration of sex information.
Collapse
Affiliation(s)
- Yoonji Joo
- Ewha Brain Institute, Ewha Womans University, Seoul, South Korea
| | - Eun Namgung
- Asan Institute for Life Sciences, Asan Medical Center, Seoul, South Korea
| | - Hyeonseok Jeong
- Department of Radiology, Incheon St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, South Korea
| | - Ilhyang Kang
- Ewha Brain Institute, Ewha Womans University, Seoul, South Korea
| | - Jinsol Kim
- Ewha Brain Institute, Ewha Womans University, Seoul, South Korea
| | - Sohyun Oh
- Ewha Brain Institute, Ewha Womans University, Seoul, South Korea
- Department of Brain and Cognitive Sciences, Ewha Womans University, Seoul, South Korea
| | - In Kyoon Lyoo
- Ewha Brain Institute, Ewha Womans University, Seoul, South Korea
- Department of Brain and Cognitive Sciences, Ewha Womans University, Seoul, South Korea
- Graduate School of Pharmaceutical Sciences, Ewha Womans University, Seoul, South Korea
| | - Sujung Yoon
- Ewha Brain Institute, Ewha Womans University, Seoul, South Korea.
- Department of Brain and Cognitive Sciences, Ewha Womans University, Seoul, South Korea.
| | - Jaeuk Hwang
- Department of Psychiatry, Soonchunhyang University College of Medicine, Seoul, South Korea.
| |
Collapse
|
32
|
Itkyal VS, Abrol A, LaGrow TJ, Fedorov A, Calhoun VD. Voxel-wise Fusion of Resting fMRI Networks and Gray Matter Volume for Alzheimer's Disease Classification using Deep Multimodal Learning. RESEARCH SQUARE 2023:rs.3.rs-3740218. [PMID: 38168287 PMCID: PMC10760243 DOI: 10.21203/rs.3.rs-3740218/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/05/2024]
Abstract
Alzheimer's disease (AD) is a prevalent neurodegenerative disorder requiring accurate and early diagnosis for effective treatment. Resting-state functional magnetic resonance imaging (rs-fMRI) and gray matter volume analysis from structural MRI have emerged as valuable tools for investigating AD-related brain alterations. However, the potential benefits of integrating these modalities using deep learning techniques remain unexplored. In this study, we propose a novel framework that fuses composite images of multiple rs-fMRI networks (called voxelwise intensity projection) and gray matter segmentation images through a deep learning approach for improved AD classification. We demonstrate the superiority of fMRI networks over commonly used metrics such as amplitude of low-frequency fluctuations (ALFF) and fractional ALFF in capturing spatial maps critical for AD classification. We use a multi-channel convolutional neural network incorporating the AlexNet dropout architecture to effectively model spatial and temporal dependencies in the integrated data. Extensive experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset of AD patients and cognitively normal (CN) validate the efficacy of our approach, showcasing improved classification performance of 94.12% test accuracy and an area under the curve (AUC) score of 97.79 compared to existing methods. Our results show that the fusion results generally outperformed the unimodal results. The saliency visualizations also show significant differences in the hippocampus, amygdala, putamen, caudate nucleus, and regions of basal ganglia which are in line with the previous neurobiological literature. Our research offers a novel method to enhance our grasp of AD pathology. By integrating data from various functional networks with structural MRI insights, we significantly improve diagnostic accuracy. This accuracy is further boosted by the effective visualization of this combined information. This lays the groundwork for further studies focused on providing a more accurate and personalized approach to AD diagnosis. The proposed framework and insights gained from fMRI networks provide a promising avenue for future research in deep multimodal fusion and neuroimaging analysis.
Collapse
|
33
|
Rahman H, Khan AR, Sadiq T, Farooqi AH, Khan IU, Lim WH. A Systematic Literature Review of 3D Deep Learning Techniques in Computed Tomography Reconstruction. Tomography 2023; 9:2158-2189. [PMID: 38133073 PMCID: PMC10748093 DOI: 10.3390/tomography9060169] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 11/27/2023] [Accepted: 12/01/2023] [Indexed: 12/23/2023] Open
Abstract
Computed tomography (CT) is used in a wide range of medical imaging diagnoses. However, the reconstruction of CT images from raw projection data is inherently complex and is subject to artifacts and noise, which compromises image quality and accuracy. In order to address these challenges, deep learning developments have the potential to improve the reconstruction of computed tomography images. In this regard, our research aim is to determine the techniques that are used for 3D deep learning in CT reconstruction and to identify the training and validation datasets that are accessible. This research was performed on five databases. After a careful assessment of each record based on the objective and scope of the study, we selected 60 research articles for this review. This systematic literature review revealed that convolutional neural networks (CNNs), 3D convolutional neural networks (3D CNNs), and deep learning reconstruction (DLR) were the most suitable deep learning algorithms for CT reconstruction. Additionally, two major datasets appropriate for training and developing deep learning systems were identified: 2016 NIH-AAPM-Mayo and MSCT. These datasets are important resources for the creation and assessment of CT reconstruction models. According to the results, 3D deep learning may increase the effectiveness of CT image reconstruction, boost image quality, and lower radiation exposure. By using these deep learning approaches, CT image reconstruction may be made more precise and effective, improving patient outcomes, diagnostic accuracy, and healthcare system productivity.
Collapse
Affiliation(s)
- Hameedur Rahman
- Department of Computer Games Development, Faculty of Computing & AI, Air University, E9, Islamabad 44000, Pakistan;
| | - Abdur Rehman Khan
- Department of Creative Technologies, Faculty of Computing & AI, Air University, E9, Islamabad 44000, Pakistan;
| | - Touseef Sadiq
- Centre for Artificial Intelligence Research, Department of Information and Communication Technology, University of Agder, Jon Lilletuns vei 9, 4879 Grimstad, Norway
| | - Ashfaq Hussain Farooqi
- Department of Computer Science, Faculty of Computing AI, Air University, Islamabad 44000, Pakistan;
| | - Inam Ullah Khan
- Department of Electronic Engineering, School of Engineering & Applied Sciences (SEAS), Isra University, Islamabad Campus, Islamabad 44000, Pakistan;
| | - Wei Hong Lim
- Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia;
| |
Collapse
|
34
|
Maghami M, Sattari SA, Tahmasbi M, Panahi P, Mozafari J, Shirbandi K. Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study. Biomed Eng Online 2023; 22:114. [PMID: 38049809 PMCID: PMC10694901 DOI: 10.1186/s12938-023-01172-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Accepted: 11/17/2023] [Indexed: 12/06/2023] Open
Abstract
BACKGROUND This systematic review and meta-analysis were conducted to objectively evaluate the evidence of machine learning (ML) in the patient diagnosis of Intracranial Hemorrhage (ICH) on computed tomography (CT) scans. METHODS Until May 2023, systematic searches were conducted in ISI Web of Science, PubMed, Scopus, Cochrane Library, IEEE Xplore Digital Library, CINAHL, Science Direct, PROSPERO, and EMBASE for studies that evaluated the diagnostic precision of ML model-assisted ICH detection. Patients with and without ICH as the target condition who were receiving CT-Scan were eligible for the research, which used ML algorithms based on radiologists' reports as the gold reference standard. For meta-analysis, pooled sensitivities, specificities, and a summary receiver operating characteristics curve (SROC) were used. RESULTS At last, after screening the title, abstract, and full paper, twenty-six retrospective and three prospective, and two retrospective/prospective studies were included. The overall (Diagnostic Test Accuracy) DTA of retrospective studies with a pooled sensitivity was 0.917 (95% CI 0.88-0.943, I2 = 99%). The pooled specificity was 0.945 (95% CI 0.918-0.964, I2 = 100%). The pooled diagnostic odds ratio (DOR) was 219.47 (95% CI 104.78-459.66, I2 = 100%). These results were significant for the specificity of the different network architecture models (p-value = 0.0289). However, the results for sensitivity (p-value = 0.6417) and DOR (p-value = 0.2187) were not significant. The ResNet algorithm has higher pooled specificity than other algorithms with 0.935 (95% CI 0.854-0.973, I2 = 93%). CONCLUSION This meta-analysis on DTA of ML algorithms for detecting ICH by assessing non-contrast CT-Scans shows the ML has an acceptable performance in diagnosing ICH. Using ResNet in ICH detection remains promising prediction was improved via training in an Architecture Learning Network (ALN).
Collapse
Affiliation(s)
- Masoud Maghami
- Medical Doctor (MD), School of Medicine, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran
| | - Shahab Aldin Sattari
- Department of Neurosurgery, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Marziyeh Tahmasbi
- Department of Medical Imaging and Radiation Sciences, School of Allied Medical Sciences, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran
| | - Pegah Panahi
- Medical Doctor (MD), School of Medicine, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran
| | - Javad Mozafari
- Department of Emergency Medicine, School of Medicine, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran
- Department of Radiology, Resident (MD), EUREGIO-KLINIK Albert-Schweitzer-Straße GmbH, Nordhorn, Germany
| | | |
Collapse
|
35
|
Wang Z, Zhang Q, Wang Y, Zhu M, Li Q. A framework for immunofluorescence image augmentation and classification based on unsupervised attention mechanism. JOURNAL OF BIOPHOTONICS 2023; 16:e202300209. [PMID: 37559356 DOI: 10.1002/jbio.202300209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 07/16/2023] [Accepted: 08/07/2023] [Indexed: 08/11/2023]
Abstract
Autoimmune encephalitis (AE) is a common neurological disorder. As a standard method for neuroautoantibody detection, pathologists use tissue matrix assays (TBA) for initial disease screening. In this study, microscopic fluorescence imaging was combined with deep learning to improve AE diagnostic accuracy. Due to the inter-class imbalance of medical data, we propose an innovative generative adversarial network supplemented with attention mechanisms to highlight key regions in images to synthesize high-quality fluorescence images. However, securing annotated medical data is both time-consuming and costly. To circumvent this problem, we employ a self-supervised learning approach that utilizes unlabeled fluorescence data to support downstream classification tasks. To better understand the fluorescence properties in the data, we introduce a multichannel input convolutional neural network that adds additional channels of fluorescence intensity. This study builds an AE immunofluorescence dataset and obtains the classification accuracy of 88.5% using our method, thus confirming the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Ziyi Wang
- Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China
- Engineering Research Center of Nanophotonics & Advanced Instrument, Ministry of Education, East China Normal University, Shanghai, China
| | - Qing Zhang
- Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China
- Engineering Research Center of Nanophotonics & Advanced Instrument, Ministry of Education, East China Normal University, Shanghai, China
| | - Yan Wang
- Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China
- Engineering Center of SHMEC for Space Information and GNSS, Shanghai, China
| | - Min Zhu
- Department of Dermatology, Huashan Hospital, Fudan University, Shanghai, China
| | - Qingli Li
- Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai, China
- Engineering Research Center of Nanophotonics & Advanced Instrument, Ministry of Education, East China Normal University, Shanghai, China
- Engineering Center of SHMEC for Space Information and GNSS, Shanghai, China
| |
Collapse
|
36
|
Matsuo H, Kitajima K, Kono AK, Kuribayashi K, Kijima T, Hashimoto M, Hasegawa S, Yamakado K, Murakami T. Prognosis prediction of patients with malignant pleural mesothelioma using conditional variational autoencoder on 3D PET images and clinical data. Med Phys 2023; 50:7548-7557. [PMID: 37651615 DOI: 10.1002/mp.16694] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2022] [Revised: 08/02/2023] [Accepted: 08/02/2023] [Indexed: 09/02/2023] Open
Abstract
BACKGROUND Deep learning (DL) has been widely used for diagnosis and prognosis prediction of numerous frequently occurring diseases. Generally, DL models require large datasets to perform accurate and reliable prognosis prediction and avoid overlearning. However, prognosis prediction of rare diseases is still limited owing to the small number of cases, resulting in small datasets. PURPOSE This paper proposes a multimodal DL method to predict the prognosis of patients with malignant pleural mesothelioma (MPM) with a small number of 3D positron emission tomography-computed tomography (PET/CT) images and clinical data. METHODS A 3D convolutional conditional variational autoencoder (3D-CCVAE), which adds a 3D-convolutional layer and conditional VAE to process 3D images, was used for dimensionality reduction of PET images. We developed a two-step model that performs dimensionality reduction using the 3D-CCVAE, which is resistant to overlearning. In the first step, clinical data were input to condition the model and perform dimensionality reduction of PET images, resulting in more efficient dimension reduction. In the second step, a subset of the dimensionally reduced features and clinical data were combined to predict 1-year survival of patients using the random forest classifier. To demonstrate the usefulness of the 3D-CCVAE, we created a model without the conditional mechanism (3D-CVAE), one without the variational mechanism (3D-CCAE), and one without an autoencoder (without AE), and compared their prediction results. We used PET images and clinical data of 520 patients with histologically proven MPM. The data were randomly split in a 2:1 ratio (train : test) and three-fold cross-validation was performed. The models were trained on the training set and evaluated based on the test set results. The area under the receiver operating characteristic curve (AUC) for all models was calculated using their 1-year survival predictions, and the results were compared. RESULTS We obtained AUC values of 0.76 (95% confidence interval [CI], 0.72-0.80) for the 3D-CCVAE model, 0.72 (95% CI, 0.68-0.77) for the 3D-CVAE model, 0.70 (95% CI, 0.66-0.75) for the 3D-CCAE model, and 0.69 (95% CI 0.65-0.74) for the without AE model. The 3D-CCVAE model performed better than the other models (3D-CVAE, p = 0.039; 3D-CCAE, p = 0.0032; and without AE, p = 0.0011). CONCLUSIONS This study demonstrates the usefulness of the 3D-CCVAE in multimodal DL models learned using a small number of datasets. Additionally, it shows that dimensionality reduction via AE can be used to learn a DL model without increasing the overlearning risk. Moreover, the VAE mechanism can overcome the uncertainty of the model parameters that commonly occurs for small datasets, thereby eliminating the risk of overlearning. Additionally, more efficient dimensionality reduction of PET images can be performed by providing clinical data as conditions and ignoring clinical data-related features.
Collapse
Affiliation(s)
- Hidetoshi Matsuo
- Department of Radiology, Kobe University Graduate School of Medicine, Kobe, Japan
| | - Kazuhiro Kitajima
- Department of Radiology, Hyogo Medical University, School of Medicine, Nishinomiya, Hyogo, Japan
| | - Atsushi K Kono
- Department of Radiology, Kobe University Graduate School of Medicine, Kobe, Japan
| | - Kozo Kuribayashi
- Department of Respiratory Medicine and Hematology, Hyogo Medical University, School of Medicine, Nishinomiya, Hyogo, Japan
| | - Takashi Kijima
- Department of Respiratory Medicine and Hematology, Hyogo Medical University, School of Medicine, Nishinomiya, Hyogo, Japan
| | - Masaki Hashimoto
- Department of Thoracic Surgery, Hyogo Medical University, School of Medicine, Nishinomiya, Hyogo, Japan
| | - Seiki Hasegawa
- Department of Thoracic Surgery, Hyogo Medical University, School of Medicine, Nishinomiya, Hyogo, Japan
| | - Koichiro Yamakado
- Department of Radiology, Hyogo Medical University, School of Medicine, Nishinomiya, Hyogo, Japan
| | - Takamichi Murakami
- Department of Radiology, Kobe University Graduate School of Medicine, Kobe, Japan
| |
Collapse
|
37
|
Allam AK, Anand A, Flores AR, Ropper AE. Computer Vision in Osteoporotic Vertebral Fracture Risk Prediction: A Systematic Review. Neurospine 2023; 20:1112-1123. [PMID: 38171281 PMCID: PMC10762393 DOI: 10.14245/ns.2347022.511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2023] [Revised: 12/03/2023] [Accepted: 12/05/2023] [Indexed: 01/05/2024] Open
Abstract
Osteoporotic vertebral fractures (OVFs) are a significant health concern linked to increased morbidity, mortality, and diminished quality of life. Traditional OVF risk assessment tools like bone mineral density (BMD) only capture a fraction of the risk profile. Artificial intelligence, specifically computer vision, has revolutionized other fields of medicine through analysis of videos, histopathology slides and radiological scans. In this review, we provide an overview of computer vision algorithms and current computer vision models used in predicting OVF risk. We highlight the clinical applications, future directions and limitations of computer vision in OVF risk prediction.
Collapse
Affiliation(s)
- Anthony K. Allam
- Department of Neurosurgery, Baylor College of Medicine, Houston, TX, USA
| | - Adrish Anand
- Department of Neurosurgery, Baylor College of Medicine, Houston, TX, USA
| | - Alex R. Flores
- Department of Neurosurgery, Baylor College of Medicine, Houston, TX, USA
| | | |
Collapse
|
38
|
Yoo YS, Kim D, Yang S, Kang SR, Kim JE, Huh KH, Lee SS, Heo MS, Yi WJ. Comparison of 2D, 2.5D, and 3D segmentation networks for maxillary sinuses and lesions in CBCT images. BMC Oral Health 2023; 23:866. [PMID: 37964229 PMCID: PMC10647072 DOI: 10.1186/s12903-023-03607-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 11/01/2023] [Indexed: 11/16/2023] Open
Abstract
BACKGROUND The purpose of this study was to compare the segmentation performances of the 2D, 2.5D, and 3D networks for maxillary sinuses (MSs) and lesions inside the maxillary sinus (MSL) with variations in sizes, shapes, and locations in cone beam CT (CBCT) images under the same constraint of memory capacity. METHODS The 2D, 2.5D, and 3D networks were compared comprehensively for the segmentation of the MS and MSL in CBCT images under the same constraint of memory capacity. MSLs were obtained by subtracting the prediction of the air region of the maxillary sinus (MSA) from that of the MS. RESULTS The 2.5D network showed the highest segmentation performances for the MS and MSA compared to the 2D and 3D networks. The performances of the Jaccard coefficient, Dice similarity coefficient, precision, and recall by the 2.5D network of U-net + + reached 0.947, 0.973, 0.974, and 0.971 for the MS, respectively, and 0.787, 0.875, 0.897, and 0.858 for the MSL, respectively. CONCLUSIONS The 2.5D segmentation network demonstrated superior segmentation performance for various MSLs with an ensemble learning approach of combining the predictions from three orthogonal planes.
Collapse
Affiliation(s)
- Yeon-Sun Yoo
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, Korea
| | - DaEl Kim
- Interdisciplinary Program in Bioengineering, Graduate School of Engineering, Seoul National University, Seoul, Korea
| | - Su Yang
- Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea
| | - Se-Ryong Kang
- Department of Biomedical Radiation Sciences, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea
| | - Jo-Eun Kim
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, Korea
| | - Kyung-Hoe Huh
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, Korea
| | - Sam-Sun Lee
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, Korea
| | - Min-Suk Heo
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, Korea
| | - Won-Jin Yi
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, Korea.
- Interdisciplinary Program in Bioengineering, Graduate School of Engineering, Seoul National University, Seoul, Korea.
- Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea.
- Department of Biomedical Radiation Sciences, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea.
| |
Collapse
|
39
|
Kuehle R, Ringwald F, Bouffleur F, Hagen N, Schaufelberger M, Nahm W, Hoffmann J, Freudlsperger C, Engel M, Eisenmann U. The Use of Artificial Intelligence for the Classification of Craniofacial Deformities. J Clin Med 2023; 12:7082. [PMID: 38002694 PMCID: PMC10672418 DOI: 10.3390/jcm12227082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2023] [Revised: 10/27/2023] [Accepted: 11/02/2023] [Indexed: 11/26/2023] Open
Abstract
Positional cranial deformities are a common finding in toddlers, yet differentiation from craniosynostosis can be challenging. The aim of this study was to train convolutional neural networks (CNNs) to classify craniofacial deformities based on 2D images generated using photogrammetry as a radiation-free imaging technique. A total of 487 patients with photogrammetry scans were included in this retrospective cohort study: children with craniosynostosis (n = 227), positional deformities (n = 206), and healthy children (n = 54). Three two-dimensional images were extracted from each photogrammetry scan. The datasets were divided into training, validation, and test sets. During the training, fine-tuned ResNet-152s were utilized. The performance was quantified using tenfold cross-validation. For the detection of craniosynostosis, sensitivity was at 0.94 with a specificity of 0.85. Regarding the differentiation of the five existing classes (trigonocephaly, scaphocephaly, positional plagiocephaly left, positional plagiocephaly right, and healthy), sensitivity ranged from 0.45 (positional plagiocephaly left) to 0.95 (scaphocephaly) and specificity ranged from 0.87 (positional plagiocephaly right) to 0.97 (scaphocephaly). We present a CNN-based approach to classify craniofacial deformities on two-dimensional images with promising results. A larger dataset would be required to identify rarer forms of craniosynostosis as well. The chosen 2D approach enables future applications for digital cameras or smartphones.
Collapse
Affiliation(s)
- Reinald Kuehle
- Department of Oral and Maxillofacial Surgery, University of Heidelberg, 69120 Heidelberg, Germany; (F.B.); (J.H.); (C.F.)
| | - Friedemann Ringwald
- Institute of Medical Informatics, University of Heidelberg, 69120 Heidelberg, Germany; (F.R.)
| | - Frederic Bouffleur
- Department of Oral and Maxillofacial Surgery, University of Heidelberg, 69120 Heidelberg, Germany; (F.B.); (J.H.); (C.F.)
| | - Niclas Hagen
- Institute of Medical Informatics, University of Heidelberg, 69120 Heidelberg, Germany; (F.R.)
| | - Matthias Schaufelberger
- Institute of Biomedical Engineering, Karlsruhe Institute for Technology, 76131 Karlsruhe, Germany
| | - Werner Nahm
- Institute of Biomedical Engineering, Karlsruhe Institute for Technology, 76131 Karlsruhe, Germany
| | - Jürgen Hoffmann
- Department of Oral and Maxillofacial Surgery, University of Heidelberg, 69120 Heidelberg, Germany; (F.B.); (J.H.); (C.F.)
| | - Christian Freudlsperger
- Department of Oral and Maxillofacial Surgery, University of Heidelberg, 69120 Heidelberg, Germany; (F.B.); (J.H.); (C.F.)
| | - Michael Engel
- Department of Oral and Maxillofacial Surgery, University of Heidelberg, 69120 Heidelberg, Germany; (F.B.); (J.H.); (C.F.)
| | - Urs Eisenmann
- Institute of Medical Informatics, University of Heidelberg, 69120 Heidelberg, Germany; (F.R.)
| |
Collapse
|
40
|
Yousefpour Shahrivar R, Karami F, Karami E. Enhancing Fetal Anomaly Detection in Ultrasonography Images: A Review of Machine Learning-Based Approaches. Biomimetics (Basel) 2023; 8:519. [PMID: 37999160 PMCID: PMC10669151 DOI: 10.3390/biomimetics8070519] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Revised: 10/05/2023] [Accepted: 10/26/2023] [Indexed: 11/25/2023] Open
Abstract
Fetal development is a critical phase in prenatal care, demanding the timely identification of anomalies in ultrasound images to safeguard the well-being of both the unborn child and the mother. Medical imaging has played a pivotal role in detecting fetal abnormalities and malformations. However, despite significant advances in ultrasound technology, the accurate identification of irregularities in prenatal images continues to pose considerable challenges, often necessitating substantial time and expertise from medical professionals. In this review, we go through recent developments in machine learning (ML) methods applied to fetal ultrasound images. Specifically, we focus on a range of ML algorithms employed in the context of fetal ultrasound, encompassing tasks such as image classification, object recognition, and segmentation. We highlight how these innovative approaches can enhance ultrasound-based fetal anomaly detection and provide insights for future research and clinical implementations. Furthermore, we emphasize the need for further research in this domain where future investigations can contribute to more effective ultrasound-based fetal anomaly detection.
Collapse
Affiliation(s)
- Ramin Yousefpour Shahrivar
- Department of Biology, College of Convergent Sciences and Technologies, Science and Research Branch, Islamic Azad University, Tehran, 14515-775, Iran
| | - Fatemeh Karami
- Department of Medical Genetics, Applied Biophotonics Research Center, Science and Research Branch, Islamic Azad University, Tehran, 14515-775, Iran
| | - Ebrahim Karami
- Department of Engineering and Applied Sciences, Memorial University of Newfoundland, St. John’s, NL A1B 3X5, Canada
| |
Collapse
|
41
|
Makaremi M, Vafaei Sadr A, Marcy B, Chraibi Kaadoud I, Mohammad-Djafari A, Sadoun S, De Brondeau F, N'kaoua B. An interpretable machine learning approach to study the relationship beetwen retrognathia and skull anatomy. Sci Rep 2023; 13:18130. [PMID: 37875537 PMCID: PMC10597995 DOI: 10.1038/s41598-023-45314-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2023] [Accepted: 10/18/2023] [Indexed: 10/26/2023] Open
Abstract
Mandibular retrognathia (C2Rm) is one of the most common oral pathologies. Acquiring a better understanding of the points of impact of C2Rm on the entire skull is of major interest in the diagnosis, treatment, and management of this dysmorphism, but also permits us to contribute to the debate on the changes undergone by the shape of the skull during human evolution. However, conventional methods have some limits in meeting these challenges, insofar as they require defining in advance the structures to be studied, and identifying them using landmarks. In this context, our work aims to answer these questions using AI tools and, in particular, machine learning, with the objective of relaying these treatments automatically. We propose an innovative methodology coupling convolutional neural networks (CNNs) and interpretability algorithms. Applied to a set of radiographs classified into physiological versus pathological categories, our methodology made it possible to: discuss the structures impacted by retrognathia and already identified in literature; identify new structures of potential interest in medical terms; highlight the dynamic evolution of impacted structures according to the level of gravity of C2Rm; provide for insights into the evolution of human anatomy. Results were discussed in terms of the major interest of this approach in the field of orthodontics and, more generally, in the field of automated processing of medical images.
Collapse
Affiliation(s)
- Masrour Makaremi
- Dentofacial Orthopedics Department (UFR de Sciences Odontologiques), University of Bordeaux, Bordeaux, France.
- Bordeaux Population Health (Team ACTIVE), INSERM U1219, University of Bordeaux, Talence, France.
| | - Alireza Vafaei Sadr
- Department of Public Health Sciences, College of Medicine, The Pennsylvania State University, Hershey, PA, 17033, USA
- Theoretical Physics Department, University of Geneva, 1211, Geneva, Switzerland
| | | | | | | | - Salomé Sadoun
- Dentofacial Orthopedics Department (UFR de Sciences Odontologiques), University of Bordeaux, Bordeaux, France
| | - François De Brondeau
- Dentofacial Orthopedics Department (UFR de Sciences Odontologiques), University of Bordeaux, Bordeaux, France
| | - Bernard N'kaoua
- Bordeaux Population Health (Team ACTIVE), INSERM U1219, University of Bordeaux, Talence, France
| |
Collapse
|
42
|
Domingo L, Djukic M, Johnson C, Borondo F. Binding affinity predictions with hybrid quantum-classical convolutional neural networks. Sci Rep 2023; 13:17951. [PMID: 37864075 PMCID: PMC10589342 DOI: 10.1038/s41598-023-45269-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Accepted: 10/17/2023] [Indexed: 10/22/2023] Open
Abstract
Central in drug design is the identification of biomolecules that uniquely and robustly bind to a target protein, while minimizing their interactions with others. Accordingly, precise binding affinity prediction, enabling the accurate selection of suitable candidates from an extensive pool of potential compounds, can greatly reduce the expenses associated to practical experimental protocols. In this respect, recent advances revealed that deep learning methods show superior performance compared to other traditional computational methods, especially with the advent of large datasets. These methods, however, are complex and very time-intensive, thus representing an important clear bottleneck for their development and practical application. In this context, the emerging realm of quantum machine learning holds promise for enhancing numerous classical machine learning algorithms. In this work, we take one step forward and present a hybrid quantum-classical convolutional neural network, which is able to reduce by 20% the complexity of the classical counterpart while still maintaining optimal performance in the predictions. Additionally, this results in a significant cost and time savings of up to 40% in the training stage, which means a substantial speed-up of the drug design process.
Collapse
Affiliation(s)
- L Domingo
- Grupo de Sistemas Complejos, Universidad Politécnica de Madrid, 28035, Madrid, Spain.
- Instituto de Ciencias Matemáticas (ICMAT), Campus de Cantoblanco UAM, Nicolás Cabrera, 13-15, 28049, Madrid, Spain.
- Departamento de Química, Universidad Autónoma de Madrid, 28049, Cantoblanco, Madrid, Spain.
- Ingenii Inc., New York, USA.
| | | | | | - F Borondo
- Departamento de Química, Universidad Autónoma de Madrid, 28049, Cantoblanco, Madrid, Spain
| |
Collapse
|
43
|
Bakasa W, Viriri S. Stacked ensemble deep learning for pancreas cancer classification using extreme gradient boosting. Front Artif Intell 2023; 6:1232640. [PMID: 37876961 PMCID: PMC10591225 DOI: 10.3389/frai.2023.1232640] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 09/04/2023] [Indexed: 10/26/2023] Open
Abstract
Ensemble learning aims to improve prediction performance by combining several models or forecasts. However, how much and which ensemble learning techniques are useful in deep learning-based pipelines for pancreas computed tomography (CT) image classification is a challenge. Ensemble approaches are the most advanced solution to many machine learning problems. These techniques entail training multiple models and combining their predictions to improve the predictive performance of a single model. This article introduces the idea of Stacked Ensemble Deep Learning (SEDL), a pipeline for classifying pancreas CT medical images. The weak learners are Inception V3, VGG16, and ResNet34, and we employed a stacking ensemble. By combining the first-level predictions, an input train set for XGBoost, the ensemble model at the second level of prediction, is created. Extreme Gradient Boosting (XGBoost), employed as a strong learner, will make the final classification. Our findings showed that SEDL performed better, with a 98.8% ensemble accuracy, after some adjustments to the hyperparameters. The Cancer Imaging Archive (TCIA) public access dataset consists of 80 pancreas CT scans with a resolution of 512 * 512 pixels, from 53 male and 27 female subjects. A sample of two hundred and twenty-two images was used for training and testing data. We concluded that implementing the SEDL technique is an effective way to strengthen the robustness and increase the performance of the pipeline for classifying pancreas CT medical images. Interestingly, grouping like-minded or talented learners does not make a difference.
Collapse
Affiliation(s)
| | - Serestina Viriri
- School of Mathematics Statistics & Computer Science, College of Agriculture, Engineering and Science, University of KwaZulu-Natal, Durban, South Africa
| |
Collapse
|
44
|
Gil J, Choi H, Paeng JC, Cheon GJ, Kang KW. Deep Learning-Based Feature Extraction from Whole-Body PET/CT Employing Maximum Intensity Projection Images: Preliminary Results of Lung Cancer Data. Nucl Med Mol Imaging 2023; 57:216-222. [PMID: 37720886 PMCID: PMC10504178 DOI: 10.1007/s13139-023-00802-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Revised: 03/20/2023] [Accepted: 04/03/2023] [Indexed: 09/19/2023] Open
Abstract
Purpose Deep learning (DL) has been widely used in various medical imaging analyses. Because of the difficulty in processing volume data, it is difficult to train a DL model as an end-to-end approach using PET volume as an input for various purposes including diagnostic classification. We suggest an approach employing two maximum intensity projection (MIP) images generated by whole-body FDG PET volume to employ pre-trained models based on 2-D images. Methods As a retrospective, proof-of-concept study, 562 [18F]FDG PET/CT images and clinicopathological factors of lung cancer patients were collected. MIP images of anterior and lateral views were used as inputs, and image features were extracted by a pre-trained convolutional neural network (CNN) model, ResNet-50. The relationship between the images was depicted on a parametric 2-D axes map using t-distributed stochastic neighborhood embedding (t-SNE), with clinicopathological factors. Results A DL-based feature map extracted by two MIP images was embedded by t-SNE. According to the visualization of the t-SNE map, PET images were clustered by clinicopathological features. The representative difference between the clusters of PET patterns according to the posture of a patient was visually identified. This map showed a pattern of clustering according to various clinicopathological factors including sex as well as tumor staging. Conclusion A 2-D image-based pre-trained model could extract image patterns of whole-body FDG PET volume by using anterior and lateral views of MIP images bypassing the direct use of 3-D PET volume that requires large datasets and resources. We suggest that this approach could be implemented as a backbone model for various applications for whole-body PET image analyses.
Collapse
Affiliation(s)
- Joonhyung Gil
- Department of Nuclear Medicine, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
| | - Hongyoon Choi
- Department of Nuclear Medicine, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
| | - Jin Chul Paeng
- Department of Nuclear Medicine, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
| | - Gi Jeong Cheon
- Department of Nuclear Medicine, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
| | - Keon Wook Kang
- Department of Nuclear Medicine, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul, 03080 Republic of Korea
| |
Collapse
|
45
|
Nader R, Bourcier R, Autrusseau F. Using deep learning for an automatic detection and classification of the vascular bifurcations along the Circle of Willis. Med Image Anal 2023; 89:102919. [PMID: 37619447 DOI: 10.1016/j.media.2023.102919] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 06/01/2023] [Accepted: 07/31/2023] [Indexed: 08/26/2023]
Abstract
Most of the intracranial aneurysms (ICA) occur on a specific portion of the cerebral vascular tree named the Circle of Willis (CoW). More particularly, they mainly arise onto fifteen of the major arterial bifurcations constituting this circular structure. Hence, for an efficient and timely diagnosis it is critical to develop some methods being able to accurately recognize each Bifurcation of Interest (BoI). Indeed, an automatic extraction of the bifurcations presenting the higher risk of developing an ICA would offer the neuroradiologists a quick glance at the most alarming areas. Due to the recent efforts on Artificial Intelligence, Deep Learning turned out to be the best performing technology for many pattern recognition tasks. Moreover, various methods have been particularly designed for medical image analysis purposes. This study intends to assist the neuroradiologists to promptly locate any bifurcation presenting a high risk of ICA occurrence. It can be seen as a Computer Aided Diagnosis scheme, where the Artificial Intelligence facilitates the access to the regions of interest within the MRI. In this work, we propose a method for a fully automatic detection and recognition of the bifurcations of interest forming the Circle of Willis. Several neural networks architectures have been tested, and we thoroughly evaluate the bifurcation recognition rate.
Collapse
Affiliation(s)
- Rafic Nader
- Nantes Université, CHU Nantes, CNRS, INSERM, l'institut du thorax, F-44000 Nantes, France
| | - Romain Bourcier
- Nantes Université, CHU Nantes, CNRS, INSERM, l'institut du thorax, F-44000 Nantes, France
| | - Florent Autrusseau
- Nantes Université, CHU Nantes, CNRS, INSERM, l'institut du thorax, F-44000 Nantes, France; Nantes Université, Polytech'Nantes, LTeN, U-6607, Rue Ch. Pauc, 44306, Nantes, France.
| |
Collapse
|
46
|
Umapathy S, Murugappan M, Bharathi D, Thakur M. Automated Computer-Aided Detection and Classification of Intracranial Hemorrhage Using Ensemble Deep Learning Techniques. Diagnostics (Basel) 2023; 13:2987. [PMID: 37761354 PMCID: PMC10527774 DOI: 10.3390/diagnostics13182987] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2023] [Revised: 09/03/2023] [Accepted: 09/14/2023] [Indexed: 09/29/2023] Open
Abstract
Diagnosing Intracranial Hemorrhage (ICH) at an early stage is difficult since it affects the blood vessels in the brain, often resulting in death. We propose an ensemble of Convolutional Neural Networks (CNNs) combining Squeeze and Excitation-based Residual Networks with the next dimension (SE-ResNeXT) and Long Short-Term Memory (LSTM) Networks in order to address this issue. This research work primarily used data from the Radiological Society of North America (RSNA) brain CT hemorrhage challenge dataset and the CQ500 dataset. Preprocessing and data augmentation are performed using the windowing technique in the proposed work. The ICH is then classified using ensembled CNN techniques after being preprocessed, followed by feature extraction in an automatic manner. ICH is classified into the following five types: epidural, intraventricular, subarachnoid, intra-parenchymal, and subdural. A gradient-weighted Class Activation Mapping method (Grad-CAM) is used for identifying the region of interest in an ICH image. A number of performance measures are used to compare the experimental results with various state-of-the-art algorithms. By achieving 99.79% accuracy with an F-score of 0.97, the proposed model proved its efficacy in detecting ICH compared to other deep learning models. The proposed ensembled model can classify epidural, intraventricular, subarachnoid, intra-parenchymal, and subdural hemorrhages with an accuracy of 99.89%, 99.65%, 98%, 99.75%, and 99.88%. Simulation results indicate that the suggested approach can categorize a variety of intracranial bleeding types. By implementing the ensemble deep learning technique using the SE-ResNeXT and LSTM models, we achieved significant classification accuracy and AUC scores.
Collapse
Affiliation(s)
- Snekhalatha Umapathy
- Department of Biomedical Engineering, SRM Institute of Science and Technology, Chennai 603203, India
- College of Engineering, Architecture, and Fine Arts, Batangas State University, Batangas 4200, Philippines
| | - Murugappan Murugappan
- Intelligent Signal Processing (ISP) Research Lab, Department of Electronics and Communication Engineering, Kuwait College of Science and Technology, Block 4, Doha 13133, Kuwait
- Department of Electronics and Communication Engineering, School of Engineering, Vels Institute of Sciences, Technology, and Advanced Studies, Chennai 600117, India
- Center of Excellence for Unmanned Aerial Systems (CoEUAS), Universiti Malaysia Perlis, Arau 02600, Perlis, Malaysia
| | - Deepa Bharathi
- Department of Electronics and Communication Engineering, SRM Institute of Science and Technology, Ramapuram, Chennai 600089, India
| | - Mahima Thakur
- Department of Biomedical Engineering, SRM Institute of Science and Technology, Chennai 603203, India
| |
Collapse
|
47
|
Ahmed R, Al Shehhi A, Hassan B, Werghi N, Seghier ML. An appraisal of the performance of AI tools for chronic stroke lesion segmentation. Comput Biol Med 2023; 164:107302. [PMID: 37572443 DOI: 10.1016/j.compbiomed.2023.107302] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2023] [Revised: 07/18/2023] [Accepted: 07/28/2023] [Indexed: 08/14/2023]
Abstract
Automated demarcation of stoke lesions from monospectral magnetic resonance imaging scans is extremely useful for diverse research and clinical applications, including lesion-symptom mapping to explain deficits and predict recovery. There is a significant surge of interest in the development of supervised artificial intelligence (AI) methods for that purpose, including deep learning, with a performance comparable to trained experts. Such AI-based methods, however, require copious amounts of data. Thanks to the availability of large datasets, the development of AI-based methods for lesion segmentation has immensely accelerated in the last decade. One of these datasets is the Anatomical Tracings of Lesions After Stroke (ATLAS) dataset which includes T1-weighted images from hundreds of chronic stroke survivors with their manually traced lesions. This systematic review offers an appraisal of the impact of the ATLAS dataset in promoting the development of AI-based segmentation of stroke lesions. An examination of all published studies, that used the ATLAS dataset to both train and test their methods, highlighted an overall moderate performance (median Dice index = 59.40%) and a huge variability across studies in terms of data preprocessing, data augmentation, AI architecture, and the mode of operation (two-dimensional versus three-dimensional methods). Perhaps most importantly, almost all AI tools were borrowed from existing AI architectures in computer vision, as 90% of all selected studies relied on conventional convolutional neural network-based architectures. Overall, current research has not led to the development of robust AI architectures than can handle spatially heterogenous lesion patterns. This review also highlights the difficulty of gauging the performance of AI tools in the presence of uncertainties in the definition of the ground truth.
Collapse
Affiliation(s)
- Ramsha Ahmed
- Department of Biomedical Engineering, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
| | - Aamna Al Shehhi
- Department of Biomedical Engineering, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates; Healthcare Engineering Innovation Center (HEIC), Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
| | - Bilal Hassan
- Department of Electrical Engineering and Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
| | - Naoufel Werghi
- Healthcare Engineering Innovation Center (HEIC), Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates; Department of Electrical Engineering and Computer Science, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
| | - Mohamed L Seghier
- Department of Biomedical Engineering, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates; Healthcare Engineering Innovation Center (HEIC), Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates.
| |
Collapse
|
48
|
Koponen M, Anwaar W, Sheikh Q, Sadiq F. Use of Artificial Intelligence in Coronary Artery Calcium Scoring. Oman Med J 2023; 38:e543. [PMID: 38053612 PMCID: PMC10694408 DOI: 10.5001/omj.2023.73] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Accepted: 11/07/2022] [Indexed: 12/07/2023] Open
Abstract
Coronary artery calcium (CAC) scoring improves traditional risk factor-based coronary heart disease (CHD) risk stratification. Here, the contribution of CAC scoring to a traditional 10-year CHD risk prediction scores and new artificial intelligence methods used to automate CAC scoring were reviewed. Research shows that traditional risk factors tend to overestimate or underestimate the actual risk of CHD, meaning that including CAC score in the risk stratification has potential to reduce over- and undertreatment. The automated CAC scoring methods are shown to be accurate and significantly more time-effective than the commonly used semi-automated method.
Collapse
Affiliation(s)
- Mia Koponen
- School of Medical Sciences, University of Aberdeen, Aberdeen, UK
| | - Waqas Anwaar
- Department of Computing, Shifa School of Computing, Shifa Tameer-e-Millat University, Islamabad, Pakistan
| | - Habib-ur-Rahman3
- School of Medical Sciences, University of Aberdeen, Aberdeen, UK
- Department of Computing, Shifa School of Computing, Shifa Tameer-e-Millat University, Islamabad, Pakistan
- Department of Cardiology, Shifa International Hospital, Islamabad, Pakistan
- Directorate of Research, Shifa Tameer-e-Millat University, Islamabad, Pakistan
| | - Qasim Sheikh
- Department of Computing, Shifa School of Computing, Shifa Tameer-e-Millat University, Islamabad, Pakistan
| | - Fouzia Sadiq
- Directorate of Research, Shifa Tameer-e-Millat University, Islamabad, Pakistan
| |
Collapse
|
49
|
Xu H, Pan H, Li J. Surface Defect Detection of Bearing Rings Based on an Improved YOLOv5 Network. SENSORS (BASEL, SWITZERLAND) 2023; 23:7443. [PMID: 37687898 PMCID: PMC10490562 DOI: 10.3390/s23177443] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 08/14/2023] [Accepted: 08/23/2023] [Indexed: 09/10/2023]
Abstract
Considering the characteristics of complex texture backgrounds, uneven brightness, varying defect sizes, and multiple defect types of the bearing surface images, a surface defect detection method for bearing rings is proposed based on improved YOLOv5. First, replacing the C3 module in the backbone network with a C2f module can effectively reduce the number of network parameters and computational complexity, thereby improving the speed and accuracy of the backbone network. Second, adding the SPD module into the backbone and neck networks enhances their ability to process low-resolution and small-object images. Next, replacing the nearest-neighbor upsampling with the lightweight and universal CARAFE operator fully utilizes feature semantic information, enriches contextual information, and reduces information loss during transmission, thereby effectively improving the model's diversity and robustness. Finally, we constructed a dataset of bearing ring surface images collected from industrial sites and conducted numerous experiments based on this dataset. Experimental results show that the mean average precision (mAP) of the network is 97.3%, especially for dents and black spot defects, improved by 2.2% and 3.9%, respectively, and that the detection speed can reach 100 frames per second (FPS). Compared with mainstream surface defect detection algorithms, the proposed method shows significant improvements in both accuracy and detection time and can meet the requirements of industrial defect detection.
Collapse
Affiliation(s)
- Haitao Xu
- School of Information Science and Engineering, Zhejiang Sci-Tech University, Hangzhou 310018, China; (H.X.); (J.L.)
- Changshan Research Institute, Zhejiang Sci-Tech University, Quzhou 324299, China
| | - Haipeng Pan
- School of Information Science and Engineering, Zhejiang Sci-Tech University, Hangzhou 310018, China; (H.X.); (J.L.)
- Changshan Research Institute, Zhejiang Sci-Tech University, Quzhou 324299, China
| | - Junfeng Li
- School of Information Science and Engineering, Zhejiang Sci-Tech University, Hangzhou 310018, China; (H.X.); (J.L.)
- Changshan Research Institute, Zhejiang Sci-Tech University, Quzhou 324299, China
| |
Collapse
|
50
|
Zhong S, Ren JX, Yu ZP, Peng YD, Yu CW, Deng D, Xie Y, He ZQ, Duan H, Wu B, Li H, Yang WZ, Bai Y, Sai K, Chen YS, Guo CC, Li DP, Cheng Y, Zhang XH, Mou YG. Predicting glioblastoma molecular subtypes and prognosis with a multimodal model integrating convolutional neural network, radiomics, and semantics. J Neurosurg 2023; 139:305-314. [PMID: 36461822 DOI: 10.3171/2022.10.jns22801] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Accepted: 10/24/2022] [Indexed: 12/24/2022]
Abstract
OBJECTIVE The aim of this study was to build a convolutional neural network (CNN)-based prediction model of glioblastoma (GBM) molecular subtype diagnosis and prognosis with multimodal features. METHODS In total, 222 GBM patients were included in the training set from Sun Yat-sen University Cancer Center (SYSUCC) and 107 GBM patients were included in the validation set from SYSUCC, Xuanwu Hospital Capital Medical University, and the First Hospital of Jilin University. The multimodal model was trained with MR images (pre- and postcontrast T1-weighted images and T2-weighted images), corresponding MRI impression, and clinical patient information. First, the original images were segmented using the Multimodal Brain Tumor Image Segmentation Benchmark toolkit. Convolutional features were extracted using 3D residual deep neural network (ResNet50) and convolutional 3D (C3D). Radiomic features were extracted using pyradiomics. Report texts were converted to word embedding using word2vec. These three types of features were then integrated to train neural networks. Accuracy, precision, recall, and F1-score were used to evaluate the model performance. RESULTS The C3D-based model yielded the highest accuracy of 91.11% in the prediction of IDH1 mutation status. Importantly, the addition of semantics improved precision by 11.21% and recall in MGMT promoter methylation status prediction by 14.28%. The areas under the receiver operating characteristic curves of the C3D-based model in the IDH1, ATRX, MGMT, and 1-year prognosis groups were 0.976, 0.953, 0.955, and 0.976, respectively. In external validation, the C3D-based model showed significant improvement in accuracy in the IDH1, ATRX, MGMT, and 1-year prognosis groups, which were 88.30%, 76.67%, 85.71%, and 85.71%, respectively (compared with 3D ResNet50: 83.51%, 66.67%, 82.14%, and 70.79%, respectively). CONCLUSIONS The authors propose a novel multimodal model integrating C3D, radiomics, and semantics, which had a great performance in predicting IDH1, ATRX, and MGMT molecular subtypes and the 1-year prognosis of GBM.
Collapse
Affiliation(s)
- Sheng Zhong
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
- 2Department of Cancer Biology, Dana-Farber Cancer Institute, Boston, Massachusetts
- 3Department of Bioinformatics, Harvard Medical School, Boston, Massachusetts
| | - Jia-Xin Ren
- 4Department of Neurology, Stroke Center, The First Hospital of Jilin University, Changchun, China
| | - Ze-Peng Yu
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Yi-Da Peng
- 5College of Computer Science and Technology, Jilin University, Changchun, China
| | - Cheng-Wei Yu
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Davy Deng
- 2Department of Cancer Biology, Dana-Farber Cancer Institute, Boston, Massachusetts
| | - YangYiran Xie
- 6Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Zhen-Qiang He
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Hao Duan
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Bo Wu
- Departments of7Orthopaedics
| | | | - Wen-Zhuo Yang
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Yang Bai
- 9Neurosurgery, The First Hospital of Jilin University, Changchun, China; and
| | - Ke Sai
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Yin-Sheng Chen
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Cheng-Cheng Guo
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - De-Pei Li
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Ye Cheng
- 10Department of Neurosurgery, The Xuanwu Hospital Capital Medical University, Beijing, China
| | - Xiang-Heng Zhang
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Yong-Gao Mou
- 1Department of Neurosurgery and Neuro-Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| |
Collapse
|