1
|
Yang X, Liu L, Yan Z, Yu J, Hu X, Yu X, Dong C, Chen J, Liu H, Yu Z, Deng X, Ni D, Huang X, Gou Z. Hierarchical online contrastive anomaly detection for fetal arrhythmia diagnosis in ultrasound. Med Image Anal 2024; 97:103229. [PMID: 38897033 DOI: 10.1016/j.media.2024.103229] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 03/10/2024] [Accepted: 06/03/2024] [Indexed: 06/21/2024]
Abstract
Arrhythmia is a major cardiac abnormality in fetuses. Therefore, early diagnosis of arrhythmia is clinically crucial. Pulsed-wave Doppler ultrasound is a commonly used diagnostic tool for fetal arrhythmia. Its key step for diagnosis involves identifying adjacent measurable cardiac cycles (MCCs). As cardiac activity is complex and the experience of sonographers is often varied, automation can improve user-independence and diagnostic-validity. However, arrhythmias pose several challenges for automation because of complex waveform variations, which can cause major localization bias and missed or false detection of MCCs. Filtering out non-MCC anomalies is difficult because of large intra-class and small inter-class variations between MCCs and non-MCCs caused by agnostic morphological waveform variations. Moreover, rare arrhythmia cases are insufficient for classification algorithms to adequately learn discriminative features. Using only normal cases for training, we propose a novel hierarchical online contrastive anomaly detection (HOCAD) framework for arrhythmia diagnosis during test time. The contribution of this study is three-fold. First, we develop a coarse-to-fine framework inspired by hierarchical diagnostic logic, which can refine localization and avoid missed detection of MCCs. Second, we propose an online learning-based contrastive anomaly detection with two new anomaly scores, which can adaptively filter out non-MCC anomalies on a single image during testing. With these complementary efforts, we precisely determine MCCs for correct measurements and diagnosis. Third, to the best of our knowledge, this is the first reported study investigating intelligent diagnosis of fetal arrhythmia on a large-scale and multi-center ultrasound dataset. Extensive experiments on 3850 cases, including 266 cases covering three typical types of arrhythmias, demonstrate the effectiveness of the proposed framework.
Collapse
Affiliation(s)
- Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Lian Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Zhongnuo Yan
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Xindi Hu
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Xuejuan Yu
- Department of Ultrasonography, Suzhou Xiangcheng People's Hospital, Suzhou, Jiangsu, China
| | - Caixia Dong
- Department of Ultrasonography, Wulin Hospital, Hangzhou, Zhejiang, China
| | - Ju Chen
- Department of Ultrasonography, Taicang First People's Hospital, Suzhou, Jiangsu, China
| | - Hongmei Liu
- Department of Ultrasonography, Panzhou Emerging Hospital, Panzhou, Guizhou, China
| | - Zhuan Yu
- Department of Ultrasonography, The Affiliated Hospital of Jiangsu University, Zhenjiang, Jiangsu, China
| | - Xuedong Deng
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Xiaoqiong Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China.
| | - Zhongshan Gou
- Center for Cardiovascular Disease, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China.
| |
Collapse
|
2
|
Guo Z, Lu X, Yao J, Zhou Y, Chen C, Chen J, Yang D, Cao Y, Zheng W, Yang X, Ni D. Fully Automated Localization and Measurement of Levator Hiatus Dimensions Using 3-D Pelvic Floor Ultrasound. ULTRASOUND IN MEDICINE & BIOLOGY 2024; 50:1329-1338. [PMID: 38845332 DOI: 10.1016/j.ultrasmedbio.2024.05.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 04/26/2024] [Accepted: 05/05/2024] [Indexed: 08/06/2024]
Abstract
OBJECTIVE To develop an algorithm for the automated localization and measurement of levator hiatus (LH) dimensions (AI-LH) using 3-D pelvic floor ultrasound. METHODS The AI-LH included a 3-D plane regression model and a 2-D segmentation model, which first achieved automated localization of the minimal LH dimension plane (C-plane) and measurement of the hiatal area (HA) on maximum Valsalva on the rendered LH images, but not on the C-plane. The dataset included 600 volumetric data. We compared AI-LH with sonographer difference (ASD) as well as the inter-sonographer differences (IESD) in the testing dataset (n = 240). The assessment encompassed the mean absolute error (MAE) for the angle and center point distance of the C-plane, along with the Dice coefficient, MAE, and intra-class correlation coefficient (ICC) for HA, and included the time consumption. RESULTS The MAE of the C-plane of ASD was 4.81 ± 2.47° with 1.92 ± 1.54 mm. AI-LH achieved a mean Dice coefficient of 0.93 for LH segmentation. The MAE on HA of ASD (1.44 ± 1.12 mm²) was lower than that of IESD (1.63 ± 1.58 mm²). The ICC on HA of ASD (0.964) was higher than that of IESD (0.949). The average time costs of AI-LH and manual measurement were 2.00 ± 0.22 s and 59.60 ± 2.63 s (t = 18.87, p < 0.01), respectively. CONCLUSION AI-LH is accurate, reliable, and robust in the localization and measurement of LH dimensions, which can shorten the time cost, simplify the operation process, and have good value in clinical applications.
Collapse
Affiliation(s)
- Zhijie Guo
- Department of Ultrasound, Shenzhen Bao'an Chinese Medicine Hospital, Guangzhou University of Chinese Medicine, Shenzhen, China
| | - Xiduo Lu
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, China
| | - Jiezhi Yao
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen, China
| | - Yongsong Zhou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen, China
| | - Chaoyu Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen, China
| | - Jiongquan Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen, China
| | - Danling Yang
- Department of Ultrasound, Shenzhen Bao'an Chinese Medicine Hospital, Guangzhou University of Chinese Medicine, Shenzhen, China
| | - Yan Cao
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, China
| | - Wei Zheng
- Department of Ultrasound, Shenzhen Bao'an Chinese Medicine Hospital, Guangzhou University of Chinese Medicine, Shenzhen, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen, China.
| |
Collapse
|
3
|
Lasala A, Fiorentino MC, Bandini A, Moccia S. FetalBrainAwareNet: Bridging GANs with anatomical insight for fetal ultrasound brain plane synthesis. Comput Med Imaging Graph 2024; 116:102405. [PMID: 38824716 DOI: 10.1016/j.compmedimag.2024.102405] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Revised: 04/25/2024] [Accepted: 05/22/2024] [Indexed: 06/04/2024]
Abstract
Over the past decade, deep-learning (DL) algorithms have become a promising tool to aid clinicians in identifying fetal head standard planes (FHSPs) during ultrasound (US) examination. However, the adoption of these algorithms in clinical settings is still hindered by the lack of large annotated datasets. To overcome this barrier, we introduce FetalBrainAwareNet, an innovative framework designed to synthesize anatomically accurate images of FHSPs. FetalBrainAwareNet introduces a cutting-edge approach that utilizes class activation maps as a prior in its conditional adversarial training process. This approach fosters the presence of the specific anatomical landmarks in the synthesized images. Additionally, we investigate specialized regularization terms within the adversarial training loss function to control the morphology of the fetal skull and foster the differentiation between the standard planes, ensuring that the synthetic images faithfully represent real US scans in both structure and overall appearance. The versatility of our FetalBrainAwareNet framework is highlighted by its ability to generate high-quality images of three predominant FHSPs using a singular, integrated framework. Quantitative (Fréchet inception distance of 88.52) and qualitative (t-SNE) results suggest that our framework generates US images with greater variability compared to state-of-the-art methods. By using the synthetic images generated with our framework, we increase the accuracy of FHSP classifiers by 3.2% compared to training the same classifiers solely with real acquisitions. These achievements suggest that using our synthetic images to increase the training set could provide benefits to enhance the performance of DL algorithms for FHSPs classification that could be integrated in real clinical scenarios.
Collapse
Affiliation(s)
- Angelo Lasala
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Pisa, Italy.
| | | | - Andrea Bandini
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Pisa, Italy; Health Science Interdisciplinary Research Center, Scuola Superiore Sant'Anna, Pisa, Italy
| | - Sara Moccia
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Pisa, Italy
| |
Collapse
|
4
|
Abbasi S, Lan H, Choupan J, Sheikh-Bahaei N, Pandey G, Varghese B. Deep learning for the harmonization of structural MRI scans: a survey. Biomed Eng Online 2024; 23:90. [PMID: 39217355 PMCID: PMC11365220 DOI: 10.1186/s12938-024-01280-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2024] [Accepted: 08/06/2024] [Indexed: 09/04/2024] Open
Abstract
Medical imaging datasets for research are frequently collected from multiple imaging centers using different scanners, protocols, and settings. These variations affect data consistency and compatibility across different sources. Image harmonization is a critical step to mitigate the effects of factors like inherent differences between various vendors, hardware upgrades, protocol changes, and scanner calibration drift, as well as to ensure consistent data for medical image processing techniques. Given the critical importance and widespread relevance of this issue, a vast array of image harmonization methodologies have emerged, with deep learning-based approaches driving substantial advancements in recent times. The goal of this review paper is to examine the latest deep learning techniques employed for image harmonization by analyzing cutting-edge architectural approaches in the field of medical image harmonization, evaluating both their strengths and limitations. This paper begins by providing a comprehensive fundamental overview of image harmonization strategies, covering three critical aspects: established imaging datasets, commonly used evaluation metrics, and characteristics of different scanners. Subsequently, this paper analyzes recent structural MRI (Magnetic Resonance Imaging) harmonization techniques based on network architecture, network learning algorithm, network supervision strategy, and network output. The underlying architectures include U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based generative models, transformer-based approaches, as well as custom-designed network architectures. This paper investigates the effectiveness of Disentangled Representation Learning (DRL) as a pivotal learning algorithm in harmonization. Lastly, the review highlights the primary limitations in harmonization techniques, specifically the lack of comprehensive quantitative comparisons across different methods. The overall aim of this review is to serve as a guide for researchers and practitioners to select appropriate architectures based on their specific conditions and requirements. It also aims to foster discussions around ongoing challenges in the field and shed light on promising future research directions with the potential for significant advancements.
Collapse
Affiliation(s)
- Soolmaz Abbasi
- Department of Computer Engineering, Yazd University, Yazd, Iran
| | - Haoyu Lan
- Department of Neurology, University of Southern California, Los Angeles, CA, USA
| | - Jeiran Choupan
- Department of Neurology, University of Southern California, Los Angeles, CA, USA
| | - Nasim Sheikh-Bahaei
- Department of Radiology, University of Southern California, Los Angeles, CA, USA
| | - Gaurav Pandey
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Bino Varghese
- Department of Radiology, University of Southern California, Los Angeles, CA, USA.
| |
Collapse
|
5
|
Chen X, Qiu RLJ, Peng J, Shelton JW, Chang CW, Yang X, Kesarwala AH. CBCT-based synthetic CT image generation using a diffusion model for CBCT-guided lung radiotherapy. Med Phys 2024. [PMID: 39088750 DOI: 10.1002/mp.17328] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Revised: 07/01/2024] [Accepted: 07/04/2024] [Indexed: 08/03/2024] Open
Abstract
BACKGROUND Although cone beam computed tomography (CBCT) has lower resolution compared to planning CTs (pCT), its lower dose, higher high-contrast resolution, and shorter scanning time support its widespread use in clinical applications, especially in ensuring accurate patient positioning during the image-guided radiation therapy (IGRT) process. PURPOSE While CBCT is critical to IGRT, CBCT image quality can be compromised by severe stripe and scattering artifacts. Tumor movement secondary to respiratory motion also decreases CBCT resolution. In order to improve the image quality of CBCT, we propose a Lung Diffusion Model (L-DM) framework. METHODS Our proposed algorithm is based on a conditional diffusion model trained on pCT and deformed CBCT (dCBCT) image pairs to synthesize lung CT images from dCBCT images and benefit CBCT-based radiotherapy. dCBCT images were used as the constraint for the L-DM. The image quality and Hounsfield unit (HU) values of the synthetic CTs (sCT) images generated by the proposed L-DM were compared to three selected mainstream generation models. RESULTS We verified our model in both an institutional lung cancer dataset and a selected public dataset. Our L-DM showed significant improvement in the four metrics of mean absolute error (MAE), peak signal-to-noise ratio (PSNR), normalized cross-correlation (NCC), and structural similarity index measure (SSIM). In our institutional dataset, our proposed L-DM decreased the MAE from 101.47 to 37.87 HU and increased the PSNR from 24.97 to 29.89 dB, the NCC from 0.81 to 0.97, and the SSIM from 0.80 to 0.93. In the public dataset, our proposed L-DM decreased the MAE from 173.65 to 58.95 HU, while increasing the PSNR, NCC, and SSIM from 13.07 to 24.05 dB, 0.68 to 0.94, and 0.41 to 0.88, respectively. CONCLUSIONS The proposed L-DM significantly improved sCT image quality compared to the pre-correction CBCT and three mainstream generative models. Our model can benefit CBCT-based IGRT and other potential clinical applications as it increases the HU accuracy and decreases the artifacts from input CBCT images.
Collapse
Affiliation(s)
- Xiaoqian Chen
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Richard L J Qiu
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Junbo Peng
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Joseph W Shelton
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Chih-Wei Chang
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Aparna H Kesarwala
- Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine, Atlanta, Georgia, USA
| |
Collapse
|
6
|
Sinha A, Kawahara J, Pakzad A, Abhishek K, Ruthven M, Ghorbel E, Kacem A, Aouada D, Hamarneh G. DermSynth3D: Synthesis of in-the-wild annotated dermatology images. Med Image Anal 2024; 95:103145. [PMID: 38615432 DOI: 10.1016/j.media.2024.103145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 02/11/2024] [Accepted: 03/18/2024] [Indexed: 04/16/2024]
Abstract
In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermatological images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.
Collapse
Affiliation(s)
- Ashish Sinha
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Jeremy Kawahara
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Arezou Pakzad
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Kumar Abhishek
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Matthieu Ruthven
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg
| | - Enjie Ghorbel
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg; Cristal Laboratory, National School of Computer Sciences, University of Manouba, 2010, Tunisia
| | - Anis Kacem
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg
| | - Djamila Aouada
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg
| | - Ghassan Hamarneh
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada.
| |
Collapse
|
7
|
Sun L, Yu J, Yao J, Cao Y, Sun N, Chen K, Lin Y, Ji C, Zhang J, Ling C, Yang Z, Pan Q, Yang R, Yang X, Ni D, Yin L, Deng X. A novel artificial intelligence model for measuring fetal intracranial markers during the first trimester based on two-dimensional ultrasound image. Int J Gynaecol Obstet 2024. [PMID: 38944698 DOI: 10.1002/ijgo.15762] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2023] [Revised: 05/30/2024] [Accepted: 06/17/2024] [Indexed: 07/01/2024]
Abstract
OBJECTIVE To establish reference ranges of fetal intracranial markers during the first trimester and develop the first novel artificial intelligence (AI) model to measure key markers automatically. METHODS This retrospective study used two-dimensional (2D) ultrasound images from 4233 singleton normal fetuses scanned at 11+0-13+6 weeks of gestation at the Affiliated Suzhou Hospital of Nanjing Medical University from January 2018 to July 2022. We analyzed 10 key markers in three important planes of the fetal head. Based on these, reference ranges of 10 fetal intracranial markers were established and an AI model was developed for automated marker measurement. AI and manual measurements were compared to evaluate differences, correlations, consistency, and time consumption based on mean error, Pearson correlation analysis, intraclass correlation coefficients (ICCs), and average measurement time. RESULTS The results of AI and manual methods had strong consistency and correlation (all ICC values >0.75, all r values >0.75, and all P values <0.001). The average absolute error of both only ranged from 0.124 to 0.178 mm. AI achieved a 100% detection rate for abnormal cases. Additionally, the average measurement time of AI was only 0.49 s, which was more than 65 times faster than the manual measurement method. CONCLUSION The present study first established the normal standard reference ranges of fetal intracranial markers based on a large Chinese population data set. Furthermore, the proposed AI model demonstrated its capability to measure multiple fetal intracranial markers automatically, serving as a highly effective tool to streamline sonographer tasks and mitigate manual measurement errors, which can be generalized to first-trimester scanning.
Collapse
Affiliation(s)
- Lingling Sun
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Jiezhi Yao
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Yan Cao
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Naimin Sun
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Keqi Chen
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Yujia Lin
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Chunya Ji
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Jun Zhang
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Chen Ling
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Zhong Yang
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Qi Pan
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Ronghao Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Linliang Yin
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Xuedong Deng
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| |
Collapse
|
8
|
Kreitner L, Paetzold JC, Rauch N, Chen C, Hagag AM, Fayed AE, Sivaprasad S, Rausch S, Weichsel J, Menze BH, Harders M, Knier B, Rueckert D, Menten MJ. Synthetic Optical Coherence Tomography Angiographs for Detailed Retinal Vessel Segmentation Without Human Annotations. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2061-2073. [PMID: 38224512 DOI: 10.1109/tmi.2024.3354408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/17/2024]
Abstract
Optical coherence tomography angiography (OCTA) is a non-invasive imaging modality that can acquire high-resolution volumes of the retinal vasculature and aid the diagnosis of ocular, neurological and cardiac diseases. Segmenting the visible blood vessels is a common first step when extracting quantitative biomarkers from these images. Classical segmentation algorithms based on thresholding are strongly affected by image artifacts and limited signal-to-noise ratio. The use of modern, deep learning-based segmentation methods has been inhibited by a lack of large datasets with detailed annotations of the blood vessels. To address this issue, recent work has employed transfer learning, where a segmentation network is trained on synthetic OCTA images and is then applied to real data. However, the previously proposed simulations fail to faithfully model the retinal vasculature and do not provide effective domain adaptation. Because of this, current methods are unable to fully segment the retinal vasculature, in particular the smallest capillaries. In this work, we present a lightweight simulation of the retinal vascular network based on space colonization for faster and more realistic OCTA synthesis. We then introduce three contrast adaptation pipelines to decrease the domain gap between real and artificial images. We demonstrate the superior segmentation performance of our approach in extensive quantitative and qualitative experiments on three public datasets that compare our method to traditional computer vision algorithms and supervised training using human annotations. Finally, we make our entire pipeline publicly available, including the source code, pretrained models, and a large dataset of synthetic OCTA images.
Collapse
|
9
|
Yang X, Huang X, Wei C, Yu J, Yu X, Dong C, Chen J, Chen R, Wu X, Yu Z, Sun B, Wang J, Liu H, Han W, Sun B, Jiang Z, Ding J, Liu Z, Peng J, Ni D, Deng X, Liu L, Gou Z. An intelligent quantification system for fetal heart rhythm assessment: A multicenter prospective study. Heart Rhythm 2024; 21:600-609. [PMID: 38266752 DOI: 10.1016/j.hrthm.2024.01.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 01/04/2024] [Accepted: 01/15/2024] [Indexed: 01/26/2024]
Abstract
BACKGROUND The motion relationship and time intervals of the pulsed-wave Doppler (PWD) spectrum are essential for diagnosing fetal arrhythmia. However, few technologies currently are available to automatically calculate fetal cardiac time intervals (CTIs). OBJECTIVE The purpose of this study was to develop a fetal heart rhythm intelligent quantification system (HR-IQS) for the automatic extraction of CTIs and establish the normal reference range for fetal CTIs. METHODS A total of 6498 PWD spectrums of 2630 fetuses over the junction between the left ventricular inflow and outflow tracts were recorded across 14 centers. E, A, and V waves were manually labeled by 3 experienced fetal cardiologists, with 17 CTIs extracted. Five-fold cross-validation was performed for training and testing of the deep learning model. Agreement between the manual and HR-IQS-based values was evaluated using the intraclass correlation coefficient and Spearman's rank correlation coefficient. The Jarque-Bera test was applied to evaluate the normality of CTIs' distributions, and the normal reference range of 17 CTIs was established with quantile regression. Arrhythmia subset was compared with the non-arrhythmia subset using the Mann-Whitney U test. RESULTS Significant positive correlation (P <.001) and moderate-to-excellent consistency (P <.001) between the manual and HR-IQS automated measurements of CTIs was found. The distribution of CTIs was non-normal (P <.001). The normal range (2.5th to 97.5th percentiles) was successfully established for the 17 CTIs. CONCLUSIONS Using our HR-IQS is feasible for the automated calculation of CTIs in practice and thus could provide a promising tool for the assessment of fetal rhythm and function.
Collapse
Affiliation(s)
- Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Xiaoqiong Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Chenchen Wei
- Center for Cardiovascular Disease, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou, Jiangsu, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Xuejuan Yu
- Department of Ultrasonography, Suzhou Xiangcheng People's Hospital, Suzhou, Jiangsu, China
| | - Caixia Dong
- Department of Ultrasonography, Wulin Hospital, Hangzhou, Zhejiang, China
| | - Ju Chen
- Department of Ultrasonography, Taicang First People's Hospital, Suzhou, Jiangsu, China
| | - Ruifeng Chen
- Department of Ultrasound Diagnosis, The Affiliated Changsha Central Hospital, Hengyang Medical School, University of South China, Changsha, Hunan, China
| | - Xiafang Wu
- Department of Ultrasonography, Taizhou Hospital of Zhejiang Province Affiliated to Wenzhou Medical University, Linhai, Zhejiang, China
| | - Zhuan Yu
- Department of Ultrasonography, The Affiliated Hospital of Jiangsu University, Zhenjiang, Jiangsu, China
| | - Baojuan Sun
- Department of Ultrasonography, Huai'an Maternal and Child Health Hospital, Huai'an, Jiangsu, China
| | - Junli Wang
- Department of Ultrasonography, Wuhu No.2 People's Hospital, Wuhu, Anhui, China
| | - Hongmei Liu
- Department of Ultrasonography, Panzhou Emerging Hospital, Panzhou, Guizhou, China
| | - Wen Han
- Department of Ultrasonography, Suzhou Gaoxin District People's Hospital, Suzhou, Jiangsu, China
| | - Biyun Sun
- Department of Ultrasonography, The Affiliated Yijishan Hospital of Wannan Medical University, Wuhu, Anhui, China
| | - Zhiyong Jiang
- Department of Ultrasonography, The Huaren Hospital, Wuhu, Zhejiang, China
| | - Jie Ding
- Department of Ultrasonography, The Affiliated Suzhou Hospital of Nanjing University, Suzhou, Jiangsu, China
| | - Zhe Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Jin Peng
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China
| | - Xuedong Deng
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou, Jiangsu, China
| | - Lian Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China.
| | - Zhongshan Gou
- Center for Cardiovascular Disease, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou, Jiangsu, China.
| |
Collapse
|
10
|
Zhang L, Wu X, Zhang J, Liu Z, Fan Y, Zheng L, Liu P, Song H, Lyu G. SEG-LUS: A novel ultrasound segmentation method for liver and its accessory structures based on muti-head self-attention. Comput Med Imaging Graph 2024; 113:102338. [PMID: 38290353 DOI: 10.1016/j.compmedimag.2024.102338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Revised: 12/13/2023] [Accepted: 01/14/2024] [Indexed: 02/01/2024]
Abstract
Although liver ultrasound (US) is quick and convenient, it presents challenges due to patient variations. Previous research has predominantly focused on computer-aided diagnosis (CAD), particularly for disease analysis. However, characterizing liver US images is complex due to structural diversity and a limited number of samples. Normal liver US images are crucial, especially for standard section diagnosis. This study explicitly addresses Liver US standard sections (LUSS) and involves detailed labeling of eight anatomical structures. We propose SEG-LUS, a US image segmentation model for the liver and its accessory structures. In SEG-LUS, we have adopted the shifted windows feature encoder combined with the cross-attention mechanism to adapt to capturing image information at different scales and resolutions and address context mismatch and sample imbalance in the segmentation task. By introducing the UUF module, we achieve the perfect fusion of shallow and deep information, making the information retained by the network in the feature extraction process more comprehensive. We have improved the Focal Loss to tackle the imbalance of pixel-level distribution. The results show that the SEG-LUS model exhibits significant performance improvement, with mPA, mDice, mIOU, and mASD reaching 85.05%, 82.60%, 74.92%, and 0.31, respectively. Compared with seven state-of-the-art semantic segmentation methods, the mPA improves by 5.32%. SEG-LUS is positioned to serve as a crucial reference for research in computer-aided modeling using liver US images, thereby advancing the field of US medicine research.
Collapse
Affiliation(s)
- Lei Zhang
- College of Physics and Electronic Engineering, Northwest Normal University, Lanzhou 730070, China
| | - Xiuming Wu
- Department of Ultrasound, Quanzhou First Hospital Affiliated to Fujian Medical University, Quanzhou 362000, China
| | - Jiansong Zhang
- College of Medicine, Huaqiao University, Quanzhou 362021, China
| | - Zhonghua Liu
- Department of Ultrasound, Quanzhou First Hospital Affiliated to Fujian Medical University, Quanzhou 362000, China
| | - Yuling Fan
- College of Engineering, Huaqiao University, Quanzhou 362021, China
| | - Lan Zheng
- College of Engineering, Huaqiao University, Quanzhou 362021, China
| | - Peizhong Liu
- College of Medicine, Huaqiao University, Quanzhou 362021, China; College of Engineering, Huaqiao University, Quanzhou 362021, China; Collaborative Innovation Center for Maternal and Infant Health Service Application Technology, Quanzhou Medical College, Quanzhou 362011, China.
| | - Haisheng Song
- College of Physics and Electronic Engineering, Northwest Normal University, Lanzhou 730070, China.
| | - Guorong Lyu
- Collaborative Innovation Center for Maternal and Infant Health Service Application Technology, Quanzhou Medical College, Quanzhou 362011, China; Department of Ultrasound, The Second Hospital of Quanzhou Affiliated to Fujian Medical University, Quanzhou, China.
| |
Collapse
|
11
|
Wang G, Zhou M, Ning X, Tiwari P, Zhu H, Yang G, Yap CH. US2Mask: Image-to-mask generation learning via a conditional GAN for cardiac ultrasound image segmentation. Comput Biol Med 2024; 172:108282. [PMID: 38503085 DOI: 10.1016/j.compbiomed.2024.108282] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2024] [Revised: 02/29/2024] [Accepted: 03/12/2024] [Indexed: 03/21/2024]
Abstract
Cardiac ultrasound (US) image segmentation is vital for evaluating clinical indices, but it often demands a large dataset and expert annotations, resulting in high costs for deep learning algorithms. To address this, our study presents a framework utilizing artificial intelligence generation technology to produce multi-class RGB masks for cardiac US image segmentation. The proposed approach directly performs semantic segmentation of the heart's main structures in US images from various scanning modes. Additionally, we introduce a novel learning approach based on conditional generative adversarial networks (CGAN) for cardiac US image segmentation, incorporating a conditional input and paired RGB masks. Experimental results from three cardiac US image datasets with diverse scan modes demonstrate that our approach outperforms several state-of-the-art models, showcasing improvements in five commonly used segmentation metrics, with lower noise sensitivity. Source code is available at https://github.com/energy588/US2mask.
Collapse
Affiliation(s)
- Gang Wang
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, Chongqing; Department of Bioengineering, Imperial College London, London, UK
| | - Mingliang Zhou
- School of Computer Science, Chongqing University, Chongqing, Chongqing.
| | - Xin Ning
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China
| | - Prayag Tiwari
- School of Information Technology, Halmstad University, Halmstad, Sweden
| | | | - Guang Yang
- Department of Bioengineering, Imperial College London, London, UK; Cardiovascular Research Centre, Royal Brompton Hospital, London, UK; National Heart and Lung Institute, Imperial College London, London, UK
| | - Choon Hwai Yap
- Department of Bioengineering, Imperial College London, London, UK
| |
Collapse
|
12
|
Zhang Y, Shi Z, Wang H, Cui S, Zhang L, Liu J, Shan X, Liu Y, Fang L. LumVertCancNet: A novel 3D lumbar vertebral body cancellous bone location and segmentation method based on hybrid Swin-transformer. Comput Biol Med 2024; 171:108237. [PMID: 38422966 DOI: 10.1016/j.compbiomed.2024.108237] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 01/12/2024] [Accepted: 02/25/2024] [Indexed: 03/02/2024]
Abstract
Lumbar vertebral body cancellous bone location and segmentation is crucial in an automated lumbar spine processing pipeline. Accurate and reliable analysis of lumbar spine image is expected to advantage practical medical diagnosis and population-based analysis of bone strength. However, the design of automated algorithms for lumbar spine processing is demanding due to significant anatomical variations and scarcity of publicly available data. In recent years, convolutional neural network (CNN) and vision transformers (Vits) have been the de facto standard in medical image segmentation. Although adept at capturing global features, the inherent bias of locality and weight sharing of CNN constrains its capacity to model long-range dependency. In contrast, Vits excel at long-range dependency modeling, but they may not generalize well with limited datasets due to the lack of inductive biases inherent to CNN. In this paper, we propose a deep learning-based two-stage coarse-to-fine solution to address the problem of automatic location and segmentation of lumbar vertebral body cancellous bone. Specifically, in the first stage, a Swin-transformer based model is applied to predict the heatmap of lumbar vertebral body centroids. Considering the characteristic anatomical structure of lumbar spine, we propose a novel loss function called LumAnatomy loss, which enforces the order and bend of the predicted vertebral body centroids. To inherit the excellence of CNN and Vits while preventing their respective limitations, in the second stage, we propose an encoder-decoder network to segment the identified lumbar vertebral body cancellous bone, which consists of two parallel encoders, i.e., a Swin-transformer encoder and a CNN encoder. To enhance the combination of CNNs and Vits, we propose a novel multi-scale attention feature fusion module (MSA-FFM), which address issues that arise when fusing features given at different encoders. To tackle the issue of lack of data, we raise the first large-scale lumbar vertebral body cancellous bone segmentation dataset called LumVBCanSeg containing a total of 185 CT scans annotated at voxel level by 3 physicians. Extensive experimental results on the LumVBCanSeg dataset demonstrate the proposed algorithm outperform other state-of-the-art medical image segmentation methods. The data is publicly available at: https://zenodo.org/record/8181250. The implementation of the proposed method is available at: https://github.com/sia405yd/LumVertCancNet.
Collapse
Affiliation(s)
- Yingdi Zhang
- Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences, Shenyang 110016, China; Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China; University of Chinese Academy of Sciences, Beijing 100049, China.
| | - Zelin Shi
- Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences, Shenyang 110016, China; Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China; University of Chinese Academy of Sciences, Beijing 100049, China
| | - Huan Wang
- Spine Surgery Department, ShengJing Hospital of China Medical University, Shenyang, China
| | - Shaoqian Cui
- Spine Surgery Department, ShengJing Hospital of China Medical University, Shenyang, China
| | - Lei Zhang
- Spine Surgery Department, ShengJing Hospital of China Medical University, Shenyang, China
| | - Jiachen Liu
- Spine Surgery Department, ShengJing Hospital of China Medical University, Shenyang, China
| | - Xiuqi Shan
- Spine Surgery Department, ShengJing Hospital of China Medical University, Shenyang, China
| | - Yunpeng Liu
- Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences, Shenyang 110016, China; Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China; University of Chinese Academy of Sciences, Beijing 100049, China
| | - Lei Fang
- Key Laboratory of Opto-Electronic Information Processing, Chinese Academy of Sciences, Shenyang 110016, China; Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China
| |
Collapse
|
13
|
Han X, Yu J, Yang X, Chen C, Zhou H, Qiu C, Cao Y, Zhang T, Peng M, Zhu G, Ni D, Zhang Y, Liu N. Artificial intelligence assistance for fetal development: evaluation of an automated software for biometry measurements in the mid-trimester. BMC Pregnancy Childbirth 2024; 24:158. [PMID: 38395822 PMCID: PMC10885506 DOI: 10.1186/s12884-024-06336-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Accepted: 02/08/2024] [Indexed: 02/25/2024] Open
Abstract
BACKGROUND This study presents CUPID, an advanced automated measurement software based on Artificial Intelligence (AI), designed to evaluate nine fetal biometric parameters in the mid-trimester. Our primary objective was to assess and compare the CUPID performance of experienced senior and junior radiologists. MATERIALS AND METHODS This prospective cross-sectional study was conducted at Shenzhen University General Hospital between September 2022 and June 2023, and focused on mid-trimester fetuses. All ultrasound images of the six standard planes, that enabled the evaluation of nine biometric measurements, were included to compare the performance of CUPID through subjective and objective assessments. RESULTS There were 642 fetuses with a mean (±SD) age of 22 ± 2.82 weeks at enrollment. In the subjective quality assessment, out of 642 images representing nine biometric measurements, 617-635 images (90.65-96.11%) of CUPID caliper placements were determined to be accurately placed and did not require any adjustments. Whereas, for the junior category, 447-691 images (69.63-92.06%) were determined to be accurately placed and did not require any adjustments. In the objective measurement indicators, across all nine biometric parameters and estimated fetal weight (EFW), the intra-class correlation coefficients (ICC) (0.843-0.990) and Pearson correlation coefficients (PCC) (0.765-0.978) between the senior radiologist and CUPID reflected good reliability compared with the ICC (0.306-0.937) and PCC (0.566-0.947) between the senior and junior radiologists. Additionally, the mean absolute error (MAE), percentage error (PE), and average error in days of gestation were lower between the senior and CUPID compared to the difference between the senior and junior radiologists. The specific differences are as follows: MAE (0.36-2.53 mm, 14.67 g) compared to (0.64- 8.13 mm, 38.05 g), PE (0.94-9.38%) compared to (1.58-16.04%), and average error in days (3.99-7.92 days) compared to (4.35-11.06 days). In the time-consuming task, CUPID only takes 0.05-0.07 s to measure nine biometric parameters, while senior and junior radiologists require 4.79-11.68 s and 4.95-13.44 s, respectively. CONCLUSIONS CUPID has proven to be highly accurate and efficient software for automatically measuring fetal biometry, gestational age, and fetal weight, providing a precise and fast tool for assessing fetal growth and development.
Collapse
Affiliation(s)
- Xuesong Han
- Department of Ultrasonography, Shenzhen University General Hospital, Shenzhen, Guangdong, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China
| | - Chaoyu Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China
| | - Han Zhou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China
| | - Chuangxin Qiu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China
| | - Yan Cao
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | | | | | - Guiyao Zhu
- Department of Ultrasonography, Shenzhen University General Hospital, Shenzhen, Guangdong, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China
| | - Yuanji Zhang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, Guangdong, China.
- Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, Guangdong, China.
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, Guangdong, China.
| | - Nana Liu
- Department of Ultrasonography, Shenzhen University General Hospital, Shenzhen, Guangdong, China.
| |
Collapse
|
14
|
Huang Y, Yang X, Liu L, Zhou H, Chang A, Zhou X, Chen R, Yu J, Chen J, Chen C, Liu S, Chi H, Hu X, Yue K, Li L, Grau V, Fan DP, Dong F, Ni D. Segment anything model for medical images? Med Image Anal 2024; 92:103061. [PMID: 38086235 DOI: 10.1016/j.media.2023.103061] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2023] [Revised: 09/28/2023] [Accepted: 12/05/2023] [Indexed: 01/12/2024]
Abstract
The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: (1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. (2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. (3) SAM performed better with manual hints, especially box, than the Everything mode. (4) SAM could help human annotation with high labeling quality and less time. (5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. (6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. (7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. (8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. Codes and models are available at: https://github.com/yuhoo0302/Segment-Anything-Model-for-Medical-Images. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.
Collapse
Affiliation(s)
- Yuhao Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Lian Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Han Zhou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Ao Chang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Xinrui Zhou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Rusi Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Jiongquan Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Chaoyu Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Sijing Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | | | - Xindi Hu
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, China
| | - Kejuan Yue
- Hunan First Normal University, Changsha, China
| | - Lei Li
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Vicente Grau
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Deng-Ping Fan
- Computer Vision Lab (CVL), ETH Zurich, Zurich, Switzerland
| | - Fajin Dong
- Ultrasound Department, the Second Clinical Medical College, Jinan University, China; First Affiliated Hospital, Southern University of Science and Technology, Shenzhen People's Hospital, Shenzhen, China.
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China.
| |
Collapse
|
15
|
Xie J, Yang Y, Jiang Z, Zhang K, Zhang X, Lin Y, Shen Y, Jia X, Liu H, Yang S, Jiang Y, Ma L. MRI radiomics-based decision support tool for a personalized classification of cervical disc degeneration: a two-center study. Front Physiol 2024; 14:1281506. [PMID: 38235385 PMCID: PMC10791783 DOI: 10.3389/fphys.2023.1281506] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Accepted: 11/24/2023] [Indexed: 01/19/2024] Open
Abstract
Objectives: To develop and validate an MRI radiomics-based decision support tool for the automated grading of cervical disc degeneration. Methods: The retrospective study included 2,610 cervical disc samples of 435 patients from two hospitals. The cervical magnetic resonance imaging (MRI) analysis of patients confirmed cervical disc degeneration grades using the Pfirrmann grading system. A training set (1,830 samples of 305 patients) and an independent test set (780 samples of 130 patients) were divided for the construction and validation of the machine learning model, respectively. We provided a fine-tuned MedSAM model for automated cervical disc segmentation. Then, we extracted 924 radiomic features from each segmented disc in T1 and T2 MRI modalities. All features were processed and selected using minimum redundancy maximum relevance (mRMR) and multiple machine learning algorithms. Meanwhile, the radiomics models of various machine learning algorithms and MRI images were constructed and compared. Finally, the combined radiomics model was constructed in the training set and validated in the test set. Radiomic feature mapping was provided for auxiliary diagnosis. Results: Of the 2,610 cervical disc samples, 794 (30.4%) were classified as low grade and 1,816 (69.6%) were classified as high grade. The fine-tuned MedSAM model achieved good segmentation performance, with the mean Dice coefficient of 0.93. Higher-order texture features contributed to the dominant force in the diagnostic task (80%). Among various machine learning models, random forest performed better than the other algorithms (p < 0.01), and the T2 MRI radiomics model showed better results than T1 MRI in the diagnostic performance (p < 0.05). The final combined radiomics model had an area under the receiver operating characteristic curve (AUC) of 0.95, an accuracy of 89.51%, a precision of 87.07%, a recall of 98.83%, and an F1 score of 0.93 in the test set, which were all better than those of other models (p < 0.05). Conclusion: The radiomics-based decision support tool using T1 and T2 MRI modalities can be used for cervical disc degeneration grading, facilitating individualized management.
Collapse
Affiliation(s)
- Jun Xie
- Information Technology Center, West China Hospital of Sichuan University, Chengdu, China
- Information Technology Center, Sanya People’s Hospital, Sanya, China
| | - Yi Yang
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| | - Zekun Jiang
- College of Computer Science, Sichuan University, Chengdu, Sichuan, China
- West China Biomedical Big Data Center, Sichuan University, Chengdu, Sichuan, China
| | - Kerui Zhang
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| | - Xiang Zhang
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| | - Yuheng Lin
- West China Biomedical Big Data Center, Sichuan University, Chengdu, Sichuan, China
| | - Yiwei Shen
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| | - Xuehai Jia
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| | - Hao Liu
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| | - Shaofen Yang
- Cadre Health Section, Hezhou People’s Hospital, Hezhou, Guangxi, China
| | - Yang Jiang
- Department of Orthopedic Spine, The Second Affiliated Hospital of Chengdu Medical College (China National Nuclear Corporation 416 Hospital), Chengdu, Sichuan, China
| | - Litai Ma
- Department of Orthopedics, Orthopedic Research Institute, West China Hospital, Sichuan University, Chengdu, Sichuan, China
| |
Collapse
|
16
|
Mendez M, Sundararaman S, Probyn L, Tyrrell PN. Approaches and Limitations of Machine Learning for Synthetic Ultrasound Generation: A Scoping Review. JOURNAL OF ULTRASOUND IN MEDICINE : OFFICIAL JOURNAL OF THE AMERICAN INSTITUTE OF ULTRASOUND IN MEDICINE 2023; 42:2695-2706. [PMID: 37772474 DOI: 10.1002/jum.16332] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 08/22/2023] [Accepted: 08/27/2023] [Indexed: 09/30/2023]
Abstract
This scoping review examines the emerging field of synthetic ultrasound generation using machine learning (ML) models in radiology. Nineteen studies were analyzed, revealing three primary methodological strategies: unconditional generation, conditional generation, and domain translation. Synthetic ultrasound is mainly used to augment training datasets and as training material for radiologists. Blind expert assessment and Fréchet Inception Distance are common evaluation methods. Current limitations include the need for large training datasets, manual annotations for controllable generation, and insufficient research on incorporating new domain knowledge. While generative ultrasound models show promise, further work is required for clinical implementation.
Collapse
Affiliation(s)
- Mauro Mendez
- Institute of Medical Science, University of Toronto, Toronto, Ontario, Canada
- Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada
| | | | - Linda Probyn
- Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada
| | - Pascal N Tyrrell
- Institute of Medical Science, University of Toronto, Toronto, Ontario, Canada
- Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada
- Department of Statistical Sciences, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
17
|
Zhang H, Meng Z, Ru J, Meng Y, Wang K. Application and prospects of AI-based radiomics in ultrasound diagnosis. Vis Comput Ind Biomed Art 2023; 6:20. [PMID: 37828411 PMCID: PMC10570254 DOI: 10.1186/s42492-023-00147-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Accepted: 09/20/2023] [Indexed: 10/14/2023] Open
Abstract
Artificial intelligence (AI)-based radiomics has attracted considerable research attention in the field of medical imaging, including ultrasound diagnosis. Ultrasound imaging has unique advantages such as high temporal resolution, low cost, and no radiation exposure. This renders it a preferred imaging modality for several clinical scenarios. This review includes a detailed introduction to imaging modalities, including Brightness-mode ultrasound, color Doppler flow imaging, ultrasound elastography, contrast-enhanced ultrasound, and multi-modal fusion analysis. It provides an overview of the current status and prospects of AI-based radiomics in ultrasound diagnosis, highlighting the application of AI-based radiomics to static ultrasound images, dynamic ultrasound videos, and multi-modal ultrasound fusion analysis.
Collapse
Affiliation(s)
- Haoyan Zhang
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China
| | - Zheling Meng
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China
| | - Jinyu Ru
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China
| | - Yaqing Meng
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China
| | - Kun Wang
- CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China.
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China.
| |
Collapse
|
18
|
Ji C, Liu K, Yang X, Cao Y, Cao X, Pan Q, Yang Z, Sun L, Yin L, Deng X, Ni D. A novel artificial intelligence model for fetal facial profile marker measurement during the first trimester. BMC Pregnancy Childbirth 2023; 23:718. [PMID: 37817098 PMCID: PMC10563312 DOI: 10.1186/s12884-023-06046-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2023] [Accepted: 10/03/2023] [Indexed: 10/12/2023] Open
Abstract
BACKGROUND To study the validity of an artificial intelligence (AI) model for measuring fetal facial profile markers, and to evaluate the clinical value of the AI model for identifying fetal abnormalities during the first trimester. METHODS This retrospective study used two-dimensional mid-sagittal fetal profile images taken during singleton pregnancies at 11-13+ 6 weeks of gestation. We measured the facial profile markers, including inferior facial angle (IFA), maxilla-nasion-mandible (MNM) angle, facial-maxillary angle (FMA), frontal space (FS) distance, and profile line (PL) distance using AI and manual measurements. Semantic segmentation and landmark localization were used to develop an AI model to measure the selected markers and evaluate the diagnostic value for fetal abnormalities. The consistency between AI and manual measurements was compared using intraclass correlation coefficients (ICC). The diagnostic value of facial markers measured using the AI model during fetal abnormality screening was evaluated using receiver operating characteristic (ROC) curves. RESULTS A total of 2372 normal fetuses and 37 with abnormalities were observed, including 18 with trisomy 21, 7 with trisomy 18, and 12 with CLP. Among them, 1872 normal fetuses were used for AI model training and validation, and the remaining 500 normal fetuses and all fetuses with abnormalities were used for clinical testing. The ICCs (95%CI) of the IFA, MNM angle, FMA, FS distance, and PL distance between the AI and manual measurement for the 500 normal fetuses were 0.812 (0.780-0.840), 0.760 (0.720-0.795), 0.766 (0.727-0.800), 0.807 (0.775-0.836), and 0.798 (0.764-0.828), respectively. IFA clinically significantly identified trisomy 21 and trisomy 18, with areas under the ROC curve (AUC) of 0.686 (95%CI, 0.585-0.788) and 0.729 (95%CI, 0.621-0.837), respectively. FMA effectively predicted trisomy 18, with an AUC of 0.904 (95%CI, 0.842-0.966). MNM angle and FS distance exhibited good predictive value in CLP, with AUCs of 0.738 (95%CI, 0.573-0.902) and 0.677 (95%CI, 0.494-0.859), respectively. CONCLUSIONS The consistency of fetal facial profile marker measurements between the AI and manual measurement was good during the first trimester. The AI model is a convenient and effective tool for the early screen for fetal trisomy 21, trisomy 18, and CLP, which can be generalized to first-trimester scanning (FTS).
Collapse
Affiliation(s)
- Chunya Ji
- Center for Medical Ultrasound, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Kai Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Xueyuan Blvd, Nanshan, Shenzhen, Guangdong, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Xueyuan Blvd, Nanshan, Shenzhen, Guangdong, China
| | - Yan Cao
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Xueyuan Blvd, Nanshan, Shenzhen, Guangdong, China
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Xiaoju Cao
- Center for Reproduction and Genetics, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, No. 26 Daoqian Street, Suzhou, 215002, Jiangsu, China
| | - Qi Pan
- Center for Medical Ultrasound, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Zhong Yang
- Center for Medical Ultrasound, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Lingling Sun
- Center for Medical Ultrasound, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Linliang Yin
- Center for Medical Ultrasound, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, Suzhou, Jiangsu, China.
| | - Xuedong Deng
- Center for Medical Ultrasound, Suzhou Municipal Hospital, Gusu School, The Affiliated Suzhou Hospital of Nanjing Medical University, Nanjing Medical University, Suzhou, Jiangsu, China.
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Xueyuan Blvd, Nanshan, Shenzhen, Guangdong, China.
| |
Collapse
|
19
|
Chen L, Liao H, Kong W, Zhang D, Chen F. Anatomy preserving GAN for realistic simulation of intraoperative liver ultrasound images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 240:107642. [PMID: 37480644 DOI: 10.1016/j.cmpb.2023.107642] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2023] [Revised: 05/31/2023] [Accepted: 06/02/2023] [Indexed: 07/24/2023]
Abstract
In ultrasound-guided liver surgery, the lack of large-scale intraoperative ultrasound images with important anatomical structures remains an obstacle hindering the successful application of AI to ultrasound guidance. In this case, intraoperative ultrasound (iUS) simulation should be conducted from preoperative magnetic resonance (pMR), which not only helps doctors understand the characteristics of iUS in advance, but also expands the iUS dataset from various imaging positions, thereby promoting the automatic iUS analysis in ultrasound guidance. Herein, a novel anatomy preserving generative adversarial network (ApGAN) framework was proposed to generate simulated intraoperative ultrasound (Sim-iUS) of liver with precise structure information from pMR. Specifically, the low-rank factors based bimodal fusion was first established focusing on the effective information of hepatic parenchyma. Then, a deformation field based correction module was introduced to learn and correct the slight structural distortion from surgical operations. Meanwhile, the multiple loss functions were designed to constrain the simulation of the content, structures, and style. Empirical results of clinical data showed that the proposed ApGAN obtained higher Structural Similarity (SSIM) of 0.74 and Fr´echet Inception Distance (FID) of 35.54 compared to existing methods. Furthermore, the average Hausdorff Distance (HD) error of the liver capsule structure was less than 0.25 mm, and the average relative (Euclidean Distance) ED error for polyps was 0.12 mm, indicating the high-level precision of this ApGAN in simulating the anatomical structures and focal areas.
Collapse
Affiliation(s)
- Lingyu Chen
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China
| | - Wentao Kong
- Department of Ultrasound, Affiliated DrumTower Hospital, Medical School of Nanjing University, Nanjing 210008, China.
| | - Daoqiang Zhang
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Fang Chen
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China.
| |
Collapse
|
20
|
Xing X, Papanastasiou G, Walsh S, Yang G. Less Is More: Unsupervised Mask-Guided Annotated CT Image Synthesis With Minimum Manual Segmentations. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2566-2576. [PMID: 37030699 DOI: 10.1109/tmi.2023.3260169] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
As a pragmatic data augmentation tool, data synthesis has generally returned dividends in performance for deep learning based medical image analysis. However, generating corresponding segmentation masks for synthetic medical images is laborious and subjective. To obtain paired synthetic medical images and segmentations, conditional generative models that use segmentation masks as synthesis conditions were proposed. However, these segmentation mask-conditioned generative models still relied on large, varied, and labeled training datasets, and they could only provide limited constraints on human anatomical structures, leading to unrealistic image features. Moreover, the invariant pixel-level conditions could reduce the variety of synthetic lesions and thus reduce the efficacy of data augmentation. To address these issues, in this work, we propose a novel strategy for medical image synthesis, namely Unsupervised Mask (UM)-guided synthesis, to obtain both synthetic images and segmentations using limited manual segmentation labels. We first develop a superpixel based algorithm to generate unsupervised structural guidance and then design a conditional generative model to synthesize images and annotations simultaneously from those unsupervised masks in a semi-supervised multi-task setting. In addition, we devise a multi-scale multi-task Fréchet Inception Distance (MM-FID) and multi-scale multi-task standard deviation (MM-STD) to harness both fidelity and variety evaluations of synthetic CT images. With multiple analyses on different scales, we could produce stable image quality measurements with high reproducibility. Compared with the segmentation mask guided synthesis, our UM-guided synthesis provided high-quality synthetic images with significantly higher fidelity, variety, and utility ( by Wilcoxon Signed Ranked test).
Collapse
|
21
|
Zhen C, Wang H, Cheng J, Yang X, Chen C, Hu X, Zhang Y, Cao Y, Ni D, Huang W, Wang P. Locating Multiple Standard Planes in First-Trimester Ultrasound Videos via the Detection and Scoring of Key Anatomical Structures. ULTRASOUND IN MEDICINE & BIOLOGY 2023:S0301-5629(23)00163-1. [PMID: 37291008 DOI: 10.1016/j.ultrasmedbio.2023.05.005] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Revised: 04/20/2023] [Accepted: 05/10/2023] [Indexed: 06/10/2023]
Abstract
OBJECTIVE This study was aimed at developing a first-trimester standard plane detection (FTSPD) system that can automatically locate nine standard planes in ultrasound videos and investigating its utility in clinical practice. METHODS The FTSPD system, based on the YOLOv3 network, was developed to detect structures and evaluate the quality of plane images by using a pre-defined scoring system. A total of 220 videos from two different ultrasound scanners were collected to compare detection performance between our FTSPD system and sonographers with different levels of experience. The quality of the detected standard planes was quantitatively rated by an expert according to a scoring protocol. Kolmogorov-Smirnov analysis was used to compare the distributions of scores across all nine standard planes. RESULTS The expert-rated scores indicated that the quality of the standard planes detected by the FTSPD system was on par with that of the planes detected by senior sonographers. There were no significant differences in the distributions of the scores across all nine standard planes. The FTSPD system performed significantly better than junior sonographers in five standard plane types. CONCLUSION The results of this study suggest that our FTSPD system has significant potential for detecting standard planes in first-trimester ultrasound screening, which may help to improve the accuracy of fetal ultrasound screening and facilitate early diagnosis of abnormalities. The quality of the standard planes selected by junior sonographers can be significantly improved with the assistance of our FTSPD system.
Collapse
Affiliation(s)
- Chaojiong Zhen
- Department of Ultrasonography, Academy of Orthopedics, Third Affiliated Hospital of Southern Medical University, Guangdong Province, Guangzhou, China; Department of Medical Ultrasonics, First People's Hospital of Foshan, Foshan, China
| | - Hongzhang Wang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Jun Cheng
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Chaoyu Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Xindi Hu
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Yuanji Zhang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Yan Cao
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, Guangdong, China; Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Weijun Huang
- Department of Medical Ultrasonics, First People's Hospital of Foshan, Foshan, China
| | - Ping Wang
- Department of Ultrasonography, Academy of Orthopedics, Third Affiliated Hospital of Southern Medical University, Guangdong Province, Guangzhou, China.
| |
Collapse
|
22
|
Huang X, Yang X, Dou H, Huang Y, Zhang L, Liu Z, Yan Z, Liu L, Zou Y, Hu X, Gao R, Zhang Y, Xiong Y, Xue W, Ni D. Test-time bi-directional adaptation between image and model for robust segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 233:107477. [PMID: 36972645 DOI: 10.1016/j.cmpb.2023.107477] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/12/2022] [Revised: 03/08/2023] [Accepted: 03/10/2023] [Indexed: 06/18/2023]
Abstract
BACKGROUND AND OBJECTIVE Deep learning models often suffer from performance degradations when deployed in real clinical environments due to appearance shifts between training and testing images. Most extant methods use training-time adaptation, which almost require target domain samples in the training phase. However, these solutions are limited by the training process and cannot guarantee the accurate prediction of test samples with unforeseen appearance shifts. Further, it is impractical to collect target samples in advance. In this paper, we provide a general method of making existing segmentation models robust to samples with unknown appearance shifts when deployed in daily clinical practice. METHODS Our proposed test-time bi-directional adaptation framework combines two complementary strategies. First, our image-to-model (I2M) adaptation strategy adapts appearance-agnostic test images to the learned segmentation model using a novel plug-and-play statistical alignment style transfer module during testing. Second, our model-to-image (M2I) adaptation strategy adapts the learned segmentation model to test images with unknown appearance shifts. This strategy applies an augmented self-supervised learning module to fine-tune the learned model with proxy labels that it generates. This innovative procedure can be adaptively constrained using our novel proxy consistency criterion. This complementary I2M and M2I framework demonstrably achieves robust segmentation against unknown appearance shifts using existing deep-learning models. RESULTS Extensive experiments on 10 datasets containing fetal ultrasound, chest X-ray, and retinal fundus images demonstrate that our proposed method achieves promising robustness and efficiency in segmenting images with unknown appearance shifts. CONCLUSIONS To address the appearance shift problem in clinically acquired medical images, we provide robust segmentation by using two complementary strategies. Our solution is general and amenable for deployment in clinical settings.
Collapse
Affiliation(s)
- Xiaoqiong Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Haoran Dou
- Centre for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB), University of Leeds, UK
| | - Yuhao Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China; RayShape Medical Technology Inc., Shenzhen, China
| | - Li Zhang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Zhendong Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Zhongnuo Yan
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Lian Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Yuxin Zou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Xindi Hu
- RayShape Medical Technology Inc., Shenzhen, China
| | - Rui Gao
- RayShape Medical Technology Inc., Shenzhen, China
| | - Yuanji Zhang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China
| | - Yi Xiong
- Department of Ultrasound, Shenzhen Luohu People's Hospital, the Third Affiliated Hospital of Shenzhen University, Shenzhen, China
| | - Wufeng Xue
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China.
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedidcal Engineering, Shenzhen University, Shenzhen, China.
| |
Collapse
|
23
|
Liu J, Zhou X, Lin H, Lu X, Zheng J, Xu E, Jiang D, Zhang H, Yang X, Zhong J, Hu X, Huang Y, Zhang Y, Liang J, Liu Q, Zhong M, Chen Y, Yan H, Deng H, Zheng R, Ni D, Ren J. Deep learning based on carotid transverse B-mode scan videos for the diagnosis of carotid plaque: a prospective multicenter study. Eur Radiol 2022; 33:3478-3487. [PMID: 36512047 DOI: 10.1007/s00330-022-09324-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2022] [Revised: 09/23/2022] [Accepted: 11/28/2022] [Indexed: 12/15/2022]
Abstract
OBJECTIVES Accurate detection of carotid plaque using ultrasound (US) is essential for preventing stroke. However, the diagnostic performance of junior radiologists (with approximately 1 year of experience in carotid US evaluation) is relatively poor. We thus aim to develop a deep learning (DL) model based on US videos to improve junior radiologists' performance in plaque detection. METHODS This multicenter prospective study was conducted at five hospitals. CaroNet-Dynamic automatically detected carotid plaque from carotid transverse US videos allowing clinical detection. Model performance was evaluated using expert annotations (with more than 10 years of experience in carotid US evaluation) as the ground truth. Model robustness was investigated on different plaque characteristics and US scanning systems. Furthermore, its clinical applicability was evaluated by comparing the junior radiologists' diagnoses with and without DL-model assistance. RESULTS A total of 1647 videos from 825 patients were evaluated. The DL model yielded high performance with sensitivities of 87.03% and 94.17%, specificities of 82.07% and 74.04%, and areas under the receiver operating characteristic curve of 0.845 and 0.841 on the internal and multicenter external test sets, respectively. Moreover, no significant difference in performance was noted among different plaque characteristics and scanning systems. Using the DL model, the performance of the junior radiologists improved significantly, especially in terms of sensitivity (largest increase from 46.3 to 94.44%). CONCLUSIONS The DL model based on US videos corresponding to real examinations showed robust performance for plaque detection and significantly improved the diagnostic performance of junior radiologists. KEY POINTS • The deep learning model based on US videos conforming to real examinations showed robust performance for plaque detection. • Computer-aided diagnosis can significantly improve the diagnostic performance of junior radiologists in clinical practice.
Collapse
|
24
|
Hu W, Chen H, Liu W, Li X, Sun H, Huang X, Grzegorzek M, Li C. A comparative study of gastric histopathology sub-size image classification: From linear regression to visual transformer. Front Med (Lausanne) 2022; 9:1072109. [PMID: 36569152 PMCID: PMC9767945 DOI: 10.3389/fmed.2022.1072109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2022] [Accepted: 11/18/2022] [Indexed: 12/12/2022] Open
Abstract
Introduction Gastric cancer is the fifth most common cancer in the world. At the same time, it is also the fourth most deadly cancer. Early detection of cancer exists as a guide for the treatment of gastric cancer. Nowadays, computer technology has advanced rapidly to assist physicians in the diagnosis of pathological pictures of gastric cancer. Ensemble learning is a way to improve the accuracy of algorithms, and finding multiple learning models with complementarity types is the basis of ensemble learning. Therefore, this paper compares the performance of multiple algorithms in anticipation of applying ensemble learning to a practical gastric cancer classification problem. Methods The complementarity of sub-size pathology image classifiers when machine performance is insufficient is explored in this experimental platform. We choose seven classical machine learning classifiers and four deep learning classifiers for classification experiments on the GasHisSDB database. Among them, classical machine learning algorithms extract five different image virtual features to match multiple classifier algorithms. For deep learning, we choose three convolutional neural network classifiers. In addition, we also choose a novel Transformer-based classifier. Results The experimental platform, in which a large number of classical machine learning and deep learning methods are performed, demonstrates that there are differences in the performance of different classifiers on GasHisSDB. Classical machine learning models exist for classifiers that classify Abnormal categories very well, while classifiers that excel in classifying Normal categories also exist. Deep learning models also exist with multiple models that can be complementarity. Discussion Suitable classifiers are selected for ensemble learning, when machine performance is insufficient. This experimental platform demonstrates that multiple classifiers are indeed complementarity and can improve the efficiency of ensemble learning. This can better assist doctors in diagnosis, improve the detection of gastric cancer, and increase the cure rate.
Collapse
Affiliation(s)
- Weiming Hu
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Haoyuan Chen
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Wanli Liu
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Xiaoyan Li
- Department of Pathology, Liaoning Cancer Hospital and Institute, Cancer Hospital, China Medical University, Shenyang, China
| | - Hongzan Sun
- Department of Radiology, Shengjing Hospital, China Medical University, Shenyang, China
| | - Xinyu Huang
- Institute of Medical Informatics, University of Luebeck, Luebeck, Germany
| | - Marcin Grzegorzek
- Institute of Medical Informatics, University of Luebeck, Luebeck, Germany
- Department of Knowledge Engineering, University of Economics in Katowice, Katowice, Poland
| | - Chen Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| |
Collapse
|
25
|
Zhu D, Li J, Li Y, Wu J, Zhu L, Li J, Wang Z, Xu J, Dong F, Cheng J. Multimodal ultrasound fusion network for differentiating between benign and malignant solid renal tumors. Front Mol Biosci 2022; 9:982703. [PMID: 36148014 PMCID: PMC9488515 DOI: 10.3389/fmolb.2022.982703] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2022] [Accepted: 07/26/2022] [Indexed: 11/13/2022] Open
Abstract
Objective: We aim to establish a deep learning model called multimodal ultrasound fusion network (MUF-Net) based on gray-scale and contrast-enhanced ultrasound (CEUS) images for classifying benign and malignant solid renal tumors automatically and to compare the model’s performance with the assessments by radiologists with different levels of experience.Methods: A retrospective study included the CEUS videos of 181 patients with solid renal tumors (81 benign and 100 malignant tumors) from June 2012 to June 2021. A total of 9794 B-mode and CEUS-mode images were cropped from the CEUS videos. The MUF-Net was proposed to combine gray-scale and CEUS images to differentiate benign and malignant solid renal tumors. In this network, two independent branches were designed to extract features from each of the two modalities, and the features were fused using adaptive weights. Finally, the network output a classification score based on the fused features. The model’s performance was evaluated using five-fold cross-validation and compared with the assessments of the two groups of radiologists with different levels of experience.Results: For the discrimination between benign and malignant solid renal tumors, the junior radiologist group, senior radiologist group, and MUF-Net achieved accuracy of 70.6%, 75.7%, and 80.0%, sensitivity of 89.3%, 95.9%, and 80.4%, specificity of 58.7%, 62.9%, and 79.1%, and area under the receiver operating characteristic curve of 0.740 (95% confidence internal (CI): 0.70–0.75), 0.794 (95% CI: 0.72–0.83), and 0.877 (95% CI: 0.83–0.93), respectively.Conclusion: The MUF-Net model can accurately classify benign and malignant solid renal tumors and achieve better performance than senior radiologists.Key points: The CEUS video data contain the entire tumor microcirculation perfusion characteristics. The proposed MUF-Net based on B-mode and CEUS-mode images can accurately distinguish between benign and malignant solid renal tumors with an area under the receiver operating characteristic curve of 0.877, which surpasses senior radiologists’ assessments by a large margin.
Collapse
Affiliation(s)
- Dongmei Zhu
- Department of Ultrasound, The Second Clinical Medical College, Jinan University, Shenzhen, China
- Department of Ultrasound, The Affiliated Nanchong Central Hospital of North Sichuan Medical College, Nanchong, China
| | - Junyu Li
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
- Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Yan Li
- Department of Ultrasound, The Affiliated Nanchong Central Hospital of North Sichuan Medical College, Nanchong, China
| | - Ji Wu
- Department of Urology Surgery, The Affiliated Nanchong Central Hospital of North Sichuan Medical College, Nanchong, China
| | - Lin Zhu
- Department of Ultrasound, The Affiliated Nanchong Central Hospital of North Sichuan Medical College, Nanchong, China
| | - Jian Li
- Department of Ultrasound, The Second Clinical Medical College, Jinan University, Shenzhen, China
| | - Zimo Wang
- Department of Ultrasound, The Second Clinical Medical College, Jinan University, Shenzhen, China
| | - Jinfeng Xu
- Department of Ultrasound, The Second Clinical Medical College, Jinan University, Shenzhen, China
- *Correspondence: Jinfeng Xu, ; Fajin Dong, ; Jun Cheng,
| | - Fajin Dong
- Department of Ultrasound, The Second Clinical Medical College, Jinan University, Shenzhen, China
- *Correspondence: Jinfeng Xu, ; Fajin Dong, ; Jun Cheng,
| | - Jun Cheng
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
- Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
- Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
- *Correspondence: Jinfeng Xu, ; Fajin Dong, ; Jun Cheng,
| |
Collapse
|