1
|
Dong G, Wang Z, Chen Y, Sun Y, Song H, Liu L, Cui H. An efficient segment anything model for the segmentation of medical images. Sci Rep 2024; 14:19425. [PMID: 39169054 DOI: 10.1038/s41598-024-70288-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2024] [Accepted: 08/14/2024] [Indexed: 08/23/2024] Open
Abstract
This paper introduces the efficient medical-images-aimed segment anything model (EMedSAM), addressing the high computational demands and limited adaptability of using SAM for medical image segmentation tasks. We present a novel, compact image encoder, DD-TinyViT, designed to enhance segmentation efficiency through an innovative parameter tuning method called med-adapter. The lightweight DD-TinyViT encoder is derived from the well-known ViT-H using a decoupled distillation approach.The segmentation and recognition capabilities of EMedSAM for specific structures are improved by med-adapter, which dynamically adjusts the model parameters specifically for medical imaging. We conducted extensive testing on EMedSAM using the public FLARE 2022 dataset and datasets from the First Hospital of Zhejiang University School of Medicine. The results demonstrate that our model outperforms existing state-of-the-art models in both multi-organ and lung segmentation tasks.
Collapse
Affiliation(s)
- Guanliang Dong
- School of Information Engineering, Huzhou University, Huzhou, 313000, China
| | - Zhangquan Wang
- College of Information Science and Technology, Zhejiang Shuren University, Hangzhou, 310015, China.
| | - Yourong Chen
- College of Information Science and Technology, Zhejiang Shuren University, Hangzhou, 310015, China
| | - Yuliang Sun
- College of Information Science and Technology, Zhejiang Shuren University, Hangzhou, 310015, China
| | - Hongbo Song
- College of Information Science and Technology, Zhejiang Shuren University, Hangzhou, 310015, China
| | - Liyuan Liu
- Department of Decision and System Sciences, Saint Joseph's University, Philadelphia, 19131, USA
| | - Haidong Cui
- Department of Breast Surgery, First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, 310003, China
| |
Collapse
|
2
|
Wang J, Zhou J, Wang M. Pan-cancer image segmentation based on feature pyramids and Mask R-CNN framework. Med Phys 2024; 51:5427-5440. [PMID: 38436455 DOI: 10.1002/mp.17014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 01/24/2024] [Accepted: 02/20/2024] [Indexed: 03/05/2024] Open
Abstract
BACKGROUND Cancer, a disease with a high mortality rate, poses a great threat to patients' physical and mental health and can lead to huge medical costs and emotional damage. With the continuous development of artificial intelligence technologies, deep learning-based cancer image segmentation techniques are becoming increasingly important in cancer detection and accurate diagnosis. However, in segmentation tasks, there are differences in efficiency between large and small objects and limited segmentation effects on objects of individual sizes. The previous segmentation frameworks still have room for improvement in multi-scale collaboration when segmenting objects. PURPOSE This paper proposes a method to train a deep learning segmentation framework using a feature pyramid processing dataset to improve the average precision (AP) index, and realizes multi-scale cooperation in target segmentation. OBJECTIVE Pan-Cancer Histology Dataset for Nuclei Instance Segmentation and Classification (PanNuke) dataset was selected to include approximately 7500 pathology images with cells from 19 different types of tissues, including five classifications of cancer, non-cancer, inflammation, death, and connective tissue. METHODS First, the method uses whole-slide images in the pan-cancer histology dataset for nuclei instance segmentation and classification (PanNuke) dataset, combined with the mask region convolutional neural network (Mask R-CNN) segmentation framework and improved loss function to segment and detect each cellular tissue in cancerous sections. Second, to address the problem of non-synergistic object segmentation at different scales in cancerous tissue segmentation, a scheme using feature pyramids to process the dataset was adopted as part of the feature extraction module. RESULTS Extensive experimental results on this dataset show that the method in this paper yields 0.269 AP and a boost of about 4% compared to the original Mask R-CNN framework. CONCLUSIONS It is effective and feasible to use feature pyramid to process data set to improve the effect of medical image segmentation.
Collapse
Affiliation(s)
- Juan Wang
- School of Computer and Information Engineering, Institute for Artificial Intelligence, Shanghai Polytechnic University, Shanghai, China
| | - Jian Zhou
- National Key Laboratory of Materials for Integrated Circuits, Shanghai Institute of Microsystem and Information Technology Chinese Academy of Science, Shanghai, China
| | - Man Wang
- School of Computer and Information Engineering, Institute for Artificial Intelligence, Shanghai Polytechnic University, Shanghai, China
| |
Collapse
|
3
|
Xu S, Chen Y, Yang S, Zhang X, Sun F. FCSU-Net: A novel full-scale Cross-dimension Self-attention U-Net with collaborative fusion of multi-scale feature for medical image segmentation. Comput Biol Med 2024; 180:108947. [PMID: 39094324 DOI: 10.1016/j.compbiomed.2024.108947] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2024] [Revised: 06/19/2024] [Accepted: 07/25/2024] [Indexed: 08/04/2024]
Abstract
Recently, ViT and CNNs based on encoder-decoder architecture have become the dominant model in the field of medical image segmentation. However, there are some deficiencies for each of them: (1) It is difficult for CNNs to capture the interaction between two locations with consideration of the longer distance. (2) ViT cannot acquire the interaction of local context information and carries high computational complexity. To optimize the above deficiencies, we propose a new network for medical image segmentation, which is called FCSU-Net. FCSU-Net uses the proposed collaborative fusion of multi-scale feature block that enables the network to obtain more abundant and more accurate features. In addition, FCSU-Net fuses full-scale feature information through the FFF (Full-scale Feature Fusion) structure instead of simple skip connections, and establishes long-range dependencies on multiple dimensions through the CS (Cross-dimension Self-attention) mechanism. Meantime, every dimension is complementary to each other. Also, CS mechanism has the advantage of convolutions capturing local contextual weights. Finally, FCSU-Net is validated on several datasets, and the results show that FCSU-Net not only has a relatively small number of parameters, but also has a leading segmentation performance.
Collapse
Affiliation(s)
- Shijie Xu
- School of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China.
| | - Yufeng Chen
- School of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China.
| | - Shukai Yang
- School of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China.
| | - Xiaoqian Zhang
- School of Information Engineering, Southwest University of Science and Technology, Mianyang 621010, China.
| | - Feng Sun
- Mianyang Central Hospital, School of Medicine, University of Electronic Science and Technology of China, Mianyang 621000, China; NHC Key Laboratory of Nuclear Technology Medical Transformation, Mianyang Central Hospital, Mianyang 621000, China.
| |
Collapse
|
4
|
Yao W, Bai J, Liao W, Chen Y, Liu M, Xie Y. From CNN to Transformer: A Review of Medical Image Segmentation Models. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:1529-1547. [PMID: 38438696 PMCID: PMC11300773 DOI: 10.1007/s10278-024-00981-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Revised: 11/13/2023] [Accepted: 11/14/2023] [Indexed: 03/06/2024]
Abstract
Medical image segmentation is an important step in medical image analysis, especially as a crucial prerequisite for efficient disease diagnosis and treatment. The use of deep learning for image segmentation has become a prevalent trend. The widely adopted approach currently is U-Net and its variants. Moreover, with the remarkable success of pre-trained models in natural language processing tasks, transformer-based models like TransUNet have achieved desirable performance on multiple medical image segmentation datasets. Recently, the Segment Anything Model (SAM) and its variants have also been attempted for medical image segmentation. In this paper, we conduct a survey of the most representative seven medical image segmentation models in recent years. We theoretically analyze the characteristics of these models and quantitatively evaluate their performance on Tuberculosis Chest X-rays, Ovarian Tumors, and Liver Segmentation datasets. Finally, we discuss the main challenges and future trends in medical image segmentation. Our work can assist researchers in the related field to quickly establish medical segmentation models tailored to specific regions.
Collapse
Affiliation(s)
- Wenjian Yao
- Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, 610054, Chengdu, China
| | - Jiajun Bai
- Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, 610054, Chengdu, China
| | - Wei Liao
- Department of Obstetrics and Gynaecology, Deyang People's Hospital, 618000, Deyang, China
| | - Yuheng Chen
- Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, 610054, Chengdu, China
| | - Mengjuan Liu
- Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, 610054, Chengdu, China.
| | - Yao Xie
- Department of Obstetrics and Gynaecology, Sichuan Provincial People's Hospital, School of Medicine, University of Electronic Science and Technology of China, Chengdu, China.
- Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, 610072, Chengdu, China.
| |
Collapse
|
5
|
Ahmad J, Akram S, Jaffar A, Ali Z, Bhatti SM, Ahmad A, Rehman SU. Deep learning empowered breast cancer diagnosis: Advancements in detection and classification. PLoS One 2024; 19:e0304757. [PMID: 38990817 PMCID: PMC11239011 DOI: 10.1371/journal.pone.0304757] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2024] [Accepted: 05/18/2024] [Indexed: 07/13/2024] Open
Abstract
Recent advancements in AI, driven by big data technologies, have reshaped various industries, with a strong focus on data-driven approaches. This has resulted in remarkable progress in fields like computer vision, e-commerce, cybersecurity, and healthcare, primarily fueled by the integration of machine learning and deep learning models. Notably, the intersection of oncology and computer science has given rise to Computer-Aided Diagnosis (CAD) systems, offering vital tools to aid medical professionals in tumor detection, classification, recurrence tracking, and prognosis prediction. Breast cancer, a significant global health concern, is particularly prevalent in Asia due to diverse factors like lifestyle, genetics, environmental exposures, and healthcare accessibility. Early detection through mammography screening is critical, but the accuracy of mammograms can vary due to factors like breast composition and tumor characteristics, leading to potential misdiagnoses. To address this, an innovative CAD system leveraging deep learning and computer vision techniques was introduced. This system enhances breast cancer diagnosis by independently identifying and categorizing breast lesions, segmenting mass lesions, and classifying them based on pathology. Thorough validation using the Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) demonstrated the CAD system's exceptional performance, with a 99% success rate in detecting and classifying breast masses. While the accuracy of detection is 98.5%, when segmenting breast masses into separate groups for examination, the method's performance was approximately 95.39%. Upon completing all the analysis, the system's classification phase yielded an overall accuracy of 99.16% for classification. The potential for this integrated framework to outperform current deep learning techniques is proposed, despite potential challenges related to the high number of trainable parameters. Ultimately, this recommended framework offers valuable support to researchers and physicians in breast cancer diagnosis by harnessing cutting-edge AI and image processing technologies, extending recent advances in deep learning to the medical domain.
Collapse
Affiliation(s)
- Jawad Ahmad
- Faculty of Computer Science & Information Technology, The Superior University, Lahore, Pakistan
- Intelligent Data Visual Computing Research (IDVCR), Lahore, Pakistan
| | - Sheeraz Akram
- Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, Saudi Arabia
| | - Arfan Jaffar
- Faculty of Computer Science & Information Technology, The Superior University, Lahore, Pakistan
- Intelligent Data Visual Computing Research (IDVCR), Lahore, Pakistan
| | - Zulfiqar Ali
- School of Computer Science and Electronic Engineering (CSEE), University of Essex, Wivenhoe Park, Colchester, United Kingdom
| | - Sohail Masood Bhatti
- Faculty of Computer Science & Information Technology, The Superior University, Lahore, Pakistan
- Intelligent Data Visual Computing Research (IDVCR), Lahore, Pakistan
| | - Awais Ahmad
- Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, Saudi Arabia
| | - Shafiq Ur Rehman
- Information Systems Department, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, Saudi Arabia
| |
Collapse
|
6
|
Sinha A, Kawahara J, Pakzad A, Abhishek K, Ruthven M, Ghorbel E, Kacem A, Aouada D, Hamarneh G. DermSynth3D: Synthesis of in-the-wild annotated dermatology images. Med Image Anal 2024; 95:103145. [PMID: 38615432 DOI: 10.1016/j.media.2024.103145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 02/11/2024] [Accepted: 03/18/2024] [Indexed: 04/16/2024]
Abstract
In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermatological images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.
Collapse
Affiliation(s)
- Ashish Sinha
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Jeremy Kawahara
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Arezou Pakzad
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Kumar Abhishek
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Matthieu Ruthven
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg
| | - Enjie Ghorbel
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg; Cristal Laboratory, National School of Computer Sciences, University of Manouba, 2010, Tunisia
| | - Anis Kacem
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg
| | - Djamila Aouada
- Computer Vision, Imaging & Machine Intelligence Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg
| | - Ghassan Hamarneh
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada.
| |
Collapse
|
7
|
Wang X, Yuan W. Nuclei-level prior knowledge constrained multiple instance learning for breast histopathology whole slide image classification. iScience 2024; 27:109826. [PMID: 38832012 PMCID: PMC11145340 DOI: 10.1016/j.isci.2024.109826] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Revised: 04/17/2024] [Accepted: 04/24/2024] [Indexed: 06/05/2024] Open
Abstract
New breast cancer cases have surpassed lung cancer, becoming the world's most prevalent cancer. Despite advancing medical image analysis, deep learning's lack of interpretability limits its adoption among pathologists. Hence, a nuclei-level prior knowledge constrained multiple instance learning (MIL) (NPKC-MIL) for breast whole slide image (WSI) classification is proposed. NPKC-MIL primarily involves the following steps: Initially, employing the transfer learning to extract patch-level features and aggregate them into slide-level features through attention pooling. Subsequently, abstract the extracted nuclei as nodes, establish nucleus topology using the K-NN (K-Nearest Neighbors, K-NN) algorithm, and create handcrafted features for nodes. Finally, combine patch-level deep learning features with nuclei-level handcrafted features to fine-tune classification results generated by slide-level deep learning features. The experimental results demonstrate that NPKC-MIL outperforms current comparable deep learning models. NPKC-MIL expands the analytical dimension of WSI classification tasks and integrates prior knowledge into deep learning models to improve interpretability.
Collapse
Affiliation(s)
- Xunping Wang
- School of Remote Sensing and Information Engineering, Wuhan University, 129 Luoyu Road, Wuhan 430079, China
| | - Wei Yuan
- Co-Creation Center for Disaster Resilience, International Research Institute of Disaster Science, Tohoku University, Aoba 468-1, Aramaki, Aoba-ku, Sendai 980-8572, Japan
| |
Collapse
|
8
|
Zimmermann N, Sieberth T, Dobay A. Automated wound segmentation and classification of seven common injuries in forensic medicine. Forensic Sci Med Pathol 2024; 20:443-451. [PMID: 37378809 PMCID: PMC11297066 DOI: 10.1007/s12024-023-00668-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/14/2023] [Indexed: 06/29/2023]
Abstract
In forensic medical investigations, physical injuries are documented with photographs accompanied by written reports. Automatic segmentation and classification of wounds on these photographs could provide forensic pathologists with a tool to improve the assessment of injuries and accelerate the reporting process. In this pilot study, we trained and compared several preexisting deep learning architectures for image segmentation and wound classification on forensically relevant photographs in our database. The best scores were a mean pixel accuracy of 69.4% and a mean intersection over union (IoU) of 48.6% when evaluating the trained models on our test set. The models had difficulty distinguishing the background from wounded areas. As an example, image pixels showing subcutaneous hematomas or skin abrasions were assigned to the background class in 31% of cases. Stab wounds, on the other hand, were reliably classified with a pixel accuracy of 93%. These results can be partially attributed to undefined wound boundaries for some types of injuries, such as subcutaneous hematoma. However, despite the large class imbalance, we demonstrate that the best trained models could reliably distinguish among seven of the most common wounds encountered in forensic medical investigations.
Collapse
Affiliation(s)
- Norio Zimmermann
- Zurich Institute of Forensic Medicine, University of Zurich, Winterthurerstrasse 190/52, CH-8057, Zurich, Switzerland
| | - Till Sieberth
- Zurich Institute of Forensic Medicine, University of Zurich, Winterthurerstrasse 190/52, CH-8057, Zurich, Switzerland
- 3D Centre Zurich, University of Zurich, Winterthurerstrasse 190/52, CH-8057, Zurich, Switzerland
| | - Akos Dobay
- Zurich Institute of Forensic Medicine, University of Zurich, Winterthurerstrasse 190/52, CH-8057, Zurich, Switzerland.
- Forensic Machine Learning Technology Center, University of Zurich, Winterthurerstrasse 190/52, CH-8057, Zurich, Switzerland.
| |
Collapse
|
9
|
Cheng Z, Wang S, Xin T, Zhou T, Zhang H, Shao L. Few-Shot Medical Image Segmentation via Generating Multiple Representative Descriptors. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2202-2214. [PMID: 38265915 DOI: 10.1109/tmi.2024.3358295] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/26/2024]
Abstract
Automatic medical image segmentation has witnessed significant development with the success of large models on massive datasets. However, acquiring and annotating vast medical image datasets often proves to be impractical due to the time consumption, specialized expertise requirements, and compliance with patient privacy standards, etc. As a result, Few-shot Medical Image Segmentation (FSMIS) has become an increasingly compelling research direction. Conventional FSMIS methods usually learn prototypes from support images and apply nearest-neighbor searching to segment the query images. However, only a single prototype cannot well represent the distribution of each class, thus leading to restricted performance. To address this problem, we propose to Generate Multiple Representative Descriptors (GMRD), which can comprehensively represent the commonality within the corresponding class distribution. In addition, we design a Multiple Affinity Maps based Prediction (MAMP) module to fuse the multiple affinity maps generated by the aforementioned descriptors. Furthermore, to address intra-class variation and enhance the representativeness of descriptors, we introduce two novel losses. Notably, our model is structured as a dual-path design to achieve a balance between foreground and background differences in medical images. Extensive experiments on four publicly available medical image datasets demonstrate that our method outperforms the state-of-the-art methods, and the detailed analysis also verifies the effectiveness of our designed module.
Collapse
|
10
|
Eyster HN, Beckage B. Applying a deep learning pipeline to classify land cover from low-quality historical RGB imagery. PeerJ Comput Sci 2024; 10:e2003. [PMID: 38855218 PMCID: PMC11157591 DOI: 10.7717/peerj-cs.2003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Accepted: 04/01/2024] [Indexed: 06/11/2024]
Abstract
Land use and land cover (LULC) classification is becoming faster and more accurate thanks to new deep learning algorithms. Moreover, new high spectral- and spatial-resolution datasets offer opportunities to classify land cover with greater accuracy and class specificity. However, deploying deep learning algorithms to characterize present-day, modern land cover based on state-of-the-art data is insufficient for understanding trends in land cover change and identifying changes in and drivers of ecological and social variables of interest. These identifications require characterizing past land cover, for which imagery is often lower-quality. We applied a deep learning pipeline to classify land cover from historical, low-quality RGB aerial imagery, using a case study of Vancouver, Canada. We deployed an atrous convolutional neural network from DeepLabv3+ (which has previously shown to outperform other networks) and trained it on modern Maxar satellite imagery using a modern land cover classification. We fine-tuned the resultant model using a small dataset of manually annotated and augmented historical imagery. This final model accurately predicted historical land cover classification at rates similar to other studies that used high-quality imagery. These predictions indicate that Vancouver has lost vegetative cover from 1995-2021, including a decrease in conifer cover, an increase in pavement cover, and an overall decrease in tree and grass cover. Our workflow may be harnessed to understand historical land cover and identify land cover change in other regions and at other times.
Collapse
Affiliation(s)
- Harold N. Eyster
- Department of Plant Biology, University of Vermont, Burlington, VT, United States of America
- Gund Institute for Environment, University of Vermont, Burlington, VT, United States of America
| | - Brian Beckage
- Department of Plant Biology, University of Vermont, Burlington, VT, United States of America
- Gund Institute for Environment, University of Vermont, Burlington, VT, United States of America
| |
Collapse
|
11
|
Su D, Luo J, Fei C. An Efficient and Rapid Medical Image Segmentation Network. IEEE J Biomed Health Inform 2024; 28:2979-2990. [PMID: 38457317 DOI: 10.1109/jbhi.2024.3374780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/10/2024]
Abstract
Accurate medical image segmentation is an essential part of the medical image analysis process that provides detailed quantitative metrics. In recent years, extensions of classical networks such as UNet have achieved state-of-the-art performance on medical image segmentation tasks. However, the high model complexity of these networks limits their applicability to devices with constrained computational resources. To alleviate this problem, we propose a shallow hierarchical Transformer for medical image segmentation, called SHFormer. By decreasing the number of transformer blocks utilized, the model complexity of SHFormer can be reduced to an acceptable level. To improve the learned attention while keeping the structure lightweight, we propose a spatial-channel connection module. This module separately learns attention in the spatial and channel dimensions of the feature while interconnecting them to produce more focused attention. To keep the decoder lightweight, the MLP-D module is proposed to progressively fuse multi-scale features in which channels are aligned using Multi-Layer Perceptron (MLP) and spatial information is fused by convolutional blocks. We first validated the performance of SHFormer on the ISIC-2018 dataset. Compared to the latest network, SHFormer exhibits comparable performance with 15 times fewer parameters, 30 times lower computational complexity and 5 times higher inference efficiency. To test the generalizability of SHFormer, we introduced the polyp dataset for additional testing. SHFormer achieves comparable segmentation accuracy to the latest network while having lower computational overhead.
Collapse
|
12
|
von Schulthess EL, Maunz A, Chakravarthy U, Holekamp N, Pauleikhoff D, Patel K, Bachmeier I, Yu S, Cohen Y, Scherb MP, Jones IL, Gibson K, Willis JR, Glittenberg C, Singh RP, Fauser S. Intraretinal Hyper-Reflective Foci Are Almost Universally Present and Co-Localize With Intraretinal Fluid in Diabetic Macular Edema. Invest Ophthalmol Vis Sci 2024; 65:26. [PMID: 38758639 PMCID: PMC11107948 DOI: 10.1167/iovs.65.5.26] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Accepted: 04/28/2024] [Indexed: 05/19/2024] Open
Abstract
Purpose In diabetic macular edema (DME), hyper-reflective foci (HRF) has been linked to disease severity and progression. Using an automated approach, we aimed to investigate the baseline distribution of HRF in DME and their co-localization with cystoid intraretinal fluid (IRF). Methods Baseline spectral-domain optical coherence tomography (SD-OCT) volume scans (N = 1527) from phase III clinical trials YOSEMITE (NCT03622580) and RHINE (NCT03622593) were segmented using a deep-learning-based algorithm (developed using B-scans from BOULEVARD NCT02699450) to detect HRF. The HRF count and volume were assessed. HRF distributions were analyzed in relation to best-corrected visual acuity (BCVA), central subfield thickness (CST), and IRF volume in quartiles, and Diabetic Retinopathy Severity Scores (DRSS) in groups. Co-localization of HRF with IRF was calculated in the central 3-mm diameter using the en face projection. Results HRF were present in most patients (up to 99.7%). Median (interquartile range [IQR]) HRF volume within the 3-mm diameter Early Treatment Diabetic Retinopathy Study ring was 1964.3 (3325.2) pL, and median count was 64.0 (IQR = 96.0). Median HRF volumes were greater with decreasing BCVA (nominal P = 0.0109), and increasing CST (nominal P < 0.0001), IRF (nominal P < 0.0001), and DRSS up to very severe nonproliferative diabetic retinopathy (nominal P < 0.0001). HRF co-localized with IRF in the en face projection. Conclusions Using automated HRF segmentation of full SD-OCT volumes, we observed that HRF are a ubiquitous feature in DME and exhibit relationships with BCVA, CST, IRF, and DRSS, supporting a potential link to disease severity. The spatial distribution of HRF closely followed that of IRF.
Collapse
Affiliation(s)
- Esther L. von Schulthess
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Andreas Maunz
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Usha Chakravarthy
- Queen's University of Belfast, Royal Victoria Hospital, Belfast, United Kingdom
| | - Nancy Holekamp
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
- Pepose Vision Institute, Chesterfield, Missouri, United States
| | | | - Katie Patel
- Roche Products Ltd., Welwyn Garden City, United Kingdom
| | - Isabel Bachmeier
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Siqing Yu
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Yaniv Cohen
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Mahnaz Parian Scherb
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Ian L. Jones
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Kara Gibson
- Roche Products Ltd., Welwyn Garden City, United Kingdom
| | | | - Carl Glittenberg
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| | - Rishi P. Singh
- Center for Ophthalmic Bioinformatics, Cole Eye Institute, Cleveland Clinic, Cleveland, Ohio, United States
| | - Sascha Fauser
- Roche Pharma Research and Early Development, Therapeutic Modalities, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd., Basel, Switzerland
| |
Collapse
|
13
|
Karthik EN, Valosek J, Smith AC, Pfyffer D, Schading-Sassenhausen S, Farner L, Weber KA, Freund P, Cohen-Adad J. SCIseg: Automatic Segmentation of T2-weighted Intramedullary Lesions in Spinal Cord Injury. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.01.03.24300794. [PMID: 38699309 PMCID: PMC11065035 DOI: 10.1101/2024.01.03.24300794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/05/2024]
Abstract
Purpose To develop a deep learning tool for the automatic segmentation of T2-weighted intramedullary lesions in spinal cord injury (SCI). Material and Methods This retrospective study included a cohort of SCI patients from three sites enrolled between July 2002 and February 2023. A deep learning model, SCIseg, was trained in a three-phase process involving active learning for the automatic segmentation of intramedullary SCI lesions and the spinal cord. The data consisted of T2-weighted MRI acquired using different scanner manufacturers with heterogeneous image resolutions (isotropic/anisotropic), orientations (axial/sagittal), lesion etiologies (traumatic/ischemic/hemorrhagic) and lesions spread across the cervical, thoracic and lumbar spine. The segmentations from the proposed model were visually and quantitatively compared with other open-source baselines. Wilcoxon signed-rank test was used to compare quantitative MRI biomarkers (lesion volume, lesion length, and maximal axial damage ratio) computed from manual lesion masks and those obtained automatically with SCIseg predictions. Results MRI data from 191 SCI patients (mean age, 48.1 years ± 17.9 [SD]; 142 males) were used for model training and evaluation. SCIseg achieved the best segmentation performance for both the cord and lesions. There was no statistically significant difference between lesion length and maximal axial damage ratio computed from manually annotated lesions and those obtained using SCIseg. Conclusion Automatic segmentation of intramedullary lesions commonly seen in SCI replaces the tedious manual annotation process and enables the extraction of relevant lesion morphometrics in large cohorts. The proposed model segments lesions across different etiologies, scanner manufacturers, and heterogeneous image resolutions. SCIseg is open-source and accessible through the Spinal Cord Toolbox.
Collapse
Affiliation(s)
- Enamundram Naga Karthik
- NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal, Montreal, QC, Canada
- Mila - Quebec AI Institute, Montreal, QC, Canada
| | - Jan Valosek
- NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal, Montreal, QC, Canada
- Mila - Quebec AI Institute, Montreal, QC, Canada
- Department of Neurosurgery, Faculty of Medicine and Dentistry, Palacký University Olomouc, Olomouc, Czechia
- Department of Neurology, Faculty of Medicine and Dentistry, Palacký University Olomouc, Olomouc, Czechia
| | - Andrew C Smith
- Department of Physical Medicine and Rehabilitation Physical Therapy Program, University of Colorado School of Medicine, Aurora, Colorado, USA
| | - Dario Pfyffer
- Spinal Cord Injury Center, Balgrist University Hospital, University of Zürich, Zürich, Switzerland
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Stanford, California, USA
| | | | - Lynn Farner
- Spinal Cord Injury Center, Balgrist University Hospital, University of Zürich, Zürich, Switzerland
| | - Kenneth A Weber
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Patrick Freund
- Spinal Cord Injury Center, Balgrist University Hospital, University of Zürich, Zürich, Switzerland
- Department of Neurophysics, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Julien Cohen-Adad
- NeuroPoly Lab, Institute of Biomedical Engineering, Polytechnique Montreal, Montreal, QC, Canada
- Mila - Quebec AI Institute, Montreal, QC, Canada
- Functional Neuroimaging Unit, CRIUGM, Université de Montréal, Montreal, QC, Canada
- Centre de Recherche du CHU Sainte-Justine, Université de Montréal, Montreal, QC, Canada
| |
Collapse
|
14
|
Zhang W, Zhang X, Li L, Liao L, Zhao F, Zhong T, Pei Y, Xu X, Yang C, Zhang H, Li G. A joint brain extraction and image quality assessment framework for fetal brain MRI slices. Neuroimage 2024; 290:120560. [PMID: 38431181 DOI: 10.1016/j.neuroimage.2024.120560] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 02/26/2024] [Accepted: 02/27/2024] [Indexed: 03/05/2024] Open
Abstract
Brain extraction and image quality assessment are two fundamental steps in fetal brain magnetic resonance imaging (MRI) 3D reconstruction and quantification. However, the randomness of fetal position and orientation, the variability of fetal brain morphology, maternal organs around the fetus, and the scarcity of data samples, all add excessive noise and impose a great challenge to automated brain extraction and quality assessment of fetal MRI slices. Conventionally, brain extraction and quality assessment are typically performed independently. However, both of them focus on the brain image representation, so they can be jointly optimized to ensure the network learns more effective features and avoid overfitting. To this end, we propose a novel two-stage dual-task deep learning framework with a brain localization stage and a dual-task stage for joint brain extraction and quality assessment of fetal MRI slices. Specifically, the dual-task module compactly contains a feature extraction module, a quality assessment head and a segmentation head with feature fusion for simultaneous brain extraction and quality assessment. Besides, a transformer architecture is introduced into the feature extraction module and the segmentation head. We utilize a multi-step training strategy to guarantee a stable and successful training of all modules. Finally, we validate our method by a 5-fold cross-validation and ablation study on a dataset with fetal brain MRI slices in different qualities, and perform a cross-dataset validation in addition. Experiments show that the proposed framework achieves very promising performance.
Collapse
Affiliation(s)
- Wenhao Zhang
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
| | - Xin Zhang
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China.
| | - Lingyi Li
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
| | - Lufan Liao
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
| | - Fenqiang Zhao
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, USA
| | - Tao Zhong
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, USA
| | - Yuchen Pei
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, USA
| | - Xiangmin Xu
- School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
| | - Chaoxiang Yang
- Department of Radiology, Guangdong Women and Children Hospital, Guangzhou, China
| | - He Zhang
- Department of Radiology, Obstetrics and Gynecology Hospital, Fudan University, Shanghai, China.
| | - Gang Li
- Department of Radiology and BRIC, University of North Carolina at Chapel Hill, USA.
| |
Collapse
|
15
|
Gao C, Shi Y, Yang S, Lei B. SAA-SDM: Neural Networks Faster Learned to Segment Organ Images. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:547-562. [PMID: 38343217 PMCID: PMC11031521 DOI: 10.1007/s10278-023-00947-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/05/2023] [Revised: 10/16/2023] [Accepted: 10/18/2023] [Indexed: 04/20/2024]
Abstract
In the field of medicine, rapidly and accurately segmenting organs in medical images is a crucial application of computer technology. This paper introduces a feature map module, Strength Attention Area Signed Distance Map (SAA-SDM), based on the principal component analysis (PCA) principle. The module is designed to accelerate neural networks' convergence speed in rapidly achieving high precision. SAA-SDM provides the neural network with confidence information regarding the target and background, similar to the signed distance map (SDM), thereby enhancing the network's understanding of semantic information related to the target. Furthermore, this paper presents a training scheme tailored for the module, aiming to achieve finer segmentation and improved generalization performance. Validation of our approach is carried out using TRUS and chest X-ray datasets. Experimental results demonstrate that our method significantly enhances neural networks' convergence speed and precision. For instance, the convergence speed of UNet and UNET + + is improved by more than 30%. Moreover, Segformer achieves an increase of over 6% and 3% in mIoU (mean Intersection over Union) on two test datasets without requiring pre-trained parameters. Our approach reduces the time and resource costs associated with training neural networks for organ segmentation tasks while effectively guiding the network to achieve meaningful learning even without pre-trained parameters.
Collapse
Affiliation(s)
- Chao Gao
- College of Computer and Information Technology, China Three Gorges University, Yichang Hubei, 443002, China
- Hubei Key Laboratory of Intelligent Vision Monitoring for Hydropower Engineering, China Three Gorges University, Yichang Hubei, 443002, China
| | - Yongtao Shi
- College of Computer and Information Technology, China Three Gorges University, Yichang Hubei, 443002, China.
- Hubei Key Laboratory of Intelligent Vision Monitoring for Hydropower Engineering, China Three Gorges University, Yichang Hubei, 443002, China.
| | - Shuai Yang
- College of Computer and Information Technology, China Three Gorges University, Yichang Hubei, 443002, China
- Hubei Key Laboratory of Intelligent Vision Monitoring for Hydropower Engineering, China Three Gorges University, Yichang Hubei, 443002, China
| | - Bangjun Lei
- College of Computer and Information Technology, China Three Gorges University, Yichang Hubei, 443002, China
- Hubei Key Laboratory of Intelligent Vision Monitoring for Hydropower Engineering, China Three Gorges University, Yichang Hubei, 443002, China
| |
Collapse
|
16
|
Abhishek K, Brown CJ, Hamarneh G. Multi-sample ζ-mixup: richer, more realistic synthetic samples from a p-series interpolant. JOURNAL OF BIG DATA 2024; 11:43. [PMID: 38528850 PMCID: PMC10960781 DOI: 10.1186/s40537-024-00898-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Accepted: 02/28/2024] [Indexed: 03/27/2024]
Abstract
Modern deep learning training procedures rely on model regularization techniques such as data augmentation methods, which generate training samples that increase the diversity of data and richness of label information. A popular recent method, mixup, uses convex combinations of pairs of original samples to generate new samples. However, as we show in our experiments, mixup can produce undesirable synthetic samples, where the data is sampled off the manifold and can contain incorrect labels. We propose ζ -mixup, a generalization of mixup with provably and demonstrably desirable properties that allows convex combinations of T ≥ 2 samples, leading to more realistic and diverse outputs that incorporate information from T original samples by using a p-series interpolant. We show that, compared to mixup, ζ -mixup better preserves the intrinsic dimensionality of the original datasets, which is a desirable property for training generalizable models. Furthermore, we show that our implementation of ζ -mixup is faster than mixup, and extensive evaluation on controlled synthetic and 26 diverse real-world natural and medical image classification datasets shows that ζ -mixup outperforms mixup, CutMix, and traditional data augmentation techniques. The code will be released at https://github.com/kakumarabhishek/zeta-mixup.
Collapse
Affiliation(s)
- Kumar Abhishek
- School of Computing Science, Simon Fraser University, 8888 University Drive, Burnaby, V5A 1S6 Canada
| | - Colin J Brown
- Engineering, Hinge Health, 455 Market Street, Suite 700, San Francisco, 94105 USA
| | - Ghassan Hamarneh
- School of Computing Science, Simon Fraser University, 8888 University Drive, Burnaby, V5A 1S6 Canada
| |
Collapse
|
17
|
Khajuria R, Sarwar A. Review of reinforcement learning applications in segmentation, chemotherapy, and radiotherapy of cancer. Micron 2024; 178:103583. [PMID: 38185018 DOI: 10.1016/j.micron.2023.103583] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 10/16/2023] [Accepted: 12/20/2023] [Indexed: 01/09/2024]
Abstract
Owing to early diagnosis and treatment of cancer as a prerequisite in recent times, the role of machine learning has been increased substantially. The mathematically powerful and optimized solutions for the detection and cure of cancer are constantly being explored and novel models based upon standard algorithms are also being developed. Leveraging one such solution is Reinforcement Learning (RL), which is a semi-supervised type of learning. The paper presents a detailed discussion on the various RL techniques, algorithms, and open issues, in addition to the review of literature for diagnosis and treatment of cancer. A smaller number of publications for diagnosis and treatment of cancer have been reported before 2011 but now after the success of Deep Learning (DL) and the advent of Deep Reinforcement Learning (DRL), the publications have grown in number from 2017 onwards. The scope of RL for cancer diagnosis and treatment is also demystified and provides the research community with the insights of how to formulate RL problem as a Cancer diagnostic problem. RL has been found successful for landmark detection in medical images and optimal control of drugs and radiations.
Collapse
|
18
|
Quinton F, Presles B, Leclerc S, Nodari G, Lopez O, Chevallier O, Pellegrinelli J, Vrigneaud JM, Popoff R, Meriaudeau F, Alberini JL. Navigating the nuances: comparative analysis and hyperparameter optimisation of neural architectures on contrast-enhanced MRI for liver and liver tumour segmentation. Sci Rep 2024; 14:3522. [PMID: 38347017 PMCID: PMC10861452 DOI: 10.1038/s41598-024-53528-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Accepted: 02/01/2024] [Indexed: 02/15/2024] Open
Abstract
In medical imaging, accurate segmentation is crucial to improving diagnosis, treatment, or both. However, navigating the multitude of available architectures for automatic segmentation can be overwhelming, making it challenging to determine the appropriate type of architecture and tune the most crucial parameters during dataset optimisation. To address this problem, we examined and refined seven distinct architectures for segmenting the liver, as well as liver tumours, with a restricted training collection of 60 3D contrast-enhanced magnetic resonance images (CE-MRI) from the ATLAS dataset. Included in these architectures are convolutional neural networks (CNNs), transformers, and hybrid CNN/transformer architectures. Bayesian search techniques were used for hyperparameter tuning to hasten convergence to the optimal parameter mixes while also minimising the number of trained models. It was unexpected that hybrid models, which typically exhibit superior performance on larger datasets, would exhibit comparable performance to CNNs. The optimisation of parameters contributed to better segmentations, resulting in an average increase of 1.7% and 5.0% in liver and tumour segmentation Dice coefficients, respectively. In conclusion, the findings of this study indicate that hybrid CNN/transformer architectures may serve as a practical substitute for CNNs even in small datasets. This underscores the significance of hyperparameter optimisation.
Collapse
Affiliation(s)
- Felix Quinton
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France.
| | - Benoit Presles
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
| | - Sarah Leclerc
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
| | - Guillaume Nodari
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Olivier Lopez
- Service de Radiologie et Imagerie Medicale Diagnostique et Therapeutique, Centre Hospitalier Universitaire, 21000, Dijon, France
| | - Olivier Chevallier
- Service de Radiologie et Imagerie Medicale Diagnostique et Therapeutique, Centre Hospitalier Universitaire, 21000, Dijon, France
| | - Julie Pellegrinelli
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Jean-Marc Vrigneaud
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Romain Popoff
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Fabrice Meriaudeau
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
| | - Jean-Louis Alberini
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| |
Collapse
|
19
|
Xu J, Wang Z. Efficient and accurate microplastics identification and segmentation in urban waters using convolutional neural networks. THE SCIENCE OF THE TOTAL ENVIRONMENT 2024; 911:168696. [PMID: 38000753 DOI: 10.1016/j.scitotenv.2023.168696] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Revised: 11/15/2023] [Accepted: 11/17/2023] [Indexed: 11/26/2023]
Abstract
Microplastics (MPs), measuring less than 5 mm, pose threats to ecological security and human health in urban waters. Additionally, they act as carriers, transporting pollutants from terrestrial systems into oceanic circulation, contributing to global pollution. Recognizing the significance of identifying MPs in urban waters, one potential solution to the time-consuming and labor-intensive manual identification process is the application of a convolutional neural network (CNN). Therefore, having a reliable CNN model that efficiently and accurately identifies MPs is essential for extensive research on MPs pollution in urban waters. In this work, an MPs dataset with complex background was acquired from urban waters in southern China. The dataset was used to train and validate CNN models, including UNet, UNet2plus, and UNet3plus. Subsequently, the computational and inference performance of the three models was evaluated using a newly collected MPs dataset. The results showed that UNet, UNet2plus, UNet3plus, after being trained for 120 epochs, provided efficient inferences within less than 1 s, 2 s, and 3 s for 100 MPs images, respectively. Accurate segmentation with mIoU of 91.45 ± 5.93 % and 91.08 ± 6.18 % was achieved using UNet and UNet2plus, respectively, while UNet3plus exhibited a lower performance with only 82.21 ± 10.33 % mIoU. This work demonstrated that UNet and UNet2plus deliver efficient and accurate identification of MPs in urban waters. Developing CNN models that efficiently and accurately identify MPs is crucial for reducing manual time, especially in large-scale investigations of MPs pollution in urban waters.
Collapse
Affiliation(s)
- Jiongji Xu
- School of Civil Engineering and Transportation, State Key Laboratory of Subtropical Building and Urban Science, South China University of Technology, Guangzhou 510641, China; Pazhou Lab, Guangzhou 510335, China
| | - Zhaoli Wang
- School of Civil Engineering and Transportation, State Key Laboratory of Subtropical Building and Urban Science, South China University of Technology, Guangzhou 510641, China; Pazhou Lab, Guangzhou 510335, China.
| |
Collapse
|
20
|
Ungureanu VI, Negirla P, Korodi A. Image-Compression Techniques: Classical and "Region-of-Interest-Based" Approaches Presented in Recent Papers. SENSORS (BASEL, SWITZERLAND) 2024; 24:791. [PMID: 38339507 PMCID: PMC10857028 DOI: 10.3390/s24030791] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Revised: 01/18/2024] [Accepted: 01/23/2024] [Indexed: 02/12/2024]
Abstract
Image compression is a vital component for domains in which the computational resources are usually scarce such as automotive or telemedicine fields. Also, when discussing real-time systems, the large amount of data that must flow through the system can represent a bottleneck. Therefore, the storage of images, alongside the compression, transmission, and decompression procedures, becomes vital. In recent years, many compression techniques that only preserve the quality of the region of interest of an image have been developed, the other parts being either discarded or compressed with major quality loss. This paper proposes a study of relevant papers from the last decade which are focused on the selection of a region of interest of an image and on the compression techniques that can be applied to that area. To better highlight the novelty of the hybrid methods, classical state-of-the-art approaches are also analyzed. The current work will provide an overview of classical and hybrid compression methods alongside a categorization based on compression ratio and other quality factors such as mean-square error and peak signal-to-noise ratio, structural similarity index measure, and so on. This overview can help researchers to develop a better idea of what compression algorithms are used in certain domains and to find out if the presented performance parameters are of interest for the intended purpose.
Collapse
Affiliation(s)
- Vlad-Ilie Ungureanu
- Automation and Applied Informatics Department, University Politehnica Timisoara, 300006 Timisoara, Romania;
| | - Paul Negirla
- Automation and Applied Informatics Department, University Politehnica Timisoara, 300006 Timisoara, Romania;
| | | |
Collapse
|
21
|
Xu W, Liu J, Fan B. Automatic segmentation of brain glioma based on XY-Net. Med Biol Eng Comput 2024; 62:153-166. [PMID: 37740132 DOI: 10.1007/s11517-023-02927-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Accepted: 09/05/2023] [Indexed: 09/24/2023]
Abstract
Glioma is a malignant primary brain tumor, which can easily lead to death if it is not detected in time. Magnetic resonance imaging is the most commonly used technique to diagnose gliomas, and precise outlining of tumor areas from magnetic resonance images (MRIs) is an important aid to physicians in understanding the patient's condition and formulating treatment plans. However, relying on radiologists to manually depict tumors is a tedious and laborious task, so it is clinically important to investigate an automated method for outlining glioma regions in MRIs. To liberate radiologists from the heavy task of outlining tumors, we propose a fully convolutional network, XY-Net, based on the most popular U-Net symmetric encoder-decoder structure to perform automatic segmentation of gliomas. We construct two symmetric sub-encoders for XY-Net and build interconnected X-shaped feature map transmission paths between the sub-encoders, while maintaining the feature map concatenation between each sub-encoder and the decoder. Moreover, a loss function composed of the balanced cross-entropy loss function and the dice loss function is used in the training task of XY-Net to solve the class unevenness problem of the medical image segmentation task. The experimental results show that the proposed XY-Net has a 2.16% improvement in dice coefficient (DC) compared to the network model with a single encoder structure, and compare with some state-of-the-art image segmentation methods, XY-Net achieves the best performance. The DC, HD, recall, and precision of our method on the test set are 74.49%, 10.89 mm, 78.06%, and 76.30%, respectively. The combination of sub-encoders and cross-transmission paths enables the model to perform better; based on this combination, the XY-Net achieves an end-to-end automatic segmentation of gliomas on 2D slices of MRIs, which can play a certain auxiliary role for doctors in grasping the state of illness.
Collapse
Affiliation(s)
- Wenbin Xu
- Nanchang Key Laboratory of Medical and Technology Research, Nanchang University, Nanchang, 330006, China
| | - Jizhong Liu
- Nanchang Key Laboratory of Medical and Technology Research, Nanchang University, Nanchang, 330006, China.
| | - Bing Fan
- Department of Radiology, Jiangxi Provincial People's Hospital, the First Affiliated Hospital of Nanchang Medical College, Nanchang, 330006, China.
| |
Collapse
|
22
|
Wang Y, Yu X, Yang Y, Zhang X, Zhang Y, Zhang L, Feng R, Xue J. A multi-branched semantic segmentation network based on twisted information sharing pattern for medical images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 243:107914. [PMID: 37992569 DOI: 10.1016/j.cmpb.2023.107914] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/21/2023] [Revised: 10/12/2023] [Accepted: 11/03/2023] [Indexed: 11/24/2023]
Abstract
BACKGROUND Semantic segmentation plays an indispensable role in clinical diagnosis support, intelligent surgical assistance, personalized treatment planning, and drug development, making it a core area of research in smart healthcare. However, the main challenge in medical image semantic segmentation lies in the accuracy bottleneck, primarily due to the low interactivity of feature information and the lack of deep exploration of local features during feature fusion. METHODS To address this issue, a novel approach called Twisted Information-sharing Pattern for Multi-branched Network (TP-MNet) has been proposed. This architecture facilitates the mutual transfer of features among neighboring branches at the next level, breaking the barrier of semantic isolation and achieving the goal of semantic fusion. Additionally, performing a secondary feature mining during the transfer process effectively enhances the detection accuracy. Building upon the Twisted Pattern transmission in the encoding and decoding stages, enhanced and refined modules for feature fusion have been developed. These modules aim to capture key features of lesions by acquiring contextual semantic information in a broader context. RESULTS The experiments extensively and objectively validated the TP-MNet on 5 medical datasets and compared it with 21 other semantic segmentation models using 7 metrics. Through metric analysis, image comparisons, process examination, and ablation tests, the superiority of TP-MNet was convincingly demonstrated. Additionally, further investigations were conducted to explore the limitations of TP-MNet, thereby clarifying the practical utility of the Twisted Information-sharing Pattern. CONCLUSIONS TP-MNet adopts the Twisted Information-sharing Pattern, leading to a substantial improvement in the semantic fusion effect and directly contributing to enhanced segmentation performance on medical images. Additionally, this semantic broadcasting mode not only underscores the importance of semantic fusion but also highlights a pivotal direction for the advancement of multi-branched architectures.
Collapse
Affiliation(s)
- Yuefei Wang
- College of Computer Science, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| | - Xi Yu
- Stirling College, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China.
| | - Yixi Yang
- Institute of Cancer Biology and Drug Discovery, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| | - Xiang Zhang
- College of Computer Science, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| | - Yutong Zhang
- College of Computer Science, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| | - Li Zhang
- College of Computer Science, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| | - Ronghui Feng
- Stirling College, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| | - Jiajing Xue
- Stirling College, Chengdu University, 2025 Chengluo Rd., Chengdu, Sichuan 610106, China
| |
Collapse
|
23
|
Xie Y, Zhang J, Liu L, Wang H, Ye Y, Verjans J, Xia Y. ReFs: A hybrid pre-training paradigm for 3D medical image segmentation. Med Image Anal 2024; 91:103023. [PMID: 37956551 DOI: 10.1016/j.media.2023.103023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Revised: 10/31/2023] [Accepted: 11/02/2023] [Indexed: 11/15/2023]
Abstract
Self-supervised learning (SSL) has achieved remarkable progress in medical image segmentation. The application of an SSL algorithm often follows a two-stage training process: using unlabeled data to perform label-free representation learning and fine-tuning the pre-trained model on the downstream tasks. One issue of this paradigm is that the SSL step is unaware of the downstream task, which may lead to sub-optimal feature representation for a target task. In this paper, we propose a hybrid pre-training paradigm that is driven by both self-supervised and supervised objectives. To achieve this, a supervised reference task is involved in self-supervised learning, aiming to improve the representation quality. Specifically, we employ the off-the-shelf medical image segmentation task as reference, and encourage learning a representation that (1) incurs low prediction loss on both SSL and reference tasks and (2) leads to a similar gradient when updating the feature extractor from either task. In this way, the reference task pilots SSL in the direction beneficial for the downstream segmentation. To this end, we propose a simple but effective gradient matching method to optimize the model towards a consistent direction, thus improving the compatibility of both SSL and supervised reference tasks. We call this hybrid pre-training paradigm reference-guided self-supervised learning (ReFs), and perform it on a large-scale unlabeled dataset and an additional reference dataset. The experimental results demonstrate its effectiveness on seven downstream medical image segmentation benchmarks.
Collapse
Affiliation(s)
| | - Jianpeng Zhang
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | | | - Hu Wang
- University of Adelaide, Australia
| | - Yiwen Ye
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | | | - Yong Xia
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China.
| |
Collapse
|
24
|
Mohammadi H, Thirunarayan K, Chen L. CVII: Enhancing Interpretability in Intelligent Sensor Systems via Computer Vision Interpretability Index. SENSORS (BASEL, SWITZERLAND) 2023; 23:9893. [PMID: 38139738 PMCID: PMC10747164 DOI: 10.3390/s23249893] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/16/2023] [Revised: 12/09/2023] [Accepted: 12/14/2023] [Indexed: 12/24/2023]
Abstract
In the realm of intelligent sensor systems, the dependence on Artificial Intelligence (AI) applications has heightened the importance of interpretability. This is particularly critical for opaque models such as Deep Neural Networks (DNN), as understanding their decisions is essential, not only for ethical and regulatory compliance, but also for fostering trust in AI-driven outcomes. This paper introduces the novel concept of a Computer Vision Interpretability Index (CVII). The CVII framework is designed to emulate human cognitive processes, specifically in tasks related to vision. It addresses the intricate challenge of quantifying interpretability, a task that is inherently subjective and varies across domains. The CVII is rigorously evaluated using a range of computer vision models applied to the COCO (Common Objects in Context) dataset, a widely recognized benchmark in the field. The findings established a robust correlation between image interpretability, model selection, and CVII scores. This research makes a substantial contribution to enhancing interpretability for human comprehension, as well as within intelligent sensor applications. By promoting transparency and reliability in AI-driven decision-making, the CVII framework empowers its stakeholders to effectively harness the full potential of AI technologies.
Collapse
Affiliation(s)
| | - Krishnaprasad Thirunarayan
- Department of Computer Science and Engineering, Wright State University, Dayton, OH 45435, USA; (H.M.); (L.C.)
| | | |
Collapse
|
25
|
Chen RQ, Joffe B, Casteleiro Costa P, Filan C, Wang B, Balakirsky S, Robles F, Roy K, Li J. Real-time semantic segmentation and anomaly detection of functional images for cell therapy manufacturing. Cytotherapy 2023; 25:1361-1369. [PMID: 37725031 PMCID: PMC10719834 DOI: 10.1016/j.jcyt.2023.08.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2023] [Revised: 08/21/2023] [Accepted: 08/24/2023] [Indexed: 09/21/2023]
Abstract
BACKGROUND AIMS Cell therapy is a promising treatment method that uses living cells to address a variety of diseases and conditions, including cardiovascular diseases, neurologic disorders and certain cancers. As interest in cell therapy grows, there is a need to shift to a more efficient, scalable and automated manufacturing process that can produce high-quality products at a lower cost. METHODS One way to achieve this is using non-invasive imaging and real-time image analysis techniques to monitor and control the manufacturing process. This work presents a machine learning-based image analysis pipeline that includes semantic segmentation and anomaly detection capabilities. RESULTS/CONCLUSIONS This method can be easily implemented even when given a limited dataset of annotated images, is able to segment cells and debris and can identify anomalies such as contamination or hardware failure.
Collapse
Affiliation(s)
- Rui Qi Chen
- H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Benjamin Joffe
- Georgia Tech Research Institute, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Paloma Casteleiro Costa
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Caroline Filan
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Bryan Wang
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Stephen Balakirsky
- Georgia Tech Research Institute, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Francisco Robles
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Krishnendu Roy
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Jing Li
- H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA.
| |
Collapse
|
26
|
Zamzmi G, Hsu LY, Rajaraman S, Li W, Sachdev V, Antani S. Evaluation of an artificial intelligence-based system for echocardiographic estimation of right atrial pressure. Int J Cardiovasc Imaging 2023; 39:2437-2450. [PMID: 37682418 PMCID: PMC10692014 DOI: 10.1007/s10554-023-02941-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 08/18/2023] [Indexed: 09/09/2023]
Abstract
Current noninvasive estimation of right atrial pressure (RAP) by inferior vena cava (IVC) measurement during echocardiography may have significant inter-rater variability due to different levels of observers' experience. Therefore, there is a need to develop new approaches to decrease the variability of IVC analysis and RAP estimation. This study aims to develop a fully automated artificial intelligence (AI)-based system for automated IVC analysis and RAP estimation. We presented a multi-stage AI system to identify the IVC view, select good quality images, delineate the IVC region and quantify its thickness, enabling temporal tracking of its diameter and collapsibility changes. The automated system was trained and tested on expert manual IVC and RAP reference measurements obtained from 255 patients during routine clinical workflow. The performance was evaluated using Pearson correlation and Bland-Altman analysis for IVC values, as well as macro accuracy and chi-square test for RAP values. Our results show an excellent agreement (r=0.96) between automatically computed versus manually measured IVC values, and Bland-Altman analysis showed a small bias of [Formula: see text]0.33 mm. Further, there is an excellent agreement ([Formula: see text]) between automatically estimated versus manually derived RAP values with a macro accuracy of 0.85. The proposed AI-based system accurately quantified IVC diameter, collapsibility index, both are used for RAP estimation. This automated system could serve as a paradigm to perform IVC analysis in routine echocardiography and support various cardiac diagnostic applications.
Collapse
Affiliation(s)
- Ghada Zamzmi
- National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, MD, 20894, USA
| | - Li-Yueh Hsu
- Clinical Center, National Institutes of Health, 10 Center Dr, Bethesda, MD, 20892, USA.
| | - Sivaramakrishnan Rajaraman
- National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, MD, 20894, USA
| | - Wen Li
- National Heart, Lung, and Blood Institute, National Institutes of Health, 9000 Rockville Pike, Bethesda, MD, 20892, USA
| | - Vandana Sachdev
- National Heart, Lung, and Blood Institute, National Institutes of Health, 9000 Rockville Pike, Bethesda, MD, 20892, USA
| | - Sameer Antani
- National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, MD, 20894, USA.
| |
Collapse
|
27
|
Lin G, Zhang Z, Long K, Zhang Y, Lu Y, Geng J, Zhou Z, Feng Q, Lu L, Cao L. GCLR: A self-supervised representation learning pretext task for glomerular filtration barrier segmentation in TEM images. Artif Intell Med 2023; 146:102720. [PMID: 38042604 DOI: 10.1016/j.artmed.2023.102720] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2022] [Revised: 10/04/2023] [Accepted: 11/14/2023] [Indexed: 12/04/2023]
Abstract
Automatic segmentation of the three substructures of glomerular filtration barrier (GFB) in transmission electron microscopy (TEM) images holds immense potential for aiding pathologists in renal disease diagnosis. However, the labor-intensive nature of manual annotations limits the training data for a fully-supervised deep learning model. Addressing this, our study harnesses self-supervised representation learning (SSRL) to utilize vast unlabeled data and mitigate annotation scarcity. Our innovation, GCLR, is a hybrid pixel-level pretext task tailored for GFB segmentation, integrating two subtasks: global clustering (GC) and local restoration (LR). GC captures the overall GFB by learning global context representations, while LR refines three substructures by learning local detail representations. Experiments on 18,928 unlabeled glomerular TEM images for self-supervised pre-training and 311 labeled ones for fine-tuning demonstrate that our proposed GCLR obtains the state-of-the-art segmentation results for all three substructures of GFB with the Dice similarity coefficient of 86.56 ± 0.16%, 75.56 ± 0.36%, and 79.41 ± 0.16%, respectively, compared with other representative self-supervised pretext tasks. Our proposed GCLR also outperforms the fully-supervised pre-training methods based on the three large-scale public datasets - MitoEM, COCO, and ImageNet - with less training data and time.
Collapse
Affiliation(s)
- Guoyu Lin
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Zhentai Zhang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Kaixing Long
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Yiwen Zhang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Yanmeng Lu
- Central Laboratory, Southern Medical University, Guangzhou, 510515, China
| | - Jian Geng
- Department of Pathology, School of Basic Medical Sciences, Southern Medical University, Guangzhou, 510515, China; Guangzhou Huayin Medical Laboratory Center, Guangzhou, 510515, China
| | - Zhitao Zhou
- Central Laboratory, Southern Medical University, Guangzhou, 510515, China
| | - Qianjin Feng
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China
| | - Lijun Lu
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China.
| | - Lei Cao
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510515, China.
| |
Collapse
|
28
|
Wang G, Ren TA, Lai J, Bai L, Ren H. Domain adaptive Sim-to-Real segmentation of oropharyngeal organs. Med Biol Eng Comput 2023; 61:2745-2755. [PMID: 37462791 DOI: 10.1007/s11517-023-02877-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2023] [Accepted: 06/25/2023] [Indexed: 09/28/2023]
Abstract
Video-assisted transoral tracheal intubation (TI) necessitates using an endoscope that helps the physician insert a tracheal tube into the glottis instead of the esophagus. The growing trend of robotic-assisted TI would require a medical robot to distinguish anatomical features like an experienced physician which can be imitated by utilizing supervised deep-learning techniques. However, the real datasets of oropharyngeal organs are often inaccessible due to limited open-source data and patient privacy. In this work, we propose a domain adaptive Sim-to-Real framework called IoU-Ranking Blend-ArtFlow (IRB-AF) for image segmentation of oropharyngeal organs. The framework includes an image blending strategy called IoU-Ranking Blend (IRB) and style-transfer method ArtFlow. Here, IRB alleviates the problem of poor segmentation performance caused by significant datasets domain differences, while ArtFlow is introduced to reduce the discrepancies between datasets further. A virtual oropharynx image dataset generated by the SOFA framework is used as the learning subject for semantic segmentation to deal with the limited availability of actual endoscopic images. We adapted IRB-AF with the state-of-the-art domain adaptive segmentation models. The results demonstrate the superior performance of our approach in further improving the segmentation accuracy and training stability.
Collapse
Affiliation(s)
- Guankun Wang
- Department of Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., 999077, Hong Kong, China
| | - Tian-Ao Ren
- College of Mechanical and Electrical Engineering, Beijing University of Chemical Technology, 15 Beisanhuan East Rd., Chaoyang, 100029, Beijing, China
- Shenzhen Research Institute, The Chinese University of Hong Kong, 2 Yuexing Rd., Nanshan, Shenzhen, 518057, Guangdong, China
| | - Jiewen Lai
- Department of Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., 999077, Hong Kong, China
| | - Long Bai
- Department of Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., 999077, Hong Kong, China
| | - Hongliang Ren
- Department of Electronic Engineering, The Chinese University of Hong Kong, Shatin, N.T., 999077, Hong Kong, China.
| |
Collapse
|
29
|
M GJ, S B. DeepNet model empowered cuckoo search algorithm for the effective identification of lung cancer nodules. FRONTIERS IN MEDICAL TECHNOLOGY 2023; 5:1157919. [PMID: 37752910 PMCID: PMC10518616 DOI: 10.3389/fmedt.2023.1157919] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 08/22/2023] [Indexed: 09/28/2023] Open
Abstract
Introduction Globally, lung cancer is a highly harmful type of cancer. An efficient diagnosis system can enable pathologists to recognize the type and nature of lung nodules and the mode of therapy to increase the patient's chance of survival. Hence, implementing an automatic and reliable system to segment lung nodules from a computed tomography (CT) image is useful in the medical industry. Methods This study develops a novel fully convolutional deep neural network (hereafter called DeepNet) model for segmenting lung nodules from CT scans. This model includes an encoder/decoder network that achieves pixel-wise image segmentation. The encoder network exploits a Visual Geometry Group (VGG-19) model as a base architecture, while the decoder network exploits 16 upsampling and deconvolution modules. The encoder used in this model has a very flexible structural design that can be modified and trained for any resolution based on the size of input scans. The decoder network upsamples and maps the low-resolution attributes of the encoder. Thus, there is a considerable drop in the number of variables used for the learning process as the network recycles the pooling indices of the encoder for segmentation. The Thresholding method and the cuckoo search algorithm determines the most useful features when categorizing cancer nodules. Results and discussion The effectiveness of the intended DeepNet model is cautiously assessed on the real-world database known as The Cancer Imaging Archive (TCIA) dataset and its effectiveness is demonstrated by comparing its representation with some other modern segmentation models in terms of selected performance measures. The empirical analysis reveals that DeepNet significantly outperforms other prevalent segmentation algorithms with 0.962 ± 0.023% of volume error, 0.968 ± 0.011 of dice similarity coefficient, 0.856 ± 0.011 of Jaccard similarity index, and 0.045 ± 0.005s average processing time.
Collapse
Affiliation(s)
- Grace John M
- Department of Electronics and Communication, Karpagam Academy of Higher Education, Coimbatore, India
| | | |
Collapse
|
30
|
Balaji K. Image Augmentation based on Variational Autoencoder for Breast Tumor Segmentation. Acad Radiol 2023; 30 Suppl 2:S172-S183. [PMID: 36804294 DOI: 10.1016/j.acra.2022.12.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 12/18/2022] [Accepted: 12/21/2022] [Indexed: 02/18/2023]
Abstract
RATIONALE AND OBJECTIVES Breast tumor segmentation based on Dynamic Contrast-Enhanced Magnetic Resonance Imaging is significant step for computable radiomics analysis of breast cancer. Manual tumor annotation is time-consuming process and involves medical acquaintance, biased, inclined to error, and inter-user discrepancy. A number of modern trainings have revealed the capability of deep learning representations in image segmentation. MATERIALS AND METHODS Here, we describe a 3D Connected-UNets for tumor segmentation from 3D Magnetic Resonance Imagings based on encoder-decoder architecture. Due to a restricted training dataset size, a variational auto-encoder outlet is supplementary to renovate the input image itself in order to identify the shared decoder and execute additional controls on its layers. Based on initial segmentation of Connected-UNets, fully connected 3D provisional unsystematic domain is used to enhance segmentation outcomes by discovering 2D neighbor areas and 3D volume statistics. Moreover, 3D connected modules evaluation is used to endure around large modules and decrease segmentation noise. RESULTS The proposed method has been assessed on two widely offered datasets, explicitly INbreast and the curated breast imaging subset of digital database for screening mammography The proposed model has also been estimated using a private dataset. CONCLUSION The experimental results show that the proposed model outperforms the state-of-the-art methods for breast tumor segmentation.
Collapse
Affiliation(s)
- K Balaji
- School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamilnadu, 632014 India.
| |
Collapse
|
31
|
Goudarzi S, Whyte J, Boily M, Towers A, Kilgour RD, Rivaz H. Segmentation of Arm Ultrasound Images in Breast Cancer-Related Lymphedema: A Database and Deep Learning Algorithm. IEEE Trans Biomed Eng 2023; 70:2552-2563. [PMID: 37028332 DOI: 10.1109/tbme.2023.3253646] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/09/2023]
Abstract
OBJECTIVE Breast cancer treatment often causes the removal of or damage to lymph nodes of the patient's lymphatic drainage system. This side effect is the origin of Breast Cancer-Related Lymphedema (BCRL), referring to a noticeable increase in excess arm volume. Ultrasound imaging is a preferred modality for the diagnosis and progression monitoring of BCRL because of its low cost, safety, and portability. As the affected and unaffected arms look similar in B-mode ultrasound images, the thickness of the skin, subcutaneous fat, and muscle have been shown to be important biomarkers for this task. The segmentation masks are also helpful in monitoring the longitudinal changes in morphology and mechanical properties of tissue layers. METHODS For the first time, a publicly available ultrasound dataset containing the Radio-Frequency (RF) data of 39 subjects and manual segmentation masks by two experts, are provided. Inter- and intra-observer reproducibility studies performed on the segmentation maps show a high Dice Score Coefficient (DSC) of 0.94±0.08 and 0.92±0.06, respectively. Gated Shape Convolutional Neural Network (GSCNN) is modified for precise automatic segmentation of tissue layers, and its generalization performance is improved by the CutMix augmentation strategy. RESULTS We got an average DSC of 0.87±0.11 on the test set, which confirms the high performance of the method. CONCLUSION Automatic segmentation can pave the way for convenient and accessible staging of BCRL, and our dataset can facilitate development and validation of those methods. SIGNIFICANCE Timely diagnosis and treatment of BCRL have crucial importance in preventing irreversible damage.
Collapse
|
32
|
Zhou P, Liu X, Xiong J. Skin lesion image segmentation based on lightweight multi-scale U-shaped network. Biomed Phys Eng Express 2023; 9:055021. [PMID: 37413980 DOI: 10.1088/2057-1976/ace4d0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Accepted: 07/06/2023] [Indexed: 07/08/2023]
Abstract
UNet, and more recently medical image segmentation methods, utilize many parameters and computational quantities to achieve higher performance. However, due to the increasing demand for real-time medical image segmentation tasks, it is important to trade between accuracy rates and computational complexity. To this end, we propose a lightweight multi-scale U-shaped network (LMUNet), a multi-scale inverted residual and an asymmetric atrous spatial pyramid pooling-based network for skin lesion image segmentation. We test LMUNet on multiple medical image segmentation datasets, which show that it reduces the number of parameters by 67X and decreases the computational complexity by 48X while obtaining better performance over the partial lightweight networks.
Collapse
Affiliation(s)
- Pengfei Zhou
- School of Electronic and Optical Engineering, Nanjing University of Science and Technology, 200 Xiaolingwei, Nanjing, 210094, People's Republic of China
| | - Xuefeng Liu
- School of Electronic and Optical Engineering, Nanjing University of Science and Technology, 200 Xiaolingwei, Nanjing, 210094, People's Republic of China
| | - Jichuan Xiong
- School of Electronic and Optical Engineering, Nanjing University of Science and Technology, 200 Xiaolingwei, Nanjing, 210094, People's Republic of China
| |
Collapse
|
33
|
Mu J, Lin Y, Meng X, Fan J, Ai D, Chen D, Qiu H, Yang J, Gu Y. M-CSAFN: Multi-Color Space Adaptive Fusion Network for Automated Port-Wine Stains Segmentation. IEEE J Biomed Health Inform 2023; 27:3924-3935. [PMID: 37027679 DOI: 10.1109/jbhi.2023.3247479] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Abstract
Automatic segmentation of port-wine stains (PWS) from clinical images is critical for accurate diagnosis and objective assessment of PWS. However, this is a challenging task due to the color heterogeneity, low contrast, and indistinguishable appearance of PWS lesions. To address such challenges, we propose a novel multi-color space adaptive fusion network (M-CSAFN) for PWS segmentation. First, a multi-branch detection model is constructed based on six typical color spaces, which utilizes rich color texture information to highlight the difference between lesions and surrounding tissues. Second, an adaptive fusion strategy is used to fuse complementary predictions, which address the significant differences within the lesions caused by color heterogeneity. Third, a structural similarity loss with color information is proposed to measure the detail error between predicted lesions and truth lesions. Additionally, a PWS clinical dataset consisting of 1413 image pairs was established for the development and evaluation of PWS segmentation algorithms. To verify the effectiveness and superiority of the proposed method, we compared it with other state-of-the-art methods on our collected dataset and four publicly available skin lesion datasets (ISIC 2016, ISIC 2017, ISIC 2018, and PH2). The experimental results show that our method achieves remarkable performance in comparison with other state-of-the-art methods on our collected dataset, achieving 92.29% and 86.14% on Dice and Jaccard metrics, respectively. Comparative experiments on other datasets also confirmed the reliability and potential capability of M-CSAFN in skin lesion segmentation.
Collapse
|
34
|
Feyisa DW, Ayano YM, Debelee TG, Schwenker F. Weak Localization of Radiographic Manifestations in Pulmonary Tuberculosis from Chest X-ray: A Systematic Review. SENSORS (BASEL, SWITZERLAND) 2023; 23:6781. [PMID: 37571564 PMCID: PMC10422452 DOI: 10.3390/s23156781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Revised: 07/03/2023] [Accepted: 07/14/2023] [Indexed: 08/13/2023]
Abstract
Pulmonary tuberculosis (PTB) is a bacterial infection that affects the lung. PTB remains one of the infectious diseases with the highest global mortalities. Chest radiography is a technique that is often employed in the diagnosis of PTB. Radiologists identify the severity and stage of PTB by inspecting radiographic features in the patient's chest X-ray (CXR). The most common radiographic features seen on CXRs include cavitation, consolidation, masses, pleural effusion, calcification, and nodules. Identifying these CXR features will help physicians in diagnosing a patient. However, identifying these radiographic features for intricate disorders is challenging, and the accuracy depends on the radiologist's experience and level of expertise. So, researchers have proposed deep learning (DL) techniques to detect and mark areas of tuberculosis infection in CXRs. DL models have been proposed in the literature because of their inherent capacity to detect diseases and segment the manifestation regions from medical images. However, fully supervised semantic segmentation requires several pixel-by-pixel labeled images. The annotation of such a large amount of data by trained physicians has some challenges. First, the annotation requires a significant amount of time. Second, the cost of hiring trained physicians is expensive. In addition, the subjectivity of medical data poses a difficulty in having standardized annotation. As a result, there is increasing interest in weak localization techniques. Therefore, in this review, we identify methods employed in the weakly supervised segmentation and localization of radiographic manifestations of pulmonary tuberculosis from chest X-rays. First, we identify the most commonly used public chest X-ray datasets for tuberculosis identification. Following that, we discuss the approaches for weakly localizing tuberculosis radiographic manifestations in chest X-rays. The weakly supervised localization of PTB can highlight the region of the chest X-ray image that contributed the most to the DL model's classification output and help pinpoint the diseased area. Finally, we discuss the limitations and challenges of weakly supervised techniques in localizing TB manifestations regions in chest X-ray images.
Collapse
Affiliation(s)
- Degaga Wolde Feyisa
- Ethiopian Artificial Intelligence Institute, Addis Ababa P.O. Box 40782, Ethiopia; (D.W.F.); (Y.M.A.); (T.G.D.)
| | - Yehualashet Megersa Ayano
- Ethiopian Artificial Intelligence Institute, Addis Ababa P.O. Box 40782, Ethiopia; (D.W.F.); (Y.M.A.); (T.G.D.)
| | - Taye Girma Debelee
- Ethiopian Artificial Intelligence Institute, Addis Ababa P.O. Box 40782, Ethiopia; (D.W.F.); (Y.M.A.); (T.G.D.)
- Department of Electrical and Computer Engineering, Addis Ababa Science and Technology University, Addis Ababa P.O. Box 120611, Ethiopia
| | - Friedhelm Schwenker
- Institute of Neural Information Processing, Ulm University, 89069 Ulm, Germany
| |
Collapse
|
35
|
Spahr A, Ståhle J, Wang C, Kaijser M. Label-efficient deep semantic segmentation of intracranial hemorrhages in CT-scans. FRONTIERS IN NEUROIMAGING 2023; 2:1157565. [PMID: 37554648 PMCID: PMC10406224 DOI: 10.3389/fnimg.2023.1157565] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/02/2023] [Accepted: 06/22/2023] [Indexed: 08/10/2023]
Abstract
Intracranial hemorrhage (ICH) is a common finding in traumatic brain injury (TBI) and computed tomography (CT) is considered the gold standard for diagnosis. Automated detection of ICH provides clinical value in diagnostics and in the ability to feed robust quantification measures into future prediction models. Several studies have explored ICH detection and segmentation but the research process is somewhat hindered due to a lack of open large and labeled datasets, making validation and comparison almost impossible. The complexity of the task is further challenged by the heterogeneity of ICH patterns, requiring a large number of labeled data to train robust and reliable models. Consequently, due to the labeling cost, there is a need for label-efficient algorithms that can exploit easily available unlabeled or weakly-labeled data. Our aims for this study were to evaluate whether transfer learning can improve ICH segmentation performance and to compare a variety of transfer learning approaches that harness unlabeled and weakly-labeled data. Three self-supervised and three weakly-supervised transfer learning approaches were explored. To be used in our comparisons, we also manually labeled a dataset of 51 CT scans. We demonstrate that transfer learning improves ICH segmentation performance on both datasets. Unlike most studies on ICH segmentation our work relies exclusively on publicly available datasets, allowing for easy comparison of performances in future studies. To further promote comparison between studies, we also present a new public dataset of ICH-labeled CT scans, Seq-CQ500.
Collapse
Affiliation(s)
- Antoine Spahr
- Department of Biomedical Engineering and Health Systems, KTH Royal Institute of Technology, Stockholm, Sweden
- Signal Processing Laboratory (LTS5), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- CHUV—Lausanne University Hospital, Lausanne, Switzerland
| | - Jennifer Ståhle
- Department of Clinical Neuroscience, Karolinska Institutet, Solna, Stockholm, Sweden
- Department of Neuroradiology, Karolinska University Hospital, Stockholm, Sweden
| | - Chunliang Wang
- Department of Biomedical Engineering and Health Systems, KTH Royal Institute of Technology, Stockholm, Sweden
| | - Magnus Kaijser
- Department of Neuroradiology, Karolinska University Hospital, Stockholm, Sweden
- Institute of Environmental Medicine, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
36
|
Jiang B, Chen Z, Tan J, Qu R, Li C, Li Y. A Real-Time Semantic Segmentation Method Based on STDC-CT for Recognizing UAV Emergency Landing Zones. SENSORS (BASEL, SWITZERLAND) 2023; 23:6514. [PMID: 37514812 PMCID: PMC10386455 DOI: 10.3390/s23146514] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 07/14/2023] [Accepted: 07/17/2023] [Indexed: 07/30/2023]
Abstract
With the accelerated growth of the UAV industry, researchers are paying close attention to the flight safety of UAVs. When a UAV loses its GPS signal or encounters unusual conditions, it must perform an emergency landing. Therefore, real-time recognition of emergency landing zones on the ground is an important research topic. This paper employs a semantic segmentation approach for recognizing emergency landing zones. First, we created a dataset of UAV aerial images, denoted as UAV-City. A total of 600 UAV aerial images were densely annotated with 12 semantic categories. Given the complex backgrounds, diverse categories, and small UAV aerial image targets, we propose the STDC-CT real-time semantic segmentation network for UAV recognition of emergency landing zones. The STDC-CT network is composed of three branches: detail guidance, small object attention extractor, and multi-scale contextual information. The fusion of detailed and contextual information branches is guided by small object attention. We conducted extensive experiments on the UAV-City, Cityscapes, and UAVid datasets to demonstrate that the STDC-CT method is superior for attaining a balance between segmentation accuracy and inference speed. Our method improves the segmentation accuracy of small objects and achieves 76.5% mIoU on the Cityscapes test set at 122.6 FPS, 68.4% mIoU on the UAVid test set, and 67.3% mIoU on the UAV-City dataset at 196.8 FPS on an NVIDIA RTX 2080Ti GPU. Finally, we deployed the STDC-CT model on Jetson TX2 for testing in a real-world environment, attaining real-time semantic segmentation with an average inference speed of 58.32 ms per image.
Collapse
Affiliation(s)
- Bo Jiang
- College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China
| | - Zhonghui Chen
- College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China
| | - Jintao Tan
- College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China
| | - Ruokun Qu
- College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China
| | - Chenglong Li
- College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China
| | - Yandong Li
- College of Air Traffic Management, Civil Aviation Flight University of China, Guanghan 618307, China
| |
Collapse
|
37
|
Pan X, Zhu H, Du J, Hu G, Han B, Jia Y. MS-DCANet: A Novel Segmentation Network For Multi-Modality COVID-19 Medical Images. J Multidiscip Healthc 2023; 16:2023-2043. [PMID: 37489133 PMCID: PMC10363353 DOI: 10.2147/jmdh.s417068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 07/10/2023] [Indexed: 07/26/2023] Open
Abstract
Aim The Coronavirus Disease 2019 (COVID-19) pandemic has increased the public health burden and brought profound disaster to humans. For the particularity of the COVID-19 medical images with blurred boundaries, low contrast and different infection sites, some researchers have improved the accuracy by adding more complexity. Also, they overlook the complexity of lesions, which hinder their ability to capture the relationship between segmentation sites and the background, as well as the edge contours and global context. However, increasing the computational complexity, parameters and inference speed is unfavorable for model transfer from laboratory to clinic. A perfect segmentation network needs to balance the above three factors completely. To solve the above issues, this paper propose a symmetric automatic segmentation framework named MS-DCANet. We introduce Tokenized MLP block, a novel attention scheme that uses a shift-window mechanism to conditionally fuse local and global features to get more continuous boundaries and spatial positioning capabilities. It has greater understanding of irregular lesion contours. MS-DCANet also uses several Dual Channel blocks and a Res-ASPP block to improve the ability to recognize small targets. On multi-modality COVID-19 tasks, MS-DCANet achieved state-of-the-art performance compared with other baselines. It can well trade off the accuracy and complexity. To prove the strong generalization ability of our proposed model, we apply it to other tasks (ISIC 2018 and BAA) and achieve satisfactory results. Patients The X-ray dataset from Qatar University which contains 3379 cases for light, normal and heavy COVID-19 lung infection. The CT dataset contains the scans of 10 patient cases with COVID-19, a total of 1562 CT axial slices. The BAA dataset is obtained from the hospital and includes 387 original images. The ISIC 2018 dataset is from the International Skin Imaging Collaborative (ISIC) containing 2594 original images. Results The proposed MS-DCANet achieved evaluation metrics (MIOU) of 73.86, 97.26, 89.54, and 79.54 on the four datasets, respectively, far exceeding other current state-of-the art baselines. Conclusion The proposed MS-DCANet can help clinicians to automate the diagnosis of COVID-19 patients with different symptoms.
Collapse
Affiliation(s)
- Xiaoyu Pan
- College of Medical Informatics, Chongqing Medical University, Chongqing, People’s Republic of China
| | - Huazheng Zhu
- College of Intelligent Technology and Engineering, Chongqing University of Science and Technology, Chongqing, People’s Republic of China
| | - Jinglong Du
- College of Medical Informatics, Chongqing Medical University, Chongqing, People’s Republic of China
| | - Guangtao Hu
- College of Medical Informatics, Chongqing Medical University, Chongqing, People’s Republic of China
| | - Baoru Han
- College of Medical Informatics, Chongqing Medical University, Chongqing, People’s Republic of China
| | - Yuanyuan Jia
- College of Medical Informatics, Chongqing Medical University, Chongqing, People’s Republic of China
| |
Collapse
|
38
|
Subramanya SK, Li R, Wang Y, Miyamoto H, Cui F. Deep learning for histopathological segmentation of smooth muscle in the urinary bladder. BMC Med Inform Decis Mak 2023; 23:122. [PMID: 37454065 PMCID: PMC10349433 DOI: 10.1186/s12911-023-02222-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Accepted: 07/03/2023] [Indexed: 07/18/2023] Open
Abstract
BACKGROUND Histological assessment of smooth muscle is a critical step particularly in staging malignant tumors in various internal organs including the urinary bladder. Nonetheless, manual segmentation and classification of muscular tissues by pathologists is often challenging. Therefore, a fully automated and reliable smooth muscle image segmentation system is in high demand. METHODS To characterize muscle fibers in the urinary bladder, including muscularis mucosa (MM) and muscularis propria (MP), we assessed 277 histological images from surgical specimens, using two well-known deep learning (DL) model groups, one including VGG16, ResNet18, SqueezeNet, and MobileNetV2, considered as a patch-based approach, and the other including U-Net, MA-Net, DeepLabv3 + , and FPN, considered as a pixel-based approach. All the trained models in both the groups were evaluated at pixel-level for their performance. RESULTS For segmenting MP and non-MP (including MM) regions, MobileNetV2, in the patch-based approach and U-Net, in the pixel-based approach outperformed their peers in the groups with mean Jaccard Index equal to 0.74 and 0.79, and mean Dice co-efficient equal to 0.82 and 0.88, respectively. We also demonstrated the strengths and weaknesses of the models in terms of speed and prediction accuracy. CONCLUSIONS This work not only creates a benchmark for future development of tools for the histological segmentation of smooth muscle but also provides an effective DL-based diagnostic system for accurate pathological staging of bladder cancer.
Collapse
Affiliation(s)
- Sridevi K Subramanya
- Thomas H. Gosnell School of Life Sciences, Rochester Institute of Technology, 1 Lomb Memorial Drive, Rochester, NY, 14623, USA
| | - Rui Li
- Golisano College of Computing and Information Sciences, Rochester Institute of Technology, 20 Lomb Memorial Drive, Rochester, NY, 14623, USA
| | - Ying Wang
- Department of Pathology and Laboratory Medicine, University of Rochester Medical Center, 601 Elmwood Avenue, Rochester, NY, 14642, USA
| | - Hiroshi Miyamoto
- Department of Pathology and Laboratory Medicine, University of Rochester Medical Center, 601 Elmwood Avenue, Rochester, NY, 14642, USA.
| | - Feng Cui
- Thomas H. Gosnell School of Life Sciences, Rochester Institute of Technology, 1 Lomb Memorial Drive, Rochester, NY, 14623, USA.
| |
Collapse
|
39
|
Jiang X, Hu Z, Wang S, Zhang Y. Deep Learning for Medical Image-Based Cancer Diagnosis. Cancers (Basel) 2023; 15:3608. [PMID: 37509272 PMCID: PMC10377683 DOI: 10.3390/cancers15143608] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 07/10/2023] [Accepted: 07/10/2023] [Indexed: 07/30/2023] Open
Abstract
(1) Background: The application of deep learning technology to realize cancer diagnosis based on medical images is one of the research hotspots in the field of artificial intelligence and computer vision. Due to the rapid development of deep learning methods, cancer diagnosis requires very high accuracy and timeliness as well as the inherent particularity and complexity of medical imaging. A comprehensive review of relevant studies is necessary to help readers better understand the current research status and ideas. (2) Methods: Five radiological images, including X-ray, ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI), positron emission computed tomography (PET), and histopathological images, are reviewed in this paper. The basic architecture of deep learning and classical pretrained models are comprehensively reviewed. In particular, advanced neural networks emerging in recent years, including transfer learning, ensemble learning (EL), graph neural network, and vision transformer (ViT), are introduced. Five overfitting prevention methods are summarized: batch normalization, dropout, weight initialization, and data augmentation. The application of deep learning technology in medical image-based cancer analysis is sorted out. (3) Results: Deep learning has achieved great success in medical image-based cancer diagnosis, showing good results in image classification, image reconstruction, image detection, image segmentation, image registration, and image synthesis. However, the lack of high-quality labeled datasets limits the role of deep learning and faces challenges in rare cancer diagnosis, multi-modal image fusion, model explainability, and generalization. (4) Conclusions: There is a need for more public standard databases for cancer. The pre-training model based on deep neural networks has the potential to be improved, and special attention should be paid to the research of multimodal data fusion and supervised paradigm. Technologies such as ViT, ensemble learning, and few-shot learning will bring surprises to cancer diagnosis based on medical images.
Collapse
Grants
- RM32G0178B8 BBSRC
- MC_PC_17171 MRC, UK
- RP202G0230 Royal Society, UK
- AA/18/3/34220 BHF, UK
- RM60G0680 Hope Foundation for Cancer Research, UK
- P202PF11 GCRF, UK
- RP202G0289 Sino-UK Industrial Fund, UK
- P202ED10, P202RE969 LIAS, UK
- P202RE237 Data Science Enhancement Fund, UK
- 24NN201 Fight for Sight, UK
- OP202006 Sino-UK Education Fund, UK
- RM32G0178B8 BBSRC, UK
- 2023SJZD125 Major project of philosophy and social science research in colleges and universities in Jiangsu Province, China
Collapse
Affiliation(s)
- Xiaoyan Jiang
- School of Mathematics and Information Science, Nanjing Normal University of Special Education, Nanjing 210038, China; (X.J.); (Z.H.)
| | - Zuojin Hu
- School of Mathematics and Information Science, Nanjing Normal University of Special Education, Nanjing 210038, China; (X.J.); (Z.H.)
| | - Shuihua Wang
- School of Computing and Mathematical Sciences, University of Leicester, Leicester LE1 7RH, UK;
| | - Yudong Zhang
- School of Computing and Mathematical Sciences, University of Leicester, Leicester LE1 7RH, UK;
| |
Collapse
|
40
|
Chen QQ, Sun ZH, Wei CF, Wu EQ, Ming D. Semi-Supervised 3D Medical Image Segmentation Based on Dual-Task Consistent Joint Learning and Task-Level Regularization. IEEE/ACM TRANSACTIONS ON COMPUTATIONAL BIOLOGY AND BIOINFORMATICS 2023; 20:2457-2467. [PMID: 35061590 DOI: 10.1109/tcbb.2022.3144428] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Semi-supervised learning has attracted wide attention from many researchers since its ability to utilize a few data with labels and relatively more data without labels to learn information. Some existing semi-supervised methods for medical image segmentation enforce the regularization of training by implicitly perturbing data or networks to perform the consistency. Most consistency regularization methods focus on data level or network structure level, and rarely of them focus on the task level. It may not directly lead to an improvement in task accuracy. To overcome the problem, this work proposes a semi-supervised dual-task consistent joint learning framework with task-level regularization for 3D medical image segmentation. Two branches are utilized to simultaneously predict the segmented and signed distance maps, and they can learn useful information from each other by constructing a consistency loss function between the two tasks. The segmentation branch learns rich information from both labeled and unlabeled data to strengthen the constraints on the geometric structure of the target. Experimental results on two benchmark datasets show that the proposed method can achieve better performance compared with other state-of-the-art works. It illustrates our method improves segmentation performance by utilizing unlabeled data and consistent regularization.
Collapse
|
41
|
Mirikharaji Z, Abhishek K, Bissoto A, Barata C, Avila S, Valle E, Celebi ME, Hamarneh G. A survey on deep learning for skin lesion segmentation. Med Image Anal 2023; 88:102863. [PMID: 37343323 DOI: 10.1016/j.media.2023.102863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 02/01/2023] [Accepted: 05/31/2023] [Indexed: 06/23/2023]
Abstract
Skin cancer is a major public health problem that could benefit from computer-aided diagnosis to reduce the burden of this common disease. Skin lesion segmentation from images is an important step toward achieving this goal. However, the presence of natural and artificial artifacts (e.g., hair and air bubbles), intrinsic factors (e.g., lesion shape and contrast), and variations in image acquisition conditions make skin lesion segmentation a challenging task. Recently, various researchers have explored the applicability of deep learning models to skin lesion segmentation. In this survey, we cross-examine 177 research papers that deal with deep learning-based segmentation of skin lesions. We analyze these works along several dimensions, including input data (datasets, preprocessing, and synthetic data generation), model design (architecture, modules, and losses), and evaluation aspects (data annotation requirements and segmentation performance). We discuss these dimensions both from the viewpoint of select seminal works, and from a systematic viewpoint, examining how those choices have influenced current trends, and how their limitations should be addressed. To facilitate comparisons, we summarize all examined works in a comprehensive table as well as an interactive table available online3.
Collapse
Affiliation(s)
- Zahra Mirikharaji
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Kumar Abhishek
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Alceu Bissoto
- RECOD.ai Lab, Institute of Computing, University of Campinas, Av. Albert Einstein 1251, Campinas 13083-852, Brazil
| | - Catarina Barata
- Institute for Systems and Robotics, Instituto Superior Técnico, Avenida Rovisco Pais, Lisbon 1049-001, Portugal
| | - Sandra Avila
- RECOD.ai Lab, Institute of Computing, University of Campinas, Av. Albert Einstein 1251, Campinas 13083-852, Brazil
| | - Eduardo Valle
- RECOD.ai Lab, School of Electrical and Computing Engineering, University of Campinas, Av. Albert Einstein 400, Campinas 13083-952, Brazil
| | - M Emre Celebi
- Department of Computer Science and Engineering, University of Central Arkansas, 201 Donaghey Ave., Conway, AR 72035, USA.
| | - Ghassan Hamarneh
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada.
| |
Collapse
|
42
|
Liu CC, Abdelhafez YG, Yap SP, Acquafredda F, Schirò S, Wong AL, Sarohia D, Bateni C, Darrow MA, Guindani M, Lee S, Zhang M, Moawad AW, Ng QKT, Shere L, Elsayes KM, Maroldi R, Link TM, Nardo L, Qi J. AI-Based Automated Lipomatous Tumor Segmentation in MR Images: Ensemble Solution to Heterogeneous Data. J Digit Imaging 2023; 36:1049-1059. [PMID: 36854923 PMCID: PMC10287587 DOI: 10.1007/s10278-023-00785-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2022] [Revised: 01/20/2023] [Accepted: 01/20/2023] [Indexed: 03/02/2023] Open
Abstract
Deep learning (DL) has been proposed to automate image segmentation and provide accuracy, consistency, and efficiency. Accurate segmentation of lipomatous tumors (LTs) is critical for correct tumor radiomics analysis and localization. The major challenge of this task is data heterogeneity, including tumor morphological characteristics and multicenter scanning protocols. To mitigate the issue, we aimed to develop a DL-based Super Learner (SL) ensemble framework with different data correction and normalization methods. Pathologically proven LTs on pre-operative T1-weighted/proton-density MR images of 185 patients were manually segmented. The LTs were categorized by tumor locations as distal upper limb (DUL), distal lower limb (DLL), proximal upper limb (PUL), proximal lower limb (PLL), or Trunk (T) and grouped by 80%/9%/11% for training, validation and testing. Six configurations of correction/normalization were applied to data for fivefold-cross-validation trainings, resulting in 30 base learners (BLs). A SL was obtained from the BLs by optimizing SL weights. The performance was evaluated by dice-similarity-coefficient (DSC), sensitivity, specificity, and Hausdorff distance (HD95). For predictions of the BLs, the average DSC, sensitivity, and specificity from the testing data were 0.72 [Formula: see text] 0.16, 0.73 [Formula: see text] 0.168, and 0.99 [Formula: see text] 0.012, respectively, while for SL predictions were 0.80 [Formula: see text] 0.184, 0.78 [Formula: see text] 0.193, and 1.00 [Formula: see text] 0.010. The average HD95 of the BLs were 11.5 (DUL), 23.2 (DLL), 25.9 (PUL), 32.1 (PLL), and 47.9 (T) mm, whereas of SL were 1.7, 8.4, 15.9, 2.2, and 36.6 mm, respectively. The proposed method could improve the segmentation accuracy and mitigate the performance instability and data heterogeneity aiding the differential diagnosis of LTs in real clinical situations.
Collapse
Affiliation(s)
- Chih-Chieh Liu
- Department of Biomedical Engineering, University of California, Davis, CA, USA
| | - Yasser G Abdelhafez
- Department of Radiology, UC Davis Health, Sacramento, CA, USA
- Radiotherapy and Nuclear Medicine Department, South Egypt Cancer Institute, Assiut University, Assiut, Egypt
| | - S Paran Yap
- Department of Radiology, UC Davis Health, Sacramento, CA, USA
| | | | - Silvia Schirò
- Section of Radiology, Department of Medicine and Surgery (DiMeC), University of Parma, Parma, Italy
| | - Andrew L Wong
- Department of Radiology, UC Davis Health, Sacramento, CA, USA
| | - Dani Sarohia
- Department of Radiology, UC Davis Health, Sacramento, CA, USA
| | - Cyrus Bateni
- Department of Radiology, UC Davis Health, Sacramento, CA, USA
| | - Morgan A Darrow
- Pathology and Laboratory Medicine, University of California Davis, Sacramento, CA, USA
| | - Michele Guindani
- Department of Biostatistics, Fielding School of Public Health, University of California, Los Angeles, CA, USA
| | - Sonia Lee
- Department of Radiological Sciences, University of California, Irvine, CA, USA
| | - Michelle Zhang
- Department of Diagnostic Radiology, McGill University Health Center, Montreal, Canada
| | - Ahmed W Moawad
- Department of Diagnostic Imaging, University of Texas MD Anderson Cancer Center, Houston, TX, USA
- Department of Diagnostic Radiology, Mercy Catholic Medical Center, Darby, PA, USA
| | | | - Layla Shere
- Department of Diagnostic Imaging, University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | - Khaled M Elsayes
- Department of Diagnostic Imaging, University of Texas MD Anderson Cancer Center, Houston, TX, USA
| | | | - Thomas M Link
- Department of Radiology and Biomedical Imaging, University of California, San Francisco, CA, USA
| | - Lorenzo Nardo
- Department of Radiology, UC Davis Health, Sacramento, CA, USA
| | - Jinyi Qi
- Department of Biomedical Engineering, University of California, Davis, CA, USA.
| |
Collapse
|
43
|
Zhukov O, He C, Soylu-Kucharz R, Cai C, Lauritzen AD, Aldana BI, Björkqvist M, Lauritzen M, Kucharz K. Preserved blood-brain barrier and neurovascular coupling in female 5xFAD model of Alzheimer's disease. Front Aging Neurosci 2023; 15:1089005. [PMID: 37261266 PMCID: PMC10228387 DOI: 10.3389/fnagi.2023.1089005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Accepted: 04/17/2023] [Indexed: 06/02/2023] Open
Abstract
Introduction Dysfunction of the cerebral vasculature is considered one of the key components of Alzheimer's disease (AD), but the mechanisms affecting individual brain vessels are poorly understood. Methods Here, using in vivo two-photon microscopy in superficial cortical layers and ex vivo imaging across brain regions, we characterized blood-brain barrier (BBB) function and neurovascular coupling (NVC) at the level of individual brain vessels in adult female 5xFAD mice, an aggressive amyloid-β (Aβ) model of AD. Results We report a lack of abnormal increase in adsorptive-mediated transcytosis of albumin and preserved paracellular barrier for fibrinogen and small molecules despite an extensive load of Aβ. Likewise, the NVC responses to somatosensory stimulation were preserved at all regulatory segments of the microvasculature: penetrating arterioles, precapillary sphincters, and capillaries. Lastly, the Aβ plaques did not affect the density of capillary pericytes. Conclusion Our findings provide direct evidence of preserved microvascular function in the 5xFAD mice and highlight the critical dependence of the experimental outcomes on the choice of preclinical models of AD. We propose that the presence of parenchymal Aβ does not warrant BBB and NVC dysfunction and that the generalized view that microvascular impairment is inherent to Aβ aggregation may need to be revised.
Collapse
Affiliation(s)
- Oleg Zhukov
- Department of Neuroscience, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Chen He
- Department of Neuroscience, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Rana Soylu-Kucharz
- Biomarkers in Brain Disease, Department of Experimental Medical Sciences, Lund University, Lund, Sweden
| | - Changsi Cai
- Department of Neuroscience, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | | | - Blanca Irene Aldana
- Department of Drug Design and Pharmacology, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Maria Björkqvist
- Biomarkers in Brain Disease, Department of Experimental Medical Sciences, Lund University, Lund, Sweden
| | - Martin Lauritzen
- Department of Neuroscience, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- Department of Clinical Neurophysiology, Rigshospitalet, Copenhagen, Denmark
| | - Krzysztof Kucharz
- Department of Neuroscience, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
44
|
Lefebvre AL, Yamamoto CAP, Shade JK, Bradley RP, Yu RA, Ali RL, Popescu DM, Prakosa A, Kholmovski EG, Trayanova NA. LASSNet: A Four Steps Deep Neural Network for Left Atrial Segmentation and Scar Quantification. LEFT ATRIAL AND SCAR QUANTIFICATION AND SEGMENTATION : FIRST CHALLENGE, LASCARQS 2022 HELD IN CONJUNCTION WITH MICCAI 2022, SINGAPORE, SEPTEMBER 18, 2022, PROCEEDINGS 2023; 13586:1-15. [PMID: 37287952 PMCID: PMC10246435 DOI: 10.1007/978-3-031-31778-1_1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Accurate quantification of left atrium (LA) scar in patients with atrial fibrillation is essential to guide successful ablation strategies. Prior to LA scar quantification, a proper LA cavity segmentation is required to ensure exact location of scar. Both tasks can be extremely time-consuming and are subject to inter-observer disagreements when done manually. We developed and validated a deep neural network to automatically segment the LA cavity and the LA scar. The global architecture uses a multi-network sequential approach in two stages which segment the LA cavity and the LA Scar. Each stage has two steps: a region of interest Neural Network and a refined segmentation network. We analysed the performances of our network according to different parameters and applied data triaging. 200+ late gadolinium enhancement magnetic resonance images were provided by the LAScarQS 2022 Challenge. Finally, we compared our performances for scar quantification to the literature and demonstrated improved performances.
Collapse
Affiliation(s)
- Arthur L Lefebvre
- Faculté polytechnique de Mons, UMONS, Mons, Belgium
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
| | - Carolyna A P Yamamoto
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Julie K Shade
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
| | - Ryan P Bradley
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
| | - Rebecca A Yu
- Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Rheeda L Ali
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Dan M Popescu
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
| | - Adityo Prakosa
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
| | - Eugene G Kholmovski
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Natalia A Trayanova
- Alliance for Cardiovascular Diagnostic and Treatment Innovation (ADVANCE), Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| |
Collapse
|
45
|
Sang S, Zhou Y, Islam MT, Xing L. Small-Object Sensitive Segmentation Using Across Feature Map Attention. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:6289-6306. [PMID: 36178991 PMCID: PMC10823909 DOI: 10.1109/tpami.2022.3211171] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
Semantic segmentation is an important step in understanding the scene for many practical applications such as autonomous driving. Although Deep Convolutional Neural Networks-based methods have significantly improved segmentation accuracy, small/thin objects remain challenging to segment due to convolutional and pooling operations that result in information loss, especially for small objects. This article presents a novel attention-based method called Across Feature Map Attention (AFMA) to address this challenge. It quantifies the inner-relationship between small and large objects belonging to the same category by utilizing the different feature levels of the original image. The AFMA could compensate for the loss of high-level feature information of small objects and improve the small/thin object segmentation. Our method can be used as an efficient plug-in for a wide range of existing architectures and produces much more interpretable feature representation than former studies. Extensive experiments on eight widely used segmentation methods and other existing small-object segmentation models on CamVid and Cityscapes demonstrate that our method substantially and consistently improves the segmentation of small/thin objects.
Collapse
|
46
|
Guan X, Zhao Y, Nyatega CO, Li Q. Brain Tumor Segmentation Network with Multi-View Ensemble Discrimination and Kernel-Sharing Dilated Convolution. Brain Sci 2023; 13:brainsci13040650. [PMID: 37190614 DOI: 10.3390/brainsci13040650] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2023] [Revised: 03/22/2023] [Accepted: 04/07/2023] [Indexed: 05/17/2023] Open
Abstract
Accurate segmentation of brain tumors from magnetic resonance 3D images (MRI) is critical for clinical decisions and surgical planning. Radiologists usually separate and analyze brain tumors by combining images of axial, coronal, and sagittal views. However, traditional convolutional neural network (CNN) models tend to use information from only a single view or one by one. Moreover, the existing models adopt a multi-branch structure with different-size convolution kernels in parallel to adapt to various tumor sizes. However, the difference in the convolution kernels' parameters cannot precisely characterize the feature similarity of tumor lesion regions with various sizes, connectivity, and convexity. To address the above problems, we propose a hierarchical multi-view convolution method that decouples the standard 3D convolution into axial, coronal, and sagittal views to provide complementary-view features. Then, every pixel is classified by ensembling the discriminant results from the three views. Moreover, we propose a multi-branch kernel-sharing mechanism with a dilated rate to obtain parameter-consistent convolution kernels with different receptive fields. We use the BraTS2018 and BraTS2020 datasets for comparison experiments. The average Dice coefficients of the proposed network on the BraTS2020 dataset can reach 78.16%, 89.52%, and 83.05% for the enhancing tumor (ET), whole tumor (WT), and tumor core (TC), respectively, while the number of parameters is only 0.5 M. Compared with the baseline network for brain tumor segmentation, the accuracy was improved by 1.74%, 0.5%, and 2.19%, respectively.
Collapse
Affiliation(s)
- Xin Guan
- School of Microelectronics, Tianjin University, Tianjin 300072, China
| | - Yushan Zhao
- School of Microelectronics, Tianjin University, Tianjin 300072, China
| | - Charles Okanda Nyatega
- School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
| | - Qiang Li
- School of Microelectronics, Tianjin University, Tianjin 300072, China
| |
Collapse
|
47
|
Islam MT, Khan HA, Naveed K, Nauman A, Gulfam SM, Kim SW. LUVS-Net: A Lightweight U-Net Vessel Segmentor for Retinal Vasculature Detection in Fundus Images. ELECTRONICS 2023; 12:1786. [DOI: 10.3390/electronics12081786] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/01/2023]
Abstract
This paper presents LUVS-Net, which is a lightweight convolutional network for retinal vessel segmentation in fundus images that is designed for resource-constrained devices that are typically unable to meet the computational requirements of large neural networks. The computational challenges arise due to low-quality retinal images, wide variance in image acquisition conditions and disparities in intensity. Consequently, the training of existing segmentation methods requires a multitude of trainable parameters for the training of networks, resulting in computational complexity. The proposed Lightweight U-Net for Vessel Segmentation Network (LUVS-Net) can achieve high segmentation performance with only a few trainable parameters. This network uses an encoder–decoder framework in which edge data are transposed from the first layers of the encoder to the last layer of the decoder, massively improving the convergence latency. Additionally, LUVS-Net’s design allows for a dual-stream information flow both inside as well as outside of the encoder–decoder pair. The network width is enhanced using group convolutions, which allow the network to learn a larger number of low- and intermediate-level features. Spatial information loss is minimized using skip connections, and class imbalances are mitigated using dice loss for pixel-wise classification. The performance of the proposed network is evaluated on the publicly available retinal blood vessel datasets DRIVE, CHASE_DB1 and STARE. LUVS-Net proves to be quite competitive, outperforming alternative state-of-the-art segmentation methods and achieving comparable accuracy using trainable parameters that are reduced by two to three orders of magnitude compared with those of comparative state-of-the-art methods.
Collapse
Affiliation(s)
- Muhammad Talha Islam
- Department of Computer Science, COMSATS University Islamabad (CUI), Islamabad 45550, Pakistan
| | - Haroon Ahmed Khan
- Department of Electrical and Computer Engineering, COMSATS University Islamabad (CUI), Islamabad 45550, Pakistan
| | - Khuram Naveed
- Department of Electrical and Computer Engineering, COMSATS University Islamabad (CUI), Islamabad 45550, Pakistan
- Department of Electrical and Computer Engineering, Aarhus University, 8000 Aarhus, Denmark
| | - Ali Nauman
- Department of Information and Communication Engineering, Yeungnam University, Gyeongsan-si 38541, Republic of Korea
| | - Sardar Muhammad Gulfam
- Department of Electrical and Computer Engineering, Abbottabad Campus, COMSATS University Islamabad (CUI), Abbottabad 22060, Pakistan
| | - Sung Won Kim
- Department of Information and Communication Engineering, Yeungnam University, Gyeongsan-si 38541, Republic of Korea
| |
Collapse
|
48
|
Les T, Markiewicz T, Dziekiewicz M, Gallego J, Swiderska-Chadaj Z, Lorent M. Localization of spleen and kidney organs from CT scans based on classification of slices in rotational views. Sci Rep 2023; 13:5709. [PMID: 37029169 PMCID: PMC10082200 DOI: 10.1038/s41598-023-32741-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2022] [Accepted: 03/31/2023] [Indexed: 04/09/2023] Open
Abstract
This article presents a novel multiple organ localization and tracking technique applied to spleen and kidney regions in computed tomography images. The proposed solution is based on a unique approach to classify regions in different spatial projections (e.g., side projection) using convolutional neural networks. Our procedure merges classification results from different projection resulting in a 3D segmentation. The proposed system is able to recognize the contour of the organ with an accuracy of 88-89% depending on the body organ. Research has shown that the use of a single method can be useful for the detection of different organs: kidney and spleen. Our solution can compete with U-Net based solutions in terms of hardware requirements, as it has significantly lower demands. Additionally, it gives better results in small data sets. Another advantage of our solution is a significantly lower training time on an equally sized data set and more capabilities to parallelize calculations. The proposed system enables visualization, localization and tracking of organs and is therefore a valuable tool in medical diagnostic problems.
Collapse
Affiliation(s)
- Tomasz Les
- University of Technology, Plac Politechniki 1, 00-661, Warsaw, Poland.
| | - Tomasz Markiewicz
- University of Technology, Plac Politechniki 1, 00-661, Warsaw, Poland
- Military Institute of Medicine, Szaserów 128, 04-141, Warsaw, Poland
| | | | - Jaime Gallego
- University of Barcelona, Gran Via de les Corts Catalanes, 08007, Barcelona, Spain
| | | | - Malgorzata Lorent
- Military Institute of Medicine, Szaserów 128, 04-141, Warsaw, Poland
| |
Collapse
|
49
|
Yu H, Wang F, Teodoro G, Chen F, Guo X, Nickerson JM, Kong J. Self-supervised semantic segmentation of retinal pigment epithelium cells in flatmount fluorescent microscopy images. Bioinformatics 2023; 39:btad191. [PMID: 37067486 PMCID: PMC10139776 DOI: 10.1093/bioinformatics/btad191] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2022] [Revised: 03/23/2023] [Accepted: 04/10/2023] [Indexed: 04/18/2023] Open
Abstract
MOTIVATION Morphological analyses with flatmount fluorescent images are essential to retinal pigment epithelial (RPE) aging studies and thus require accurate RPE cell segmentation. Although rapid technology advances in deep learning semantic segmentation have achieved great success in many biomedical research, the performance of these supervised learning methods for RPE cell segmentation is still limited by inadequate training data with high-quality annotations. RESULTS To address this problem, we develop a Self-Supervised Semantic Segmentation (S4) method that utilizes a self-supervised learning strategy to train a semantic segmentation network with an encoder-decoder architecture. We employ a reconstruction and a pairwise representation loss to make the encoder extract structural information, while we create a morphology loss to produce the segmentation map. In addition, we develop a novel image augmentation algorithm (AugCut) to produce multiple views for self-supervised learning and enhance the network training performance. To validate the efficacy of our method, we applied our developed S4 method for RPE cell segmentation to a large set of flatmount fluorescent microscopy images, we compare our developed method for RPE cell segmentation with other state-of-the-art deep learning approaches. Compared with other state-of-the-art deep learning approaches, our method demonstrates better performance in both qualitative and quantitative evaluations, suggesting its promising potential to support large-scale cell morphological analyses in RPE aging investigations. AVAILABILITY AND IMPLEMENTATION The codes and the documentation are available at: https://github.com/jkonglab/S4_RPE.
Collapse
Affiliation(s)
- Hanyi Yu
- Department of Computer Science, Emory University, Atlanta, GA 30322, USA
| | - Fusheng Wang
- Department of Computer Science, Stony Brook University, Stony Brook, NY 11794, USA
| | - George Teodoro
- Department of Computer Science, Federal University of Minas Gerais, Belo Horizonte 31270, Brazil
| | - Fan Chen
- Huangpu Branch, Guangzhou Urban Planning & Design Survey Research Institute, Guangzhou 510060, China
| | - Xiaoyuan Guo
- Department of Computer Science, Emory University, Atlanta, GA 30322, USA
| | - John M Nickerson
- Department of Ophthalmology, Emory University, Atlanta, GA 30322, USA
| | - Jun Kong
- Department of Computer Science, Emory University, Atlanta, GA 30322, USA
- Department of Mathematics and Statistics, Georgia State University, Atlanta, GA 30303, USA
| |
Collapse
|
50
|
Deane O, Toth E, Yeo SH. Deep-SAGA: a deep-learning-based system for automatic gaze annotation from eye-tracking data. Behav Res Methods 2023; 55:1372-1391. [PMID: 35650384 PMCID: PMC10126076 DOI: 10.3758/s13428-022-01833-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/09/2022] [Indexed: 11/08/2022]
Abstract
With continued advancements in portable eye-tracker technology liberating experimenters from the restraints of artificial laboratory designs, research can now collect gaze data from real-world, natural navigation. However, the field lacks a robust method for achieving this, as past approaches relied upon the time-consuming manual annotation of eye-tracking data, while previous attempts at automation lack the necessary versatility for in-the-wild navigation trials consisting of complex and dynamic scenes. Here, we propose a system capable of informing researchers of where and what a user's gaze is focused upon at any one time. The system achieves this by first running footage recorded on a head-mounted camera through a deep-learning-based object detection algorithm called Masked Region-based Convolutional Neural Network (Mask R-CNN). The algorithm's output is combined with frame-by-frame gaze coordinates measured by an eye-tracking device synchronized with the head-mounted camera to detect and annotate, without any manual intervention, what a user looked at for each frame of the provided footage. The effectiveness of the presented methodology was legitimized by a comparison between the system output and that of manual coders. High levels of agreement between the two validated the system as a preferable data collection technique as it was capable of processing data at a significantly faster rate than its human counterpart. Support for the system's practicality was then further demonstrated via a case study exploring the mediatory effects of gaze behaviors on an environment-driven attentional bias.
Collapse
Affiliation(s)
- Oliver Deane
- School of Sport, Exercise and Rehabilitation Sciences, The University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK
| | - Eszter Toth
- School of Psychology, The University of Birmingham, Birmingham, UK
| | - Sang-Hoon Yeo
- School of Sport, Exercise and Rehabilitation Sciences, The University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK.
| |
Collapse
|