101
|
Oza P, Sindagi VA, Vs V, Patel VM. Unsupervised Domain Adaptation of Object Detectors: A Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:4018-4040. [PMID: 37030853 DOI: 10.1109/tpami.2022.3217046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Recent advances in deep learning have led to the development of accurate and efficient models for various computer vision applications such as classification, segmentation, and detection. However, learning highly accurate models relies on the availability of large-scale annotated datasets. Due to this, model performance drops drastically when evaluated on label-scarce datasets having visually distinct images, termed as domain adaptation problem. There are a plethora of works to adapt classification and segmentation models to label-scarce target dataset through unsupervised domain adaptation. Considering that detection is a fundamental task in computer vision, many recent works have focused on developing novel domain adaptive detection techniques. Here, we describe in detail the domain adaptation problem for detection and present an extensive survey of the various methods. Furthermore, we highlight strategies proposed and the associated shortcomings. Subsequently, we identify multiple aspects of the problem that are most promising for future research. We believe that this survey shall be valuable to the pattern recognition experts working in the fields of computer vision, biometrics, medical imaging, and autonomous navigation by introducing them to the problem, and familiarizing them with the current status of the progress while providing promising directions for future research.
Collapse
|
102
|
Wang Z, Liu Z, Yu J, Gao Y, Liu M. Multi-scale nested UNet with transformer for colorectal polyp segmentation. J Appl Clin Med Phys 2024; 25:e14351. [PMID: 38551396 PMCID: PMC11163511 DOI: 10.1002/acm2.14351] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 02/13/2024] [Accepted: 02/19/2024] [Indexed: 06/11/2024] Open
Abstract
BACKGROUND Polyp detection and localization are essential tasks for colonoscopy. U-shape network based convolutional neural networks have achieved remarkable segmentation performance for biomedical images, but lack of long-range dependencies modeling limits their receptive fields. PURPOSE Our goal was to develop and test a novel architecture for polyp segmentation, which takes advantage of learning local information with long-range dependencies modeling. METHODS A novel architecture combining with multi-scale nested UNet structure integrated transformer for polyp segmentation was developed. The proposed network takes advantage of both CNN and transformer to extract distinct feature information. The transformer layer is embedded between the encoder and decoder of a U-shape net to learn explicit global context and long-range semantic information. To address the challenging of variant polyp sizes, a MSFF unit was proposed to fuse features with multiple resolution. RESULTS Four public datasets and one in-house dataset were used to train and test the model performance. Ablation study was also conducted to verify each component of the model. For dataset Kvasir-SEG and CVC-ClinicDB, the proposed model achieved mean dice score of 0.942 and 0.950 respectively, which were more accurate than the other methods. To show the generalization of different methods, we processed two cross dataset validations, the proposed model achieved the highest mean dice score. The results demonstrate that the proposed network has powerful learning and generalization capability, significantly improving segmentation accuracy and outperforming state-of-the-art methods. CONCLUSIONS The proposed model produced more accurate polyp segmentation than current methods on four different public and one in-house datasets. Its capability of polyps segmentation in different sizes shows the potential clinical application.
Collapse
Affiliation(s)
- Zenan Wang
- Department of Gastroenterology, Beijing Chaoyang Hospitalthe Third Clinical Medical College of Capital Medical UniversityBeijingChina
| | - Zhen Liu
- Department of Gastroenterology, Beijing Chaoyang Hospitalthe Third Clinical Medical College of Capital Medical UniversityBeijingChina
| | - Jianfeng Yu
- Department of Gastroenterology, Beijing Chaoyang Hospitalthe Third Clinical Medical College of Capital Medical UniversityBeijingChina
| | - Yingxin Gao
- Department of Gastroenterology, Beijing Chaoyang Hospitalthe Third Clinical Medical College of Capital Medical UniversityBeijingChina
| | - Ming Liu
- Hunan Key Laboratory of Nonferrous Resources and Geological Hazard ExplorationChangshaChina
| |
Collapse
|
103
|
Zhao Z, Chen X, Cao J, Zhao Q, Liu W. FE-Net: Feature enhancement segmentation network. Neural Netw 2024; 174:106232. [PMID: 38490116 DOI: 10.1016/j.neunet.2024.106232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2023] [Revised: 01/29/2024] [Accepted: 03/07/2024] [Indexed: 03/17/2024]
Abstract
Semantic segmentation is one of the directions in image research. It aims to obtain the contours of objects of interest, facilitating subsequent engineering tasks such as measurement and feature selection. However, existing segmentation methods still lack precision in class edge, particularly in multi-class mixed region. To this end, we present the Feature Enhancement Network (FE-Net), a novel approach that leverages edge label and pixel-wise weights to enhance segmentation performance in complex backgrounds. Firstly, we propose a Smart Edge Head (SE-Head) to process shallow-level information from the backbone network. It is combined with the FCN-Head and SepASPP-Head, located at deeper layers, to form a transitional structure where the loss weights gradually transition from edge labels to semantic labels and a mixed loss is also designed to support this structure. Additionally, we propose a pixel-wise weight evaluation method, a pixel-wise weight block, and a feature enhancement loss to improve training effectiveness in multi-class regions. FE-Net achieves significant performance improvements over baselines on publicly datasets Pascal VOC2012, SBD, and ATR, with best mIoU enhancements of 15.19%, 1.42% and 3.51%, respectively. Furthermore, experiments conducted on Pole&Hole match dataset from our laboratory environment demonstrate the superior effectiveness of FE-Net in segmenting defined key pixels.
Collapse
Affiliation(s)
- Zhangyan Zhao
- School of Transportation and Logistics, Wuhan University of Technology, Wuhan, China
| | - Xiaoming Chen
- School of Transportation and Logistics, Wuhan University of Technology, Wuhan, China
| | - Jingjing Cao
- School of Transportation and Logistics, Wuhan University of Technology, Wuhan, China; State Key Laboratory of Maritime Technology, Wuhan University of Technology, Wuhan, China.
| | - Qiangwei Zhao
- School of Transportation and Logistics, Wuhan University of Technology, Wuhan, China
| | - Wenxi Liu
- College of Computer and Data Science, Fuzhou University, Fuzhou, China
| |
Collapse
|
104
|
Li L, Li Y, Lin Q, Liu S, Zhou J, Ming Z, Coello Coello CA. Neural Net-Enhanced Competitive Swarm Optimizer for Large-Scale Multiobjective Optimization. IEEE TRANSACTIONS ON CYBERNETICS 2024; 54:3502-3515. [PMID: 37486827 DOI: 10.1109/tcyb.2023.3287596] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/26/2023]
Abstract
The competitive swarm optimizer (CSO) classifies swarm particles into loser and winner particles and then uses the winner particles to efficiently guide the search of the loser particles. This approach has very promising performance in solving large-scale multiobjective optimization problems (LMOPs). However, most studies of CSOs ignore the evolution of the winner particles, although their quality is very important for the final optimization performance. Aiming to fill this research gap, this article proposes a new neural net-enhanced CSO for solving LMOPs, called NN-CSO, which not only guides the loser particles via the original CSO strategy, but also applies our trained neural network (NN) model to evolve winner particles. First, the swarm particles are classified into winner and loser particles by the pairwise competition. Then, the loser particles and winner particles are, respectively, treated as the input and desired output to train the NN model, which tries to learn promising evolutionary dynamics by driving the loser particles toward the winners. Finally, when model training is complete, the winner particles are evolved by the well-trained NN model, while the loser particles are still guided by the winner particles to maintain the search pattern of CSOs. To evaluate the performance of our designed NN-CSO, several LMOPs with up to ten objectives and 1000 decision variables are adopted, and the experimental results show that our designed NN model can significantly improve the performance of CSOs and shows some advantages over several state-of-the-art large-scale multiobjective evolutionary algorithms as well as over model-based evolutionary algorithms.
Collapse
|
105
|
Yeung C, Ungi T, Hu Z, Jamzad A, Kaufmann M, Walker R, Merchant S, Engel CJ, Jabs D, Rudan J, Mousavi P, Fichtinger G. From quantitative metrics to clinical success: assessing the utility of deep learning for tumor segmentation in breast surgery. Int J Comput Assist Radiol Surg 2024; 19:1193-1201. [PMID: 38642296 DOI: 10.1007/s11548-024-03133-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 03/28/2024] [Indexed: 04/22/2024]
Abstract
PURPOSE Preventing positive margins is essential for ensuring favorable patient outcomes following breast-conserving surgery (BCS). Deep learning has the potential to enable this by automatically contouring the tumor and guiding resection in real time. However, evaluation of such models with respect to pathology outcomes is necessary for their successful translation into clinical practice. METHODS Sixteen deep learning models based on established architectures in the literature are trained on 7318 ultrasound images from 33 patients. Models are ranked by an expert based on their contours generated from images in our test set. Generated contours from each model are also analyzed using recorded cautery trajectories of five navigated BCS cases to predict margin status. Predicted margins are compared with pathology reports. RESULTS The best-performing model using both quantitative evaluation and our visual ranking framework achieved a mean Dice score of 0.959. Quantitative metrics are positively associated with expert visual rankings. However, the predictive value of generated contours was limited with a sensitivity of 0.750 and a specificity of 0.433 when tested against pathology reports. CONCLUSION We present a clinical evaluation of deep learning models trained for intraoperative tumor segmentation in breast-conserving surgery. We demonstrate that automatic contouring is limited in predicting pathology margins despite achieving high performance on quantitative metrics.
Collapse
Affiliation(s)
- Chris Yeung
- School of Computing, Queen's University, Kingston, ON, Canada.
| | - Tamas Ungi
- School of Computing, Queen's University, Kingston, ON, Canada
| | - Zoe Hu
- School of Medicine, Queen's University, Kingston, ON, Canada
| | - Amoon Jamzad
- School of Computing, Queen's University, Kingston, ON, Canada
| | - Martin Kaufmann
- Department of Surgery, Queen's University, Kingston, ON, Canada
| | - Ross Walker
- Department of Surgery, Queen's University, Kingston, ON, Canada
| | - Shaila Merchant
- Department of Surgery, Queen's University, Kingston, ON, Canada
| | - Cecil Jay Engel
- Department of Surgery, Queen's University, Kingston, ON, Canada
| | - Doris Jabs
- Department of Radiology, Queen's University, Kingston, ON, Canada
| | - John Rudan
- Department of Surgery, Queen's University, Kingston, ON, Canada
| | - Parvin Mousavi
- School of Computing, Queen's University, Kingston, ON, Canada
| | | |
Collapse
|
106
|
Kusakunniran W, Imaromkul T, Mongkolluksamee S, Thongkanchorn K, Ritthipravat P, Tuakta P, Benjapornlert P. Deep Upscale U-Net for automatic tongue segmentation. Med Biol Eng Comput 2024; 62:1751-1762. [PMID: 38372910 DOI: 10.1007/s11517-024-03051-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 02/13/2024] [Indexed: 02/20/2024]
Abstract
In a treatment or diagnosis related to oral health conditions such as oral cancer and oropharyngeal cancer, an investigation of tongue's movements is a major part. In an automatic measurement of such movement, it must first start with a task of tongue segmentation. This paper proposes a solution of tongue segmentation based on a decoder-encoder CNN-based structure i.e., U-Net. However, it could suffer from a problem of feature loss in deep layers. This paper proposes a Deep Upscale U-Net (DU-UNET). An additional up-sampling of the feature map from a contracting path is concatenated to an upper layer of an expansive path, based on an original U-Net structure. The segmentation model is constructed by training DU-UNET on the two publicly available datasets, and transferred to the self-collected dataset of tongue images with five tongue postures which were recorded at a far distance from a camera under a real-world scenario. The proposed DU-UNET outperforms the other existing methods in our literature reviews, with accuracy of 99.2%, mean IoU of 97.8%, Dice score of 96.8%, and Jaccard score of 96.8%.
Collapse
Affiliation(s)
- Worapan Kusakunniran
- Faculty of Information and Communication Technology, Mahidol University, 999 Phuttamonthon 4 Road, Salaya, 73170, Nakhon Pathom, Thailand.
| | - Thanandon Imaromkul
- Faculty of Information and Communication Technology, Mahidol University, 999 Phuttamonthon 4 Road, Salaya, 73170, Nakhon Pathom, Thailand
| | - Sophon Mongkolluksamee
- Department of Computer Science, Faculty of Science, Srinakharinwirot University, 114 Sukhumvit 23, 10110, Bangkok, Thailand
| | - Kittikhun Thongkanchorn
- Faculty of Information and Communication Technology, Mahidol University, 999 Phuttamonthon 4 Road, Salaya, 73170, Nakhon Pathom, Thailand
| | - Panrasee Ritthipravat
- Department of Biomedical Engineering, Faculty of Engineering, Mahidol University, 999 Phuttamonthon 4 Road, Salaya, 73170, Nakhon Pathom, Thailand.
| | - Pimchanok Tuakta
- Department of Rehabilitation Medicine, Faculty of Medicine Ramathibodi Hospital, Mahidol University, 270 Rama 6 Road, 10400, Bangkok, Thailand
| | - Paitoon Benjapornlert
- Department of Rehabilitation Medicine, Faculty of Medicine Ramathibodi Hospital, Mahidol University, 270 Rama 6 Road, 10400, Bangkok, Thailand
| |
Collapse
|
107
|
Yüksel B, Özveren N, Yeşil Ç. Evaluation of Dental Plaque Area with Artificial Intelligence Model. Niger J Clin Pract 2024; 27:759-765. [PMID: 38943301 DOI: 10.4103/njcp.njcp_862_23] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2023] [Accepted: 03/06/2024] [Indexed: 07/01/2024]
Abstract
OBJECTIVES This study aims to assess the diagnostic accuracy of an artificial intelligence (AI) system employing deep learning for identifying dental plaque, utilizing a dataset comprising photographs of permanent teeth. MATERIALS AND METHODS In this study, photographs of 168 teeth belonging to 20 patients aged between 10 and 15 years, who met our criteria, were included. Intraoral photographs were taken of the patients in two stages, before and after the application of the plaque staining agent. To train the AI system to identify plaque on teeth with dental plaque that is not discolored, plaque and teeth were marked on photos with exposed dental plaque. One hundred forty teeth were used to construct the training group, while 28 teeth were used to create the test group. Another dentist reviewed images of teeth with dental plaque that was not discolored, and the effectiveness of AI in detecting plaque was evaluated using pertinent performance indicators. To compare the AI model and the dentist's evaluation outcomes, the mean intersection over union (IoU) values were evaluated by the Wilcoxon test. RESULTS The AI system showed higher performance in our study with a precision of 82% accuracy, 84% sensitivity, 83% F1 score, 87% accuracy, and 89% specificity in plaque detection. The area under the curve (AUC) value was found to be 0.922, and the IoU value was 76%. Subsequently, the dentist's plaque diagnosis performance was also evaluated. The IoU value was 0.71, and the AUC was 0.833. The AI model showed statistically significantly higher performance than the dentist (P < 0.05). CONCLUSIONS The AI algorithm that we developed has achieved promising results and demonstrated clinically acceptable performance in detecting dental plaque compared to a dentist.
Collapse
Affiliation(s)
- B Yüksel
- Department of Pediatric Dentistry, Faculty of Dentistry, Trakya University, Edirne, Turkey
| | - N Özveren
- Department of Pediatric Dentistry, Faculty of Dentistry, Trakya University, Edirne, Turkey
| | - Ç Yeşil
- Department of Computer Engineering, Faculty of Engineering, Yeditepe University, İstanbul, Turkey
| |
Collapse
|
108
|
Zhang B, Chen K, Yuan H, Liao Z, Zhou T, Guo W, Zhao S, Wang R, Su P. Automatic Lenke classification of adolescent idiopathic scoliosis with deep learning. JOR Spine 2024; 7:e1327. [PMID: 38690524 PMCID: PMC11058480 DOI: 10.1002/jsp2.1327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Revised: 02/26/2024] [Accepted: 03/10/2024] [Indexed: 05/02/2024] Open
Abstract
Purpose The Lenke classification system is widely utilized as the preoperative evaluation protocol for adolescent idiopathic scoliosis (AIS). However, manual measurement is susceptible to observer-induced variability, which consequently impacts the evaluation of progression. The goal of this investigation was to develop an automated Lenke classification system utilizing innovative deep learning algorithms. Methods Using the database from the First Affiliated Hospital of Sun Yat-sen University, the whole spinal x-rays images were retrospectively collected. Specifically, images collection was divided into AIS and control group. The control group consisted of individuals who underwent routine health checks and did not have scoliosis. Afterwards, relative features of all images were annotated. Deep learning was implemented through the utilization of the key-point based detection method to realize the vertebral detection, and Cobb angle measurement and scoliosis classification were performed based on relevant standards. Besides, the segmentation method was employed to achieve the recognition of lumbar vertebral pedicle to determine the type of lumbar spine modifier. Finally, the model performance was further quantitatively analyzed. Results In the study, a total of 2082 spinal x-ray images were collected from 407 AIS patients and 227 individuals in the control group. The model for vertebral detection achieved an F1-score of 0.809 for curve type evaluation and an F1-score of 0.901 for thoracic sagittal profile. The intraclass correlation efficient (ICC) of the Cobb angle measurement was 0.925. In the analysis of performance for vertebra pedicle segmentation model, the F1-score of lumbar modification profile was 0.942, the intersection over union (IOU) of the target pixels was 0.827, and the Hausdorff distance (HD) was 6.565 ± 2.583 mm. Specifically, the F1-score for ultimate Lenke type classifier was 0.885. Conclusions This study has constructed an automated Lenke classification system by employing the deep learning networks to achieve the recognition pattern and feature extraction. Our models require further validation in additional cases in the future.
Collapse
Affiliation(s)
- Baolin Zhang
- Department of Orthopaedic SurgeryFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Guangdong Provincial Key Laboratory of Orthopedics and TraumatologyFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Kanghao Chen
- School of Computer Science and EngineeringSun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Haodong Yuan
- Department of Orthopaedic SurgeryFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Guangdong Provincial Key Laboratory of Orthopedics and TraumatologyFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Graduate School of Biomedical EngineeringUNSW SydneySydneyNew South WalesAustralia
| | - Zhiheng Liao
- Department of Orthopaedic SurgeryFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Guangdong Provincial Key Laboratory of Orthopedics and TraumatologyFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Taifeng Zhou
- Department of Orthopaedic SurgeryFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Guangdong Provincial Key Laboratory of Orthopedics and TraumatologyFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Weiming Guo
- Department of Orthopaedic SurgeryFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Guangdong Provincial Key Laboratory of Orthopedics and TraumatologyFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Shen Zhao
- School of Intelligent Systems EngineeringSun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Ruixuan Wang
- School of Computer Science and EngineeringSun Yat‐sen UniversityGuangzhouGuangdongChina
| | - Peiqiang Su
- Department of Orthopaedic SurgeryFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
- Guangdong Provincial Key Laboratory of Orthopedics and TraumatologyFirst Affiliated Hospital, Sun Yat‐sen UniversityGuangzhouGuangdongChina
| |
Collapse
|
109
|
Ma J, Xie R, Ayyadhury S, Ge C, Gupta A, Gupta R, Gu S, Zhang Y, Lee G, Kim J, Lou W, Li H, Upschulte E, Dickscheid T, de Almeida JG, Wang Y, Han L, Yang X, Labagnara M, Gligorovski V, Scheder M, Rahi SJ, Kempster C, Pollitt A, Espinosa L, Mignot T, Middeke JM, Eckardt JN, Li W, Li Z, Cai X, Bai B, Greenwald NF, Van Valen D, Weisbart E, Cimini BA, Cheung T, Brück O, Bader GD, Wang B. The multimodality cell segmentation challenge: toward universal solutions. Nat Methods 2024; 21:1103-1113. [PMID: 38532015 PMCID: PMC11210294 DOI: 10.1038/s41592-024-02233-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 03/04/2024] [Indexed: 03/28/2024]
Abstract
Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multimodality cell segmentation benchmark, comprising more than 1,500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.
Collapse
Affiliation(s)
- Jun Ma
- Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Ronald Xie
- Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
- Department of Molecular Genetics, University of Toronto, Toronto, Ontario, Canada
| | - Shamini Ayyadhury
- Donnelly Centre, University of Toronto, Toronto, Ontario, Canada
- Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
| | - Cheng Ge
- School of Medicine and Pharmacy, Ocean University of China, Qingdao, China
| | - Anubha Gupta
- Department of Electronics and Communications Engineering, Indraprastha Institute of Information Technology Delhi (IIITD), New Delhi, India
| | - Ritu Gupta
- Laboratory Oncology Unit, Dr. BRAIRCH, All India Institute of Medical Sciences, New Delhi, India
| | - Song Gu
- Department of Image Reconstruction, Nanjing Anke Medical Technology Co., Nanjing, China
| | - Yao Zhang
- Shanghai Artificial Intelligence Laboratory, Shanghai, China
| | - Gihun Lee
- Graduate School of AI, KAIST, Seoul, South Korea
| | - Joonkee Kim
- Graduate School of AI, KAIST, Seoul, South Korea
| | - Wei Lou
- Shenzhen Research Institute of Big Data, Shenzhen, China
- Chinese University of Hong Kong (Shenzhen), Shenzhen, China
| | - Haofeng Li
- Shenzhen Research Institute of Big Data, Shenzhen, China
| | - Eric Upschulte
- Institute of Neuroscience and Medicine (INM-1) and Helmholtz AI, Research Center Jülich, Jülich, Germany
| | - Timo Dickscheid
- Institute of Neuroscience and Medicine (INM-1) and Helmholtz AI, Research Center Jülich, Jülich, Germany
- Faculty of Mathematics and Natural Sciences - Institute of Computer Science, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - José Guilherme de Almeida
- European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK
- Champalimaud Foundation - Centre for the Unknown, Lisbon, Portugal
| | - Yixin Wang
- Department of Bioengineering, Stanford University, Palo Alto, CA, USA
| | - Lin Han
- Tandon School of Engineering, New York University, New York, NY, USA
| | - Xin Yang
- School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China
| | - Marco Labagnara
- Laboratory of the Physics of Biological Systems, Institute of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Vojislav Gligorovski
- Laboratory of the Physics of Biological Systems, Institute of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Maxime Scheder
- Laboratory of the Physics of Biological Systems, Institute of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Sahand Jamal Rahi
- Laboratory of the Physics of Biological Systems, Institute of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Carly Kempster
- School of Biological Sciences, University of Reading, Reading, UK
| | - Alice Pollitt
- School of Biological Sciences, University of Reading, Reading, UK
| | - Leon Espinosa
- Laboratoire de Chimie Bactérienne, CNRS-Université Aix-Marseille UMR, Institut de Microbiologie de la Méditerranée, Marseille, France
| | - Tâm Mignot
- Laboratoire de Chimie Bactérienne, CNRS-Université Aix-Marseille UMR, Institut de Microbiologie de la Méditerranée, Marseille, France
| | - Jan Moritz Middeke
- Department of Internal Medicine I, University Hospital Dresden, Technical University Dresden, Dresden, Germany
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Jan-Niklas Eckardt
- Department of Internal Medicine I, University Hospital Dresden, Technical University Dresden, Dresden, Germany
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Wangkai Li
- Department of Automation, University of Science and Technology of China, Hefei, China
| | - Zhaoyang Li
- Institute of Advanced Technology, University of Science and Technology of China, Hefei, China
| | - Xiaochen Cai
- Department of Computer Science and Technology, Nanjing University, Nanjing, China
| | - Bizhe Bai
- School of EECS, The University of Queensland, Brisbane, Queensland, Australia
| | | | - David Van Valen
- Division of Computing and Mathematical Science, Caltech, Pasadena, CA, USA
- Howard Hughes Medical Institute, Chevy Chase, MD, USA
| | - Erin Weisbart
- Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA
| | - Beth A Cimini
- Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA
| | - Trevor Cheung
- Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada
- Department of Computer Science, University of Waterloo, Waterloo, Ontario, Canada
| | - Oscar Brück
- Hematoscope Laboratory, Comprehensive Cancer Center & Center of Diagnostics, Helsinki University Hospital, Helsinki, Finland
- Department of Oncology, University of Helsinki, Helsinki, Finland
| | - Gary D Bader
- Department of Molecular Genetics, University of Toronto, Toronto, Ontario, Canada
- Donnelly Centre, University of Toronto, Toronto, Ontario, Canada
- Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Lunenfeld-Tanenbaum Research Institute, Sinai Health System, Toronto, Ontario, Canada
- CIFAR Multiscale Human Program, CIFAR, Toronto, Ontario, Canada
| | - Bo Wang
- Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada.
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada.
- Vector Institute, Toronto, Ontario, Canada.
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.
- UHN AI Hub, University Health Network, Toronto, Ontario, Canada.
| |
Collapse
|
110
|
Batool A, Byun YC. Brain tumor detection with integrating traditional and computational intelligence approaches across diverse imaging modalities - Challenges and future directions. Comput Biol Med 2024; 175:108412. [PMID: 38691914 DOI: 10.1016/j.compbiomed.2024.108412] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2023] [Revised: 03/18/2024] [Accepted: 04/02/2024] [Indexed: 05/03/2024]
Abstract
Brain tumor segmentation and classification play a crucial role in the diagnosis and treatment planning of brain tumors. Accurate and efficient methods for identifying tumor regions and classifying different tumor types are essential for guiding medical interventions. This study comprehensively reviews brain tumor segmentation and classification techniques, exploring various approaches based on image processing, machine learning, and deep learning. Furthermore, our study aims to review existing methodologies, discuss their advantages and limitations, and highlight recent advancements in this field. The impact of existing segmentation and classification techniques for automated brain tumor detection is also critically examined using various open-source datasets of Magnetic Resonance Images (MRI) of different modalities. Moreover, our proposed study highlights the challenges related to segmentation and classification techniques and datasets having various MRI modalities to enable researchers to develop innovative and robust solutions for automated brain tumor detection. The results of this study contribute to the development of automated and robust solutions for analyzing brain tumors, ultimately aiding medical professionals in making informed decisions and providing better patient care.
Collapse
Affiliation(s)
- Amreen Batool
- Department of Electronic Engineering, Institute of Information Science & Technology, Jeju National University, Jeju, 63243, South Korea
| | - Yung-Cheol Byun
- Department of Computer Engineering, Major of Electronic Engineering, Jeju National University, Institute of Information Science Technology, Jeju, 63243, South Korea.
| |
Collapse
|
111
|
Yang J, Lai S, Wang X, Wang Y, Qian X. Diversity-Learning Block: Conquer Feature Homogenization of Multibranch. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:7563-7576. [PMID: 36322499 DOI: 10.1109/tnnls.2022.3214993] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Visual Geometry Group (VGG)-style ConvNet is an neural-network process units (NPU)-friendly network; however, the accuracy of this architecture cannot keep up with other well-designed network structures. Although some reparameterization methods are proposed to remedy this weakness, their performance suffers from the homogenization issue of parallel branches, and the preset shape of convolution kernels also influences spatial perception. To address this problem, we propose a diversity-learning (DL) block to build the DLNet, which could adaptively learn various features to enrich the feature space. To balance floating point of operations (FLOPs) and accuracy, groupwise operation is introduced and finally, a lightweight DL ConvNet DLGNet is obtained. Extensive evaluations have been conducted on different computer vision tasks, e.g., image classification [Canadian Institute For Advanced Research (CIFAR) and ImageNet], object detection [PASCAL visual object classes (VOC) and Microsoft Common Objects in Context (MS COCO)], and semantic segmentation (Cityscapes). The experimental results show that our proposed DLGNet can achieve comparable performance with the state-of-the-art networks while the speed is 183% faster than GhostNet and even over 600% faster than MobileNetV3 with similar accuracy when running on NPU.
Collapse
|
112
|
Liu T, He Z, Lin Z, Cao GZ, Su W, Xie S. An Adaptive Image Segmentation Network for Surface Defect Detection. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:8510-8523. [PMID: 37015643 DOI: 10.1109/tnnls.2022.3230426] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Surface defect detection plays an essential role in industry, and it is challenging due to the following problems: 1) the similarity between defect and nondefect texture is very high, which eventually leads to recognition or classification errors and 2) the size of defects is tiny, which are much more difficult to be detected than larger ones. To address such problems, this article proposes an adaptive image segmentation network (AIS-Net) for pixelwise segmentation of surface defects. It consists of three main parts: multishuffle-block dilated convolution (MSDC), dual attention context guidance (DACG), and adaptive category prediction (ACP) modules, where MSDC is designed to merge the multiscale defect features for avoiding the loss of tiny defect feature caused by model depth, DACG is designed to capture more contextual information from the defect feature map for locating defect regions and obtaining clear segmentation boundaries, and ACP is used to make classification and regression for predicting defect categories. Experimental results show that the proposed AIS-Net is superior to the state-of-the-art approaches on four actual surface defect datasets (NEU-DET: 98.38% ± 0.03%, DAGM: 99.25% ± 0.02%, Magnetic-tile: 98.73% ± 0.13%, and MVTec: 99.72% ± 0.02%).
Collapse
|
113
|
Fang T, Jiang Z, Zhou Y, Jia S, Zhao J, Nie S. Automatic assessment of DWI-ASPECTS for acute ischemic stroke based on deep learning. Med Phys 2024; 51:4351-4364. [PMID: 38687043 DOI: 10.1002/mp.17101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2023] [Revised: 04/04/2024] [Accepted: 04/12/2024] [Indexed: 05/02/2024] Open
Abstract
BACKGROUND Alberta Stroke Program Early Computed Tomography Score (ASPECTS) is a standardized semi-quantitative method for early ischemic changes in acute ischemic stroke. PURPOSE However, ASPECTS is still affected by expert experience and inconsistent results between readers in clinical. This study aims to propose an automatic ASPECTS scoring model based on diffusion-weighted imaging (DWI) mode to help clinicians make accurate treatment plans. METHODS Eighty-two patients with stroke were included in the study. First, we designed a new deep learning network for segmenting ASPECTS scoring brain regions. The network is improved based on U-net, which integrates multiple modules. Second, we proposed using hybrid classifiers to classify brain regions. For brain regions with larger areas, we used brain grayscale comparison algorithm to train machine learning classifiers, while using hybrid feature training for brain regions with smaller areas. RESULTS The average DICE coefficient of the segmented hindbrain area can reach 0.864. With the proposed hybrid classifier, our method performs significantly on both region-level ASPECTS and dichotomous ASPECTS. The sensitivity and accuracy on the test set are 95.51% and 93.43%, respectively. For dichotomous ASPECTS, the intraclass correlation coefficient (ICC) between our automated ASPECTS score and the expert reading was 0.87. CONCLUSIONS This study proposed an automated model for ASPECTS scoring of patients with acute ischemic stroke based on DWI images. Experimental results show that the method of segmentation first and then classification is feasible. Our method has the potential to assist physicians in the Alberta Stroke Program with early CT scoring and clinical stroke diagnosis.
Collapse
Affiliation(s)
- Ting Fang
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Zhuoyun Jiang
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Yuxi Zhou
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Shouqiang Jia
- Department of Imaging, Jinan People's Hospital affiliated to Shandong First Medical University, Shandong, China
| | - Jiaqi Zhao
- Department of Ultrasound, Shanghai Fourth People's Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Shengdong Nie
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, China
| |
Collapse
|
114
|
Alahmari SS, Goldgof D, Hall LO, Mouton PR. A Review of Nuclei Detection and Segmentation on Microscopy Images Using Deep Learning With Applications to Unbiased Stereology Counting. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:7458-7477. [PMID: 36327184 DOI: 10.1109/tnnls.2022.3213407] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
The detection and segmentation of stained cells and nuclei are essential prerequisites for subsequent quantitative research for many diseases. Recently, deep learning has shown strong performance in many computer vision problems, including solutions for medical image analysis. Furthermore, accurate stereological quantification of microscopic structures in stained tissue sections plays a critical role in understanding human diseases and developing safe and effective treatments. In this article, we review the most recent deep learning approaches for cell (nuclei) detection and segmentation in cancer and Alzheimer's disease with an emphasis on deep learning approaches combined with unbiased stereology. Major challenges include accurate and reproducible cell detection and segmentation of microscopic images from stained sections. Finally, we discuss potential improvements and future trends in deep learning applied to cell detection and segmentation.
Collapse
|
115
|
Lin H, Falahkheirkhah K, Kindratenko V, Bhargava R. INSTRAS: INfrared Spectroscopic imaging-based TRAnsformers for medical image Segmentation. MACHINE LEARNING WITH APPLICATIONS 2024; 16:100549. [PMID: 39036499 PMCID: PMC11258863 DOI: 10.1016/j.mlwa.2024.100549] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/23/2024] Open
Abstract
Infrared (IR) spectroscopic imaging is of potentially wide use in medical imaging applications due to its ability to capture both chemical and spatial information. This complexity of the data both necessitates using machine intelligence as well as presents an opportunity to harness a high-dimensionality data set that offers far more information than today's manually-interpreted images. While convolutional neural networks (CNNs), including the well-known U-Net model, have demonstrated impressive performance in image segmentation, the inherent locality of convolution limits the effectiveness of these models for encoding IR data, resulting in suboptimal performance. In this work, we propose an INfrared Spectroscopic imaging-based TRAnsformers for medical image Segmentation (INSTRAS). This novel model leverages the strength of the transformer encoders to segment IR breast images effectively. Incorporating skip-connection and transformer encoders, INSTRAS overcomes the issue of pure convolution models, such as the difficulty of capturing long-range dependencies. To evaluate the performance of our model and existing convolutional models, we conducted training on various encoder-decoder models using a breast dataset of IR images. INSTRAS, utilizing 9 spectral bands for segmentation, achieved a remarkable AUC score of 0.9788, underscoring its superior capabilities compared to purely convolutional models. These experimental results attest to INSTRAS's advanced and improved segmentation abilities for IR imaging.
Collapse
Affiliation(s)
- Hangzheng Lin
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, IL, United States
| | | | - Volodymyr Kindratenko
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, IL, United States
- Center for Artificial Intelligence Innovation, National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign, IL, United States
| | - Rohit Bhargava
- Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, IL, United States
- Beckman Institute, University of Illinois at Urbana-Champaign, IL, United States
- Departments of Bioengineering, Mechanical Science and Engineering and Cancer Center at Illinois, University of Illinois at Urbana-Champaign, IL, United States
| |
Collapse
|
116
|
Sun Y, Gu X, Zhou X, Yang J, Shen W, Cheng Y, Zhang JM, Chen Y. DPIF-Net: a dual path network for rural road extraction based on the fusion of global and local information. PeerJ Comput Sci 2024; 10:e2079. [PMID: 38855245 PMCID: PMC11157547 DOI: 10.7717/peerj-cs.2079] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Accepted: 05/03/2024] [Indexed: 06/11/2024]
Abstract
Background Automatic extraction of roads from remote sensing images can facilitate many practical applications. However, thus far, thousands of kilometers or more of roads worldwide have not been recorded, especially low-grade roads in rural areas. Moreover, rural roads have different shapes and are influenced by complex environments and other interference factors, which has led to a scarcity of dedicated low level category road datasets. Methods To address these issues, based on convolutional neural networks (CNNs) and tranformers, this article proposes the Dual Path Information Fusion Network (DPIF-Net). In addition, given the severe lack of low-grade road datasets, we constructed the GaoFen-2 (GF-2) rural road dataset to address this challenge, which spans three regions in China and covers an area of over 2,300 km, almost entirely composed of low-grade roads. To comprehensively test the low-grade road extraction performance and generalization ability of the model, comparative experiments are carried out on the DeepGlobe, and Massachusetts regular road datasets. Results The results show that DPIF-Net achieves the highest IoU and F1 score on three datasets compared with methods such as U-Net, SegNet, DeepLabv3+, and D-LinkNet, with notable performance on the GF-2 dataset, reaching 0.6104 and 0.7608, respectively. Furthermore, multiple validation experiments demonstrate that DPIF-Net effectively preserves improved connectivity in low-grade road extraction with a modest parameter count of 63.9 MB. The constructed low-grade road dataset and proposed methods will facilitate further research on rural roads, which holds promise for assisting governmental authorities in making informed decisions and strategies to enhance rural road infrastructure.
Collapse
Affiliation(s)
- Yuan Sun
- Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China
| | - Xingfa Gu
- Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China
| | - Xiang Zhou
- Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China
| | - Jian Yang
- Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, China
| | - Wangyao Shen
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Yuanlei Cheng
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Jin Ming Zhang
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| | - Yunping Chen
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China
| |
Collapse
|
117
|
Zhang Q, Chen S, Wu Y, Ji Z, Yan F, Huang S, Liu Y. Improved U-net network asphalt pavement crack detection method. PLoS One 2024; 19:e0300679. [PMID: 38820536 PMCID: PMC11142699 DOI: 10.1371/journal.pone.0300679] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2024] [Accepted: 03/01/2024] [Indexed: 06/02/2024] Open
Abstract
Road crack detection is one of the important parts of road safety detection. Aiming at the problems such as weak segmentation effect of basic U-Net on pavement crack, insufficient precision of crack contour segmentation, difficult to identify narrow crack and low segmentation accuracy, this paper proposes an improved U-net network pavement crack segmentation method. VGG16 and Up_Conv (Upsampling Convolution) modules are introduced as backbone network and feature enhancement network respectively, and the more abstract features in the image are extracted by using the Block depth separable convolution blocks, and the multi-scale features are captured and enhanced by higher level semantic information to improve the recognition accuracy of narrow cracks in the road surface. The improved network embedded the Ca(Channel Attention) attention mechanism in U-net's jump connection to enhance the crack portion to suppress background noise. At the same time, DG_Conv(Depthwise GSConv Convolution) module and UnetUp(Unet Upsampling) module are added in the decoding part to extract richer features through more convolutional layers in the network, so that the model pays more attention to the detailed part of the crack, so the segmentation accuracy can be improved. In order to verify the model's ability to detect cracks in complex backgrounds, experiments were carried out on CFD and Deepcrack datasets. The experimental results show that compared with the traditional U-net network F1-score and mIoU have increased by 13.6% and 9.9% respectively. Superior to advanced models such as U-net, Segnet and Linknet in accuracy and generalization ability, the improved model provides a new method for asphalt pavement crack detection. The model is more conducive to practical application and ground deployment, and can be applied in road maintenance projects.
Collapse
Affiliation(s)
- Qiong Zhang
- Changchun University of Science and Technology, Changchun, Jilin, China
- Jilin Provincial Science and Technology Innovation Center of Intelligent Perception and Information Processing, Changchun, Jilin, China
| | - Shanshan Chen
- Changchun University of Science and Technology, Changchun, Jilin, China
| | - Yue Wu
- Changchun University of Science and Technology, Changchun, Jilin, China
| | - Zhonghang Ji
- Changchun University of Science and Technology, Changchun, Jilin, China
- Jilin Provincial Science and Technology Innovation Center of Intelligent Perception and Information Processing, Changchun, Jilin, China
| | - Fei Yan
- Changchun University of Science and Technology, Changchun, Jilin, China
- Jilin Provincial Science and Technology Innovation Center of Intelligent Perception and Information Processing, Changchun, Jilin, China
| | - Shiling Huang
- Changchun University of Science and Technology, Changchun, Jilin, China
| | - Yunqing Liu
- Changchun University of Science and Technology, Changchun, Jilin, China
- Jilin Provincial Science and Technology Innovation Center of Intelligent Perception and Information Processing, Changchun, Jilin, China
| |
Collapse
|
118
|
Wang R, Ma L, He G, Johnson BA, Yan Z, Chang M, Liang Y. Transformers for Remote Sensing: A Systematic Review and Analysis. SENSORS (BASEL, SWITZERLAND) 2024; 24:3495. [PMID: 38894286 PMCID: PMC11175147 DOI: 10.3390/s24113495] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/22/2024] [Revised: 05/20/2024] [Accepted: 05/27/2024] [Indexed: 06/21/2024]
Abstract
Research on transformers in remote sensing (RS), which started to increase after 2021, is facing the problem of a relative lack of review. To understand the trends of transformers in RS, we undertook a quantitative analysis of the major research on transformers over the past two years by dividing the application of transformers into eight domains: land use/land cover (LULC) classification, segmentation, fusion, change detection, object detection, object recognition, registration, and others. Quantitative results show that transformers achieve a higher accuracy in LULC classification and fusion, with more stable performance in segmentation and object detection. Combining the analysis results on LULC classification and segmentation, we have found that transformers need more parameters than convolutional neural networks (CNNs). Additionally, further research is also needed regarding inference speed to improve transformers' performance. It was determined that the most common application scenes for transformers in our database are urban, farmland, and water bodies. We also found that transformers are employed in the natural sciences such as agriculture and environmental protection rather than the humanities or economics. Finally, this work summarizes the analysis results of transformers in remote sensing obtained during the research process and provides a perspective on future directions of development.
Collapse
Affiliation(s)
- Ruikun Wang
- Beijing Institute of Satellite Information Engineering, Beijing 100095, China
- State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd., Beijing 100095, China
| | - Lei Ma
- Jiangsu Provincial Key Laboratory of Geographic Information Science and Technology, Key Laboratory for Land Satellite Remote Sensing Applications of Ministry of Natural Resources, School of Geography and Ocean Science, Nanjing University, Nanjing 210023, China
| | - Guangjun He
- Beijing Institute of Satellite Information Engineering, Beijing 100095, China
- State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd., Beijing 100095, China
| | - Brian Alan Johnson
- Natural Resources and Ecosystem Services, Institute for Global Environmental Strategies, 2108-11, Kamiyamaguchi, Hayama, Kanagawa 240-0115, Japan
| | - Ziyun Yan
- Jiangsu Provincial Key Laboratory of Geographic Information Science and Technology, Key Laboratory for Land Satellite Remote Sensing Applications of Ministry of Natural Resources, School of Geography and Ocean Science, Nanjing University, Nanjing 210023, China
| | - Ming Chang
- Beijing Institute of Satellite Information Engineering, Beijing 100095, China
- State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd., Beijing 100095, China
| | - Ying Liang
- Beijing Institute of Satellite Information Engineering, Beijing 100095, China
- State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd., Beijing 100095, China
| |
Collapse
|
119
|
Gao H, Qi M, Du B, Yang S, Li H, Wang T, Zhong W, Tang Y. An accurate semantic segmentation model for bean seedlings and weeds identification based on improved ERFnet. Sci Rep 2024; 14:12288. [PMID: 38811674 PMCID: PMC11136954 DOI: 10.1038/s41598-024-61981-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Accepted: 05/13/2024] [Indexed: 05/31/2024] Open
Abstract
In agricultural production activities, the growth of crops always accompanies the competition of weeds for nutrients and sunlight. In order to mitigate the adverse effects of weeds on yield, we apply semantic segmentation techniques to differentiate between seedlings and weeds, leading to precision weeding. The proposed EPAnet employs a loss function coupled with Cross-entropy loss and Dice loss to enhance attention to feature information. A multi-Decoder cooperative module based on ERFnet is designed to enhance information transfer during feature mapping. The SimAM is introduced to enhance position recognition. DO-CONV is used to replace the traditional convolution Feature Pyramid Networks (FPN) connection layer to integrate feature information, improving the model's performance on leaf edge processing, and is named FDPN. Moreover, the Overall Accuracy has been improved by 0.65%, the mean Intersection over Union (mIoU) by 1.91%, and the Frequency-Weighted Intersection over Union (FWIoU) by 1.19%. Compared to other advanced methods, EPAnet demonstrates superior image segmentation results in complex natural environments with uneven lighting, leaf interference, and shadows.
Collapse
Affiliation(s)
- Haozhang Gao
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China
- School of Information and Control Engineering, Jilin Institute of Chemical Technology, Jilin, 132022, China
| | - Mingyang Qi
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China
| | - Baoxia Du
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China
- School of Information and Control Engineering, Jilin Institute of Chemical Technology, Jilin, 132022, China
| | - Shuang Yang
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China
- School of Information and Control Engineering, Jilin Institute of Chemical Technology, Jilin, 132022, China
| | - Han Li
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China
- School of Information and Control Engineering, Jilin Institute of Chemical Technology, Jilin, 132022, China
| | - Tete Wang
- R &D Department, Jilin Province Electric Innovation Information Technology Limited Company, Changchun, 130117, China
| | - Wenyu Zhong
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China.
| | - You Tang
- Electrical and Information Engineering College, Jilin Agricultural Science and Technology University, Jilin, 132101, China.
- School of Information and Control Engineering, Jilin Institute of Chemical Technology, Jilin, 132022, China.
| |
Collapse
|
120
|
Kong D, Hu X, Gong Z, Zhang D. Segmentation of void defects in X-ray images of chip solder joints based on PCB-DeepLabV3 algorithm. Sci Rep 2024; 14:11925. [PMID: 38789447 PMCID: PMC11126598 DOI: 10.1038/s41598-024-61643-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Accepted: 05/08/2024] [Indexed: 05/26/2024] Open
Abstract
Defects within chip solder joints are usually inspected visually for defects using X-ray imaging to obtain images. The phenomenon of voids inside solder joints is one of the most likely types of defects in the soldering process, and accurate detection of voids becomes difficult due to their irregular shapes, varying sizes, and defocused edges. To address this problem, an X-ray void image segmentation algorithm based on improved PCB-DeepLabV3 is proposed. Firstly, to meet the demand for lightweight and easy deployment in industrial scenarios, mobilenetv2 is used as the feature extraction backbone network of the PCB-DeepLabV3 model; then, Attentional multi-scale two-space pyramid pooling network (AMTPNet) is designed to optimize the shallow feature edges and to improve the ability to capture detailed information; finally, image cropping and cleaning methods are designed to enhance the training dataset, and the improved PCB-DeepLabV3 is applied to the training dataset. The improved PCB-DeepLabV3 model is used to segment the void regions within the solder joints and compared with the classical semantic segmentation models such as Unet, SegNet, PSPNet, and DeeplabV3. The proposed new method enables the solder joint void inspection to get rid of the traditional way of visual inspection, realize intelligent upgrading, and effectively improve the problem of difficult segmentation of the target virtual edges, to obtain the inspection results with higher accuracy.
Collapse
Affiliation(s)
- Defeng Kong
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China
| | - Xinyu Hu
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China.
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China.
| | - Ziang Gong
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China
| | - Daode Zhang
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China
- School of Mechanical Engineering, Hubei University of Technology, Wuhan, China
| |
Collapse
|
121
|
Li D, Huo H, Jiao S, Sun X, Chen S. Automated thorax disease diagnosis using multi-branch residual attention network. Sci Rep 2024; 14:11865. [PMID: 38789592 PMCID: PMC11126636 DOI: 10.1038/s41598-024-62813-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2024] [Accepted: 05/21/2024] [Indexed: 05/26/2024] Open
Abstract
Chest X-ray (CXR) is an extensively utilized radiological modality for supporting the diagnosis of chest diseases. However, existing research approaches suffer from limitations in effectively integrating multi-scale CXR image features and are also hindered by imbalanced datasets. Therefore, there is a pressing need for further advancement in computer-aided diagnosis (CAD) of thoracic diseases. To tackle these challenges, we propose a multi-branch residual attention network (MBRANet) for thoracic disease diagnosis. MBRANet comprises three components. Firstly, to address the issue of inadequate extraction of spatial and positional information by the convolutional layer, a novel residual structure incorporating a coordinate attention (CA) module is proposed to extract features at multiple scales. Next, based on the concept of a Feature Pyramid Network (FPN), we perform multi-scale feature fusion in the following manner. Thirdly, we propose a novel Multi-Branch Feature Classifier (MFC) approach, which leverages the class-specific residual attention (CSRA) module for classification instead of relying solely on the fully connected layer. In addition, the designed BCEWithLabelSmoothing loss function improves the generalization ability and mitigates the problem of class imbalance by introducing a smoothing factor. We evaluated MBRANet on the ChestX-Ray14, CheXpert, MIMIC-CXR, and IU X-Ray datasets and achieved average AUCs of 0.841, 0.895, 0.805, and 0.745, respectively. Our method outperformed state-of-the-art baselines on these benchmark datasets.
Collapse
Affiliation(s)
- Dongfang Li
- School of Information Engineering, Henan University of Science and Technology, Luoyang, 471000, Henan, China
| | - Hua Huo
- School of Information Engineering, Henan University of Science and Technology, Luoyang, 471000, Henan, China.
| | - Shupei Jiao
- School of Information Engineering, Henan University of Science and Technology, Luoyang, 471000, Henan, China
| | - Xiaowei Sun
- School of Information Engineering, Henan University of Science and Technology, Luoyang, 471000, Henan, China
| | - Shuya Chen
- School of Information Engineering, Henan University of Science and Technology, Luoyang, 471000, Henan, China
| |
Collapse
|
122
|
Liu J, Zhang W, Liu Y, Zhang Q. Polyp segmentation based on implicit edge-guided cross-layer fusion networks. Sci Rep 2024; 14:11678. [PMID: 38778219 PMCID: PMC11111678 DOI: 10.1038/s41598-024-62331-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Accepted: 05/15/2024] [Indexed: 05/25/2024] Open
Abstract
Polyps are abnormal tissue clumps growing primarily on the inner linings of the gastrointestinal tract. While such clumps are generally harmless, they can potentially evolve into pathological tumors, and thus require long-term observation and monitoring. Polyp segmentation in gastrointestinal endoscopy images is an important stage for polyp monitoring and subsequent treatment. However, this segmentation task faces multiple challenges: the low contrast of the polyp boundaries, the varied polyp appearance, and the co-occurrence of multiple polyps. So, in this paper, an implicit edge-guided cross-layer fusion network (IECFNet) is proposed for polyp segmentation. The codec pair is used to generate an initial saliency map, the implicit edge-enhanced context attention module aggregates the feature graph output from the encoding and decoding to generate the rough prediction, and the multi-scale feature reasoning module is used to generate final predictions. Polyp segmentation experiments have been conducted on five popular polyp image datasets (Kvasir, CVC-ClinicDB, ETIS, CVC-ColonDB, and CVC-300), and the experimental results show that the proposed method significantly outperforms a conventional method, especially with an accuracy margin of 7.9% on the ETIS dataset.
Collapse
Affiliation(s)
- Junqing Liu
- Hubei Engineering and Technology Research Center for Construction Quality Inspection Equipment, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China
- College of Computer and Information Technology, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China
| | - Weiwei Zhang
- Hubei Engineering and Technology Research Center for Construction Quality Inspection Equipment, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China.
- College of Computer and Information Technology, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China.
| | - Yong Liu
- Hubei Engineering and Technology Research Center for Construction Quality Inspection Equipment, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China
- College of Computer and Information Technology, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China
| | - Qinghe Zhang
- Hubei Engineering and Technology Research Center for Construction Quality Inspection Equipment, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China
- College of Computer and Information Technology, China Three Gorges University, Yichang, 443002, Hubei, People's Republic of China
| |
Collapse
|
123
|
Keshavamurthy KN, Eickhoff C, Ziv E. Pre-operative lung ablation prediction using deep learning. Eur Radiol 2024:10.1007/s00330-024-10767-8. [PMID: 38775950 DOI: 10.1007/s00330-024-10767-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2024] [Revised: 02/09/2024] [Accepted: 03/07/2024] [Indexed: 07/31/2024]
Abstract
OBJECTIVE Microwave lung ablation (MWA) is a minimally invasive and inexpensive alternative cancer treatment for patients who are not candidates for surgery/radiotherapy. However, a major challenge for MWA is its relatively high tumor recurrence rates, due to incomplete treatment as a result of inaccurate planning. We introduce a patient-specific, deep-learning model to accurately predict post-treatment ablation zones to aid planning and enable effective treatments. MATERIALS AND METHODS Our IRB-approved retrospective study consisted of ablations with a single applicator/burn/vendor between 01/2015 and 01/2019. The input data included pre-procedure computerized tomography (CT), ablation power/time, and applicator position. The ground truth ablation zone was segmented from follow-up CT post-treatment. Novel deformable image registration optimized for ablation scans and an applicator-centric co-ordinate system for data analysis were applied. Our prediction model was based on the U-net architecture. The registrations were evaluated using target registration error (TRE) and predictions using Bland-Altman plots, Dice co-efficient, precision, and recall, compared against the applicator vendor's estimates. RESULTS The data included 113 unique ablations from 72 patients (median age 57, interquartile range (IQR) (49-67); 41 women). We obtained a TRE ≤ 2 mm on 52 ablations. Our prediction had no bias from ground truth ablation volumes (p = 0.169) unlike the vendor's estimate (p < 0.001) and had smaller limits of agreement (p < 0.001). An 11% improvement was achieved in the Dice score. The ability to account for patient-specific in-vivo anatomical effects due to vessels, chest wall, heart, lung boundaries, and fissures was shown. CONCLUSIONS We demonstrated a patient-specific deep-learning model to predict the ablation treatment effect prior to the procedure, with the potential for improved planning, achieving complete treatments, and reduce tumor recurrence. CLINICAL RELEVANCE STATEMENT Our method addresses the current lack of reliable tools to estimate ablation extents, required for ensuring successful ablation treatments. The potential clinical implications include improved treatment planning, ensuring complete treatments, and reducing tumor recurrence.
Collapse
Affiliation(s)
| | - Carsten Eickhoff
- University of Tübingen Geschwister-Scholl-Platz, 72074, Tübingen, Germany
| | - Etay Ziv
- Memorial Sloan Kettering Cancer Center, 1275 York Avenue, New York, NY, 10065, USA
| |
Collapse
|
124
|
Ma J, Iddir SP, Ganesh S, Yi D, Heiferman MJ. Automated segmentation for early detection of uveal melanoma. CANADIAN JOURNAL OF OPHTHALMOLOGY 2024:S0008-4182(24)00103-0. [PMID: 38768649 DOI: 10.1016/j.jcjo.2024.04.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Revised: 03/22/2024] [Accepted: 04/01/2024] [Indexed: 05/22/2024]
Abstract
OBJECTIVE Uveal melanoma is the most common intraocular malignancy in adults. Current screening and triaging methods for melanocytic choroidal tumours face inherent limitations, particularly in regions with limited access to specialized ocular oncologists. This study explores the potential of machine learning to automate tumour segmentation. We develop and evaluate a machine-learning model for lesion segmentation using ultra-wide-field fundus photography. METHOD A retrospective chart review was conducted of patients diagnosed with uveal melanoma, choroidal nevi, or congenital hypertrophy of the retinal pigmented epithelium at a tertiary academic medical centre. Included patients had a single ultra-wide-field fundus photograph (Optos PLC, Dunfermline, Fife, Scotland) of adequate quality to visualize the lesion of interest, as confirmed by a single ocular oncologist. These images were used to develop and test a machine-learning algorithm for lesion segmentation. RESULTS A total of 396 images were used to develop a machine-learning algorithm for lesion segmentation. Ninety additional images were used in the testing data set along with images of 30 healthy control individuals. Of the images with successfully detected lesions, the machine-learning segmentation yielded Dice coefficients of 0.86, 0.81, and 0.85 for uveal melanoma, choroidal nevi, and congenital hypertrophy of the retinal pigmented epithelium, respectively. Sensitivities for any lesion detection per image were 1.00, 0.90, and 0.87, respectively. For images without lesions, specificity was 0.93. CONCLUSION Our study demonstrates a novel machine-learning algorithm's performance, suggesting its potential clinical utility as a widely accessible method of screening choroidal tumours. Additional evaluation methods are necessary to further enhance the model's lesion classification and diagnostic accuracy.
Collapse
Affiliation(s)
- Jiechao Ma
- Department of Biomedical Engineering, University of Illinois at Chicago, Chicago, IL
| | - Sabrina P Iddir
- Department of Ophthalmology and Visual Sciences, University of Illinois at Chicago, Chicago, IL
| | - Sanjay Ganesh
- Department of Ophthalmology and Visual Sciences, University of Illinois at Chicago, Chicago, IL
| | - Darvin Yi
- Department of Ophthalmology and Visual Sciences, University of Illinois at Chicago, Chicago, IL
| | - Michael J Heiferman
- Department of Ophthalmology and Visual Sciences, University of Illinois at Chicago, Chicago, IL.
| |
Collapse
|
125
|
You S, Lin S, Feng Y, Fan J, Yan Z, Liu S, Ji Y. ISLS: An Illumination-Aware Sauce-Packet Leakage Segmentation Method. SENSORS (BASEL, SWITZERLAND) 2024; 24:3216. [PMID: 38794069 PMCID: PMC11126124 DOI: 10.3390/s24103216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/08/2024] [Revised: 05/04/2024] [Accepted: 05/16/2024] [Indexed: 05/26/2024]
Abstract
The segmentation of abnormal regions is vital in smart manufacturing. The blurring sauce-packet leakage segmentation task (BSLST) is designed to distinguish the sauce packet and the leakage's foreground and background at the pixel level. However, the existing segmentation system for detecting sauce-packet leakage on intelligent sensors encounters an issue of imaging blurring caused by uneven illumination. This issue adversely affects segmentation performance, thereby hindering the measurements of leakage area and impeding the automated sauce-packet production. To alleviate this issue, we propose the two-stage illumination-aware sauce-packet leakage segmentation (ISLS) method for intelligent sensors. The ISLS comprises two main stages: illumination-aware region enhancement and leakage region segmentation. In the first stage, YOLO-Fastestv2 is employed to capture the Region of Interest (ROI), which reduces redundancy computations. Additionally, we propose image enhancement to relieve the impact of uneven illumination, enhancing the texture details of the ROI. In the second stage, we propose a novel feature extraction network. Specifically, we propose the multi-scale feature fusion module (MFFM) and the Sequential Self-Attention Mechanism (SSAM) to capture discriminative representations of leakage. The multi-level features are fused by the MFFM with a small number of parameters, which capture leakage semantics at different scales. The SSAM realizes the enhancement of valid features and the suppression of invalid features by the adaptive weighting of spatial and channel dimensions. Furthermore, we generate a self-built dataset of sauce packets, including 606 images with various leakage areas. Comprehensive experiments demonstrate that our ISLS method shows better results than several state-of-the-art methods, with additional performance analyses deployed on intelligent sensors to affirm the effectiveness of our proposed method.
Collapse
Affiliation(s)
- Shuai You
- School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing 210023, China; (S.Y.); (Y.F.)
| | - Shijun Lin
- School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China; (S.L.); (S.L.)
| | - Yujian Feng
- School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing 210023, China; (S.Y.); (Y.F.)
| | - Jianhua Fan
- The 63rd Research Institute, National University of Defense Technology, Nanjing 210007, China;
| | - Zhenzheng Yan
- Northern Information Control Research Academy Group Co., Ltd., Nanjing 211153, China;
| | - Shangdong Liu
- School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China; (S.L.); (S.L.)
| | - Yimu Ji
- School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China; (S.L.); (S.L.)
| |
Collapse
|
126
|
Huang Z, Zhao Y, Yu Z, Qin P, Han X, Wang M, Liu M, Gregersen H. BiU-net: A dual-branch structure based on two-stage fusion strategy for biomedical image segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 252:108235. [PMID: 38776830 DOI: 10.1016/j.cmpb.2024.108235] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Revised: 04/28/2024] [Accepted: 05/17/2024] [Indexed: 05/25/2024]
Abstract
BACKGROUND AND OBJECTIVE Computer-based biomedical image segmentation plays a crucial role in planning of assisted diagnostics and therapy. However, due to the variable size and irregular shape of the segmentation target, it is still a challenge to construct an effective medical image segmentation structure. Recently, hybrid architectures based on convolutional neural networks (CNNs) and transformers were proposed. However, most current backbones directly replace one or all convolutional layers with transformer blocks, regardless of the semantic gap between features. Thus, how to sufficiently and effectively eliminate the semantic gap as well as combine the global and local information is a critical challenge. METHODS To address the challenge, we propose a novel structure, called BiU-Net, which integrates CNNs and transformers with a two-stage fusion strategy. In the first fusion stage, called Single-Scale Fusion (SSF) stage, the encoding layers of the CNNs and transformers are coupled, with both having the same feature map size. The SSF stage aims to reconstruct local features based on CNNs and long-range information based on transformers in each encoding block. In the second stage, Multi-Scale Fusion (MSF), BiU-Net interacts with multi-scale features from various encoding layers to eliminate the semantic gap between deep and shallow layers. Furthermore, a Context-Aware Block (CAB) is embedded in the bottleneck to reinforce multi-scale features in the decoder. RESULTS Experiments on four public datasets were conducted. On the BUSI dataset, our BiU-Net achieved 85.50 % on Dice coefficient (Dice), 76.73 % on intersection over union (IoU), and 97.23 % on accuracy (ACC). Compared to the state-of-the-art method, BiU-Net improves Dice by 1.17 %. For the Monuseg dataset, the proposed method attained the highest scores, reaching 80.27 % and 67.22 % for Dice and IoU. The BiU-Net achieves 95.33 % and 81.22 % Dice on the PH2 and DRIVE datasets. CONCLUSIONS The results of our experiments showed that BiU-Net transcends existing state-of-the-art methods on four publicly available biomedical datasets. Due to the powerful multi-scale feature extraction ability, our proposed BiU-Net is a versatile medical image segmentation framework for various types of medical images. The source code is released on (https://github.com/ZYLandy/BiU-Net).
Collapse
Affiliation(s)
- Zhiyong Huang
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China.
| | - Yunlan Zhao
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
| | - Zhi Yu
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
| | - Pinzhong Qin
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
| | - Xiao Han
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
| | - Mengyao Wang
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
| | - Man Liu
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China
| | - Hans Gregersen
- California Medical Innovations Institute, San Diego 92121, California
| |
Collapse
|
127
|
Makram AW, Salem NM, El-Wakad MT, Al-Atabany W. Robust detection and refinement of saliency identification. Sci Rep 2024; 14:11076. [PMID: 38744990 DOI: 10.1038/s41598-024-61105-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Accepted: 05/02/2024] [Indexed: 05/16/2024] Open
Abstract
Salient object detection is an increasingly popular topic in the computer vision field, particularly for images with complex backgrounds and diverse object parts. Background information is an essential factor in detecting salient objects. This paper suggests a robust and effective methodology for salient object detection. This method involves two main stages. The first stage is to produce a saliency detection map based on the dense and sparse reconstruction of image regions using a refined background dictionary. The refined background dictionary uses a boundary conductivity measurement to exclude salient object regions near the image's boundary from a background dictionary. In the second stage, the CascadePSP network is integrated to refine and correct the local boundaries of the saliency mask to highlight saliency objects more uniformly. Using six evaluation indexes, experimental outcomes conducted on three datasets show that the proposed approach performs effectively compared to the state-of-the-art methods in salient object detection, particularly in identifying the challenging salient objects located near the image's boundary. These results demonstrate the potential of the proposed framework for various computer vision applications.
Collapse
Affiliation(s)
- Abram W Makram
- Biomedical Engineering Department, Faculty of Engineering, Helwan University, Helwan, Egypt.
| | - Nancy M Salem
- Biomedical Engineering Department, Faculty of Engineering, Helwan University, Helwan, Egypt
| | | | - Walid Al-Atabany
- Biomedical Engineering Department, Faculty of Engineering, Helwan University, Helwan, Egypt
- Information Technology and Computer Science School, Nile University, Giza, Egypt
| |
Collapse
|
128
|
Li X, Zhao H, Wu D, Liu Q, Tang R, Li L, Xu Z, Lyu X. SLMFNet: Enhancing land cover classification of remote sensing images through selective attentions and multi-level feature fusion. PLoS One 2024; 19:e0301134. [PMID: 38743645 PMCID: PMC11093330 DOI: 10.1371/journal.pone.0301134] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 03/08/2024] [Indexed: 05/16/2024] Open
Abstract
Land cover classification (LCC) is of paramount importance for assessing environmental changes in remote sensing images (RSIs) as it involves assigning categorical labels to ground objects. The growing availability of multi-source RSIs presents an opportunity for intelligent LCC through semantic segmentation, offering a comprehensive understanding of ground objects. Nonetheless, the heterogeneous appearances of terrains and objects contribute to significant intra-class variance and inter-class similarity at various scales, adding complexity to this task. In response, we introduce SLMFNet, an innovative encoder-decoder segmentation network that adeptly addresses this challenge. To mitigate the sparse and imbalanced distribution of RSIs, we incorporate selective attention modules (SAMs) aimed at enhancing the distinguishability of learned representations by integrating contextual affinities within spatial and channel domains through a compact number of matrix operations. Precisely, the selective position attention module (SPAM) employs spatial pyramid pooling (SPP) to resample feature anchors and compute contextual affinities. In tandem, the selective channel attention module (SCAM) concentrates on capturing channel-wise affinity. Initially, feature maps are aggregated into fewer channels, followed by the generation of pairwise channel attention maps between the aggregated channels and all channels. To harness fine-grained details across multiple scales, we introduce a multi-level feature fusion decoder with data-dependent upsampling (MLFD) to meticulously recover and merge feature maps at diverse scales using a trainable projection matrix. Empirical results on the ISPRS Potsdam and DeepGlobe datasets underscore the superior performance of SLMFNet compared to various state-of-the-art methods. Ablation studies affirm the efficacy and precision of SAMs in the proposed model.
Collapse
Affiliation(s)
- Xin Li
- College of Computer and Information, Hohai University, Nanjing, Jiangsu, China
| | - Hejing Zhao
- Water History Department, China Institute of Water Resources and Hydropower Research, Beijing, China
- Research Center on Flood and Drought Disaster Reduction of Ministry of Water Resource, China Institute of Water Resources and Hydropower Research, Beijing, China
| | - Dan Wu
- Information Engineering Center, Yellow River Institute of Hydraulic Research, Yellow River Conservancy Commission of the Ministry of Water Resources, Zhengzhou, Henan, China
- Key Laboratory of Yellow River Sediment Research, MWR (Ministry of Water Resources), Zhengzhou, Henan, China
- Henan Engineering Research Center of Smart Water Conservancy, Yellow River Institute of Hydraulic Research, Zhengzhou, Henan, China
| | - Qixing Liu
- Information Engineering Center, Yellow River Institute of Hydraulic Research, Yellow River Conservancy Commission of the Ministry of Water Resources, Zhengzhou, Henan, China
- Key Laboratory of Yellow River Sediment Research, MWR (Ministry of Water Resources), Zhengzhou, Henan, China
- Henan Engineering Research Center of Smart Water Conservancy, Yellow River Institute of Hydraulic Research, Zhengzhou, Henan, China
| | - Rui Tang
- Department of Orthopedics, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Linyang Li
- School of Geodesy and Geomatics, Wuhan University, Wuhan, Hubei, China
| | - Zhennan Xu
- College of Computer and Information, Hohai University, Nanjing, Jiangsu, China
| | - Xin Lyu
- College of Computer and Information, Hohai University, Nanjing, Jiangsu, China
| |
Collapse
|
129
|
Liang L, Ma H, Zhao L, Xie X, Hua C, Zhang M, Zhang Y. Vehicle Detection Algorithms for Autonomous Driving: A Review. SENSORS (BASEL, SWITZERLAND) 2024; 24:3088. [PMID: 38793942 PMCID: PMC11125132 DOI: 10.3390/s24103088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2024] [Revised: 05/07/2024] [Accepted: 05/10/2024] [Indexed: 05/26/2024]
Abstract
Autonomous driving, as a pivotal technology in modern transportation, is progressively transforming the modalities of human mobility. In this domain, vehicle detection is a significant research direction that involves the intersection of multiple disciplines, including sensor technology and computer vision. In recent years, many excellent vehicle detection methods have been reported, but few studies have focused on summarizing and analyzing these algorithms. This work provides a comprehensive review of existing vehicle detection algorithms and discusses their practical applications in the field of autonomous driving. First, we provide a brief description of the tasks, evaluation metrics, and datasets for vehicle detection. Second, more than 200 classical and latest vehicle detection algorithms are summarized in detail, including those based on machine vision, LiDAR, millimeter-wave radar, and sensor fusion. Finally, this article discusses the strengths and limitations of different algorithms and sensors, and proposes future trends.
Collapse
Affiliation(s)
- Liang Liang
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
| | - Haihua Ma
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
- Key Laboratory of Grain Information Processing and Control of Ministry of Education, Henan University of Technology, Zhengzhou 450001, China
| | - Le Zhao
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
- Key Laboratory of Grain Information Processing and Control of Ministry of Education, Henan University of Technology, Zhengzhou 450001, China
| | - Xiaopeng Xie
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
| | - Chengxin Hua
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
| | - Miao Zhang
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
- Key Laboratory of Grain Information Processing and Control of Ministry of Education, Henan University of Technology, Zhengzhou 450001, China
| | - Yonghui Zhang
- College of Information Science and Engineering, Henan University of Technology, Zhengzhou 450001, China; (L.L.); (L.Z.); (X.X.)
| |
Collapse
|
130
|
Wen C, Li B, Yang Y, Feng Y, Liu J, Zhang L, Zhang Y, Li N, Liu J, Wang L, Zhang M, Liu Y. WITHDRAWN: Coronary artery segmentation based on ACMA-Net and unscented Kalman filter algorithm. Comput Biol Med 2024:108615. [PMID: 38910075 DOI: 10.1016/j.compbiomed.2024.108615] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 03/30/2024] [Accepted: 05/11/2024] [Indexed: 06/25/2024]
Abstract
This article has been withdrawn at the request of the author(s) and/or editor. The Publisher apologizes for any inconvenience this may cause. The full Elsevier Policy on Article Withdrawal can be found at https://www.elsevier.com/about/policies/article-withdrawal.
Collapse
Affiliation(s)
- Chuanqi Wen
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China.
| | - Bao Li
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| | - Yang Yang
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| | - Yili Feng
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| | - Jincheng Liu
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| | - Liyuan Zhang
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| | - Yanping Zhang
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| | - Na Li
- Shandong First Medical University & Shandong Academy of Medical Sciences, Taian, 271016, China
| | - Jian Liu
- Department of Cardiology, Peking University People's Hospital, Beijing, 100444, China
| | - Lihua Wang
- Department of Radiology, The Second Affiliated Hospital of Zhejiang University School of Medicine, Zhejiang, 310003, China
| | - Mingzi Zhang
- Department of Biomedical Sciences, Macquarie Medical School, Macquarie University, Sydney, Australia
| | - Youjun Liu
- Department of Biomedical Engineering, Faculty of Environment and Life, Beijing University of Technology, Beijing, 100124, China
| |
Collapse
|
131
|
Alajaji SA, Khoury ZH, Jessri M, Sciubba JJ, Sultan AS. An Update on the Use of Artificial Intelligence in Digital Pathology for Oral Epithelial Dysplasia Research. Head Neck Pathol 2024; 18:38. [PMID: 38727841 PMCID: PMC11087425 DOI: 10.1007/s12105-024-01643-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 03/30/2024] [Indexed: 05/13/2024]
Abstract
INTRODUCTION Oral epithelial dysplasia (OED) is a precancerous histopathological finding which is considered the most important prognostic indicator for determining the risk of malignant transformation into oral squamous cell carcinoma (OSCC). The gold standard for diagnosis and grading of OED is through histopathological examination, which is subject to inter- and intra-observer variability, impacting accurate diagnosis and prognosis. The aim of this review article is to examine the current advances in digital pathology for artificial intelligence (AI) applications used for OED diagnosis. MATERIALS AND METHODS We included studies that used AI for diagnosis, grading, or prognosis of OED on histopathology images or intraoral clinical images. Studies utilizing imaging modalities other than routine light microscopy (e.g., scanning electron microscopy), or immunohistochemistry-stained histology slides, or immunofluorescence were excluded from the study. Studies not focusing on oral dysplasia grading and diagnosis, e.g., to discriminate OSCC from normal epithelial tissue were also excluded. RESULTS A total of 24 studies were included in this review. Nineteen studies utilized deep learning (DL) convolutional neural networks for histopathological OED analysis, and 4 used machine learning (ML) models. Studies were summarized by AI method, main study outcomes, predictive value for malignant transformation, strengths, and limitations. CONCLUSION ML/DL studies for OED grading and prediction of malignant transformation are emerging as promising adjunctive tools in the field of digital pathology. These adjunctive objective tools can ultimately aid the pathologist in more accurate diagnosis and prognosis prediction. However, further supportive studies that focus on generalization, explainable decisions, and prognosis prediction are needed.
Collapse
Affiliation(s)
- Shahd A Alajaji
- Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, 650 W. Baltimore Street, 7 Floor, Baltimore, MD, 21201, USA
- Department of Oral Medicine and Diagnostic Sciences, College of Dentistry, King Saud University, Riyadh, Saudi Arabia
- Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA
| | - Zaid H Khoury
- Department of Oral Diagnostic Sciences and Research, Meharry Medical College School of Dentistry, Nashville, TN, USA
| | - Maryam Jessri
- Oral Medicine and Pathology Department, School of Dentistry, University of Queensland, Herston, QLD, Australia
- Oral Medicine Department, Metro North Hospital and Health Services, Queensland Health, Brisbane, QLD, Australia
| | - James J Sciubba
- Department of Otolaryngology, Head & Neck Surgery, The Johns Hopkins University, Baltimore, MD, USA
| | - Ahmed S Sultan
- Department of Oncology and Diagnostic Sciences, University of Maryland School of Dentistry, 650 W. Baltimore Street, 7 Floor, Baltimore, MD, 21201, USA.
- Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA.
- University of Maryland Marlene and Stewart Greenebaum Comprehensive Cancer Center, Baltimore, MD, USA.
| |
Collapse
|
132
|
Kujawa A, Dorent R, Connor S, Thomson S, Ivory M, Vahedi A, Guilhem E, Wijethilake N, Bradford R, Kitchen N, Bisdas S, Ourselin S, Vercauteren T, Shapey J. Deep learning for automatic segmentation of vestibular schwannoma: a retrospective study from multi-center routine MRI. Front Comput Neurosci 2024; 18:1365727. [PMID: 38784680 PMCID: PMC11111906 DOI: 10.3389/fncom.2024.1365727] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 04/17/2024] [Indexed: 05/25/2024] Open
Abstract
Automatic segmentation of vestibular schwannoma (VS) from routine clinical MRI has potential to improve clinical workflow, facilitate treatment decisions, and assist patient management. Previous work demonstrated reliable automatic segmentation performance on datasets of standardized MRI images acquired for stereotactic surgery planning. However, diagnostic clinical datasets are generally more diverse and pose a larger challenge to automatic segmentation algorithms, especially when post-operative images are included. In this work, we show for the first time that automatic segmentation of VS on routine MRI datasets is also possible with high accuracy. We acquired and publicly release a curated multi-center routine clinical (MC-RC) dataset of 160 patients with a single sporadic VS. For each patient up to three longitudinal MRI exams with contrast-enhanced T1-weighted (ceT1w) (n = 124) and T2-weighted (T2w) (n = 363) images were included and the VS manually annotated. Segmentations were produced and verified in an iterative process: (1) initial segmentations by a specialized company; (2) review by one of three trained radiologists; and (3) validation by an expert team. Inter- and intra-observer reliability experiments were performed on a subset of the dataset. A state-of-the-art deep learning framework was used to train segmentation models for VS. Model performance was evaluated on a MC-RC hold-out testing set, another public VS datasets, and a partially public dataset. The generalizability and robustness of the VS deep learning segmentation models increased significantly when trained on the MC-RC dataset. Dice similarity coefficients (DSC) achieved by our model are comparable to those achieved by trained radiologists in the inter-observer experiment. On the MC-RC testing set, median DSCs were 86.2(9.5) for ceT1w, 89.4(7.0) for T2w, and 86.4(8.6) for combined ceT1w+T2w input images. On another public dataset acquired for Gamma Knife stereotactic radiosurgery our model achieved median DSCs of 95.3(2.9), 92.8(3.8), and 95.5(3.3), respectively. In contrast, models trained on the Gamma Knife dataset did not generalize well as illustrated by significant underperformance on the MC-RC routine MRI dataset, highlighting the importance of data variability in the development of robust VS segmentation models. The MC-RC dataset and all trained deep learning models were made available online.
Collapse
Affiliation(s)
- Aaron Kujawa
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Reuben Dorent
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Steve Connor
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
- Department of Neuroradiology, King's College Hospital, London, United Kingdom
- Department of Radiology, Guy's and St Thomas' Hospital, London, United Kingdom
| | - Suki Thomson
- Department of Neuroradiology, King's College Hospital, London, United Kingdom
| | - Marina Ivory
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Ali Vahedi
- Department of Neuroradiology, King's College Hospital, London, United Kingdom
| | - Emily Guilhem
- Department of Neuroradiology, King's College Hospital, London, United Kingdom
| | - Navodini Wijethilake
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Robert Bradford
- Queen Square Radiosurgery Centre (Gamma Knife), National Hospital for Neurology and Neurosurgery, London, United Kingdom
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, United Kingdom
| | - Neil Kitchen
- Queen Square Radiosurgery Centre (Gamma Knife), National Hospital for Neurology and Neurosurgery, London, United Kingdom
- Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, United Kingdom
| | - Sotirios Bisdas
- Department of Neuroradiology, National Hospital for Neurology and Neurosurgery, London, United Kingdom
| | - Sebastien Ourselin
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Tom Vercauteren
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
| | - Jonathan Shapey
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
- Department of Neurosurgery, King's College Hospital, London, United Kingdom
| |
Collapse
|
133
|
Huang H, Wang Y, Cai M, Wang R, Wen F, Hu X. Adaptive temporal compression for reduction of computational complexity in human behavior recognition. Sci Rep 2024; 14:10560. [PMID: 38720020 DOI: 10.1038/s41598-024-61286-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 05/03/2024] [Indexed: 05/12/2024] Open
Abstract
The research on video analytics especially in the area of human behavior recognition has become increasingly popular recently. It is widely applied in virtual reality, video surveillance, and video retrieval. With the advancement of deep learning algorithms and computer hardware, the conventional two-dimensional convolution technique for training video models has been replaced by three-dimensional convolution, which enables the extraction of spatio-temporal features. Specifically, the use of 3D convolution in human behavior recognition has been the subject of growing interest. However, the increased dimensionality has led to challenges such as the dramatic increase in the number of parameters, increased time complexity, and a strong dependence on GPUs for effective spatio-temporal feature extraction. The training speed can be considerably slow without the support of powerful GPU hardware. To address these issues, this study proposes an Adaptive Time Compression (ATC) module. Functioning as an independent component, ATC can be seamlessly integrated into existing architectures and achieves data compression by eliminating redundant frames within video data. The ATC module effectively reduces GPU computing load and time complexity with negligible loss of accuracy, thereby facilitating real-time human behavior recognition.
Collapse
Affiliation(s)
- Haixin Huang
- School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, 110159, China
| | - Yuyao Wang
- School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, 110159, China
| | - Mingqi Cai
- School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, 110159, China
| | - Ruipeng Wang
- School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, 110159, China
| | - Feng Wen
- School of Information Science and Engineering, Shenyang Ligong University, Shenyang, 110159, China
| | - Xiaojie Hu
- School of Information Science and Engineering, Shenyang Ligong University, Shenyang, 110159, China.
| |
Collapse
|
134
|
Zhang G, He Z, Zhang Y, Li Z, Wu L. SC-Net: Symmetrical conical network for colorectal pathology image segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 248:108119. [PMID: 38520785 DOI: 10.1016/j.cmpb.2024.108119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/04/2023] [Revised: 02/25/2024] [Accepted: 03/04/2024] [Indexed: 03/25/2024]
Abstract
BACKGROUND AND OBJECTIVE Image segmentation of histopathology of colorectal cancer is a core task of computer aided medical image diagnosis system. Existing convolutional neural networks generally extract multi-scale information in linear flow structures by inserting multi-branch modules, which is difficult to extract heterogeneous semantic information under multi-level and different receptive field and tough to establish context dependency among different receptive field features. METHODS To address these issues, we propose a symmetric spiral progressive feature fusion encoder-decoder network called the Symmetric Conical Network (SC-Net). First, we design a Multi-scale Feature Extraction Block (MFEB) matching with the Symmetric Conical Network to obtain multi-branch heterogeneous semantic information under different receptive fields, so as to enrich the diversity of extracted feature information. The encoder is composed of MFEB through spiral and multi-branch arrangement to enhance context dependence between different information flow. Secondly, the information loss of contour, color and others in high-level semantic information through causally stacking MFEB, the Feature Mapping Layer (FML) is designed to map low-level features to high-level semantic features along the down-sampling branch and solve the problem of insufficient global feature extraction in deep levels. RESULTS The SC-Net was evaluated on our self-constructed colorectal cancer dataset, a publicly available breast cancer dataset and a polyp dataset. The results revealed that the mDice of segmentation reached 0.8611, 0.7259 and 0.7144. We compare our model with the state-of-art semantic segmentation UNet++, PSPNet, Attention U-Net, R2U-Net and other advanced segmentation networks. The experimental results demonstrate that we achieve the most advanced performance. CONCLUSIONS The results indicate that the proposed SC-Net excels in segmenting H&E stained pathology images, effectively preserving morphological features and spatial information even in scenarios with weak texture, poor contrast, and variations in appearance.
Collapse
Affiliation(s)
- Gang Zhang
- Faculty of Mechanical and Electrical Engineering, Kunming University of Science and Technology, Kunming 650500, China.
| | - Zifen He
- Faculty of Mechanical and Electrical Engineering, Kunming University of Science and Technology, Kunming 650500, China.
| | - Yinhui Zhang
- Faculty of Mechanical and Electrical Engineering, Kunming University of Science and Technology, Kunming 650500, China.
| | - Zhenhui Li
- Yunnan Cancer Hospital, Department of Radiology, The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Center, Kunming 650118, China.
| | - Lin Wu
- Yunnan Cancer Hospital, Department of Radiology, The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Center, Kunming 650118, China.
| |
Collapse
|
135
|
Jiang X, Zhuang X, Chen J, Zhang J, Zhang Y. YOLOv8-MU: An Improved YOLOv8 Underwater Detector Based on a Large Kernel Block and a Multi-Branch Reparameterization Module. SENSORS (BASEL, SWITZERLAND) 2024; 24:2905. [PMID: 38733010 PMCID: PMC11086134 DOI: 10.3390/s24092905] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/09/2024] [Revised: 04/25/2024] [Accepted: 04/30/2024] [Indexed: 05/13/2024]
Abstract
Underwater visual detection technology is crucial for marine exploration and monitoring. Given the growing demand for accurate underwater target recognition, this study introduces an innovative architecture, YOLOv8-MU, which significantly enhances the detection accuracy. This model incorporates the large kernel block (LarK block) from UniRepLKNet to optimize the backbone network, achieving a broader receptive field without increasing the model's depth. Additionally, the integration of C2fSTR, which combines the Swin transformer with the C2f module, and the SPPFCSPC_EMA module, which blends Cross-Stage Partial Fast Spatial Pyramid Pooling (SPPFCSPC) with attention mechanisms, notably improves the detection accuracy and robustness for various biological targets. A fusion block from DAMO-YOLO further enhances the multi-scale feature extraction capabilities in the model's neck. Moreover, the adoption of the MPDIoU loss function, designed around the vertex distance, effectively addresses the challenges of localization accuracy and boundary clarity in underwater organism detection. The experimental results on the URPC2019 dataset indicate that YOLOv8-MU achieves an mAP@0.5 of 78.4%, showing an improvement of 4.0% over the original YOLOv8 model. Additionally, on the URPC2020 dataset, it achieves 80.9%, and, on the Aquarium dataset, it reaches 75.5%, surpassing other models, including YOLOv5 and YOLOv8n, thus confirming the wide applicability and generalization capabilities of our proposed improved model architecture. Furthermore, an evaluation on the improved URPC2019 dataset demonstrates leading performance (SOTA), with an mAP@0.5 of 88.1%, further verifying its superiority on this dataset. These results highlight the model's broad applicability and generalization capabilities across various underwater datasets.
Collapse
Affiliation(s)
| | | | | | - Jian Zhang
- School of Tropical Agriculture and Forestry (School of Agricultural and Rural, School of Rural Revitalization), Hainan University, Danzhou 571737, China; (X.J.); (X.Z.); (J.C.); (Y.Z.)
| | | |
Collapse
|
136
|
Peng J, Wang P, Pedersoli M, Desrosiers C. Boundary-aware information maximization for self-supervised medical image segmentation. Med Image Anal 2024; 94:103150. [PMID: 38574545 DOI: 10.1016/j.media.2024.103150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 02/24/2024] [Accepted: 03/20/2024] [Indexed: 04/06/2024]
Abstract
Self-supervised representation learning can boost the performance of a pre-trained network on downstream tasks for which labeled data is limited. A popular method based on this paradigm, known as contrastive learning, works by constructing sets of positive and negative pairs from the data, and then pulling closer the representations of positive pairs while pushing apart those of negative pairs. Although contrastive learning has been shown to improve performance in various classification tasks, its application to image segmentation has been more limited. This stems in part from the difficulty of defining positive and negative pairs for dense feature maps without having access to pixel-wise annotations. In this work, we propose a novel self-supervised pre-training method that overcomes the challenges of contrastive learning in image segmentation. Our method leverages Information Invariant Clustering (IIC) as an unsupervised task to learn a local representation of images in the decoder of a segmentation network, but addresses three important drawbacks of this approach: (i) the difficulty of optimizing the loss based on mutual information maximization; (ii) the lack of clustering consistency for different random transformations of the same image; (iii) the poor correspondence of clusters obtained by IIC with region boundaries in the image. Toward this goal, we first introduce a regularized mutual information maximization objective that encourages the learned clusters to be balanced and consistent across different image transformations. We also propose a boundary-aware loss based on cross-correlation, which helps the learned clusters to be more representative of important regions in the image. Compared to contrastive learning applied in dense features, our method does not require computing positive and negative pairs and also enhances interpretability through the visualization of learned clusters. Comprehensive experiments involving four different medical image segmentation tasks reveal the high effectiveness of our self-supervised representation learning method. Our results show the proposed method to outperform by a large margin several state-of-the-art self-supervised and semi-supervised approaches for segmentation, reaching a performance close to full supervision with only a few labeled examples.
Collapse
Affiliation(s)
- Jizong Peng
- ETS Montréal, 1100 Notre-Dame St W, Montreal H3C 1K3, QC, Canada.
| | - Ping Wang
- ETS Montréal, 1100 Notre-Dame St W, Montreal H3C 1K3, QC, Canada
| | - Marco Pedersoli
- ETS Montréal, 1100 Notre-Dame St W, Montreal H3C 1K3, QC, Canada
| | | |
Collapse
|
137
|
Kunhimon S, Shaker A, Naseer M, Khan S, Khan FS. Learnable weight initialization for volumetric medical image segmentation. Artif Intell Med 2024; 151:102863. [PMID: 38593682 DOI: 10.1016/j.artmed.2024.102863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Revised: 03/27/2024] [Accepted: 03/30/2024] [Indexed: 04/11/2024]
Abstract
Hybrid volumetric medical image segmentation models, combining the advantages of local convolution and global attention, have recently received considerable attention. While mainly focusing on architectural modifications, most existing hybrid approaches still use conventional data-independent weight initialization schemes which restrict their performance due to ignoring the inherent volumetric nature of the medical data. To address this issue, we propose a learnable weight initialization approach that utilizes the available medical training data to effectively learn the contextual and structural cues via the proposed self-supervised objectives. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach, leading to state-of-the-art segmentation performance. Our proposed data-dependent initialization approach performs favorably as compared to the Swin-UNETR model pretrained using large-scale datasets on multi-organ segmentation task. Our source code and models are available at: https://github.com/ShahinaKK/LWI-VMS.
Collapse
Affiliation(s)
- Shahina Kunhimon
- Mohammed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates.
| | - Abdelrahman Shaker
- Mohammed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
| | - Muzammal Naseer
- Mohammed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
| | - Salman Khan
- Mohammed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
| | - Fahad Shahbaz Khan
- Mohammed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates; Linkoping University, Sweden
| |
Collapse
|
138
|
Zhao X, Lai L, Li Y, Zhou X, Cheng X, Chen Y, Huang H, Guo J, Wang G. A lightweight bladder tumor segmentation method based on attention mechanism. Med Biol Eng Comput 2024; 62:1519-1534. [PMID: 38308022 DOI: 10.1007/s11517-024-03018-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Accepted: 01/05/2024] [Indexed: 02/04/2024]
Abstract
In the endoscopic images of bladder, accurate segmentation of different grade bladder tumor from blurred boundary regions and highly variable shapes is of great significance for doctors' diagnosis and patients' later treatment. We propose a nested attentional feature fusion segmentation network (NAFF-Net) based on the encoder-decoder structure formed by the combination of weighted pyramid pooling module (WPPM) and nested attentional feature fusion (NAFF). Among them, WPPM applies the cascade of atrous convolution to enhance the overall perceptual field while introducing adaptive weights to optimize multi-scale feature extraction, NAFF integrates deep semantic information into shallow feature maps, effectively focusing on edge and detail information in bladder tumor images. Additionally, a weighted mixed loss function is constructed to alleviate the impact of imbalance between positive and negative sample distribution on segmentation accuracy. Experiments illustrate the proposed NAFF-Net achieves better segmentation results compared to other mainstream models, with a MIoU of 84.05%, MPrecision of 91.52%, MRecall of 90.81%, and F1-score of 91.16%, and also achieves good results on the public datasets Kvasir-SEG and CVC-ClinicDB. Compared to other models, NAFF-Net has a smaller number of parameters, which is a significant advantage in model deployment.
Collapse
Affiliation(s)
- Xiushun Zhao
- School of Automation, Guangdong University of Technology, Guangzhou, 510006, China
| | - Libing Lai
- Department of Urology, The First Affiliated Hospital of Nanchang University, Nanchang, 330006, China
| | - Yunjiao Li
- School of Automation, Guangdong University of Technology, Guangzhou, 510006, China
| | - Xiaochen Zhou
- Department of Urology, The First Affiliated Hospital of Nanchang University, Nanchang, 330006, China
| | - Xiaofeng Cheng
- Department of Urology, The First Affiliated Hospital of Nanchang University, Nanchang, 330006, China
| | - Yujun Chen
- Department of Urology, The First Affiliated Hospital of Nanchang University, Nanchang, 330006, China
| | - Haohui Huang
- School of Automation, Guangdong University of Technology, Guangzhou, 510006, China
| | - Jing Guo
- School of Automation, Guangdong University of Technology, Guangzhou, 510006, China.
| | - Gongxian Wang
- Department of Urology, The First Affiliated Hospital of Nanchang University, Nanchang, 330006, China.
| |
Collapse
|
139
|
Jeong H, Lim H, Yoon C, Won J, Lee GY, de la Rosa E, Kirschke JS, Kim B, Kim N, Kim C. Robust Ensemble of Two Different Multimodal Approaches to Segment 3D Ischemic Stroke Segmentation Using Brain Tumor Representation Among Multiple Center Datasets. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01099-6. [PMID: 38693333 DOI: 10.1007/s10278-024-01099-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/23/2023] [Revised: 03/18/2024] [Accepted: 03/22/2024] [Indexed: 05/03/2024]
Abstract
Ischemic stroke segmentation at an acute stage is vital in assessing the severity of patients' impairment and guiding therapeutic decision-making for reperfusion. Although many deep learning studies have shown attractive performance in medical segmentation, it is difficult to use these models trained on public data with private hospitals' datasets. Here, we demonstrate an ensemble model that employs two different multimodal approaches for generalization, a more effective way to perform on external datasets. First, after we jointly train a segmentation model on diffusion-weighted imaging (DWI) and apparent diffusion coefficient (ADC) MR modalities, the model is inferred on the DWI images. Second, a channel-wise segmentation model is trained by concatenating the DWI and ADC images as input, and then is inferred using both MR modalities. Before training with ischemic stroke data, we utilized BraTS 2021, a public brain tumor dataset, for transfer learning. An extensive ablation study evaluates which strategy learns better representations for ischemic stroke segmentation. In our study, nnU-Net well-known for robustness is selected as our baseline model. Our proposed method is evaluated on three different datasets: the Asan Medical Center (AMC) I and II, and the 2022 Ischemic Stroke Lesion Segmentation (ISLES). Our experiments are widely validated over a large, multi-center, and multi-scanner dataset with a huge amount of 846 scans. Not only stroke lesion models can benefit from transfer learning using brain tumor data, but combining the MR modalities using different training schemes also highly improves segmentation performance. The method achieved a top-1 ranking in the ongoing ISLES'22 challenge and performed particularly well on lesion-wise metrics of interest to neuroradiologists, achieving a Dice coefficient of 78.69% and a lesion-wise F1 score of 82.46%. Also, the method was relatively robust on the AMC I (Dice, 60.35%; lesion-wise F1, 68.30%) and II (Dice; 74.12%; lesion-wise F1, 67.53%) datasets in different settings. The high segmentation accuracy of our proposed method could improve radiologists' ability to detect ischemic stroke lesions in MRI images. Our model weights and inference code are available on https://github.com/MDOpx/ISLES22-model-inference .
Collapse
Affiliation(s)
- Hyunsu Jeong
- Graduate School of Artificial Intelligence (GSAI), Department of Electrical Engineering, Medical Science and Engineering, and Medical Device Innovation Center, Convergence IT Engineering, Mechanical Engineering, Pohang University of Science and Technology (POSTECH), Pohang, South Korea
| | - Hyunseok Lim
- Graduate School of Artificial Intelligence (GSAI), Department of Electrical Engineering, Medical Science and Engineering, and Medical Device Innovation Center, Convergence IT Engineering, Mechanical Engineering, Pohang University of Science and Technology (POSTECH), Pohang, South Korea
| | - Chiho Yoon
- Graduate School of Artificial Intelligence (GSAI), Department of Electrical Engineering, Medical Science and Engineering, and Medical Device Innovation Center, Convergence IT Engineering, Mechanical Engineering, Pohang University of Science and Technology (POSTECH), Pohang, South Korea
| | - Jongjun Won
- Department of Medical Science, Asan Medical Center, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Seoul, Republic of Korea
| | - Grace Yoojin Lee
- Department of Medical Science, Asan Medical Center, Asan Medical Institute of Convergence Science and Technology, University of Ulsan College of Medicine, Seoul, Republic of Korea
| | - Ezequiel de la Rosa
- icometrix, Leuven, Belgium
- Department of Informatics, Technical University of Munich, Neuroradiology Munich, Germany
| | - Jan S Kirschke
- Department of Informatics, Technical University of Munich, Neuroradiology Munich, Germany
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum Rechtsder Isar, Technical University of Munich, Munich, Germany
| | - Bumjoon Kim
- Department of Biomedical Engineering, Convergence Medicine, Radiology, Neurology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea.
| | - Namkug Kim
- Department of Biomedical Engineering, Convergence Medicine, Radiology, Neurology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, South Korea.
| | - Chulhong Kim
- Graduate School of Artificial Intelligence (GSAI), Department of Electrical Engineering, Medical Science and Engineering, and Medical Device Innovation Center, Convergence IT Engineering, Mechanical Engineering, Pohang University of Science and Technology (POSTECH), Pohang, South Korea.
| |
Collapse
|
140
|
Dombrowski AK, Gerken JE, Muller KR, Kessel P. Diffeomorphic Counterfactuals With Generative Models. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:3257-3274. [PMID: 38055368 DOI: 10.1109/tpami.2023.3339980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/08/2023]
Abstract
Counterfactuals can explain classification decisions of neural networks in a human interpretable way. We propose a simple but effective method to generate such counterfactuals. More specifically, we perform a suitable diffeomorphic coordinate transformation and then perform gradient ascent in these coordinates to find counterfactuals which are classified with great confidence as a specified target class. We propose two methods to leverage generative models to construct such suitable coordinate systems that are either exactly or approximately diffeomorphic. We analyze the generation process theoretically using Riemannian differential geometry and validate the quality of the generated counterfactuals using various qualitative and quantitative measures.
Collapse
|
141
|
Li J, Chen P, Yu S, Liu S, Jia J. BAL: Balancing Diversity and Novelty for Active Learning. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:3653-3664. [PMID: 38133981 DOI: 10.1109/tpami.2023.3345844] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/24/2023]
Abstract
The objective of Active Learning is to strategically label a subset of the dataset to maximize performance within a predetermined labeling budget. In this study, we harness features acquired through self-supervised learning. We introduce a straightforward yet potent metric, Cluster Distance Difference, to identify diverse data. Subsequently, we introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data. Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%. Moreover, we assess the efficacy of our proposed framework under extended settings, encompassing both larger and smaller labeling budgets. Experimental results demonstrate that, when labeling 80% of the samples, the performance of the current SOTA method declines by 0.74%, whereas our proposed BAL achieves performance comparable to the full dataset.
Collapse
|
142
|
Wu Y, Xing M, Zhang Y, Xie Y, Wu Z, Qu Y. Perturbed Progressive Learning for Semisupervised Defect Segmentation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:6118-6132. [PMID: 37874733 DOI: 10.1109/tnnls.2023.3324188] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/26/2023]
Abstract
Recently, with the development of intelligent manufacturing, the demand for surface defect inspection has been increasing. Deep learning has achieved promising results in defect inspection. However, due to the rareness of defect data and the difficulties of pixelwise annotation, the existing supervised defect inspection methods are too inferior to be implemented in practice. To solve the problem of defect segmentation with few labeled data, we propose a simple and efficient method for semisupervised defect segmentation (SSDS), named perturbed progressive learning (PPL). On the one hand, PPL decouples the predictions of student and teacher networks as well as alleviates overfitting on noisy pseudo-labels. On the other hand, PPL encourages consistency across various perturbations in a broader stagewise scope, alleviating drift caused by the noisy pseudo-labels. Specifically, PPL contains two training stages. In the first stage, the teacher network gives the unlabeled data with pseudo-labels that are divided into the easy and hard groups. The labeled data and the unlabeled data in the easy group with their perturbation are both used to train for a better-performing student network. In the second stage, the unlabeled data in the hard group are predicted by the obtained student network, so the refined pseudo-labeled data are enlarged. All the pseudo-labeling data and labeled data with their perturbation are used to retrain the student network, progressively improving the defect feature representation. We build a mobile screen defect dataset (MSDD-3) with three classes of defects. PPL is implemented on MSDD-3 as well as other public datasets. Extensive experimental results demonstrate that PPL significantly surpasses the state-of-the-art methods across all evaluation partition protocols.
Collapse
|
143
|
Ji X, Zhu G, Gou J, Chen S, Zhao W, Sun Z, Fu H, Wang H. A fully automatic deep learning-based method for segmenting regions of interest and predicting renal function in pediatric dynamic renal scintigraphy. Ann Nucl Med 2024; 38:382-390. [PMID: 38376629 DOI: 10.1007/s12149-024-01907-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 01/17/2024] [Indexed: 02/21/2024]
Abstract
OBJECTIVE Accurate delineation of renal regions of interest (ROIs) is critical for the assessment of renal function in pediatric dynamic renal scintigraphy (DRS). The purpose of this study was to develop and evaluate a deep learning (DL) model that can fully automatically delineate renal ROIs and calculate renal function in pediatric 99mTechnetium-ethylenedicysteine (99mTc-EC) DRS. METHODS This study retrospectively analyzed 1,283 pediatric DRS data at a single center from January to December 2018. These patients were divided into training set (n = 1027), validation set (n = 128), and testing set (n = 128). A fully automatic segmentation of ROIs (FASR) model was developed and evaluated. The pixel values of the automatically segmented ROIs were calculated to predict renal blood perfusion rate (BPR) and differential renal function (DRF). Precision, recall rate, intersection over union (IOU), and Dice similarity coefficient (DSC) were used to evaluate the performance of FASR model. Intraclass correlation (ICC) and Pearson correlation analysis were used to compare the consistency of automatic and manual method in assessing the renal function parameters in the testing set. RESULTS The FASR model achieved a precision of 0.88, recall rate of 0.94, IOU of 0.83, and DSC of 0.91. In the testing set, the r values of BPR and DRF calculated by the two methods were 0.94 (P < 0.01) and 0.97 (P < 0.01), and the ICCs (95% confidence interval CI) were 0.94 (0.90-0.96) and 0.94 (0.91-0.96). CONCLUSION We propose a reliable and stable DL model that can fully automatically segment ROIs and accurately predict renal function in pediatric 99mTc-EC DRS.
Collapse
Affiliation(s)
- Xueli Ji
- Department of Nuclear Medicine, Xinhua Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200092, China
| | - Guohui Zhu
- Institute of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200092, China
| | - Jinyu Gou
- Department of Nuclear Medicine, Xinhua Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200092, China
| | - Suyun Chen
- Department of Nuclear Medicine, Xinhua Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200092, China
| | - Wenyu Zhao
- Department of Nuclear Medicine, Xinhua Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200092, China
| | - Zhanquan Sun
- Institute of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200092, China.
| | - Hongliang Fu
- Department of Nuclear Medicine, Xinhua Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200092, China.
| | - Hui Wang
- Department of Nuclear Medicine, Xinhua Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200092, China.
| |
Collapse
|
144
|
Su D, Luo J, Fei C. An Efficient and Rapid Medical Image Segmentation Network. IEEE J Biomed Health Inform 2024; 28:2979-2990. [PMID: 38457317 DOI: 10.1109/jbhi.2024.3374780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/10/2024]
Abstract
Accurate medical image segmentation is an essential part of the medical image analysis process that provides detailed quantitative metrics. In recent years, extensions of classical networks such as UNet have achieved state-of-the-art performance on medical image segmentation tasks. However, the high model complexity of these networks limits their applicability to devices with constrained computational resources. To alleviate this problem, we propose a shallow hierarchical Transformer for medical image segmentation, called SHFormer. By decreasing the number of transformer blocks utilized, the model complexity of SHFormer can be reduced to an acceptable level. To improve the learned attention while keeping the structure lightweight, we propose a spatial-channel connection module. This module separately learns attention in the spatial and channel dimensions of the feature while interconnecting them to produce more focused attention. To keep the decoder lightweight, the MLP-D module is proposed to progressively fuse multi-scale features in which channels are aligned using Multi-Layer Perceptron (MLP) and spatial information is fused by convolutional blocks. We first validated the performance of SHFormer on the ISIC-2018 dataset. Compared to the latest network, SHFormer exhibits comparable performance with 15 times fewer parameters, 30 times lower computational complexity and 5 times higher inference efficiency. To test the generalizability of SHFormer, we introduced the polyp dataset for additional testing. SHFormer achieves comparable segmentation accuracy to the latest network while having lower computational overhead.
Collapse
|
145
|
Ma X, Ouyang W, Simonelli A, Ricci E. 3D Object Detection From Images for Autonomous Driving: A Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:3537-3556. [PMID: 38145536 DOI: 10.1109/tpami.2023.3346386] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/27/2023]
Abstract
3D object detection from images, one of the fundamental and challenging problems in autonomous driving, has received increasing attention from both industry and academia in recent years. Benefiting from the rapid development of deep learning technologies, image-based 3D detection has achieved remarkable progress. Particularly, more than 200 works have studied this problem from 2015 to 2021, encompassing a broad spectrum of theories, algorithms, and applications. However, to date no recent survey exists to collect and organize this knowledge. In this paper, we fill this gap in the literature and provide the first comprehensive survey of this novel and continuously growing research field, summarizing the most commonly used pipelines for image-based 3D detection and deeply analyzing each of their components. Additionally, we also propose two new taxonomies to organize the state-of-the-art methods into different categories, with the intent of providing a more systematic review of existing methods and facilitating fair comparisons with future works. In retrospect of what has been achieved so far, we also analyze the current challenges in the field and discuss future directions for image-based 3D detection research.
Collapse
|
146
|
Ghamsarian N, Wolf S, Zinkernagel M, Schoeffmann K, Sznitman R. DeepPyramid+: medical image segmentation using Pyramid View Fusion and Deformable Pyramid Reception. Int J Comput Assist Radiol Surg 2024; 19:851-859. [PMID: 38189905 DOI: 10.1007/s11548-023-03046-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 12/07/2023] [Indexed: 01/09/2024]
Abstract
PURPOSE Semantic segmentation plays a pivotal role in many applications related to medical image and video analysis. However, designing a neural network architecture for medical image and surgical video segmentation is challenging due to the diverse features of relevant classes, including heterogeneity, deformability, transparency, blunt boundaries, and various distortions. We propose a network architecture, DeepPyramid+, which addresses diverse challenges encountered in medical image and surgical video segmentation. METHODS The proposed DeepPyramid+ incorporates two major modules, namely "Pyramid View Fusion" (PVF) and "Deformable Pyramid Reception" (DPR), to address the outlined challenges. PVF replicates a deduction process within the neural network, aligning with the human visual system, thereby enhancing the representation of relative information at each pixel position. Complementarily, DPR introduces shape- and scale-adaptive feature extraction techniques using dilated deformable convolutions, enhancing accuracy and robustness in handling heterogeneous classes and deformable shapes. RESULTS Extensive experiments conducted on diverse datasets, including endometriosis videos, MRI images, OCT scans, and cataract and laparoscopy videos, demonstrate the effectiveness of DeepPyramid+ in handling various challenges such as shape and scale variation, reflection, and blur degradation. DeepPyramid+ demonstrates significant improvements in segmentation performance, achieving up to a 3.65% increase in Dice coefficient for intra-domain segmentation and up to a 17% increase in Dice coefficient for cross-domain segmentation. CONCLUSIONS DeepPyramid+ consistently outperforms state-of-the-art networks across diverse modalities considering different backbone networks, showcasing its versatility. Accordingly, DeepPyramid+ emerges as a robust and effective solution, successfully overcoming the intricate challenges associated with relevant content segmentation in medical images and surgical videos. Its consistent performance and adaptability indicate its potential to enhance precision in computerized medical image and surgical video analysis applications.
Collapse
Affiliation(s)
- Negin Ghamsarian
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland.
| | - Sebastian Wolf
- Department of Ophthalmology, Inselspital, Bern, Switzerland
| | | | - Klaus Schoeffmann
- Department of Information Technology, University of Klagenfurt, Klagenfurt, Austria
| | - Raphael Sznitman
- ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland
| |
Collapse
|
147
|
Song Q, Li J, Guo H, Huang R. Denoised Non-Local Neural Network for Semantic Segmentation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:7162-7174. [PMID: 37021852 DOI: 10.1109/tnnls.2022.3214216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
The non-local (NL) network has become a widely used technique for semantic segmentation, which computes an attention map to measure the relationships of each pixel pair. However, most of the current popular NL models tend to ignore the phenomenon that the calculated attention map appears to be very noisy, containing interclass and intraclass inconsistencies, which lowers the accuracy and reliability of the NL methods. In this article, we figuratively denote these inconsistencies as attention noises and explore the solutions to denoise them. Specifically, we inventively propose a denoised NL network, which consists of two primary modules, i.e., the global rectifying (GR) block and the local retention (LR) block, to eliminate the interclass and intraclass noises, respectively. First, GR adopts the class-level predictions to capture a binary map to distinguish whether the selected two pixels belong to the same category. Second, LR captures the ignored local dependencies and further uses them to rectify the unwanted hollows in the attention map. The experimental results on two challenging semantic segmentation datasets demonstrate the superior performance of our model. Without any external training data, our proposed denoised NL can achieve the state-of-the-art performance of 83.5% and 46.69% mean of classwise intersection over union (mIoU) on Cityscapes and ADE20K, respectively.
Collapse
|
148
|
G S, Gopalakrishnan U, Parthinarupothi RK, Madathil T. Deep learning supported echocardiogram analysis: A comprehensive review. Artif Intell Med 2024; 151:102866. [PMID: 38593684 DOI: 10.1016/j.artmed.2024.102866] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2023] [Revised: 03/20/2024] [Accepted: 03/30/2024] [Indexed: 04/11/2024]
Abstract
An echocardiogram is a sophisticated ultrasound imaging technique employed to diagnose heart conditions. The transthoracic echocardiogram, one of the most prevalent types, is instrumental in evaluating significant cardiac diseases. However, interpreting its results heavily relies on the clinician's expertise. In this context, artificial intelligence has emerged as a vital tool for helping clinicians. This study critically analyzes key state-of-the-art research that uses deep learning techniques to automate transthoracic echocardiogram analysis and support clinical judgments. We have systematically organized and categorized articles that proffer solutions for view classification, enhancement of image quality and dataset, segmentation and identification of cardiac structures, detection of cardiac function abnormalities, and quantification of cardiac functions. We compared the performance of various deep learning approaches within each category, identifying the most promising methods. Additionally, we highlight limitations in current research and explore promising avenues for future exploration. These include addressing generalizability issues, incorporating novel AI approaches, and tackling the analysis of rare cardiac diseases.
Collapse
Affiliation(s)
- Sanjeevi G
- Center for Wireless Networks & Applications (WNA), Amrita Vishwa Vidyapeetham, Amritapuri, India
| | - Uma Gopalakrishnan
- Center for Wireless Networks & Applications (WNA), Amrita Vishwa Vidyapeetham, Amritapuri, India.
| | | | - Thushara Madathil
- Department of Cardiac Anesthesiology, Amrita Institute of Medical Sciences and Research Center, Kochi, India
| |
Collapse
|
149
|
Wang YRJ, Yang K, Wen Y, Wang P, Hu Y, Lai Y, Wang Y, Zhao K, Tang S, Zhang A, Zhan H, Lu M, Chen X, Yang S, Dong Z, Wang Y, Liu H, Zhao L, Huang L, Li Y, Wu L, Chen Z, Luo Y, Liu D, Zhao P, Lin K, Wu JC, Zhao S. Screening and diagnosis of cardiovascular disease using artificial intelligence-enabled cardiac magnetic resonance imaging. Nat Med 2024; 30:1471-1480. [PMID: 38740996 PMCID: PMC11108784 DOI: 10.1038/s41591-024-02971-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Accepted: 04/03/2024] [Indexed: 05/16/2024]
Abstract
Cardiac magnetic resonance imaging (CMR) is the gold standard for cardiac function assessment and plays a crucial role in diagnosing cardiovascular disease (CVD). However, its widespread application has been limited by the heavy resource burden of CMR interpretation. Here, to address this challenge, we developed and validated computerized CMR interpretation for screening and diagnosis of 11 types of CVD in 9,719 patients. We propose a two-stage paradigm consisting of noninvasive cine-based CVD screening followed by cine and late gadolinium enhancement-based diagnosis. The screening and diagnostic models achieved high performance (area under the curve of 0.988 ± 0.3% and 0.991 ± 0.0%, respectively) in both internal and external datasets. Furthermore, the diagnostic model outperformed cardiologists in diagnosing pulmonary arterial hypertension, demonstrating the ability of artificial intelligence-enabled CMR to detect previously unidentified CMR features. This proof-of-concept study holds the potential to substantially advance the efficiency and scalability of CMR interpretation, thereby improving CVD screening and diagnosis.
Collapse
Affiliation(s)
| | - Kai Yang
- Department of Magnetic Resonance Imaging, Fuwai Hospital and National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yi Wen
- Changhong AI Research (CHAIR), Sichuan Changhong Electronics Holding Group, Mianyang, China
| | - Pengcheng Wang
- Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, USA
| | - Yuepeng Hu
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
| | - Yongfan Lai
- School of Engineering, University of Science and Technology of China, Hefei, China
| | - Yufeng Wang
- Department of Computer Science, Stony Brook University, New York, NY, USA
| | - Kankan Zhao
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
| | - Siyi Tang
- School of Medicine, Stanford University, Stanford, CA, USA
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Angela Zhang
- School of Medicine, Stanford University, Stanford, CA, USA
- Stanford Cardiovascular Institute, School of Medicine (Division of Cardiology), Stanford University, Stanford, CA, USA
| | - Huayi Zhan
- Changhong AI Research (CHAIR), Sichuan Changhong Electronics Holding Group, Mianyang, China
| | - Minjie Lu
- Department of Magnetic Resonance Imaging, Fuwai Hospital and National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Xiuyu Chen
- Department of Magnetic Resonance Imaging, Fuwai Hospital and National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Shujuan Yang
- Department of Magnetic Resonance Imaging, Fuwai Hospital and National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Zhixiang Dong
- Department of Magnetic Resonance Imaging, Fuwai Hospital and National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yining Wang
- Peking Union Medical College Hospital, Beijing, China
| | - Hui Liu
- Guangdong Provincial People's Hospital, Guangzhou, China
| | - Lei Zhao
- Beijing Anzhen Hospital, Beijing, China
| | | | - Yunling Li
- The Second Affiliated Hospital of Harbin Medical University, Harbin, China
| | | | - Zixian Chen
- The First Hospital of Lanzhou University, Lanzhou, China
| | - Yi Luo
- The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
| | - Dongbo Liu
- Changhong AI Research (CHAIR), Sichuan Changhong Electronics Holding Group, Mianyang, China
| | - Pengbo Zhao
- Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA
| | - Keldon Lin
- Mayo Clinic Alix School of Medicine, Phoenix, AZ, USA
| | - Joseph C Wu
- School of Medicine, Stanford University, Stanford, CA, USA
- Stanford Cardiovascular Institute, School of Medicine (Division of Cardiology), Stanford University, Stanford, CA, USA
| | - Shihua Zhao
- Department of Magnetic Resonance Imaging, Fuwai Hospital and National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.
| |
Collapse
|
150
|
Yan T, Tang G, Zhang H, Liang L, Ma J, Gao Y, Zhou C, Li S. Multiscale and multiperception feature learning for pancreatic lesion detection based on noncontrast CT. Phys Med Biol 2024; 69:105014. [PMID: 38588676 DOI: 10.1088/1361-6560/ad3c0c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2023] [Accepted: 04/08/2024] [Indexed: 04/10/2024]
Abstract
Background. Pancreatic cancer is one of the most malignant tumours, demonstrating a poor prognosis and nearly identically high mortality and morbidity, mainly because of the difficulty of early diagnosis and timely treatment for localized stages.Objective. To develop a noncontrast CT (NCCT)-based pancreatic lesion detection model that could serve as an intelligent tool for diagnosing pancreatic cancer early, overcoming the challenges associated with low contrast intensities and complex anatomical structures present in NCCT images.Approach.We design a multiscale and multiperception (MSMP) feature learning network with ResNet50 coupled with a feature pyramid network as the backbone for strengthening feature expressions. We added multiscale atrous convolutions to expand different receptive fields, contextual attention to perceive contextual information, and channel and spatial attention to focus on important channels and spatial regions, respectively. The MSMP network then acts as a feature extractor for proposing an NCCT-based pancreatic lesion detection model with image patches covering the pancreas as its input; Faster R-CNN is employed as the detection method for accurately detecting pancreatic lesions.Main results. By using the new MSMP network as a feature extractor, our model outperforms the conventional object detection algorithms in terms of the recall (75.40% and 90.95%), precision (40.84% and 68.21%), F1 score (52.98% and 77.96%), F2 score (64.48% and 85.26%) and Ap50 metrics (53.53% and 70.14%) at the image and patient levels, respectively.Significance.The good performance of our new model implies that MSMP can mine NCCT imaging features for detecting pancreatic lesions from complex backgrounds well. The proposed detection model is expected to be further developed as an intelligent method for the early detection of pancreatic cancer.
Collapse
Affiliation(s)
- Tian Yan
- School of Biomedical Engineering, Southern Medical University, Guangzhou, People's Republic of China
- Guangdong Provincial Key Laboratory of Medical Image Processing, Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, People's Republic of China
- General Surgery Center, Department of Hepatobiliary Surgery II, Guangdong Provincial Research Center for Artificial Organ and Tissue Engineering, Guangzhou Clinical Research and Transformation Center for Artificial Liver, Institute of Regenerative Medicine, Zhujiang Hospital, Southern Medical University, Guangzhou, People's Republic of China
| | - Geye Tang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, People's Republic of China
- Guangdong Provincial Key Laboratory of Medical Image Processing, Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, People's Republic of China
| | - Haojie Zhang
- Department of General Surgery, Seventh People's Hospital of Nanhai District, Foshan, People's Republic of China
| | - Lidu Liang
- Equipment and Materials Department, Xinchang Hospital of Traditional Chinese Medicine, Zhejiang, People's Republic of China
| | - Jianhua Ma
- School of Biomedical Engineering, Southern Medical University, Guangzhou, People's Republic of China
- Guangdong Provincial Key Laboratory of Medical Image Processing, Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, People's Republic of China
| | - Yi Gao
- General Surgery Center, Department of Hepatobiliary Surgery II, Guangdong Provincial Research Center for Artificial Organ and Tissue Engineering, Guangzhou Clinical Research and Transformation Center for Artificial Liver, Institute of Regenerative Medicine, Zhujiang Hospital, Southern Medical University, Guangzhou, People's Republic of China
- State Key Laboratory of Organ Failure Research, Southern Medical University, Guangzhou, People's Republic of China
| | - Chenjie Zhou
- Department of Pancreatic Surgery, Fudan University Shanghai Cancer Center, Shanghai, People's Republic of China
| | - Shulong Li
- School of Biomedical Engineering, Southern Medical University, Guangzhou, People's Republic of China
- Guangdong Provincial Key Laboratory of Medical Image Processing, Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, People's Republic of China
| |
Collapse
|