1
|
Queisner M, Eisenträger K. Surgical planning in virtual reality: a systematic review. J Med Imaging (Bellingham) 2024; 11:062603. [PMID: 38680654 PMCID: PMC11043584 DOI: 10.1117/1.jmi.11.6.062603] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Revised: 03/05/2024] [Accepted: 03/18/2024] [Indexed: 05/01/2024] Open
Abstract
Purpose Virtual reality (VR) technology has emerged as a promising tool for physicians, offering the ability to assess anatomical data in 3D with visuospatial interaction qualities. The last decade has witnessed a remarkable increase in the number of studies focusing on the application of VR to assess patient-specific image data. This systematic review aims to provide an up-to-date overview of the latest research on VR in the field of surgical planning. Approach A comprehensive literature search was conducted based on the preferred reporting items for systematic reviews and meta-analyses covering the period from April 1, 2021 to May 10, 2023. It includes research articles reporting on preoperative surgical planning using patient-specific medical images in virtual reality using head-mounted displays. The review summarizes the current state of research in this field, identifying key findings, technologies, study designs, methods, and potential directions for future research. Results The selected studies show a positive impact on surgical decision-making and anatomy understanding compared to other visualization modalities. A substantial number of studies are reporting anecdotal evidence and case-specific outcomes. Notably, surgical planning using VR led to more frequent changes in surgical plans compared to planning with other visualization methods when surgeons reassessed their initial plans. VR demonstrated benefits in reducing planning time and improving spatial localization of pathologies. Conclusions Results show that the application of VR for surgical planning is still in an experimental stage but is gradually advancing toward clinical use. The diverse study designs, methodologies, and varying reporting hinder a comprehensive analysis. Some findings lack statistical evidence and rely on subjective assumptions. To strengthen evaluation, future research should focus on refining study designs, improving technical reporting, defining visual and technical proficiency requirements, and enhancing VR software usability and design. Addressing these areas could pave the way for an effective implementation of VR in clinical settings.
Collapse
Affiliation(s)
- Moritz Queisner
- Charité – Universitätsmedizin Berlin, Department of Surgery, Experimental Surgery, Berlin, Germany
- Humboldt Universität zu Berlin, Cluster of Excellence Matters of Activity, Berlin, Germany
| | - Karl Eisenträger
- Charité – Universitätsmedizin Berlin, Department of Surgery, Experimental Surgery, Berlin, Germany
| |
Collapse
|
2
|
Cho MJ, Lee JS. Semi-supervised abdominal multi-organ segmentation by object-redrawing. Med Phys 2024. [PMID: 39167059 DOI: 10.1002/mp.17364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2024] [Accepted: 08/04/2024] [Indexed: 08/23/2024] Open
Abstract
BACKGROUND Multi-organ segmentation is a critical task in medical imaging, with wide-ranging applications in both clinical practice and research. Accurate delineation of organs from high-resolution 3D medical images, such as CT scans, is essential for radiation therapy planning, enhancing treatment outcomes, and minimizing radiation toxicity risks. Additionally, it plays a pivotal role in quantitative image analysis, supporting various medical research studies. Despite its significance, manual segmentation of multiple organs from 3D images is labor-intensive and prone to low reproducibility due to high interoperator variability. Recent advancements in deep learning have led to several automated segmentation methods, yet many rely heavily on labeled data and human anatomy expertise. PURPOSE In this study, our primary objective is to address the limitations of existing semi-supervised learning (SSL) methods for abdominal multi-organ segmentation. We aim to introduce a novel SSL approach that leverages unlabeled data to enhance the performance of deep neural networks in segmenting abdominal organs. Specifically, we propose a method that incorporates a redrawing network into the segmentation process to correct errors and improve accuracy. METHODS Our proposed method comprises three interconnected neural networks: a segmentation network for image segmentation, a teacher network for consistency regularization, and a redrawing network for object redrawing. During training, the segmentation network undergoes two rounds of optimization: basic training and readjustment. We adopt the Mean-Teacher model as our baseline SSL approach, utilizing labeled and unlabeled data. However, recognizing significant errors in abdominal multi-organ segmentation using this method alone, we introduce the redrawing network to generate redrawn images based on CT scans, preserving original anatomical information. Our approach is grounded in the generative process hypothesis, encompassing segmentation, drawing, and assembling stages. Correct segmentation is crucial for generating accurate images. In the basic training phase, the segmentation network is trained using both labeled and unlabeled data, incorporating consistency learning to ensure consistent predictions before and after perturbations. The readjustment phase focuses on reducing segmentation errors by optimizing the segmentation network parameters based on the differences between redrawn and original CT images. RESULTS We evaluated our method using two publicly available datasets: the beyond the cranial vault (BTCV) segmentation dataset (training: 44, validation: 6) and the abdominal multi-organ segmentation (AMOS) challenge 2022 dataset (training:138, validation:16). Our results were compared with state-of-the-art SSL methods, including MT and dual-task consistency (DTC), using the Dice similarity coefficient (DSC) as an accuracy metric. On both datasets, our proposed SSL method consistently outperformed other methods, including supervised learning, achieving superior segmentation performance for various abdominal organs. These findings demonstrate the effectiveness of our approach, even with a limited number of labeled data. CONCLUSIONS Our novel semi-supervised learning approach for abdominal multi-organ segmentation addresses the challenges associated with this task. By integrating a redrawing network and leveraging unlabeled data, we achieve remarkable improvements in accuracy. Our method demonstrates superior performance compared to existing SSL and supervised learning methods. This approach holds great promise in enhancing the precision and efficiency of multi-organ segmentation in medical imaging applications.
Collapse
Affiliation(s)
- Min Jeong Cho
- Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, South Korea
- Artificial Intelligence Institute, Seoul National University, Seoul, South Korea
- Integrated Major in Innovative Medical Science, Seoul National University College of Medicine, Seoul, South Korea
| | - Jae Sung Lee
- Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, South Korea
- Artificial Intelligence Institute, Seoul National University, Seoul, South Korea
- Integrated Major in Innovative Medical Science, Seoul National University College of Medicine, Seoul, South Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, Seoul, South Korea
- Brightonix Imaging Inc., Seoul, South Korea
| |
Collapse
|
3
|
Barekatrezaei S, Kozegar E, Salamati M, Soryani M. Mass detection in automated three dimensional breast ultrasound using cascaded convolutional neural networks. Phys Med 2024; 124:103433. [PMID: 39002423 DOI: 10.1016/j.ejmp.2024.103433] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 07/03/2024] [Accepted: 07/08/2024] [Indexed: 07/15/2024] Open
Abstract
PURPOSE Early detection of breast cancer has a significant effect on reducing its mortality rate. For this purpose, automated three-dimensional breast ultrasound (3-D ABUS) has been recently used alongside mammography. The 3-D volume produced in this imaging system includes many slices. The radiologist must review all the slices to find the mass, a time-consuming task with a high probability of mistakes. Therefore, many computer-aided detection (CADe) systems have been developed to assist radiologists in this task. In this paper, we propose a novel CADe system for mass detection in 3-D ABUS images. METHODS The proposed system includes two cascaded convolutional neural networks. The goal of the first network is to achieve the highest possible sensitivity, and the second network's goal is to reduce false positives while maintaining high sensitivity. In both networks, an improved version of 3-D U-Net architecture is utilized in which two types of modified Inception modules are used in the encoder section. In the second network, new attention units are also added to the skip connections that receive the results of the first network as saliency maps. RESULTS The system was evaluated on a dataset containing 60 3-D ABUS volumes from 43 patients and 55 masses. A sensitivity of 91.48% and a mean false positive of 8.85 per patient were achieved. CONCLUSIONS The suggested mass detection system is fully automatic without any user interaction. The results indicate that the sensitivity and the mean FP per patient of the CADe system outperform competing techniques.
Collapse
Affiliation(s)
- Sepideh Barekatrezaei
- School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran.
| | - Ehsan Kozegar
- Department of Computer Engineering and Engineering Sciences, Faculty of Technology and Engineering, University of Guilan, Rudsar-Vajargah, Guilan, Iran.
| | - Masoumeh Salamati
- Department of Reproductive Imaging, Reproductive Biomedicine Research Center, Royan Institute for Reproductive Biomedicine, ACECR, Tehran, Iran.
| | - Mohsen Soryani
- School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran.
| |
Collapse
|
4
|
Saeed SU, Ramalhinho J, Pinnock M, Shen Z, Fu Y, Montaña-Brown N, Bonmati E, Barratt DC, Pereira SP, Davidson B, Clarkson MJ, Hu Y. Active learning using adaptable task-based prioritisation. Med Image Anal 2024; 95:103181. [PMID: 38640779 DOI: 10.1016/j.media.2024.103181] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 04/03/2024] [Accepted: 04/12/2024] [Indexed: 04/21/2024]
Abstract
Supervised machine learning-based medical image computing applications necessitate expert label curation, while unlabelled image data might be relatively abundant. Active learning methods aim to prioritise a subset of available image data for expert annotation, for label-efficient model training. We develop a controller neural network that measures priority of images in a sequence of batches, as in batch-mode active learning, for multi-class segmentation tasks. The controller is optimised by rewarding positive task-specific performance gain, within a Markov decision process (MDP) environment that also optimises the task predictor. In this work, the task predictor is a segmentation network. A meta-reinforcement learning algorithm is proposed with multiple MDPs, such that the pre-trained controller can be adapted to a new MDP that contains data from different institutes and/or requires segmentation of different organs or structures within the abdomen. We present experimental results using multiple CT datasets from more than one thousand patients, with segmentation tasks of nine different abdominal organs, to demonstrate the efficacy of the learnt prioritisation controller function and its cross-institute and cross-organ adaptability. We show that the proposed adaptable prioritisation metric yields converging segmentation accuracy for a new kidney segmentation task, unseen in training, using between approximately 40% to 60% of labels otherwise required with other heuristic or random prioritisation metrics. For clinical datasets of limited size, the proposed adaptable prioritisation offers a performance improvement of 22.6% and 10.2% in Dice score, for tasks of kidney and liver vessel segmentation, respectively, compared to random prioritisation and alternative active sampling strategies.
Collapse
Affiliation(s)
- Shaheer U Saeed
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK.
| | - João Ramalhinho
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Mark Pinnock
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Ziyi Shen
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Yunguan Fu
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; InstaDeep, London, UK
| | - Nina Montaña-Brown
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Ester Bonmati
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; School of Computer Science and Engineering, University of Westminster, London, UK
| | - Dean C Barratt
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Stephen P Pereira
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; Institute for Liver and Digestive Health, University College London, London, UK
| | - Brian Davidson
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; Centre for Surgical Innovation, Organ Regeneration and Transplantation (CISORT), Division of Surgery & Interventional Science, University College London, London, UK
| | - Matthew J Clarkson
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Yipeng Hu
- Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional & Surgical Sciences, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| |
Collapse
|
5
|
Alajaji SA, Amarin R, Masri R, Tavares T, Kumar V, Price JB, Sultan AS. Detection of extracranial and intracranial calcified carotid artery atheromas in cone beam computed tomography using a deep learning convolutional neural network image segmentation approach. Oral Surg Oral Med Oral Pathol Oral Radiol 2024; 138:162-172. [PMID: 37770329 DOI: 10.1016/j.oooo.2023.08.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Revised: 07/16/2023] [Accepted: 08/04/2023] [Indexed: 09/30/2023]
Abstract
OBJECTIVE We leveraged an artificial intelligence deep-learning convolutional neural network (DL CNN) to detect calcified carotid artery atheromas (CCAAs) on cone beam computed tomography (CBCT) images. STUDY DESIGN We obtained 137 full-volume CBCT scans with previously diagnosed CCAAs. The DL model was trained on 170 single axial CBCT slices, 90 with extracranial CCAAs and 80 with intracranial CCAAs. A board-certified oral and maxillofacial radiologist confirmed the presence of each CCAA. Transfer learning through a U-Net-based CNN architecture was utilized. Data allocation was 60% training, 10% validation, and 30% testing. We determined the accuracy of the DL model in detecting CCAA by calculating the mean training and validation accuracy and the area under the receiver operating characteristic curve (AUC). We reserved 5 randomly selected unseen full CBCT volumes for final testing. RESULTS The mean training and validation accuracy of the model in detecting extracranial CCAAs was 92% and 82%, respectively, and the AUC was 0.84 with 1.0 sensitivity and 0.69 specificity. The mean training and validation accuracy in detecting intracranial CCAAs was 61% and 70%, respectively, and the AUC was 0.5 with 0.93 sensitivity and 0.08 specificity. Testing of full-volume scans yielded an AUC of 0.72 and 0.55 for extracranial and intracranial CCAAs, respectively. CONCLUSION Our DL model showed excellent discrimination in detecting extracranial CCAAs on axial CBCT images and acceptable discrimination on full-volumes but poor discrimination in detecting intracranial CCAAs, for which further research is required.
Collapse
Affiliation(s)
- Shahd A Alajaji
- Department of Oncology and Diagnostic Sciences, School of Dentistry, University of Maryland Baltimore, MD, USA; Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA; Department of Oral Medicine and Diagnostic Sciences, College of Dentistry, King Saud University, Riyadh, Saudi Arabia
| | - Rula Amarin
- Department of Advanced Oral Sciences and Therapeutics, School of Dentistry, University of Maryland, Baltimore, MD, USA
| | - Radi Masri
- Department of Advanced Oral Sciences and Therapeutics, School of Dentistry, University of Maryland, Baltimore, MD, USA
| | - Tiffany Tavares
- Department of Comprehensive Dentistry, UT Health San Antonio, School of Dentistry, San Antonio, TX, USA
| | - Vandana Kumar
- Department of Oncology and Diagnostic Sciences, School of Dentistry, University of Maryland Baltimore, MD, USA
| | - Jeffery B Price
- Department of Oncology and Diagnostic Sciences, School of Dentistry, University of Maryland Baltimore, MD, USA; Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA
| | - Ahmed S Sultan
- Department of Oncology and Diagnostic Sciences, School of Dentistry, University of Maryland Baltimore, MD, USA; Division of Artificial Intelligence Research, University of Maryland School of Dentistry, Baltimore, MD, USA; Marlene and Stewart Greenebaum Comprehensive Cancer Center, University of Maryland, Baltimore, MD, USA.
| |
Collapse
|
6
|
Bendazzoli S, Bäcklin E, Smedby Ö, Janerot-Sjoberg B, Connolly B, Wang C. Lung vessel connectivity map as anatomical prior knowledge for deep learning-based lung lobe segmentation. J Med Imaging (Bellingham) 2024; 11:044001. [PMID: 38988990 PMCID: PMC11231955 DOI: 10.1117/1.jmi.11.4.044001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Revised: 05/31/2024] [Accepted: 06/05/2024] [Indexed: 07/12/2024] Open
Abstract
Purpose Our study investigates the potential benefits of incorporating prior anatomical knowledge into a deep learning (DL) method designed for the automated segmentation of lung lobes in chest CT scans. Approach We introduce an automated DL-based approach that leverages anatomical information from the lung's vascular system to guide and enhance the segmentation process. This involves utilizing a lung vessel connectivity (LVC) map, which encodes relevant lung vessel anatomical data. Our study explores the performance of three different neural network architectures within the nnU-Net framework: a standalone U-Net, a multitasking U-Net, and a cascade U-Net. Results Experimental findings suggest that the inclusion of LVC information in the DL model can lead to improved segmentation accuracy, particularly, in the challenging boundary regions of expiration chest CT volumes. Furthermore, our study demonstrates the potential for LVC to enhance the model's generalization capabilities. Finally, the method's robustness is evaluated through the segmentation of lung lobes in 10 cases of COVID-19, demonstrating its applicability in the presence of pulmonary diseases. Conclusions Incorporating prior anatomical information, such as LVC, into the DL model shows promise for enhancing segmentation performance, particularly in the boundary regions. However, the extent of this improvement has limitations, prompting further exploration of its practical applicability.
Collapse
Affiliation(s)
- Simone Bendazzoli
- KTH Royal Institute of Technology, Department of Biomedical Engineering and Health Systems, Huddinge, Sweden
- Karolinska Institutet, Department of Clinical Science, Intervention and Technology, Solna, Sweden
| | - Emelie Bäcklin
- KTH Royal Institute of Technology, Department of Biomedical Engineering and Health Systems, Huddinge, Sweden
- Karolinska Institutet, Department of Clinical Science, Intervention and Technology, Solna, Sweden
| | - Örjan Smedby
- KTH Royal Institute of Technology, Department of Biomedical Engineering and Health Systems, Huddinge, Sweden
| | - Birgitta Janerot-Sjoberg
- Karolinska Institutet, Department of Clinical Science, Intervention and Technology, Solna, Sweden
| | - Bryan Connolly
- Karolinska Institutet, Department of Radiology, Solna, Sweden
| | - Chunliang Wang
- KTH Royal Institute of Technology, Department of Biomedical Engineering and Health Systems, Huddinge, Sweden
| |
Collapse
|
7
|
Bannone E, Collins T, Esposito A, Cinelli L, De Pastena M, Pessaux P, Felli E, Andreotti E, Okamoto N, Barberio M, Felli E, Montorsi RM, Ingaglio N, Rodríguez-Luna MR, Nkusi R, Marescaux J, Hostettler A, Salvia R, Diana M. Surgical optomics: hyperspectral imaging and deep learning towards precision intraoperative automatic tissue recognition-results from the EX-MACHYNA trial. Surg Endosc 2024; 38:3758-3772. [PMID: 38789623 DOI: 10.1007/s00464-024-10880-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2024] [Accepted: 04/23/2024] [Indexed: 05/26/2024]
Abstract
BACKGROUND Hyperspectral imaging (HSI), combined with machine learning, can help to identify characteristic tissue signatures enabling automatic tissue recognition during surgery. This study aims to develop the first HSI-based automatic abdominal tissue recognition with human data in a prospective bi-center setting. METHODS Data were collected from patients undergoing elective open abdominal surgery at two international tertiary referral hospitals from September 2020 to June 2021. HS images were captured at various time points throughout the surgical procedure. Resulting RGB images were annotated with 13 distinct organ labels. Convolutional Neural Networks (CNNs) were employed for the analysis, with both external and internal validation settings utilized. RESULTS A total of 169 patients were included, 73 (43.2%) from Strasbourg and 96 (56.8%) from Verona. The internal validation within centers combined patients from both centers into a single cohort, randomly allocated to the training (127 patients, 75.1%, 585 images) and test sets (42 patients, 24.9%, 181 images). This validation setting showed the best performance. The highest true positive rate was achieved for the skin (100%) and the liver (97%). Misclassifications included tissues with a similar embryological origin (omentum and mesentery: 32%) or with overlaying boundaries (liver and hepatic ligament: 22%). The median DICE score for ten tissue classes exceeded 80%. CONCLUSION To improve automatic surgical scene segmentation and to drive clinical translation, multicenter accurate HSI datasets are essential, but further work is needed to quantify the clinical value of HSI. HSI might be included in a new omics science, namely surgical optomics, which uses light to extract quantifiable tissue features during surgery.
Collapse
Affiliation(s)
- Elisa Bannone
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France.
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy.
| | - Toby Collins
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
| | - Alessandro Esposito
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy
| | - Lorenzo Cinelli
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- Department of Gastrointestinal Surgery, San Raffaele Hospital IRCCS, Milan, Italy
| | - Matteo De Pastena
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy
| | - Patrick Pessaux
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- Department of General, Digestive, and Endocrine Surgery, University Hospital of Strasbourg, Strasbourg, France
- Institut of Viral and Liver Disease, Inserm U1110, University of Strasbourg, Strasbourg, France
| | - Emanuele Felli
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- Department of General, Digestive, and Endocrine Surgery, University Hospital of Strasbourg, Strasbourg, France
- Institut of Viral and Liver Disease, Inserm U1110, University of Strasbourg, Strasbourg, France
| | - Elena Andreotti
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy
| | - Nariaki Okamoto
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- Photonics Instrumentation for Health, iCube Laboratory, University of Strasbourg, Strasbourg, France
| | - Manuel Barberio
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- General Surgery Department, Ospedale Cardinale G. Panico, Tricase, Italy
| | - Eric Felli
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- Department of Visceral Surgery and Medicine, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Roberto Maria Montorsi
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy
| | - Naomi Ingaglio
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy
| | - María Rita Rodríguez-Luna
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
- Photonics Instrumentation for Health, iCube Laboratory, University of Strasbourg, Strasbourg, France
| | - Richard Nkusi
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
| | - Jacque Marescaux
- Research Institute Against Digestive Cancer (IRCAD), 67000, Strasbourg, France
| | | | - Roberto Salvia
- Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, P.Le Scuro 10, 37134, Verona, Italy
| | - Michele Diana
- Photonics Instrumentation for Health, iCube Laboratory, University of Strasbourg, Strasbourg, France
- Department of Surgery, University Hospital of Geneva, Geneva, Switzerland
| |
Collapse
|
8
|
Al Hasan MM, Ghazimoghadam S, Tunlayadechanont P, Mostafiz MT, Gupta M, Roy A, Peters K, Hochhegger B, Mancuso A, Asadizanjani N, Forghani R. Automated Segmentation of Lymph Nodes on Neck CT Scans Using Deep Learning. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01114-w. [PMID: 38937342 DOI: 10.1007/s10278-024-01114-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/16/2023] [Revised: 04/01/2024] [Accepted: 04/03/2024] [Indexed: 06/29/2024]
Abstract
Early and accurate detection of cervical lymph nodes is essential for the optimal management and staging of patients with head and neck malignancies. Pilot studies have demonstrated the potential for radiomic and artificial intelligence (AI) approaches in increasing diagnostic accuracy for the detection and classification of lymph nodes, but implementation of many of these approaches in real-world clinical settings would necessitate an automated lymph node segmentation pipeline as a first step. In this study, we aim to develop a non-invasive deep learning (DL) algorithm for detecting and automatically segmenting cervical lymph nodes in 25,119 CT slices from 221 normal neck contrast-enhanced CT scans from patients without head and neck cancer. We focused on the most challenging task of segmentation of small lymph nodes, evaluated multiple architectures, and employed U-Net and our adapted spatial context network to detect and segment small lymph nodes measuring 5-10 mm. The developed algorithm achieved a Dice score of 0.8084, indicating its effectiveness in detecting and segmenting cervical lymph nodes despite their small size. A segmentation framework successful in this task could represent an essential initial block for future algorithms aiming to evaluate small objects such as lymph nodes in different body parts, including small lymph nodes looking normal to the naked human eye but harboring early nodal metastases.
Collapse
Affiliation(s)
- Md Mahfuz Al Hasan
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Electrical and Computer Engineering, University of Florida College of Medicine, Gainesville, FL, USA
| | - Saba Ghazimoghadam
- Augmented Intelligence and Precision Health Laboratory, Research Institute of the McGill University Health Centre, Montreal, QC, Canada
| | - Padcha Tunlayadechanont
- Augmented Intelligence and Precision Health Laboratory, Research Institute of the McGill University Health Centre, Montreal, QC, Canada
- Department of Diagnostic and Therapeutic Radiology and Research, Faculty of Medicine Ramathibodi Hospital, Ratchathewi, Bangkok, Thailand
| | - Mohammed Tahsin Mostafiz
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Electrical and Computer Engineering, University of Florida College of Medicine, Gainesville, FL, USA
| | - Manas Gupta
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
| | - Antika Roy
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Electrical and Computer Engineering, University of Florida College of Medicine, Gainesville, FL, USA
| | - Keith Peters
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Radiology, University of Florida College of Medicine, Gainesville, FL, USA
| | - Bruno Hochhegger
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Radiology, University of Florida College of Medicine, Gainesville, FL, USA
| | - Anthony Mancuso
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Radiology, University of Florida College of Medicine, Gainesville, FL, USA
| | - Navid Asadizanjani
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA
- Department of Electrical and Computer Engineering, University of Florida College of Medicine, Gainesville, FL, USA
| | - Reza Forghani
- Radiomics and Augmented Intelligence Laboratory (RAIL), Department of Radiology and the Norman Fixel Institute for Neurological Diseases, University of Florida College of Medicine, 1600 SW Archer Road, Gainesville, FL, 32610-0374, USA.
- Department of Radiology, University of Florida College of Medicine, Gainesville, FL, USA.
- Division of Medical Physics, University of Florida College of Medicine, Gainesville, FL, USA.
- Department of Neurology, Division of Movement Disorders, University of Florida College of Medicine, Gainesville, FL, USA.
- Augmented Intelligence and Precision Health Laboratory, Research Institute of the McGill University Health Centre, Montreal, QC, Canada.
| |
Collapse
|
9
|
Yang M, Yang M, Yang L, Wang Z, Ye P, Chen C, Fu L, Xu S. Deep learning for MRI lesion segmentation in rectal cancer. Front Med (Lausanne) 2024; 11:1394262. [PMID: 38983364 PMCID: PMC11231084 DOI: 10.3389/fmed.2024.1394262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Accepted: 06/14/2024] [Indexed: 07/11/2024] Open
Abstract
Rectal cancer (RC) is a globally prevalent malignant tumor, presenting significant challenges in its management and treatment. Currently, magnetic resonance imaging (MRI) offers superior soft tissue contrast and radiation-free effects for RC patients, making it the most widely used and effective detection method. In early screening, radiologists rely on patients' medical radiology characteristics and their extensive clinical experience for diagnosis. However, diagnostic accuracy may be hindered by factors such as limited expertise, visual fatigue, and image clarity issues, resulting in misdiagnosis or missed diagnosis. Moreover, the distribution of surrounding organs in RC is extensive with some organs having similar shapes to the tumor but unclear boundaries; these complexities greatly impede doctors' ability to diagnose RC accurately. With recent advancements in artificial intelligence, machine learning techniques like deep learning (DL) have demonstrated immense potential and broad prospects in medical image analysis. The emergence of this approach has significantly enhanced research capabilities in medical image classification, detection, and segmentation fields with particular emphasis on medical image segmentation. This review aims to discuss the developmental process of DL segmentation algorithms along with their application progress in lesion segmentation from MRI images of RC to provide theoretical guidance and support for further advancements in this field.
Collapse
Affiliation(s)
- Mingwei Yang
- Department of General Surgery, Nanfang Hospital Zengcheng Campus, Guangzhou, Guangdong, China
| | - Miyang Yang
- Department of Radiology, Fuzong Teaching Hospital, Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
- Department of Radiology, 900th Hospital of Joint Logistics Support Force, Fuzhou, Fujian, China
| | - Lanlan Yang
- Department of Radiology, Fuzong Teaching Hospital, Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
| | - Zhaochu Wang
- Department of Radiology, Fuzong Teaching Hospital, Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
| | - Peiyun Ye
- Department of Radiology, Fuzong Teaching Hospital, Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
- Department of Radiology, 900th Hospital of Joint Logistics Support Force, Fuzhou, Fujian, China
| | - Chujie Chen
- Department of Radiology, Fuzong Teaching Hospital, Fujian University of Traditional Chinese Medicine, Fuzhou, Fujian, China
- Department of Radiology, 900th Hospital of Joint Logistics Support Force, Fuzhou, Fujian, China
| | - Liyuan Fu
- Department of Radiology, 900th Hospital of Joint Logistics Support Force, Fuzhou, Fujian, China
| | - Shangwen Xu
- Department of Radiology, 900th Hospital of Joint Logistics Support Force, Fuzhou, Fujian, China
| |
Collapse
|
10
|
Hassan R, Mondal MRH, Ahamed SI. UDBRNet: A novel uncertainty driven boundary refined network for organ at risk segmentation. PLoS One 2024; 19:e0304771. [PMID: 38885241 PMCID: PMC11182520 DOI: 10.1371/journal.pone.0304771] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Accepted: 05/19/2024] [Indexed: 06/20/2024] Open
Abstract
Organ segmentation has become a preliminary task for computer-aided intervention, diagnosis, radiation therapy, and critical robotic surgery. Automatic organ segmentation from medical images is a challenging task due to the inconsistent shape and size of different organs. Besides this, low contrast at the edges of organs due to similar types of tissue confuses the network's ability to segment the contour of organs properly. In this paper, we propose a novel convolution neural network based uncertainty-driven boundary-refined segmentation network (UDBRNet) that segments the organs from CT images. The CT images are segmented first and produce multiple segmentation masks from multi-line segmentation decoder. Uncertain regions are identified from multiple masks and the boundaries of the organs are refined based on uncertainty data. Our method achieves remarkable performance, boasting dice accuracies of 0.80, 0.95, 0.92, and 0.94 for Esophagus, Heart, Trachea, and Aorta respectively on the SegThor dataset, and 0.71, 0.89, 0.85, 0.97, and 0.97 for Esophagus, Spinal Cord, Heart, Left-Lung, and Right-Lung respectively on the LCTSC dataset. These results demonstrate the superiority of our uncertainty-driven boundary refinement technique over state-of-the-art segmentation networks such as UNet, Attention UNet, FC-denseNet, BASNet, UNet++, R2UNet, TransUNet, and DS-TransUNet. UDBRNet presents a promising network for more precise organ segmentation, particularly in challenging, uncertain conditions. The source code of our proposed method will be available at https://github.com/riadhassan/UDBRNet.
Collapse
Affiliation(s)
- Riad Hassan
- Institute of Information and Communication Technology, Bangladesh University of Engineering and Technology, Palashi, Dhaka, Bangladesh
| | - M. Rubaiyat Hossain Mondal
- Institute of Information and Communication Technology, Bangladesh University of Engineering and Technology, Palashi, Dhaka, Bangladesh
| | - Sheikh Iqbal Ahamed
- Department of Computer Science, Marquette University, Milwaukee, Wisconsin, United States of America
| |
Collapse
|
11
|
Salimi Y, Mansouri Z, Hajianfar G, Sanaat A, Shiri I, Zaidi H. Fully automated explainable abdominal CT contrast media phase classification using organ segmentation and machine learning. Med Phys 2024; 51:4095-4104. [PMID: 38629779 DOI: 10.1002/mp.17076] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 03/19/2024] [Accepted: 04/02/2024] [Indexed: 06/05/2024] Open
Abstract
BACKGROUND Contrast-enhanced computed tomography (CECT) provides much more information compared to non-enhanced CT images, especially for the differentiation of malignancies, such as liver carcinomas. Contrast media injection phase information is usually missing on public datasets and not standardized in the clinic even in the same region and language. This is a barrier to effective use of available CECT images in clinical research. PURPOSE The aim of this study is to detect contrast media injection phase from CT images by means of organ segmentation and machine learning algorithms. METHODS A total number of 2509 CT images split into four subsets of non-contrast (class #0), arterial (class #1), venous (class #2), and delayed (class #3) after contrast media injection were collected from two CT scanners. Seven organs including the liver, spleen, heart, kidneys, lungs, urinary bladder, and aorta along with body contour masks were generated by pre-trained deep learning algorithms. Subsequently, five first-order statistical features including average, standard deviation, 10, 50, and 90 percentiles extracted from the above-mentioned masks were fed to machine learning models after feature selection and reduction to classify the CT images in one of four above mentioned classes. A 10-fold data split strategy was followed. The performance of our methodology was evaluated in terms of classification accuracy metrics. RESULTS The best performance was achieved by Boruta feature selection and RF model with average area under the curve of more than 0.999 and accuracy of 0.9936 averaged over four classes and 10 folds. Boruta feature selection selected all predictor features. The lowest classification was observed for class #2 (0.9888), which is already an excellent result. In the 10-fold strategy, only 33 cases from 2509 cases (∼1.4%) were misclassified. The performance over all folds was consistent. CONCLUSIONS We developed a fast, accurate, reliable, and explainable methodology to classify contrast media phases which may be useful in data curation and annotation in big online datasets or local datasets with non-standard or no series description. Our model containing two steps of deep learning and machine learning may help to exploit available datasets more effectively.
Collapse
Affiliation(s)
- Yazdan Salimi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Zahra Mansouri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Ghasem Hajianfar
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Amirhossein Sanaat
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Isaac Shiri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
- Department of Cardiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
- Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen, Groningen, Netherlands
- Department of Nuclear Medicine, University of Southern Denmark, Odense, Denmark
- University Research and Innovation Center, Óbuda University, Budapest, Hungary
| |
Collapse
|
12
|
Ferreira Silvério N, van den Wollenberg W, Betgen A, Wiersema L, Marijnen C, Peters F, van der Heide UA, Simões R, Janssen T. Evaluation of Deep Learning Clinical Target Volumes Auto-Contouring for Magnetic Resonance Imaging-Guided Online Adaptive Treatment of Rectal Cancer. Adv Radiat Oncol 2024; 9:101483. [PMID: 38706833 PMCID: PMC11066509 DOI: 10.1016/j.adro.2024.101483] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Accepted: 02/11/2024] [Indexed: 05/07/2024] Open
Abstract
Purpose Segmentation of clinical target volumes (CTV) on medical images can be time-consuming and is prone to interobserver variation (IOV). This is a problem for online adaptive radiation therapy, where CTV segmentation must be performed every treatment fraction, leading to longer treatment times and logistic challenges. Deep learning (DL)-based auto-contouring has the potential to speed up CTV contouring, but its current clinical use is limited. One reason for this is that it can be time-consuming to verify the accuracy of CTV contours produced using auto-contouring, and there is a risk of bias being introduced. To be accepted by clinicians, auto-contouring must be trustworthy. Therefore, there is a need for a comprehensive commissioning framework when introducing DL-based auto-contouring in clinical practice. We present such a framework and apply it to an in-house developed DL model for auto-contouring of the CTV in rectal cancer patients treated with MRI-guided online adaptive radiation therapy. Methods and Materials The framework for evaluating DL-based auto-contouring consisted of 3 steps: (1) Quantitative evaluation of the model's performance and comparison with IOV; (2) Expert observations and corrections; and (3) Evaluation of the impact on expected volumetric target coverage. These steps were performed on independent data sets. The framework was applied to an in-house trained nnU-Net model, using the data of 44 rectal cancer patients treated at our institution. Results The framework established that the model's performance after expert corrections was comparable to IOV, and although the model introduced a bias, this had no relevant impact on clinical practice. Additionally, we found a substantial time gain without reducing quality as determined by volumetric target coverage. Conclusions Our framework provides a comprehensive evaluation of the performance and clinical usability of target auto-contouring models. Based on the results, we conclude that the model is eligible for clinical use.
Collapse
Affiliation(s)
| | | | - Anja Betgen
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Lisa Wiersema
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Corrie Marijnen
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Femke Peters
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Uulke A. van der Heide
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Rita Simões
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Tomas Janssen
- Department of Radiation Oncology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
| |
Collapse
|
13
|
Rivetti L, Studen A, Sharma M, Chan J, Jeraj R. Uncertainty estimation and evaluation of deformation image registration based convolutional neural networks. Phys Med Biol 2024; 69:115045. [PMID: 38749468 DOI: 10.1088/1361-6560/ad4c4f] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Accepted: 05/15/2024] [Indexed: 05/31/2024]
Abstract
Objective.Fast and accurate deformable image registration (DIR), including DIR uncertainty estimation, is essential for safe and reliable clinical deployment. While recent deep learning models have shown promise in predicting DIR with its uncertainty, challenges persist in proper uncertainty evaluation and hyperparameter optimization for these methods. This work aims to develop and evaluate a model that can perform fast DIR and predict its uncertainty in seconds.Approach.This study introduces a novel probabilistic multi-resolution image registration model utilizing convolutional neural networks to estimate a multivariate normal distributed dense displacement field (DDF) in a multimodal image registration problem. To assess the quality of the DDF distribution predicted by the model, we propose a new metric based on the Kullback-Leibler divergence. The performance of our approach was evaluated against three other DIR algorithms (VoxelMorph, Monte Carlo dropout, and Monte Carlo B-spline) capable of predicting uncertainty. The evaluation of the models included not only the quality of the deformation but also the reliability of the estimated uncertainty. Our application investigated the registration of a treatment planning computed tomography (CT) to follow-up cone beam CT for daily adaptive radiotherapy.Main results.The hyperparameter tuning of the models showed a trade-off between the estimated uncertainty's reliability and the deformation's accuracy. In the optimal trade-off, our model excelled in contour propagation and uncertainty estimation (p <0.05) compared to existing uncertainty estimation models. We obtained an average dice similarity coefficient of 0.89 and a KL-divergence of 0.15.Significance.By addressing challenges in DIR uncertainty estimation and evaluation, our work showed that both the DIR and its uncertainty can be reliably predicted, paving the way for safe deployment in a clinical environment.
Collapse
Affiliation(s)
- Luciano Rivetti
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
| | - Andrej Studen
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
- Jožef Stefan Institute, Ljubljana, Slovenia
| | - Manju Sharma
- Department of Radiation Oncology, University of California, San Francisco, CA, United States of America
| | - Jason Chan
- Department of Radiation Oncology, University of California, San Francisco, CA, United States of America
| | - Robert Jeraj
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
- Jožef Stefan Institute, Ljubljana, Slovenia
- School of Medicine and Public Health, Department of Medical Physics, University of Wisconsin, Madison, WI, United States of America
| |
Collapse
|
14
|
Li S, Wang H, Meng Y, Zhang C, Song Z. Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation. Phys Med Biol 2024; 69:11TR01. [PMID: 38479023 DOI: 10.1088/1361-6560/ad33b5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 03/13/2024] [Indexed: 05/21/2024]
Abstract
Precise delineation of multiple organs or abnormal regions in the human body from medical images plays an essential role in computer-aided diagnosis, surgical simulation, image-guided interventions, and especially in radiotherapy treatment planning. Thus, it is of great significance to explore automatic segmentation approaches, among which deep learning-based approaches have evolved rapidly and witnessed remarkable progress in multi-organ segmentation. However, obtaining an appropriately sized and fine-grained annotated dataset of multiple organs is extremely hard and expensive. Such scarce annotation limits the development of high-performance multi-organ segmentation models but promotes many annotation-efficient learning paradigms. Among these, studies on transfer learning leveraging external datasets, semi-supervised learning including unannotated datasets and partially-supervised learning integrating partially-labeled datasets have led the dominant way to break such dilemmas in multi-organ segmentation. We first review the fully supervised method, then present a comprehensive and systematic elaboration of the 3 abovementioned learning paradigms in the context of multi-organ segmentation from both technical and methodological perspectives, and finally summarize their challenges and future trends.
Collapse
Affiliation(s)
- Shiman Li
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Haoran Wang
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Yucong Meng
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Chenxi Zhang
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Zhijian Song
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| |
Collapse
|
15
|
Khodadadi Shoushtari F, Dehkordi ANV, Sina S. Quantitative and Visual Analysis of Data Augmentation and Hyperparameter Optimization in Deep Learning-Based Segmentation of Low-Grade Glioma Tumors Using Grad-CAM. Ann Biomed Eng 2024; 52:1359-1377. [PMID: 38409433 DOI: 10.1007/s10439-024-03461-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Accepted: 01/29/2024] [Indexed: 02/28/2024]
Abstract
This study executes a quantitative and visual investigation on the effectiveness of data augmentation and hyperparameter optimization on the accuracy of deep learning-based segmentation of LGG tumors. The study employed the MobileNetV2 and ResNet backbones with atrous convolution in DeepLabV3+ structure. The Grad-CAM tool was also used to interpret the effect of augmentation and network optimization on segmentation performance. A wide investigation was performed to optimize the network hyperparameters. In addition, the study examined 35 different models to evaluate different data augmentation techniques. The results of the study indicated that incorporating data augmentation techniques and optimization can improve the performance of segmenting brain LGG tumors up to 10%. Our extensive investigation of the data augmentation techniques indicated that enlargement of data from 90° and 225° rotated data,up to down and left to right flipping are the most effective techniques. MobilenetV2 as the backbone,"Focal Loss" as the loss function and "Adam" as the optimizer showed the superior results. The optimal model (DLG-Net) achieved an overall accuracy of 96.1% with a loss value of 0.006. Specifically, the segmentation performance for Whole Tumor (WT), Tumor Core (TC), and Enhanced Tumor (ET) reached a Dice Similarity Coefficient (DSC) of 89.4%, 70.1%, and 49.9%, respectively. Simultaneous visual and quantitative assessment of data augmentation and network optimization can lead to an optimal model with a reasonable performance in segmenting the LGG tumors.
Collapse
Affiliation(s)
| | - Azimeh N V Dehkordi
- Department of Physics, Faculty of Computer Engineering, Najafabad Branch, Islamic Azad University, Najafabad, Iran.
- Najafabad Branch, Islamic Azad University, Najafabad, 8514143131, Iran.
| | - Sedigheh Sina
- Nuclear Engineering Department, Shiraz University, Shiraz, Iran
- Radiation Research Center, Shiraz University, Shiraz, Iran
| |
Collapse
|
16
|
Yu X, Qin Y, Zhang F, Zhang Z. A recurrent positional encoding circular attention mechanism network for biomedical image segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 246:108054. [PMID: 38350190 DOI: 10.1016/j.cmpb.2024.108054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Revised: 01/16/2024] [Accepted: 01/28/2024] [Indexed: 02/15/2024]
Abstract
Deep-learning-based medical image segmentation techniques can assist doctors in disease diagnosis and rapid treatment. However, existing medical image segmentation models do not fully consider the dependence between feature segments in the feature extraction process, and the correlated features can be further extracted. Therefore, a recurrent positional encoding circular attention mechanism network (RPECAMNet) is proposed based on relative positional encoding for medical image segmentation. Multiple residual modules are used to extract the primary features of the medical images, which are thereafter converted into one-dimensional data for relative positional encoding. The recursive former is used to further extract features from medical images, and decoding is performed using deconvolution. An adaptive loss function is designed to train the model and achieve accurate medical-image segmentation. Finally, the proposed model is used to conduct comparative experiments on the synapse and self-constructed kidney datasets to verify the accuracy of the proposed model for medical image segmentation.
Collapse
Affiliation(s)
- Xiaoxia Yu
- College of Mechanical Engineering, Chongqing University of Technology, Chongqing, 400054, China.
| | - Yong Qin
- Ph.D. degree at Children's Hospital of Chongqing Medical University, Chongqing, 400016, China
| | - Fanghong Zhang
- Ghent University in Belgium in 2012. She is now a professor at Chongqing Normal University, Chongqing, 400054, China
| | - Zhigang Zhang
- College of Mechanical Engineering, Chongqing University of Technology, Chongqing, 400045, China
| |
Collapse
|
17
|
Zhang C, Wang J. PIS-Net: Efficient Medical Image Segmentation Network with Multivariate Downsampling for Point-of-Care. ENTROPY (BASEL, SWITZERLAND) 2024; 26:284. [PMID: 38667838 PMCID: PMC11049141 DOI: 10.3390/e26040284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 03/19/2024] [Accepted: 03/24/2024] [Indexed: 04/28/2024]
Abstract
Recently, with more portable diagnostic devices being moved to people anywhere, point-of-care (PoC) imaging has become more convenient and more popular than the traditional "bed imaging". Instant image segmentation, as an important technology of computer vision, is receiving more and more attention in PoC diagnosis. However, the image distortion caused by image preprocessing and the low resolution of medical images extracted by PoC devices are urgent problems that need to be solved. Moreover, more efficient feature representation is necessary in the design of instant image segmentation. In this paper, a new feature representation considering the relationships among local features with minimal parameters and a lower computational complexity is proposed. Since a feature window sliding along a diagonal can capture more pluralistic features, a Diagonal-Axial Multi-Layer Perceptron is designed to obtain the global correlation among local features for a more comprehensive feature representation. Additionally, a new multi-scale feature fusion is proposed to integrate nonlinear features with linear ones to obtain a more precise feature representation. Richer features are figured out. In order to improve the generalization of the models, a dynamic residual spatial pyramid pooling based on various receptive fields is constructed according to different sizes of images, which alleviates the influence of image distortion. The experimental results show that the proposed strategy has better performance on instant image segmentation. Notably, it yields an average improvement of 1.31% in Dice than existing strategies on the BUSI, ISIC2018 and MoNuSeg datasets.
Collapse
Affiliation(s)
| | - Jia Wang
- School of Computer Science and Technology, Xinjiang University, Urumqi 830046, China;
| |
Collapse
|
18
|
Roberfroid B, Lee JA, Geets X, Sterpin E, Barragán-Montero AM. DIVE-ART: A tool to guide clinicians towards dosimetrically informed volume editions of automatically segmented volumes in adaptive radiation therapy. Radiother Oncol 2024; 192:110108. [PMID: 38272315 DOI: 10.1016/j.radonc.2024.110108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Revised: 01/17/2024] [Accepted: 01/17/2024] [Indexed: 01/27/2024]
Affiliation(s)
- Benjamin Roberfroid
- Université catholique de Louvain - Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Brussels, Belgium.
| | - John A Lee
- Université catholique de Louvain - Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Brussels, Belgium
| | - Xavier Geets
- Université catholique de Louvain - Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Brussels, Belgium; Cliniques universitaires Saint-Luc, Department of Radiation Oncology, Brussels, Belgium
| | - Edmond Sterpin
- Université catholique de Louvain - Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Brussels, Belgium; KU Leuven - Department of Oncology, Laboratory of Experimental Radiotherapy, Leuven, Belgium; Particle Therapy Interuniversity Center Leuven - PARTICLE, Leuven, Belgium
| | - Ana M Barragán-Montero
- Université catholique de Louvain - Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Brussels, Belgium
| |
Collapse
|
19
|
Singh S, Singh BK, Kumar A. Multi-organ segmentation of organ-at-risk (OAR's) of head and neck site using ensemble learning technique. Radiography (Lond) 2024; 30:673-680. [PMID: 38364707 DOI: 10.1016/j.radi.2024.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 11/25/2023] [Accepted: 02/05/2024] [Indexed: 02/18/2024]
Abstract
INTRODUCTION This paper presents a novel approach to automate the segmentation of Organ-at-Risk (OAR) in Head and Neck cancer patients using Deep Learning models combined with Ensemble Learning techniques. The study aims to improve the accuracy and efficiency of OAR segmentation, essential for radiotherapy treatment planning. METHODS The dataset comprised computed tomography (CT) scans of 182 patients in DICOM format, obtained from an institutional image bank. Experienced Radiation Oncologists manually segmented seven OARs for each scan. Two models, 3D U-Net and 3D DenseNet-FCN, were trained on reduced CT scans (192 × 192 x 128) due to memory limitations. Ensemble Learning techniques were employed to enhance accuracy and segmentation metrics. Testing was conducted on 78 patients from the institutional dataset and an open-source dataset (TCGA-HNSC and Head-Neck Cetuximab) consisting of 31 patient scans. RESULTS Using the Ensemble Learning technique, the average dice similarity coefficient for OARs ranged from 0.990 to 0.994, indicating high segmentation accuracy. The 95% Hausdorff distance (mm) ranged from 1.3 to 2.1, demonstrating precise segmentation boundaries. CONCLUSION The proposed automated segmentation method achieved efficient and accurate OAR segmentation, surpassing human expert performance in terms of time and accuracy. IMPLICATIONS FOR PRACTICE This approach has implications for improving treatment planning and patient care in radiotherapy. By reducing manual segmentation reliance, the proposed method offers significant time savings and potential improvements in treatment planning efficiency and precision for head and neck cancer patients.
Collapse
Affiliation(s)
- S Singh
- Department of Physics, GLA University, Mathura, Uttar Pradesh, India; Department of Radiation Oncology, Division of Medical Physics, Rajiv Gandhi Cancer Institute and Research Center, New Delhi, India.
| | - B K Singh
- Department of Physics, GLA University, Mathura, Uttar Pradesh, India.
| | - A Kumar
- Department of Radiotherapy, S N. Medical College, Agra, Uttar Pradesh, India.
| |
Collapse
|
20
|
Eidex Z, Ding Y, Wang J, Abouei E, Qiu RLJ, Liu T, Wang T, Yang X. Deep learning in MRI-guided radiation therapy: A systematic review. J Appl Clin Med Phys 2024; 25:e14155. [PMID: 37712893 PMCID: PMC10860468 DOI: 10.1002/acm2.14155] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 05/10/2023] [Accepted: 08/21/2023] [Indexed: 09/16/2023] Open
Abstract
Recent advances in MRI-guided radiation therapy (MRgRT) and deep learning techniques encourage fully adaptive radiation therapy (ART), real-time MRI monitoring, and the MRI-only treatment planning workflow. Given the rapid growth and emergence of new state-of-the-art methods in these fields, we systematically review 197 studies written on or before December 31, 2022, and categorize the studies into the areas of image segmentation, image synthesis, radiomics, and real time MRI. Building from the underlying deep learning methods, we discuss their clinical importance and current challenges in facilitating small tumor segmentation, accurate x-ray attenuation information from MRI, tumor characterization and prognosis, and tumor motion tracking. In particular, we highlight the recent trends in deep learning such as the emergence of multi-modal, visual transformer, and diffusion models.
Collapse
Affiliation(s)
- Zach Eidex
- Department of Radiation Oncology and Winship Cancer InstituteEmory UniversityAtlantaGeorgiaUSA
- School of Mechanical EngineeringGeorgia Institute of TechnologyAtlantaGeorgiaUSA
| | - Yifu Ding
- Department of Radiation Oncology and Winship Cancer InstituteEmory UniversityAtlantaGeorgiaUSA
| | - Jing Wang
- Department of Radiation Oncology and Winship Cancer InstituteEmory UniversityAtlantaGeorgiaUSA
| | - Elham Abouei
- Department of Radiation Oncology and Winship Cancer InstituteEmory UniversityAtlantaGeorgiaUSA
| | - Richard L. J. Qiu
- Department of Radiation Oncology and Winship Cancer InstituteEmory UniversityAtlantaGeorgiaUSA
| | - Tian Liu
- Department of Radiation OncologyIcahn School of Medicine at Mount SinaiNew YorkNew YorkUSA
| | - Tonghe Wang
- Department of Medical PhysicsMemorial Sloan Kettering Cancer CenterNew YorkNew YorkUSA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer InstituteEmory UniversityAtlantaGeorgiaUSA
- School of Mechanical EngineeringGeorgia Institute of TechnologyAtlantaGeorgiaUSA
| |
Collapse
|
21
|
Krokos G, Kotwal T, Malaih A, Barrington S, Jackson P, Hicks RJ, Marsden PK, Fischer BM. Evaluation of manual and automated approaches for segmentation and extraction of quantitative indices from [ 18F]FDG PET-CT images. Biomed Phys Eng Express 2024; 10:025007. [PMID: 38100790 PMCID: PMC10767880 DOI: 10.1088/2057-1976/ad160e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Revised: 11/28/2023] [Accepted: 12/15/2023] [Indexed: 12/17/2023]
Abstract
Utilisation of whole organ volumes to extract anatomical and functional information from computed tomography (CT) and positron emission tomography (PET) images may provide key information for the treatment and follow-up of cancer patients. However, manual organ segmentation, is laborious and time-consuming. In this study, a CT-based deep learning method and a multi-atlas method were evaluated for segmenting the liver and spleen on CT images to extract quantitative tracer information from Fluorine-18 fluorodeoxyglucose ([18F]FDG) PET images of 50 patients with advanced Hodgkin lymphoma (HL). Manual segmentation was used as the reference method. The two automatic methods were also compared with a manually defined volume of interest (VOI) within the organ, a technique commonly performed in clinical settings. Both automatic methods provided accurate CT segmentations, with the deep learning method outperforming the multi-atlas with a DICE coefficient of 0.93 ± 0.03 (mean ± standard deviation) in liver and 0.87 ± 0.17 in spleen compared to 0.87 ± 0.05 (liver) and 0.78 ± 0.11 (spleen) for the multi-atlas. Similarly, a mean relative error of -3.2% for the liver and -3.4% for the spleen across patients was found for the mean standardized uptake value (SUVmean) using the deep learning regions while the corresponding errors for the multi-atlas method were -4.7% and -9.2%, respectively. For the maximum SUV (SUVmax), both methods resulted in higher than 20% overestimation due to the extension of organ boundaries to include neighbouring, high-uptake regions. The conservative VOI method which did not extend into neighbouring tissues, provided a more accurate SUVmaxestimate. In conclusion, the automatic, and particularly the deep learning method could be used to rapidly extract information of the SUVmeanwithin the liver and spleen. However, activity from neighbouring organs and lesions can lead to high biases in SUVmaxand current practices of manually defining a volume of interest in the organ should be considered instead.
Collapse
Affiliation(s)
- Georgios Krokos
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Tejas Kotwal
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Afnan Malaih
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Sally Barrington
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | | | - Rodney J Hicks
- Department of Medicine, St Vincent’s Hospital Medical School, the University of Melbourne, Australia
| | - Paul K Marsden
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Barbara Malene Fischer
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
- Dept. Clinical Physiology and Nuclear Medicine, Rigshospitalet, Copenhagen, Denmark
- Dept. of Clinical Medicine, University of Copenhagen, Denmark
| |
Collapse
|
22
|
Wang C, Wang L, Wang N, Wei X, Feng T, Wu M, Yao Q, Zhang R. CFATransUnet: Channel-wise cross fusion attention and transformer for 2D medical image segmentation. Comput Biol Med 2024; 168:107803. [PMID: 38064854 DOI: 10.1016/j.compbiomed.2023.107803] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Revised: 11/23/2023] [Accepted: 11/29/2023] [Indexed: 01/10/2024]
Abstract
Medical image segmentation faces current challenges in effectively extracting and fusing long-distance and local semantic information, as well as mitigating or eliminating semantic gaps during the encoding and decoding process. To alleviate the above two problems, we propose a new U-shaped network structure, called CFATransUnet, with Transformer and CNN blocks as the backbone network, equipped with Channel-wise Cross Fusion Attention and Transformer (CCFAT) module, containing Channel-wise Cross Fusion Transformer (CCFT) and Channel-wise Cross Fusion Attention (CCFA). Specifically, we use a Transformer and CNN blocks to construct the encoder and decoder for adequate extraction and fusion of long-range and local semantic features. The CCFT module utilizes the self-attention mechanism to reintegrate semantic information from different stages into cross-level global features to reduce the semantic asymmetry between features at different levels. The CCFA module adaptively acquires the importance of each feature channel based on a global perspective in a network learning manner, enhancing effective information grasping and suppressing non-important features to mitigate semantic gaps. The combination of CCFT and CCFA can guide the effective fusion of different levels of features more powerfully with a global perspective. The consistent architecture of the encoder and decoder also alleviates the semantic gap. Experimental results suggest that the proposed CFATransUnet achieves state-of-the-art performance on four datasets. The code is available at https://github.com/CPU0808066/CFATransUnet.
Collapse
Affiliation(s)
- Cheng Wang
- Department of Optical Science and Engineering, Fudan University, Shanghai 200433, China
| | - Le Wang
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
| | - Nuoqi Wang
- Department of Optical Science and Engineering, Fudan University, Shanghai 200433, China
| | - Xiaoling Wei
- Department of Endodontics, Shanghai Stomatological Hospital, Fudan University, Shanghai 200001, China
| | - Ting Feng
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
| | - Minfeng Wu
- Department of Dermatology, Huadong Hospital Affiliated to Fudan University, Shanghai, 200040, China.
| | - Qi Yao
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, China.
| | - Rongjun Zhang
- Department of Optical Science and Engineering, Fudan University, Shanghai 200433, China; Academy for Engineering and Technology, Fudan University, Shanghai 200433, China; Zhuhai Fudan Innovation Institute, Zhuhai 519031, China.
| |
Collapse
|
23
|
Xie F, Ju J, Zhang T, Wang H, Liu J, Wang J, Zhou Y, Zhao X. A Small Intestinal Stromal Tumor Detection Method Based on an Attention Balance Feature Pyramid. SENSORS (BASEL, SWITZERLAND) 2023; 23:9723. [PMID: 38139569 PMCID: PMC10747994 DOI: 10.3390/s23249723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Revised: 11/05/2023] [Accepted: 11/15/2023] [Indexed: 12/24/2023]
Abstract
Small intestinal stromal tumor (SIST) is a common gastrointestinal tumor. Currently, SIST diagnosis relies on clinical radiologists reviewing CT images from medical imaging sensors. However, this method is inefficient and greatly affected by subjective factors. The automatic detection method for stromal tumors based on computer vision technology can better solve these problems. However, in CT images, SIST have different shapes and sizes, blurred edge texture, and little difference from surrounding normal tissues, which to a large extent challenges the use of computer vision technology for the automatic detection of stromal tumors. Furthermore, there are the following issues in the research on the detection and recognition of SIST. After analyzing mainstream target detection models on SIST data, it was discovered that there is an imbalance in the features at different levels during the feature fusion stage of the network model. Therefore, this paper proposes an algorithm, based on the attention balance feature pyramid (ABFP), for detecting SIST with unbalanced feature fusion in the target detection model. By combining weighted multi-level feature maps from the backbone network, the algorithm creates a balanced semantic feature map. Spatial attention and channel attention modules are then introduced to enhance this map. In the feature fusion stage, the algorithm scales the enhanced balanced semantic feature map to the size of each level feature map and enhances the original feature information with the original feature map, effectively addressing the imbalance between deep and shallow features. Consequently, the SIST detection model's detection performance is significantly improved, and the method is highly versatile. Experimental results show that the ABFP method can enhance traditional target detection methods, and is compatible with various models and feature fusion strategies.
Collapse
Affiliation(s)
- Fei Xie
- Xi’an Key Laboratory of Human–Machine Integration and Control Technology for Intelligent Rehabilitation, Xijing University, Xi’an 710123, China; (F.X.); (J.W.)
- School of AOAIR, Xidian University, Xi’an 710075, China
| | - Jianguo Ju
- School of Information Science and Technology, Northwest University, Xi’an 710126, China; (T.Z.); (J.L.); (Y.Z.)
| | - Tongtong Zhang
- School of Information Science and Technology, Northwest University, Xi’an 710126, China; (T.Z.); (J.L.); (Y.Z.)
| | - Hexu Wang
- Xi’an Key Laboratory of Human–Machine Integration and Control Technology for Intelligent Rehabilitation, Xijing University, Xi’an 710123, China; (F.X.); (J.W.)
| | - Jindong Liu
- School of Information Science and Technology, Northwest University, Xi’an 710126, China; (T.Z.); (J.L.); (Y.Z.)
| | - Juan Wang
- Xi’an Key Laboratory of Human–Machine Integration and Control Technology for Intelligent Rehabilitation, Xijing University, Xi’an 710123, China; (F.X.); (J.W.)
| | - Yang Zhou
- School of Information Science and Technology, Northwest University, Xi’an 710126, China; (T.Z.); (J.L.); (Y.Z.)
| | - Xuesong Zhao
- Departments of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| |
Collapse
|
24
|
Kikuchi T, Hanaoka S, Nakao T, Nomura Y, Yoshikawa T, Alam MA, Mori H, Hayashi N. Relationship between Thyroid CT Density, Volume, and Future TSH Elevation: A 5-Year Follow-Up Study. Life (Basel) 2023; 13:2303. [PMID: 38137904 PMCID: PMC10744487 DOI: 10.3390/life13122303] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Revised: 11/30/2023] [Accepted: 12/04/2023] [Indexed: 12/24/2023] Open
Abstract
This study aimed to explore the relationship between thyroid-stimulating hormone (TSH) elevation and the baseline computed tomography (CT) density and volume of the thyroid. We examined 86 cases with new-onset hypothyroidism (TSH > 4.5 IU/mL) and 1071 controls from a medical check-up database over 5 years. A deep learning-based thyroid segmentation method was used to assess CT density and volume. Statistical tests and logistic regression were employed to determine differences and odds ratios. Initially, the case group showed a higher CT density (89.8 vs. 81.7 Hounsfield units (HUs)) and smaller volume (13.0 vs. 15.3 mL) than those in the control group. For every +10 HU in CT density and -3 mL in volume, the odds of developing hypothyroidism increased by 1.40 and 1.35, respectively. Over the course of the study, the case group showed a notable CT density reduction (median: -8.9 HU), whereas the control group had a minor decrease (-2.9 HU). Thyroid volume remained relatively stable for both groups. Higher CT density and smaller thyroid volume at baseline are correlated with future TSH elevation. Over time, there was a substantial and minor decrease in CT density in the case and control groups, respectively. Thyroid volumes remained consistent in both cohorts.
Collapse
Affiliation(s)
- Tomohiro Kikuchi
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo 113-8655, Japan (M.A.A.)
- Department of Radiology, School of Medicine, Jichi Medical University, Tochigi 329-0498, Japan
| | - Shouhei Hanaoka
- Department of Radiology, The University of Tokyo Hospital, Tokyo 113-8655, Japan
| | - Takahiro Nakao
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo 113-8655, Japan (M.A.A.)
| | - Yukihiro Nomura
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo 113-8655, Japan (M.A.A.)
- Center for Frontier Medical Engineering, Chiba University, Chiba 263-8522, Japan
| | - Takeharu Yoshikawa
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo 113-8655, Japan (M.A.A.)
| | - Md Ashraful Alam
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo 113-8655, Japan (M.A.A.)
| | - Harushi Mori
- Department of Radiology, School of Medicine, Jichi Medical University, Tochigi 329-0498, Japan
| | - Naoto Hayashi
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo 113-8655, Japan (M.A.A.)
| |
Collapse
|
25
|
Lei Y, Ding Y, Qiu RLJ, Wang T, Roper J, Fu Y, Shu HK, Mao H, Yang X. Hippocampus substructure segmentation using morphological vision transformer learning. Phys Med Biol 2023; 68:235013. [PMID: 37972414 PMCID: PMC10690959 DOI: 10.1088/1361-6560/ad0d45] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Revised: 11/01/2023] [Accepted: 11/15/2023] [Indexed: 11/19/2023]
Abstract
The hippocampus plays a crucial role in memory and cognition. Because of the associated toxicity from whole brain radiotherapy, more advanced treatment planning techniques prioritize hippocampal avoidance, which depends on an accurate segmentation of the small and complexly shaped hippocampus. To achieve accurate segmentation of the anterior and posterior regions of the hippocampus from T1 weighted (T1w) MR images, we developed a novel model, Hippo-Net, which uses a cascaded model strategy. The proposed model consists of two major parts: (1) a localization model is used to detect the volume-of-interest (VOI) of hippocampus. (2) An end-to-end morphological vision transformer network (Franchietal2020Pattern Recognit.102107246, Ranemetal2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW) pp 3710-3719) is used to perform substructures segmentation within the hippocampus VOI. The substructures include the anterior and posterior regions of the hippocampus, which are defined as the hippocampus proper and parts of the subiculum. The vision transformer incorporates the dominant features extracted from MR images, which are further improved by learning-based morphological operators. The integration of these morphological operators into the vision transformer increases the accuracy and ability to separate hippocampus structure into its two distinct substructures. A total of 260 T1w MRI datasets from medical segmentation decathlon dataset were used in this study. We conducted a five-fold cross-validation on the first 200 T1w MR images and then performed a hold-out test on the remaining 60 T1w MR images with the model trained on the first 200 images. In five-fold cross-validation, the Dice similarity coefficients were 0.900 ± 0.029 and 0.886 ± 0.031 for the hippocampus proper and parts of the subiculum, respectively. The mean surface distances (MSDs) were 0.426 ± 0.115 mm and 0.401 ± 0.100 mm for the hippocampus proper and parts of the subiculum, respectively. The proposed method showed great promise in automatically delineating hippocampus substructures on T1w MR images. It may facilitate the current clinical workflow and reduce the physicians' effort.
Collapse
Affiliation(s)
- Yang Lei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, United States of America
| | - Yifu Ding
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, United States of America
| | - Richard L J Qiu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, United States of America
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States of America
| | - Justin Roper
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, United States of America
| | - Yabo Fu
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States of America
| | - Hui-Kuo Shu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, United States of America
| | - Hui Mao
- Department of Radiology and Imaging Sciences and Winship Cancer Institute, Atlanta, GA 30308, United States of America
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30308, United States of America
| |
Collapse
|
26
|
Roest C, Kloet RW, Lamers MJ, Yakar D, Kwee TC. Focused view CT angiography for selective visualization of stroke related arteries: technical feasibility. Eur Radiol 2023; 33:9099-9108. [PMID: 37438639 PMCID: PMC10667412 DOI: 10.1007/s00330-023-09904-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2022] [Revised: 04/18/2023] [Accepted: 05/02/2023] [Indexed: 07/14/2023]
Abstract
OBJECTIVES This study investigated the technical feasibility of focused view CTA for the selective visualization of stroke related arteries. METHODS A total of 141 CTA examinations for acute ischemic stroke evaluation were divided into a set of 100 cases to train a deep learning algorithm (dubbed "focused view CTA") that selectively extracts brain (including intracranial arteries) and extracranial arteries, and a test set of 41 cases. The visibility of anatomic structures at focused view and unmodified CTA was assessed using the following scoring system: 5 = completely visible, diagnostically sufficient; 4 = nearly completely visible, diagnostically sufficient; 3 = incompletely visible, barely diagnostically sufficient; 2 = hardly visible, diagnostically insufficient; 1 = not visible, diagnostically insufficient. RESULTS At focused view CTA, median scores for the aortic arch, subclavian arteries, common carotid arteries, C1, C6, and C7 segments of the internal carotid arteries, V4 segment of the vertebral arteries, basilar artery, cerebellum including cerebellar arteries, cerebrum including cerebral arteries, and dural venous sinuses, were all 4. Median scores for the C2 to C5 segments of the internal carotid arteries, and V1 to V3 segments of the vertebral arteries ranged between 3 and 2. At unmodified CTA, median score for all above-mentioned anatomic structures was 5, which was significantly higher (p < 0.0001) than that at focused view CTA. CONCLUSION Focused view CTA shows promise for the selective visualization of stroke-related arteries. Further improvements should focus on more accurately visualizing the smaller and tortuous internal carotid and vertebral artery segments close to bone. CLINICAL RELEVANCE Focused view CTA may speed up image interpretation time for LVO detection and may potentially be used as a tool to study the clinical relevance of incidental findings in future prospective long-term follow-up studies. KEY POINTS • A deep learning-based algorithm ("focused view CTA") was developed to selectively visualize relevant structures for acute ischemic stroke evaluation at CTA. • The elimination of unrequested anatomic background information was complete in all cases. • Focused view CTA may be used to study the clinical relevance of incidental findings.
Collapse
Affiliation(s)
- Christian Roest
- Medical Imaging Center, Department of Radiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands.
| | - Reina W Kloet
- Medical Imaging Center, Department of Radiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands
| | - Maria J Lamers
- Medical Imaging Center, Department of Radiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands
| | - Derya Yakar
- Medical Imaging Center, Department of Radiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands
- Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands
| | - Thomas C Kwee
- Medical Imaging Center, Department of Radiology, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands
| |
Collapse
|
27
|
Salimi Y, Akhavanallaf A, Mansouri Z, Shiri I, Zaidi H. Real-time, acquisition parameter-free voxel-wise patient-specific Monte Carlo dose reconstruction in whole-body CT scanning using deep neural networks. Eur Radiol 2023; 33:9411-9424. [PMID: 37368113 PMCID: PMC10667156 DOI: 10.1007/s00330-023-09839-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 03/28/2023] [Accepted: 04/14/2023] [Indexed: 06/28/2023]
Abstract
OBJECTIVE We propose a deep learning-guided approach to generate voxel-based absorbed dose maps from whole-body CT acquisitions. METHODS The voxel-wise dose maps corresponding to each source position/angle were calculated using Monte Carlo (MC) simulations considering patient- and scanner-specific characteristics (SP_MC). The dose distribution in a uniform cylinder was computed through MC calculations (SP_uniform). The density map and SP_uniform dose maps were fed into a residual deep neural network (DNN) to predict SP_MC through an image regression task. The whole-body dose maps reconstructed by the DNN and MC were compared in the 11 test cases scanned with two tube voltages through transfer learning with/without tube current modulation (TCM). The voxel-wise and organ-wise dose evaluations, such as mean error (ME, mGy), mean absolute error (MAE, mGy), relative error (RE, %), and relative absolute error (RAE, %), were performed. RESULTS The model performance for the 120 kVp and TCM test set in terms of ME, MAE, RE, and RAE voxel-wise parameters was - 0.0302 ± 0.0244 mGy, 0.0854 ± 0.0279 mGy, - 1.13 ± 1.41%, and 7.17 ± 0.44%, respectively. The organ-wise errors for 120 kVp and TCM scenario averaged over all segmented organs in terms of ME, MAE, RE, and RAE were - 0.144 ± 0.342 mGy, and 0.23 ± 0.28 mGy, - 1.11 ± 2.90%, 2.34 ± 2.03%, respectively. CONCLUSION Our proposed deep learning model is able to generate voxel-level dose maps from a whole-body CT scan with reasonable accuracy suitable for organ-level absorbed dose estimation. CLINICAL RELEVANCE STATEMENT We proposed a novel method for voxel dose map calculation using deep neural networks. This work is clinically relevant since accurate dose calculation for patients can be carried out within acceptable computational time compared to lengthy Monte Carlo calculations. KEY POINTS • We proposed a deep neural network approach as an alternative to Monte Carlo dose calculation. • Our proposed deep learning model is able to generate voxel-level dose maps from a whole-body CT scan with reasonable accuracy, suitable for organ-level dose estimation. • By generating a dose distribution from a single source position, our model can generate accurate and personalized dose maps for a wide range of acquisition parameters.
Collapse
Affiliation(s)
- Yazdan Salimi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva, Switzerland
| | - Azadeh Akhavanallaf
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva, Switzerland
| | - Zahra Mansouri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva, Switzerland
| | - Isaac Shiri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva, Switzerland
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva, Switzerland.
- Geneva University Neurocenter, Geneva University, CH_1205, Geneva, Switzerland.
- Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen, Groningen, Netherlands.
- Department of Nuclear Medicine, University of Southern Denmark, DK-500, Odense, Denmark.
| |
Collapse
|
28
|
Alruily M, Said W, Mostafa AM, Ezz M, Elmezain M. Breast Ultrasound Images Augmentation and Segmentation Using GAN with Identity Block and Modified U-Net 3. SENSORS (BASEL, SWITZERLAND) 2023; 23:8599. [PMID: 37896692 PMCID: PMC10610596 DOI: 10.3390/s23208599] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 10/10/2023] [Accepted: 10/16/2023] [Indexed: 10/29/2023]
Abstract
One of the most prevalent diseases affecting women in recent years is breast cancer. Early breast cancer detection can help in the treatment, lower the infection risk, and worsen the results. This paper presents a hybrid approach for augmentation and segmenting breast cancer. The framework contains two main stages: augmentation and segmentation of ultrasound images. The augmentation of the ultrasounds is applied using generative adversarial networks (GAN) with nonlinear identity block, label smoothing, and a new loss function. The segmentation of the ultrasounds applied a modified U-Net 3+. The hybrid approach achieves efficient results in the segmentation and augmentation steps compared with the other available methods for the same task. The modified version of the GAN with the nonlinear identity block overcomes different types of modified GAN in the ultrasound augmentation process, such as speckle GAN, UltraGAN, and deep convolutional GAN. The modified U-Net 3+ also overcomes the different architectures of U-Nets in the segmentation process. The GAN with nonlinear identity blocks achieved an inception score of 14.32 and a Fréchet inception distance of 41.86 in the augmenting process. The GAN with identity achieves a smaller value in Fréchet inception distance (FID) and a bigger value in inception score; these results prove the model's efficiency compared with other versions of GAN in the augmentation process. The modified U-Net 3+ architecture achieved a Dice Score of 95.49% and an Accuracy of 95.67%.
Collapse
Affiliation(s)
- Meshrif Alruily
- College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia; (M.A.); (M.E.)
| | - Wael Said
- Computer Science Department, Faculty of Computers and Informatics, Zagazig University, Zagazig 44511, Egypt;
- Computer Science Department, College of Computer Science and Engineering, Taibah University, Medina 42353, Saudi Arabia
| | - Ayman Mohamed Mostafa
- College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia; (M.A.); (M.E.)
| | - Mohamed Ezz
- College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia; (M.A.); (M.E.)
| | - Mahmoud Elmezain
- Computer Science Department, Faculty of Science, Tanta University, Tanta 31527, Egypt;
- Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 966144, Saudi Arabia
| |
Collapse
|
29
|
Sahu A, Das PK, Meher S. Recent advancements in machine learning and deep learning-based breast cancer detection using mammograms. Phys Med 2023; 114:103138. [PMID: 37914431 DOI: 10.1016/j.ejmp.2023.103138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/11/2023] [Revised: 07/22/2023] [Accepted: 09/14/2023] [Indexed: 11/03/2023] Open
Abstract
OBJECTIVE Mammogram-based automatic breast cancer detection has a primary role in accurate cancer diagnosis and treatment planning to save valuable lives. Mammography is one basic yet efficient test for screening breast cancer. Very few comprehensive surveys have been presented to briefly analyze methods for detecting breast cancer with mammograms. In this article, our objective is to give an overview of recent advancements in machine learning (ML) and deep learning (DL)-based breast cancer detection systems. METHODS We give a structured framework to categorize mammogram-based breast cancer detection techniques. Several publicly available mammogram databases and different performance measures are also mentioned. RESULTS After deliberate investigation, we find most of the works classify breast tumors either as normal-abnormal or malignant-benign rather than classifying them into three classes. Furthermore, DL-based features are more significant than hand-crafted features. However, transfer learning is preferred over others as it yields better performance in small datasets, unlike classical DL techniques. SIGNIFICANCE AND CONCLUSION In this article, we have made an attempt to give recent advancements in artificial intelligence (AI)-based breast cancer detection systems. Furthermore, a number of challenging issues and possible research directions are mentioned, which will help researchers in further scopes of research in this field.
Collapse
Affiliation(s)
- Adyasha Sahu
- Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, Odisha, 769008, India.
| | - Pradeep Kumar Das
- School of Electronics Engineering (SENSE), VIT Vellore, Tamil Nadu, 632014, India.
| | - Sukadev Meher
- Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, Odisha, 769008, India.
| |
Collapse
|
30
|
Jin S, Han H, Huang Z, Xiang Y, Du M, Hua F, Guan X, Liu J, Chen F, He H. Automatic three-dimensional nasal and pharyngeal airway subregions identification via Vision Transformer. J Dent 2023; 136:104595. [PMID: 37343616 DOI: 10.1016/j.jdent.2023.104595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Revised: 06/06/2023] [Accepted: 06/19/2023] [Indexed: 06/23/2023] Open
Abstract
OBJECTIVES Upper airway assessment requires a fully-automated segmentation system for complete or sub-regional identification. This study aimed to develop a novel Deep Learning (DL) model for accurate segmentation of the upper airway and achieve entire and subregional identification. METHODS Fifty cone-beam computed tomography (CBCT) scans, including 24,502 slices, were labelled as the ground truth by one orthodontist and two otorhinolaryngologists. A novel model, a lightweight multitask network based on the Swin Transformer and U-Net, was built for automatic segmentation of the entire upper airway and subregions. Segmentation performance was evaluated using Precision, Recall, Dice similarity coefficient (DSC) and Intersection over union (IoU). The clinical implications of the precision errors were quantitatively analysed, and comparisons between the AI model and Dolphin software were conducted. RESULTS Our model achieved good performance with a precision of 85.88-94.25%, recall of 93.74-98.44%, DSC of 90.95-96.29%, IoU of 83.68-92.85% in the overall and subregions of three-dimensional (3D) upper airway, and a precision of 91.22-97.51%, recall of 90.70-97.62%, DSC of 90.92-97.55%, and IoU of 83.41-95.29% in the subregions of two-dimensional (2D) crosssections. Discrepancies in volume and area caused by precision errors did not affect clinical outcomes. Both our AI model and the Dolphin software provided clinically acceptable consistency for pharyngeal airway assessments. CONCLUSION The novel DL model not only achieved segmentation of the entire upper airway, including the nasal cavity and subregion identification, but also performed exceptionally well, making it well suited for 3D upper airway assessment from the nasal cavity to the hypopharynx, especially for intricate structures. CLINICAL SIGNIFICANCE This system provides insights into the aetiology, risk, severity, treatment effect, and prognosis of dentoskeletal deformities and obstructive sleep apnea. It achieves rapid assessment of the entire upper airway and its subregions, making airway management-an integral part of orthodontic treatment, orthognathic surgery, and ENT surgery-easier.
Collapse
Affiliation(s)
- Suhan Jin
- Department of Orthodontics, Hubei-MOST KLOS & KLOBM, School & Hospital of Stomatology, Wuhan University,Wuhan, China; Department of Orthodontics, Affiliated Stomatological Hospital of Zunyi Medical University, Zunyi, China
| | - Haojie Han
- College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China
| | - Zhiqun Huang
- Department of Otolaryngology-Head and Neck Surgery, Renmin Hospital of Wuhan University, Wuhan, China
| | - Yuandi Xiang
- Department of Otolaryngology-Head and Neck Surgery, Renmin Hospital of Wuhan University, Wuhan, China
| | - Mingyuan Du
- Department of Orthodontics, Hubei-MOST KLOS & KLOBM, School & Hospital of Stomatology, Wuhan University,Wuhan, China
| | - Fang Hua
- Department of Orthodontics, Hubei-MOST KLOS & KLOBM, School & Hospital of Stomatology, Wuhan University,Wuhan, China
| | - Xiaoyan Guan
- Department of Orthodontics, Affiliated Stomatological Hospital of Zunyi Medical University, Zunyi, China
| | - Jianguo Liu
- School of Stomatology, Zunyi Medical University, Zunyi, China; Special Key Laboratory of Oral Diseases Research, Higher Education Institution, Zunyi, China
| | - Fang Chen
- College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China.
| | - Hong He
- Department of Orthodontics, Hubei-MOST KLOS & KLOBM, School & Hospital of Stomatology, Wuhan University,Wuhan, China.
| |
Collapse
|
31
|
Isaksson LJ, Summers P, Mastroleo F, Marvaso G, Corrao G, Vincini MG, Zaffaroni M, Ceci F, Petralia G, Orecchia R, Jereczek-Fossa BA. Automatic Segmentation with Deep Learning in Radiotherapy. Cancers (Basel) 2023; 15:4389. [PMID: 37686665 PMCID: PMC10486603 DOI: 10.3390/cancers15174389] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 08/28/2023] [Accepted: 08/30/2023] [Indexed: 09/10/2023] Open
Abstract
This review provides a formal overview of current automatic segmentation studies that use deep learning in radiotherapy. It covers 807 published papers and includes multiple cancer sites, image types (CT/MRI/PET), and segmentation methods. We collect key statistics about the papers to uncover commonalities, trends, and methods, and identify areas where more research might be needed. Moreover, we analyzed the corpus by posing explicit questions aimed at providing high-quality and actionable insights, including: "What should researchers think about when starting a segmentation study?", "How can research practices in medical image segmentation be improved?", "What is missing from the current corpus?", and more. This allowed us to provide practical guidelines on how to conduct a good segmentation study in today's competitive environment that will be useful for future research within the field, regardless of the specific radiotherapeutic subfield. To aid in our analysis, we used the large language model ChatGPT to condense information.
Collapse
Affiliation(s)
- Lars Johannes Isaksson
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
- Department of Oncology and Hemato-Oncology, University of Milan, 20141 Milan, Italy; (F.C.); (G.P.)
| | - Paul Summers
- Division of Radiology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy;
| | - Federico Mastroleo
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
- Department of Translational Medicine, University of Piemonte Orientale (UPO), 20188 Novara, Italy
| | - Giulia Marvaso
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
| | - Giulia Corrao
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
| | - Maria Giulia Vincini
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
| | - Mattia Zaffaroni
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
| | - Francesco Ceci
- Department of Oncology and Hemato-Oncology, University of Milan, 20141 Milan, Italy; (F.C.); (G.P.)
- Division of Nuclear Medicine, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy
| | - Giuseppe Petralia
- Department of Oncology and Hemato-Oncology, University of Milan, 20141 Milan, Italy; (F.C.); (G.P.)
- Precision Imaging and Research Unit, Department of Medical Imaging and Radiation Sciences, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy
| | - Roberto Orecchia
- Scientific Directorate, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy;
| | - Barbara Alicja Jereczek-Fossa
- Division of Radiation Oncology, IEO European Institute of Oncology IRCCS, 20141 Milan, Italy; (L.J.I.); (F.M.); (G.C.); (M.G.V.); (M.Z.); (B.A.J.-F.)
- Department of Oncology and Hemato-Oncology, University of Milan, 20141 Milan, Italy; (F.C.); (G.P.)
| |
Collapse
|
32
|
Fiandra C, Rosati S, Arcadipane F, Dinapoli N, Fato M, Franco P, Gallio E, Scaffidi Gennarino D, Silvetti P, Zara S, Ricardi U, Balestra G. Active bone marrow segmentation based on computed tomography imaging in anal cancer patients: A machine-learning-based proof of concept. Phys Med 2023; 113:102657. [PMID: 37567068 DOI: 10.1016/j.ejmp.2023.102657] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Revised: 06/30/2023] [Accepted: 08/05/2023] [Indexed: 08/13/2023] Open
Abstract
PURPOSE Different methods are available to identify haematopoietically active bone marrow (ActBM). However, their use can be challenging for radiotherapy routine treatments, since they require specific equipment and dedicated time. A machine learning (ML) approach, based on radiomic features as inputs to three different classifiers, was applied to computed tomography (CT) images to identify haematopoietically active bone marrow in anal cancer patients. METHODS A total of 40 patients was assigned to the construction set (training set + test set). Fluorine-18-Fluorodeoxyglucose Positron Emission Tomography (18FDG-PET) images were used to detect the active part of the pelvic bone marrow (ActPBM) and stored as ground-truth for three subregions: iliac, lower pelvis and lumbosacral bone marrow (ActIBM, ActLPBM, ActLSBM). Three parameters were used for the correspondence analyses between 18FDG-PET and ML classifiers: DICE index, Precision and Recall. RESULTS For the 40-patient cohort, median values [min; max] of the Dice index were 0.69 [0.20; 0.84], 0.76 [0.25; 0.89], and 0.36 [0.15; 0.67] for ActIBM, ActLSBM, and ActLPBM, respectively. The Precision/Recall (P/R) ratio median value for the ActLPBM structure was 0.59 [0.20; 1.84] (over segmentation), while for the other two subregions the P/R ratio median has values of 1.249 [0.43; 4.15] for ActIBM and 1.093 [0.24; 1.91] for ActLSBM (under segmentation). CONCLUSION A satisfactory degree of overlap compared to 18FDG-PET was found for 2 out of the 3 subregions within pelvic bones. Further optimization and generalization of the process is required before clinical implementation.
Collapse
Affiliation(s)
- C Fiandra
- Department of Oncology, University of Turin, Turin, Italy.
| | - S Rosati
- Department of Electronics and Telecommunications, Politecnico di Torino, Turin, Italy
| | - F Arcadipane
- Department of Oncology, University of Turin, Turin, Italy
| | - N Dinapoli
- UOC Radioterapia Oncologica, Dipartimento Diagnostica per Immagini, Radioterapia Oncologica ed Ematologia, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - M Fato
- Department of Informatics, Bioengineering, Robotics and System Engineering (DIBRIS), University of Genova, Genova, Italy
| | - P Franco
- Department of Oncology, University of Turin, Turin, Italy
| | - E Gallio
- Medical Physics Unit, A.O.U. Città della Salute e della Scienza, Turin, Italy
| | - D Scaffidi Gennarino
- Department of Electronics and Telecommunications, Politecnico di Torino, Turin, Italy
| | - P Silvetti
- Department of Oncology, University of Turin, Turin, Italy
| | - S Zara
- Tecnologie Avanzate, Torino, Italy
| | - U Ricardi
- Department of Oncology, University of Turin, Turin, Italy
| | - G Balestra
- Department of Electronics and Telecommunications, Politecnico di Torino, Turin, Italy
| |
Collapse
|
33
|
Boehringer AS, Sanaat A, Arabi H, Zaidi H. An active learning approach to train a deep learning algorithm for tumor segmentation from brain MR images. Insights Imaging 2023; 14:141. [PMID: 37620554 PMCID: PMC10449747 DOI: 10.1186/s13244-023-01487-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2023] [Accepted: 07/22/2023] [Indexed: 08/26/2023] Open
Abstract
PURPOSE This study focuses on assessing the performance of active learning techniques to train a brain MRI glioma segmentation model. METHODS The publicly available training dataset provided for the 2021 RSNA-ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge was used in this study, consisting of 1251 multi-institutional, multi-parametric MR images. Post-contrast T1, T2, and T2 FLAIR images as well as ground truth manual segmentation were used as input for the model. The data were split into a training set of 1151 cases and testing set of 100 cases, with the testing set remaining constant throughout. Deep convolutional neural network segmentation models were trained using the NiftyNet platform. To test the viability of active learning in training a segmentation model, an initial reference model was trained using all 1151 training cases followed by two additional models using only 575 cases and 100 cases. The resulting predicted segmentations of these two additional models on the remaining training cases were then addended to the training dataset for additional training. RESULTS It was demonstrated that an active learning approach for manual segmentation can lead to comparable model performance for segmentation of brain gliomas (0.906 reference Dice score vs 0.868 active learning Dice score) while only requiring manual annotation for 28.6% of the data. CONCLUSION The active learning approach when applied to model training can drastically reduce the time and labor spent on preparation of ground truth training data. CRITICAL RELEVANCE STATEMENT Active learning concepts were applied to a deep learning-assisted segmentation of brain gliomas from MR images to assess their viability in reducing the required amount of manually annotated ground truth data in model training. KEY POINTS • This study focuses on assessing the performance of active learning techniques to train a brain MRI glioma segmentation model. • The active learning approach for manual segmentation can lead to comparable model performance for segmentation of brain gliomas. • Active learning when applied to model training can drastically reduce the time and labor spent on preparation of ground truth training data.
Collapse
Affiliation(s)
- Andrew S Boehringer
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1205, Geneva, Switzerland
| | - Amirhossein Sanaat
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1205, Geneva, Switzerland
| | - Hossein Arabi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1205, Geneva, Switzerland
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1205, Geneva, Switzerland.
- Geneva University Neurocenter, University of Geneva, CH-1211, Geneva, Switzerland.
- Department of Nuclear Medicine and Molecular Imaging, University of Groningen, Groningen, Netherlands.
- Department of Nuclear Medicine, University of Southern Denmark, Odense, Denmark.
| |
Collapse
|
34
|
Tian F, Tian Z, Chen Z, Zhang D, Du S. Surface-GCN: Learning interaction experience for organ segmentation in 3D medical images. Med Phys 2023; 50:5030-5044. [PMID: 36738103 DOI: 10.1002/mp.16280] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2022] [Revised: 12/26/2022] [Accepted: 01/13/2023] [Indexed: 02/05/2023] Open
Abstract
BACKGROUND Accurate segmentation of organs has a great significance for clinical diagnosis, but it is still hard work due to the obscure imaging boundaries caused by tissue adhesion on medical images. Based on the image continuity in medical image volumes, segmentation on these slices could be inferred from adjacent slices with a clear organ boundary. Radiologists can delineate a clear organ boundary by observing adjacent slices. PURPOSE Inspired by the radiologists' delineating procedure, we design an organ segmentation model based on boundary information of adjacent slices and a human-machine interactive learning strategy to introduce clinical experience. METHODS We propose an interactive organ segmentation method for medical image volume based on Graph Convolution Network (GCN) called Surface-GCN. First, we propose a Surface Feature Extraction Network (SFE-Net) to capture surface features of a target organ, and supervise it by a Mini-batch Adaptive Surface Matching (MBASM) module. Then, to predict organ boundaries precisely, we design an automatic segmentation module based on a Surface Convolution Unit (SCU), which propagates information on organ surfaces to refine the generated boundaries. In addition, an interactive segmentation module is proposed to learn radiologists' experience of interactive corrections on organ surfaces to reduce interaction clicks. RESULTS We evaluate the proposed method on one prostate MR image dataset and two abdominal multi-organ CT datasets. The experimental results show that our method outperforms other state-of-the-art methods. For prostate segmentation, the proposed method conducts a DSC score of 94.49% on PROMISE12 test dataset. For abdominal multi-organ segmentation, the proposed method achieves DSC scores of 95, 91, 95, and 88% for the left kidney, gallbladder, spleen, and esophagus, respectively. For interactive segmentation, the proposed method reduces 5-10 interaction clicks to reach the same accuracy. CONCLUSIONS To overcome the medical organ segmentation challenge, we propose a Graph Convolutional Network called Surface-GCN by imitating radiologist interactions and learning clinical experience. On single and multiple organ segmentation tasks, the proposed method could obtain more accurate segmentation boundaries compared with other state-of-the-art methods.
Collapse
Affiliation(s)
- Fengrui Tian
- School of Software Engineering, Xi'an Jiaotong University, Xi'an, China
- Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, China
| | - Zhiqiang Tian
- School of Software Engineering, Xi'an Jiaotong University, Xi'an, China
| | - Zhang Chen
- School of Software Engineering, Xi'an Jiaotong University, Xi'an, China
| | - Dong Zhang
- Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, China
- School of Automation Science and Engineering, Xi'an Jiaotong University, Xi'an, China
| | - Shaoyi Du
- Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, China
| |
Collapse
|
35
|
Song Y, Hu J, Wang Q, Yu C, Su J, Chen L, Jiang X, Chen B, Zhang L, Yu Q, Li P, Wang F, Bai S, Luo Y, Yi Z. Young oncologists benefit more than experts from deep learning-based organs-at-risk contouring modeling in nasopharyngeal carcinoma radiotherapy: A multi-institution clinical study exploring working experience and institute group style factor. Clin Transl Radiat Oncol 2023; 41:100635. [PMID: 37251619 PMCID: PMC10213188 DOI: 10.1016/j.ctro.2023.100635] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2022] [Revised: 04/26/2023] [Accepted: 05/01/2023] [Indexed: 05/31/2023] Open
Abstract
Background To comprehensively investigate the behaviors of oncologists with different working experiences and institute group styles in deep learning-based organs-at-risk (OAR) contouring. Methods A deep learning-based contouring system (DLCS) was modeled from 188 CT datasets of patients with nasopharyngeal carcinoma (NPC) in institute A. Three institute oncology groups, A, B, and C, were included; each contained a beginner and an expert. For each of the 28 OARs, two trials were performed with manual contouring first and post-DLCS edition later, for ten test cases. Contouring performance and group consistency were quantified by volumetric and surface Dice coefficients. A volume-based and a surface-based oncologist satisfaction rate (VOSR and SOSR) were defined to evaluate the oncologists' acceptance of DLCS. Results Based on DLCS, experience inconsistency was eliminated. Intra-institute consistency was eliminated for group C but still existed for group A and group B. Group C benefits most from DLCS with the highest number of improved OARs (8 for volumetric Dice and 10 for surface Dice), followed by group B. Beginners obtained more numbers of improved OARs than experts (7 v.s. 4 in volumetric Dice and 5 v.s. 4 in surface Dice). VOSR and SOSR varied for institute groups, but the rates of beginners were all significantly higher than those of experts for OARs with experience group significance. A remarkable positive linear relationship was found between VOSR and post-DLCS edition volumetric Dice with a coefficient of 0.78. Conclusions The DLCS was effective for various institutes and the beginners benefited more than the experts.
Collapse
Affiliation(s)
- Ying Song
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, No. 24, South Section 1 of the First Ring Road, Chengdu 610065, PR China
| | - Junjie Hu
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, No. 24, South Section 1 of the First Ring Road, Chengdu 610065, PR China
| | - Qiang Wang
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, No. 24, South Section 1 of the First Ring Road, Chengdu 610065, PR China
| | - Chengrong Yu
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, No. 24, South Section 1 of the First Ring Road, Chengdu 610065, PR China
| | - Jiachong Su
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
| | - Lin Chen
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
| | - Xiaorui Jiang
- Department of Oncology, First People's Hospital of Chengdu, No. 18, Wanxiang North Road, High-tech Zone, Chengdu 610041, PR China
| | - Bo Chen
- Department of Oncology, First People's Hospital of Chengdu, No. 18, Wanxiang North Road, High-tech Zone, Chengdu 610041, PR China
| | - Lei Zhang
- Department of Oncology, Second People's Hospital of Chengdu, Chengdu, PR China
| | - Qian Yu
- Department of Oncology, Second People's Hospital of Chengdu, Chengdu, PR China
| | - Ping Li
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
| | - Feng Wang
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
| | - Sen Bai
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
| | - Yong Luo
- Cancer Center, West China Hospital, Sichuan University, No. 37 Guo Xue Alley, Chengdu 610065, PR China
| | - Zhang Yi
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, No. 24, South Section 1 of the First Ring Road, Chengdu 610065, PR China
| |
Collapse
|
36
|
Automatic abdominal segmentation using novel 3D self-adjustable organ aware deep network in CT images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104691] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/27/2023]
|
37
|
Tao B, Yu X, Wang W, Wang H, Chen X, Wang F, Wu Y. A deep learning-based automatic segmentation of zygomatic bones from cone-beam computed tomography images: A proof of concept. J Dent 2023:104582. [PMID: 37321334 DOI: 10.1016/j.jdent.2023.104582] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Revised: 05/28/2023] [Accepted: 06/06/2023] [Indexed: 06/17/2023] Open
Abstract
OBJECTIVES To investigate the efficiency and accuracy of a deep learning-based automatic segmentation method for zygomatic bones from cone-beam computed tomography (CBCT) images. METHODS One hundred thirty CBCT scans were included and randomly divided into three subsets (training, validation, and test) in a 6:2:2 ratio. A deep learning-based model was developed, and it included a classification network and a segmentation network, where an edge supervision module was added to increase the attention of the edges of zygomatic bones. Attention maps were generated by the Grad-CAM and Guided Grad-CAM algorithms to improve the interpretability of the model. The performance of the model was then compared with that of four dentists on 10 CBCT scans from the test dataset. A p value <.05 was considered statistically significant. RESULTS The accuracy of the classification network was 99.64%. The Dice coefficient (Dice) of the deep learning-based model for the test dataset was 92.34 ± 2.04%, the average surface distance (ASD) was 0.1 ± 0.15 mm, and the 95% Hausdorff distance (HD) was 0.98 ± 0.42 mm. The model required 17.03 seconds on average to segment zygomatic bones, whereas this task took 49.3 minutes for dentists to complete. The Dice score of the model for the 10 CBCT scans was 93.2 ± 1.3%, while that of the dentists was 90.37 ± 3.32%. CONCLUSIONS The proposed deep learning-based model could segment zygomatic bones with high accuracy and efficiency compared with those of dentists. CLINICAL SIGNIFICANCE The proposed automatic segmentation model for zygomatic bone could generate an accurate 3D model for the preoperative digital planning of zygoma reconstruction, orbital surgery, zygomatic implant surgery, and orthodontics.
Collapse
Affiliation(s)
- Baoxin Tao
- Department of Second Dental Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology; Shanghai Research Institute of Stomatology, Shanghai, China
| | - Xinbo Yu
- Department of Second Dental Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology; Shanghai Research Institute of Stomatology, Shanghai, China
| | - Wenying Wang
- Department of Second Dental Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology; Shanghai Research Institute of Stomatology, Shanghai, China
| | - Haowei Wang
- Department of Second Dental Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology; Shanghai Research Institute of Stomatology, Shanghai, China
| | - Xiaojun Chen
- Institute of Biomedical Manufacturing and Life Quality Engineering, State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Room 805, Dongchuan Road 800, Minhang District, Shanghai, 200240, China..
| | - Feng Wang
- Department of Second Dental Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology; Shanghai Research Institute of Stomatology, Shanghai, China..
| | - Yiqun Wu
- Department of Second Dental Center, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology; Shanghai Research Institute of Stomatology, Shanghai, China..
| |
Collapse
|
38
|
Sistaninejhad B, Rasi H, Nayeri P. A Review Paper about Deep Learning for Medical Image Analysis. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2023; 2023:7091301. [PMID: 37284172 PMCID: PMC10241570 DOI: 10.1155/2023/7091301] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/03/2023] [Revised: 02/12/2023] [Accepted: 04/21/2023] [Indexed: 06/08/2023]
Abstract
Medical imaging refers to the process of obtaining images of internal organs for therapeutic purposes such as discovering or studying diseases. The primary objective of medical image analysis is to improve the efficacy of clinical research and treatment options. Deep learning has revamped medical image analysis, yielding excellent results in image processing tasks such as registration, segmentation, feature extraction, and classification. The prime motivations for this are the availability of computational resources and the resurgence of deep convolutional neural networks. Deep learning techniques are good at observing hidden patterns in images and supporting clinicians in achieving diagnostic perfection. It has proven to be the most effective method for organ segmentation, cancer detection, disease categorization, and computer-assisted diagnosis. Many deep learning approaches have been published to analyze medical images for various diagnostic purposes. In this paper, we review the work exploiting current state-of-the-art deep learning approaches in medical image processing. We begin the survey by providing a synopsis of research works in medical imaging based on convolutional neural networks. Second, we discuss popular pretrained models and general adversarial networks that aid in improving convolutional networks' performance. Finally, to ease direct evaluation, we compile the performance metrics of deep learning models focusing on COVID-19 detection and child bone age prediction.
Collapse
Affiliation(s)
| | - Habib Rasi
- Sahand University of Technology, East Azerbaijan, New City of Sahand, Iran
| | - Parisa Nayeri
- Khoy University of Medical Sciences, West Azerbaijan, Khoy, Iran
| |
Collapse
|
39
|
Tong N, Xu Y, Zhang J, Gou S, Li M. Robust and efficient abdominal CT segmentation using shape constrained multi-scale attention network. Phys Med 2023; 110:102595. [PMID: 37178624 DOI: 10.1016/j.ejmp.2023.102595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Revised: 03/02/2023] [Accepted: 04/17/2023] [Indexed: 05/15/2023] Open
Abstract
PURPOSE Although many deep learning-based abdominal multi-organ segmentation networks have been proposed, the various intensity distributions and organ shapes of the CT images from multi-center, multi-phase with various diseases introduce new challenges for robust abdominal CT segmentation. To achieve robust and efficient abdominal multi-organ segmentation, a new two-stage method is presented in this study. METHODS A binary segmentation network is used for coarse localization, followed by a multi-scale attention network for the fine segmentation of liver, kidney, spleen, and pancreas. To constrain the organ shapes produced by the fine segmentation network, an additional network is pre-trained to learn the shape features of the organs with serious diseases and then employed to constrain the training of the fine segmentation network. RESULTS The performance of the presented segmentation method was extensively evaluated on the multi-center data set from the Fast and Low GPU Memory Abdominal oRgan sEgmentation (FLARE) challenge, which was held in conjunction with International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2021. Dice Similarity Coefficient (DSC) and Normalized Surface Dice (NSD) were calculated to quantitatively evaluate the segmentation accuracy and efficiency. An average DSC and NSD of 83.7% and 64.4% were achieved, and our method finally won the second place among more than 90 participating teams. CONCLUSIONS The evaluation results on the public challenge demonstrate that our method shows promising performance in robustness and efficiency, which may promote the clinical application of the automatic abdominal multi-organ segmentation.
Collapse
Affiliation(s)
- Nuo Tong
- AI-based Big Medical Imaging Data Frontier Research Center, Academy of Advanced Interdisciplinary Research, Xidian University, Xi'an, Shaanxi 710071, China
| | - Yinan Xu
- Key Lab of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an, Shaanxi, 710071, China
| | - Jinsong Zhang
- Xijing Hospital of Air Force Military Medical University, Xian, Shaanxi 710032, China
| | - Shuiping Gou
- AI-based Big Medical Imaging Data Frontier Research Center, Academy of Advanced Interdisciplinary Research, Xidian University, Xi'an, Shaanxi 710071, China; Key Lab of Intelligent Perception and Image Understanding of Ministry of Education, Xidian University, Xi'an, Shaanxi, 710071, China.
| | - Mengbin Li
- Xijing Hospital of Air Force Military Medical University, Xian, Shaanxi 710032, China.
| |
Collapse
|
40
|
Luu MH, Mai HS, Pham XL, Le QA, Le QK, Walsum TV, Le NH, Franklin D, Le VH, Moelker A, Chu DT, Trung NL. Quantification of liver-Lung shunt fraction on 3D SPECT/CT images for selective internal radiation therapy of liver cancer using CNN-based segmentations and non-rigid registration. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 233:107453. [PMID: 36921463 DOI: 10.1016/j.cmpb.2023.107453] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 01/25/2023] [Accepted: 02/27/2023] [Indexed: 06/18/2023]
Abstract
PURPOSE Selective internal radiation therapy (SIRT) has been proven to be an effective treatment for hepatocellular carcinoma (HCC) patients. In clinical practice, the treatment planning for SIRT using 90Y microspheres requires estimation of the liver-lung shunt fraction (LSF) to avoid radiation pneumonitis. Currently, the manual segmentation method to draw a region of interest (ROI) of the liver and lung in 2D planar imaging of 99mTc-MAA and 3D SPECT/CT images is inconvenient, time-consuming and observer-dependent. In this study, we propose and evaluate a nearly automatic method for LSF quantification using 3D SPECT/CT images, offering improved performance compared with the current manual segmentation method. METHODS We retrospectively acquired 3D SPECT with non-contrast-enhanced CT images (nCECT) of 60 HCC patients from a SPECT/CT scanning machine, along with the corresponding diagnostic contrast-enhanced CT images (CECT). Our approach for LSF quantification is to use CNN-based methods for liver and lung segmentations in the nCECT image. We first apply 3D ResUnet to coarsely segment the liver. If the liver segmentation contains a large error, we dilate the coarse liver segmentation into the liver mask as a ROI in the nCECT image. Subsequently, non-rigid registration is applied to deform the liver in the CECT image to fit that obtained in the nCECT image. The final liver segmentation is obtained by segmenting the liver in the deformed CECT image using nnU-Net. In addition, the lung segmentations are obtained using 2D ResUnet. Finally, LSF quantitation is performed based on the number of counts in the SPECT image inside the segmentations. Evaluations and Results: To evaluate the liver segmentation accuracy, we used Dice similarity coefficient (DSC), asymmetric surface distance (ASSD), and max surface distance (MSD) and compared the proposed method to five well-known CNN-based methods for liver segmentation. Furthermore, the LSF error obtained by the proposed method was compared to a state-of-the-art method, modified Deepmedic, and the LSF quantifications obtained by manual segmentation. The results show that the proposed method achieved a DSC score for the liver segmentation that is comparable to other state-of-the-art methods, with an average of 0.93, and the highest consistency in segmentation accuracy, yielding a standard deviation of the DSC score of 0.01. The proposed method also obtains the lowest ASSD and MSD scores on average (2.6 mm and 31.5 mm, respectively). Moreover, for the proposed method, a median LSF error of 0.14% is obtained, which is a statically significant improvement to the state-of-the-art-method (p=0.004), and is much smaller than the median error in LSF manual determination by the medical experts using 2D planar image (1.74% and p<0.001). CONCLUSIONS A method for LSF quantification using 3D SPECT/CT images based on CNNs and non-rigid registration was proposed, evaluated and compared to state-of-the-art techniques. The proposed method can quantitatively determine the LSF with high accuracy and has the potential to be applied in clinical practice.
Collapse
Affiliation(s)
- Manh Ha Luu
- AVITECH, VNU University of Engineering and Technology, Hanoi, Vietnam; FET, VNU University of Engineering and Technology, Hanoi, Vietnam; Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, the Netherlands.
| | - Hong Son Mai
- Department of Nuclear Medicine, Hospital 108, Hanoi, Vietnam
| | - Xuan Loc Pham
- FET, VNU University of Engineering and Technology, Hanoi, Vietnam
| | - Quoc Anh Le
- AVITECH, VNU University of Engineering and Technology, Hanoi, Vietnam
| | - Quoc Khanh Le
- Department of Nuclear Medicine, Hospital 108, Hanoi, Vietnam
| | - Theo van Walsum
- Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, the Netherlands
| | - Ngoc Ha Le
- Department of Nuclear Medicine, Hospital 108, Hanoi, Vietnam
| | - Daniel Franklin
- School of Electrical and Data Engineering, University of Technology Sydney, Sydney, Australia
| | - Vu Ha Le
- AVITECH, VNU University of Engineering and Technology, Hanoi, Vietnam; FET, VNU University of Engineering and Technology, Hanoi, Vietnam
| | - Adriaan Moelker
- Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, the Netherlands
| | - Duc Trinh Chu
- FET, VNU University of Engineering and Technology, Hanoi, Vietnam
| | - Nguyen Linh Trung
- AVITECH, VNU University of Engineering and Technology, Hanoi, Vietnam
| |
Collapse
|
41
|
Yu Z, Rahman MA, Abbey CK, Siegel BA, Jha AK. Development and task-based evaluation of a scatter-window projection and deep learning-based transmission-less attenuation compensation method for myocardial perfusion SPECT. ARXIV 2023:arXiv:2303.00197v2. [PMID: 36911280 PMCID: PMC10002798] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 03/14/2023]
Abstract
Attenuation compensation (AC) is beneficial for visual interpretation tasks in single-photon emission computed tomography (SPECT) myocardial perfusion imaging (MPI). However, traditional AC methods require the availability of a transmission scan, most often a CT scan. This approach has the disadvantages of increased radiation dose, increased scanner cost, and the possibility of inaccurate diagnosis in cases of misregistration between the SPECT and CT images. Further, many SPECT systems do not include a CT component. To address these issues, we developed a Scatter-window projection and deep Learning-based AC (SLAC) method to perform AC without a separate transmission scan. To investigate the clinical efficacy of this method, we then objectively evaluated the performance of this method on the clinical task of detecting perfusion defects on MPI in a retrospective study with anonymized clinical SPECT/CT stress MPI images. The proposed method was compared with CT-based AC (CTAC) and no-AC (NAC) methods. Our results showed that the SLAC method yielded an almost overlapping receiver operating characteristic (ROC) plot and a similar area under the ROC (AUC) to the CTAC method on this task. These results demonstrate the capability of the SLAC method for transmission-less AC in SPECT and motivate further clinical evaluation.
Collapse
Affiliation(s)
- Zitong Yu
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, USA
| | - Md Ashequr Rahman
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, USA
| | - Craig K. Abbey
- Department of Psychological & Brain Sciences, University of California Santa Barbara, Santa Barbara, USA
| | - Barry A. Siegel
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, USA
| | - Abhinav K. Jha
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, USA
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, USA
| |
Collapse
|
42
|
Podobnik G, Strojan P, Peterlin P, Ibragimov B, Vrtovec T. HaN-Seg: The head and neck organ-at-risk CT and MR segmentation dataset. Med Phys 2023; 50:1917-1927. [PMID: 36594372 DOI: 10.1002/mp.16197] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2022] [Revised: 11/17/2022] [Accepted: 12/07/2022] [Indexed: 01/04/2023] Open
Abstract
PURPOSE For the cancer in the head and neck (HaN), radiotherapy (RT) represents an important treatment modality. Segmentation of organs-at-risk (OARs) is the starting point of RT planning, however, existing approaches are focused on either computed tomography (CT) or magnetic resonance (MR) images, while multimodal segmentation has not been thoroughly explored yet. We present a dataset of CT and MR images of the same patients with curated reference HaN OAR segmentations for an objective evaluation of segmentation methods. ACQUISITION AND VALIDATION METHODS The cohort consists of HaN images of 56 patients that underwent both CT and T1-weighted MR imaging for image-guided RT. For each patient, reference segmentations of up to 30 OARs were obtained by experts performing manual pixel-wise image annotation. By maintaining the distribution of patient age and gender, and annotation type, the patients were randomly split into training Set 1 (42 cases or 75%) and test Set 2 (14 cases or 25%). Baseline auto-segmentation results are also provided by training the publicly available deep nnU-Net architecture on Set 1, and evaluating its performance on Set 2. DATA FORMAT AND USAGE NOTES The data are publicly available through an open-access repository under the name HaN-Seg: The Head and Neck Organ-at-Risk CT & MR Segmentation Dataset. Images and reference segmentations are stored in the NRRD file format, where the OAR filenames correspond to the nomenclature recommended by the American Association of Physicists in Medicine, and OAR and demographics information is stored in separate comma-separated value files. POTENTIAL APPLICATIONS The HaN-Seg: The Head and Neck Organ-at-Risk CT & MR Segmentation Challenge is launched in parallel with the dataset release to promote the development of automated techniques for OAR segmentation in the HaN. Other potential applications include out-of-challenge algorithm development and benchmarking, as well as external validation of the developed algorithms.
Collapse
Affiliation(s)
- Gašper Podobnik
- Faculty Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia
| | | | | | - Bulat Ibragimov
- Faculty Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia
- Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
| | - Tomaž Vrtovec
- Faculty Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia
| |
Collapse
|
43
|
Koike Y, Takegawa H, Anetai Y, Ohira S, Nakamura S, Tanigawa N. Patient-specific three-dimensional dose distribution prediction via deep learning for prostate cancer therapy: Improvement with the structure loss. Phys Med 2023; 107:102544. [PMID: 36774846 DOI: 10.1016/j.ejmp.2023.102544] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 01/18/2023] [Accepted: 02/03/2023] [Indexed: 02/12/2023] Open
Abstract
PURPOSE Deep learning (DL)-based dose distribution prediction can potentially reduce the cost of inverse planning process. We developed and introduced a structure-focused loss (Lstruct) for 3D dose prediction to improve prediction accuracy. This study investigated the influence of Lstruct on DL-based dose prediction for patients with prostate cancer. The proposed Lstruct, which is similar in concept to dose-volume histogram (DVH)-based optimization in clinical practice, has the potential to provide more interpretable and accurate DL-based optimization. METHODS This study involved 104 patients who underwent prostate radiotherapy. We used 3D U-Net-based architecture to predict dose distributions from computed tomography and contours of the planning target volume and organs-at-risk. We trained two models using different loss functions: L2 loss and Lstruct. Predicted doses were compared in terms of dose-volume parameters and the Dice similarity coefficient of isodose volume. RESULTS DVH analysis showed that the Lstruct model had smaller errors from the ground truth than the L2 model. The Lstruct model achieved more consistent dose distributions than the L2 model, with errors close to zero. The isodose Dice score of the Lstruct model was greater than that of the L2 model by >20% of the prescribed dose. CONCLUSIONS We developed Lstruct using labels of inputted contours for DL-based dose prediction for prostate radiotherapy. Lstruct can be generalized to any DL architecture, thereby enhancing the dose prediction accuracy.
Collapse
Affiliation(s)
- Yuhei Koike
- Department of Radiology, Kansai Medical University, 2-5-1 Shinmachi, Hirakata, Osaka 573-1010, Japan.
| | - Hideki Takegawa
- Department of Radiology, Kansai Medical University, 2-5-1 Shinmachi, Hirakata, Osaka 573-1010, Japan
| | - Yusuke Anetai
- Department of Radiology, Kansai Medical University, 2-5-1 Shinmachi, Hirakata, Osaka 573-1010, Japan
| | - Shingo Ohira
- Department of Radiation Oncology, Osaka International Cancer Institute, 3-1-69 Otemae, Chuo-ku, Osaka 537-8567, Japan
| | - Satoaki Nakamura
- Department of Radiology, Kansai Medical University, 2-5-1 Shinmachi, Hirakata, Osaka 573-1010, Japan
| | - Noboru Tanigawa
- Department of Radiology, Kansai Medical University, 2-5-1 Shinmachi, Hirakata, Osaka 573-1010, Japan
| |
Collapse
|
44
|
Shen J, Gu P, Wang Y, Wang Z. Advances in automatic delineation of target volume and cardiac substructure in breast cancer radiotherapy (Review). Oncol Lett 2023; 25:110. [PMID: 36817059 PMCID: PMC9932716 DOI: 10.3892/ol.2023.13697] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Accepted: 01/06/2023] [Indexed: 02/05/2023] Open
Abstract
Postoperative adjuvant radiotherapy plays an important role in the treatment of patients with breast cancer. With the continuous development of radiotherapeutic technologies, the requirements for radiotherapeutic accuracy are increasingly high. The accuracy of target volume and organ at risk delineation significantly affects the effect of radiotherapy. Automatic delineation software has been continuously developed for the automatic delineation of target areas and organs at risk. Automatic segmentation based on an atlas and deep learning is a hot topic in current clinical research. Automatic delineation can not only reduce the workload and delineation times, but also establish a uniform delineation standard and reduce inter-observer and intra-observer differences. In patients with breast cancer, especially in patients who undergo left breast radiotherapy, the protection of the heart is particularly important. Treating the whole heart as an organ at risk cannot meet the clinical needs, and it is necessary to limit the dose to specific cardiac substructures. The present review discusses the importance of automatic delineation of target volume and cardiac substructure in radiotherapy for patients with breast cancer.
Collapse
Affiliation(s)
- Jingjing Shen
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai 200438, P.R. China
| | - Peihua Gu
- Department of Oncology and Radiotherapy, Shidong Hospital Affiliated to University of Shanghai for Science and Technology, Shanghai 200438, P.R. China
| | - Yun Wang
- Department of Oncology and Radiotherapy, Shidong Hospital Affiliated to University of Shanghai for Science and Technology, Shanghai 200438, P.R. China
| | - Zhongming Wang
- Department of Oncology and Radiotherapy, Shidong Hospital Affiliated to University of Shanghai for Science and Technology, Shanghai 200438, P.R. China,Correspondence to: Dr Zhongming Wang, Department of Oncology and Radiotherapy, Shidong Hospital Affiliated to University of Shanghai for Science and Technology, 999 Shiguang Road, Shanghai 200438, P.R. China, E-mail:
| |
Collapse
|
45
|
Yu Z, Rahman MA, Abbey CK, Siegel BA, Jha AK. Development and task-based evaluation of a scatter-window projection and deep learning-based transmission-less attenuation compensation method for myocardial perfusion SPECT. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12463:124631E. [PMID: 37274423 PMCID: PMC10238080 DOI: 10.1117/12.2654500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
Attenuation compensation (AC) is beneficial for visual interpretation tasks in single-photon emission computed tomography (SPECT) myocardial perfusion imaging (MPI). However, traditional AC methods require the availability of a transmission scan, most often a CT scan. This approach has the disadvantage of increased radiation dose, increased scanner costs, and the possibility of inaccurate diagnosis in cases of misregistration between the SPECT and CT images. Further, many SPECT systems do not include a CT component. To address these issues, we developed a Scatter-window projection and deep Learning-based AC (SLAC) method to perform AC without a separate transmission scan. To investigate the clinical efficacy of this method, we then objectively evaluated the performance of this method on the clinical task of detecting perfusion defects on MPI in a retrospective study with anonymized clinical SPECT/CT stress MPI images. The proposed method was compared with CT-based AC (CTAC) and no-AC (NAC) methods. Our results showed that the SLAC method yielded an almost overlapping receiver operating characteristic (ROC) plot and a similar area under the ROC (AUC) to the CTAC method on this task. These results demonstrate the capability of the SLAC method for transmission-less AC in SPECT and motivate further clinical evaluation.
Collapse
Affiliation(s)
- Zitong Yu
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, USA
| | - Md Ashequr Rahman
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, USA
| | - Craig K. Abbey
- Department of Psychological & Brain Sciences, University of California Santa Barbara, Santa Barbara, USA
| | - Barry A. Siegel
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, USA
| | - Abhinav K. Jha
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, USA
- Mallinckrodt Institute of Radiology, Washington University in St. Louis, St. Louis, USA
| |
Collapse
|
46
|
Wang J, Zhao H, Liang W, Wang S, Zhang Y. Cross-convolutional transformer for automated multi-organs segmentation in a variety of medical images. Phys Med Biol 2023; 68. [PMID: 36623323 DOI: 10.1088/1361-6560/acb19a] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2022] [Accepted: 01/09/2023] [Indexed: 01/11/2023]
Abstract
Objective.It is a huge challenge for multi-organs segmentation in various medical images based on a consistent algorithm with the development of deep learning methods. We therefore develop a deep learning method based on cross-convolutional transformer for these automated- segmentation to obtain better generalization and accuracy.Approach.We propose a cross-convolutional transformer network (C2Former) to solve the segmentation problem. Specifically, we first redesign a novel cross-convolutional self-attention mechanism in terms of the algorithm to integrate local and global contexts and model long-distance and short-distance dependencies to enhance the semantic feature understanding of images. Then multi-scale feature edge fusion module is proposed to combine the image edge features, which effectively form multi-scale feature streams and establish reliable relational connections in the global context. Finally, we use three different modalities, imaging three different anatomical regions to train and test multi organs and evaluate segmentation performance.Main results.We use the evaluation metrics of Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) for each dataset. Experiments showed the average DSC of 83.22% and HD95 of 17.55 mm on the Synapse dataset (CT images of abdominal multi-organ), the average DSC of 91.42% and HD95 of 1.06 mm on the ACDC dataset (MRI of cardiac substructures) and the average DSC of 86.78% and HD95 of 16.85 mm on the ISIC 2017 dataset (skin cancer images). In each dataset, our proposed method consistently outperforms the compared networks.Significance.The proposed deep learning network provides a generalized and accurate solution method for multi-organ segmentation in the three different datasets. It has the potential to be applied to a variety of medical datasets for structural segmentation.
Collapse
Affiliation(s)
- Jing Wang
- School of Information Science and Engineering Department, Shandong University, 72 Binghai Road, Jimo, Qingdao, Shandong, People's Republic of China
| | - Haiyue Zhao
- Shandong Youth University of Political Science, No.31699 Jing Shi East Road, Li Xia District, Jinan, Shandong, People's Republic of China
| | - Wei Liang
- Department of ecological environment of Shandong, No.3377 Jing Shi Road, Jinan, People's Republic of China
| | - Shuyu Wang
- Shandong Youth University of Political Science, No.31699 Jing Shi East Road, Li Xia District, Jinan, Shandong, People's Republic of China
| | - Yan Zhang
- Shandong Mental Health Center, No.49 Wen Hua Dong Road, Li Xia District, Jinan, Shandong, People's Republic of China
| |
Collapse
|
47
|
Fan X, Zhu Q, Tu P, Joskowicz L, Chen X. A review of advances in image-guided orthopedic surgery. Phys Med Biol 2023; 68. [PMID: 36595258 DOI: 10.1088/1361-6560/acaae9] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2022] [Accepted: 12/12/2022] [Indexed: 12/15/2022]
Abstract
Orthopedic surgery remains technically demanding due to the complex anatomical structures and cumbersome surgical procedures. The introduction of image-guided orthopedic surgery (IGOS) has significantly decreased the surgical risk and improved the operation results. This review focuses on the application of recent advances in artificial intelligence (AI), deep learning (DL), augmented reality (AR) and robotics in image-guided spine surgery, joint arthroplasty, fracture reduction and bone tumor resection. For the pre-operative stage, key technologies of AI and DL based medical image segmentation, 3D visualization and surgical planning procedures are systematically reviewed. For the intra-operative stage, the development of novel image registration, surgical tool calibration and real-time navigation are reviewed. Furthermore, the combination of the surgical navigation system with AR and robotic technology is also discussed. Finally, the current issues and prospects of the IGOS system are discussed, with the goal of establishing a reference and providing guidance for surgeons, engineers, and researchers involved in the research and development of this area.
Collapse
Affiliation(s)
- Xingqi Fan
- Institute of Biomedical Manufacturing and Life Quality Engineering, State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Qiyang Zhu
- Institute of Biomedical Manufacturing and Life Quality Engineering, State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Puxun Tu
- Institute of Biomedical Manufacturing and Life Quality Engineering, State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Leo Joskowicz
- School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel
| | - Xiaojun Chen
- Institute of Biomedical Manufacturing and Life Quality Engineering, State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China.,Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| |
Collapse
|
48
|
Shan T, Yan J, Cui X, Xie L. DSCA-Net: A depthwise separable convolutional neural network with attention mechanism for medical image segmentation. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2023; 20:365-382. [PMID: 36650770 DOI: 10.3934/mbe.2023017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
Accurate segmentation is a basic and crucial step for medical image processing and analysis. In the last few years, U-Net, and its variants, have become widely adopted models in medical image segmentation tasks. However, the multiple training parameters of these models determines high computation complexity, which is impractical for further applications. In this paper, by introducing depthwise separable convolution and attention mechanism into U-shaped architecture, we propose a novel lightweight neural network (DSCA-Net) for medical image segmentation. Three attention modules are created to improve its segmentation performance. Firstly, Pooling Attention (PA) module is utilized to reduce the loss of consecutive down-sampling operations. Secondly, for capturing critical context information, based on attention mechanism and convolution operation, we propose Context Attention (CA) module instead of concatenation operations. Finally, Multiscale Edge Attention (MEA) module is used to emphasize multi-level representative scale edge features for final prediction. The number of parameters in our network is 2.2 M, which is 71.6% less than U-Net. Experiment results across four public datasets show the potential and the dice coefficients are improved by 5.49% for ISIC 2018, 4.28% for thyroid, 1.61% for lung and 9.31% for nuclei compared with U-Net.
Collapse
Affiliation(s)
- Tong Shan
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China
| | - Jiayong Yan
- School of Medical Instruments, Shanghai University of Medicine and Health Sciences, Shanghai 201318, China
- Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Xiaoyao Cui
- Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China
| | - Lijian Xie
- Children's Hospital of Shanghai, Shanghai 200062, China
| |
Collapse
|
49
|
Lin D, Wang Z, Li H, Zhang H, Deng L, Ren H, Sun S, Zheng F, Zhou J, Wang M. Automated Measurement of Pancreatic Fat Deposition on Dixon MRI Using nnU-Net. J Magn Reson Imaging 2023; 57:296-307. [PMID: 35635494 DOI: 10.1002/jmri.28275] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2022] [Revised: 05/12/2022] [Accepted: 05/13/2022] [Indexed: 02/04/2023] Open
Abstract
BACKGROUND Pancreatic fat accumulation may cause or aggravate the process of acute pancreatitis, β-cell dysfunction, T2DM disease, and even be associated with pancreatic tumors. The pathophysiology of fatty pancreas remains overlooked and lacks effective imaging diagnostics. PURPOSE To automatically measure the distribution of pancreatic fat deposition on Dixon MRI in multicenter/population datasets using nnU-Net models. STUDY TYPE Retrospective. POPULATION A total of 176 obese/nonobese subjects (90 males, 86 females; mean age, 27.2 ± 19.7) were enrolled, including a training set (N = 132) and a testing set (N = 44). FIELD STRENGTH/SEQUENCE A 3 T and 1.5 T/gradient echo T1 dual-echo Dixon. ASSESSMENT The segmentation results of four types of nnU-Net models were compared using dice similarity coefficient (DSC), positive predicted value (PPV), and sensitivity. The ground truth was the manual delineation by two radiologists according to in-phase (IP) and opposed-phase (OP) images. STATISTICAL TESTS The group difference of segmentation results of four models were assessed by the Kruskal-Wallis H test with Dunn-Bonferroni comparisons. The interobserver agreement of pancreatic fat fraction measurements across three observers and test-retest reliability of human and machine were assessed by intragroup correlation coefficient (ICC). P < 0.05 was considered statistically significant. RESULTS The three-dimensional (3D) dual-contrast model had significantly improved performance than 2D dual-contrast (DSC/sensitivity) and 3D one-contrast (IP) models (DSC/PPV/sensitivity) and had less errors than 3D one-contrast (OP) model according to higher DSC and PPV (not significant), with a mean DSC of 0.9158, PPV of 0.9105 and sensitivity of 0.9232 in the testing set. The test-retest ICC of this model was above 0.900 in all pancreatic regions, exceeded human. DATA CONCLUSION 3D Dual-contrast nnU-Net aided segmentation of pancreas on Dixon images appears to be adaptable to multicenter/population datasets. It fully automates the assessment of pancreatic fat distribution and has high reliability. EVIDENCE LEVEL 3 TECHNICAL EFFICACY: Stage 3.
Collapse
Affiliation(s)
- Dingyi Lin
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China
| | - Ziyan Wang
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China
| | - Hong Li
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Hongxi Zhang
- School of Medicine, Children's Hospital Binjiang Campus, Department of Radiology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Liping Deng
- School of Medicine, Sir Run Run Shaw Hospital, Department of Radiology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Hong Ren
- School of Medicine, Sir Run Run Shaw Hospital, Department of Radiology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Shuiya Sun
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Fenping Zheng
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Jiaqiang Zhou
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Min Wang
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China.,School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| |
Collapse
|
50
|
Leveraging image complexity in macro-level neural network design for medical image segmentation. Sci Rep 2022; 12:22286. [PMID: 36566313 PMCID: PMC9790020 DOI: 10.1038/s41598-022-26482-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Accepted: 12/15/2022] [Indexed: 12/25/2022] Open
Abstract
Recent progress in encoder-decoder neural network architecture design has led to significant performance improvements in a wide range of medical image segmentation tasks. However, state-of-the-art networks for a given task may be too computationally demanding to run on affordable hardware, and thus users often resort to practical workarounds by modifying various macro-level design aspects. Two common examples are downsampling of the input images and reducing the network depth or size to meet computer memory constraints. In this paper, we investigate the effects of these changes on segmentation performance and show that image complexity can be used as a guideline in choosing what is best for a given dataset. We consider four statistical measures to quantify image complexity and evaluate their suitability on ten different public datasets. For the purpose of our illustrative experiments, we use DeepLabV3+ (deep large-size), M2U-Net (deep lightweight), U-Net (shallow large-size), and U-Net Lite (shallow lightweight). Our results suggest that median frequency is the best complexity measure when deciding on an acceptable input downsampling factor and using a deep versus shallow, large-size versus lightweight network. For high-complexity datasets, a lightweight network running on the original images may yield better segmentation results than a large-size network running on downsampled images, whereas the opposite may be the case for low-complexity images.
Collapse
|