1
|
Meng Y, Zhang Y, Xie J, Duan J, Joddrell M, Madhusudhan S, Peto T, Zhao Y, Zheng Y. Multi-granularity learning of explicit geometric constraint and contrast for label-efficient medical image segmentation and differentiable clinical function assessment. Med Image Anal 2024; 95:103183. [PMID: 38692098 DOI: 10.1016/j.media.2024.103183] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2023] [Revised: 01/26/2024] [Accepted: 04/18/2024] [Indexed: 05/03/2024]
Abstract
Automated segmentation is a challenging task in medical image analysis that usually requires a large amount of manually labeled data. However, most current supervised learning based algorithms suffer from insufficient manual annotations, posing a significant difficulty for accurate and robust segmentation. In addition, most current semi-supervised methods lack explicit representations of geometric structure and semantic information, restricting segmentation accuracy. In this work, we propose a hybrid framework to learn polygon vertices, region masks, and their boundaries in a weakly/semi-supervised manner that significantly advances geometric and semantic representations. Firstly, we propose multi-granularity learning of explicit geometric structure constraints via polygon vertices (PolyV) and pixel-wise region (PixelR) segmentation masks in a semi-supervised manner. Secondly, we propose eliminating boundary ambiguity by using an explicit contrastive objective to learn a discriminative feature space of boundary contours at the pixel level with limited annotations. Thirdly, we exploit the task-specific clinical domain knowledge to differentiate the clinical function assessment end-to-end. The ground truth of clinical function assessment, on the other hand, can serve as auxiliary weak supervision for PolyV and PixelR learning. We evaluate the proposed framework on two tasks, including optic disc (OD) and cup (OC) segmentation along with vertical cup-to-disc ratio (vCDR) estimation in fundus images; left ventricle (LV) segmentation at end-diastolic and end-systolic frames along with ejection fraction (LVEF) estimation in two-dimensional echocardiography images. Experiments on nine large-scale datasets of the two tasks under different label settings demonstrate our model's superior performance on segmentation and clinical function assessment.
Collapse
Affiliation(s)
- Yanda Meng
- Department of Eye and Vision Sciences, University of Liverpool, Liverpool, United Kingdom
| | - Yuchen Zhang
- Center for Bioinformatics, Peking University, Beijing, China
| | - Jianyang Xie
- Department of Eye and Vision Sciences, University of Liverpool, Liverpool, United Kingdom
| | - Jinming Duan
- School of Computer Science, University of Birmingham, Birmingham, United Kingdom
| | - Martha Joddrell
- Liverpool Centre for Cardiovascular Science, University of Liverpool and Liverpool Heart & Chest Hospital, Liverpool, United Kingdom; Department of Cardiovascular and Metabolic Medicine, University of Liverpool, Liverpool, United Kingdom
| | - Savita Madhusudhan
- St Paul's Eye Unit, Liverpool University Hospitals NHS Foundation Trust, Liverpool, United Kingdom
| | - Tunde Peto
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, United Kingdom
| | - Yitian Zhao
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Science, Ningbo, China; Ningbo Eye Hospital, Ningbo, China.
| | - Yalin Zheng
- Department of Eye and Vision Sciences, University of Liverpool, Liverpool, United Kingdom; Liverpool Centre for Cardiovascular Science, University of Liverpool and Liverpool Heart & Chest Hospital, Liverpool, United Kingdom.
| |
Collapse
|
2
|
Li H, Liu D, Zeng Y, Liu S, Gan T, Rao N, Yang J, Zeng B. Single-Image-Based Deep Learning for Segmentation of Early Esophageal Cancer Lesions. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:2676-2688. [PMID: 38530733 DOI: 10.1109/tip.2024.3379902] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/28/2024]
Abstract
Accurate segmentation of lesions is crucial for diagnosis and treatment of early esophageal cancer (EEC). However, neither traditional nor deep learning-based methods up to today can meet the clinical requirements, with the mean Dice score - the most important metric in medical image analysis - hardly exceeding 0.75. In this paper, we present a novel deep learning approach for segmenting EEC lesions. Our method stands out for its uniqueness, as it relies solely on a single input image from a patient, forming the so-called "You-Only-Have-One" (YOHO) framework. On one hand, this "one-image-one-network" learning ensures complete patient privacy as it does not use any images from other patients as the training data. On the other hand, it avoids nearly all generalization-related problems since each trained network is applied only to the same input image itself. In particular, we can push the training to "over-fitting" as much as possible to increase the segmentation accuracy. Our technical details include an interaction with clinical doctors to utilize their expertise, a geometry-based data augmentation over a single lesion image to generate the training dataset (the biggest novelty), and an edge-enhanced UNet. We have evaluated YOHO over an EEC dataset collected by ourselves and achieved a mean Dice score of 0.888, which is much higher as compared to the existing deep-learning methods, thus representing a significant advance toward clinical applications. The code and dataset are available at: https://github.com/lhaippp/YOHO.
Collapse
|
3
|
Yap BP, Ng BK. Coarse-to-fine visual representation learning for medical images via class activation maps. Comput Biol Med 2024; 171:108203. [PMID: 38430741 DOI: 10.1016/j.compbiomed.2024.108203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Revised: 01/29/2024] [Accepted: 02/19/2024] [Indexed: 03/05/2024]
Abstract
The value of coarsely labeled datasets in learning transferable representations for medical images is investigated in this work. Compared to fine labels which require meticulous effort to annotate, coarse labels can be acquired at a significantly lower cost and can provide useful training signals for data-hungry deep neural networks. We consider coarse labels in the form of binary labels differentiating a normal (healthy) image from an abnormal (diseased) image and propose CAMContrast, a two-stage representation learning framework for medical images. Using class activation maps, CAMContrast makes use of the binary labels to generate heatmaps as positive views for contrastive representation learning. Specifically, the learning objective is optimized to maximize the agreement within fixed crops of image-heatmap pair to learn fine-grained representations that are generalizable to different downstream tasks. We empirically validate the transfer learning performance of CAMContrast on several public datasets, covering classification and segmentation tasks on fundus photographs and chest X-ray images. The experimental results showed that our method outperforms other self-supervised and supervised pretrain methods in terms of data efficiency and downstream performance.
Collapse
Affiliation(s)
- Boon Peng Yap
- School of Electrical and Electronic Engineering, Nanyang Technological University, 50 Nanyang Ave, 639798, Singapore; Centre for OptoElectronics and Biophotonics, Nanyang Technological University, 50 Nanyang Ave, 639798, Singapore.
| | - Beng Koon Ng
- School of Electrical and Electronic Engineering, Nanyang Technological University, 50 Nanyang Ave, 639798, Singapore; Centre for OptoElectronics and Biophotonics, Nanyang Technological University, 50 Nanyang Ave, 639798, Singapore.
| |
Collapse
|
4
|
Ilyas T, Ahmad K, Arsa DMS, Jeong YC, Kim H. Enhancing medical image analysis with unsupervised domain adaptation approach across microscopes and magnifications. Comput Biol Med 2024; 170:108055. [PMID: 38295480 DOI: 10.1016/j.compbiomed.2024.108055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Revised: 01/05/2024] [Accepted: 01/26/2024] [Indexed: 02/02/2024]
Abstract
In the domain of medical image analysis, deep learning models are heralding a revolution, especially in detecting complex and nuanced features characteristic of diseases like tumors and cancers. However, the robustness and adaptability of these models across varied imaging conditions and magnifications remain a formidable challenge. This paper introduces the Fourier Adaptive Recognition System (FARS), a pioneering model primarily engineered to address adaptability in malarial parasite recognition. Yet, the foundational principles guiding FARS lend themselves seamlessly to broader applications, including tumor and cancer diagnostics. FARS capitalizes on the untapped potential of transitioning from bounding box labels to richer semantic segmentation labels, enabling a more refined examination of microscopy slides. With the integration of adversarial training and the Color Domain Aware Fourier Domain Adaptation (F2DA), the model ensures consistent feature extraction across diverse microscopy configurations. The further inclusion of category-dependent context attention amplifies FARS's cross-domain versatility. Evidenced by a substantial elevation in cross-magnification performance from 31.3% mAP to 55.19% mAP and a 15.68% boost in cross-domain adaptability, FARS positions itself as a significant advancement in malarial parasite recognition. Furthermore, the core methodologies of FARS can serve as a blueprint for enhancing precision in other realms of medical image analysis, especially in the complex terrains of tumor and cancer imaging. The code is available at; https://github.com/Mr-TalhaIlyas/FARS.
Collapse
Affiliation(s)
- Talha Ilyas
- Division of Electronics and Information Engineering, Jeonbuk National University, Jeonju, 54896, Republic of Korea; Core Research Institute of Intelligent Robots, Jeonbuk National University, Jeonju, 54896, Republic of Korea.
| | - Khubaib Ahmad
- Division of Electronics and Information Engineering, Jeonbuk National University, Jeonju, 54896, Republic of Korea; Core Research Institute of Intelligent Robots, Jeonbuk National University, Jeonju, 54896, Republic of Korea
| | - Dewa Made Sri Arsa
- Division of Electronics and Information Engineering, Jeonbuk National University, Jeonju, 54896, Republic of Korea; Core Research Institute of Intelligent Robots, Jeonbuk National University, Jeonju, 54896, Republic of Korea; Department of Information Technology, Universitas Udayana, Bali, 80361, Indonesia
| | - Yong Chae Jeong
- Core Research Institute of Intelligent Robots, Jeonbuk National University, Jeonju, 54896, Republic of Korea; Division of Electronics Engineering, Jeonbuk National University, Jeonju, 54896, Republic of Korea
| | - Hyongsuk Kim
- Division of Electronics and Information Engineering, Jeonbuk National University, Jeonju, 54896, Republic of Korea; Core Research Institute of Intelligent Robots, Jeonbuk National University, Jeonju, 54896, Republic of Korea.
| |
Collapse
|
5
|
Wang J, Xia B. Weakly supervised image segmentation beyond tight bounding box annotations. Comput Biol Med 2024; 169:107913. [PMID: 38176213 DOI: 10.1016/j.compbiomed.2023.107913] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Revised: 11/21/2023] [Accepted: 12/24/2023] [Indexed: 01/06/2024]
Abstract
Weakly supervised image segmentation approaches in the literature usually achieve high segmentation performance using tight bounding box supervision and decrease the performance greatly when supervised by loose bounding boxes. However, compared with loose bounding box, it is much more difficult to acquire tight bounding box due to its strict requirements on the precise locations of the four sides of the box. To resolve this issue, this study investigates whether it is possible to maintain good segmentation performance when loose bounding boxes are used as supervision. For this purpose, this work extends our previous parallel transformation based multiple instance learning (MIL) for tight bounding box supervision by integrating an MIL strategy based on polar transformation to assist image segmentation. The proposed polar transformation based MIL formulation works for both tight and loose bounding boxes, in which a positive bag is defined as pixels in a polar line of a bounding box with one endpoint located inside the object enclosed by the box and the other endpoint located at one of the four sides of the box. Moreover, a weighted smooth maximum approximation is introduced to incorporate the observation that pixels closer to the origin of the polar transformation are more likely to belong to the object in the box. The proposed approach was evaluated on two public datasets using dice coefficient when bounding boxes at different precision levels were considered in the experiments. The results demonstrate that the proposed approach achieves state-of-the-art performance for bounding boxes at all precision levels and is robust to mild and moderate errors in the loose bounding box annotations. The codes are available at https://github.com/wangjuan313/wsis-beyond-tightBB.
Collapse
Affiliation(s)
- Juan Wang
- Horizon Med Innovation Inc., 23421 South Pointe Dr., Laguna Hills, CA 92653, USA.
| | - Bin Xia
- Shenzhen SiBright Co. Ltd., Tinwe Industrial Park, No. 6 Liufang Rd., Shenzhen, Guangdong 518052, China.
| |
Collapse
|
6
|
Chen J, Lu R, Ye S, Guang M, Tassew TM, Jing B, Zhang G, Chen G, Shen D. Image Recovery Matters: A Recovery-Extraction Framework for Robust Fetal Brain Extraction From MR Images. IEEE J Biomed Health Inform 2024; 28:823-834. [PMID: 37995170 DOI: 10.1109/jbhi.2023.3333953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2023]
Abstract
The extraction of the fetal brain from magnetic resonance (MR) images is a challenging task. In particular, fetal MR images suffer from different kinds of artifacts introduced during the image acquisition. Among those artifacts, intensity inhomogeneity is a common one affecting brain extraction. In this work, we propose a deep learning-based recovery-extraction framework for fetal brain extraction, which is particularly effective in handling fetal MR images with intensity inhomogeneity. Our framework involves two stages. First, the artifact-corrupted images are recovered with the proposed generative adversarial learning-based image recovery network with a novel region-of-darkness discriminator that enforces the network focusing on artifacts of the images. Second, we propose a brain extraction network for more effective fetal brain segmentation by strengthening the association between lower- and higher-level features as well as suppressing task-irrelevant features. Thanks to the proposed recovery-extraction strategy, our framework is able to accurately segment fetal brains from artifact-corrupted MR images. The experiments show that our framework achieves promising performance in both quantitative and qualitative evaluations, and outperforms state-of-the-art methods in both image recovery and fetal brain extraction.
Collapse
|
7
|
Ji T, Li J, Fang H, Zhang R, Yang J, Fan L. Rapid dataset generation methods for stacked construction solid waste based on machine vision and deep learning. PLoS One 2024; 19:e0296666. [PMID: 38227593 PMCID: PMC10790992 DOI: 10.1371/journal.pone.0296666] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2023] [Accepted: 12/13/2023] [Indexed: 01/18/2024] Open
Abstract
The development of urbanization has brought convenience to people, but it has also brought a lot of harmful construction solid waste. The machine vision detection algorithm is the crucial technology for finely sorting solid waste, which is faster and more stable than traditional methods. However, accurate identification relies on large datasets, while the datasets from the field working conditions are scarce, and the manual annotation cost of datasets is high. To rapidly and automatically generate datasets for stacked construction waste, an acquisition and detection platform was built to automatically collect different groups of RGB-D images for instances labeling. Then, based on the distribution points generation theory and data augmentation algorithm, a rapid-generation method for synthetic construction solid waste datasets was proposed. Additionally, two automatic annotation methods for real stacked construction solid waste datasets based on semi-supervised self-training and RGB-D fusion edge detection were proposed, and datasets under real-world conditions yield better models training results. Finally, two different working conditions were designed to validate these methods. Under the simple working condition, the generated dataset achieved an F1-score of 95.98, higher than 94.81 for the manually labeled dataset. In the complicated working condition, the F1-score obtained by the rapid generation method reached 97.74. In contrast, the F1-score of the dataset obtained manually labeled was only 85.97, which demonstrates the effectiveness of proposed approaches.
Collapse
Affiliation(s)
- Tianchen Ji
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen, Fujian, China
| | - Jiantao Li
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen, Fujian, China
| | - Huaiying Fang
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen, Fujian, China
| | - RenCheng Zhang
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen, Fujian, China
| | - Jianhong Yang
- College of Mechanical Engineering and Automation, Huaqiao University, Xiamen, Fujian, China
| | - Lulu Fan
- Shenzhen Municipal Engineering Corporation, Shenzhen, Guangdong, China
| |
Collapse
|
8
|
Sharma P, Nayak DR, Balabantaray BK, Tanveer M, Nayak R. A survey on cancer detection via convolutional neural networks: Current challenges and future directions. Neural Netw 2024; 169:637-659. [PMID: 37972509 DOI: 10.1016/j.neunet.2023.11.006] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Revised: 10/21/2023] [Accepted: 11/04/2023] [Indexed: 11/19/2023]
Abstract
Cancer is a condition in which abnormal cells uncontrollably split and damage the body tissues. Hence, detecting cancer at an early stage is highly essential. Currently, medical images play an indispensable role in detecting various cancers; however, manual interpretation of these images by radiologists is observer-dependent, time-consuming, and tedious. An automatic decision-making process is thus an essential need for cancer detection and diagnosis. This paper presents a comprehensive survey on automated cancer detection in various human body organs, namely, the breast, lung, liver, prostate, brain, skin, and colon, using convolutional neural networks (CNN) and medical imaging techniques. It also includes a brief discussion about deep learning based on state-of-the-art cancer detection methods, their outcomes, and the possible medical imaging data used. Eventually, the description of the dataset used for cancer detection, the limitations of the existing solutions, future trends, and challenges in this domain are discussed. The utmost goal of this paper is to provide a piece of comprehensive and insightful information to researchers who have a keen interest in developing CNN-based models for cancer detection.
Collapse
Affiliation(s)
- Pallabi Sharma
- School of Computer Science, UPES, Dehradun, 248007, Uttarakhand, India.
| | - Deepak Ranjan Nayak
- Department of Computer Science and Engineering, Malaviya National Institute of Technology, Jaipur, 302017, Rajasthan, India.
| | - Bunil Kumar Balabantaray
- Computer Science and Engineering, National Institute of Technology Meghalaya, Shillong, 793003, Meghalaya, India.
| | - M Tanveer
- Department of Mathematics, Indian Institute of Technology Indore, Simrol, 453552, Indore, India.
| | - Rajashree Nayak
- School of Applied Sciences, Birla Global University, Bhubaneswar, 751029, Odisha, India.
| |
Collapse
|
9
|
Zhuang M, Chen Z, Yang Y, Kettunen L, Wang H. Annotation-efficient training of medical image segmentation network based on scribble guidance in difficult areas. Int J Comput Assist Radiol Surg 2024; 19:87-96. [PMID: 37233894 DOI: 10.1007/s11548-023-02931-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2023] [Accepted: 04/19/2023] [Indexed: 05/27/2023]
Abstract
PURPOSE The training of deep medical image segmentation networks usually requires a large amount of human-annotated data. To alleviate the burden of human labor, many semi- or non-supervised methods have been developed. However, due to the complexity of clinical scenario, insufficient training labels still causes inaccurate segmentation in some difficult local areas such as heterogeneous tumors and fuzzy boundaries. METHODS We propose an annotation-efficient training approach, which only requires scribble guidance in the difficult areas. A segmentation network is initially trained with a small amount of fully annotated data and then used to produce pseudo labels for more training data. Human supervisors draw scribbles in the areas of incorrect pseudo labels (i.e., difficult areas), and the scribbles are converted into pseudo label maps using a probability-modulated geodesic transform. To reduce the influence of the potential errors in the pseudo labels, a confidence map of the pseudo labels is generated by jointly considering the pixel-to-scribble geodesic distance and the network output probability. The pseudo labels and confidence maps are iteratively optimized with the update of the network, and the network training is promoted by the pseudo labels and the confidence maps in turn. RESULTS Cross-validation based on two data sets (brain tumor MRI and liver tumor CT) showed that our method significantly reduces the annotation time while maintains the segmentation accuracy of difficult areas (e.g., tumors). Using 90 scribble-annotated training images (annotated time: ~ 9 h), our method achieved the same performance as using 45 fully annotated images (annotation time: > 100 h) but required much shorter annotation time. CONCLUSION Compared to the conventional full annotation approaches, the proposed method significantly saves the annotation efforts by focusing the human supervisions on the most difficult regions. It provides an annotation-efficient way for training medical image segmentation networks in complex clinical scenario.
Collapse
Affiliation(s)
- Mingrui Zhuang
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China
| | - Zhonghua Chen
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China
- Faculty of Information Technology, University of Jyväskylä, 40100, Jyvaskyla, Finland
| | - Yuxin Yang
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China
| | - Lauri Kettunen
- Faculty of Information Technology, University of Jyväskylä, 40100, Jyvaskyla, Finland
| | - Hongkai Wang
- School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, 116024, China.
- Liaoning Key Laboratory of Integrated Circuit and Biomedical Electronic System, Dalian, China.
| |
Collapse
|
10
|
Yang H, Tan T, Tegzes P, Dong X, Tamada R, Ferenczi L, Avinash G. Light mixed-supervised segmentation for 3D medical image data. Med Phys 2024; 51:167-178. [PMID: 37909833 DOI: 10.1002/mp.16816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Revised: 10/03/2023] [Accepted: 10/16/2023] [Indexed: 11/03/2023] Open
Abstract
BACKGROUND Accurate 3D semantic segmentation models are essential for many clinical applications. To train a model for 3D segmentation, voxel-level annotation is necessary, which is expensive to obtain due to laborious work and privacy protection. To accurately annotate 3D medical data, such as MRI, a common practice is to annotate the volumetric data in a slice-by-slice contouring way along principal axes. PURPOSE In order to reduce the annotation effort in slices, weakly supervised learning with a bounding box (Bbox) was proposed to leverage the discriminating information via a tightness prior assumption. Nevertheless, this method requests accurate and tight Bboxes, which will significantly drop the performance when tightness is not held, that is when a relaxed Bbox is applied. Therefore, there is a need to train a stable model based on relaxed Bbox annotation. METHODS This paper presents a mixed-supervised training strategy to reduce the annotation effort for 3D segmentation tasks. In the proposed approach, a fully annotated contour is only required for a single slice of the volume. In contrast, the rest of the slices with targets are annotated with relaxed Bboxes. This mixed-supervised method adopts fully supervised learning, relaxed Bbox prior, and contrastive learning during the training, which ensures the network exploits the discriminative information of the training volumes properly. The proposed method was evaluated on two public 3D medical imaging datasets (MRI prostate dataset and Vestibular Schwannoma [VS] dataset). RESULTS The proposed method obtained a high segmentation Dice score of 85.3% on an MRI prostate dataset and 83.3% on a VS dataset with relaxed Bbox annotation, which are close to a fully supervised model. Moreover, with the same relaxed Bbox annotations, the proposed method outperforms the state-of-the-art methods. More importantly, the model performance is stable when the accuracy of Bbox annotation varies. CONCLUSIONS The presented study proposes a method based on a mixed-supervised learning method in 3D medical imaging. The benefit will be stable segmentation of the target in 3D images with low accurate annotation requirement, which leads to easier model training on large-scale datasets.
Collapse
Affiliation(s)
| | - Tao Tan
- GE Healthcare, Eindhoven, The Netherlands
| | | | | | | | | | | |
Collapse
|
11
|
Camarasa R, Kervadec H, Kooi ME, Hendrikse J, Nederkoorn PJ, Bos D, de Bruijne M. Nested star-shaped objects segmentation using diameter annotations. Med Image Anal 2023; 90:102934. [PMID: 37688981 DOI: 10.1016/j.media.2023.102934] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Revised: 08/02/2023] [Accepted: 08/10/2023] [Indexed: 09/11/2023]
Abstract
Most current deep learning based approaches for image segmentation require annotations of large datasets, which limits their application in clinical practice. We observe a mismatch between the voxelwise ground-truth that is required to optimize an objective at a voxel level and the commonly used, less time-consuming clinical annotations seeking to characterize the most important information about the patient (diameters, counts, etc.). In this study, we propose to bridge this gap for the case of multiple nested star-shaped objects (e.g., a blood vessel lumen and its outer wall) by optimizing a deep learning model based on diameter annotations. This is achieved by extracting in a differentiable manner the boundary points of the objects at training time, and by using this extraction during the backpropagation. We evaluate the proposed approach on segmentation of the carotid artery lumen and wall from multisequence MR images, thus reducing the annotation burden to only four annotated landmarks required to measure the diameters in the direction of the vessel's maximum narrowing. Our experiments show that training based on diameter annotations produces state-of-the-art weakly supervised segmentations and performs reasonably compared to full supervision. We made our code publicly available at https://gitlab.com/radiology/aim/carotid-artery-image-analysis/nested-star-shaped-objects.
Collapse
Affiliation(s)
- Robin Camarasa
- Biomedical Imaging Group Rotterdam, Erasmus MC, Rotterdam, The Netherlands; Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands.
| | - Hoel Kervadec
- Biomedical Imaging Group Rotterdam, Erasmus MC, Rotterdam, The Netherlands; Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands
| | - M Eline Kooi
- Department of Radiology and Nuclear Medicine, Maastricht University Medical Center, Maastricht, The Netherlands; Cardiovascular Research Institute Maastricht (CARIM), Maastricht University, Maastricht, The Netherlands
| | - Jeroen Hendrikse
- Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Paul J Nederkoorn
- Department of Neurology, Academic Medical Center University of Amsterdam, Amsterdam, The Netherlands
| | - Daniel Bos
- Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands; Department of Epidemiology, Erasmus MC, Rotterdam, The Netherlands
| | - Marleen de Bruijne
- Biomedical Imaging Group Rotterdam, Erasmus MC, Rotterdam, The Netherlands; Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, The Netherlands; Department of Computer Science, University of Copenhagen, Denmark.
| |
Collapse
|
12
|
Ying J, Huang W, Fu L, Yang H, Cheng J. Weakly supervised segmentation of uterus by scribble labeling on endometrial cancer MR images. Comput Biol Med 2023; 167:107582. [PMID: 37922606 DOI: 10.1016/j.compbiomed.2023.107582] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 09/28/2023] [Accepted: 10/15/2023] [Indexed: 11/07/2023]
Abstract
Uterine segmentation of endometrial cancer MR images can be a valuable diagnostic tool for gynecologists. However, uterine segmentation based on deep learning relies on artificial pixel-level annotation, which is time-consuming, laborious and subjective. To reduce the dependence on pixel-level annotation, a method of weakly supervised uterine segmentation on endometrial cancer MRI slices is proposed, which only requires scribble label and is enhanced by pseudo-label technology, exponential geodesic distance loss and input disturbance strategy. Specifically, the limitations caused by the shortage of supervision are addressed by dynamically mixing the two outputs of the dual branch network to generate pseudo-labels, expanding supervision information and promoting mutual supervision training. On the other hand, considering the large difference of grayscale intensity between the uterus and surrounding tissues, the exponential geodesic distance loss is introduced to enhance the ability of the network to capture the edge of the uterus. Input disturbance strategies are incorporated to adapt to the flexible and variable characteristics of the uterus and further improve the segmentation performance of the network. The proposed method is evaluated on MRI images from 135 cases of endometrial cancer. Compared with other four weakly supervised segmentation methods, the performance of the proposed method is the best, whose mean DI, HD95, Recall, Precision, ADP are 92.8%, 11.632, 92.7%, 93.6%, 6.5% and increasing by 2.1%, 9.144, 0.6%, 2.4%, 2.9% respectively. The experimental results demonstrate that the proposed method is more effective than other weakly supervised methods and achieves similar performance as those fully supervised.
Collapse
Affiliation(s)
- Jie Ying
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China.
| | - Wei Huang
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Le Fu
- Department of Radiology, Shanghai First Maternity and Infant Hospital, Tongji University School of Medicine, Shanghai, China.
| | - Haima Yang
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China
| | - Jiangzihao Cheng
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, China
| |
Collapse
|
13
|
Lyu Y, Bennamoun M, Sharif N, Lip GYH, Dwivedi G. Artificial Intelligence in the Image-Guided Care of Atrial Fibrillation. Life (Basel) 2023; 13:1870. [PMID: 37763273 PMCID: PMC10532509 DOI: 10.3390/life13091870] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Revised: 08/19/2023] [Accepted: 08/30/2023] [Indexed: 09/29/2023] Open
Abstract
Atrial fibrillation arises mainly due to abnormalities in the cardiac conduction system and is associated with anatomical remodeling of the atria and the pulmonary veins. Cardiovascular imaging techniques, such as echocardiography, computed tomography, and magnetic resonance imaging, are crucial in the management of atrial fibrillation, as they not only provide anatomical context to evaluate structural alterations but also help in determining treatment strategies. However, interpreting these images requires significant human expertise. The potential of artificial intelligence in analyzing these images has been repeatedly suggested due to its ability to automate the process with precision comparable to human experts. This review summarizes the benefits of artificial intelligence in enhancing the clinical care of patients with atrial fibrillation through cardiovascular image analysis. It provides a detailed overview of the two most critical steps in image-guided AF management, namely, segmentation and classification. For segmentation, the state-of-the-art artificial intelligence methodologies and the factors influencing the segmentation performance are discussed. For classification, the applications of artificial intelligence in the diagnosis and prognosis of atrial fibrillation are provided. Finally, this review also scrutinizes the current challenges hindering the clinical applicability of these methods, with the aim of guiding future research toward more effective integration into clinical practice.
Collapse
Affiliation(s)
- Yiheng Lyu
- Department of Computer Science and Software Engineering, School of Physics, Mathematics and Computing, The University of Western Australia, Perth, WA 6009, Australia; (Y.L.); (M.B.)
- Harry Perkins Institute of Medical Research, The University of Western Australia, Perth, WA 6009, Australia
| | - Mohammed Bennamoun
- Department of Computer Science and Software Engineering, School of Physics, Mathematics and Computing, The University of Western Australia, Perth, WA 6009, Australia; (Y.L.); (M.B.)
| | - Naeha Sharif
- Department of Computer Science and Software Engineering, School of Physics, Mathematics and Computing, The University of Western Australia, Perth, WA 6009, Australia; (Y.L.); (M.B.)
| | - Gregory Y. H. Lip
- Liverpool Centre for Cardiovascular Science, University of Liverpool, Liverpool L69 3BX, UK
- Liverpool John Moores University, Liverpool L3 5UX, UK
- Liverpool Heart and Chest Hospital, Liverpool L14 3PE, UK
- Danish Center for Health Services Research, Department of Clinical Medicine, Aalborg University, 9220 Aalborg, Denmark
| | - Girish Dwivedi
- Harry Perkins Institute of Medical Research, The University of Western Australia, Perth, WA 6009, Australia
- Department of Cardiology, Fiona Stanley Hospital, Perth, WA 6150, Australia
- Medical School, The University of Western Australia, Perth, WA 6009, Australia
| |
Collapse
|
14
|
Swinburne NC, Yadav V, Murthy KNK, Elnajjar P, Shih HH, Panyam PK, Santilli A, Gutman DC, Pike L, Moss NS, Stone J, Hatzoglou V, Shah A, Juluru K, Shah SP, Holodny AI, Young RJ. Fast, light, and scalable: harnessing data-mined line annotations for automated tumor segmentation on brain MRI. Eur Radiol 2023; 33:6582-6591. [PMID: 37042979 PMCID: PMC10523913 DOI: 10.1007/s00330-023-09583-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2022] [Revised: 02/04/2023] [Accepted: 02/16/2023] [Indexed: 04/13/2023]
Abstract
OBJECTIVES While fully supervised learning can yield high-performing segmentation models, the effort required to manually segment large training sets limits practical utility. We investigate whether data mined line annotations can facilitate brain MRI tumor segmentation model development without requiring manually segmented training data. METHODS In this retrospective study, a tumor detection model trained using clinical line annotations mined from PACS was leveraged with unsupervised segmentation to generate pseudo-masks of enhancing tumors on T1-weighted post-contrast images (9911 image slices; 3449 adult patients). Baseline segmentation models were trained and employed within a semi-supervised learning (SSL) framework to refine the pseudo-masks. Following each self-refinement cycle, a new model was trained and tested on a held-out set of 319 manually segmented image slices (93 adult patients), with the SSL cycles continuing until Dice score coefficient (DSC) peaked. DSCs were compared using bootstrap resampling. Utilizing the best-performing models, two inference methods were compared: (1) conventional full-image segmentation, and (2) a hybrid method augmenting full-image segmentation with detection plus image patch segmentation. RESULTS Baseline segmentation models achieved DSC of 0.768 (U-Net), 0.831 (Mask R-CNN), and 0.838 (HRNet), improving with self-refinement to 0.798, 0.871, and 0.873 (each p < 0.001), respectively. Hybrid inference outperformed full image segmentation alone: DSC 0.884 (Mask R-CNN) vs. 0.873 (HRNet), p < 0.001. CONCLUSIONS Line annotations mined from PACS can be harnessed within an automated pipeline to produce accurate brain MRI tumor segmentation models without manually segmented training data, providing a mechanism to rapidly establish tumor segmentation capabilities across radiology modalities. KEY POINTS • A brain MRI tumor detection model trained using clinical line measurement annotations mined from PACS was leveraged to automatically generate tumor segmentation pseudo-masks. • An iterative self-refinement process automatically improved pseudo-mask quality, with the best-performing segmentation pipeline achieving a Dice score of 0.884 on a held-out test set. • Tumor line measurement annotations generated in routine clinical radiology practice can be harnessed to develop high-performing segmentation models without manually segmented training data, providing a mechanism to rapidly establish tumor segmentation capabilities across radiology modalities.
Collapse
Affiliation(s)
- Nathaniel C Swinburne
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA.
| | - Vivek Yadav
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | | | - Pierre Elnajjar
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Hao-Hsin Shih
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Prashanth Kumar Panyam
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Alice Santilli
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - David C Gutman
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Luke Pike
- Department of Radiation Oncology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Nelson S Moss
- Department of Neurosurgery, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Jacqueline Stone
- Department of Neurology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Vaios Hatzoglou
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Akash Shah
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Krishna Juluru
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Sohrab P Shah
- Computational Oncology, Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Andrei I Holodny
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| | - Robert J Young
- Department of Radiology, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY, 10065, USA
| |
Collapse
|
15
|
Ciceri T, Squarcina L, Giubergia A, Bertoldo A, Brambilla P, Peruzzo D. Review on deep learning fetal brain segmentation from Magnetic Resonance images. Artif Intell Med 2023; 143:102608. [PMID: 37673558 DOI: 10.1016/j.artmed.2023.102608] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 05/31/2023] [Accepted: 06/06/2023] [Indexed: 09/08/2023]
Abstract
Brain segmentation is often the first and most critical step in quantitative analysis of the brain for many clinical applications, including fetal imaging. Different aspects challenge the segmentation of the fetal brain in magnetic resonance imaging (MRI), such as the non-standard position of the fetus owing to his/her movements during the examination, rapid brain development, and the limited availability of imaging data. In recent years, several segmentation methods have been proposed for automatically partitioning the fetal brain from MR images. These algorithms aim to define regions of interest with different shapes and intensities, encompassing the entire brain, or isolating specific structures. Deep learning techniques, particularly convolutional neural networks (CNNs), have become a state-of-the-art approach in the field because they can provide reliable segmentation results over heterogeneous datasets. Here, we review the deep learning algorithms developed in the field of fetal brain segmentation and categorize them according to their target structures. Finally, we discuss the perceived research gaps in the literature of the fetal domain, suggesting possible future research directions that could impact the management of fetal MR images.
Collapse
Affiliation(s)
- Tommaso Ciceri
- NeuroImaging Laboratory, Scientific Institute IRCCS Eugenio Medea, Bosisio Parini, Italy; Department of Information Engineering, University of Padua, Padua, Italy
| | - Letizia Squarcina
- Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy
| | - Alice Giubergia
- NeuroImaging Laboratory, Scientific Institute IRCCS Eugenio Medea, Bosisio Parini, Italy; Department of Information Engineering, University of Padua, Padua, Italy
| | - Alessandra Bertoldo
- Department of Information Engineering, University of Padua, Padua, Italy; University of Padua, Padova Neuroscience Center, Padua, Italy
| | - Paolo Brambilla
- Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy; Department of Neurosciences and Mental Health, Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico, Milan, Italy.
| | - Denis Peruzzo
- NeuroImaging Laboratory, Scientific Institute IRCCS Eugenio Medea, Bosisio Parini, Italy
| |
Collapse
|
16
|
Rathore A, Sharma A, Shah S, Sharma N, Torney C, Guttal V. Multi-Object Tracking in Heterogeneous environments (MOTHe) for animal video recordings. PeerJ 2023; 11:e15573. [PMID: 37397020 PMCID: PMC10309051 DOI: 10.7717/peerj.15573] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 05/25/2023] [Indexed: 07/04/2023] Open
Abstract
Aerial imagery and video recordings of animals are used for many areas of research such as animal behaviour, behavioural neuroscience and field biology. Many automated methods are being developed to extract data from such high-resolution videos. Most of the available tools are developed for videos taken under idealised laboratory conditions. Therefore, the task of animal detection and tracking for videos taken in natural settings remains challenging due to heterogeneous environments. Methods that are useful for field conditions are often difficult to implement and thus remain inaccessible to empirical researchers. To address this gap, we present an open-source package called Multi-Object Tracking in Heterogeneous environments (MOTHe), a Python-based application that uses a basic convolutional neural network for object detection. MOTHe offers a graphical interface to automate the various steps related to animal tracking such as training data generation, animal detection in complex backgrounds and visually tracking animals in the videos. Users can also generate training data and train a new model which can be used for object detection tasks for a completely new dataset. MOTHe doesn't require any sophisticated infrastructure and can be run on basic desktop computing units. We demonstrate MOTHe on six video clips in varying background conditions. These videos are from two species in their natural habitat-wasp colonies on their nests (up to 12 individuals per colony) and antelope herds in four different habitats (up to 156 individuals in a herd). Using MOTHe, we are able to detect and track individuals in all these videos. MOTHe is available as an open-source GitHub repository with a detailed user guide and demonstrations at: https://github.com/tee-lab/MOTHe-GUI.
Collapse
Affiliation(s)
- Akanksha Rathore
- Centre for Ecological Sciences, Indian Institute of Science, Bangalore, India
| | - Ananth Sharma
- Centre for Ecological Sciences, Indian Institute of Science, Bangalore, India
| | - Shaan Shah
- Department of Electrical Engineering, Indian Institute of Technology, Bombay, Mumbai, India
| | - Nitika Sharma
- Centre for Ecological Sciences, Indian Institute of Science, Bangalore, India
- Department of Ecology and Evolutionary Biology, University of California, Los Angeles, Los Angeles, United States of America
| | - Colin Torney
- School of Mathematics and Statistics, University of Glasgow, Glasgow, United Kingdom
| | - Vishwesha Guttal
- Centre for Ecological Sciences, Indian Institute of Science, Bangalore, India
| |
Collapse
|
17
|
Karimi D, Rollins CK, Velasco-Annis C, Ouaalam A, Gholipour A. Learning to segment fetal brain tissue from noisy annotations. Med Image Anal 2023; 85:102731. [PMID: 36608414 PMCID: PMC9974964 DOI: 10.1016/j.media.2022.102731] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2022] [Revised: 11/17/2022] [Accepted: 12/23/2022] [Indexed: 01/03/2023]
Abstract
Automatic fetal brain tissue segmentation can enhance the quantitative assessment of brain development at this critical stage. Deep learning methods represent the state of the art in medical image segmentation and have also achieved impressive results in brain segmentation. However, effective training of a deep learning model to perform this task requires a large number of training images to represent the rapid development of the transient fetal brain structures. On the other hand, manual multi-label segmentation of a large number of 3D images is prohibitive. To address this challenge, we segmented 272 training images, covering 19-39 gestational weeks, using an automatic multi-atlas segmentation strategy based on deformable registration and probabilistic atlas fusion, and manually corrected large errors in those segmentations. Since this process generated a large training dataset with noisy segmentations, we developed a novel label smoothing procedure and a loss function to train a deep learning model with smoothed noisy segmentations. Our proposed methods properly account for the uncertainty in tissue boundaries. We evaluated our method on 23 manually-segmented test images of a separate set of fetuses. Results show that our method achieves an average Dice similarity coefficient of 0.893 and 0.916 for the transient structures of younger and older fetuses, respectively. Our method generated results that were significantly more accurate than several state-of-the-art methods including nnU-Net that achieved the closest results to our method. Our trained model can serve as a valuable tool to enhance the accuracy and reproducibility of fetal brain analysis in MRI.
Collapse
Affiliation(s)
- Davood Karimi
- Department of Radiology, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA.
| | - Caitlin K Rollins
- Department of Neurology, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Clemente Velasco-Annis
- Department of Radiology, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Abdelhakim Ouaalam
- Department of Radiology, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| | - Ali Gholipour
- Department of Radiology, Boston Children's Hospital, Harvard Medical School, Boston, MA, USA
| |
Collapse
|
18
|
Li K, Qian Z, Han Y, Chang EIC, Wei B, Lai M, Liao J, Fan Y, Xu Y. Weakly supervised histopathology image segmentation with self-attention. Med Image Anal 2023; 86:102791. [PMID: 36933385 DOI: 10.1016/j.media.2023.102791] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2022] [Revised: 01/09/2023] [Accepted: 02/24/2023] [Indexed: 03/13/2023]
Abstract
Accurate segmentation in histopathology images at pixel-level plays a critical role in the digital pathology workflow. The development of weakly supervised methods for histopathology image segmentation liberates pathologists from time-consuming and labor-intensive works, opening up possibilities of further automated quantitative analysis of whole-slide histopathology images. As an effective subgroup of weakly supervised methods, multiple instance learning (MIL) has achieved great success in histopathology images. In this paper, we specially treat pixels as instances so that the histopathology image segmentation task is transformed into an instance prediction task in MIL. However, the lack of relations between instances in MIL limits the further improvement of segmentation performance. Therefore, we propose a novel weakly supervised method called SA-MIL for pixel-level segmentation in histopathology images. SA-MIL introduces a self-attention mechanism into the MIL framework, which captures global correlation among all instances. In addition, we use deep supervision to make the best use of information from limited annotations in the weakly supervised method. Our approach makes up for the shortcoming that instances are independent of each other in MIL by aggregating global contextual information. We demonstrate state-of-the-art results compared to other weakly supervised methods on two histopathology image datasets. It is evident that our approach has generalization ability for the high performance on both tissue and cell histopathology datasets. There is potential in our approach for various applications in medical images.
Collapse
Affiliation(s)
- Kailu Li
- School of Biological Science and Medical Engineering, State Key Laboratory of Software Development Environment, Key Laboratory of Biomechanics, Mechanobiology of Ministry of Education and Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing 100191, China.
| | - Ziniu Qian
- School of Biological Science and Medical Engineering, State Key Laboratory of Software Development Environment, Key Laboratory of Biomechanics, Mechanobiology of Ministry of Education and Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing 100191, China.
| | - Yingnan Han
- School of Biological Science and Medical Engineering, State Key Laboratory of Software Development Environment, Key Laboratory of Biomechanics, Mechanobiology of Ministry of Education and Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing 100191, China.
| | | | | | - Maode Lai
- Department of Pathology, School of Medicine, Zhejiang University, Hangzhou 310027, China.
| | - Jing Liao
- Department of Computer Science, City University of Hong Kong, 999077, Hong Kong SAR, China.
| | - Yubo Fan
- School of Biological Science and Medical Engineering, State Key Laboratory of Software Development Environment, Key Laboratory of Biomechanics, Mechanobiology of Ministry of Education and Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing 100191, China.
| | - Yan Xu
- School of Biological Science and Medical Engineering, State Key Laboratory of Software Development Environment, Key Laboratory of Biomechanics, Mechanobiology of Ministry of Education and Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing 100191, China; Microsoft Research, Beijing 100080, China.
| |
Collapse
|
19
|
Urru A, Nakaki A, Benkarim O, Crovetto F, Segalés L, Comte V, Hahner N, Eixarch E, Gratacos E, Crispi F, Piella G, González Ballester MA. An automatic pipeline for atlas-based fetal and neonatal brain segmentation and analysis. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 230:107334. [PMID: 36682108 DOI: 10.1016/j.cmpb.2023.107334] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2022] [Revised: 11/29/2022] [Accepted: 01/02/2023] [Indexed: 06/17/2023]
Abstract
BACKGROUND AND OBJECTIVE The automatic segmentation of perinatal brain structures in magnetic resonance imaging (MRI) is of utmost importance for the study of brain growth and related complications. While different methods exist for adult and pediatric MRI data, there is a lack for automatic tools for the analysis of perinatal imaging. METHODS In this work, a new pipeline for fetal and neonatal segmentation has been developed. We also report the creation of two new fetal atlases, and their use within the pipeline for atlas-based segmentation, based on novel registration methods. The pipeline is also able to extract cortical and pial surfaces and compute features, such as curvature, local gyrification index, sulcal depth, and thickness. RESULTS Results show that the introduction of the new templates together with our segmentation strategy leads to accurate results when compared to expert annotations, as well as better performances when compared to a reference pipeline (developing Human Connectome Project (dHCP)), for both early and late-onset fetal brains. CONCLUSIONS These findings show the potential of the presented atlases and the whole pipeline for application in both fetal, neonatal, and longitudinal studies, which could lead to dramatic improvements in the understanding of perinatal brain development.
Collapse
Affiliation(s)
- Andrea Urru
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain
| | - Ayako Nakaki
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain; Institut d'Investigacions Biomédiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain
| | - Oualid Benkarim
- McConnell Brain Imaging Centre, Montreal Neurological Institute and Hospital, McGill University, Montreal, Quebec, Canada
| | - Francesca Crovetto
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain
| | - Laura Segalés
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain
| | - Valentin Comte
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain
| | - Nadine Hahner
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain; Institut d'Investigacions Biomédiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain
| | - Elisenda Eixarch
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain; Institut d'Investigacions Biomédiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Centre for Biomedical Research on Rare Diseases (CIBERER), Barcelona, Spain
| | - Eduard Gratacos
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain; Institut d'Investigacions Biomédiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Centre for Biomedical Research on Rare Diseases (CIBERER), Barcelona, Spain
| | - Fàtima Crispi
- BCNatal | Fetal Medicine Research Center (Hospital Clínic and Hospital Sant Joan de Déu), University of Barcelona, Barcelona, Spain; Institut d'Investigacions Biomédiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain; Centre for Biomedical Research on Rare Diseases (CIBERER), Barcelona, Spain
| | - Gemma Piella
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain
| | - Miguel A González Ballester
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain; ICREA, Barcelona, Spain.
| |
Collapse
|
20
|
Zhao Z, Zhou F, Xu K, Zeng Z, Guan C, Zhou SK. LE-UDA: Label-Efficient Unsupervised Domain Adaptation for Medical Image Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:633-646. [PMID: 36227829 DOI: 10.1109/tmi.2022.3214766] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
While deep learning methods hitherto have achieved considerable success in medical image segmentation, they are still hampered by two limitations: (i) reliance on large-scale well-labeled datasets, which are difficult to curate due to the expert-driven and time-consuming nature of pixel-level annotations in clinical practices, and (ii) failure to generalize from one domain to another, especially when the target domain is a different modality with severe domain shifts. Recent unsupervised domain adaptation (UDA) techniques leverage abundant labeled source data together with unlabeled target data to reduce the domain gap, but these methods degrade significantly with limited source annotations. In this study, we address this underexplored UDA problem, investigating a challenging but valuable realistic scenario, where the source domain not only exhibits domain shift w.r.t. the target domain but also suffers from label scarcity. In this regard, we propose a novel and generic framework called "Label-Efficient Unsupervised Domain Adaptation" (LE-UDA). In LE-UDA, we construct self-ensembling consistency for knowledge transfer between both domains, as well as a self-ensembling adversarial learning module to achieve better feature alignment for UDA. To assess the effectiveness of our method, we conduct extensive experiments on two different tasks for cross-modality segmentation between MRI and CT images. Experimental results demonstrate that the proposed LE-UDA can efficiently leverage limited source labels to improve cross-domain segmentation performance, outperforming state-of-the-art UDA approaches in the literature.
Collapse
|
21
|
Meng Y, Zhang H, Zhao Y, Gao D, Hamill B, Patri G, Peto T, Madhusudhan S, Zheng Y. Dual Consistency Enabled Weakly and Semi-Supervised Optic Disc and Cup Segmentation With Dual Adaptive Graph Convolutional Networks. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:416-429. [PMID: 36044486 DOI: 10.1109/tmi.2022.3203318] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Glaucoma is a progressive eye disease that results in permanent vision loss, and the vertical cup to disc ratio (vCDR) in colour fundus images is essential in glaucoma screening and assessment. Previous fully supervised convolution neural networks segment the optic disc (OD) and optic cup (OC) from color fundus images and then calculate the vCDR offline. However, they rely on a large set of labeled masks for training, which is expensive and time-consuming to acquire. To address this, we propose a weakly and semi-supervised graph-based network that investigates geometric associations and domain knowledge between segmentation probability maps (PM), modified signed distance function representations (mSDF), and boundary region of interest characteristics (B-ROI) in three aspects. Firstly, we propose a novel Dual Adaptive Graph Convolutional Network (DAGCN) to reason the long-range features of the PM and the mSDF w.r.t. the regional uniformity. Secondly, we propose a dual consistency regularization-based semi-supervised learning paradigm. The regional consistency between the PM and the mSDF, and the marginal consistency between the derived B-ROI from each of them boost the proposed model's performance due to the inherent geometric associations. Thirdly, we exploit the task-specific domain knowledge via the oval shapes of OD & OC, where a differentiable vCDR estimating layer is proposed. Furthermore, without additional annotations, the supervision on vCDR serves as weakly-supervisions for segmentation tasks. Experiments on six large-scale datasets demonstrate our model's superior performance on OD & OC segmentation and vCDR estimation. The implementation code has been made available.https://github.com/smallmax00/Dual_Adaptive_Graph_Reasoning.
Collapse
|
22
|
Hadjiiski L, Cha K, Chan HP, Drukker K, Morra L, Näppi JJ, Sahiner B, Yoshida H, Chen Q, Deserno TM, Greenspan H, Huisman H, Huo Z, Mazurchuk R, Petrick N, Regge D, Samala R, Summers RM, Suzuki K, Tourassi G, Vergara D, Armato SG. AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer-aided diagnosis in medical imaging. Med Phys 2023; 50:e1-e24. [PMID: 36565447 DOI: 10.1002/mp.16188] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 11/13/2022] [Accepted: 11/22/2022] [Indexed: 12/25/2022] Open
Abstract
Rapid advances in artificial intelligence (AI) and machine learning, and specifically in deep learning (DL) techniques, have enabled broad application of these methods in health care. The promise of the DL approach has spurred further interest in computer-aided diagnosis (CAD) development and applications using both "traditional" machine learning methods and newer DL-based methods. We use the term CAD-AI to refer to this expanded clinical decision support environment that uses traditional and DL-based AI methods. Numerous studies have been published to date on the development of machine learning tools for computer-aided, or AI-assisted, clinical tasks. However, most of these machine learning models are not ready for clinical deployment. It is of paramount importance to ensure that a clinical decision support tool undergoes proper training and rigorous validation of its generalizability and robustness before adoption for patient care in the clinic. To address these important issues, the American Association of Physicists in Medicine (AAPM) Computer-Aided Image Analysis Subcommittee (CADSC) is charged, in part, to develop recommendations on practices and standards for the development and performance assessment of computer-aided decision support systems. The committee has previously published two opinion papers on the evaluation of CAD systems and issues associated with user training and quality assurance of these systems in the clinic. With machine learning techniques continuing to evolve and CAD applications expanding to new stages of the patient care process, the current task group report considers the broader issues common to the development of most, if not all, CAD-AI applications and their translation from the bench to the clinic. The goal is to bring attention to the proper training and validation of machine learning algorithms that may improve their generalizability and reliability and accelerate the adoption of CAD-AI systems for clinical decision support.
Collapse
Affiliation(s)
- Lubomir Hadjiiski
- Department of Radiology, University of Michigan, Ann Arbor, Michigan, USA
| | - Kenny Cha
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Heang-Ping Chan
- Department of Radiology, University of Michigan, Ann Arbor, Michigan, USA
| | - Karen Drukker
- Department of Radiology, University of Chicago, Chicago, Illinois, USA
| | - Lia Morra
- Department of Control and Computer Engineering, Politecnico di Torino, Torino, Italy
| | - Janne J Näppi
- 3D Imaging Research, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Berkman Sahiner
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Hiroyuki Yoshida
- 3D Imaging Research, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Quan Chen
- Department of Radiation Medicine, University of Kentucky, Lexington, Kentucky, USA
| | - Thomas M Deserno
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Braunschweig, Germany
| | - Hayit Greenspan
- Department of Biomedical Engineering, Faculty of Engineering, Tel Aviv, Israel & Department of Radiology, Ichan School of Medicine, Tel Aviv University, Mt Sinai, New York, New York, USA
| | - Henkjan Huisman
- Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Zhimin Huo
- Tencent America, Palo Alto, California, USA
| | - Richard Mazurchuk
- Division of Cancer Prevention, National Cancer Institute, National Institutes of Health, Bethesda, Maryland, USA
| | | | - Daniele Regge
- Radiology Unit, Candiolo Cancer Institute, FPO-IRCCS, Candiolo, Italy.,Department of Surgical Sciences, University of Turin, Turin, Italy
| | - Ravi Samala
- U.S. Food and Drug Administration, Silver Spring, Maryland, USA
| | - Ronald M Summers
- Radiology and Imaging Sciences, National Institutes of Health Clinical Center, Bethesda, Maryland, USA
| | - Kenji Suzuki
- Institute of Innovative Research, Tokyo Institute of Technology, Tokyo, Japan
| | | | - Daniel Vergara
- Department of Radiology, Yale New Haven Hospital, New Haven, Connecticut, USA
| | - Samuel G Armato
- Department of Radiology, University of Chicago, Chicago, Illinois, USA
| |
Collapse
|
23
|
Zhou T, Li L, Bredell G, Li J, Unkelbach J, Konukoglu E. Volumetric memory network for interactive medical image segmentation. Med Image Anal 2023; 83:102599. [PMID: 36327652 DOI: 10.1016/j.media.2022.102599] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2022] [Revised: 06/23/2022] [Accepted: 08/24/2022] [Indexed: 12/13/2022]
Abstract
Despite recent progress of automatic medical image segmentation techniques, fully automatic results usually fail to meet clinically acceptable accuracy, thus typically require further refinement. To this end, we propose a novel Volumetric Memory Network, dubbed as VMN, to enable segmentation of 3D medical images in an interactive manner. Provided by user hints on an arbitrary slice, a 2D interaction network is firstly employed to produce an initial 2D segmentation for the chosen slice. Then, the VMN propagates the initial segmentation mask bidirectionally to all slices of the entire volume. Subsequent refinement based on additional user guidance on other slices can be incorporated in the same manner. To facilitate smooth human-in-the-loop segmentation, a quality assessment module is introduced to suggest the next slice for interaction based on the segmentation quality of each slice produced in the previous round. Our VMN demonstrates two distinctive features: First, the memory-augmented network design offers our model the ability to quickly encode past segmentation information, which will be retrieved later for the segmentation of other slices; Second, the quality assessment module enables the model to directly estimate the quality of each segmentation prediction, which allows for an active learning paradigm where users preferentially label the lowest-quality slice for multi-round refinement. The proposed network leads to a robust interactive segmentation engine, which can generalize well to various types of user annotations (e.g., scribble, bounding box, extreme clicking). Extensive experiments have been conducted on three public medical image segmentation datasets (i.e., MSD, KiTS19, CVC-ClinicDB), and the results clearly confirm the superiority of our approach in comparison with state-of-the-art segmentation models. The code is made publicly available at https://github.com/0liliulei/Mem3D.
Collapse
Affiliation(s)
- Tianfei Zhou
- Computer Vision Laboratory, ETH Zurich, Switzerland.
| | - Liulei Li
- School of Computer Science and Technology, Beijing Institute of Technology, China
| | | | - Jianwu Li
- School of Computer Science and Technology, Beijing Institute of Technology, China
| | - Jan Unkelbach
- Department of Radiation Oncology, University Hospital of Zurich, Zurich, Switzerland
| | | |
Collapse
|
24
|
Kim HY, Cho GJ, Kwon HS. Applications of artificial intelligence in obstetrics. Ultrasonography 2023; 42:2-9. [PMID: 36588179 PMCID: PMC9816710 DOI: 10.14366/usg.22063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2022] [Accepted: 06/20/2022] [Indexed: 01/13/2023] Open
Abstract
Artificial intelligence, which has been applied as an innovative technology in multiple fields of healthcare, analyzes large amounts of data to assist in disease prediction, prevention, and diagnosis, as well as in patient monitoring. In obstetrics, artificial intelligence has been actively applied and integrated into our daily medical practice. This review provides an overview of artificial intelligence systems currently used for obstetric diagnostic purposes, such as fetal cardiotocography, ultrasonography, and magnetic resonance imaging, and demonstrates how these methods have been developed and clinically applied.
Collapse
Affiliation(s)
- Ho Yeon Kim
- Department of Obstetrics and Gynecology, Korea University College of Medicine, Seoul, Korea
| | - Geum Joon Cho
- Department of Obstetrics and Gynecology, Korea University College of Medicine, Seoul, Korea,Correspondence to: Geum Joon Cho, MD, PhD, Department of Obstetrics and Gynecology, Korea University Guro Hospital, Korea University College of Medicine, 148 Gurodong-ro, Guro-gu, Seoul 08308, Korea Tel. +82-2-2626-3141 Fax. +82-2-838-1560 E-mail:
| | - Han Sung Kwon
- Division of Maternal and Fetal Medicine, Department of Obstetrics and Gynecology, Research Institute of Medical Science, Konkuk University School of Medicine, Seoul, Korea
| |
Collapse
|
25
|
Meng X, Fan J, Yu H, Mu J, Li Z, Yang A, Liu B, Lv K, Ai D, Lin Y, Song H, Fu T, Xiao D, Ma G, Yang J, Gu Y. Volume-awareness and outlier-suppression co-training for weakly-supervised MRI breast mass segmentation with partial annotations. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109988] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
26
|
Li Z, Zhou S, Chang C, Wang Y, Guo Y. A Weakly Supervised Deep Active Contour Model for Nodule Segmentation in Thyroid Ultrasound Images. Pattern Recognit Lett 2022. [DOI: 10.1016/j.patrec.2022.12.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
|
27
|
Shen Y, Shen D, Ke J. Identify Representative Samples by Conditional Random Field of Cancer Histology Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:3835-3848. [PMID: 35951579 DOI: 10.1109/tmi.2022.3198526] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Pathology analysis is crucial to precise cancer diagnoses and the succeeding treatment plan as well. To detect abnormality in histopathology images with prevailing patch-based convolutional neural networks (CNNs), contextual information often serves as a powerful cue. However, as whole-slide images (WSIs) are characterized by intense morphological heterogeneity and extensive tissue scale, a straightforward visual span to a larger context may not well capture the information closely associated with the focal patch. In this paper, we propose a novel pixel-offset based patch-location method to identify high-representative tissues, with a CNN backbone. Pathology Deformable Conditional Random Field (PDCRF) is proposed to learn the offsets and weights of neighboring contexts in a spatial-adaptive manner, to search for high-representative patches. A CNN structure with the localized patches as training input is then capable of consistently reaching superior classification outcomes for histology images. Overall, the proposed method has achieved state-of-the-art performance, in terms of the test classification accuracy improvement to the baseline by 1.15-2.60%, 0.78-1.78%, and 1.47-2.18% on TCGA public datasets of TCGA-STAD, TCGA-COAD, and TCGA-READ respectively. It also achieves 88.95% test accuracy and 0.920 test AUC on Camelyon 16. To show the effectiveness of the proposed framework on downstream tasks, we take a further step by incorporating an active learning model, which noticeably reduces the number of manual annotations by PDCRF to reach a parallel patch-based histology classifier.
Collapse
|
28
|
Artificial intelligence and machine learning in cancer imaging. COMMUNICATIONS MEDICINE 2022; 2:133. [PMID: 36310650 PMCID: PMC9613681 DOI: 10.1038/s43856-022-00199-0] [Citation(s) in RCA: 52] [Impact Index Per Article: 26.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2020] [Accepted: 10/06/2022] [Indexed: 11/16/2022] Open
Abstract
An increasing array of tools is being developed using artificial intelligence (AI) and machine learning (ML) for cancer imaging. The development of an optimal tool requires multidisciplinary engagement to ensure that the appropriate use case is met, as well as to undertake robust development and testing prior to its adoption into healthcare systems. This multidisciplinary review highlights key developments in the field. We discuss the challenges and opportunities of AI and ML in cancer imaging; considerations for the development of algorithms into tools that can be widely used and disseminated; and the development of the ecosystem needed to promote growth of AI and ML in cancer imaging.
Collapse
|
29
|
Jin G, Liu C, Chen X. An efficient deep neural network framework for COVID-19 lung infection segmentation. Inf Sci (N Y) 2022; 612:745-758. [PMID: 36068814 PMCID: PMC9436790 DOI: 10.1016/j.ins.2022.08.059] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Revised: 08/11/2022] [Accepted: 08/13/2022] [Indexed: 02/04/2023]
Abstract
Since the outbreak of Coronavirus Disease 2019 (COVID-19) in 2020, it has significantly affected the global health system. The use of deep learning technology to automatically segment pneumonia lesions from Computed Tomography (CT) images can greatly reduce the workload of physicians and expand traditional diagnostic methods. However, there are still some challenges to tackle the task, including obtaining high-quality annotations and subtle differences between classes. In the present study, a novel deep neural network based on Resnet architecture is proposed to automatically segment infected areas from CT images. To reduce the annotation cost, a Vector Quantized Variational AutoEncoder (VQ-VAE) branch is added to reconstruct the input images for purpose of regularizing the shared decoder and the latent maps of the VQ-VAE are utilized to further improve the feature representation. Moreover, a novel proportions loss is presented for mitigating class imbalance and enhance the generalization ability of the model. In addition, a semi-supervised mechanism based on adversarial learning to the network has been proposed, which can utilize the information of the trusted region in unlabeled images to further regularize the network. Extensive experiments on the COVID-SemiSeg are performed to verify the superiority of the proposed method, and the results are in line with expectations.
Collapse
Affiliation(s)
- Ge Jin
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
| | - Chuancai Liu
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
- Collaborative Innovation Center of IoT Technology and Intelligent Systems, Minjiang University, Fuzhou 350108, China
| | - Xu Chen
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
| |
Collapse
|
30
|
Bateson M, Kervadec H, Dolz J, Lombaert H, Ben Ayed I. Source-free domain adaptation for image segmentation. Med Image Anal 2022; 82:102617. [DOI: 10.1016/j.media.2022.102617] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2021] [Revised: 07/25/2022] [Accepted: 09/02/2022] [Indexed: 11/25/2022]
|
31
|
Wang Z, Chen W, Li T, Zhang S, Xiong R. Improved YOLOv3 detection method for PCB plug-in solder joint defects based on ordered probability density weighting and attention mechanism. AI COMMUN 2022. [DOI: 10.3233/aic-210245] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Printed Circuit Board (PCB) is the heart component of electronic products, and its defect detection is the basic requirement of PCB quality control in the production process. Traditional visual detection methods need artificial design features, so their detection accuracy is poor, and the rate of missed and false detection is high. To solve the above problems, this paper proposes an improved YOLOv3 (You Only Look Once) detection method for PCB plug-in solder spot defects based on the combination of the ordered probability density weighting and the attention mechanism. First, in order to obtain a higher priority priori box, the ordered probability density weighting (OWA) method is used to optimize the multiple sets of a priori boxes generated by K-means. Then, to get more effective feature information, the Squeeze-and-Excitation mechanism (SE) is added to the backbone network. In the feature detection network, the Convolutional Block Attention Module (CBAM) attention mechanism is joined, at the same time in the inspection network output layer three layer feature are fusions. Finally, in order to accelerate the convergence speed of model and improve the accuracy of the model, the network loss function was improved by using the generalized joint generalized intersection over union (GIoU), and the COCO data model was applied to PCB solder spot defect training by transfer learning method. After testing, the average detection accuracy of improved network is improved from 84.35% to 96.69%, and the improved network has better convergence than the original network. The study shows that the improved method based on YOLOv3 is more suitable for industrial application of PCB plug-in solder spot defect detection.
Collapse
Affiliation(s)
- Zheng Wang
- School of Electrical Engineering Chongqing University of Science & Technology, Chongqing University of Science and Technology, China
| | - Wenbin Chen
- School of Electrical Engineering Chongqing University of Science & Technology, Chongqing University of Science and Technology, China
| | - Taifu Li
- School of Electrical Engineering Chongqing University of Science & Technology, Chongqing University of Science and Technology, China
| | - Shaolin Zhang
- School of Electrical Engineering Chongqing University of Science & Technology, Chongqing University of Science and Technology, China
| | - Rui Xiong
- School of Electrical Engineering Chongqing University of Science & Technology, Chongqing University of Science and Technology, China
| |
Collapse
|
32
|
Schumacher M, Siebert H, Genz A, Bade R, Heinrich M. Learning-based three-dimensional registration with weak bounding box supervision. J Med Imaging (Bellingham) 2022; 9:044001. [DOI: 10.1117/1.jmi.9.4.044001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2021] [Accepted: 06/28/2022] [Indexed: 11/14/2022] Open
Affiliation(s)
- Mona Schumacher
- University of Luebeck, Institute of Medical Informatics, Luebeck
| | - Hanna Siebert
- University of Luebeck, Institute of Medical Informatics, Luebeck
| | | | | | - Mattias Heinrich
- University of Luebeck, Institute of Medical Informatics, Luebeck
| |
Collapse
|
33
|
Kuang X, Cheung JPY, Wong KYK, Lam WY, Lam CH, Choy RW, Cheng CP, Wu H, Yang C, Wang K, Li Y, Zhang T. Spine-GFlow: A hybrid learning framework for robust multi-tissue segmentation in lumbar MRI without manual annotation. Comput Med Imaging Graph 2022; 99:102091. [PMID: 35803034 DOI: 10.1016/j.compmedimag.2022.102091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2022] [Revised: 05/30/2022] [Accepted: 06/13/2022] [Indexed: 10/18/2022]
Abstract
Most learning-based magnetic resonance image (MRI) segmentation methods rely on the manual annotation to provide supervision, which is extremely tedious, especially when multiple anatomical structures are required. In this work, we aim to develop a hybrid framework named Spine-GFlow that combines the image features learned by a CNN model and anatomical priors for multi-tissue segmentation in a sagittal lumbar MRI. Our framework does not require any manual annotation and is robust against image feature variation caused by different image settings and/or underlying pathology. Our contributions include: 1) a rule-based method that automatically generates the weak annotation (initial seed area), 2) a novel proposal generation method that integrates the multi-scale image features and anatomical prior, 3) a comprehensive loss for CNN training that optimizes the pixel classification and feature distribution simultaneously. Our Spine-GFlow has been validated on 2 independent datasets: HKDDC (containing images obtained from 3 different machines) and IVDM3Seg. The segmentation results of vertebral bodies (VB), intervertebral discs (IVD), and spinal canal (SC) are evaluated quantitatively using intersection over union (IoU) and the Dice coefficient. Results show that our method, without requiring manual annotation, has achieved a segmentation performance comparable to a model trained with full supervision (mean Dice 0.914 vs 0.916).
Collapse
Affiliation(s)
- Xihe Kuang
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China
| | - Jason Pui Yin Cheung
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China
| | - Kwan-Yee K Wong
- Department of Computer Science, Faculty of Engineering, University of Hong Kong, Hong Kong, China
| | - Wai Yi Lam
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China
| | - Chak Hei Lam
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China
| | - Richard W Choy
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China
| | | | - Honghan Wu
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China
| | - Cao Yang
- Department of Orthopedics, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China
| | - Kun Wang
- Department of Orthopedics, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430022, China
| | - Yang Li
- School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China.
| | - Teng Zhang
- Department of Orthopaedics and Traumatology, Li Ka Shing Faculty of Medicine, University of Hong Kong, Hong Kong, China.
| |
Collapse
|
34
|
Adiga V S, Dolz J, Lombaert H. Attention-based Dynamic Subspace Learners for Medical Image Analysis. IEEE J Biomed Health Inform 2022; 26:4599-4610. [PMID: 35763468 DOI: 10.1109/jbhi.2022.3186882] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Learning similarity is a key aspect in medical image analysis, particularly in recommendation systems or in uncovering the interpretation of anatomical data in images. Most existing methods learn such similarities in the embedding space over image sets using a single metric learner. Images, however, have a variety of object attributes such as color, shape, or artifacts. Encoding such attributes using a single metric learner is inadequate and may fail to generalize. Instead, multiple learners could focus on separate aspects of these attributes in subspaces of an overarching embedding. This, however, implies the number of learners to be found empirically for each new dataset. This work, Dynamic Subspace Learners, proposes to dynamically exploit multiple learners by removing the need of knowing apriori the number of learners and aggregating new subspace learners during training. Furthermore, the visual interpretability of such subspace learning is enforced by integrating an attention module into our method. This integrated attention mechanism provides a visual insight of discriminative image features that contribute to the clustering of image sets and a visual explanation of the embedding features. The benefits of our attention-based dynamic subspace learners are evaluated in the application of image clustering, image retrieval, and weakly supervised segmentation. Our method achieves competitive results with the performances of multiple learners baselines and significantly outperforms the classification network in terms of clustering and retrieval scores on three different public benchmark datasets. Moreover, our method also provides an attention map generated directly during inference to illustrate the visual interpretability of the embedding features. These attention maps offer a proxy-labels, which improves the segmentation accuracy up to 15% in Dice scores when compared to state-of-the-art interpretation techniques.
Collapse
|
35
|
A teacher-student framework for liver and tumor segmentation under mixed supervision from abdominal CT scans. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07240-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
36
|
Trimpl MJ, Primakov S, Lambin P, Stride EPJ, Vallis KA, Gooding MJ. Beyond automatic medical image segmentation-the spectrum between fully manual and fully automatic delineation. Phys Med Biol 2022; 67. [PMID: 35523158 DOI: 10.1088/1361-6560/ac6d9c] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Accepted: 05/06/2022] [Indexed: 12/19/2022]
Abstract
Semi-automatic and fully automatic contouring tools have emerged as an alternative to fully manual segmentation to reduce time spent contouring and to increase contour quality and consistency. Particularly, fully automatic segmentation has seen exceptional improvements through the use of deep learning in recent years. These fully automatic methods may not require user interactions, but the resulting contours are often not suitable to be used in clinical practice without a review by the clinician. Furthermore, they need large amounts of labelled data to be available for training. This review presents alternatives to manual or fully automatic segmentation methods along the spectrum of variable user interactivity and data availability. The challenge lies to determine how much user interaction is necessary and how this user interaction can be used most effectively. While deep learning is already widely used for fully automatic tools, interactive methods are just at the starting point to be transformed by it. Interaction between clinician and machine, via artificial intelligence, can go both ways and this review will present the avenues that are being pursued to improve medical image segmentation.
Collapse
Affiliation(s)
- Michael J Trimpl
- Mirada Medical Ltd, Oxford, United Kingdom
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, United Kingdom
- Oxford Institute for Radiation Oncology, University of Oxford, Oxford, United Kingdom
| | - Sergey Primakov
- The D-Lab, Department of Precision Medicine, GROW-School for Oncology, Maastricht University, Maastricht, NL, The Netherlands
| | - Philippe Lambin
- The D-Lab, Department of Precision Medicine, GROW-School for Oncology, Maastricht University, Maastricht, NL, The Netherlands
| | - Eleanor P J Stride
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, United Kingdom
| | - Katherine A Vallis
- Oxford Institute for Radiation Oncology, University of Oxford, Oxford, United Kingdom
| | | |
Collapse
|
37
|
Moser F, Huang R, Papież BW, Namburete AIL. BEAN: Brain Extraction and Alignment Network for 3D Fetal Neurosonography. Neuroimage 2022; 258:119341. [PMID: 35654376 DOI: 10.1016/j.neuroimage.2022.119341] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2021] [Revised: 04/08/2022] [Accepted: 05/28/2022] [Indexed: 01/18/2023] Open
Abstract
Brain extraction (masking of extra-cerebral tissues) and alignment are fundamental first steps of most neuroimage analysis pipelines. The lack of automated solutions for 3D ultrasound (US) has therefore limited its potential as a neuroimaging modality for studying fetal brain development using routinely acquired scans. In this work, we propose a convolutional neural network (CNN) that accurately and consistently aligns and extracts the fetal brain from minimally pre-processed 3D US scans. Our multi-task CNN, Brain Extraction and Alignment Network (BEAN), consists of two independent branches: 1) a fully-convolutional encoder-decoder branch for brain extraction of unaligned scans, and 2) a two-step regression-based branch for similarity alignment of the brain to a common coordinate space. BEAN was tested on 356 fetal head 3D scans spanning the gestational range of 14 to 30 weeks, significantly outperforming all current alternatives for fetal brain extraction and alignment. BEAN achieved state-of-the-art performance for both tasks, with a mean Dice Similarity Coefficient (DSC) of 0.94 for the brain extraction masks, and a mean DSC of 0.93 for the alignment of the target brain masks. The presented experimental results show that brain structures such as the thalamus, choroid plexus, cavum septum pellucidum, and Sylvian fissure, are consistently aligned throughout the dataset and remain clearly visible when the scans are averaged together. The BEAN implementation and related code can be found under www.github.com/felipemoser/kelluwen.
Collapse
Affiliation(s)
- Felipe Moser
- Oxford Machine Learning in Neuroimaging laboratory, OMNI, Department of Computer Science, University of Oxford, Oxford, UK.
| | - Ruobing Huang
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
| | -
- Nuffield Department of Women's and Reproductive Health, John Radcliffe Hospital, University of Oxford, Oxford, UK
| | - Bartłomiej W Papież
- Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK; Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, University of Oxford, Oxford, UK
| | - Ana I L Namburete
- Oxford Machine Learning in Neuroimaging laboratory, OMNI, Department of Computer Science, University of Oxford, Oxford, UK; Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, United Kingdom
| |
Collapse
|
38
|
Lyu F, Ma AJ, Yip TCF, Wong GLH, Yuen PC. Weakly Supervised Liver Tumor Segmentation Using Couinaud Segment Annotation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:1138-1149. [PMID: 34871168 DOI: 10.1109/tmi.2021.3132905] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Automatic liver tumor segmentation is of great importance for assisting doctors in liver cancer diagnosis and treatment planning. Recently, deep learning approaches trained with pixel-level annotations have contributed many breakthroughs in image segmentation. However, acquiring such accurate dense annotations is time-consuming and labor-intensive, which limits the performance of deep neural networks for medical image segmentation. We note that Couinaud segment is widely used by radiologists when recording liver cancer-related findings in the reports, since it is well-suited for describing the localization of tumors. In this paper, we propose a novel approach to train convolutional networks for liver tumor segmentation using Couinaud segment annotations. Couinaud segment annotations are image-level labels with values ranging from 1 to 8, indicating a specific region of the liver. Our proposed model, namely CouinaudNet, can estimate pseudo tumor masks from the Couinaud segment annotations as pixel-wise supervision for training a fully supervised tumor segmentation model, and it is composed of two components: 1) an inpainting network with Couinaud segment masks which can effectively remove tumors for pathological images by filling the tumor regions with plausible healthy-looking intensities; 2) a difference spotting network for segmenting the tumors, which is trained with healthy-pathological pairs generated by an effective tumor synthesis strategy. The proposed method is extensively evaluated on two liver tumor segmentation datasets. The experimental results demonstrate that our method can achieve competitive performance compared to the fully supervised counterpart and the state-of-the-art methods while requiring significantly less annotation effort.
Collapse
|
39
|
Introduction of Lazy Luna an automatic software-driven multilevel comparison of ventricular function quantification in cardiovascular magnetic resonance imaging. Sci Rep 2022; 12:6629. [PMID: 35459270 PMCID: PMC9033783 DOI: 10.1038/s41598-022-10464-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2022] [Accepted: 04/06/2022] [Indexed: 11/25/2022] Open
Abstract
Cardiovascular magnetic resonance imaging is the gold standard for cardiac function assessment. Quantification of clinical results (CR) requires precise segmentation. Clinicians statistically compare CRs to ensure reproducibility. Convolutional Neural Network developers compare their results via metrics. Aim: Introducing software capable of automatic multilevel comparison. A multilevel analysis covering segmentations and CRs builds on a generic software backend. Metrics and CRs are calculated with geometric accuracy. Segmentations and CRs are connected to track errors and their effects. An interactive GUI makes the software accessible to different users. The software’s multilevel comparison was tested on a use case based on cardiac function assessment. The software shows good reader agreement in CRs and segmentation metrics (Dice > 90%). Decomposing differences by cardiac position revealed excellent agreement in midventricular slices: > 90% but poorer segmentations in apical (> 71%) and basal slices (> 74%). Further decomposition by contour type locates the largest millilitre differences in the basal right cavity (> 3 ml). Visual inspection shows these differences being caused by different basal slice choices. The software illuminated reader differences on several levels. Producing spreadsheets and figures concerning metric values and CR differences was automated. A multilevel reader comparison is feasible and extendable to other cardiac structures in the future.
Collapse
|
40
|
Yan Y, Shi M, Fannin R, Yu L, Liu J, Castro L, Dixon D. Prolonged Cadmium Exposure Alters Migration Dynamics and Increases Heterogeneity of Human Uterine Fibroid Cells—Insights from Time Lapse Analysis. Biomedicines 2022; 10:biomedicines10040917. [PMID: 35453667 PMCID: PMC9031958 DOI: 10.3390/biomedicines10040917] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2022] [Revised: 04/03/2022] [Accepted: 04/05/2022] [Indexed: 11/16/2022] Open
Abstract
Cadmium (Cd) is one of the most prevalent environmental heavy metal contaminants and is considered an endocrine disruptor and carcinogen. In women with uterine fibroids, there is a correlation between blood Cd levels and fibroid tumor size. In this study, fibroid cells were exposed to 10 µM CdCl2 for 6 months and a fast-growing Cd-Resistant Leiomyoma culture, termed CR-LM6, was recovered. To characterize the morphological and mechanodynamic features of uterine fibroid cells associated with prolonged Cd exposure, we conducted time lapse imaging using a Zeiss confocal microscope and analyzed data by Imaris and RStudio. Our experiments recorded more than 64,000 trackable nuclear surface objects, with each having multiple parameters such as nuclear size and shape, speed, location, orientation, track length, and track straightness. Quantitative analysis revealed that prolonged Cd exposure significantly altered cell migration behavior, such as increased track length and reduced track straightness. Cd exposure also significantly increased the heterogeneity in nuclear size. Additionally, Cd significantly increased the median and variance of instantaneous speed, indicating that Cd exposure results in higher speed and greater variation in motility. Profiling of mRNA by NanoString analysis and Ingenuity Pathway Analysis (IPA) strongly suggested that the direction of gene expression changes due to Cd exposure enhanced cell movement and invasion. The altered expression of extracellular matrix (ECM) genes such as collagens, matrix metallopeptidases (MMPs), secreted phosphoprotein 1 (SPP1), which are important for migration contact guidance, may be responsible for the greater heterogeneity. The significantly increased heterogeneity of nuclear size, speed, and altered migration patterns may be a prerequisite for fibroid cells to attain characteristics favorable for cancer progression, invasion, and metastasis.
Collapse
Affiliation(s)
- Yitang Yan
- Molecular Pathogenesis Group, Mechanistic Toxicology Branch, Division of the National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA; (Y.Y.); (L.Y.); (J.L.); (L.C.)
| | - Min Shi
- Biostatistics & Computational Biology Branch, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA;
| | - Rick Fannin
- Molecular Genomics Core Laboratory, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA;
| | - Linda Yu
- Molecular Pathogenesis Group, Mechanistic Toxicology Branch, Division of the National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA; (Y.Y.); (L.Y.); (J.L.); (L.C.)
| | - Jingli Liu
- Molecular Pathogenesis Group, Mechanistic Toxicology Branch, Division of the National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA; (Y.Y.); (L.Y.); (J.L.); (L.C.)
| | - Lysandra Castro
- Molecular Pathogenesis Group, Mechanistic Toxicology Branch, Division of the National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA; (Y.Y.); (L.Y.); (J.L.); (L.C.)
| | - Darlene Dixon
- Molecular Pathogenesis Group, Mechanistic Toxicology Branch, Division of the National Toxicology Program, National Institute of Environmental Health Sciences, National Institutes of Health, 111 TW Alexander Drive, Durham, NC 27709, USA; (Y.Y.); (L.Y.); (J.L.); (L.C.)
- Correspondence: ; Tel.: +1-984-287-3848
| |
Collapse
|
41
|
Bilodeau A, Delmas CVL, Parent M, De Koninck P, Durand A, Lavoie-Cardinal F. Microscopy analysis neural network to solve detection, enumeration and segmentation from image-level annotations. NAT MACH INTELL 2022. [DOI: 10.1038/s42256-022-00472-w] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
|
42
|
Astley JR, Wild JM, Tahir BA. Deep learning in structural and functional lung image analysis. Br J Radiol 2022; 95:20201107. [PMID: 33877878 PMCID: PMC9153705 DOI: 10.1259/bjr.20201107] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022] Open
Abstract
The recent resurgence of deep learning (DL) has dramatically influenced the medical imaging field. Medical image analysis applications have been at the forefront of DL research efforts applied to multiple diseases and organs, including those of the lungs. The aims of this review are twofold: (i) to briefly overview DL theory as it relates to lung image analysis; (ii) to systematically review the DL research literature relating to the lung image analysis applications of segmentation, reconstruction, registration and synthesis. The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. 479 studies were initially identified from the literature search with 82 studies meeting the eligibility criteria. Segmentation was the most common lung image analysis DL application (65.9% of papers reviewed). DL has shown impressive results when applied to segmentation of the whole lung and other pulmonary structures. DL has also shown great potential for applications in image registration, reconstruction and synthesis. However, the majority of published studies have been limited to structural lung imaging with only 12.9% of reviewed studies employing functional lung imaging modalities, thus highlighting significant opportunities for further research in this field. Although the field of DL in lung image analysis is rapidly expanding, concerns over inconsistent validation and evaluation strategies, intersite generalisability, transparency of methodological detail and interpretability need to be addressed before widespread adoption in clinical lung imaging workflow.
Collapse
Affiliation(s)
| | - Jim M Wild
- Department of Oncology and Metabolism, The University of Sheffield, Sheffield, United Kingdom
| | | |
Collapse
|
43
|
Weakly-supervised learning for catheter segmentation in 3D frustum ultrasound. Comput Med Imaging Graph 2022; 96:102037. [DOI: 10.1016/j.compmedimag.2022.102037] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 11/15/2021] [Accepted: 01/13/2022] [Indexed: 11/21/2022]
|
44
|
Wang Z, Ma Y. Detection and recognition of stationary vehicles and seat belts in intelligent Internet of Things traffic management system. Neural Comput Appl 2022. [DOI: 10.1007/s00521-021-05870-6] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
45
|
Zhao L, Asis-Cruz JD, Feng X, Wu Y, Kapse K, Largent A, Quistorff J, Lopez C, Wu D, Qing K, Meyer C, Limperopoulos C. Automated 3D Fetal Brain Segmentation Using an Optimized Deep Learning Approach. AJNR Am J Neuroradiol 2022; 43:448-454. [PMID: 35177547 PMCID: PMC8910820 DOI: 10.3174/ajnr.a7419] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Accepted: 12/06/2021] [Indexed: 01/01/2023]
Abstract
BACKGROUND AND PURPOSE MR imaging provides critical information about fetal brain growth and development. Currently, morphologic analysis primarily relies on manual segmentation, which is time-intensive and has limited repeatability. This work aimed to develop a deep learning-based automatic fetal brain segmentation method that provides improved accuracy and robustness compared with atlas-based methods. MATERIALS AND METHODS A total of 106 fetal MR imaging studies were acquired prospectively from fetuses between 23 and 39 weeks of gestation. We trained a deep learning model on the MR imaging scans of 65 healthy fetuses and compared its performance with a 4D atlas-based segmentation method using the Wilcoxon signed-rank test. The trained model was also evaluated on data from 41 fetuses diagnosed with congenital heart disease. RESULTS The proposed method showed high consistency with the manual segmentation, with an average Dice score of 0.897. It also demonstrated significantly improved performance (P < .001) based on the Dice score and 95% Hausdorff distance in all brain regions compared with the atlas-based method. The performance of the proposed method was consistent across gestational ages. The segmentations of the brains of fetuses with high-risk congenital heart disease were also highly consistent with the manual segmentation, though the Dice score was 7% lower than that of healthy fetuses. CONCLUSIONS The proposed deep learning method provides an efficient and reliable approach for fetal brain segmentation, which outperformed segmentation based on a 4D atlas and has been used in clinical and research settings.
Collapse
Affiliation(s)
- L Zhao
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
- Department of Biomedical Engineering (L.Z., D.W.), Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering & Instrument Science, Zhejiang University, China
| | - J D Asis-Cruz
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| | - X Feng
- Department of Biomedical Engineering (X.F., C.M.), University of Virginia, Charlottesville, Virginia
| | - Y Wu
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| | - K Kapse
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| | - A Largent
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| | - J Quistorff
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| | - C Lopez
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| | - D Wu
- Department of Biomedical Engineering (L.Z., D.W.), Key Laboratory for Biomedical Engineering of Ministry of Education, College of Biomedical Engineering & Instrument Science, Zhejiang University, China
| | - K Qing
- Department of Radiation Oncology (K.Q.), City of Hope National Center, Duarte, California
| | - C Meyer
- Department of Biomedical Engineering (X.F., C.M.), University of Virginia, Charlottesville, Virginia
| | - C Limperopoulos
- From the Department of Diagnostic Imaging and Radiology (L.Z., J.D.A.-C., Y.W., K.K., A.L., J.Q., C. Lopez, C. Limperopoulos), Developing Brain Institute, Children's National, Washington, DC
| |
Collapse
|
46
|
A Multi-Agent Deep Reinforcement Learning Approach for Enhancement of COVID-19 CT Image Segmentation. J Pers Med 2022; 12:jpm12020309. [PMID: 35207796 PMCID: PMC8880720 DOI: 10.3390/jpm12020309] [Citation(s) in RCA: 31] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Revised: 02/14/2022] [Accepted: 02/15/2022] [Indexed: 11/18/2022] Open
Abstract
Currently, most mask extraction techniques are based on convolutional neural networks (CNNs). However, there are still numerous problems that mask extraction techniques need to solve. Thus, the most advanced methods to deploy artificial intelligence (AI) techniques are necessary. The use of cooperative agents in mask extraction increases the efficiency of automatic image segmentation. Hence, we introduce a new mask extraction method that is based on multi-agent deep reinforcement learning (DRL) to minimize the long-term manual mask extraction and to enhance medical image segmentation frameworks. A DRL-based method is introduced to deal with mask extraction issues. This new method utilizes a modified version of the Deep Q-Network to enable the mask detector to select masks from the image studied. Based on COVID-19 computed tomography (CT) images, we used DRL mask extraction-based techniques to extract visual features of COVID-19 infected areas and provide an accurate clinical diagnosis while optimizing the pathogenic diagnostic test and saving time. We collected CT images of different cases (normal chest CT, pneumonia, typical viral cases, and cases of COVID-19). Experimental validation achieved a precision of 97.12% with a Dice of 80.81%, a sensitivity of 79.97%, a specificity of 99.48%, a precision of 85.21%, an F1 score of 83.01%, a structural metric of 84.38%, and a mean absolute error of 0.86%. Additionally, the results of the visual segmentation clearly reflected the ground truth. The results reveal the proof of principle for using DRL to extract CT masks for an effective diagnosis of COVID-19.
Collapse
|
47
|
Protonotarios NE, Katsamenis I, Sykiotis S, Dikaios N, Kastis GA, Chatziioannou SN, Metaxas M, Doulamis N, Doulamis A. A few-shot U-Net deep learning model for lung cancer lesion segmentation via PET/CT imaging. Biomed Phys Eng Express 2022; 8. [PMID: 35144242 DOI: 10.1088/2057-1976/ac53bd] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2021] [Accepted: 02/10/2022] [Indexed: 11/12/2022]
Abstract
Over the past few years, positron emission tomography/computed tomography (PET/CT) imaging for computer-aided diagnosis has received increasing attention. Supervised deep learning architectures are usually employed for the detection of abnormalities, with anatomical localization, especially in the case of CT scans. However, the main limitations of the supervised learning paradigm include (i) large amounts of data required for model training, and (ii) the assumption of fixed network weights upon training completion, implying that the performance of the model cannot be further improved after training. In order to overcome these limitations, we apply a few-shot learning (FSL) scheme. Contrary to traditional deep learning practices, in FSL the model is provided with less data during training. The model then utilizes end-user feedback after training to constantly improve its performance. We integrate FSL in a U-Net architecture for lung cancer lesion segmentation on PET/CT scans, allowing for dynamic model weight fine-tuning and resulting in an online supervised learning scheme. Constant online readjustments of the model weights according to the user's feedback, increase the detection and classification accuracy, especially in cases where low detection performance is encountered. Our proposed method is validated on the Lung-PET-CT-DX TCIA database. PET/CT scans from 87 patients were included in the dataset and were acquired 60 minutes after intravenous18F-FDG injection. Experimental results indicate the superiority of our approach compared to other state of the art methods.
Collapse
Affiliation(s)
- Nicholas E Protonotarios
- Department of Applied Mathematics and Theoretical Physics (DAMTP), University of Cambridge, University of Cambridge, Cambridge, CB3 0WA, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Iason Katsamenis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| | - Stavros Sykiotis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| | - Nikolaos Dikaios
- Mathematics Research Center, Academy of Athens, 4, Soranou Efesiou, Athens, 115 27, GREECE
| | - George Anthony Kastis
- Mathematics Research Center, Academy of Athens, 4, Soranou Efesiou, Athens, Attica, 115 27, GREECE
| | - Sofia N Chatziioannou
- PET/CT, Biomedical Research Foundation of the Academy of Athens, 4, Soranou Efesiou, Athens, Attica, 115 27, GREECE
| | - Marinos Metaxas
- PET/CT, Biomedical Research Foundation of the Academy of Athens, 4, Soranou Efesiou, Athens, Attica, 115 27, GREECE
| | - Nikolaos Doulamis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| | - Anastasios Doulamis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| |
Collapse
|
48
|
Abdou MA. Literature review: efficient deep neural networks techniques for medical image analysis. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-06960-9] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
|
49
|
|
50
|
Xiong H, Liu S, Sharan RV, Coiera E, Berkovsky S. Weak label based Bayesian U-Net for optic disc segmentation in fundus images. Artif Intell Med 2022; 126:102261. [DOI: 10.1016/j.artmed.2022.102261] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2021] [Revised: 01/18/2022] [Accepted: 02/20/2022] [Indexed: 01/27/2023]
|