1
|
Madadi Y, Abu-Serhan H, Yousefi S. Domain Adaptation-Based Deep Learning Model for Forecasting and Diagnosis of Glaucoma Disease. Biomed Signal Process Control 2024; 92:106061. [PMID: 38463435 PMCID: PMC10922017 DOI: 10.1016/j.bspc.2024.106061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
The main factor causing irreversible blindness is glaucoma. Early detection greatly reduces the risk of further vision loss. To address this problem, we developed a domain adaptation-based deep learning model called Glaucoma Domain Adaptation (GDA) based on 66,742 fundus photographs collected from 3272 eyes of 1636 subjects. GDA learns domain-invariant and domain-specific representations to extract both general and specific features. We also developed a progressive weighting mechanism to accurately transfer the source domain knowledge while mitigating the transfer of negative knowledge from the source to the target domain. We employed low-rank coding for aligning the source and target distributions. We trained GDA based on three different scenarios including eyes annotated as glaucoma due to 1) optic disc abnormalities regardless of visual field abnormalities, 2) optic disc or visual field abnormalities except ones that are glaucoma due to both optic disc and visual field abnormalities at the same time, and 3) visual field abnormalities regardless of optic disc abnormalities We then evaluate the generalizability of GDA based on two independent datasets. The AUCs of GDA in forecasting glaucoma based on the first, second, and third scenarios were 0.90, 0.88, and 0.80 and the Accuracies were 0.82, 0.78, and 0.72, respectively. The AUCs of GDA in diagnosing glaucoma based on the first, second, and third scenarios were 0.98, 0.96, and 0.93 and the Accuracies were 0.93, 0.91, and 0.88, respectively. The proposed GDA model achieved high performance and generalizability for forecasting and diagnosis of glaucoma disease from fundus photographs. GDA may augment glaucoma research and clinical practice in identifying patients with glaucoma and forecasting those who may develop glaucoma thus preventing future vision loss.
Collapse
Affiliation(s)
- Yeganeh Madadi
- Department of Ophthalmology, University of Tennessee Health Science Center, Memphis, TN, USA
| | | | - Siamak Yousefi
- Department of Ophthalmology, University of Tennessee Health Science Center, Memphis, TN, USA
- Department of Genetics, Genomics, and Informatics, University of Tennessee Health Science Center, Memphis, TN, USA
| |
Collapse
|
2
|
Zhu S, Wu C, Du B, Zhang L. Adversarial pair-wise distribution matching for remote sensing image cross-scene classification. Neural Netw 2024; 174:106241. [PMID: 38508050 DOI: 10.1016/j.neunet.2024.106241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Revised: 10/18/2023] [Accepted: 03/13/2024] [Indexed: 03/22/2024]
Abstract
Remarkable achievements have been made in the field of remote sensing cross-scene classification in recent years. However, most methods directly align the entire image features for cross-scene knowledge transfer. They usually ignore the high background complexity and low category consistency of remote sensing images, which can significantly impair the performance of distribution alignment. Besides, shortcomings of the adversarial training paradigm and the inability to guarantee the prediction discriminability and diversity can also hinder cross-scene classification performance. To alleviate the above problems, we propose a novel cross-scene classification framework in a discriminator-free adversarial paradigm, called Adversarial Pair-wise Distribution Matching (APDM), to avoid irrelevant knowledge transfer and enable effective cross-domain modeling. Specifically, we propose the pair-wise cosine discrepancy for both inter-domain and intra-domain prediction measurements to fully leverage the prediction information, which can suppress negative semantic features and implicitly align the cross-scene distributions. Nuclear-norm maximization and minimization are introduced to enhance the target prediction quality and increase the applicability of the source knowledge, respectively. As a general cross-scene framework, APDM can be easily embedded with existing methods to boost the performance. Experimental results and analyses demonstrate that APDM can achieve competitive and effective performance on cross-scene classification tasks.
Collapse
Affiliation(s)
- Sihan Zhu
- The State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan 430079, China.
| | - Chen Wu
- The State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan 430079, China.
| | - Bo Du
- The National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan 430079, China.
| | - Liangpei Zhang
- The State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan 430079, China.
| |
Collapse
|
3
|
Fang Y, Yap PT, Lin W, Zhu H, Liu M. Source-free unsupervised domain adaptation: A survey. Neural Netw 2024; 174:106230. [PMID: 38490115 PMCID: PMC11015964 DOI: 10.1016/j.neunet.2024.106230] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 01/14/2024] [Accepted: 03/07/2024] [Indexed: 03/17/2024]
Abstract
Unsupervised domain adaptation (UDA) via deep learning has attracted appealing attention for tackling domain-shift problems caused by distribution discrepancy across different domains. Existing UDA approaches highly depend on the accessibility of source domain data, which is usually limited in practical scenarios due to privacy protection, data storage and transmission cost, and computation burden. To tackle this issue, many source-free unsupervised domain adaptation (SFUDA) methods have been proposed recently, which perform knowledge transfer from a pre-trained source model to the unlabeled target domain with source data inaccessible. A comprehensive review of these works on SFUDA is of great significance. In this paper, we provide a timely and systematic literature review of existing SFUDA approaches from a technical perspective. Specifically, we categorize current SFUDA studies into two groups, i.e., white-box SFUDA and black-box SFUDA, and further divide them into finer subcategories based on different learning strategies they use. We also investigate the challenges of methods in each subcategory, discuss the advantages/disadvantages of white-box and black-box SFUDA methods, conclude the commonly used benchmark datasets, and summarize the popular techniques for improved generalizability of models learned without using source data. We finally discuss several promising future directions in this field.
Collapse
Affiliation(s)
- Yuqi Fang
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States
| | - Pew-Thian Yap
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States
| | - Weili Lin
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States
| | - Hongtu Zhu
- Department of Biostatistics and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States
| | - Mingxia Liu
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, United States.
| |
Collapse
|
4
|
Hetz MJ, Bucher TC, Brinker TJ. Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images. Med Image Anal 2024; 94:103149. [PMID: 38574542 DOI: 10.1016/j.media.2024.103149] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Revised: 12/11/2023] [Accepted: 03/20/2024] [Indexed: 04/06/2024]
Abstract
The variation in histologic staining between different medical centers is one of the most profound challenges in the field of computer-aided diagnosis. The appearance disparity of pathological whole slide images causes algorithms to become less reliable, which in turn impedes the wide-spread applicability of downstream tasks like cancer diagnosis. Furthermore, different stainings lead to biases in the training which in case of domain shifts negatively affect the test performance. Therefore, in this paper we propose MultiStain-CycleGAN, a multi-domain approach to stain normalization based on CycleGAN. Our modifications to CycleGAN allow us to normalize images of different origins without retraining or using different models. We perform an extensive evaluation of our method using various metrics and compare it to commonly used methods that are multi-domain capable. First, we evaluate how well our method fools a domain classifier that tries to assign a medical center to an image. Then, we test our normalization on the tumor classification performance of a downstream classifier. Furthermore, we evaluate the image quality of the normalized images using the Structural similarity index and the ability to reduce the domain shift using the Fréchet inception distance. We show that our method proves to be multi-domain capable, provides a very high image quality among the compared methods, and can most reliably fool the domain classifier while keeping the tumor classifier performance high. By reducing the domain influence, biases in the data can be removed on the one hand and the origin of the whole slide image can be disguised on the other, thus enhancing patient data privacy.
Collapse
Affiliation(s)
- Martin J Hetz
- Division of Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tabea-Clara Bucher
- Division of Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Division of Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
5
|
Jiang S, Chen Q, Xiang Y, Pan Y, Wu X, Lin Y. Confounder balancing in adversarial domain adaptation for pre-trained large models fine-tuning. Neural Netw 2024; 173:106173. [PMID: 38387200 DOI: 10.1016/j.neunet.2024.106173] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Revised: 01/07/2024] [Accepted: 02/08/2024] [Indexed: 02/24/2024]
Abstract
The excellent generalization, contextual learning, and emergence abilities in the pre-trained large models (PLMs) handle specific tasks without direct training data, making them the better foundation models in the adversarial domain adaptation (ADA) methods to transfer knowledge learned from the source domain to target domains. However, existing ADA methods fail to account for the confounder properly, which is the root cause of the source data distribution that differs from the target domains. This study proposes a confounder balancing method in adversarial domain adaptation for PLMs fine-tuning (CadaFT), which includes a PLM as the foundation model for a feature extractor, a domain classifier and a confounder classifier, and they are jointly trained with an adversarial loss. This loss is designed to improve the domain-invariant representation learning by diluting the discrimination in the domain classifier. At the same time, the adversarial loss also balances the confounder distribution among source and unmeasured domains in training. Compared to newest ADA methods, CadaFT can correctly identify confounders in domain-invariant features, thereby eliminating the confounder biases in the extracted features from PLMs. The confounder classifier in CadaFT is designed as a plug-and-play and can be applied in the confounder measurable, unmeasurable, or partially measurable environments. Empirical results on natural language processing and computer vision downstream tasks show that CadaFT outperforms the newest GPT-4, LLaMA2, ViT and ADA methods.
Collapse
Affiliation(s)
- Shuoran Jiang
- Haibin Institute of Technology, ShenZhen, Harbin Institute of Technology campus, Taoyuan Street, Nanshan District, Shenzhen, 518055, GuangDong, China
| | - Qingcai Chen
- Haibin Institute of Technology, ShenZhen, Harbin Institute of Technology campus, Taoyuan Street, Nanshan District, Shenzhen, 518055, GuangDong, China; Peng Cheng Laboratory, No. 2, Xingke 1st Street, Nanshan District, Shenzhen, 518055, Guangdong, China.
| | - Yang Xiang
- Peng Cheng Laboratory, No. 2, Xingke 1st Street, Nanshan District, Shenzhen, 518055, Guangdong, China.
| | - Youcheng Pan
- Peng Cheng Laboratory, No. 2, Xingke 1st Street, Nanshan District, Shenzhen, 518055, Guangdong, China
| | - Xiangping Wu
- Haibin Institute of Technology, ShenZhen, Harbin Institute of Technology campus, Taoyuan Street, Nanshan District, Shenzhen, 518055, GuangDong, China
| | - Yukang Lin
- Haibin Institute of Technology, ShenZhen, Harbin Institute of Technology campus, Taoyuan Street, Nanshan District, Shenzhen, 518055, GuangDong, China
| |
Collapse
|
6
|
Steingrimsson JA, Barker DH, Bie R, Dahabreh IJ. Systematically missing data in causally interpretable meta-analysis. Biostatistics 2024; 25:289-305. [PMID: 36977366 PMCID: PMC11017122 DOI: 10.1093/biostatistics/kxad006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 02/15/2023] [Accepted: 03/13/2023] [Indexed: 03/30/2023] Open
Abstract
Causally interpretable meta-analysis combines information from a collection of randomized controlled trials to estimate treatment effects in a target population in which experimentation may not be possible but from which covariate information can be obtained. In such analyses, a key practical challenge is the presence of systematically missing data when some trials have collected data on one or more baseline covariates, but other trials have not, such that the covariate information is missing for all participants in the latter. In this article, we provide identification results for potential (counterfactual) outcome means and average treatment effects in the target population when covariate data are systematically missing from some of the trials in the meta-analysis. We propose three estimators for the average treatment effect in the target population, examine their asymptotic properties, and show that they have good finite-sample performance in simulation studies. We use the estimators to analyze data from two large lung cancer screening trials and target population data from the National Health and Nutrition Examination Survey (NHANES). To accommodate the complex survey design of the NHANES, we modify the methods to incorporate survey sampling weights and allow for clustering.
Collapse
Affiliation(s)
- Jon A Steingrimsson
- Department of Biostatistics, Brown University, 121 South Main Street, Providence, RI 02903, USA
| | - David H Barker
- Department of Psychiatry, Rhode Island Hospital, Providence, RI 02904, USA
| | - Ruofan Bie
- Department of Biostatistics, Brown University, 121 South Main Street, Providence, RI 02903, USA
| | - Issa J Dahabreh
- Departments of Epidemiology and Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA and CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA
| |
Collapse
|
7
|
Zeng X, Chen W, Lei B. CAT-DTI: cross-attention and Transformer network with domain adaptation for drug-target interaction prediction. BMC Bioinformatics 2024; 25:141. [PMID: 38566002 DOI: 10.1186/s12859-024-05753-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2024] [Accepted: 03/19/2024] [Indexed: 04/04/2024] Open
Abstract
Accurate and efficient prediction of drug-target interaction (DTI) is critical to advance drug development and reduce the cost of drug discovery. Recently, the employment of deep learning methods has enhanced DTI prediction precision and efficacy, but it still encounters several challenges. The first challenge lies in the efficient learning of drug and protein feature representations alongside their interaction features to enhance DTI prediction. Another important challenge is to improve the generalization capability of the DTI model within real-world scenarios. To address these challenges, we propose CAT-DTI, a model based on cross-attention and Transformer, possessing domain adaptation capability. CAT-DTI effectively captures the drug-target interactions while adapting to out-of-distribution data. Specifically, we use a convolution neural network combined with a Transformer to encode the distance relationship between amino acids within protein sequences and employ a cross-attention module to capture the drug-target interaction features. Generalization to new DTI prediction scenarios is achieved by leveraging a conditional domain adversarial network, aligning DTI representations under diverse distributions. Experimental results within in-domain and cross-domain scenarios demonstrate that CAT-DTI model overall improves DTI prediction performance compared with previous methods.
Collapse
Affiliation(s)
- Xiaoting Zeng
- School of Computer and Software, Shenzhen University, Shenzhen, 518060, China
| | - Weilin Chen
- Marshall Laboratory of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, 518055, China.
| | - Baiying Lei
- School of Biomedical Engineering, Shenzhen University, Shenzhen, 518055, China.
| |
Collapse
|
8
|
Ren CX, Xu GX, Dai DQ, Lin L, Sun Y, Liu QS. Cross-site prognosis prediction for nasopharyngeal carcinoma from incomplete multi-modal data. Med Image Anal 2024; 93:103103. [PMID: 38368752 DOI: 10.1016/j.media.2024.103103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 12/05/2023] [Accepted: 02/05/2024] [Indexed: 02/20/2024]
Abstract
Accurate prognosis prediction for nasopharyngeal carcinoma based on magnetic resonance (MR) images assists in the guidance of treatment intensity, thus reducing the risk of recurrence and death. To reduce repeated labor and sufficiently explore domain knowledge, aggregating labeled/annotated data from external sites enables us to train an intelligent model for a clinical site with unlabeled data. However, this task suffers from the challenges of incomplete multi-modal examination data fusion and image data heterogeneity among sites. This paper proposes a cross-site survival analysis method for prognosis prediction of nasopharyngeal carcinoma from domain adaptation viewpoint. Utilizing a Cox model as the basic framework, our method equips it with a cross-attention based multi-modal fusion regularization. This regularization model effectively fuses the multi-modal information from multi-parametric MR images and clinical features onto a domain-adaptive space, despite the absence of some modalities. To enhance the feature discrimination, we also extend the contrastive learning technique to censored data cases. Compared with the conventional approaches which directly deploy a trained survival model in a new site, our method achieves superior prognosis prediction performance in cross-site validation experiments. These results highlight the key role of cross-site adaptability of our method and support its value in clinical practice.
Collapse
Affiliation(s)
- Chuan-Xian Ren
- School of Mathematics, Sun Yat-sen University, Guangzhou 510275, China.
| | - Geng-Xin Xu
- School of Mathematics, Sun Yat-sen University, Guangzhou 510275, China
| | - Dao-Qing Dai
- School of Mathematics, Sun Yat-sen University, Guangzhou 510275, China
| | - Li Lin
- Department of Radiation Oncology, Sun Yat-sen University Cancer Center; State Key Laboratory of Oncology in South China; Collaborative Innovation Center for Cancer Medicine; Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy, Guangzhou 510060, China
| | - Ying Sun
- Department of Radiation Oncology, Sun Yat-sen University Cancer Center; State Key Laboratory of Oncology in South China; Collaborative Innovation Center for Cancer Medicine; Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy, Guangzhou 510060, China
| | - Qing-Shan Liu
- School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
| |
Collapse
|
9
|
Chen J, Wang J, Lin W, Zhang K, de Silva CW. Preserving domain private information via mutual information maximization. Neural Netw 2024; 172:106112. [PMID: 38218025 DOI: 10.1016/j.neunet.2024.106112] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Revised: 10/15/2023] [Accepted: 01/05/2024] [Indexed: 01/15/2024]
Abstract
Recent advances in unsupervised domain adaptation have shown that mitigating the domain divergence by extracting the domain-invariant features could significantly improve the generalization of a model with respect to a new data domain. However, current methodologies often neglect to retain domain private information, which is the unique information inherent to the unlabeled new domain, compromising generalization. This paper presents a novel method that utilizes mutual information to protect this domain-specific information, ensuring that the latent features of the unlabeled data not only remain domain-invariant but also reflect the unique statistics of the unlabeled domain. We show that simultaneous maximization of mutual information and reduction of domain divergence can effectively preserve domain-private information. We further illustrate that a neural estimator can aptly estimate the mutual information between the unlabeled input space and its latent feature space. Both theoretical analysis and empirical results validate the significance of preserving such unique information of the unlabeled domain for cross-domain generalization. Comparative evaluations reveal our method's superiority over existing state-of-the-art techniques across multiple benchmark datasets.
Collapse
Affiliation(s)
- Jiahong Chen
- Department of Mechanical Engineering, University of British Columbia, Vancouver, BC, Canada.
| | - Jing Wang
- Department of Mechanical Engineering, University of British Columbia, Vancouver, BC, Canada.
| | - Weipeng Lin
- School of Artificial Intelligence, Shenzhen Polytechnic, Shenzhen, Guangdong, China.
| | - Kuangen Zhang
- Department of Mechanical Engineering, University of British Columbia, Vancouver, BC, Canada.
| | - Clarence W de Silva
- Department of Mechanical Engineering, University of British Columbia, Vancouver, BC, Canada.
| |
Collapse
|
10
|
Abad M, Casas-Roma J, Prados F. Generalizable disease detection using model ensemble on chest X-ray images. Sci Rep 2024; 14:5890. [PMID: 38467705 PMCID: PMC10928229 DOI: 10.1038/s41598-024-56171-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Accepted: 03/03/2024] [Indexed: 03/13/2024] Open
Abstract
In the realm of healthcare, the demand for swift and precise diagnostic tools has been steadily increasing. This study delves into a comprehensive performance analysis of three pre-trained convolutional neural network (CNN) architectures: ResNet50, DenseNet121, and Inception-ResNet-v2. To ensure the broad applicability of our approach, we curated a large-scale dataset comprising a diverse collection of chest X-ray images, that included both positive and negative cases of COVID-19. The models' performance was evaluated using separate datasets for internal validation (from the same source as the training images) and external validation (from different sources). Our examination uncovered a significant drop in network efficacy, registering a 10.66% reduction for ResNet50, a 36.33% decline for DenseNet121, and a 19.55% decrease for Inception-ResNet-v2 in terms of accuracy. Best results were obtained with DenseNet121 achieving the highest accuracy at 96.71% in internal validation and Inception-ResNet-v2 attaining 76.70% accuracy in external validation. Furthermore, we introduced a model ensemble approach aimed at improving network performance when making inferences on images from diverse sources beyond their training data. The proposed method uses uncertainty-based weighting by calculating the entropy in order to assign appropriate weights to the outputs of each network. Our results showcase the effectiveness of the ensemble method in enhancing accuracy up to 97.38% for internal validation and 81.18% for external validation, while maintaining a balanced ability to detect both positive and negative cases.
Collapse
Affiliation(s)
- Maider Abad
- Universitat Oberta de Catalunya, e-Health Center, Barcelona, Spain.
| | - Jordi Casas-Roma
- Universitat Oberta de Catalunya, e-Health Center, Barcelona, Spain
- Department of Computer Science, Universitat Autònoma de Barcelona (UAB), Bellaterra, Spain
- Computer Vision Center (CVC), Universitat Autònoma de Barcelona (UAB), Bellaterra, Spain
| | - Ferran Prados
- Universitat Oberta de Catalunya, e-Health Center, Barcelona, Spain
- Queen Square MS Centre, Department of Neuroinflammation, UCL Queen Square Institute of Neurology, Faculty of Brain Science, University College of London, London, WC1N 3BG, UK
- Centre for Medical Image Computing (CMIC), Department of Medical Physics and Bioengineering, University College London, London, WC1V 6LJ, UK
| |
Collapse
|
11
|
Gao Y, Ma C, Guo L, Liu G, Zhang X, Ji X. Adversarial learning-based domain adaptation algorithm for intracranial artery stenosis detection on multi-source datasets. Comput Biol Med 2024; 170:108001. [PMID: 38280254 DOI: 10.1016/j.compbiomed.2024.108001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Revised: 12/26/2023] [Accepted: 01/13/2024] [Indexed: 01/29/2024]
Abstract
Intracranial arterial stenosis (ICAS) is characterized by the pathological narrowing or occlusion of the inner lumen of intracranial blood vessels. However, the retina can indirectly react to cerebrovascular disease. Therefore, retinal fundus images (RFI) serve as valuable noninvasive and easily accessible screening tools for early detection and diagnosis of ICAS. This paper introduces an adversarial learning-based domain adaptation algorithm (ALDA) specifically designed for ICAS detection in multi-source datasets. The primary objective is to achieve accurate detection and enhanced generalization of ICAS based on RFI. Given the limitations of traditional algorithms in meeting the accuracy and generalization requirements, ALDA overcomes these challenges by leveraging RFI datasets from multiple sources and employing the concept of adversarial learning to facilitate feature representation sharing and distinguishability learning. In order to evaluate the performance of the ALDA algorithm, we conducted experimental validation on multi-source datasets. We compared its results with those obtained from other deep learning algorithms in the ICAS detection task. Furthermore, we validated the potential of ALDA for detecting diabetic retinopathy. The experimental results clearly demonstrate the significant improvements achieved by the ALDA algorithm. By leveraging information from diverse datasets, ALDA learns feature representations that exhibit enhanced generalizability and robustness. This makes it a reliable auxiliary diagnostic tool for clinicians, thereby facilitating the prevention and treatment of cerebrovascular diseases.
Collapse
Affiliation(s)
- Yuan Gao
- Department of Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, 100191, Beijing, China; Department of Ophthalmology, Xuanwu Hospital, Capital Medical University, 100053, Beijing, China.
| | - Chenbin Ma
- Department of Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, 100191, Beijing, China; Shen Yuan Honors College, Beihang University, 100191, Beijing, China.
| | - Lishuang Guo
- Department of Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, 100191, Beijing, China.
| | - Guiyou Liu
- Department of Ophthalmology, Beijing Tiantan Hospital, Capital Medical University, 100050, Beijing, China.
| | - Xuxiang Zhang
- Beijing Institute for Brain Disorders, Capital Medical University, 100069, Beijing, China.
| | - Xunming Ji
- Department of Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University, 100191, Beijing, China.
| |
Collapse
|
12
|
Zhang K, Lin PC, Pan J, Shao R, Xu PX, Cao R, Wu CG, Crookes D, Hua L, Wang L. DeepmdQCT: A multitask network with domain invariant features and comprehensive attention mechanism for quantitative computer tomography diagnosis of osteoporosis. Comput Biol Med 2024; 170:107916. [PMID: 38237237 DOI: 10.1016/j.compbiomed.2023.107916] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Revised: 12/18/2023] [Accepted: 12/29/2023] [Indexed: 02/28/2024]
Abstract
In the medical field, the application of machine learning technology in the automatic diagnosis and monitoring of osteoporosis often faces challenges related to domain adaptation in drug therapy research. The existing neural networks used for the diagnosis of osteoporosis may experience a decrease in model performance when applied to new data domains due to changes in radiation dose and equipment. To address this issue, in this study, we propose a new method for multi domain diagnostic and quantitative computed tomography (QCT) images, called DeepmdQCT. This method adopts a domain invariant feature strategy and integrates a comprehensive attention mechanism to guide the fusion of global and local features, effectively improving the diagnostic performance of multi domain CT images. We conducted experimental evaluations on a self-created OQCT dataset, and the results showed that for dose domain images, the average accuracy reached 91%, while for device domain images, the accuracy reached 90.5%. our method successfully estimated bone density values, with a fit of 0.95 to the gold standard. Our method not only achieved high accuracy in CT images in the dose and equipment fields, but also successfully estimated key bone density values, which is crucial for evaluating the effectiveness of osteoporosis drug treatment. In addition, we validated the effectiveness of our architecture in feature extraction using three publicly available datasets. We also encourage the application of the DeepmdQCT method to a wider range of medical image analysis fields to improve the performance of multi-domain images.
Collapse
Affiliation(s)
- Kun Zhang
- School of Electrical Engineering, Nantong University, Nantong, Jiangsu, 226001, China; Nantong Key Laboratory of Intelligent Control and Intelligent Computing, Nantong, Jiangsu, 226001, China; Nantong Key Laboratory of Intelligent Medicine Innovation and Transformation, Nantong, Jiangsu, 226001, China
| | - Peng-Cheng Lin
- School of Electrical Engineering, Nantong University, Nantong, Jiangsu, 226001, China
| | - Jing Pan
- Department of Radiology, Affiliated Hospital 2 of Nantong University, Nantong, Jiangsu, 226001, China
| | - Rui Shao
- School of Electrical Engineering, Nantong University, Nantong, Jiangsu, 226001, China
| | - Pei-Xia Xu
- School of Electrical Engineering, Nantong University, Nantong, Jiangsu, 226001, China
| | - Rui Cao
- Department of Radiology, Affiliated Hospital 2 of Nantong University, Nantong, Jiangsu, 226001, China
| | - Cheng-Gang Wu
- School of Electrical Engineering, Nantong University, Nantong, Jiangsu, 226001, China
| | - Danny Crookes
- School of Electronics, Electrical Engineering and Computer Science, Queen's University Belfast, Belfast, BT7 1NN, UK
| | - Liang Hua
- School of Electrical Engineering, Nantong University, Nantong, Jiangsu, 226001, China.
| | - Lin Wang
- Department of Radiology, Affiliated Hospital 2 of Nantong University, Nantong, Jiangsu, 226001, China.
| |
Collapse
|
13
|
Yu W, Xu N, Huang N, Chen H. Bridging the gap: Geometry-centric discriminative manifold distribution alignment for enhanced classification in colorectal cancer imaging. Comput Biol Med 2024; 170:107998. [PMID: 38266468 DOI: 10.1016/j.compbiomed.2024.107998] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Revised: 12/19/2023] [Accepted: 01/13/2024] [Indexed: 01/26/2024]
Abstract
The early detection of colorectal cancer (CRC) through medical image analysis is a pivotal concern in healthcare, with the potential to significantly reduce mortality rates. Current Domain Adaptation (DA) methods strive to mitigate the discrepancies between different imaging modalities that are critical in identifying CRC, yet they often fall short in addressing the complexity of cancer's presentation within these images. These conventional techniques typically overlook the intricate geometrical structures and the local variations within the data, leading to suboptimal diagnostic performance. This study introduces an innovative application of the Discriminative Manifold Distribution Alignment (DMDA) method, which is specifically engineered to enhance the medical image diagnosis of colorectal cancer. DMDA transcends traditional DA approaches by focusing on both local and global distribution alignments and by intricately learning the intrinsic geometrical characteristics present in manifold space. This is achieved without depending on the potentially misleading pseudo-labels, a common pitfall in existing methodologies. Our implementation of DMDA on three distinct datasets, involving several unique DA tasks, has consistently demonstrated superior classification accuracy and computational efficiency. The method adeptly captures the complex morphological and textural nuances of CRC lesions, leading to a significant leap in domain adaptation technology. DMDA's ability to reconcile global and local distributional disparities, coupled with its manifold-based geometrical structure learning, signals a paradigm shift in medical imaging analysis. The results obtained are not only promising in terms of advancing domain adaptation theory but also in their practical implications, offering the prospect of substantially improved diagnostic accuracy and faster clinical workflows. This heralds a transformative approach in personalized oncology care, aligning with the pressing need for early and accurate CRC detection.
Collapse
Affiliation(s)
- Weiwei Yu
- Department of Gastroenterology, Wenzhou TCM Hospital of Zhejiang Chinese Medical University, Wenzhou, Zhejiang, 325000, China.
| | - Nuo Xu
- Department of Medical Oncology, Wenzhou TCM Hospital of Zhejiang Chinese Medical University, Wenzhou, Zhejiang, 325000, China.
| | - Nuanhui Huang
- Department of Medical Oncology, Wenzhou TCM Hospital of Zhejiang Chinese Medical University, Wenzhou, Zhejiang, 325000, China.
| | - Houliang Chen
- Department of Medical Oncology, Wenzhou TCM Hospital of Zhejiang Chinese Medical University, Wenzhou, Zhejiang, 325000, China.
| |
Collapse
|
14
|
Kumari S, Singh P. Deep learning for unsupervised domain adaptation in medical imaging: Recent advancements and future perspectives. Comput Biol Med 2024; 170:107912. [PMID: 38219643 DOI: 10.1016/j.compbiomed.2023.107912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Revised: 11/02/2023] [Accepted: 12/24/2023] [Indexed: 01/16/2024]
Abstract
Deep learning has demonstrated remarkable performance across various tasks in medical imaging. However, these approaches primarily focus on supervised learning, assuming that the training and testing data are drawn from the same distribution. Unfortunately, this assumption may not always hold true in practice. To address these issues, unsupervised domain adaptation (UDA) techniques have been developed to transfer knowledge from a labeled domain to a related but unlabeled domain. In recent years, significant advancements have been made in UDA, resulting in a wide range of methodologies, including feature alignment, image translation, self-supervision, and disentangled representation methods, among others. In this paper, we provide a comprehensive literature review of recent deep UDA approaches in medical imaging from a technical perspective. Specifically, we categorize current UDA research in medical imaging into six groups and further divide them into finer subcategories based on the different tasks they perform. We also discuss the respective datasets used in the studies to assess the divergence between the different domains. Finally, we discuss emerging areas and provide insights and discussions on future research directions to conclude this survey.
Collapse
Affiliation(s)
- Suruchi Kumari
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, India.
| | - Pravendra Singh
- Department of Computer Science and Engineering, Indian Institute of Technology Roorkee, India.
| |
Collapse
|
15
|
S S, Dharani Devi G, V R, Jeyalakshmi J. Privacy-Preserving Breast Cancer Classification: A Federated Transfer Learning Approach. J Imaging Inform Med 2024:10.1007/s10278-024-01035-8. [PMID: 38424280 DOI: 10.1007/s10278-024-01035-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Revised: 01/11/2024] [Accepted: 01/30/2024] [Indexed: 03/02/2024]
Abstract
Breast cancer is deadly cancer causing a considerable number of fatalities among women in worldwide. To enhance patient outcomes as well as survival rates, early and accurate detection is crucial. Machine learning techniques, particularly deep learning, have demonstrated impressive success in various image recognition tasks, including breast cancer classification. However, the reliance on large labeled datasets poses challenges in the medical domain due to privacy issues and data silos. This study proposes a novel transfer learning approach integrated into a federated learning framework to solve the limitations of limited labeled data and data privacy in collaborative healthcare settings. For breast cancer classification, the mammography and MRO images were gathered from three different medical centers. Federated learning, an emerging privacy-preserving paradigm, empowers multiple medical institutions to jointly train the global model while maintaining data decentralization. Our proposed methodology capitalizes on the power of pre-trained ResNet, a deep neural network architecture, as a feature extractor. By fine-tuning the higher layers of ResNet using breast cancer datasets from diverse medical centers, we enable the model to learn specialized features relevant to different domains while leveraging the comprehensive image representations acquired from large-scale datasets like ImageNet. To overcome domain shift challenges caused by variations in data distributions across medical centers, we introduce domain adversarial training. The model learns to minimize the domain discrepancy while maximizing classification accuracy, facilitating the acquisition of domain-invariant features. We conducted extensive experiments on diverse breast cancer datasets obtained from multiple medical centers. Comparative analysis was performed to evaluate the proposed approach against traditional standalone training and federated learning without domain adaptation. When compared with traditional models, our proposed model showed a classification accuracy of 98.8% and a computational time of 12.22 s. The results showcase promising enhancements in classification accuracy and model generalization, underscoring the potential of our method in improving breast cancer classification performance while upholding data privacy in a federated healthcare environment.
Collapse
Affiliation(s)
- Selvakanmani S
- Department of Information Technology, R.M.K Engineering College, Chennai, Tamil Nadu, India.
| | - G Dharani Devi
- Department of Computer Science and Engineering, Rajalakshmi Engineering College, Chennai, Tamil Nadu, India
| | - Rekha V
- Department of Artificial Intelligence and Data Science, Panimalar Engineering College, Chennai, Tamil Nadu, India
| | - J Jeyalakshmi
- Department of Computer Science and Engineering, Amrita School of Computing, Amrita Vishwa Vidhyapeetham, Chennai, India
| |
Collapse
|
16
|
Davidashvilly S, Cardei M, Hssayeni M, Chi C, Ghoraani B. Deep neural networks for wearable sensor-based activity recognition in Parkinson's disease: investigating generalizability and model complexity. Biomed Eng Online 2024; 23:17. [PMID: 38336781 PMCID: PMC10858599 DOI: 10.1186/s12938-024-01214-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Accepted: 01/25/2024] [Indexed: 02/12/2024] Open
Abstract
BACKGROUND The research gap addressed in this study is the applicability of deep neural network (NN) models on wearable sensor data to recognize different activities performed by patients with Parkinson's Disease (PwPD) and the generalizability of these models to PwPD using labeled healthy data. METHODS The experiments were carried out utilizing three datasets containing wearable motion sensor readings on common activities of daily living. The collected readings were from two accelerometer sensors. PAMAP2 and MHEALTH are publicly available datasets collected from 10 and 9 healthy, young subjects, respectively. A private dataset of a similar nature collected from 14 PwPD patients was utilized as well. Deep NN models were implemented with varying levels of complexity to investigate the impact of data augmentation, manual axis reorientation, model complexity, and domain adaptation on activity recognition performance. RESULTS A moderately complex model trained on the augmented PAMAP2 dataset and adapted to the Parkinson domain using domain adaptation achieved the best activity recognition performance with an accuracy of 73.02%, which was significantly higher than the accuracy of 63% reported in previous studies. The model's F1 score of 49.79% significantly improved compared to the best cross-testing of 33.66% F1 score with only data augmentation and 2.88% F1 score without data augmentation or domain adaptation. CONCLUSION These findings suggest that deep NN models originating on healthy data have the potential to recognize activities performed by PwPD accurately and that data augmentation and domain adaptation can improve the generalizability of models in the healthy-to-PwPD transfer scenario. The simple/moderately complex architectures tested in this study could generalize better to the PwPD domain when trained on a healthy dataset compared to the most complex architectures used. The findings of this study could contribute to the development of accurate wearable-based activity monitoring solutions for PwPD, improving clinical decision-making and patient outcomes based on patient activity levels.
Collapse
Affiliation(s)
- Shelly Davidashvilly
- Electrical and Computer Engineering, Florida Atlantic University, Boca Raton, FL, 33431, US
| | - Maria Cardei
- Electrical and Computer Engineering, Florida Atlantic University, Boca Raton, FL, 33431, US
- Biomedical Engineering, University of Florida, Gainesville, FL, US
| | - Murtadha Hssayeni
- Electrical and Computer Engineering, Florida Atlantic University, Boca Raton, FL, 33431, US
- Computer Engineering, University of Technology, Baghdad, Iraq
| | - Christopher Chi
- Electrical and Computer Engineering, Florida Atlantic University, Boca Raton, FL, 33431, US
| | - Behnaz Ghoraani
- Electrical and Computer Engineering, Florida Atlantic University, Boca Raton, FL, 33431, US.
| |
Collapse
|
17
|
Hognon C, Conze PH, Bourbonne V, Gallinato O, Colin T, Jaouen V, Visvikis D. Contrastive image adaptation for acquisition shift reduction in medical imaging. Artif Intell Med 2024; 148:102747. [PMID: 38325919 DOI: 10.1016/j.artmed.2023.102747] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2022] [Revised: 10/21/2023] [Accepted: 12/10/2023] [Indexed: 02/09/2024]
Abstract
The domain shift, or acquisition shift in medical imaging, is responsible for potentially harmful differences between development and deployment conditions of medical image analysis techniques. There is a growing need in the community for advanced methods that could mitigate this issue better than conventional approaches. In this paper, we consider configurations in which we can expose a learning-based pixel level adaptor to a large variability of unlabeled images during its training, i.e. sufficient to span the acquisition shift expected during the training or testing of a downstream task model. We leverage the ability of convolutional architectures to efficiently learn domain-agnostic features and train a many-to-one unsupervised mapping between a source collection of heterogeneous images from multiple unknown domains subjected to the acquisition shift and a homogeneous subset of this source set of lower cardinality, potentially constituted of a single image. To this end, we propose a new cycle-free image-to-image architecture based on a combination of three loss functions : a contrastive PatchNCE loss, an adversarial loss and an edge preserving loss allowing for rich domain adaptation to the target image even under strong domain imbalance and low data regimes. Experiments support the interest of the proposed contrastive image adaptation approach for the regularization of downstream deep supervised segmentation and cross-modality synthesis models.
Collapse
Affiliation(s)
- Clément Hognon
- UMR U1101 Inserm LaTIM, IMT Atlantique, Université de Bretagne Occidentale, France; SOPHiA Genetics, Pessac, France
| | - Pierre-Henri Conze
- UMR U1101 Inserm LaTIM, IMT Atlantique, Université de Bretagne Occidentale, France
| | - Vincent Bourbonne
- UMR U1101 Inserm LaTIM, IMT Atlantique, Université de Bretagne Occidentale, France
| | | | | | - Vincent Jaouen
- UMR U1101 Inserm LaTIM, IMT Atlantique, Université de Bretagne Occidentale, France.
| | - Dimitris Visvikis
- UMR U1101 Inserm LaTIM, IMT Atlantique, Université de Bretagne Occidentale, France
| |
Collapse
|
18
|
Zhu L, Yu F, Huang A, Ying N, Zhang J. Instance-representation transfer method based on joint distribution and deep adaptation for EEG emotion recognition. Med Biol Eng Comput 2024; 62:479-493. [PMID: 37914959 DOI: 10.1007/s11517-023-02956-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Accepted: 10/20/2023] [Indexed: 11/03/2023]
Abstract
Electroencephalogram (EEG) emotion recognition technology is essential for improving human-computer interaction. However, the practical application of emotion recognition technology is limited due to the variety of subjects and sessions. Transfer learning has been applied to address this issue and has received extensive research and application. Studies mainly concentrate on either instance transfer or representation transfer methods. This paper proposes an emotion recognition method called Joint Distributed Instances Represent Transfer (JD-IRT), which includes two core components: Joint Distribution Deep Adaptation (JDDA) and Instance-Representation Transfer (I-RT). Specifically, JDDA is different from common representation transfer methods in transfer learning. It bridges the discrepancies of marginal and conditional distributions simultaneously and combines multiple adaptive layers and kernels for deep domain adaptation. On the other hand, I-RT utilizes instance transfer to select source domain data for better representation transfer. We performed experiments and compared them with other representative methods in the SEED, SEED-IV, and SEED-V datasets. In cross-subject experiments, our approach achieved an average accuracy of 83.21% in SEED, 52.12% in SEED-IV, and 60.17% in SEED-V. Similarly, in cross-session experiments, the accuracy was 91.29% in SEED, 59.02% in SEED-IV, and 65.91% in SEED-V. These results demonstrate the improvement in the accuracy of EEG emotion recognition using the proposed approach.
Collapse
Affiliation(s)
- Lei Zhu
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310000, China.
| | - Fei Yu
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310000, China
| | - Aiai Huang
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310000, China
| | - Nanjiao Ying
- School of Automation, Hangzhou Dianzi University, Hangzhou, 310000, China
- Center for Drug Inspection of Zhejiang Province, Hangzhou, 310000, China
| | - Jianhai Zhang
- School of Computer Science, Hangzhou Dianzi University, Hangzhou, 310000, China
- Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province, Hangzhou, 310000, China
| |
Collapse
|
19
|
Velikova Y, Simson W, Azampour MF, Paprottka P, Navab N. CACTUSS: Common Anatomical CT-US Space for US examinations. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03060-y. [PMID: 38270811 DOI: 10.1007/s11548-024-03060-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 01/03/2024] [Indexed: 01/26/2024]
Abstract
PURPOSE The detection and treatment of abdominal aortic aneurysm (AAA), a vascular disorder with life-threatening consequences, is challenging due to its lack of symptoms until it reaches a critical size. Abdominal ultrasound (US) is utilized for diagnosis; however, its inherent low image quality and reliance on operator expertise make computed tomography (CT) the preferred choice for monitoring and treatment. Moreover, CT datasets have been effectively used for training deep neural networks for aorta segmentation. In this work, we demonstrate how leveraging CT labels can be used to improve segmentation in ultrasound and hence save manual annotations. METHODS We introduce CACTUSS: a common anatomical CT-US space that inherits properties from both CT and ultrasound modalities to produce an image in intermediate representation (IR) space. CACTUSS acts as a virtual third modality between CT and US to address the scarcity of annotated ultrasound training data. The generation of IR images is facilitated by re-parametrizing a physics-based US simulator. In CACTUSS we use IR images as training data for ultrasound segmentation, eliminating the need for manual labeling. In addition, an image-to-image translation network is employed for the model's application on real B-modes. RESULTS The model's performance is evaluated quantitatively for the task of aorta segmentation by comparison against a fully supervised method in terms of Dice Score and diagnostic metrics. CACTUSS outperforms the fully supervised network in segmentation and meets clinical requirements for AAA screening and diagnosis. CONCLUSION CACTUSS provides a promising approach to improve US segmentation accuracy by leveraging CT labels, reducing the need for manual annotations. We generate IRs that inherit properties from both modalities while preserving the anatomical structure and are optimized for the task of aorta segmentation. Future work involves integrating CACTUSS into robotic ultrasound platforms for automated screening and conducting clinical feasibility studies.
Collapse
Affiliation(s)
- Yordanka Velikova
- Computer Aided Medical Procedures, Technical University of Munich, Garching, Germany.
| | - Walter Simson
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Mohammad Farid Azampour
- Computer Aided Medical Procedures, Technical University of Munich, Garching, Germany
- Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran
| | | | - Nassir Navab
- Computer Aided Medical Procedures, Technical University of Munich, Garching, Germany
- Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
20
|
Susmelj AK, Lafci B, Ozdemir F, Davoudi N, Deán-Ben XL, Perez-Cruz F, Razansky D. Signal domain adaptation network for limited-view optoacoustic tomography. Med Image Anal 2024; 91:103012. [PMID: 37922769 DOI: 10.1016/j.media.2023.103012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 09/19/2023] [Accepted: 10/18/2023] [Indexed: 11/07/2023]
Abstract
Optoacoustic (OA) imaging is based on optical excitation of biological tissues with nanosecond-duration laser pulses and detection of ultrasound (US) waves generated by thermoelastic expansion following light absorption. The image quality and fidelity of OA images critically depend on the extent of tomographic coverage provided by the US detector arrays. However, full tomographic coverage is not always possible due to experimental constraints. One major challenge concerns an efficient integration between OA and pulse-echo US measurements using the same transducer array. A common approach toward the hybridization consists in using standard linear transducer arrays, which readily results in arc-type artifacts and distorted shapes in OA images due to the limited angular coverage. Deep learning methods have been proposed to mitigate limited-view artifacts in OA reconstructions by mapping artifactual to artifact-free (ground truth) images. However, acquisition of ground truth data with full angular coverage is not always possible, particularly when using handheld probes in a clinical setting. Deep learning methods operating in the image domain are then commonly based on networks trained on simulated data. This approach is yet incapable of transferring the learned features between two domains, which results in poor performance on experimental data. Here, we propose a signal domain adaptation network (SDAN) consisting of i) a domain adaptation network to reduce the domain gap between simulated and experimental signals and ii) a sides prediction network to complement the missing signals in limited-view OA datasets acquired from a human forearm by means of a handheld linear transducer array. The proposed method showed improved performance in reducing limited-view artifacts without the need for ground truth signals from full tomographic acquisitions.
Collapse
Affiliation(s)
| | - Berkan Lafci
- Institute for Biomedical Engineering, Department of Information Technology and Electrical Engineering, ETH Zurich, Switzerland; Institute of Pharmacology and Toxicology and Institute for Biomedical Engineering, Faculty of Medicine, University of Zurich, Switzerland
| | - Firat Ozdemir
- Swiss Data Science Center, ETH Zürich and EPFL, Switzerland
| | - Neda Davoudi
- Institute for Biomedical Engineering, Department of Information Technology and Electrical Engineering, ETH Zurich, Switzerland; Institute of Pharmacology and Toxicology and Institute for Biomedical Engineering, Faculty of Medicine, University of Zurich, Switzerland
| | - Xosé Luís Deán-Ben
- Institute for Biomedical Engineering, Department of Information Technology and Electrical Engineering, ETH Zurich, Switzerland; Institute of Pharmacology and Toxicology and Institute for Biomedical Engineering, Faculty of Medicine, University of Zurich, Switzerland
| | - Fernando Perez-Cruz
- Swiss Data Science Center, ETH Zürich and EPFL, Switzerland; Institute for Machine Learning, Department of Computer Science, ETH Zurich, Switzerland
| | - Daniel Razansky
- Institute for Biomedical Engineering, Department of Information Technology and Electrical Engineering, ETH Zurich, Switzerland; Institute of Pharmacology and Toxicology and Institute for Biomedical Engineering, Faculty of Medicine, University of Zurich, Switzerland.
| |
Collapse
|
21
|
Muffoletto M, Xu H, Kunze KP, Neji R, Botnar R, Prieto C, Rückert D, Young AA. Combining generative modelling and semi-supervised domain adaptation for whole heart cardiovascular magnetic resonance angiography segmentation. J Cardiovasc Magn Reson 2023; 25:80. [PMID: 38124106 PMCID: PMC10734115 DOI: 10.1186/s12968-023-00981-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 11/12/2023] [Indexed: 12/23/2023] Open
Abstract
BACKGROUND Quantification of three-dimensional (3D) cardiac anatomy is important for the evaluation of cardiovascular diseases. Changes in anatomy are indicative of remodeling processes as the heart tissue adapts to disease. Although robust segmentation methods exist for computed tomography angiography (CTA), few methods exist for whole-heart cardiovascular magnetic resonance angiograms (CMRA) which are more challenging due to variable contrast, lower signal to noise ratio and a limited amount of labeled data. METHODS Two state-of-the-art unsupervised generative deep learning domain adaptation architectures, generative adversarial networks and variational auto-encoders, were applied to 3D whole heart segmentation of both conventional (n = 20) and high-resolution (n = 45) CMRA (target) images, given segmented CTA (source) images for training. An additional supervised loss function was implemented to improve performance given 10%, 20% and 30% segmented CMRA cases. A fully supervised nn-UNet trained on the given CMRA segmentations was used as the benchmark. RESULTS The addition of a small number of segmented CMRA training cases substantially improved performance in both generative architectures in both standard and high-resolution datasets. Compared with the nn-UNet benchmark, the generative methods showed substantially better performance in the case of limited labelled cases. On the standard CMRA dataset, an average 12% (adversarial method) and 10% (variational method) improvement in Dice score was obtained. CONCLUSIONS Unsupervised domain-adaptation methods for CMRA segmentation can be boosted by the addition of a small number of supervised target training cases. When only few labelled cases are available, semi-supervised generative modelling is superior to supervised methods.
Collapse
Affiliation(s)
- Marica Muffoletto
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK.
| | - Hao Xu
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Karl P Kunze
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
- MR Research Collaborations, Siemens Healthcare Limited, Frimley, UK
| | - Radhouene Neji
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
- MR Research Collaborations, Siemens Healthcare Limited, Frimley, UK
| | - René Botnar
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Claudia Prieto
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Daniel Rückert
- Biomedical Image Analysis Group, Department of Computing, Imperial College London, London, UK
- Institute for Artificial Intelligence and Informatics in Medicine, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany
| | - Alistair A Young
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| |
Collapse
|
22
|
Ding X, Fan Y, Li Y, Ge J. Urban surface classification using semi-supervised domain adaptive deep learning models and its application in urban environment studies. Environ Sci Pollut Res Int 2023; 30:123507-123526. [PMID: 37989945 DOI: 10.1007/s11356-023-30843-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/21/2023] [Accepted: 10/29/2023] [Indexed: 11/23/2023]
Abstract
High-resolution urban surface information, e.g., the fraction of impervious/pervious surface, is pivotal in studies of local thermal/wind environments and air pollution. In this study, we introduced and validated a domain adaptive land cover classification model, to automatically classify Google Earth images into pixel-based land cover maps. By combining domain adaptation (DA) and semi-supervised learning (SSL) techniques, our model demonstrates its effectiveness even when trained with a limited dataset derived from Gaofen2 (GF2) satellite images. The model's overall accuracy on the translated GF2 dataset improved significantly from 19.5% to 75.2%, and on the Google Earth image dataset from 23.1% to 61.5%. The overall accuracy is 2.9% and 3.4% higher than when using only DA. Furthermore, with this model, we derived land cover maps and investigated the impact of land surface composition on the local meteorological parameters and air pollutant concentrations in the three most developed urban agglomerations in China, i.e., Beijing, Shanghai and the Great Bay Area (GBA). Our correlation analysis reveals that air temperature exhibits a strong positive correlation with neighboring artificial impervious surfaces, with Pearson correlation coefficients higher than 0.6 in all areas except during the spring in the GBA. However, the correlation between air pollutants and land surface composition is notably weaker and more variable. The primary contribution of this paper is to provide an efficient method for urban land cover extraction which will be of great value for assessing the urban surface composition, quantifying the impact of land use/land cover, and facilitating the development of informed policies.
Collapse
Affiliation(s)
- Xiaotian Ding
- College of Civil Engineering and Architecture, Zhejiang University, Hangzhou, China
- Center for Balance Architecture, Zhejiang University, Hangzhou, China
- International Research Center for Green Building and Low-Carbon City, International Campus, Zhejiang University, Haining, China
| | - Yifan Fan
- College of Civil Engineering and Architecture, Zhejiang University, Hangzhou, China.
- Center for Balance Architecture, Zhejiang University, Hangzhou, China.
- International Research Center for Green Building and Low-Carbon City, International Campus, Zhejiang University, Haining, China.
| | - Yuguo Li
- Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China
| | - Jian Ge
- College of Civil Engineering and Architecture, Zhejiang University, Hangzhou, China
- International Research Center for Green Building and Low-Carbon City, International Campus, Zhejiang University, Haining, China
| |
Collapse
|
23
|
Xing F, Yang X, Cornish TC, Ghosh D. Learning with limited target data to detect cells in cross-modality images. Med Image Anal 2023; 90:102969. [PMID: 37802010 DOI: 10.1016/j.media.2023.102969] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Revised: 08/16/2023] [Accepted: 09/11/2023] [Indexed: 10/08/2023]
Abstract
Deep neural networks have achieved excellent cell or nucleus quantification performance in microscopy images, but they often suffer from performance degradation when applied to cross-modality imaging data. Unsupervised domain adaptation (UDA) based on generative adversarial networks (GANs) has recently improved the performance of cross-modality medical image quantification. However, current GAN-based UDA methods typically require abundant target data for model training, which is often very expensive or even impossible to obtain for real applications. In this paper, we study a more realistic yet challenging UDA situation, where (unlabeled) target training data is limited and previous work seldom delves into cell identification. We first enhance a dual GAN with task-specific modeling, which provides additional supervision signals to assist with generator learning. We explore both single-directional and bidirectional task-augmented GANs for domain adaptation. Then, we further improve the GAN by introducing a differentiable, stochastic data augmentation module to explicitly reduce discriminator overfitting. We examine source-, target-, and dual-domain data augmentation for GAN enhancement, as well as joint task and data augmentation in a unified GAN-based UDA framework. We evaluate the framework for cell detection on multiple public and in-house microscopy image datasets, which are acquired with different imaging modalities, staining protocols and/or tissue preparations. The experiments demonstrate that our method significantly boosts performance when compared with the reference baseline, and it is superior to or on par with fully supervised models that are trained with real target annotations. In addition, our method outperforms recent state-of-the-art UDA approaches by a large margin on different datasets.
Collapse
Affiliation(s)
- Fuyong Xing
- Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, 13001 E 17th Pl, Aurora, CO 80045, USA.
| | - Xinyi Yang
- Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, 13001 E 17th Pl, Aurora, CO 80045, USA
| | - Toby C Cornish
- Department of Pathology, University of Colorado Anschutz Medical Campus, 13001 E 17th Pl, Aurora, CO 80045, USA
| | - Debashis Ghosh
- Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, 13001 E 17th Pl, Aurora, CO 80045, USA
| |
Collapse
|
24
|
Voina D, Shea-Brown E, Mihalas S. A biologically inspired architecture with switching units can learn to generalize across backgrounds. Neural Netw 2023; 168:615-630. [PMID: 37839332 PMCID: PMC10843013 DOI: 10.1016/j.neunet.2023.09.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2022] [Revised: 08/24/2023] [Accepted: 09/07/2023] [Indexed: 10/17/2023]
Abstract
Humans and other animals navigate different environments effortlessly, their brains rapidly and accurately generalizing across contexts. Despite recent progress in deep learning, this flexibility remains a challenge for many artificial systems. Here, we show how a bio-inspired network motif can explicitly address this issue. We do this using a dataset of MNIST digits of varying transparency, set on one of two backgrounds of different statistics that define two contexts: a pixel-wise noise or a more naturalistic background from the CIFAR-10 dataset. After learning digit classification when both contexts are shown sequentially, we find that both shallow and deep networks have sharply decreased performance when returning to the first background - an instance of the catastrophic forgetting phenomenon known from continual learning. To overcome this, we propose the bottleneck-switching network or switching network for short. This is a bio-inspired architecture analogous to a well-studied network motif in the visual cortex, with additional "switching" units that are activated in the presence of a new background, assuming a priori a contextual signal to turn these units on or off. Intriguingly, only a few of these switching units are sufficient to enable the network to learn the new context without catastrophic forgetting through inhibition of redundant background features. Further, the bottleneck-switching network can generalize to novel contexts similar to contexts it has learned. Importantly, we find that - again as in the underlying biological network motif, recurrently connecting the switching units to network layers is advantageous for context generalization.
Collapse
Affiliation(s)
- Doris Voina
- Department of Applied Mathematics, Computational Neuroscience Center, University of Washington, Seattle, WA 98195, USA.
| | - Eric Shea-Brown
- Department of Applied Mathematics, Computational Neuroscience Center, University of Washington, Seattle, WA 98195, USA; Allen Institute for Brain Science, 615 Westlake Ave N, Seattle, WA 98109, USA
| | - Stefan Mihalas
- Department of Applied Mathematics, Computational Neuroscience Center, University of Washington, Seattle, WA 98195, USA; Allen Institute for Brain Science, 615 Westlake Ave N, Seattle, WA 98109, USA
| |
Collapse
|
25
|
Liu Y, Yang B, Chen X, Zhu J, Ji G, Liu Y, Chen B, Lu N, Yi J, Wang S, Li Y, Dai J, Men K. Efficient segmentation using domain adaptation for MRI-guided and CBCT-guided online adaptive radiotherapy. Radiother Oncol 2023; 188:109871. [PMID: 37634767 DOI: 10.1016/j.radonc.2023.109871] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2023] [Revised: 07/31/2023] [Accepted: 08/20/2023] [Indexed: 08/29/2023]
Abstract
BACKGROUND Delineation of regions of interest (ROIs) is important for adaptive radiotherapy (ART) but it is also time consuming and labor intensive. AIM This study aims to develop efficient segmentation methods for magnetic resonance imaging-guided ART (MRIgART) and cone-beam computed tomography-guided ART (CBCTgART). MATERIALS AND METHODS MRIgART and CBCTgART studies enrolled 242 prostate cancer patients and 530 nasopharyngeal carcinoma patients, respectively. A public dataset of CBCT from 35 pancreatic cancer patients was adopted to test the framework. We designed two domain adaption methods to learn and adapt the features from planning computed tomography (pCT) to MRI or CBCT modalities. The pCT was transformed to synthetic MRI (sMRI) for MRIgART, while CBCT was transformed to synthetic CT (sCT) for CBCTgART. Generalized segmentation models were trained with large popular data in which the inputs were sMRI for MRIgART and pCT for CBCTgART. Finally, the personalized models for each patient were established by fine-tuning the generalized model with the contours on pCT of that patient. The proposed method was compared with deformable image registration (DIR), a regular deep learning (DL) model trained on the same modality (DL-regular), and a generalized model in our framework (DL-generalized). RESULTS The proposed method achieved better or comparable performance. For MRIgART of the prostate cancer patients, the mean dice similarity coefficient (DSC) of four ROIs was 87.2%, 83.75%, 85.36%, and 92.20% for the DIR, DL-regular, DL-generalized, and proposed method, respectively. For CBCTgART of the nasopharyngeal carcinoma patients, the mean DSC of two target volumes were 90.81% and 91.18%, 75.17% and 58.30%, for the DIR, DL-regular, DL-generalized, and the proposed method, respectively. For CBCTgART of the pancreatic cancer patients, the mean DSC of two ROIs were 61.94% and 61.44%, 63.94% and 81.56%, for the DIR, DL-regular, DL-generalized, and the proposed method, respectively. CONCLUSION The proposed method utilizing personalized modeling improved the segmentation accuracy of ART.
Collapse
Affiliation(s)
- Yuxiang Liu
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Bining Yang
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Xinyuan Chen
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Ji Zhu
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Guangqian Ji
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Yueping Liu
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Bo Chen
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Ningning Lu
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Junlin Yi
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Shulian Wang
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Yexiong Li
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
| | - Jianrong Dai
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China.
| | - Kuo Men
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China.
| |
Collapse
|
26
|
Dan J, Jin T, Chi H, Dong S, Xie H, Cao K, Yang X. Trust-aware conditional adversarial domain adaptation with feature norm alignment. Neural Netw 2023; 168:518-530. [PMID: 37832319 DOI: 10.1016/j.neunet.2023.10.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Revised: 08/19/2023] [Accepted: 10/02/2023] [Indexed: 10/15/2023]
Abstract
Adversarial learning has proven to be an effective method for capturing transferable features for unsupervised domain adaptation. However, some existing conditional adversarial domain adaptation methods assign equal importance to different samples, ignoring the fact that hard-to-transfer samples might damage the conditional adversarial adaptation procedure. Meanwhile, some methods can only roughly align marginal distributions across domains, but cannot ensure category distributions alignment, causing classifiers to make uncertain or even wrong predictions for some target data. Furthermore, we find that the feature norms of real images usually follow a complex distribution, so directly matching the mean feature norms of two domains cannot effectively reduce the statistical discrepancy of feature norms and may potentially induce feature degradation. In this paper, we develop a Trust-aware Conditional Adversarial Domain Adaptation (TCADA) method for solving the aforementioned issues. To quantify data transferability, we suggest utilizing posterior probability modeled by a Gaussian-uniform mixture, which effectively facilitates conditional domain alignment. Based on this posterior probability, a confidence-guided alignment strategy is presented to promote precise alignment of category distributions and accelerate the learning of shared features. Moreover, a novel optimal transport-based strategy is introduced to align the feature norms and facilitate shared features becoming more informative. To encourage classifiers to make more accurate predictions for target data, we also design a mixed information-guided entropy regularization term to promote deep features being away from the decision boundaries. Extensive experiments show that our method greatly improves transfer performance on various tasks.
Collapse
Affiliation(s)
- Jun Dan
- College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, 310027, China.
| | - Tao Jin
- College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, 310027, China.
| | - Hao Chi
- School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, 310018, China.
| | - Shunjie Dong
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China; College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China.
| | - Haoran Xie
- Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China.
| | - Keying Cao
- College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, 310027, China.
| | - Xinjing Yang
- College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, 310027, China.
| |
Collapse
|
27
|
She B, Liang W, Qin F, Wang X. Known classes aware and emerging unknown classes rejection based on adversarial training for open set fault diagnosis. ISA Trans 2023; 141:455-469. [PMID: 37453891 DOI: 10.1016/j.isatra.2023.06.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/07/2023] [Revised: 06/29/2023] [Accepted: 06/30/2023] [Indexed: 07/18/2023]
Abstract
Most domain adaptation diagnosis approaches presume that the label spaces of different domains are identical. However, novel fault states may emerge in real-world applications, and traditional closed-set approaches only rely on marginal distribution alignment, making them difficult to resolve the open-set domain adaptation issue. One typical open-set problem is that the label spaces of the source and target domains are partially overlapped. To tackle this issue, this paper proposes an approach called known classes aware and emerging unknown classes rejection (KAEUR) based on adversarial training. First, an adaptive weighted learning scheme based on the entropy is introduced to the maximum classifier discrepancy method, which aims to align the target known-type samples with each class of the source known-type samples and suppress the influence of unknown-type samples during feature alignment, thereby extracting domain-invariant features through interactive adversarial training. Second, two binary cross-entropy schemes and the entropy modules are constructed to enhance the divergence between the known and unknown types. Then, an integrated criterion is established to reject the target unknown classes. Finally, three machinery datasets are constructed to demonstrate the effectiveness and superior performance of the proposed approach.
Collapse
Affiliation(s)
- Bo She
- Department of Weaponry Engineering, Naval University of Engineering, Wuhan 430000, China
| | - Weige Liang
- Department of Weaponry Engineering, Naval University of Engineering, Wuhan 430000, China.
| | - Fenqi Qin
- 713 Research Institute of China Shipbuilding, Zhengzhou 450000, China
| | - Xuan Wang
- Department of Weaponry Engineering, Naval University of Engineering, Wuhan 430000, China
| |
Collapse
|
28
|
Zhang Y, Zhao L, Wang Q. MiDA: Membership inference attacks against domain adaptation. ISA Trans 2023; 141:103-112. [PMID: 36702690 DOI: 10.1016/j.isatra.2023.01.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/30/2022] [Revised: 01/03/2023] [Accepted: 01/14/2023] [Indexed: 06/18/2023]
Abstract
Domain adaption has become an effective solution to train neural networks with insufficient training data. In this paper, we investigate the vulnerability of domain adaption that potentially breaches sensitive information about the training dataset. We propose a new membership inference attack against domain adaption models, to infer the membership information of samples from the target domain. By leveraging the background knowledge about an additional source-domain in domain adaptation tasks, our attack can exploit the similar distributions between the target and source domain data to determine if a specific data sample belongs in the training set with high efficiency and accuracy. In particular, the proposed attack can be deployed in a practical scenario where the attacker cannot obtain any details of the model. We conduct extensive evaluations for object and digit recognition tasks. Experimental results show that our method can achieve the attack against domain adaptation models with a high success rate.
Collapse
Affiliation(s)
- Yuanjie Zhang
- Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, 430072 Wuhan, PR China.
| | - Lingchen Zhao
- Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, 430072 Wuhan, PR China.
| | - Qian Wang
- Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, 430072 Wuhan, PR China.
| |
Collapse
|
29
|
Bigalke A, Hansen L, Diesel J, Hennigs C, Rostalski P, Heinrich MP. Anatomy-guided domain adaptation for 3D in-bed human pose estimation. Med Image Anal 2023; 89:102887. [PMID: 37453235 DOI: 10.1016/j.media.2023.102887] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Revised: 06/16/2023] [Accepted: 06/28/2023] [Indexed: 07/18/2023]
Abstract
3D human pose estimation is a key component of clinical monitoring systems. The clinical applicability of deep pose estimation models, however, is limited by their poor generalization under domain shifts along with their need for sufficient labeled training data. As a remedy, we present a novel domain adaptation method, adapting a model from a labeled source to a shifted unlabeled target domain. Our method comprises two complementary adaptation strategies based on prior knowledge about human anatomy. First, we guide the learning process in the target domain by constraining predictions to the space of anatomically plausible poses. To this end, we embed the prior knowledge into an anatomical loss function that penalizes asymmetric limb lengths, implausible bone lengths, and implausible joint angles. Second, we propose to filter pseudo labels for self-training according to their anatomical plausibility and incorporate the concept into the Mean Teacher paradigm. We unify both strategies in a point cloud-based framework applicable to unsupervised and source-free domain adaptation. Evaluation is performed for in-bed pose estimation under two adaptation scenarios, using the public SLP dataset and a newly created dataset. Our method consistently outperforms various state-of-the-art domain adaptation methods, surpasses the baseline model by 31%/66%, and reduces the domain gap by 65%/82%. Source code is available at https://github.com/multimodallearning/da-3dhpe-anatomy.
Collapse
Affiliation(s)
- Alexander Bigalke
- Institute of Medical Informatics, University of Lübeck, Ratzeburger Allee 160, 23538 Lübeck, Germany.
| | - Lasse Hansen
- EchoScout GmbH, Maria-Goeppert-Str. 3, 23562 Lübeck, Germany
| | - Jasper Diesel
- Drägerwerk AG & Co. KGaA, Moislinger Allee 53-55, 23558 Lübeck, Germany
| | - Carlotta Hennigs
- Institute for Electrical Engineering in Medicine, University of Lübeck, Moislinger Allee 53-55, 23558 Lübeck, Germany
| | - Philipp Rostalski
- Institute for Electrical Engineering in Medicine, University of Lübeck, Moislinger Allee 53-55, 23558 Lübeck, Germany
| | - Mattias P Heinrich
- Institute of Medical Informatics, University of Lübeck, Ratzeburger Allee 160, 23538 Lübeck, Germany
| |
Collapse
|
30
|
Fan L, Gong X, Guo Y. General Multiscenario Ultrasound Image Tumor Diagnosis Method Based on Unsupervised Domain Adaptation. Ultrasound Med Biol 2023; 49:2291-2301. [PMID: 37532633 DOI: 10.1016/j.ultrasmedbio.2023.06.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Revised: 06/18/2023] [Accepted: 06/23/2023] [Indexed: 08/04/2023]
Abstract
OBJECTIVE The utilization of computer-aided diagnosis (CAD) in breast ultrasound image classification has been limited by small sample sizes and domain shift. Current ultrasound classification methods perform inadequately when exposed to cross-domain scenarios, as they struggle with data sets from unobserved domains. In the medical field, there are situations in which all images must share the same networks as they capture the same symptom of the same participant, implying that they share identical structural content. Nevertheless, most domain adaptation methods are not suitable for medical images as they overlook the common features among the images. METHODS To overcome these challenges, we propose a novel diverse-domain 2-D feature selection network (FSN), which uses the similarities among medical images and extracts features with a reconstruction network with shared weights. Additionally, it penalizes the feature domain distance through two adversarial learning modules that align the feature space and select common features. Our experiments illustrate that the proposed method is robust and can be applied to ultrasound images of various diseases. RESULTS Compared with the latest domain adaptive methods, 2-D FSN markedly enhances the accuracy of classification of breast, thyroid and endoscopic ultrasound images, achieving accuracies of 82.4%, 96.4% and 89.7%, respectively. Furthermore, the model was evaluated on an unsupervised domain adaptation task using ultrasound images from multiple sources and achieved an average accuracy of 77.3% across widely varying domains. CONCLUSION In general, 2-D FSN improves the classification ability of the model on multidomain ultrasound data sets through the learning of common features and the combination of multimodule intelligence. The algorithm has good clinical guidance value.
Collapse
Affiliation(s)
- Lin Fan
- School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, P.R. China; Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, Chengdu 611756, P.R. China; National Engineering Laboratory of Integrated Transportation Big Data Application Technology, Southwest Jiaotong University, Chengdu 611756, P.R. China; Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, Southwest Jiaotong University, Chengdu 611756, P.R. China
| | - Xun Gong
- School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756, P.R. China; Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education, Chengdu 611756, P.R. China; National Engineering Laboratory of Integrated Transportation Big Data Application Technology, Southwest Jiaotong University, Chengdu 611756, P.R. China; Manufacturing Industry Chains Collaboration and Information Support Technology Key Laboratory of Sichuan Province, Southwest Jiaotong University, Chengdu 611756, P.R. China.
| | - Ying Guo
- North China University of Science and Technology Affiliated Hospital, Tangshan, Hebei, China
| |
Collapse
|
31
|
Syed AA, Gaol FL, Boediman A, Matsuo T, Budiharto W. A data package for abstractive opinion summarization, title generation, and rating-based sentiment prediction for airline reviews. Data Brief 2023; 50:109535. [PMID: 37720686 PMCID: PMC10504493 DOI: 10.1016/j.dib.2023.109535] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2023] [Revised: 08/09/2023] [Accepted: 08/28/2023] [Indexed: 09/19/2023] Open
Abstract
Customer reviews are valuable resources containing customer opinions and sentiments toward the product. The reviews are informative but can be quite lengthy or may contain repetitive information calling for opinion summarization systems that retain only the significant opinion information from the review. Abstractive summarization is a form of text summarization that generates a summary mimicking a human-written summary [1]. When pretrained language models are finetuned for abstractive review summarization, there usually occurs a problem known as the 'domain shift', because the source and target domains exhibit data from varying distributions [2]. This issue results in performance degradation of the model at the target end. This paper contributes a data package comprising of an annotated abstractive summarization dataset (annotated_abs_summ) of airline reviews having 500 reviews and abstractive summary pairs, a dataset (review_titles_data) consisting of 7079 reviews and review title pairs for review title generatioon or domain adaptive training [3] to address the domain shift problem for abstractive opinion summarization and, an annotated reviews dataset (annotated_sentiment) for rating-based sentiment classification. All datasets have been collected from the Skytrax Review Portal via web scraping using Python programming language. The datasets have several potential use cases. The abstractive summarization dataset can serve as a benchmark dataset for airline review summarization. The dataset for domain adaptive training can be used as a standalone dataset for review title generation. The dataset for sentiment analysis is multipurpose having columns like user rating and recommendation value, that can be used for statistical analysis like finding correlation between these data items as well as for other Natural Language Processing (NLP) tasks like predicting rating or recommendation value from the customer reviews. The datasets can be extended using various data augmentation techniques [4,5]. Moreover, the datasets are related and can be collectively used to develop a multi-task learning model [6] for better learning efficiency and improved performance.
Collapse
Affiliation(s)
- Ayesha Ayub Syed
- Department of Doctor of Computer Science – BINUS Graduate Program, Bina Nusantara University, Jakarta, Indonesia
| | - Ford Lumban Gaol
- Department of Doctor of Computer Science – BINUS Graduate Program, Bina Nusantara University, Jakarta, Indonesia
| | - Alfred Boediman
- Department of Econometrics and Statistics - The University of Chicago, Booth School of Business, USA
| | - Tokuro Matsuo
- Graduate School of Industrial Technology, Advanced Institute of Industrial Technology, Tokyo 140-0011, Japan
- Department of M-Commerce and Multimedia Applications, Asia University, Taichung 41354, Taiwan
| | - Widodo Budiharto
- Computer Science Department, School of Computer Science, Bina Nusantara University, Jakarta 11480, Indonesia
| |
Collapse
|
32
|
Li X, Zhang X, Chen X, Chen X, Liu A. Cross-user gesture recognition from sEMG signals using an optimal transport assisted student-teacher framework. Comput Biol Med 2023; 165:107327. [PMID: 37619326 DOI: 10.1016/j.compbiomed.2023.107327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Revised: 07/14/2023] [Accepted: 08/07/2023] [Indexed: 08/26/2023]
Abstract
The cross-user gesture recognition is a puzzle in the myoelectric control system, owing to great variability in muscle activities across different users. To address this problem, a novel optimal transport (OT) assisted student-teacher (ST) framework (termed OT-ST) was proposed in this paper to facilitate transfer across user domains in an unsupervised domain adaptation (UDA) manner. In this framework, the initial parameters of the ST models were trained with the labeled data from users in the source domain. In the model transfer stage for a new user in the target domain, the teacher model was utilized to generate pseudo labels for unlabeled testing samples, providing guidance to the adaptation of the student model. The OT algorithm was employed to optimize the pseudo labels generated from the teacher model, avoiding the model bias and further improving the effect of domain adaptation. The performance of the proposed OT-ST framework was evaluated via experiments of classifying seven hand gestures using high-density surface electromyogram (HD-sEMG) recordings from extensor digitorum muscles of eight intact-limbed subjects. The OT-ST framework yielded a high accuracy of 96.50 ± 2.88% for new users, and outperformed other common machine learning and UDA methods significantly (p < 0.01), demonstrating its effectiveness. The OT-ST framework does not require special repetitive training or any labeled data for calibration. In addition, it can incrementally learn from new testing samples and improve the recognition ability. This study provides a promising method for developing user-generic myoelectric pattern recognition, with wide applications in human-computer interaction, consumer electronics and prosthesis control.
Collapse
Affiliation(s)
- Xinhui Li
- School of Microelectronics, University of Science and Technology of China, Hefei, 230027, China
| | - Xu Zhang
- School of Microelectronics, University of Science and Technology of China, Hefei, 230027, China.
| | - Xiang Chen
- School of Microelectronics, University of Science and Technology of China, Hefei, 230027, China
| | - Xun Chen
- School of Information Science and Technology, University of Science and Technology of China, Hefei, 230027, China
| | - Aiping Liu
- School of Information Science and Technology, University of Science and Technology of China, Hefei, 230027, China
| |
Collapse
|
33
|
Wanjiku RN, Nderu L, Kimwele M. Improved transfer learning using textural features conflation and dynamically fine-tuned layers. PeerJ Comput Sci 2023; 9:e1601. [PMID: 37810335 PMCID: PMC10557498 DOI: 10.7717/peerj-cs.1601] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Accepted: 08/29/2023] [Indexed: 10/10/2023]
Abstract
Transfer learning involves using previously learnt knowledge of a model task in addressing another task. However, this process works well when the tasks are closely related. It is, therefore, important to select data points that are closely relevant to the previous task and fine-tune the suitable pre-trained model's layers for effective transfer. This work utilises the least divergent textural features of the target datasets and pre-trained model's layers, minimising the lost knowledge during the transfer learning process. This study extends previous works on selecting data points with good textural features and dynamically selected layers using divergence measures by combining them into one model pipeline. Five pre-trained models are used: ResNet50, DenseNet169, InceptionV3, VGG16 and MobileNetV2 on nine datasets: CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, Stanford Dogs, Caltech 256, ISIC 2016, ChestX-ray8 and MIT Indoor Scenes. Experimental results show that data points with lower textural feature divergence and layers with more positive weights give better accuracy than other data points and layers. The data points with lower divergence give an average improvement of 3.54% to 6.75%, while the layers improve by 2.42% to 13.04% for the CIFAR-100 dataset. Combining the two methods gives an extra accuracy improvement of 1.56%. This combined approach shows that data points with lower divergence from the source dataset samples can lead to a better adaptation for the target task. The results also demonstrate that selecting layers with more positive weights reduces instances of trial and error in selecting fine-tuning layers for pre-trained models.
Collapse
Affiliation(s)
| | - Lawrence Nderu
- Computing, Jomo Kenyatta University of Agriculture and Technology, Nairobi, Kenya
| | - Michael Kimwele
- Computing, Jomo Kenyatta University of Agriculture and Technology, Nairobi, Kenya
| |
Collapse
|
34
|
Huang J, Chen K, Ren Y, Sun J, Wang Y, Tao T, Pu X. CDDnet: Cross-domain denoising network for low-dose CT image via local and global information alignment. Comput Biol Med 2023; 163:107219. [PMID: 37422942 DOI: 10.1016/j.compbiomed.2023.107219] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2023] [Revised: 05/21/2023] [Accepted: 06/25/2023] [Indexed: 07/11/2023]
Abstract
The domain shift problem has emerged as a challenge in cross-domain low-dose CT (LDCT) image denoising task, where the acquisition of a sufficient number of medical images from multiple sources may be constrained by privacy concerns. In this study, we propose a novel cross-domain denoising network (CDDnet) that incorporates both local and global information of CT images. To address the local component, a local information alignment module has been proposed to regularize the similarity between extracted target and source features from selected patches. To align the general information of the semantic structure from a global perspective, an autoencoder is adopted to learn the latent correlation between the source label and the estimated target label generated by the pre-trained denoiser. Experimental results demonstrate that our proposed CDDnet effectively alleviates the domain shift problem, outperforming other deep learning-based and domain adaptation-based methods under cross-domain scenarios.
Collapse
Affiliation(s)
- Jiaxin Huang
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China
| | - Kecheng Chen
- Department of Electrical Engineering, City University of Hong Kong, 999077, Hong Kong Special Administrative Region of China
| | - Yazhou Ren
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China; Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, 518110, China
| | - Jiayu Sun
- West China Hospital, Sichuan University, Chengdu, 610044, China
| | - Yanmei Wang
- Institute of Traditional Chinese Medicine, Sichuan College of Traditional Chinese Medicine (Sichuan Second Hospital of TCM), Chengdu, 610075, China
| | - Tao Tao
- Institute of Traditional Chinese Medicine, Sichuan College of Traditional Chinese Medicine (Sichuan Second Hospital of TCM), Chengdu, 610075, China
| | - Xiaorong Pu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China; Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China, Shenzhen, 518110, China; NHC Key Laboratory of Nuclear Technology Medical Transformation, Mianyang Central Hospital, Mianyang, 621000, China.
| |
Collapse
|
35
|
Gu J, Qian X, Zhang Q, Zhang H, Wu F. Unsupervised domain adaptation for Covid-19 classification based on balanced slice Wasserstein distance. Comput Biol Med 2023; 164:107207. [PMID: 37480680 DOI: 10.1016/j.compbiomed.2023.107207] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 06/06/2023] [Accepted: 06/25/2023] [Indexed: 07/24/2023]
Abstract
Covid-19 has swept the world since 2020, taking millions of lives. In order to seek a rapid diagnosis of Covid-19, deep learning-based Covid-19 classification methods have been extensively developed. However, deep learning relies on many samples with high-quality labels, which is expensive. To this end, we propose a novel unsupervised domain adaptation method to process many different but related Covid-19 X-ray images. Unlike existing unsupervised domain adaptation methods that cannot handle conditional class distributions, we adopt a balanced Slice Wasserstein distance as the metric for unsupervised domain adaptation to solve this problem. Multiple standard datasets for domain adaptation and X-ray datasets of different Covid-19 are adopted to verify the effectiveness of our proposed method. Experimented by cross-adopting multiple datasets as source and target domains, respectively, our proposed method can effectively capture discriminative and domain-invariant representations with better data distribution matching.
Collapse
Affiliation(s)
- Jiawei Gu
- Affiliated Hospital of Nantong University, Nantong, 226001, China.
| | - Xuan Qian
- Affiliated Hospital of Nantong University, Nantong, 226001, China.
| | - Qian Zhang
- School of Data Science and Artificial Intelligence, Wenzhou University of Technology, Wenzhou, 325035, China.
| | - Hongliang Zhang
- College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, 325035, China.
| | - Fang Wu
- Department of Gastroenterology, The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, 325035, China.
| |
Collapse
|
36
|
Zhang D, Li H, Xie J. MI-CAT: A transformer-based domain adaptation network for motor imagery classification. Neural Netw 2023; 165:451-462. [PMID: 37336030 DOI: 10.1016/j.neunet.2023.06.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 04/03/2023] [Accepted: 06/02/2023] [Indexed: 06/21/2023]
Abstract
Due to its convenience and safety, electroencephalography (EEG) data is one of the most widely used signals in motor imagery (MI) brain-computer interfaces (BCIs). In recent years, methods based on deep learning have been widely applied to the field of BCIs, and some studies have gradually tried to apply Transformer to EEG signal decoding due to its superior global information focusing ability. However, EEG signals vary from subject to subject. Based on Transformer, how to effectively use data from other subjects (source domain) to improve the classification performance of a single subject (target domain) remains a challenge. To fill this gap, we propose a novel architecture called MI-CAT. The architecture innovatively utilizes Transformer's self-attention and cross-attention mechanisms to interact features to resolve differential distribution between different domains. Specifically, we adopt a patch embedding layer for the extracted source and target features to divide the features into multiple patches. Then, we comprehensively focus on the intra-domain and inter-domain features by stacked multiple Cross-Transformer Blocks (CTBs), which can adaptively conduct bidirectional knowledge transfer and information exchange between domains. Furthermore, we also utilize two non-shared domain-based attention blocks to efficiently capture domain-dependent information, optimizing the features extracted from the source and target domains to assist in feature alignment. To evaluate our method, we conduct extensive experiments on two real public EEG datasets, Dataset IIb and Dataset IIa, achieving competitive performance with an average classification accuracy of 85.26% and 76.81%, respectively. Experimental results demonstrate that our method is a powerful model for decoding EEG signals and facilitates the development of the Transformer for brain-computer interfaces (BCIs).
Collapse
Affiliation(s)
- Dongxue Zhang
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Huiying Li
- Jilin University, College of Computer Science and Technology, Changchun, Jilin Province, China; Key Laboratory of Symbol Computation and Knowledge Engineering, Jilin University, Changchun 130012, China.
| | - Jingmeng Xie
- Xi'an Jiaotong University, College of Electronic information, Xi'an, Shanxi Province, China.
| |
Collapse
|
37
|
Liu S, Ozay M. Task guided representation learning using compositional models for zero-shot domain adaptation. Neural Netw 2023; 165:370-380. [PMID: 37329781 DOI: 10.1016/j.neunet.2023.05.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2022] [Revised: 04/29/2023] [Accepted: 05/17/2023] [Indexed: 06/19/2023]
Abstract
Zero-shot domain adaptation (ZDA) methods aim to transfer knowledge about a task learned in a source domain to a target domain, while task-relevant data from target domain are not available. In this work, we address learning feature representations which are invariant to and shared among different domains considering task characteristics for ZDA. To this end, we propose a method for task-guided ZDA (TG-ZDA) which employs multi-branch deep neural networks to learn feature representations exploiting their domain invariance and shareability properties. The proposed TG-ZDA models can be trained end-to-end without requiring synthetic tasks and data generated from estimated representations of target domains. The proposed TG-ZDA has been examined using benchmark ZDA tasks on image classification datasets. Experimental results show that our proposed TG-ZDA outperforms state-of-the-art ZDA methods for different domains and tasks.
Collapse
Affiliation(s)
- Shuang Liu
- RIKEN Center for AIP, Nihonbashi 1-chome Mitsui Building, Tokyo, 1030027, Japan.
| | - Mete Ozay
- Middle East Technical University, Dumlupınar Bulvarı No:1, Ankara, 06800, Turkey.
| |
Collapse
|
38
|
Steingrimsson JA. Extending prediction models for use in a new target population with failure time outcomes. Biostatistics 2023; 24:728-742. [PMID: 35389429 DOI: 10.1093/biostatistics/kxac011] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 03/14/2022] [Accepted: 03/21/2022] [Indexed: 07/20/2023] Open
Abstract
Prediction models are often built and evaluated using data from a population that differs from the target population where model-derived predictions are intended to be used in. In this article, we present methods for evaluating model performance in the target population when some observations are right censored. The methods assume that outcome and covariate data are available from a source population used for model development and covariates, but no outcome data, are available from the target population. We evaluate the finite sample performance of the proposed estimators using simulations and apply the methods to transport a prediction model built using data from a lung cancer screening trial to a nationally representative population of participants eligible for lung cancer screening.
Collapse
Affiliation(s)
- Jon A Steingrimsson
- Department of Biostatistics, Brown University, 121 South Main Street, Providence, RI 02903, USA
| |
Collapse
|
39
|
Shen X, Pan S, Choi KS, Zhou X. Domain-adaptive message passing graph neural network. Neural Netw 2023; 164:439-454. [PMID: 37182346 DOI: 10.1016/j.neunet.2023.04.038] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2022] [Revised: 04/18/2023] [Accepted: 04/21/2023] [Indexed: 05/16/2023]
Abstract
Cross-network node classification (CNNC), which aims to classify nodes in a label-deficient target network by transferring the knowledge from a source network with abundant labels, draws increasing attention recently. To address CNNC, we propose a domain-adaptive message passing graph neural network (DM-GNN), which integrates graph neural network (GNN) with conditional adversarial domain adaptation. DM-GNN is capable of learning informative representations for node classification that are also transferrable across networks. Firstly, a GNN encoder is constructed by dual feature extractors to separate ego-embedding learning from neighbor-embedding learning so as to jointly capture commonality and discrimination between connected nodes. Secondly, a label propagation node classifier is proposed to refine each node's label prediction by combining its own prediction and its neighbors' prediction. In addition, a label-aware propagation scheme is devised for the labeled source network to promote intra-class propagation while avoiding inter-class propagation, thus yielding label-discriminative source embeddings. Thirdly, conditional adversarial domain adaptation is performed to take the neighborhood-refined class-label information into account during adversarial domain adaptation, so that the class-conditional distributions across networks can be better matched. Comparisons with eleven state-of-the-art methods demonstrate the effectiveness of the proposed DM-GNN.
Collapse
Affiliation(s)
- Xiao Shen
- School of Computer Science and Technology, Hainan University, Haikou, China.
| | - Shirui Pan
- School of ICT, Griffith University, Gold Coast, Australia.
| | - Kup-Sze Choi
- Centre for Smart Health, The Hong Kong Polytechnic University, Hong Kong, China.
| | - Xi Zhou
- College of Tropical Crops, Hainan University, Haikou, China.
| |
Collapse
|
40
|
Goli R, Hubig N, Min H, Gong Y, Sittig DF, Rennert L, Robinson D, Biondich P, Wright A, Nøhr C, Law T, Faxvaag A, Weaver A, Gimbel R, Jing X. Keyphrase Identification Using Minimal Labeled Data with Hierarchical Context and Transfer Learning. medRxiv 2023:2023.01.26.23285060. [PMID: 37292830 PMCID: PMC10246160 DOI: 10.1101/2023.01.26.23285060] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Interoperable clinical decision support system (CDSS) rules provide a pathway to interoperability, a well-recognized challenge in health information technology. Building an ontology facilitates creating interoperable CDSS rules, which can be achieved by identifying the keyphrases (KP) from the existing literature. However, KP identification for data labeling requires human expertise, consensus, and contextual understanding. This paper aims to present a semi-supervised KP identification framework using minimal labeled data based on hierarchical attention over the documents and domain adaptation. Our method outperforms the prior neural architectures by learning through synthetic labels for initial training, document-level contextual learning, language modeling, and fine-tuning with limited gold standard label data. To the best of our knowledge, this is the first functional framework for the CDSS sub-domain to identify KPs, which is trained on limited labeled data. It contributes to the general natural language processing (NLP) architectures in areas such as clinical NLP, where manual data labeling is challenging, and light-weighted deep learning models play a role in real-time KP identification as a complementary approach to human experts' effort.
Collapse
Affiliation(s)
- Rohan Goli
- School of Computing, College of Engineering, Computing and Applied Science, Clemson University, Clemson, SC, USA
| | - Nina Hubig
- School of Computing, College of Engineering, Computing and Applied Science, Clemson University, Clemson, SC, USA
| | - Hua Min
- Department of Health Administration and Policy, College of Public Health, George Mason University, Fairfax, VA, USA
| | - Yang Gong
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Dean F. Sittig
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Lior Rennert
- Department of Public Health Sciences, College of Behavioral, Social, and Health Sciences, Clemson University, Clemson, SC, USA
| | - David Robinson
- General Practitioner/Independent Consultant, Cumbria, UK
| | - Paul Biondich
- Clem McDonald Biomedical Informatics Center, Regenstrief Institute, Department of Pediatrics, Indiana University School of Medicine, Indianapolis, IN, USA
| | - Adam Wright
- Vanderbilt University Medical Center, Nashville, TN, USA
| | - Christian Nøhr
- Department of Planning, Faculty of Engineering, Aalborg University, Aalborg, Denmark
| | - Timothy Law
- Ohio Musculoskeletal and Neurologic Institute, Ohio University, Athens, OH, USA
| | - Arild Faxvaag
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, Trondheim, Norway
| | - Aneesa Weaver
- Department of Public Health Sciences, College of Behavioral, Social, and Health Sciences, Clemson University, Clemson, SC, USA
| | - Ronald Gimbel
- Department of Public Health Sciences, College of Behavioral, Social, and Health Sciences, Clemson University, Clemson, SC, USA
| | - Xia Jing
- Department of Public Health Sciences, College of Behavioral, Social, and Health Sciences, Clemson University, Clemson, SC, USA
| |
Collapse
|
41
|
Ebadi N, Li R, Das A, Roy A, Nikos P, Najafirad P. CBCT-guided adaptive radiotherapy using self-supervised sequential domain adaptation with uncertainty estimation. Med Image Anal 2023; 86:102800. [PMID: 37003101 DOI: 10.1016/j.media.2023.102800] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2022] [Revised: 01/29/2023] [Accepted: 03/14/2023] [Indexed: 03/17/2023]
Abstract
Adaptive radiotherapy (ART) is an advanced technology in modern cancer treatment that incorporates progressive changes in patient anatomy into active plan/dose adaption during the fractionated treatment. However, the clinical application relies on the accurate segmentation of cancer tumors on low-quality on-board images, which has posed challenges for both manual delineation and deep learning-based models. In this paper, we propose a novel sequence transduction deep neural network with an attention mechanism to learn the shrinkage of the cancer tumor based on patients' weekly cone-beam computed tomography (CBCT). We design a self-supervised domain adaption (SDA) method to learn and adapt the rich textural and spatial features from pre-treatment high-quality computed tomography (CT) to CBCT modality in order to address the poor image quality and lack of labels. We also provide uncertainty estimation for sequential segmentation, which aids not only in the risk management of treatment planning but also in the calibration and reliability of the model. Our experimental results based on a clinical non-small cell lung cancer (NSCLC) dataset with sixteen patients and ninety-six longitudinal CBCTs show that our model correctly learns weekly deformation of the tumor over time with an average dice score of 0.92 on the immediate next step, and is able to predict multiple steps (up to 5 weeks) for future patient treatments with an average dice score reduction of 0.05. By incorporating the tumor shrinkage predictions into a weekly re-planning strategy, our proposed method demonstrates a significant decrease in the risk of radiation-induced pneumonitis up to 35% while maintaining the high tumor control probability.
Collapse
Affiliation(s)
- Nima Ebadi
- Department of Electrical and Computer Engineering, The University of Texas at San Antonio, San Antonio, TX 78249, United States of America.
| | - Ruiqi Li
- Department of Radiation Oncology, UT Health San Antonio, San Antonio, TX 78229, United States of America.
| | - Arun Das
- Department of Electrical and Computer Engineering, The University of Texas at San Antonio, San Antonio, TX 78249, United States of America; Department of Medicine, The University of Pittsburgh, Pittsburgh, PA 15260, United States of America.
| | - Arkajyoti Roy
- Department of Management Science and Statistics, The University of Texas at San Antonio, San Antonio, TX 78249, United States of America.
| | - Papanikolaou Nikos
- Department of Radiation Oncology, UT Health San Antonio, San Antonio, TX 78229, United States of America.
| | - Peyman Najafirad
- Department of Computer Science, The University of Texas at San Antonio, San Antonio, TX 78249, United States of America.
| |
Collapse
|
42
|
van Tulder G, de Bruijne M. Unpaired, unsupervised domain adaptation assumes your domains are already similar. Med Image Anal 2023; 87:102825. [PMID: 37116296 DOI: 10.1016/j.media.2023.102825] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Revised: 03/30/2023] [Accepted: 04/17/2023] [Indexed: 04/30/2023]
Abstract
Unsupervised domain adaptation is a popular method in medical image analysis, but it can be tricky to make it work: without labels to link the domains, domains must be matched using feature distributions. If there is no additional information, this often leaves a choice between multiple possibilities to map the data that may be equally likely but not equally correct. In this paper we explore the fundamental problems that may arise in unsupervised domain adaptation, and discuss conditions that might still make it work. Focusing on medical image analysis, we argue that images from different domains may have similar class balance, similar intensities, similar spatial structure, or similar textures. We demonstrate how these implicit conditions can affect domain adaptation performance in experiments with synthetic data, MNIST digits, and medical images. We observe that practical success of unsupervised domain adaptation relies on existing similarities in the data, and is anything but guaranteed in the general case. Understanding these implicit assumptions is a key step in identifying potential problems in domain adaptation and improving the reliability of the results.
Collapse
Affiliation(s)
- Gijs van Tulder
- Data Science group, Faculty of Science, Radboud University, Postbus 9010, 6500 GL Nijmegen, The Netherlands; Biomedical Imaging Group, Erasmus MC, Postbus 2040, 3000 CA Rotterdam, The Netherlands.
| | - Marleen de Bruijne
- Biomedical Imaging Group, Erasmus MC, Postbus 2040, 3000 CA Rotterdam, The Netherlands; Department of Computer Science, University of Copenhagen, Universitetsparken 1, 2100 Copenhagen, Denmark.
| |
Collapse
|
43
|
Tian K, Zhang C, Wang Y, Xiang S. Domain adaptive object detection with model-agnostic knowledge transferring. Neural Netw 2023; 161:213-227. [PMID: 36774861 DOI: 10.1016/j.neunet.2023.01.028] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2022] [Revised: 12/30/2022] [Accepted: 01/21/2023] [Indexed: 01/30/2023]
Abstract
The development of deep learning techniques has greatly benefited CNN-based object detectors, leading to unprecedented progress in recent years. However, the distribution variance between training and testing domains causes significant performance degradation. Labeling data for new scenarios is costly and time-consuming, so most existing domain adaptation methods perform feature alignment through adversarial training. While this can improve the accuracy of detectors in unlabeled target domains, the unconstrained domain alignment also negatively transfers the feature distribution, which compromises the recognition ability of the model. To address this problem, we propose the Knowledge Transfer Network (KTNet), which consists of object intrinsic knowledge mining and category relational knowledge constraint modules. Specifically, a binary classifier shared by the source and target domains is designed to extract common attribute knowledge of objects, which can align foreground and background features from different data domains adaptively. Then, we construct relational knowledge graphs to explicitly constrain the category correlations in the source, target, and cross-domain settings. These two modules guide the detector to learn object-related and domain-invariant representations, enabling the proposed KTNet to perform well in four commonly-used cross-domain scenarios. Furthermore, the ablation experiments show that our method is scalable to more complex backbone networks and different detection architectures.
Collapse
Affiliation(s)
- Kun Tian
- NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China.
| | - Chenghao Zhang
- NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China.
| | - Ying Wang
- NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
| | - Shiming Xiang
- NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China.
| |
Collapse
|
44
|
Feng Y, Luo Y, Yang J. Cross-platform privacy-preserving CT image COVID-19 diagnosis based on source-free domain adaptation. Knowl Based Syst 2023; 264:110324. [PMID: 36713615 PMCID: PMC9869622 DOI: 10.1016/j.knosys.2023.110324] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2022] [Revised: 01/05/2023] [Accepted: 01/17/2023] [Indexed: 01/24/2023]
Abstract
In the wake of the Coronavirus disease (COVID-19) pandemic, chest computed tomography (CT) has become an invaluable component in the rapid and accurate detection of COVID-19. CT scans traditionally require manual inspections from medical professionals, which is expensive and tedious. With advancements in machine learning, deep neural networks have been applied to classify CT scans for efficient diagnosis. However, three challenges hinder this application of deep learning: (1) Domain shift across CT platforms and human subjects impedes the performance of neural networks in different hospitals. (2) Unsupervised Domain Adaptation (UDA), the traditional method to overcome domain shift, typically requires access to both source and target data. This is not realistic in COVID-19 diagnosis due to the sensitivity of medical data. The privacy of patients must be protected. (3) Data imbalance may exist between easy/hard samples and between data classes which can overwhelm the training of deep networks, causing degenerate models. To overcome these challenges, we propose a Cross-Platform Privacy-Preserving COVID-19 diagnosis network (CP 3 Net) that integrates domain adaptation, self-supervised learning, imbalanced label learning, and rotation classifier training into one synergistic framework. We also create a new CT benchmark by combining real-world datasets from multiple medical platforms to facilitate the cross-domain evaluation of our method. Through extensive experiments, we demonstrate that CP 3 Net outperforms many popular UDA methods and achieves state-of-the-art results in diagnosing COVID-19 using CT scans.
Collapse
Affiliation(s)
| | - Yuemei Luo
- School of Artificial Intelligence, Nanjing University of Information Science & Technology, China
| | - Jianfei Yang
- School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore,Corresponding author
| |
Collapse
|
45
|
Cai H, Zhang Q, Long Y. Prototype-guided multi-scale domain adaptation for Alzheimer's disease detection. Comput Biol Med 2023; 154:106570. [PMID: 36739819 DOI: 10.1016/j.compbiomed.2023.106570] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2022] [Revised: 01/02/2023] [Accepted: 01/22/2023] [Indexed: 01/24/2023]
Abstract
Alzheimer's disease (AD) is the most common form of dementia and there is no effective treatment currently. Using artificial intelligence technology to assist the diagnosis and intervention as early as possible is of great significance to delay the development of AD. Structural Magnetic Resonance Imaging (sMRI) has shown great practical values on computer-aided AD diagnosis. Affected by data from different sources or acquisition domains in realistic scenarios, MRI data often suffer from domain shift problem. In this paper, we propose a deep Prototype-Guided Multi-Scale Domain Adaptation (PMDA) framework to handle MRI data with domain shift problem, and realize automatic auxiliary diagnosis of AD, Mild Cognitive Impairment (MCI) and Cognitively Normal (CN). PMDA is composed of three modules: (1) MRI multi-scale feature extraction module combines the advantages of 3D convolution and self-attention to effectively extract multi-scale features in high-dimensional space, (2) Prototype Maximum Density Divergence (Pro-MDD) module adopts prototype learning to constrain the feature outlier samples in a mini-batch when MDD is used to align source domain and target domain, and (3) Adversarial Domain Adaptation module is applied to achieve global feature alignment of the source domain and target domain and co-training two distinctive discriminators to mitigate the over-fitting issue. Experiments have been performed on 3T and 1.5T sMRI with domain shift in ADNI dataset. The experimental results demonstrated that the proposed framework PMDA outperforms supervised learning methods and several state-of-the-art domain adaptation methods and achieves a superior accuracy of 92.11%, 76.01% and 82.37% on AD vs. CN, AD vs. MCI, and MCI vs. CN tasks, respectively.
Collapse
Affiliation(s)
- Hongshun Cai
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, 400054, China
| | - Qiongmin Zhang
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, 400054, China.
| | - Ying Long
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, 400054, China
| |
Collapse
|
46
|
Chu L, Li Q, Yang B, Chen L, Shen C, Wang D. Exploring the essence of compound fault diagnosis: A novel multi-label domain adaptation method and its application to bearings. Heliyon 2023; 9:e14545. [PMID: 36950628 PMCID: PMC10025141 DOI: 10.1016/j.heliyon.2023.e14545] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2022] [Revised: 03/05/2023] [Accepted: 03/09/2023] [Indexed: 03/13/2023] Open
Abstract
Compound fault diagnosis in essence is a fundamental but difficult problem to be solved. The separation and extraction of compound fault features remain great challenges in industrial applications due to the lack of labeled fault data. This paper proposes a novel multi-label domain adaptation method applicable to compound fault diagnosis of bearings. Firstly, multi-layer domain adaptation is designed based on a fault feature extractor with customized residual blocks. In that way, features from discrepant domain can be transformed into domain-invariant features. Furthermore, a multi-label classifier is applied to decompose compound fault features into corresponding single fault feature, and diagnoses them separately. The application on bearing datasets demonstrates that the proposed method could enhance the detachable degree of compound faults and achieve greater diagnostic performance than other existing methods.
Collapse
Affiliation(s)
- Liuxing Chu
- School of Mechanical and Electric Engineering, Soochow University, Suzhou, 215000, China
| | - Qi Li
- School of Mechanical and Electric Engineering, Soochow University, Suzhou, 215000, China
| | - Bingru Yang
- School of Mechanical and Electric Engineering, Soochow University, Suzhou, 215000, China
| | - Liang Chen
- School of Mechanical and Electric Engineering, Soochow University, Suzhou, 215000, China
- Corresponding author.
| | - Changqing Shen
- School of Mechanical and Electric Engineering, Soochow University, Suzhou, 215000, China
| | - Dong Wang
- The State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, 200240, China
| |
Collapse
|
47
|
Guan H, Liu M. DomainATM: Domain adaptation toolbox for medical data analysis. Neuroimage 2023; 268:119863. [PMID: 36610676 PMCID: PMC9908850 DOI: 10.1016/j.neuroimage.2023.119863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 12/11/2022] [Accepted: 01/03/2023] [Indexed: 01/06/2023] Open
Abstract
Domain adaptation (DA) is an important technique for modern machine learning-based medical data analysis, which aims at reducing distribution differences between different medical datasets. A proper domain adaptation method can significantly enhance the statistical power by pooling data acquired from multiple sites/centers. To this end, we have developed the Domain Adaptation Toolbox for Medical data analysis (DomainATM) - an open-source software package designed for fast facilitation and easy customization of domain adaptation methods for medical data analysis. The DomainATM is implemented in MATLAB with a user-friendly graphical interface, and it consists of a collection of popular data adaptation algorithms that have been extensively applied to medical image analysis and computer vision. With DomainATM, researchers are able to facilitate fast feature-level and image-level adaptation, visualization and performance evaluation of different adaptation methods for medical data analysis. More importantly, the DomainATM enables the users to develop and test their own adaptation methods through scripting, greatly enhancing its utility and extensibility. An overview characteristic and usage of DomainATM is presented and illustrated with three example experiments, demonstrating its effectiveness, simplicity, and flexibility. The software, source code, and manual are available online.
Collapse
Affiliation(s)
| | - Mingxia Liu
- The Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA.
| |
Collapse
|
48
|
Li X, Huang J, Wang C, Yu X, Zhao T, Huang C, Gao Y. Expectation-maximization algorithm leads to domain adaptation for a perineural invasion and nerve extraction task in whole slide digital pathology images. Med Biol Eng Comput 2023; 61:457-73. [PMID: 36496513 DOI: 10.1007/s11517-022-02711-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Accepted: 10/22/2022] [Indexed: 12/14/2022]
Abstract
In addition to lymphatic and vascular channels, tumor cells can also spread via nerves, i.e., perineural invasion (PNI). PNI serves as an independent prognostic indicator in many malignancies. As a result, identifying and determining the extent of PNI is an important yet extremely tedious task in surgical pathology. In this work, we present a computational approach to extract nerves and PNI from whole slide histopathology images. We make manual annotations on selected prostate cancer slides once but then apply the trained model for nerve segmentation to both prostate cancer slides and head and neck cancer slides. For the purpose of multi-domain learning/prediction and investigation on the generalization capability of deep neural network, an expectation-maximization (EM)-based domain adaptation approach is proposed to improve the segmentation performance, in particular for the head and neck cancer slides. Experiments are conducted to demonstrate the segmentation performances. The average Dice coefficient for prostate cancer slides is 0.82 and 0.79 for head and neck cancer slides. Comparisons are then made for segmentations with and without the proposed EM-based domain adaptation on prostate cancer and head and neck cancer whole slide histopathology images from The Cancer Genome Atlas (TCGA) database and significant improvements are observed.
Collapse
|
49
|
Jiménez-Sánchez A, Tardy M, González Ballester MA, Mateus D, Piella G. Memory-aware curriculum federated learning for breast cancer classification. Comput Methods Programs Biomed 2023; 229:107318. [PMID: 36592580 DOI: 10.1016/j.cmpb.2022.107318] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/23/2022] [Revised: 11/25/2022] [Accepted: 12/17/2022] [Indexed: 06/17/2023]
Abstract
BACKGROUND AND OBJECTIVE For early breast cancer detection, regular screening with mammography imaging is recommended. Routine examinations result in datasets with a predominant amount of negative samples. The limited representativeness of positive cases can be problematic for learning Computer-Aided Diagnosis (CAD) systems. Collecting data from multiple institutions is a potential solution to mitigate this problem. Recently, federated learning has emerged as an effective tool for collaborative learning. In this setting, local models perform computation on their private data to update the global model. The order and the frequency of local updates influence the final global model. In the context of federated adversarial learning to improve multi-site breast cancer classification, we investigate the role of the order in which samples are locally presented to the optimizers. METHODS We define a novel memory-aware curriculum learning method for the federated setting. We aim to improve the consistency of the local models penalizing inconsistent predictions, i.e., forgotten samples. Our curriculum controls the order of the training samples prioritizing those that are forgotten after the deployment of the global model. Our approach is combined with unsupervised domain adaptation to deal with domain shift while preserving data privacy. RESULTS Two classification metrics: area under the receiver operating characteristic curve (ROC-AUC) and area under the curve for the precision-recall curve (PR-AUC) are used to evaluate the performance of the proposed method. Our method is evaluated with three clinical datasets from different vendors. An ablation study showed the improvement of each component of our method. The AUC and PR-AUC are improved on average by 5% and 6%, respectively, compared to the conventional federated setting. CONCLUSIONS We demonstrated the benefits of curriculum learning for the first time in a federated setting. Our results verified the effectiveness of the memory-aware curriculum federated learning for the multi-site breast cancer classification. Our code is publicly available at: https://github.com/ameliajimenez/curriculum-federated-learning.
Collapse
Affiliation(s)
- Amelia Jiménez-Sánchez
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain; IT University of Copenhagen, Copenhagen, Denmark.
| | - Mickael Tardy
- École Centrale Nantes, LS2N, UMR 6004, Nantes, France; Hera-MI SAS, Nantes, France
| | - Miguel A González Ballester
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain; ICREA, Barcelona, Spain
| | - Diana Mateus
- École Centrale Nantes, LS2N, UMR 6004, Nantes, France
| | - Gemma Piella
- BCN MedTech, Department of Information and Communication Technologies, Universitat Pompeu Fabra, Barcelona, Spain
| |
Collapse
|
50
|
Zhou Y, Koyuncu C, Lu C, Grobholz R, Katz I, Madabhushi A, Janowczyk A. Multi-site cross-organ calibrated deep learning (MuSClD): Automated diagnosis of non-melanoma skin cancer. Med Image Anal 2023; 84:102702. [PMID: 36516556 PMCID: PMC9825103 DOI: 10.1016/j.media.2022.102702] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2022] [Revised: 11/09/2022] [Accepted: 11/21/2022] [Indexed: 11/25/2022]
Abstract
Although deep learning (DL) has demonstrated impressive diagnostic performance for a variety of computational pathology tasks, this performance often markedly deteriorates on whole slide images (WSI) generated at external test sites. This phenomenon is due in part to domain shift, wherein differences in test-site pre-analytical variables (e.g., slide scanner, staining procedure) result in WSI with notably different visual presentations compared to training data. To ameliorate pre-analytic variances, approaches such as CycleGAN can be used to calibrate visual properties of images between sites, with the intent of improving DL classifier generalizability. In this work, we present a new approach termed Multi-Site Cross-Organ Calibration based Deep Learning (MuSClD) that employs WSIs of an off-target organ for calibration created at the same site as the on-target organ, based off the assumption that cross-organ slides are subjected to a common set of pre-analytical sources of variance. We demonstrate that by using an off-target organ from the test site to calibrate training data, the domain shift between training and testing data can be mitigated. Importantly, this strategy uniquely guards against potential data leakage introduced during calibration, wherein information only available in the testing data is imparted on the training data. We evaluate MuSClD in the context of the automated diagnosis of non-melanoma skin cancer (NMSC). Specifically, we evaluated MuSClD for identifying and distinguishing (a) basal cell carcinoma (BCC), (b) in-situ squamous cell carcinomas (SCC-In Situ), and (c) invasive squamous cell carcinomas (SCC-Invasive), using an Australian (training, n = 85) and a Swiss (held-out testing, n = 352) cohort. Our experiments reveal that MuSCID reduces the Wasserstein distances between sites in terms of color, contrast, and brightness metrics, without imparting noticeable artifacts to training data. The NMSC-subtyping performance is statistically improved as a result of MuSCID in terms of one-vs. rest AUC: BCC (0.92 vs 0.87, p = 0.01), SCC-In Situ (0.87 vs 0.73, p = 0.15) and SCC-Invasive (0.92 vs 0.82, p = 1e-5). Compared to baseline NMSC-subtyping with no calibration, the internal validation results of MuSClD (BCC (0.98), SCC-In Situ (0.92), and SCC-Invasive (0.97)) suggest that while domain shift indeed degrades classification performance, our on-target calibration using off-target tissue can safely compensate for pre-analytical variabilities, while improving the robustness of the model.
Collapse
Affiliation(s)
- Yufei Zhou
- Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, OH, USA
| | - Can Koyuncu
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA,Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, USA
| | - Cheng Lu
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA
| | - Rainer Grobholz
- Institute of Pathology, Cantonal Hospital Aarau, Aarau, Switzerland,Medical Faculty University of Zurich, Zurich, Switzerland
| | - Ian Katz
- Southern Sun Pathology, Sydney, NSW, Australia,University of Queensland, Brisbane, Qld, Australia
| | - Anant Madabhushi
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA; Atlanta VA Medical Center, Atlanta, USA.
| | - Andrew Janowczyk
- Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA, USA,Department of Oncology, Lausanne University Hospital,Department of Diagnostics, Division of Clinical Pathology, Geneva University Hospitals
| |
Collapse
|