1
|
Sun H, Chen P, Hu Z, Wei L. Multi-objective evolutionary multitasking algorithm based on cross-task transfer solution matching strategy. ISA TRANSACTIONS 2023; 138:504-520. [PMID: 36948908 DOI: 10.1016/j.isatra.2023.03.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/16/2022] [Revised: 03/12/2023] [Accepted: 03/12/2023] [Indexed: 06/16/2023]
Abstract
The superior performance of evolutionary multitasking (EMT) algorithms is largely owing to the potential synergy between tasks. Current EMT algorithms only involve a unidirectional process of transferring individuals from the source task to the target task. This method does not consider the search preference of the target task in the process of finding transferred individuals; therefore, the potential synergy between tasks is not fully utilized. Herein, we propose a bidirectional knowledge transfer method, which refers to the search preference of the target task in the process of finding transferred individuals. These transferred individuals fit the search process well for the target task. In addition, an adaptive strategy for adjusting the intensity of the knowledge transfer is proposed. This method enables the algorithm to adjust the intensity of knowledge transfer independently according to the living conditions of the individuals to be transferred to balance the convergence of the population with the computational intensity of the algorithm. The proposed algorithm is compared with comparison algorithms on 38 multi-objective multitasking optimization benchmarks. Experimental results show that the proposed algorithm is not only outperforming other comparison algorithms in more than 30 benchmarks, but also has considerable convergence efficiency.
Collapse
Affiliation(s)
- Hao Sun
- Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, China.
| | - Pengfei Chen
- Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, China
| | - Ziyu Hu
- Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, China
| | - Lixin Wei
- Engineering Research Center of the Ministry of Education for Intelligent Control System and Intelligent Equipment, Yanshan University, Qinhuangdao, China
| |
Collapse
|
2
|
Rahimi A, Gonen M. Efficient Multitask Multiple Kernel Learning With Application to Cancer Research. IEEE TRANSACTIONS ON CYBERNETICS 2022; 52:8716-8728. [PMID: 33705328 DOI: 10.1109/tcyb.2021.3052357] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Multitask multiple kernel learning (MKL) algorithms combine the capabilities of incorporating different data sources into the prediction model and using the data from one task to improve the accuracy on others. However, these methods do not necessarily produce interpretable results. Restricting the solutions to the set of interpretable solutions increases the computational burden of the learning problem significantly, leading to computationally prohibitive run times for some important biomedical applications. That is why we propose a multitask MKL formulation with a clustering of tasks and develop a highly time-efficient solution approach for it. Our solution method is based on the Benders decomposition and treating the clustering problem as finding a given number of tree structures in a graph; hence, it is called the forest formulation. We use our method to discriminate early-stage and late-stage cancers using genomic data and gene sets and compare our algorithm against two other algorithms. The two other algorithms are based on different approaches for linearization of the problem while all algorithms make use of the cutting-plane method. Our results indicate that as the number of tasks and/or the number of desired clusters increase, the forest formulation becomes increasingly favorable in terms of computational performance.
Collapse
|
3
|
Xue X, Zhang K, Tan KC, Feng L, Wang J, Chen G, Zhao X, Zhang L, Yao J. Affine Transformation-Enhanced Multifactorial Optimization for Heterogeneous Problems. IEEE TRANSACTIONS ON CYBERNETICS 2022; 52:6217-6231. [PMID: 33320820 DOI: 10.1109/tcyb.2020.3036393] [Citation(s) in RCA: 21] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Evolutionary multitasking (EMT) is a newly emerging research topic in the community of evolutionary computation, which aims to improve the convergence characteristic across multiple distinct optimization tasks simultaneously by triggering knowledge transfer among them. Unfortunately, most of the existing EMT algorithms are only capable of boosting the optimization performance for homogeneous problems which explicitly share the same (or similar) fitness landscapes. Seldom efforts have been devoted to generalize the EMT for solving heterogeneous problems. A few preliminary studies employ domain adaptation techniques to enhance the transferability between two distinct tasks. However, almost all of these methods encounter a severe issue which is the so-called degradation of intertask mapping. Keeping this in mind, a novel rank loss function for acquiring a superior intertask mapping is proposed in this article. In particular, with an evolutionary-path-based representation model for optimization instance, an analytical solution of affine transformation for bridging the gap between two distinct problems is mathematically derived from the proposed rank loss function. It is worth mentioning that the proposed mapping-based transferability enhancement technique can be seamlessly embedded into an EMT paradigm. Finally, the efficacy of our proposed method against several state-of-the-art EMTs is verified experimentally on a number of synthetic multitasking and many-tasking benchmark problems, as well as a practical case study.
Collapse
|
4
|
Bao G, Chen H, Liu T, Gong G, Yin Y, Wang L, Wang X. COVID-MTL: Multitask learning with Shift3D and random-weighted loss for COVID-19 diagnosis and severity assessment. PATTERN RECOGNITION 2022; 124:108499. [PMID: 34924632 PMCID: PMC8666107 DOI: 10.1016/j.patcog.2021.108499] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/31/2020] [Revised: 11/11/2021] [Accepted: 12/10/2021] [Indexed: 05/07/2023]
Abstract
There is an urgent need for automated methods to assist accurate and effective assessment of COVID-19. Radiology and nucleic acid test (NAT) are complementary COVID-19 diagnosis methods. In this paper, we present an end-to-end multitask learning (MTL) framework (COVID-MTL) that is capable of automated and simultaneous detection (against both radiology and NAT) and severity assessment of COVID-19. COVID-MTL learns different COVID-19 tasks in parallel through our novel random-weighted loss function, which assigns learning weights under Dirichlet distribution to prevent task dominance; our new 3D real-time augmentation algorithm (Shift3D) introduces space variances for 3D CNN components by shifting low-level feature representations of volumetric inputs in three dimensions; thereby, the MTL framework is able to accelerate convergence and improve joint learning performance compared to single-task models. By only using chest CT scans, COVID-MTL was trained on 930 CT scans and tested on separate 399 cases. COVID-MTL achieved AUCs of 0.939 and 0.846, and accuracies of 90.23% and 79.20% for detection of COVID-19 against radiology and NAT, respectively, which outperformed the state-of-the-art models. Meanwhile, COVID-MTL yielded AUC of 0.800 ± 0.020 and 0.813 ± 0.021 (with transfer learning) for classifying control/suspected, mild/regular, and severe/critically-ill cases. To decipher the recognition mechanism, we also identified high-throughput lung features that were significantly related (P < 0.001) to the positivity and severity of COVID-19.
Collapse
Affiliation(s)
- Guoqing Bao
- School of Computer Science, The University of Sydney, J12/1 Cleveland St, Darlington, Sydney, NSW 2008, Australia
| | - Huai Chen
- Department of Automation, Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Tongliang Liu
- School of Computer Science, The University of Sydney, J12/1 Cleveland St, Darlington, Sydney, NSW 2008, Australia
| | - Guanzhong Gong
- Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Jinan 250117, China
| | - Yong Yin
- Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Jinan 250117, China
| | - Lisheng Wang
- Department of Automation, Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Xiuying Wang
- School of Computer Science, The University of Sydney, J12/1 Cleveland St, Darlington, Sydney, NSW 2008, Australia
| |
Collapse
|
5
|
Multitask transfer learning with kernel representation. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07126-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
6
|
Abhadiomhen SE, Nzeh RC, Ganaa ED, Nwagwu HC, Okereke GE, Routray S. Supervised Shallow Multi-task Learning: Analysis of Methods. Neural Process Lett 2022. [DOI: 10.1007/s11063-021-10703-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
7
|
Xu Z, Liu X, Zhang K, He J. Cultural transmission based multi-objective evolution strategy for evolutionary multitasking. Inf Sci (N Y) 2022. [DOI: 10.1016/j.ins.2021.09.007] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
8
|
Zhang C, Tao D, Hu T, Liu B. Generalization Bounds of Multitask Learning From Perspective of Vector-Valued Function Learning. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2021; 32:1906-1919. [PMID: 32497006 DOI: 10.1109/tnnls.2020.2995428] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
In this article, we study the generalization performance of multitask learning (MTL) by considering MTL as a learning process of vector-valued functions (VFs). We will answer two theoretical questions, given a small size training sample: 1) under what conditions does MTL perform better than single-task learning (STL)? And 2) under what conditions does MTL guarantee the consistency of all tasks during learning? In contrast to the conventional task-summation based MTL, the introduction of VF form enables us to detect the behavior of each task and the task-group relatedness in MTL. Specifically, the task-group relatedness examines how the success (or failure) of some tasks affects the performance of the other tasks. By deriving the specific deviation and symmetrization inequalities for VFs, we obtain a generalization bound for MTL to the upper bound of the joint probability that there is at least one task with a large generalization gap. To answer the first question, we discuss how the synergic relatedness between task groups affects the generalization performance of MTL and shows that MTL outperforms STL if almost any pair of complementary task groups is predominantly synergic. Moreover, to answer the second question, we present a sufficient condition to guarantee the consistency of each task in MTL, which requires that the function class of each task should not have high complexity. In addition, our findings provide a strategy to examine whether the task settings will enjoy the advantages of MTL.
Collapse
|
9
|
Zheng Q, Wang Y, Heng PA. Multitask Feature Learning Meets Robust Tensor Decomposition for EEG Classification. IEEE TRANSACTIONS ON CYBERNETICS 2021; 51:2242-2252. [PMID: 31689229 DOI: 10.1109/tcyb.2019.2946914] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
In this article, we study a tensor-based multitask learning (MTL) method for classification. Taking into account the fact that in many real-world applications, the given training samples are limited and can be inherently arranged into multidimensional arrays (tensors), we are motivated by the advantages of MTL, where the shared structural information among related tasks can be leveraged to produce better generalization performance. We propose a regularized tensor-based MTL method for joint feature selection and classification. For feature selection, we employ the Fisher discriminant criterion to both select discriminative features and control the within-class nonstationarity. For classification, we take both shared and task-specific structural information into consideration. We decompose the regression tensor for each task into a linear combination of a shared tensor and a task-specific tensor and propose a composite tensor norm. Specifically, we use the scaled latent trace norm for regularizing the shared tensor and the l1 -norm for task-specific tensor. Further, we give a computationally efficient optimization algorithm based on the alternating direction method of multipliers (ADMMs) to tackle the joint learning of discriminative features and multitask classification. The experimental results on real electroencephalography (EEG) datasets demonstrate the superiority of our method over the state-of-the-art techniques.
Collapse
|
10
|
He F, Liu T, Tao D. Why ResNet Works? Residuals Generalize. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2020; 31:5349-5362. [PMID: 32031953 DOI: 10.1109/tnnls.2020.2966319] [Citation(s) in RCA: 49] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Residual connections significantly boost the performance of deep neural networks. However, few theoretical results address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This article studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We first present an upper bound of the covering number of networks with residual connections. This bound shares a similar structure with that of neural networks without residual connections. This result suggests that moving a weight matrix or nonlinear activation from the bone to a vine would not increase the hypothesis space. Afterward, an O(1 / √N) margin-based multiclass generalization bound is obtained for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straightforward. According to the obtained generalization bound, we should introduce regularization terms to control the magnitude of the norms of weight matrices not to increase too much, in practice, to ensure a good generalization ability, which justifies the technique of weight decay.
Collapse
|
11
|
Ding L, Liao S, Liu Y, Liu L, Zhu F, Yao Y, Shao L, Gao X. Approximate Kernel Selection via Matrix Approximation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2020; 31:4881-4891. [PMID: 31945003 DOI: 10.1109/tnnls.2019.2958922] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Kernel selection is of fundamental importance for the generalization of kernel methods. This article proposes an approximate approach for kernel selection by exploiting the approximability of kernel selection and the computational virtue of kernel matrix approximation. We define approximate consistency to measure the approximability of the kernel selection problem. Based on the analysis of approximate consistency, we solve the theoretical problem of whether, under what conditions, and at what speed, the approximate criterion is close to the accurate one, establishing the foundations of approximate kernel selection. We introduce two selection criteria based on error estimation and prove the approximate consistency of the multilevel circulant matrix (MCM) approximation and Nyström approximation under these criteria. Under the theoretical guarantees of the approximate consistency, we design approximate algorithms for kernel selection, which exploits the computational advantages of the MCM and Nyström approximations to conduct kernel selection in a linear or quasi-linear complexity. We experimentally validate the theoretical results for the approximate consistency and evaluate the effectiveness of the proposed kernel selection algorithms.
Collapse
|