Eigenfunction-Based Multitask Learning in a Reproducing Kernel Hilbert Space

Multi-objective evolutionary multitasking algorithm based on cross-task transfer solution matching strategy

The superior performance of evolutionary multitasking (EMT) algorithms is largely owing to the potential synergy between tasks. Current EMT algorithms only involve a unidirectional process of transferring individuals from the source task to the target task. This method does not consider the search preference of the target task in the process of finding transferred individuals; therefore, the potential synergy between tasks is not fully utilized. Herein, we propose a bidirectional knowledge transfer method, which refers to the search preference of the target task in the process of finding transferred individuals. These transferred individuals fit the search process well for the target task. In addition, an adaptive strategy for adjusting the intensity of the knowledge transfer is proposed. This method enables the algorithm to adjust the intensity of knowledge transfer independently according to the living conditions of the individuals to be transferred to balance the convergence of the population with the computational intensity of the algorithm. The proposed algorithm is compared with comparison algorithms on 38 multi-objective multitasking optimization benchmarks. Experimental results show that the proposed algorithm is not only outperforming other comparison algorithms in more than 30 benchmarks, but also has considerable convergence efficiency.

Efficient Multitask Multiple Kernel Learning With Application to Cancer Research

Multitask multiple kernel learning (MKL) algorithms combine the capabilities of incorporating different data sources into the prediction model and using the data from one task to improve the accuracy on others. However, these methods do not necessarily produce interpretable results. Restricting the solutions to the set of interpretable solutions increases the computational burden of the learning problem significantly, leading to computationally prohibitive run times for some important biomedical applications. That is why we propose a multitask MKL formulation with a clustering of tasks and develop a highly time-efficient solution approach for it. Our solution method is based on the Benders decomposition and treating the clustering problem as finding a given number of tree structures in a graph; hence, it is called the forest formulation. We use our method to discriminate early-stage and late-stage cancers using genomic data and gene sets and compare our algorithm against two other algorithms. The two other algorithms are based on different approaches for linearization of the problem while all algorithms make use of the cutting-plane method. Our results indicate that as the number of tasks and/or the number of desired clusters increase, the forest formulation becomes increasingly favorable in terms of computational performance.

Affine Transformation-Enhanced Multifactorial Optimization for Heterogeneous Problems

Evolutionary multitasking (EMT) is a newly emerging research topic in the community of evolutionary computation, which aims to improve the convergence characteristic across multiple distinct optimization tasks simultaneously by triggering knowledge transfer among them. Unfortunately, most of the existing EMT algorithms are only capable of boosting the optimization performance for homogeneous problems which explicitly share the same (or similar) fitness landscapes. Seldom efforts have been devoted to generalize the EMT for solving heterogeneous problems. A few preliminary studies employ domain adaptation techniques to enhance the transferability between two distinct tasks. However, almost all of these methods encounter a severe issue which is the so-called degradation of intertask mapping. Keeping this in mind, a novel rank loss function for acquiring a superior intertask mapping is proposed in this article. In particular, with an evolutionary-path-based representation model for optimization instance, an analytical solution of affine transformation for bridging the gap between two distinct problems is mathematically derived from the proposed rank loss function. It is worth mentioning that the proposed mapping-based transferability enhancement technique can be seamlessly embedded into an EMT paradigm. Finally, the efficacy of our proposed method against several state-of-the-art EMTs is verified experimentally on a number of synthetic multitasking and many-tasking benchmark problems, as well as a practical case study.

COVID-MTL: Multitask learning with Shift3D and random-weighted loss for COVID-19 diagnosis and severity assessment

There is an urgent need for automated methods to assist accurate and effective assessment of COVID-19. Radiology and nucleic acid test (NAT) are complementary COVID-19 diagnosis methods. In this paper, we present an end-to-end multitask learning (MTL) framework (COVID-MTL) that is capable of automated and simultaneous detection (against both radiology and NAT) and severity assessment of COVID-19. COVID-MTL learns different COVID-19 tasks in parallel through our novel random-weighted loss function, which assigns learning weights under Dirichlet distribution to prevent task dominance; our new 3D real-time augmentation algorithm (Shift3D) introduces space variances for 3D CNN components by shifting low-level feature representations of volumetric inputs in three dimensions; thereby, the MTL framework is able to accelerate convergence and improve joint learning performance compared to single-task models. By only using chest CT scans, COVID-MTL was trained on 930 CT scans and tested on separate 399 cases. COVID-MTL achieved AUCs of 0.939 and 0.846, and accuracies of 90.23% and 79.20% for detection of COVID-19 against radiology and NAT, respectively, which outperformed the state-of-the-art models. Meanwhile, COVID-MTL yielded AUC of 0.800 ± 0.020 and 0.813 ± 0.021 (with transfer learning) for classifying control/suspected, mild/regular, and severe/critically-ill cases. To decipher the recognition mechanism, we also identified high-throughput lung features that were significantly related (P < 0.001) to the positivity and severity of COVID-19.

Generalization Bounds of Multitask Learning From Perspective of Vector-Valued Function Learning

In this article, we study the generalization performance of multitask learning (MTL) by considering MTL as a learning process of vector-valued functions (VFs). We will answer two theoretical questions, given a small size training sample: 1) under what conditions does MTL perform better than single-task learning (STL)? And 2) under what conditions does MTL guarantee the consistency of all tasks during learning? In contrast to the conventional task-summation based MTL, the introduction of VF form enables us to detect the behavior of each task and the task-group relatedness in MTL. Specifically, the task-group relatedness examines how the success (or failure) of some tasks affects the performance of the other tasks. By deriving the specific deviation and symmetrization inequalities for VFs, we obtain a generalization bound for MTL to the upper bound of the joint probability that there is at least one task with a large generalization gap. To answer the first question, we discuss how the synergic relatedness between task groups affects the generalization performance of MTL and shows that MTL outperforms STL if almost any pair of complementary task groups is predominantly synergic. Moreover, to answer the second question, we present a sufficient condition to guarantee the consistency of each task in MTL, which requires that the function class of each task should not have high complexity. In addition, our findings provide a strategy to examine whether the task settings will enjoy the advantages of MTL.

Multitask Feature Learning Meets Robust Tensor Decomposition for EEG Classification

In this article, we study a tensor-based multitask learning (MTL) method for classification. Taking into account the fact that in many real-world applications, the given training samples are limited and can be inherently arranged into multidimensional arrays (tensors), we are motivated by the advantages of MTL, where the shared structural information among related tasks can be leveraged to produce better generalization performance. We propose a regularized tensor-based MTL method for joint feature selection and classification. For feature selection, we employ the Fisher discriminant criterion to both select discriminative features and control the within-class nonstationarity. For classification, we take both shared and task-specific structural information into consideration. We decompose the regression tensor for each task into a linear combination of a shared tensor and a task-specific tensor and propose a composite tensor norm. Specifically, we use the scaled latent trace norm for regularizing the shared tensor and the l₁ -norm for task-specific tensor. Further, we give a computationally efficient optimization algorithm based on the alternating direction method of multipliers (ADMMs) to tackle the joint learning of discriminative features and multitask classification. The experimental results on real electroencephalography (EEG) datasets demonstrate the superiority of our method over the state-of-the-art techniques.

Why ResNet Works? Residuals Generalize

Residual connections significantly boost the performance of deep neural networks. However, few theoretical results address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This article studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We first present an upper bound of the covering number of networks with residual connections. This bound shares a similar structure with that of neural networks without residual connections. This result suggests that moving a weight matrix or nonlinear activation from the bone to a vine would not increase the hypothesis space. Afterward, an O(1 / √N) margin-based multiclass generalization bound is obtained for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straightforward. According to the obtained generalization bound, we should introduce regularization terms to control the magnitude of the norms of weight matrices not to increase too much, in practice, to ensure a good generalization ability, which justifies the technique of weight decay.

Approximate Kernel Selection via Matrix Approximation

Kernel selection is of fundamental importance for the generalization of kernel methods. This article proposes an approximate approach for kernel selection by exploiting the approximability of kernel selection and the computational virtue of kernel matrix approximation. We define approximate consistency to measure the approximability of the kernel selection problem. Based on the analysis of approximate consistency, we solve the theoretical problem of whether, under what conditions, and at what speed, the approximate criterion is close to the accurate one, establishing the foundations of approximate kernel selection. We introduce two selection criteria based on error estimation and prove the approximate consistency of the multilevel circulant matrix (MCM) approximation and Nyström approximation under these criteria. Under the theoretical guarantees of the approximate consistency, we design approximate algorithms for kernel selection, which exploits the computational advantages of the MCM and Nyström approximations to conduct kernel selection in a linear or quasi-linear complexity. We experimentally validate the theoretical results for the approximate consistency and evaluate the effectiveness of the proposed kernel selection algorithms.