Multiple Kernel Clustering With Neighbor-Kernel Subspace Segmentation

Asymmetric double-winged multi-view clustering network for exploring diverse and consistent information

In unsupervised scenarios, deep contrastive multi-view clustering (DCMVC) is becoming a hot research spot, which aims to mine the potential relationships between different views. Most existing DCMVC algorithms focus on exploring the consistency information for the deep semantic features, while ignoring the diverse information on shallow features. To fill this gap, we propose a novel multi-view clustering network termed CodingNet to explore the diverse and consistent information simultaneously in this paper. Specifically, instead of utilizing the conventional auto-encoder, we design an asymmetric structure network to extract shallow and deep features separately. Then, by approximating the similarity matrix on the shallow feature to the zero matrix, we ensure the diversity for the shallow features, thus offering a better description of multi-view data. Moreover, we propose a dual contrastive mechanism that maintains consistency for deep features at both view-feature and pseudo-label levels. Our framework's efficacy is validated through extensive experiments on six widely used benchmark datasets, outperforming most state-of-the-art multi-view clustering algorithms.

A Review of Convex Clustering From Multiple Perspectives: Models, Optimizations, Statistical Properties, Applications, and Connections

Traditional partition-based clustering is very sensitive to the initialized centroids, which are easily stuck in the local minimum due to their nonconvex objectives. To this end, convex clustering is proposed by relaxing K -means clustering or hierarchical clustering. As an emerging and excellent clustering technology, convex clustering can solve the instability problems of partition-based clustering methods. Generally, convex clustering objective consists of the fidelity and the shrinkage terms. The fidelity term encourages the cluster centroids to estimate the observations and the shrinkage term shrinks the cluster centroids matrix so that their observations share the same cluster centroid in the same category. Regularized by the lpn -norm ( pn ∈ {1,2,+∞} ), the convex objective guarantees the global optimal solution of the cluster centroids. This survey conducts a comprehensive review of convex clustering. It starts with the convex clustering as well as its nonconvex variants and then concentrates on the optimization algorithms and the hyperparameters setting. In particular, the statistical properties, the applications, and the connections of convex clustering with other methods are reviewed and discussed thoroughly for a better understanding the convex clustering. Finally, we briefly summarize the development of convex clustering and present some potential directions for future research.

Deep Double Incomplete Multi-View Multi-Label Learning With Incomplete Labels and Missing Views

View missing and label missing are two challenging problems in the applications of multi-view multi-label classification scenery. In the past years, many efforts have been made to address the incomplete multi-view learning or incomplete multi-label learning problem. However, few works can simultaneously handle the challenging case with both the incomplete issues. In this article, we propose a new incomplete multi-view multi-label learning network to address this challenging issue. The proposed method is composed of four major parts: view-specific deep feature extraction network, weighted representation fusion module, classification module, and view-specific deep decoder network. By, respectively, integrating the view missing information and label missing information into the weighted fusion module and classification module, the proposed method can effectively reduce the negative influence caused by two such incomplete issues and sufficiently explore the available data and label information to obtain the most discriminative feature extractor and classifier. Furthermore, our method can be trained in both supervised and semi-supervised manners, which has important implications for flexible deployment. Experimental results on five benchmarks in supervised and semi-supervised cases demonstrate that the proposed method can greatly enhance the classification performance on the difficult incomplete multi-view multi-label classification tasks with missing labels and missing views.

Fast Incomplete Multi-View Clustering With View-Independent Anchors

Multi-view clustering (MVC) methods aim to exploit consistent and complementary information among each view and achieve encouraging performance improvement than single-view counterparts. In practical applications, it is common to obtain instances with partially available information, raising researches of incomplete multi-view clustering (IMC) issues. Recently, several fast IMC methods have been proposed to process the large-scale partial data. Though with considerable acceleration, these methods seek view-shared anchors and ignore specific information among single views. To tackle the above issue, we propose a fast IMC with view-independent anchors (FIMVC-VIA) method in this article. Specifically, we learn individual anchors based on the diversity of distribution among each incomplete view and construct a unified anchor graph following the principle of consistent clustering structure. By constructing an anchor graph instead of pairwise full graph, the time and space complexities of our proposed FIMVC-VIA are proven to be linearly related to the number of samples, which can efficiently solve the large-scale task. The experiment performed on benchmarks with different missing rate illustrates the improvement in complexity and effectiveness of our method compared with other IMC methods. Our code is publicly available at https://github.com/Tracesource/ FIMVC-VIA.

Interpolation-Based Contrastive Learning for Few-Label Semi-Supervised Learning

Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. In the existing literature, consistency regularization-based methods, which force the perturbed samples to have similar predictions with the original ones have attracted much attention for their promising accuracy. However, we observe that the performance of such methods decreases drastically when the labels get extremely limited, e.g., 2 or 3 labels for each category. Our empirical study finds that the main problem lies with the drift of semantic information in the procedure of data augmentation. The problem can be alleviated when enough supervision is provided. However, when little guidance is available, the incorrect regularization would mislead the network and undermine the performance of the algorithm. To tackle the problem, we: 1) propose an interpolation-based method to construct more reliable positive sample pairs and 2) design a novel contrastive loss to guide the embedding of the learned network to change linearly between samples so as to improve the discriminative capability of the network by enlarging the margin decision boundaries. Since no destructive regularization is introduced, the performance of our proposed algorithm is largely improved. Specifically, the proposed algorithm outperforms the second best algorithm (Comatch) with 5.3% by achieving 88.73% classification accuracy when only two labels are available for each class on the CIFAR-10 dataset. Moreover, we further prove the generality of the proposed method by improving the performance of the existing state-of-the-art algorithms considerably with our proposed strategy. The corresponding code is available at https://github.com/xihongyang1999/ICL_SSL.

Local Sample-Weighted Multiple Kernel Clustering With Consensus Discriminative Graph

Multiple kernel clustering (MKC) is committed to achieving optimal information fusion from a set of base kernels. Constructing precise and local kernel matrices is proven to be of vital significance in applications since the unreliable distant-distance similarity estimation would degrade clustering performance. Although existing localized MKC algorithms exhibit improved performance compared with globally designed competitors, most of them widely adopt the KNN mechanism to localize kernel matrix by accounting for τ -nearest neighbors. However, such a coarse manner follows an unreasonable strategy that the ranking importance of different neighbors is equal, which is impractical in applications. To alleviate such problems, this article proposes a novel local sample-weighted MKC (LSWMKC) model. We first construct a consensus discriminative affinity graph in kernel space, revealing the latent local structures. Furthermore, an optimal neighborhood kernel for the learned affinity graph is output with naturally sparse property and clear block diagonal structure. Moreover, LSWMKC implicitly optimizes adaptive weights on different neighbors with corresponding samples. Experimental results demonstrate that our LSWMKC possesses better local manifold representation and outperforms existing kernel or graph-based clustering algorithms. The source code of LSWMKC can be publicly accessed from https://github.com/liliangnudt/LSWMKC.

Robust Discriminant Subspace Clustering With Adaptive Local Structure Embedding

Unsupervised dimension reduction and clustering are frequently used as two separate steps to conduct clustering tasks in subspace. However, the two-step clustering methods may not necessarily reflect the cluster structure in the subspace. In addition, the existing subspace clustering methods do not consider the relationship between the low-dimensional representation and local structure in the input space. To address the above issues, we propose a robust discriminant subspace (RDS) clustering model with adaptive local structure embedding. Specifically, unlike the existing methods which incorporate dimension reduction and clustering via regularizer, thereby introducing extra parameters, RDS first integrates them into a unified matrix factorization (MF) model through theoretical proof. Furthermore, a similarity graph is constructed to learn the local structure. A constraint is imposed on the graph to guarantee that it has the same connected components with low-dimensional representation. In this spirit, the similarity graph serves as a tradeoff that adaptively balances the learning process between the low-dimensional space and the original space. Finally, RDS adopts the l_2,1 -norm to measure the residual error, which enhances the robustness to noise. Using the property of the l_2,1 -norm, RDS can be optimized efficiently without introducing more penalty terms. Experimental results on real-world benchmark datasets show that RDS can provide more interpretable clustering results and also outperform other state-of-the-art alternatives.

Logarithmic Schatten- p Norm Minimization for Tensorial Multi-View Subspace Clustering

The low-rank tensor could characterize inner structure and explore high-order correlation among multi-view representations, which has been widely used in multi-view clustering. Existing approaches adopt the tensor nuclear norm (TNN) as a convex approximation of non-convex tensor rank function. However, TNN treats the different singular values equally and over-penalizes the main rank components, leading to sub-optimal tensor representation. In this paper, we devise a better surrogate of tensor rank, namely the tensor logarithmic Schatten- p norm ([Formula: see text]N), which fully considers the physical difference between singular values by the non-convex and non-linear penalty function. Further, a tensor logarithmic Schatten- p norm minimization ([Formula: see text]NM)-based multi-view subspace clustering ([Formula: see text]NM-MSC) model is proposed. Specially, the proposed [Formula: see text]NM can not only protect the larger singular values encoded with useful structural information, but also remove the smaller ones encoded with redundant information. Thus, the learned tensor representation with compact low-rank structure will well explore the complementary information and accurately characterize the high-order correlation among multi-views. The alternating direction method of multipliers (ADMM) is used to solve the non-convex multi-block [Formula: see text]NM-MSC model where the challenging [Formula: see text]NM problem is carefully handled. Importantly, the algorithm convergence analysis is mathematically established by showing that the sequence generated by the algorithm is of Cauchy and converges to a Karush-Kuhn-Tucker (KKT) point. Experimental results on nine benchmark databases reveal the superiority of the [Formula: see text]NM-MSC model.

Multiple Kernel Clustering With Compressed Subspace Alignment

Multiple kernel clustering (MKC) has recently achieved remarkable progress in fusing multisource information to boost the clustering performance. However, the O(n²) memory consumption and O(n³) computational complexity prohibit these methods from being applied into median- or large-scale applications, where n denotes the number of samples. To address these issues, we carefully redesign the formulation of subspace segmentation-based MKC, which reduces the memory and computational complexity to O(n) and O(n²) , respectively. The proposed algorithm adopts a novel sampling strategy to enhance the performance and accelerate the speed of MKC. Specifically, we first mathematically model the sampling process and then learn it simultaneously during the procedure of information fusion. By this way, the generated anchor point set can better serve data reconstruction across different views, leading to improved discriminative capability of the reconstruction matrix and boosted clustering performance. Although the integrated sampling process makes the proposed algorithm less efficient than the linear complexity algorithms, the elaborate formulation makes our algorithm straightforward for parallelization. Through the acceleration of GPU and multicore techniques, our algorithm achieves superior performance against the compared state-of-the-art methods on six datasets with comparable time cost to the linear complexity algorithms.

DLnet With Training Task Conversion Stream for Precise Semantic Segmentation in Actual Traffic Scene

Many successful semantic segmentation models trained on certain datasets experience a performance gap when they are applied to the actual scene images, expressing weak robustness of these models in the actual scene. The training task conversion (TTC) and domain adaption field have been originally proposed to solve the performance gap problem. Unfortunately, many existing models for TTC and domain adaptation have defects, and even if the TTC is completed, the performance is far from the original task model. Thus, how to maintain excellent performance while completing TTC is the main challenge. In order to address this challenge, a deep learning model named DLnet is proposed for TTC from the existing image dataset-based training task to the actual scene image-based training task. The proposed network, named the DLnet, contains three main innovations. The proposed network is verified by experiments. The experimental results show that the proposed DLnet not only can achieve state-of-the-art quantitative performance on four popular datasets but also can obtain outstanding qualitative performance in four actual urban scenes, which demonstrates the robustness and performance of the proposed DLnet. In addition, although the proposed DLnet cannot achieve outstanding performance in real time, it can still achieve a moderate performance in real time, which is within an acceptable range.

Multiview Subspace Clustering via Co-Training Robust Data Representation

Taking the assumption that data samples are able to be reconstructed with the dictionary formed by themselves, recent multiview subspace clustering (MSC) algorithms aim to find a consensus reconstruction matrix via exploring complementary information across multiple views. Most of them directly operate on the original data observations without preprocessing, while others operate on the corresponding kernel matrices. However, they both ignore that the collected features may be designed arbitrarily and hard guaranteed to be independent and nonoverlapping. As a result, original data observations and kernel matrices would contain a large number of redundant details. To address this issue, we propose an MSC algorithm that groups samples and removes data redundancy concurrently. In specific, eigendecomposition is employed to obtain the robust data representation of low redundancy for later clustering. By utilizing the two processes into a unified model, clustering results will guide eigendecomposition to generate more discriminative data representation, which, as feedback, helps obtain better clustering results. In addition, an alternate and convergent algorithm is designed to solve the optimization problem. Extensive experiments are conducted on eight benchmarks, and the proposed algorithm outperforms comparative ones in recent literature by a large margin, verifying its superiority. At the same time, its effectiveness, computational efficiency, and robustness to noise are validated experimentally.

Agglomerative Neural Networks for Multiview Clustering

Conventional multiview clustering methods seek a view consensus through minimizing the pairwise discrepancy between the consensus and subviews. However, pairwise comparison cannot portray the interview relationship precisely if some of the subviews can be further agglomerated. To address the above challenge, we propose the agglomerative analysis to approximate the optimal consensus view, thereby describing the subview relationship within a view structure. We present an agglomerative neural network (ANN) based on constrained Laplacian rank to cluster multiview data directly without a dedicated postprocessing step (e.g., using K -means). We further extend ANN with a learnable data space to handle data of complex scenarios. Our evaluations against several state-of-the-art multiview clustering approaches on four popular data sets show the promising view-consensus analysis ability of ANN. We further demonstrate ANN's capability in analyzing complex view structures, extensibility through our case study and robustness and effectiveness of data-driven modifications.

Deep structure integrative representation of multi-omics data for cancer subtyping

MOTIVATION

Cancer is a heterogeneous group of diseases. Cancer subtyping is crucial and critical step to diagnosis, prognosis and treatment. Since high-throughput sequencing technologies provide unprecedented opportunity to rapid collect multi-omics data for the same individuals, an urgent need in current is how to effectively represent and integrate these multi-omics data to achieve clinically meaningful cancer subtyping.

RESULTS

We propose a novel deep learning model, called Deep Structure Integrative Representation (DSIR), for cancer subtypes dentification by integrating representation and clustering multi-omics data. DSIR simultaneously captures the global structures in sparse subspace and local structures in manifold subspace from multi-omics data and constructs consensus similarity matrix by utilizing deep neural networks. Extensive tests are performed in twelve different cancers on three levels of omics data from The Cancer Genome Atlas. The results demonstrate that DSIR obtains more significant performances than the state-of-the-art integrative methods.

AVAILABILITY

https://github.com/Polytech-bioinf/Deep-structure-integrative-representation.git.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Robust Matrix Factorization With Spectral Embedding

Nonnegative matrix factorization (NMF) and spectral clustering are two of the most widely used clustering techniques. However, NMF cannot deal with the nonlinear data, and spectral clustering relies on the postprocessing. In this article, we propose a Robust Matrix factorization with Spectral embedding (RMS) approach for data clustering, which inherits the advantages of NMF and spectral clustering, while avoiding their shortcomings. In addition, to cluster the data represented by multiple views, we present the multiview version of RMS (M-RMS), and the weights of different views are self-tuned. The main contributions of this research are threefold: 1) by integrating spectral clustering and matrix factorization, the proposed methods are able to capture the nonlinear data structure and obtain the cluster indicator directly; 2) instead of using the squared Frobenius-norm, the objectives are developed with the l_2,1 -norm, such that the effects of the outliers are alleviated; and 3) the proposed methods are totally parameter-free, which increases the applicability for various real-world problems. Extensive experiments on several single-view/multiview data sets demonstrate the effectiveness of our methods and verify their superior clustering performance over the state of the arts.

Consensus Affinity Graph Learning for Multiple Kernel Clustering

Significant attention to multiple kernel graph-based clustering (MKGC) has emerged in recent years, primarily due to the superiority of multiple kernel learning (MKL) and the outstanding performance of graph-based clustering. However, many existing MKGC methods design a fat model that poses challenges for computational cost and clustering performance, as they learn both an affinity graph and an extra consensus kernel cumbersomely. To tackle this challenging problem, this article proposes a new MKGC method to learn a consensus affinity graph directly. By using the self-expressiveness graph learning and an adaptive local structure learning term, the local manifold structure of the data in kernel space is preserved for learning multiple candidate affinity graphs from a kernel pool first. After that, these candidate affinity graphs are synthesized to learn a consensus affinity graph via a thin autoweighted fusion model, in which a self-tuned Laplacian rank constraint and a top- k neighbors sparse strategy are introduced to improve the quality of the consensus affinity graph for accurate clustering purposes. The experimental results on ten benchmark datasets and two synthetic datasets show that the proposed method consistently and significantly outperforms the state-of-the-art methods.

V[Formula: see text]H: View Variation and View Heredity for Incomplete Multiview Clustering

Real data often appear in the form of multiple incomplete views. Incomplete multiview clustering is an effective method to integrate these incomplete views. Previous methods only learn the consistent information between different views and ignore the unique information of each view, which limits their clustering performance and generalizations. To overcome this limitation, we propose a novel View Variation and View Heredity approach (V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H). Inspired by the variation and the heredity in genetics, V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H first decomposes each subspace into a variation matrix for the corresponding view and a heredity matrix for all the views to represent the unique information and the consistent information respectively. Then, by aligning different views based on their cluster indicator matrices, V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H integrates the unique information from different views to improve the clustering performance. Finally, with the help of the adjustable low-rank representation based on the heredity matrix, V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H recovers the underlying true data structure to reduce the influence of the large incompleteness. More importantly, V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H presents possibly the first work to introduce genetics to clustering algorithms for learning simultaneously the consistent information and the unique information from incomplete multiview data. Extensive experimental results on fifteen benchmark datasets validate its superiority over other state-of-the-arts.

Impact Statement—Incomplete multiview clustering is a popular technology to cluster incomplete datasets from multiple sources. The technology is becoming more significant due to the absence of the expensive requirement of labeling these datasets. However, previous algorithms cannot fully learn the information of each view. Inspired by variation and heredity in genetics, our proposed algorithm V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H fully learns the information of each view. Compared with the state-of-the-art algorithms, V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H improves clustering performance by more than 20% in representative cases. With the large improvement on multiple datasets, V\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$^3$\end{document}H has wide potential applications including the analysis of pandemic, financial and election datasets. The DOI of our codes is 10.24 433/CO.2 119 636.v1.

Adaptive Weighted Graph Fusion Incomplete Multi-View Subspace Clustering

With the enormous amount of multi-source data produced by various sensors and feature extraction approaches, multi-view clustering (MVC) has attracted developing research attention and is widely exploited in data analysis. Most of the existing multi-view clustering methods hold on the assumption that all of the views are complete. However, in many real scenarios, multi-view data are often incomplete for many reasons, e.g., hardware failure or incomplete data collection. In this paper, we propose an adaptive weighted graph fusion incomplete multi-view subspace clustering (AWGF-IMSC) method to solve the incomplete multi-view clustering problem. Firstly, to eliminate the noise existing in the original space, we transform complete original data into latent representations which contribute to better graph construction for each view. Then, we incorporate feature extraction and incomplete graph fusion into a unified framework, whereas two processes can negotiate with each other, serving for graph learning tasks. A sparse regularization is imposed on the complete graph to make it more robust to the view-inconsistency. Besides, the importance of different views is automatically learned, further guiding the construction of the complete graph. An effective iterative algorithm is proposed to solve the resulting optimization problem with convergence. Compared with the existing state-of-the-art methods, the experiment results on several real-world datasets demonstrate the effectiveness and advancement of our proposed method.