Deep clustering with a Dynamic Autoencoder: From reconstruction towards centroids construction

T-distributed Stochastic Neighbor Network for unsupervised representation learning

Unsupervised representation learning (URL) is still lack of a reasonable operator (e.g. convolution kernel) for exploring meaningful structural information from generic data including vector, image and tabular data. In this paper, we propose a simple end-to-end T-distributed Stochastic Neighbor Network (TsNet) for URL with clustering downstream task. Concretely, our TsNet model has three major components: (1) an adaptive connectivity distribution learning module is presented to construct a pairwise graph for preserving the local structure of generic data; (2) a T-distributed stochastic neighbor embedding based loss function is designed to learn a transformation between embeddings and original data, which improves the discrimination of representations; (3) a nonlinear parametric mapping is learned via our TsNet on an unsupervised generalized manner, which can address the "out-of-sample" issue. By combining these components, our method is able to considerably outperform previous related unsupervised learning approaches on visualization and clustering of generic data. A simple deep neural network equipped on our model respectively achieves 74.90%, 76.56% ACC and NMI, which is 8% relative improvement over previous state-of-the-art on real single-cell RNA-sequencing (scRNA-seq) datasets clustering.

iCVI-ARTMAP: Using Incremental Cluster Validity Indices and Adaptive Resonance Theory Reset Mechanism to Accelerate Validation and Achieve Multiprototype Unsupervised Representations

This article presents an adaptive resonance theory predictive mapping (ARTMAP) model, which uses incremental cluster validity indices (iCVIs) to perform unsupervised learning, namely, iCVI-ARTMAP. Incorporating iCVIs to the decision-making and many-to-one mapping capabilities of this adaptive resonance theory (ART)-based model can improve the choices of clusters to which samples are incrementally assigned. These improvements are accomplished by intelligently performing the operations of swapping sample assignments between clusters, splitting and merging clusters, and caching the values of variables when iCVI values need to be recomputed. Using recursive formulations enables iCVI-ARTMAP to considerably reduce the computational burden associated with cluster validity index (CVI)-based offline clustering. In this work, six iCVI-ARTMAP variants were realized via the integration of one information-theoretic and five sum-of-squares-based iCVIs into fuzzy ARTMAP. With proper choice of iCVI, iCVI-ARTMAP either outperformed or performed comparably to three ART-based and four non-ART-based clustering algorithms in experiments using benchmark datasets of different natures. Naturally, the performance of iCVI-ARTMAP is subject to the selected iCVI and its suitability to the data at hand; fortunately, it is a general model in which other iCVIs can be easily embedded.

Exploring Temporal Community Structure via Network Embedding

Temporal community detection is helpful to discover and analyze significant groups or clusters hidden in dynamic networks in the real world. A variety of methods, such as modularity optimization, spectral method, and statistical network model, has been developed from diversified perspectives. Recently, network embedding-based technologies have made significant progress, and one can exploit deep learning superiority to network tasks. Although some methods for static networks have shown promising results in boosting community detection by integrating community embedding, they are not suitable for temporal networks and unable to capture their dynamics. Furthermore, the dynamic embedding methods only model network varying without considering community structures. Hence, in this article, we propose a novel unsupervised dynamic community detection model, which is based on network embedding and can effectively discover temporal communities and model dynamic networks. More specifically, we propose the community prior by introducing the Gaussian mixture model (GMM) in the variational autoencoder, which can obtain community information and better model the evolutionary characteristics of community structure and node embedding by utilizing the variant of gated recurrent unit (GRU). Extensive experiments conducted in real-world and artificial networks demonstrate that our proposed model has a better effect on improving the accuracy of dynamic community detection.

ELINAC: Autoencoder Approach for Electronic Invoices Data Clustering

The most common method used to document monetary transactions in Brazil is by issuing electronic invoices (NF-e). The audit of electronic invoices is essential, and this can be improved by using data mining solutions, such as clustering and anomaly detection. However, applying these solutions is not a simple task because NF-e data contains millions of records with noisy fields and nonstandard documents, especially short text descriptions. In addition to these challenges, it is costly to extract information from short texts to identify traces of mismanagement, embezzlement, commercial fraud or tax evasion. Analyzing such data can be more effective when divided into well-defined groups. However, efficient solutions for clustering data with characteristics similar to NF-es have not yet been proposed in the literature. We developed ELINAC, a service for clustering short-text data in NF-es that uses an automatic encoder to cluster data. ELINAC aids in auditing transactions documented in NF-e, clustering similar data by short-text descriptions and making anomaly detection in numeric fields easier. For this, ELINAC explores how to model the automatic encoder without increasing the calculation costs to suppress a large number of short text data. In the worst case, the results show that ELINAC efficiently groups data while performing three times faster than solutions previously adopted in the literature.

Deformation Analysis of an Ultra-High Arch Dam under Different Water Level Conditions Based on Optimized Dynamic Panel Clustering

During the operation period, the deformation of an ultra-high arch dam is affected by the large fluctuation of the reservoir water level. Under the dual coupling of the ultra-high dam and the complex water level conditions, the traditional variational analysis method cannot be sufficiently applied to its deformation analysis. The deformation analysis of the ultra-high arch dam, however, is very important in order to judge the dam safety state. To analyze the deformation law of different parts of an ultra-high arch dam, the panel data clustering theory is used to construct a Spatio-temporal characteristic model of dam deformation. In order to solve the difficult problem of the fluctuating displacement of dam deformation with water level effect, three displacement component indexes (absolute quantity, growing, and fluctuation) are proposed to characterize dam deformation. To further optimize the panel clustering deformation model, the objective weight coefficient of clustering comprehensive distance is calculated based on the CRITIC (CRiteria Importance Through Inter-criteria Correlation) method. The zoning rules of the ultra-high arch dam are established by using the idea of the CSP (Constraint Satisfaction Problem) index, and the complex water level of the reservoir is simulated in the whole process. Finally, the dynamic cluster analysis of dam deformation is realized. Through a case study, three typical working conditions including the rapid rise and fall of water level and the normal operation are calculated, and the deformation laws of different deformation zones are analyzed. The results show that the model can reasonably describe the deformation law of an ultra-high arch dam under different water levels, conveniently and intuitively select representative measuring points and key monitoring parts, effectively reducing the analysis workload of lots of measuring points, and improve the reliability of arch dam deformation analysis.

A Deep Clustering Method For Analyzing Uterine Cervix Images Across Imaging Devices

Visual inspection of the cervix with acetic acid (VIA), though error prone, has long been used for screening women and to guide management for cervical cancer. The automated visual evaluation (AVE) technique, in which deep learning is used to predict precancer based on a digital image of the acetowhitened cervix, has demonstrated its promise as a low-cost method to improve on human performance. However, there are several challenges in moving AVE beyond proof-of-concept and deploying it as a practical adjunct tool in visual screening. One of them is making AVE robust across images captured using different devices. We propose a new deep learning based clustering approach to investigate whether the images taken by three different devices (a common smartphone, a custom smartphone-based handheld device for cervical imaging, and a clinical colposcope equipped with SLR digital camera-based imaging capability) can be well distinguished from each other with respect to the visual appearance/content within their cervix regions. We argue that disparity in visual appearance of a cervix across devices could be a significant confounding factor in training and generalizing AVE performance. Our method consists of four components: cervix region detection, feature extraction, feature encoding, and clustering. Multiple experiments are conducted to demonstrate the effectiveness of each component and compare alternative methods in each component. Our proposed method achieves high clustering accuracy (97%) and significantly outperforms several representative deep clustering methods on our dataset. The high clustering performance indicates the images taken from these three devices are different with respect to visual appearance. Our results and analysis establish a need for developing a method that minimizes such variance among the images acquired from different devices. It also recognizes the need for large number of training images from different sources for robust device-independent AVE performance worldwide.

Pseudo-Supervised Deep Subspace Clustering

Auto-Encoder (AE)-based deep subspace clustering (DSC) methods have achieved impressive performance due to the powerful representation extracted using deep neural networks while prioritizing categorical separability. However, self-reconstruction loss of an AE ignores rich useful relation information and might lead to indiscriminative representation, which inevitably degrades the clustering performance. It is also challenging to learn high-level similarity without feeding semantic labels. Another unsolved problem facing DSC is the huge memory cost due to n×n similarity matrix, which is incurred by the self-expression layer between an encoder and decoder. To tackle these problems, we use pairwise similarity to weigh the reconstruction loss to capture local structure information, while a similarity is learned by the self-expression layer. Pseudo-graphs and pseudo-labels, which allow benefiting from uncertain knowledge acquired during network training, are further employed to supervise similarity learning. Joint learning and iterative training facilitate to obtain an overall optimal solution. Extensive experiments on benchmark datasets demonstrate the superiority of our approach. By combining with the k -nearest neighbors algorithm, we further show that our method can address the large-scale and out-of-sample problems. The source code of our method is available: https://github.com/sckangz/SelfsupervisedSC.

End-to-end novel visual categories learning via auxiliary self-supervision

Semi-supervised learning has largely alleviated the strong demand for large amount of annotations in deep learning. However, most of the methods have adopted a common assumption that there is always labeled data from the same class of unlabeled data, which is impractical and restricted for real-world applications. In this research work, our focus is on semi-supervised learning when the categories of unlabeled data and labeled data are disjoint from each other. The main challenge is how to effectively leverage knowledge in labeled data to unlabeled data when they are independent from each other, and not belonging to the same categories. Previous state-of-the-art methods have proposed to construct pairwise similarity pseudo labels as supervising signals. However, two issues are commonly inherent in these methods: (1) All of previous methods are comprised of multiple training phases, which makes it difficult to train the model in an end-to-end fashion. (2) Strong dependence on the quality of pairwise similarity pseudo labels limits the performance as pseudo labels are vulnerable to noise and bias. Therefore, we propose to exploit the use of self-supervision as auxiliary task during model training such that labeled data and unlabeled data will share the same set of surrogate labels and overall supervising signals can have strong regularization. By doing so, all modules in the proposed algorithm can be trained simultaneously, which will boost the learning capability as end-to-end learning can be achieved. Moreover, we propose to utilize local structure information in feature space during pairwise pseudo label construction, as local properties are more robust to noise. Extensive experiments have been conducted on three frequently used visual datasets, i.e., CIFAR-10, CIFAR-100 and SVHN, in this paper. Experiment results have indicated the effectiveness of our proposed algorithm as we have achieved new state-of-the-art performance for novel visual categories learning for these three datasets.