Human Motion Segmentation via Robust Kernel Sparse Subspace Clustering

Segmentation and Completion of Human Motion Sequence via Temporal Learning of Subspace Variety Model

Subspace-based models have been extensively employed in unsupervised segmentation and completion of human motion sequence (HMS). However, existing approaches often neglect the incorporation of temporal priors embedded in HMS, resulting in suboptimal results. This paper presents a subspace variety model for HMS, along with an innovative Temporal Learning of Subspace Variety Model (TL-SVM) method for enhanced segmentation and completion in HMS. The key idea is to segment incomplete HMS into motion clusters and extracting the subspace features of each motion through the temporal learning of the subspace variety model. Subsequently, the HMS is completed based on the extracted subspace features. Thus, the main challenge is to learn the subspace variety model with temporal priors when confronted with missing entries. To tackle this, the paper develops a spatio-temporal assignment consistency (STAC) constraint for the subspace variety model, leveraging temporal priors embedded in HMS. In addition, a subspace clustering approach under the STAC constraint is proposed to learn the subspace variety model by extracting subspace features from HMS and segmenting HMS into motion clusters alternatively. The proposed subspace clustering model can also handle missing entries with theoretical guarantees. Furthermore, the missing entries of HMS are completed by minimizing the distance between each human motion frame and its corresponding subspace. Extensive experimental results, along with comparisons to state-of-the-art methods on four benchmark datasets, underscore the advantages of the proposed method.

A Deep Learning Framework for Start-End Frame Pair-Driven Motion Synthesis

A start-end frame pair and a motion pattern-based motion synthesis scheme can provide more control to the synthesis process and produce content-various motion sequences. However, the data preparation for the motion training is intractable, and concatenating feature spaces of the start-end frame pair and the motion pattern lacks theoretical rationality in previous works. In this article, we propose a deep learning framework that completes automatic data preparation and learns the nonlinear mapping from start-end frame pairs to motion patterns. The proposed model consists of three modules: action detection, motion extraction, and motion synthesis networks. The action detection network extends the deep subspace learning framework to a supervised version, i.e., uses the local self-expression (LSE) of the motion data to supervise feature learning and complement the classification error. A long short-term memory (LSTM)-based network is used to efficiently extract the motion patterns to address the speed deficiency reflected in the previous optimization-based method. A motion synthesis network consists of a group of LSTM-based blocks, where each of them is to learn the nonlinear relation between the start-end frame pairs and the motion patterns of a certain joint. The superior performances in action detection accuracy, motion pattern extraction efficiency, and motion synthesis quality show the effectiveness of each module in the proposed framework.

Fuzzy Sparse Subspace Clustering for Infrared Image Segmentation

Infrared image segmentation is a challenging task, due to interference of complex background and appearance inhomogeneity of foreground objects. A critical defect of fuzzy clustering for infrared image segmentation is that the method treats image pixels or fragments in isolation. In this paper, we propose to adopt self-representation from sparse subspace clustering in fuzzy clustering, aiming to introduce global correlation information into fuzzy clustering. Meanwhile, to apply sparse subspace clustering for non-linear samples from an infrared image, we leverage membership from fuzzy clustering to improve conventional sparse subspace clustering. The contributions of this paper are fourfold. First, by introducing self-representation coefficients modeled in sparse subspace clustering based on high-dimensional features, fuzzy clustering is capable of utilizing global information to resist complex background as well as intensity inhomogeneity of objects, so as to improve clustering accuracy. Second, fuzzy membership is tactfully exploited in the sparse subspace clustering framework. Thereby, the bottleneck of conventional sparse subspace clustering methods, that they could be barely applied to nonlinear samples, can be surmounted. Third, as we integrate fuzzy clustering and subspace clustering in a unified framework, features from two different aspects are employed, contributing to precise clustering results. Finally, we further incorporate neighbor information into clustering, thus effectively solving the uneven intensity problem in infrared image segmentation. Experiments examine the feasibility of proposed methods on various infrared images. Segmentation results demonstrate the effectiveness and efficiency of the proposed methods, which proves the superiority compared to other fuzzy clustering methods and sparse space clustering methods.

Consistency and Diversity Induced Human Motion Segmentation

Subspace clustering is a classical technique that has been widely used for human motion segmentation and other related tasks. However, existing segmentation methods often cluster data without guidance from prior knowledge, resulting in unsatisfactory segmentation results. To this end, we propose a novel Consistency and Diversity induced human Motion Segmentation (CDMS) algorithm. Specifically, our model factorizes the source and target data into distinct multi-layer feature spaces, in which transfer subspace learning is conducted on different layers to capture multi-level information. A multi-mutual consistency learning strategy is carried out to reduce the domain gap between the source and target data. In this way, the domain-specific knowledge and domain-invariant properties can be explored simultaneously. Besides, a novel constraint based on the Hilbert Schmidt Independence Criterion (HSIC) is introduced to ensure the diversity of multi-level subspace representations, which enables the complementarity of multi-level representations to be explored to boost the transfer learning performance. Moreover, to preserve the temporal correlations, an enhanced graph regularizer is imposed on the learned representation coefficients and the multi-level representations of the source data. The proposed model can be efficiently solved using the Alternating Direction Method of Multipliers (ADMM) algorithm. Extensive experimental results on public human motion datasets demonstrate the effectiveness of our method against several state-of-the-art approaches.

A Structure Constraint Matrix Factorization Framework for Human Behavior Segmentation

This article presents a structure constraint matrix factorization framework for different behavior segmentation of the human behavior sequential data. This framework is based on the structural information of the behavior continuity and the high similarity between neighboring frames. Due to the high similarity and high dimensionality of human behavior data, the high-precision segmentation of human behavior is hard to achieve from the perspective of application and academia. By making the behavior continuity hypothesis, first, the effective constraint regular terms are constructed. Subsequently, the clustering framework based on constrained non-negative matrix factorization is established. Finally, the segmentation result can be obtained by using the spectral clustering and graph segmentation algorithm. For illustration, the proposed framework is applied to the Weiz dataset, Keck dataset, mo_86 dataset, and mo_86_9 dataset. Empirical experiments on several public human behavior datasets demonstrate that the structure constraint matrix factorization framework can automatically segment human behavior sequences. Compared to the classical algorithm, the proposed framework can ensure consistent segmentation of sequential points within behavior actions and provide better performance in accuracy.

Multi-Attribute Subspace Clustering via Auto-Weighted Tensor Nuclear Norm Minimization

Self-expressiveness based subspace clustering methods have received wide attention for unsupervised learning tasks. However, most existing subspace clustering methods consider data features as a whole and then focus only on one single self-representation. These approaches ignore the intrinsic multi-attribute information embedded in the original data feature and result in one-attribute self-representation. This paper proposes a novel multi-attribute subspace clustering (MASC) model that understands data from multiple attributes. MASC simultaneously learns multiple subspace representations corresponding to each specific attribute by exploiting the intrinsic multi-attribute features drawn from original data. In order to better capture the high-order correlation among multi-attribute representations, we represent them as a tensor in low-rank structure and propose the auto-weighted tensor nuclear norm (AWTNN) as a superior low-rank tensor approximation. Especially, the non-convex AWTNN fully considers the difference between singular values through the implicit and adaptive weights splitting during the AWTNN optimization procedure. We further develop an efficient algorithm to optimize the non-convex and multi-block MASC model and establish the convergence guarantees. A more comprehensive subspace representation can be obtained via aggregating these multi-attribute representations, which can be used to construct a clustering-friendly affinity matrix. Extensive experiments on eight real-world databases reveal that the proposed MASC exhibits superior performance over other subspace clustering methods.

A Time-Division Position-Sensitive Detector Image System for High-Speed Multitarget Trajectory Tracking

High-speed trajectory tracking with real-time processing capability is particularly important in the fields of pilotless automobiles, guidance systems, robotics, and filmmaking. The conventional optical approach to high-speed trajectory tracking involves charge coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) image sensors, which suffer from trade-offs between resolution and framerates, complexity of the system, and enormous data-analysis processes. Here, a high-speed trajectory tracking system is designed by using a time-division position-sensitive detector (TD-PSD) based on a graphene-silicon Schottky heterojunction. Benefiting from the high-speed optoelectronic response and sub-micrometer positional accuracy of the TD-PSD, multitarget real-time trajectory tracking is realized, with a maximum image output framerate of up to 62 000 frames per second. Moreover, multichannel trajectory tracking and image-distortion correction functionalities are realized by TD-PSD systems through frequency-related image preprocessing, which significantly improves the capacity of real-time information processing and image quality in complicated light environments.

Local Self-Expression Subspace Learning Network for Motion Capture Data

Deep subspace learning is an important branch of self-supervised learning and has been a hot research topic in recent years, but current methods do not fully consider the individualities of temporal data and related tasks. In this paper, by transforming the individualities of motion capture data and segmentation task as the supervision, we propose the local self-expression subspace learning network. Specifically, considering the temporality of motion data, we use the temporal convolution module to extract temporal features. To implement the local validity of self-expression in temporal tasks, we design the local self-expression layer which only maintains the representation relations with temporally adjacent motion frames. To simulate the interpolatability of motion data in the feature space, we impose a group sparseness constraint on the local self-expression layer to impel the representations only using selected keyframes. Besides, based on the subspace assumption, we propose the subspace projection loss, which is induced from distances of each frame projected to the fitted subspaces, to penalize the potential clustering errors. The superior performances of the proposed model on the segmentation task of synthetic data and three tasks of real motion capture data demonstrate the feature learning ability of our model.

Autoencoder-Based Latent Block-Diagonal Representation for Subspace Clustering

Block-diagonal representation (BDR) is an effective subspace clustering method. The existing BDR methods usually obtain a self-expression coefficient matrix from the original features by a shallow linear model. However, the underlying structure of real-world data is often nonlinear, thus those methods cannot faithfully reflect the intrinsic relationship among samples. To address this problem, we propose a novel latent BDR (LBDR) model to perform the subspace clustering on a nonlinear structure, which jointly learns an autoencoder and a BDR matrix. The autoencoder, which consists of a nonlinear encoder and a linear decoder, plays an important role to learn features from the nonlinear samples. Meanwhile, the learned features are used as a new dictionary for a linear model with block-diagonal regularization, which can ensure good performances for spectral clustering. Moreover, we theoretically prove that the learned features are located in the linear space, thus ensuring the effectiveness of the linear model using self-expression. Extensive experiments on various real-world datasets verify the superiority of our LBDR over the state-of-the-art subspace clustering approaches.

Dynamic Sparse Subspace Clustering for Evolving High-Dimensional Data Streams

In an era of ubiquitous large-scale evolving data streams, data stream clustering (DSC) has received lots of attention because the scale of the data streams far exceeds the ability of expert human analysts. It has been observed that high-dimensional data are usually distributed in a union of low-dimensional subspaces. In this article, we propose a novel sparse representation-based DSC algorithm, called evolutionary dynamic sparse subspace clustering (EDSSC). It can cope with the time-varying nature of subspaces underlying the evolving data streams, such as subspace emergence, disappearance, and recurrence. The proposed EDSSC consists of two phases: 1) static learning and 2) online clustering. During the first phase, a data structure for storing the statistic summary of data streams, called EDSSC summary, is proposed which can better address the dilemma between the two conflicting goals: 1) saving more points for accuracy of subspace clustering (SC) and 2) discarding more points for the efficiency of DSC. By further proposing an algorithm to estimate the subspace number, the proposed EDSSC does not need to know the number of subspaces. In the second phase, a more suitable index, called the average sparsity concentration index (ASCI), is proposed, which dramatically promotes the clustering accuracy compared to the conventionally utilized SCI index. In addition, the subspace evolution detection model based on the Page-Hinkley test is proposed where the appearing, disappearing, and recurring subspaces can be detected and adapted. Extinct experiments on real-world data streams show that the EDSSC outperforms the state-of-the-art online SC approaches.

Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks

Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.

ARMA-Based Segmentation of Human Limb Motion Sequences

With the development of human motion capture (MoCap) equipment and motion analysis technologies, MoCap systems have been widely applied in many fields, including biomedicine, computer vision, virtual reality, etc. With the rapid increase in MoCap data collection in different scenarios and applications, effective segmentation of MoCap data is becoming a crucial issue for further human motion posture and behavior analysis, which requires both robustness and computation efficiency in the algorithm design. In this paper, we propose an unsupervised segmentation algorithm based on limb-bone partition angle body structural representation and autoregressive moving average (ARMA) model fitting. The collected MoCap data were converted into the angle sequence formed by the human limb-bone partition segment and the central spine segment. The limb angle sequences are matched by the ARMA model, and the segmentation points of the limb angle sequences are distinguished by analyzing the good of fitness of the ARMA model. A medial filtering algorithm is proposed to ensemble the segmentation results from individual limb motion sequences. A set of MoCap measurements were also conducted to evaluate the algorithm including typical body motions collected from subjects of different heights, and were labeled by manual segmentation. The proposed algorithm is compared with the principle component analysis (PCA), K-means clustering algorithm (K-means), and back propagation (BP) neural-network-based segmentation algorithms, which shows higher segmentation accuracy due to a more semantic description of human motions by limb-bone partition angles. The results highlight the efficiency and performance of the proposed algorithm, and reveals the potentials of this segmentation model on analyzing inter- and intra-motion sequence distinguishing.

Nonconvex Low-Rank Kernel Sparse Subspace Learning for Keyframe Extraction and Motion Segmentation

By exploiting the kernel trick, the sparse subspace model is extended to the nonlinear version with one or a combination of predefined kernels, but the high-dimensional space induced by predefined kernels is not guaranteed to be able to capture the features of the nonlinear data in theory. In this article, we propose a nonconvex low-rank learning framework in an unsupervised way to learn a kernel to replace the predefined kernel in the sparse subspace model. The learned kernel by a nonconvex relaxation of rank can better exploiting the low-rank property of nonlinear data to induce a high-dimensional Hilbert space that more closely approaches the true feature space. Furthermore, we give a global closed-form optimal solution of the nonconvex rank minimization and prove it. Considering the low-rank and sparseness characteristics of motion capture data in its feature space, we use them to verify the better representation of nonlinear data with the learned kernel via two tasks: keyframe extraction and motion segmentation. The performances on both tasks demonstrate the advantage of our model over the sparse subspace model with predefined kernels and some other related state-of-art methods.

Identifying ADHD Individuals From Resting-State Functional Connectivity Using Subspace Clustering and Binary Hypothesis Testing

Objective: This study focused on the ADHD classification through functional connectivity (FC) analysis. Method: An ADHD classification method was proposed with subspace clustering and binary hypothesis testing, wherein partial information of test data was adopted for training. By hypothesizing the binary label (ADHD or control) for the test data, two feature sets of training FC data were generated during the feature selection procedure that employed both training and test data. Then, a multi-affinity subspace clustering approach was performed to obtain the corresponding subspace-projected feature sets. With the energy comparison of projected feature sets, we finally identified ADHD individuals for the test data. Results: Our method outperformed several state-of-the-art methods with the above 90% average identification accuracy. By the discriminative FC contribution analysis, it also proved the reliability of our method. Conclusion: Results demonstrate the remarkable classification performance of our method and reveal some useful brain circuits to identify ADHD.

Dimensionality Reduction and Motion Clustering During Activities of Daily Living: Three-, Four-, and Seven-Degree-of-Freedom Arm Movements

This paper is the first in a two-part series analyzing human arm and hand motion during a wide range of unstructured tasks. The wide variety of motions performed by the human arm during daily tasks makes it desirable to find representative subsets to reduce the dimensionality of these movements for a variety of applications, including the design and control of robotic and prosthetic devices. This paper presents a novel method and the results of an extensive human subjects study to obtain representative arm joint angle trajectories that span naturalistic motions during Activities of Daily Living (ADLs). In particular, we seek to identify sets of useful motion trajectories of the upper limb that are functions of a single variable, allowing, for instance, an entire prosthetic or robotic arm to be controlled with a single input from a user, along with a means to select between motions for different tasks. Data driven approaches are used to discover clusters and representative motion averages for the wrist 3 degree of freedom (DOF), elbow-wrist 4 DOF, and full-arm 7 DOF motions. The proposed method makes use of well-known techniques such as dynamic time warping (DTW) to obtain a divergence measure between motion segments, Ward's distance criterion to build hierarchical trees, and functional principal component analysis (fPCA) to evaluate cluster variability. The emerging clusters associate various recorded motions into primarily hand start and end location for the full-arm system, motion direction for the wrist-only system, and an intermediate between the two qualities for the elbow-wrist system.

Learning the Distribution Preserving Semantic Subspace for Clustering

This paper proposes a new clustering method for images called distribution preserving indexing (DPI). It aims to find a lower dimensional semantic space approximating the original image space in the sense of preserving the distribution of the data. In the theory, the intrinsic structure of the data clusters can be described by the distribution of the data effectively. Therefore, the cluster structure of the data in a lower dimensional semantic space derived by the DPI becomes clear. Unlike these distance-based clustering methods, which reveal the intrinsic Euclidean structure of data, our method attempts to discover the intrinsic cluster structure of the data space that actually is the union of some sub-manifolds. Moreover, we propose a revised kernel density estimator for the case of high-dimensional data, which is a crucial step in DPI. In addition, we provide a theoretical analysis of the bound of our method. Finally, the extensive experiments compared with other algorithms, on COIL20, CBCL, and MNIST demonstrate the effectiveness of our proposed approach.