A framework for gait-based recognition using Kinect

Data augmentation by separating identity and emotion representations for emotional gait recognition

Human-centered intelligent human–robot interaction can transcend the traditional keyboard and mouse and have the capacity to understand human communicative intentions by actively mining implicit human clues (e.g., identity information and emotional information) to meet individuals’ needs. Gait is a unique biometric feature that can provide reliable information to recognize emotions even when viewed from a distance. However, the insufficient amount and diversity of training data annotated with emotions severely hinder the application of gait emotion recognition. In this paper, we propose an adversarial learning framework for emotional gait dataset augmentation, with which a two-stage model can be trained to generate a number of synthetic emotional samples by separating identity and emotion representations from gait trajectories. To our knowledge, this is the first work to realize the mutual transformation between natural gait and emotional gait. Experimental results reveal that the synthetic gait samples generated by the proposed networks are rich in emotional information. As a result, the emotion classifier trained on the augmented dataset is competitive with state-of-the-art gait emotion recognition works.

A Gait-Based Real-Time Gender Classification System Using Whole Body Joints

Gait-based gender classification is a challenging task since people may walk in different directions with varying speed, gait style, and occluded joints. The majority of research studies in the literature focused on gender-specific joints, while there is less attention on the comparison of all of a body's joints. To consider all of the joints, it is essential to determine a person's gender based on their gait using a Kinect sensor. This paper proposes a logistic-regression-based machine learning model using whole body joints for gender classification. The proposed method consists of different phases including gait feature extraction based on three dimensional (3D) positions, feature selection, and classification of human gender. The Kinect sensor is used to extract 3D features of different joints. Different statistical tools such as Cronbach's alpha, correlation, t-test, and ANOVA techniques are exploited to select significant joints. The Coronbach's alpha technique yields an average result of 99.74%, which indicates the reliability of joints. Similarly, the correlation results indicate that there is significant difference between male and female joints during gait. t-test and ANOVA approaches demonstrate that all twenty joints are statistically significant for gender classification, because the p-value for each joint is zero and less than 1%. Finally, classification is performed based on the selected features using binary logistic regression model. A total of hundred (100) volunteers participated in the experiments in real scenario. The suggested method successfully classifies gender based on 3D features recorded in real-time using machine learning classifier with an accuracy of 98.0% using all body joints. The proposed method outperformed the existing systems which mostly rely on digital images.

ST-DeepGait: A Spatiotemporal Deep Learning Model for Human Gait Recognition

Human gait analysis presents an opportunity to study complex spatiotemporal data transpiring as co-movement patterns of multiple moving objects (i.e., human joints). Such patterns are acknowledged as movement signatures specific to an individual, offering the possibility to identify each individual based on unique gait patterns. We present a spatiotemporal deep learning model, dubbed ST-DeepGait, to featurize spatiotemporal co-movement patterns of human joints, and accordingly classify such patterns to enable human gait recognition. To this end, the ST-DeepGait model architecture is designed according to the spatiotemporal human skeletal graph in order to impose learning the salient local spatial dynamics of gait as they occur over time. Moreover, we employ a multi-layer RNN architecture to induce a sequential notion of gait cycles in the model. Our experimental results show that ST-DeepGait can achieve recognition accuracy rates over 90%. Furthermore, we qualitatively evaluate the model with the class embeddings to show interpretable separability of the features in geometric latent space. Finally, to evaluate the generalizability of our proposed model, we perform a zero-shot detection on 10 classes of data completely unseen during training and achieve a recognition accuracy rate of 88% overall. With this paper, we also contribute our gait dataset captured with an RGB-D sensor containing approximately 30 video samples of each subject for 100 subjects totaling 3087 samples. While we use human gait analysis as a motivating application to evaluate ST-DeepGait, we believe that this model can be simply adopted and adapted to study co-movement patterns of multiple moving objects in other applications such as in sports analytics and traffic pattern analysis.

KinectGaitNet: Kinect-Based Gait Recognition Using Deep Convolutional Neural Network

Over the past decade, gait recognition had gained a lot of attention in various research and industrial domains. These include remote surveillance, border control, medical rehabilitation, emotion detection from posture, fall detection, and sports training. The main advantages of identifying a person by their gait include unobtrusiveness, acceptance, and low costs. This paper proposes a convolutional neural network KinectGaitNet for Kinect-based gait recognition. The 3D coordinates of each of the body joints over the gait cycle are transformed to create a unique input representation. The proposed KinectGaitNet is trained directly using the 3D input representation without the necessity of the handcrafted features. The KinectGaitNet design allows avoiding gait cycle resampling, and the residual learning method ensures high accuracy without the degradation problem. The proposed deep learning architecture surpasses the recognition performance of all state-of-the-art methods for Kinect-based gait recognition by achieving 96.91% accuracy on UPCV and 99.33% accuracy on the KGB dataset. The method is the first, to the best of our knowledge, deep learning-based architecture that is based on a unique 3D input representation of joint coordinates. It achieves performance higher than previous traditional and deep learning methods, with fewer parameters and shorter inference time.

Geometric Regularization of Local Activations for Knowledge Transfer in Convolutional Neural Networks

In this work, we propose a mechanism for knowledge transfer between Convolutional Neural Networks via the geometric regularization of local features produced by the activations of convolutional layers. We formulate appropriate loss functions, driving a “student” model to adapt such that its local features exhibit similar geometrical characteristics to those of an “instructor” model, at corresponding layers. The investigated functions, inspired by manifold-to-manifold distance measures, are designed to compare the neighboring information inside the feature space of the involved activations without any restrictions in the features’ dimensionality, thus enabling knowledge transfer between different architectures. Experimental evidence demonstrates that the proposed technique is effective in different settings, including knowledge-transfer to smaller models, transfer between different deep architectures and harnessing knowledge from external data, producing models with increased accuracy compared to a typical training. Furthermore, results indicate that the presented method can work synergistically with methods such as knowledge distillation, further increasing the accuracy of the trained models. Finally, experiments on training with limited data show that a combined regularization scheme can achieve the same generalization as a non-regularized training with 50% of the data in the CIFAR-10 classification task.

Frailty Level Classification of the Community Elderly Using Microsoft Kinect-Based Skeleton Pose: A Machine Learning Approach

Frailty is one of the most important geriatric syndromes, which can be associated with increased risk for incident disability and hospitalization. Developing a real-time classification model of elderly frailty level could be beneficial for designing a clinical predictive assessment tool. Hence, the objective of this study was to predict the elderly frailty level utilizing the machine learning approach on skeleton data acquired from a Kinect sensor. Seven hundred and eighty-seven community elderly were recruited in this study. The Kinect data were acquired from the elderly performing different functional assessment exercises including: (1) 30-s arm curl; (2) 30-s chair sit-to-stand; (3) 2-min step; and (4) gait analysis tests. The proposed methodology was successfully validated by gender classification with accuracies up to 84 percent. Regarding frailty level evaluation and prediction, the results indicated that support vector classifier (SVC) and multi-layer perceptron (MLP) are the most successful estimators in prediction of the Fried’s frailty level with median accuracies up to 97.5 percent. The high level of accuracy achieved with the proposed methodology indicates that ML modeling can identify the risk of frailty in elderly individuals based on evaluating the real-time skeletal movements using the Kinect sensor.

Two-dimensional video-based analysis of human gait using pose estimation

Human gait analysis is often conducted in clinical and basic research, but many common approaches (e.g., three-dimensional motion capture, wearables) are expensive, immobile, data-limited, and require expertise. Recent advances in video-based pose estimation suggest potential for gait analysis using two-dimensional video collected from readily accessible devices (e.g., smartphones). To date, several studies have extracted features of human gait using markerless pose estimation. However, we currently lack evaluation of video-based approaches using a dataset of human gait for a wide range of gait parameters on a stride-by-stride basis and a workflow for performing gait analysis from video. Here, we compared spatiotemporal and sagittal kinematic gait parameters measured with OpenPose (open-source video-based human pose estimation) against simultaneously recorded three-dimensional motion capture from overground walking of healthy adults. When assessing all individual steps in the walking bouts, we observed mean absolute errors between motion capture and OpenPose of 0.02 s for temporal gait parameters (i.e., step time, stance time, swing time and double support time) and 0.049 m for step lengths. Accuracy improved when spatiotemporal gait parameters were calculated as individual participant mean values: mean absolute error was 0.01 s for temporal gait parameters and 0.018 m for step lengths. The greatest difference in gait speed between motion capture and OpenPose was less than 0.10 m s⁻¹. Mean absolute error of sagittal plane hip, knee and ankle angles between motion capture and OpenPose were 4.0°, 5.6° and 7.4°. Our analysis workflow is freely available, involves minimal user input, and does not require prior gait analysis expertise. Finally, we offer suggestions and considerations for future applications of pose estimation for human gait analysis.

There is a growing interest among clinicians and researchers to use novel pose estimation algorithms that automatically track human movement to analyze human gait. Gait analysis is routinely conducted in designated laboratories with specialized equipment. On the other hand, pose estimation relies on digital videos that can be recorded from household devices such as a smartphone. As a result, these new techniques make it possible to move beyond the laboratory and perform gait analysis in other settings such as the home or clinic. Before such techniques are adopted, we identify a critical need for comparing outcome parameters against three-dimensional motion capture and to evaluate how camera viewpoint affect outcome parameters. We used simultaneous motion capture and left- and right-side video recordings of healthy human gait and calculated spatiotemporal gait parameters and lower-limb joint angles. We find that our provided workflow estimates spatiotemporal gait parameters together with hip and knee angles with the accuracy and precision needed to detect changes in the gait pattern. We demonstrate that the position of the participant relative to the camera affect spatial measures such as step length and discuss the limitations posed by the current approach.

Gait Recognition and Understanding Based on Hierarchical Temporal Memory Using 3D Gait Semantic Folding

Gait recognition and understanding systems have shown a wide-ranging application prospect. However, their use of unstructured data from image and video has affected their performance, e.g., they are easily influenced by multi-views, occlusion, clothes, and object carrying conditions. This paper addresses these problems using a realistic 3-dimensional (3D) human structural data and sequential pattern learning framework with top-down attention modulating mechanism based on Hierarchical Temporal Memory (HTM). First, an accurate 2-dimensional (2D) to 3D human body pose and shape semantic parameters estimation method is proposed, which exploits the advantages of an instance-level body parsing model and a virtual dressing method. Second, by using gait semantic folding, the estimated body parameters are encoded using a sparse 2D matrix to construct the structural gait semantic image. In order to achieve time-based gait recognition, an HTM Network is constructed to obtain the sequence-level gait sparse distribution representations (SL-GSDRs). A top-down attention mechanism is introduced to deal with various conditions including multi-views by refining the SL-GSDRs, according to prior knowledge. The proposed gait learning model not only aids gait recognition tasks to overcome the difficulties in real application scenarios but also provides the structured gait semantic images for visual cognition. Experimental analyses on CMU MoBo, CASIA B, TUM-IITKGP, and KY4D datasets show a significant performance gain in terms of accuracy and robustness.

Score Level and Rank Level Fusion for Kinect-Based Multi-Modal Biometric System

Computational intelligence firmly made its way into the areas of consumer applications, banking, education, social networks, and security. Among all the applications, biometric systems play a significant role in ensuring an uncompromised and secure access to resources and facilities. This article presents a first multimodal biometric system that combines KINECT gait modality with KINECT face modality utilizing the rank level and the score level fusion. For the KINECT gait modality, a new approach is proposed based on the skeletal information processing. The gait cycle is calculated using three consecutive local minima computed for the distance between left and right ankles. The feature distance vectors are calculated for each person’s gait cycle, which allows extracting the biometric features such as the mean and the variance of the feature distance vector. For Kinect face recognition, a novel method based on HOG features has been developed. Then, K-nearest neighbors feature matching algorithm is applied as feature classification for both gait and face biometrics. Two fusion algorithms are implemented. The combination of Borda count and logistic regression approaches are used in the rank level fusion. The weighted sum method is used for score level fusion. The recognition accuracy obtained for multi-modal biometric recognition system tested on KINECT Gait and KINECT Eurocom Face datasets is 93.33% for Borda count rank level fusion, 96.67% for logistic regression rank-level fusion and 96.6% for score level fusion.

Riemannian Spatio-Temporal Features of Locomotion for Individual Recognition

Individual recognition based on skeletal sequence is a challenging computer vision task with multiple important applications, such as public security, human⁻computer interaction, and surveillance. However, much of the existing work usually fails to provide any explicit quantitative differences between different individuals. In this paper, we propose a novel 3D spatio-temporal geometric feature representation of locomotion on Riemannian manifold, which explicitly reveals the intrinsic differences between individuals. To this end, we construct mean sequence by aligning related motion sequences on the Riemannian manifold. The differences in respect to this mean sequence are modeled as spatial state descriptors. Subsequently, a temporal hierarchy of covariance are imposed on the state descriptors, making it a higher-order statistical spatio-temporal feature representation, showing unique biometric characteristics for individuals. Finally, we introduce a kernel metric learning method to improve the classification accuracy. We evaluated our method on two public databases: the CMU Mocap database and the UPCV Gait database. Furthermore, we also constructed a new database for evaluating running and analyzing two major influence factors of walking. As a result, the proposed approach achieves promising results in all experiments.

Understanding the Physiological Significance of Four Inertial Gait Features in Multiple Sclerosis

Gait impairment in multiple sclerosis (MS) can result from muscle weakness, physical fatigue, lack of coordination, and other symptoms. Walking speed, as measured by a number of clinician-administered walking tests, is the primary measure of gait impairment used by clinical researchers, but inertial gait features from body-worn sensors have been proven to add clinical value. This paper seeks to understand and differentiate the physiological significance of four such features with proven value in MS to facilitate adoption by clinical researchers and incorporation in gait monitoring and analysis systems. In addition, this information can be used to select features that might be appropriate in other forms of disability. Two of the four features are computed using the dynamic time warping (DTW) algorithm: The "DTW Score" is based on the usual DTW distance, and the "Warp Score" is based on the warping length. The third feature, based on kernel density estimation (KDE), is the "KDE Peak" value. Finally, the "Causality Index" is based on the phase slope index between inertial signals from different body parts. Relationships between these measures and the aforementioned gait-related symptoms are determined by applying factor analysis to three common, clinical walking outcomes, then correlating the inertial measures as well as walking speed to each extracted factor. Statistically significant differences in correlation coefficients to the three extracted clinical factors support their distinct physiological meaning and suggest they may have complimentary roles in the analysis of MS-related walking disability.

A review of the historical use and criticisms of gait analysis evidence

The use of gait analysis is a well-established facet of practice for many professions and a fundamental aspect of clinical practice. In recent times, gait analysis evidence has emerged as a new area of forensic practice. As its use has continued to spread and develop, the area of work has come under close scrutiny and subsequent criticism. The purpose of this paper is to examine the historical use of gait analysis evidence and consider the criticisms of this work. Through the use of the historical records of cases within the public domain it has been determined that gait analysis as evidence was first presented in court over 175 years ago, although it has only been utilized by experts in more recent times. The quality of analysis underpinning such evidence has been variable, and has been undertaken by both non-expert and expert witnesses. The work undertaken by expert witnesses appears to have been both non-scientific and scientific in nature, though there is limited reporting of cases involving scientific approaches. Given the variation in the quality of the methodologies utilized, there is the potential for confusion within the courts, where it may be difficult for the judge or jury to determine the appropriate weight that can be attributed to the evidence. It is concluded that future publications should explore the scientific basis of forensic gait analysis to evaluate standards, reliability and validity, as well as reporting the methodologies utilized in relevant cases in the field. It is also recommended that courts consider in greater depth an expert's theoretical approach and experience prior to admitting their evidence. The publication of 'Forensic gait analysis: a primer for courts', although limited in some aspects of its consideration of practice, is a welcome addition to the information available for guidance.