Individual recognition using gait energy image

Pattern analysis using lower body human walking data to identify the gaitprint

All people have a fingerprint that is unique to them and persistent throughout life. Similarly, we propose that people have a gaitprint, a persistent walking pattern that contains unique information about an individual. To provide evidence of a unique gaitprint, we aimed to identify individuals based on basic spatiotemporal variables. 81 adults were recruited to walk overground on an indoor track at their own pace for four minutes wearing inertial measurement units. A total of 18 trials per participant were completed between two days, one week apart. Four methods of pattern analysis, a) Euclidean distance, b) cosine similarity, c) random forest, and d) support vector machine, were applied to our basic spatiotemporal variables such as step and stride lengths to accurately identify people. Our best accuracy (98.63%) was achieved by random forest, followed by support vector machine (98.40%), and the top 10 most similar trials from cosine similarity (98.40%). Our results clearly demonstrate a persistent walking pattern with sufficient information about the individual to make them identifiable, suggesting the existence of a gaitprint.

Emerging trends in gait recognition based on deep learning: a survey

Gait recognition, a biometric identification method, has garnered significant attention due to its unique attributes, including non-invasiveness, long-distance capture, and resistance to impersonation. Gait recognition has undergone a revolution driven by the remarkable capacity of deep learning to extract complicated features from data. An overview of the current developments in deep learning-based gait identification methods is provided in this work. We explore and analyze the development of gait recognition and highlight its uses in forensics, security, and criminal investigations. The article delves into the challenges associated with gait recognition, such as variations in walking conditions, viewing angles, and clothing as well. We discuss about the effectiveness of deep neural networks in addressing these challenges by providing a comprehensive analysis of state-of-the-art architectures, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and attention mechanisms. Diverse neural network-based gait recognition models, such as Gate Controlled and Shared Attention ICDNet (GA-ICDNet), Multi-Scale Temporal Feature Extractor (MSTFE), GaitNet, and various CNN-based approaches, demonstrate impressive accuracy across different walking conditions, showcasing the effectiveness of these models in capturing unique gait patterns. GaitNet achieved an exceptional identification accuracy of 99.7%, whereas GA-ICDNet showed high precision with an equal error rate of 0.67% in verification tasks. GaitGraph (ResGCN+2D CNN) achieved rank-1 accuracies ranging from 66.3% to 87.7%, whereas a Fully Connected Network with Koopman Operator achieved an average rank-1 accuracy of 74.7% for OU-MVLP across various conditions. However, GCPFP (GCN with Graph Convolution-Based Part Feature Polling) utilizing graph convolutional network (GCN) and GaitSet achieves the lowest average rank-1 accuracy of 62.4% for CASIA-B, while MFINet (Multiple Factor Inference Network) exhibits the lowest accuracy range of 11.72% to 19.32% under clothing variation conditions on CASIA-B. In addition to an across-the-board analysis of recent breakthroughs in gait recognition, the scope for potential future research direction is also assessed.

An end-to-end gait recognition system for covariate conditions using custom kernel CNN

Gait recognition is the identification of individuals based on how they walk. It can identify an individual of interest without their intervention, making it better suited for surveillance from afar. Computer-aided silhouette-based gait analysis is frequently employed due to its efficiency and effectiveness. However, covariate conditions have a significant influence on individual recognition because they conceal essential features that are helpful in recognizing individuals from their walking style. To address such issues, we proposed a novel deep-learning framework to tackle covariate conditions in gait by proposing regions subject to covariate conditions. The features extracted from those regions will be neglected to keep the model's performance effective with custom kernels. The proposed technique sets aside static and dynamic areas of interest, where static areas contain covariates, and then features are learnt from the dynamic regions unaffected by covariates to effectively recognize individuals. The features were extracted using three customized kernels, and the results were concatenated to produce a fused feature map. Afterward, CNN learns and extracts the features from the proposed regions to recognize an individual. The suggested approach is an end-to-end system that eliminates the requirement for manual region proposal and feature extraction, which would improve gait-based identification of individuals in real-world scenarios. The experimentation is performed on publicly available dataset i.e. CASIA A, and CASIA C. The findings indicate that subjects wearing bags produced 90 % accuracy, and subjects wearing coats produced 58 % accuracy. Likewise, recognizing individuals with different walking speeds also exhibited excellent results, with an accuracy of 94 % for fast and 96 % for slow-paced walk patterns, which shows improvement compared to previous deep learning methods.© 2017 Elsevier Inc. All rights reserved.

Machine vision-based gait scan method for identifying cognitive impairment in older adults

Objective

Early identification of cognitive impairment in older adults could reduce the burden of age-related disabilities. Gait parameters are associated with and predictive of cognitive decline. Although a variety of sensors and machine learning analysis methods have been used in cognitive studies, a deep optimized machine vision-based method for analyzing gait to identify cognitive decline is needed.

Methods

This study used a walking footage dataset of 158 adults named West China Hospital Elderly Gait, which was labelled by performance on the Short Portable Mental Status Questionnaire. We proposed a novel recognition network, Deep Optimized GaitPart (DO-GaitPart), based on silhouette and skeleton gait images. Three improvements were applied: short-term temporal template generator (STTG) in the template generation stage to decrease computational cost and minimize loss of temporal information; depth-wise spatial feature extractor (DSFE) to extract both global and local fine-grained spatial features from gait images; and multi-scale temporal aggregation (MTA), a temporal modeling method based on attention mechanism, to improve the distinguishability of gait patterns.

Results

An ablation test showed that each component of DO-GaitPart was essential. DO-GaitPart excels in backpack walking scene on CASIA-B dataset, outperforming comparison methods, which were GaitSet, GaitPart, MT3D, 3D Local, TransGait, CSTL, GLN, GaitGL and SMPLGait on Gait3D dataset. The proposed machine vision gait feature identification method achieved a receiver operating characteristic/area under the curve (ROCAUC) of 0.876 (0.852-0.900) on the cognitive state classification task.

Conclusion

The proposed method performed well identifying cognitive decline from the gait video datasets, making it a prospective prototype tool in cognitive assessment.

Railway contactless checkout process with identification assisted by gait recognition

With business process optimization, technological advancement, equipment capability enhancement, and other means, the Railway Passenger Service Department in China is consistently working to improve the efficiency and convenience of passenger entry and exit procedures at railway stations. Concerning passengers' checkout, not only conventional identification approaches based on manual control, identification card, and magnetic thermal paper ticket are supported, but also a recent contactless identification process based on face recognition is applied in some stations. To further improve the contactless identification ability for checkout, an advanced contactless checkout process based on gait-augmented face recognition is innovatively put forward, in which a weakly-supervised body segmentation network named Dwsegnet and an improved GaitSet model are proposed. Through comparison with various models, the effectiveness of both Dwsegnet and the improved GaitSet is validated. Specifically, the contactless identification rate of gait-augmented face recognition is improved by 2.31% when compared to single-modal face recognition, which demonstrates the superiority of the proposed process.

Gait Impairment Analysis Using Silhouette Sinogram Signals and Assisted Knowledge Learning

The analysis of body motion is a valuable tool in the assessment and diagnosis of gait impairments, particularly those related to neurological disorders. In this study, we propose a novel automated system leveraging artificial intelligence for efficiently analyzing gait impairment from video-recorded images. The proposed methodology encompasses three key aspects. First, we generate a novel one-dimensional representation of each silhouette image, termed a silhouette sinogram, by computing the distance and angle between the centroid and each detected boundary points. This process enables us to effectively utilize relative variations in motion at different angles to detect gait patterns. Second, a one-dimensional convolutional neural network (1D CNN) model is developed and trained by incorporating the consecutive silhouette sinogram signals of silhouette frames to capture spatiotemporal information via assisted knowledge learning. This process allows the network to capture a broader context and temporal dependencies within the gait cycle, enabling a more accurate diagnosis of gait abnormalities. This study conducts training and an evaluation utilizing the publicly accessible INIT GAIT database. Finally, two evaluation schemes are employed: one leveraging individual silhouette frames and the other operating at the subject level, utilizing a majority voting technique. The outcomes of the proposed method showed superior enhancements in gait impairment recognition, with overall F1-scores of 100%, 90.62%, and 77.32% when evaluated based on sinogram signals, and 100%, 100%, and 83.33% when evaluated based on the subject level, for cases involving two, four, and six gait abnormalities, respectively. In conclusion, by comparing the observed locomotor function to a conventional gait pattern often seen in healthy individuals, the recommended approach allows for a quantitative and non-invasive evaluation of locomotion.

A comparative analysis of multi-biometrics performance in human and action recognition using silhouette thermal-face and skeletal data

Biometrics is a field that has been given importance in recent years and has been extensively studied. Biometrics can use physical and behavioural differences that are unique to individuals to recognize and identify them. Today, biometric information is used in many areas such as computer vision systems, entrance systems, security and recognition. In this study, a new biometrics database containing silhouette, thermal face and skeletal data based on the distance between the joints was created to be used in behavioural and physical biometrics studies. The fact that many cameras were used in previous studies increases both the processing intensity and the material cost. This study aimed to both increase the recognition performance and reduce material costs by adding thermal face data in addition to soft and behavioural biometrics with the optimum camera. The presented data set was created in accordance with both motion recognition and person identification. Various data loss scenarios and multi-biometrics approaches based on data fusion have been tried on the created data sets and the results have been given comparatively. In addition, the correlation coefficient of the motion frames method to obtain energy images from silhouette data was tested on this dataset and yielded high-accuracy results for both motion and person recognition.

Single-Belt Versus Split-Belt: Intelligent Treadmill Control via Microphase Gait Capture for Poststroke Rehabilitation

Stroke is the leading long-term disability and causes a significant financial burden associated with rehabilitation. In poststroke rehabilitation, individuals with hemiparesis have a specialized demand for coordinated movement between the paretic and the nonparetic legs. The split-belt treadmill can effectively facilitate the paretic leg by slowing down the belt speed for that leg while the patient is walking on a split-belt treadmill. Although studies have found that split-belt treadmills can produce better gait recovery outcomes than traditional single-belt treadmills, the high cost of split-belt treadmills is a significant barrier to stroke rehabilitation in clinics. In this article, we design an AI-based system for the single-belt treadmill to make it act like a split-belt by adjusting the belt speed instantaneously according to the patient's microgait phases. This system only requires a low-cost RGB camera to capture human gait patterns. A novel microgait classification pipeline model is used to detect gait phases in real time. The pipeline is based on self-supervised learning that can calibrate the anchor video with the real-time video. We then use a ResNet-LSTM module to handle temporal information and increase accuracy. A real-time filtering algorithm is used to smoothen the treadmill control. We have tested the developed system with 34 healthy individuals and four stroke patients. The results show that our system is able to detect the gait microphase accurately and requires less human annotation in training, compared to the ResNet50 classifier. Our system "Splicer" is boosted by AI modules and performs comparably as a split-belt system, in terms of timely varying left/right foot speed, creating a hemiparetic gait in healthy individuals, and promoting paretic side symmetry in force exertion for stroke patients. This innovative design can potentially provide cost-effective rehabilitation treatment for hemiparetic patients.

Learning Gait Representation From Massive Unlabelled Walking Videos: A Benchmark

Gait depicts individuals' unique and distinguishing walking patterns and has become one of the most promising biometric features for human identification. As a fine-grained recognition task, gait recognition is easily affected by many factors and usually requires a large amount of completely annotated data that is costly and insatiable. This paper proposes a large-scale self-supervised benchmark for gait recognition with contrastive learning, aiming to learn the general gait representation from massive unlabelled walking videos for practical applications via offering informative walking priors and diverse real-world variations. Specifically, we collect a large-scale unlabelled gait dataset GaitLU-1M consisting of 1.02M walking sequences and propose a conceptually simple yet empirically powerful baseline model GaitSSB. Experimentally, we evaluate the pre-trained model on four widely-used gait benchmarks, CASIA-B, OU-MVLP, GREW and Gait3D with or without transfer learning. The unsupervised results are comparable to or even better than the early model-based and GEI-based methods. After transfer learning, GaitSSB outperforms existing methods by a large margin in most cases, and also showcases the superior generalization capacity. Further experiments indicate that the pre-training can save about 50% and 80% annotation costs of GREW and Gait3D. Theoretically, we discuss the critical issues for gait-specific contrastive framework and present some insights for further study. As far as we know, GaitLU-1M is the first large-scale unlabelled gait dataset, and GaitSSB is the first method that achieves remarkable unsupervised results on the aforementioned benchmarks.

Cross-View Gait Recognition Method Based on Multi-Teacher Joint Knowledge Distillation

Aiming at challenges such as the high complexity of the network model, the large number of parameters, and the slow speed of training and testing in cross-view gait recognition, this paper proposes a solution: Multi-teacher Joint Knowledge Distillation (MJKD). The algorithm employs multiple complex teacher models to train gait images from a single view, extracting inter-class relationships that are then weighted and integrated into the set of inter-class relationships. These relationships guide the training of a lightweight student model, improving its gait feature extraction capability and recognition accuracy. To validate the effectiveness of the proposed Multi-teacher Joint Knowledge Distillation (MJKD), the paper performs experiments on the CASIA_B dataset using the ResNet network as the benchmark. The experimental results show that the student model trained by Multi-teacher Joint Knowledge Distillation (MJKD) achieves 98.24% recognition accuracy while significantly reducing the number of parameters and computational cost.

Gender Recognition Based on Gradual and Ensemble Learning from Multi-View Gait Energy Images and Poses

Image-based gender classification is very useful in many applications, such as intelligent surveillance, micromarketing, etc. One common approach is to adopt a machine learning algorithm to recognize the gender class of the captured subject based on spatio-temporal gait features extracted from the image. The image input can be generated from the video of the walking cycle, e.g., gait energy image (GEI). Recognition accuracy depends on the similarity of intra-class GEIs, as well as the dissimilarity of inter-class GEIs. However, we observe that, at some viewing angles, the GEIs of both gender classes are very similar. Moreover, the GEI does not exhibit a clear appearance of posture. We postulate that distinctive postures of the walking cycle can provide additional and valuable information for gender classification. This paper proposes a gender classification framework that exploits multiple inputs of the GEI and the characteristic poses of the walking cycle. The proposed framework is a cascade network that is capable of gradually learning the gait features from images acquired in multiple views. The cascade network contains a feature extractor and gender classifier. The multi-stream feature extractor network is trained to extract features from the multiple input images. Features are then fed to the classifier network, which is trained with ensemble learning. We evaluate and compare the performance of our proposed framework with state-of-the-art gait-based gender classification methods on benchmark datasets. The proposed framework outperforms other methods that only utilize a single input of the GEI or pose.

Gait Quality Aware Network: Toward the Interpretability of Silhouette-Based Gait Recognition

Gait recognition receives increasing attention since it can be conducted at a long distance in a nonintrusive way and applied to the condition of changing clothes. Most existing methods take the silhouettes of gait sequences as the input and learn a unified representation from multiple silhouettes to match probe and gallery. However, these models are all faced with the lack of interpretability, e.g., it is not clear which silhouette in a gait sequence and which part in the human body are relatively more important for recognition. In this work, we propose a gait quality aware network (GQAN) for gait recognition which explicitly assesses the quality of each silhouette and each part via two blocks: frame quality block (FQBlock) and part quality block (PQBlock). Specifically, FQBlock works in a squeeze-and-excitation style to recalibrate the features for each silhouette, and the scores of all the channels are added as frame quality indicator. PQBlock predicts a score for each part which is used to compute the weighted distance between the probe and gallery. Particularly, we propose a part quality loss (PQLoss) which enables GQAN to be trained in an end-to-end manner with only sequence-level identity annotations. This work is meaningful by moving toward the interpretability of silhouette-based gait recognition, and our method also achieves very competitive performance on CASIA-B and OUMVLP.

Elephants and algorithms: a review of the current and future role of AI in elephant monitoring

Artificial intelligence (AI) and machine learning (ML) present revolutionary opportunities to enhance our understanding of animal behaviour and conservation strategies. Using elephants, a crucial species in Africa and Asia's protected areas, as our focal point, we delve into the role of AI and ML in their conservation. Given the increasing amounts of data gathered from a variety of sensors like cameras, microphones, geophones, drones and satellites, the challenge lies in managing and interpreting this vast data. New AI and ML techniques offer solutions to streamline this process, helping us extract vital information that might otherwise be overlooked. This paper focuses on the different AI-driven monitoring methods and their potential for improving elephant conservation. Collaborative efforts between AI experts and ecological researchers are essential in leveraging these innovative technologies for enhanced wildlife conservation, setting a precedent for numerous other species.

GaitSG: Gait Recognition with SMPLs in Graph Structure

Gait recognition aims to identify a person based on his unique walking pattern. Compared with silhouettes and skeletons, skinned multi-person linear (SMPL) models can simultaneously provide human pose and shape information and are robust to viewpoint and clothing variances. However, previous approaches have only considered SMPL parameters as a whole and are yet to explore their potential for gait recognition thoroughly. To address this problem, we concentrate on SMPL representations and propose a novel SMPL-based method named GaitSG for gait recognition, which takes SMPL parameters in the graph structure as input. Specifically, we represent the SMPL model as graph nodes and employ graph convolution techniques to effectively model the human model topology and generate discriminative gait features. Further, we utilize prior knowledge of the human body and elaborately design a novel part graph pooling block, PGPB, to encode viewpoint information explicitly. The PGPB also alleviates the physical distance-unaware limitation of the graph structure. Comprehensive experiments on public gait recognition datasets, Gait3D and CASIA-B, demonstrate that GaitSG can achieve better performance and faster convergence than existing model-based approaches. Specifically, compared with the baseline SMPLGait (3D only), our model achieves approximately twice the Rank-1 accuracy and requires three times fewer training iterations on Gait3D.

Integration of force and IMU sensors for developing low-cost portable gait measurement system in lower extremities

Gait analysis is the method to accumulate walking data. It is useful in diagnosing diseases, follow-up of symptoms, and rehabilitation post-treatment. Several techniques have been developed to assess human gait. In the laboratory, gait parameters are analyzed by using a camera capture and a force plate. However, there are several limitations, such as high operating costs, the need for a laboratory and a specialist to operate the system, and long preparation time. This paper presents the development of a low-cost portable gait measurement system by using the integration of flexible force sensors and IMU sensors in outdoor applications for early detection of abnormal gait in daily living. The developed device is designed to measure ground reaction force, acceleration, angular velocity, and joint angles of the lower extremities. The commercialized device, including the motion capture system (Motive-OptiTrack) and force platform (MatScan), is used as the reference system to validate the performance of the developed system. The results of the system show that it has high accuracy in measuring gait parameters such as ground reaction force and joint angles in lower limbs. The developed device has a strong correlation coefficient compared with the commercialized system. The percent error of the motion sensor is below 8%, and the force sensor is lower than 3%. The low-cost portable device with a user interface was successfully developed to measure gait parameters for non-laboratory applications to support healthcare applications.

Gait-CNN-ViT: Multi-Model Gait Recognition with Convolutional Neural Networks and Vision Transformer

Gait recognition, the task of identifying an individual based on their unique walking style, can be difficult because walking styles can be influenced by external factors such as clothing, viewing angle, and carrying conditions. To address these challenges, this paper proposes a multi-model gait recognition system that integrates Convolutional Neural Networks (CNNs) and Vision Transformer. The first step in the process is to obtain a gait energy image, which is achieved by applying an averaging technique to a gait cycle. The gait energy image is then fed into three different models, DenseNet-201, VGG-16, and a Vision Transformer. These models are pre-trained and fine-tuned to encode the salient gait features that are specific to an individual's walking style. Each model provides prediction scores for the classes based on the encoded features, and these scores are then summed and averaged to produce the final class label. The performance of this multi-model gait recognition system was evaluated on three datasets, CASIA-B, OU-ISIR dataset D, and OU-ISIR Large Population dataset. The experimental results showed substantial improvement compared to existing methods on all three datasets. The integration of CNNs and ViT allows the system to learn both the pre-defined and distinct features, providing a robust solution for gait recognition even under the influence of covariates.

Pre-set estimation-based in-silico silhouette-based methodology for improving the robustness to viewing direction difference for assisting forensic gait analysis

Forensic gait analysis is used to visually and quantitatively analyze information regarding the appearance and style of walking that can be presented as evidence in the court. The demand for analyzing CCTV pedestrian footage in video surveillance has been increasing. The dependence of the accuracy of semiautomatic silhouette-based analysis, often used in forensic science, on the differences in the viewing directions is a very challenging issue that is yet to be resolved for real case applications. Currently, the different viewing directions used in comparison footage significantly decrease the accuracy of same person analysis when using the silhouette-based method, often used in the Japanese forensic science domain. A calibration-based method was previously prosed to resolve this problem, but it requires performing an elaborate measurement procedure at the camera installation site for an accurate analysis. In this study, we propose a novel in-silico silhouette-based analysis method that significantly expands the number of viewing direction pre-set settings to 900 from the 24 used in the previous method. Several software tools have been developed to ensure that all the procedures can be executed on a computer. The experimental results confirm that the accuracy of the proposed method is comparable to that of the calibration-based method. Furthermore, the practical comparison results from actual consultation confirmed the effectiveness of the proposed method under existing viewing direction differences. We therefore anticipate that the proposed method will be beneficial for improving the analysis accuracy in real cases and therefore serve as a substitute of the previous method.

CASIA-E: A Large Comprehensive Dataset for Gait Recognition

Gait recognition plays a special role in visual surveillance due to its unique advantage, e.g., long-distance, cross-view and non-cooperative recognition. However, it has not yet been widely applied. One reason for this awkwardness is the lack of a truly big dataset captured in practical outdoor scenarios. Here, the "big" at least means: (1) huge amount of gait videos; (2) sufficient subjects; (3) rich attributes; and (4) spatial and temporal variations. Moreover, most existing large-scale gait datasets are collected indoors, which have few challenges from real scenes, such as the dynamic and complex background clutters, illumination variations, vertical view variations, etc. In this article, we introduce a newly built big outdoor gait dataset, called CASIA-E. It contains more than one thousand people distributed over near one million videos. Each person involves 26 view angles and varied appearances caused by changes of bag carrying, dressing and walking styles. The videos are captured across five months and across three kinds of outdoor scenes. Soft biometric features are also recorded for all subjects including age, gender, height, weight, and nationality. Besides, we report an experimental benchmark and examine some meaningful problems that have not been well studied previously, e.g., the influence of million-level training videos, vertical view angles, walking styles, and the thermal infrared modality. We believe that such a big outdoor dataset and the experimental benchmark will promote the development of gait recognition in both academic research and industrial applications.

Design and Development of Polymer-Based Optical Fiber Sensor for GAIT Analysis

In the present scenario like COVID-19 pandemic, to maintain physical distance, the gait-based biometric is a must. Human gait identification is a very difficult process, but it is a suitable distance biometric that also gives good results at low resolution conditions even with face features that are not clear. This study describes the construction of a smart carpet that measures ground response force (GRF) and spatio-temporal gait parameters (STGP) using a polymer optical fiber sensor (POFS). The suggested carpet contains two light detection units for acquiring signals. Each unit obtains response from 10 nearby sensors. There are 20 intensity deviation sensors on a fiber. Light-emitting diodes (LED) are triggered successively, using the multiplexing approach that is being employed. Multiplexing is dependent on coupling among the LED and POFS sections. Results of walking experiments performed on the smart carpet suggested that certain parameters, including step length, stride length, cadence, and stance time, might be used to estimate the GRF and STGP. The results enable the detection of gait, including the swing phase, stance, stance length, and double supporting periods. The suggested carpet is dependable, reasonably priced equipment for gait acquisition in a variety of applications. Using the sensor data, gait recognition is performed using genetic algorithm (GA) and particle swarm optimization (PSO) technique. GA- and PSO-based gait template analyses are performed to extract the features with respect to the gait signals obtained from polymer optical gait sensors (POGS). The techniques used for classification of the obtained signals are random forest (RF) and support vector machine (SVM). The accuracy, sensitivity, and specificity results are obtained using SVM classifier and RF classifier. The results obtained using both classifiers are compared.

Deep Gait Recognition: A Survey

Gait recognition is an appealing biometric modality which aims to identify individuals based on the way they walk. Deep learning has reshaped the research landscape in this area since 2015 through the ability to automatically learn discriminative representations. Gait recognition methods based on deep learning now dominate the state-of-the-art in the field and have fostered real-world applications. In this paper, we present a comprehensive overview of breakthroughs and recent developments in gait recognition with deep learning, and cover broad topics including datasets, test protocols, state-of-the-art solutions, challenges, and future research directions. We first review the commonly used gait datasets along with the principles designed for evaluating them. We then propose a novel taxonomy made up of four separate dimensions namely body representation, temporal representation, feature representation, and neural architecture, to help characterize and organize the research landscape and literature in this area. Following our proposed taxonomy, a comprehensive survey of gait recognition methods using deep learning is presented with discussions on their performances, characteristics, advantages, and limitations. We conclude this survey with a discussion on current challenges and mention a number of promising directions for future research in gait recognition.

An autonomous decision-making framework for gait recognition systems against adversarial attack using reinforcement learning

Gait identification based on Deep Learning (DL) techniques has recently emerged as biometric technology for surveillance. We leveraged the vulnerabilities and decision-making abilities of the DL model in gait-based autonomous surveillance systems when attackers have no access to underlying model gradients/structures using a patch-based black-box adversarial attack with Reinforcement Learning (RL). These automated surveillance systems are secured, blocking the attacker's access. Therefore, the attack can be conducted in an RL framework where the agent's goal is determining the optimal image location, causing the model to perform incorrectly when perturbed with random pixels. Furthermore, the proposed adversarial attack presents encouraging results (maximum success rate = 77.59%). Researchers should explore system resilience scenarios (e.g., when attackers have no system access) before using these models in surveillance applications.

Deep Learning in Diverse Intelligent Sensor Based Systems

Deep learning has become a predominant method for solving data analysis problems in virtually all fields of science and engineering. The increasing complexity and the large volume of data collected by diverse sensor systems have spurred the development of deep learning methods and have fundamentally transformed the way the data are acquired, processed, analyzed, and interpreted. With the rapid development of deep learning technology and its ever-increasing range of successful applications across diverse sensor systems, there is an urgent need to provide a comprehensive investigation of deep learning in this domain from a holistic view. This survey paper aims to contribute to this by systematically investigating deep learning models/methods and their applications across diverse sensor systems. It also provides a comprehensive summary of deep learning implementation tips and links to tutorials, open-source codes, and pretrained models, which can serve as an excellent self-contained reference for deep learning practitioners and those seeking to innovate deep learning in this space. In addition, this paper provides insights into research topics in diverse sensor systems where deep learning has not yet been well-developed, and highlights challenges and future opportunities. This survey serves as a catalyst to accelerate the application and transformation of deep learning in diverse sensor systems.

A Gait-Based Real-Time Gender Classification System Using Whole Body Joints

Gait-based gender classification is a challenging task since people may walk in different directions with varying speed, gait style, and occluded joints. The majority of research studies in the literature focused on gender-specific joints, while there is less attention on the comparison of all of a body's joints. To consider all of the joints, it is essential to determine a person's gender based on their gait using a Kinect sensor. This paper proposes a logistic-regression-based machine learning model using whole body joints for gender classification. The proposed method consists of different phases including gait feature extraction based on three dimensional (3D) positions, feature selection, and classification of human gender. The Kinect sensor is used to extract 3D features of different joints. Different statistical tools such as Cronbach's alpha, correlation, t-test, and ANOVA techniques are exploited to select significant joints. The Coronbach's alpha technique yields an average result of 99.74%, which indicates the reliability of joints. Similarly, the correlation results indicate that there is significant difference between male and female joints during gait. t-test and ANOVA approaches demonstrate that all twenty joints are statistically significant for gender classification, because the p-value for each joint is zero and less than 1%. Finally, classification is performed based on the selected features using binary logistic regression model. A total of hundred (100) volunteers participated in the experiments in real scenario. The suggested method successfully classifies gender based on 3D features recorded in real-time using machine learning classifier with an accuracy of 98.0% using all body joints. The proposed method outperformed the existing systems which mostly rely on digital images.

Gait Recognition by Combining the Long-Short-Term Attention Network and Personal Physiological Features

Although gait recognition has been greatly improved by efforts from many researchers in recent years, its performance is still unsatisfactory due to the lack of gait information under the real scenariowhere only one or two images may be used for recognition. In this paper, a new gait recognition framework is brought about which can combine the long-short-term attention modules on silhouette images over the whole sequence and the real human physiological information calculated by a monocular image. The contributions of this work include the following: (1) Fusing the global long-term attention (GLTA) and local short-term attention (LSTA) over the whole query sequence to improve the gait recognition accuracy, where both the short-term gait feature (from two or three frames) and long-term feature (from the whole sequence) are extracted; (2) presenting a method to calculate the real personal static and dynamic physiological features through a single monocular image; (3) by efficiently applying the human physiological information, a new physiological feature extraction (PFE) network is proposed to concatenate the physiological information with silhouette for gait recognition. Through the experiments between the CASIA-B and Multi-state Gait datasets, the effectiveness and efficiency of the proposed method are proven. Under three different walking conditions of the CASIA-B dataset, the mean accuracy of rank-1 in our method is up to 89.6%, and in the Multi-state Gait dataset, wearing different clothes, the mean accuracy of rank-1 in our method is 2.4% higher than the other works.

ST-DeepGait: A Spatiotemporal Deep Learning Model for Human Gait Recognition

Human gait analysis presents an opportunity to study complex spatiotemporal data transpiring as co-movement patterns of multiple moving objects (i.e., human joints). Such patterns are acknowledged as movement signatures specific to an individual, offering the possibility to identify each individual based on unique gait patterns. We present a spatiotemporal deep learning model, dubbed ST-DeepGait, to featurize spatiotemporal co-movement patterns of human joints, and accordingly classify such patterns to enable human gait recognition. To this end, the ST-DeepGait model architecture is designed according to the spatiotemporal human skeletal graph in order to impose learning the salient local spatial dynamics of gait as they occur over time. Moreover, we employ a multi-layer RNN architecture to induce a sequential notion of gait cycles in the model. Our experimental results show that ST-DeepGait can achieve recognition accuracy rates over 90%. Furthermore, we qualitatively evaluate the model with the class embeddings to show interpretable separability of the features in geometric latent space. Finally, to evaluate the generalizability of our proposed model, we perform a zero-shot detection on 10 classes of data completely unseen during training and achieve a recognition accuracy rate of 88% overall. With this paper, we also contribute our gait dataset captured with an RGB-D sensor containing approximately 30 video samples of each subject for 100 subjects totaling 3087 samples. While we use human gait analysis as a motivating application to evaluate ST-DeepGait, we believe that this model can be simply adopted and adapted to study co-movement patterns of multiple moving objects in other applications such as in sports analytics and traffic pattern analysis.

Gait Recognition with Self-Supervised Learning of Gait Features Based on Vision Transformers

Gait is a unique biometric trait with several useful properties. It can be recognized remotely and without the cooperation of the individual, with low-resolution cameras, and it is difficult to obscure. Therefore, it is suitable for crime investigation, surveillance, and access control. Existing approaches for gait recognition generally belong to the supervised learning domain, where all samples in the dataset are annotated. In the real world, annotation is often expensive and time-consuming. Moreover, convolutional neural networks (CNNs) have dominated the field of gait recognition for many years and have been extensively researched, while other recent methods such as vision transformer (ViT) remain unexplored. In this manuscript, we propose a self-supervised learning (SSL) approach for pretraining the feature extractor using the DINO model to automatically learn useful gait features with the vision transformer architecture. The feature extractor is then used for extracting gait features on which the fully connected neural network classifier is trained using the supervised approach. Experiments on CASIA-B and OU-MVLP gait datasets show the effectiveness of the proposed approach.

Learning Gait Representations with Noisy Multi-Task Learning

Gait analysis is proven to be a reliable way to perform person identification without relying on subject cooperation. Walking is a biometric that does not significantly change in short periods of time and can be regarded as unique to each person. So far, the study of gait analysis focused mostly on identification and demographics estimation, without considering many of the pedestrian attributes that appearance-based methods rely on. In this work, alongside gait-based person identification, we explore pedestrian attribute identification solely from movement patterns. We propose DenseGait, the largest dataset for pretraining gait analysis systems containing 217 K anonymized tracklets, annotated automatically with 42 appearance attributes. DenseGait is constructed by automatically processing video streams and offers the full array of gait covariates present in the real world. We make the dataset available to the research community. Additionally, we propose GaitFormer, a transformer-based model that after pretraining in a multi-task fashion on DenseGait, achieves 92.5% accuracy on CASIA-B and 85.33% on FVG, without utilizing any manually annotated data. This corresponds to a +14.2% and +9.67% accuracy increase compared to similar methods. Moreover, GaitFormer is able to accurately identify gender information and a multitude of appearance attributes utilizing only movement patterns. The code to reproduce the experiments is made publicly.

Utilizing Spatio Temporal Gait Pattern and Quadratic SVM for Gait Recognition

This study aimed to develop a vision-based gait recognition system for person identification. Gait is the soft biometric trait recognizable from low-resolution surveillance videos, where the face and other hard biometrics are not even extractable. The gait is a cycle pattern of human body locomotion that consists of two sequential phases: swing and stance. The gait features of the complete gait cycle, referred to as gait signature, can be used for person identification. The proposed work utilizes gait dynamics for gait feature extraction. For this purpose, the spatio temporal power spectral gait features are utilized for gait dynamics captured through sub-pixel motion estimation, and they are less affected by the subject’s appearance. The spatio temporal power spectral gait features are utilized for a quadratic support vector machine classifier for gait recognition aiming for person identification. Spatio temporal power spectral preserves the spatiotemporal gait features and is adaptable for a quadratic support vector machine classifier-based gait recognition across different views and appearances. We have evaluated the gait features and support vector machine classifier-based gait recognition on a locally collected gait dataset that captures the effect of view variance in high scene depth videos. The proposed gait recognition technique achieves significant accuracy across all appearances and views.

Advances in Vision-Based Gait Recognition: From Handcrafted to Deep Learning

Identifying people’s identity by using behavioral biometrics has attracted many researchers’ attention in the biometrics industry. Gait is a behavioral trait, whereby an individual is identified based on their walking style. Over the years, gait recognition has been performed by using handcrafted approaches. However, due to several covariates’ effects, the competence of the approach has been compromised. Deep learning is an emerging algorithm in the biometrics field, which has the capability to tackle the covariates and produce highly accurate results. In this paper, a comprehensive overview of the existing deep learning-based gait recognition approach is presented. In addition, a summary of the performance of the approach on different gait datasets is provided.

VGG16-MLP: Gait Recognition with Fine-Tuned VGG-16 and Multilayer Perceptron

Gait is a pattern of a person’s walking. The body movements of a person while walking makes the gait unique. Regardless of the uniqueness, the gait recognition process suffers under various factors, namely the viewing angle, carrying condition, and clothing. In this paper, a pre-trained VGG-16 model is incorporated with a multilayer perceptron to enhance the performance under various covariates. At first, the gait energy image is obtained by averaging the silhouettes over a gait cycle. Transfer learning and fine-tuning techniques are then applied on the pre-trained VGG-16 model to learn the gait features of the attained gait energy image. Subsequently, a multilayer perceptron is utilized to determine the relationship among the gait features and the corresponding subject. Lastly, the classification layer identifies the corresponding subject. Experiments are conducted to evaluate the performance of the proposed method on the CASIA-B dataset, the OU-ISIR dataset D, and the OU-ISIR large population dataset. The comparison with the state-of-the-art methods shows that the proposed method outperforms the methods on all the datasets.

Multimodal Biometrics Recognition Using a Deep Convolutional Neural Network with Transfer Learning in Surveillance Videos

Biometric recognition is a critical task in security control systems. Although the face has long been widely accepted as a practical biometric for human recognition, it can be easily stolen and imitated. Moreover, in video surveillance, it is a challenge to obtain reliable facial information from an image taken at a long distance with a low-resolution camera. Gait, on the other hand, has been recently used for human recognition because gait is not easy to replicate, and reliable information can be obtained from a low-resolution camera at a long distance. However, the gait biometric alone still has constraints due to its intrinsic factors. In this paper, we propose a multimodal biometrics system by combining information from both the face and gait. Our proposed system uses a deep convolutional neural network with transfer learning. Our proposed network model learns discriminative spatiotemporal features from gait and facial features from face images. The two extracted features are fused into a common feature space at the feature level. This study conducted experiments on the publicly available CASIA-B gait and Extended Yale-B databases and a dataset of walking videos of 25 users. The proposed model achieves a 97.3 percent classification accuracy with an F1 score of 0.97and an equal error rate (EER) of 0.004.

Event-Stream Representation for Human Gaits Identification Using Deep Neural Networks

Dynamic vision sensors (event cameras) have recently been introduced to solve a number of different vision tasks such as object recognition, activities recognition, tracking, etc. Compared with the traditional RGB sensors, the event cameras have many unique advantages such as ultra low resources consumption, high temporal resolution and much larger dynamic range. However, these cameras only produce noisy and asynchronous events of intensity changes, i.e., event-streams rather than frames, where conventional computer vision algorithms can't be directly applied. In our opinion the key challenge for improving the performance of event cameras in vision tasks is finding the appropriate representations of the event-streams so that cutting-edge learning approaches can be applied to fully uncover the spatio-temporal information contained in the event-streams. In this paper, we focus on the event-based human gait identification task and investigate the possible representations of the event-streams when deep neural networks are applied as the classifier. We propose new event-based gait recognition approaches basing on two different representations of the event-stream, i.e., graph and image-like representations, and use graph-based convolutional network (GCN) and convolutional neural networks (CNN) respectively to recognize gait from the event-streams. The two approaches are termed as EV-Gait-3DGraph and EV-Gait-IMG. To evaluate the performance of the proposed approaches, we collect two event-based gait datasets, one from real-world experiments and the other by converting the publicly available RGB gait recognition benchmark CASIA-B. Extensive experiments show that EV-Gait-3DGraph achieves significantly higher recognition accuracy than other competing methods when sufficient training samples are available. However, EV-Gait-IMG converges more quickly than graph-based approaches while training and shows good accuracy with only few number of training samples (less than ten). So image-like presentation is preferable when the amount of training data is limited.

GaitSet: Cross-View Gait Recognition Through Utilizing Gait As a Deep Set

Gait is a unique biometric feature that can be recognized at a distance; thus, it has broad applications in crime prevention, forensic identification, and social security. To portray a gait, existing gait recognition methods utilize either a gait template which makes it difficult to preserve temporal information, or a gait sequence that maintains unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper, we present a novel perspective that utilizes gait as a deep set, which means that a set of gait frames are integrated by a global-local fused deep network inspired by the way our left- and right-hemisphere processes information to learn information that can be used in identification. Based on this deep set perspective, our method is immune to frame permutations, and can naturally integrate frames from different videos that have been acquired under different scenarios, such as diverse viewing angles, different clothes, or different item-carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 96.1 percent on the CASIA-B gait dataset and an accuracy of 87.9 percent on the OU-MVLP gait dataset. Under various complex scenarios, our model also exhibits a high level of robustness. It achieves accuracies of 90.8 and 70.3 percent on CASIA-B under bag-carrying and coat-wearing walking conditions respectively, significantly outperforming the best existing methods. Moreover, the proposed method maintains a satisfactory accuracy even when only small numbers of frames are available in the test samples; for example, it achieves 85.0 percent on CASIA-B even when using only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.

Deep supervised hashing for gait retrieval

Background: Gait recognition is perceived as the most promising biometric approach for future decades especially because of its efficient applicability in surveillance systems. Due to recent growth in the use of gait biometrics across surveillance systems, the ability to rapidly search for the required data has become an emerging need. Therefore, we addressed the gait retrieval problem, which retrieves people with gaits similar to a query subject from a large-scale dataset. Methods: This paper presents the deep gait retrieval hashing (DGRH) model to address the gait retrieval problem for large-scale datasets. Our proposed method is based on a supervised hashing method with a deep convolutional network. We use the ability of the convolutional neural network (CNN) to capture the semantic gait features for feature representation and learn the compact hash codes with the compatible hash function. Therefore, our DGRH model combines gait feature learning with binary hash codes. In addition, the learning loss is designed with a classification loss function that learns to preserve similarity and a quantization loss function that controls the quality of the hash codes Results: The proposed method was evaluated against the CASIA-B, OUISIR-LP, and OUISIR-MVLP benchmark datasets and received the promising result for gait retrieval tasks. Conclusions: The end-to-end deep supervised hashing model is able to learn discriminative gait features and is efficient in terms of the storage memory and speed for gait retrieval.

Robust and Lightweight System for Gait-Based Gender Classification toward Viewing Angle Variations

In computer vision applications, gait-based gender classification is a challenging task as a person may walk at various angles with respect to the camera viewpoint. In some of the viewing angles, the person’s limb movement can be occluded from the camera, preventing the perception of the gait-based features. To solve this problem, this study proposes a robust and lightweight system for gait-based gender classification. It uses a gait energy image (GEI) for representing the gait of an individual. A discrete cosine transform (DCT) is applied on GEI to generate a gait-based feature vector. Further, this DCT feature vector is applied to XGBoost classifier for performing gender classification. To improve the classification results, the XGBoost parameters are tuned. Finally, the results are compared with the other state-of-the-art approaches. The performance of the proposed system is evaluated on the OU-MVLP dataset. The experiment results show a mean CCR (correct classification rate) of 95.33% for the gender classification. The results obtained from various viewpoints of OU-MVLP illustrate the robustness of the proposed system for gait-based gender classification.

Robust clothing-independent gait recognition using hybrid part-based gait features

Recently, gait has been gathering extensive interest for the non-fungible position in applications. Although various methods have been proposed for gait recognition, most of them can only attain an excellent recognition performance when the probe and gallery gaits are in a similar condition. Once external factors (e.g., clothing variations) influence people's gaits and changes happen in human appearances, a significant performance degradation occurs. Hence, in our article, a robust hybrid part-based spatio-temporal feature learning method is proposed for gait recognition to handle this cloth-changing problem. First, human bodies are segmented into the affected and non/less unaffected parts based on the anatomical studies. Then, a well-designed network is proposed in our method to formulate our required hybrid features from the non/less unaffected body parts. This network contains three sub-networks, aiming to generate features independently. Each sub-network emphasizes individual aspects of gait, hence an effective hybrid gait feature can be created through their concatenation. In addition, temporal information can be used as complement to enhance the recognition performance, a sub-network is specifically proposed to establish the temporal relationship between consecutive short-range frames. Also, since local features are more discriminative than global features in gait recognition, in this network a sub-network is specifically proposed to generate features of local refined differences. The effectiveness of our proposed method has been evaluated by experiments on the CASIA Gait Dataset B and OU-ISIR Treadmill Gait Dataset B. Related experiments illustrate that compared with other gait recognition methods, our proposed method can achieve a prominent result when handling this cloth-changing gait recognition problem.

Long-Term Person Re-Identification Based on Appearance and Gait Feature Fusion under Covariate Changes

Person re-identification(Re-ID) technology has been a research hotspot in intelligent video surveillance, which accurately retrieves specific pedestrians from massive video data. Most research focuses on the short-term scenarios of person Re-ID to deal with general problems, such as occlusion, illumination change, and view variance. The appearance change or similar appearance problem in the long-term scenarios has has not been the focus of past research. This paper proposes a novel Re-ID framework consisting of a two-branch model to fuse the appearance and gait feature to overcome covariate changes. Firstly, we extract the appearance features from a video sequence by ResNet50 and leverage average pooling to aggregate the features. Secondly, we design an improved gait representation to obtain a person’s motion information and exclude the effects of external covariates. Specifically, we accumulate the difference between silhouettes to form an active energy image (AEI) and then mask the mid-body part in the image with the Improved-Sobel-Masking operator to extract the final gait representation called ISMAEI. Thirdly, we combine appearance features with gait features to generate discriminative and robust fused features. Finally, the Euclidean norm is adopted to calculate the distance between probe and gallery samples for person Re-ID. The proposed method is evaluated on the CASIA Gait Database B and TUM-GAID datasets. Compared with state-of-the-art methods, experimental results demonstrate that it can perform better in both Rank-1 and mAP.

CNN-based Multimodal Touchless Biometric Recognition System using Gait and Speech

Person identification using biometric features is an effective method for recognizing and authenticating the identity of a person. Multimodal biometric systems combine different biometric modalities in order to make better predictions as well as for achieving increased robustness. This paper proposes a touchless multimodal person identification model using deep learning techniques by combining the gait and speech modalities. Separate pipelines for both the modalities were developed using Convolutional Neural Networks. The paper also explores various fusion strategies for combining the two pipelines and shows how various metrics get affected with different fusion strategies. Results show that weighted average and product fusion rules work best for the data used in the experiments.

A Survey of Human Gait-Based Artificial Intelligence Applications

We performed an electronic database search of published works from 2012 to mid-2021 that focus on human gait studies and apply machine learning techniques. We identified six key applications of machine learning using gait data: 1) Gait analysis where analyzing techniques and certain biomechanical analysis factors are improved by utilizing artificial intelligence algorithms, 2) Health and Wellness, with applications in gait monitoring for abnormal gait detection, recognition of human activities, fall detection and sports performance, 3) Human Pose Tracking using one-person or multi-person tracking and localization systems such as OpenPose, Simultaneous Localization and Mapping (SLAM), etc., 4) Gait-based biometrics with applications in person identification, authentication, and re-identification as well as gender and age recognition 5) “Smart gait” applications ranging from smart socks, shoes, and other wearables to smart homes and smart retail stores that incorporate continuous monitoring and control systems and 6) Animation that reconstructs human motion utilizing gait data, simulation and machine learning techniques. Our goal is to provide a single broad-based survey of the applications of machine learning technology in gait analysis and identify future areas of potential study and growth. We discuss the machine learning techniques that have been used with a focus on the tasks they perform, the problems they attempt to solve, and the trade-offs they navigate.

On Learning Disentangled Representations for Gait Recognition

Gait, the walking pattern of individuals, is one of the important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and viewing angle. To remedy this issue, we propose a novel AutoEncoder framework, GaitNet, to explicitly disentangle appearance, canonical and pose features from RGB imagery. The LSTM integrates pose features over time as a dynamic gait feature while canonical features are averaged as a static gait feature. Both of them are utilized as classification features. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF, and FVG datasets, our method demonstrates superior performance to the SOTA quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency. We further compare our GaitNet with state-of-the-art face recognition to demonstrate the advantages of gait biometrics identification under certain scenarios, e.g., long-distance/lower resolutions, cross viewing angles. Source code is available at http://cvlab.cse.msu.edu/project-gaitnet.html.