1
|
Fan C, Hou S, Wang J, Huang Y, Yu S. Learning Gait Representation From Massive Unlabelled Walking Videos: A Benchmark. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:14920-14937. [PMID: 37672380 DOI: 10.1109/tpami.2023.3312419] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/08/2023]
Abstract
Gait depicts individuals' unique and distinguishing walking patterns and has become one of the most promising biometric features for human identification. As a fine-grained recognition task, gait recognition is easily affected by many factors and usually requires a large amount of completely annotated data that is costly and insatiable. This paper proposes a large-scale self-supervised benchmark for gait recognition with contrastive learning, aiming to learn the general gait representation from massive unlabelled walking videos for practical applications via offering informative walking priors and diverse real-world variations. Specifically, we collect a large-scale unlabelled gait dataset GaitLU-1M consisting of 1.02M walking sequences and propose a conceptually simple yet empirically powerful baseline model GaitSSB. Experimentally, we evaluate the pre-trained model on four widely-used gait benchmarks, CASIA-B, OU-MVLP, GREW and Gait3D with or without transfer learning. The unsupervised results are comparable to or even better than the early model-based and GEI-based methods. After transfer learning, GaitSSB outperforms existing methods by a large margin in most cases, and also showcases the superior generalization capacity. Further experiments indicate that the pre-training can save about 50% and 80% annotation costs of GREW and Gait3D. Theoretically, we discuss the critical issues for gait-specific contrastive framework and present some insights for further study. As far as we know, GaitLU-1M is the first large-scale unlabelled gait dataset, and GaitSSB is the first method that achieves remarkable unsupervised results on the aforementioned benchmarks.
Collapse
|
2
|
Nirenberg M, Vernon W, Birch I. A review of the historical use and criticisms of gait analysis evidence. Sci Justice 2018; 58:292-298. [PMID: 29895463 DOI: 10.1016/j.scijus.2018.03.002] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/21/2017] [Revised: 02/19/2018] [Accepted: 03/04/2018] [Indexed: 10/17/2022]
Abstract
The use of gait analysis is a well-established facet of practice for many professions and a fundamental aspect of clinical practice. In recent times, gait analysis evidence has emerged as a new area of forensic practice. As its use has continued to spread and develop, the area of work has come under close scrutiny and subsequent criticism. The purpose of this paper is to examine the historical use of gait analysis evidence and consider the criticisms of this work. Through the use of the historical records of cases within the public domain it has been determined that gait analysis as evidence was first presented in court over 175 years ago, although it has only been utilized by experts in more recent times. The quality of analysis underpinning such evidence has been variable, and has been undertaken by both non-expert and expert witnesses. The work undertaken by expert witnesses appears to have been both non-scientific and scientific in nature, though there is limited reporting of cases involving scientific approaches. Given the variation in the quality of the methodologies utilized, there is the potential for confusion within the courts, where it may be difficult for the judge or jury to determine the appropriate weight that can be attributed to the evidence. It is concluded that future publications should explore the scientific basis of forensic gait analysis to evaluate standards, reliability and validity, as well as reporting the methodologies utilized in relevant cases in the field. It is also recommended that courts consider in greater depth an expert's theoretical approach and experience prior to admitting their evidence. The publication of 'Forensic gait analysis: a primer for courts', although limited in some aspects of its consideration of practice, is a welcome addition to the information available for guidance.
Collapse
Affiliation(s)
- Michael Nirenberg
- Friendly Foot Care, PC, 50 W. 94th Place, Crown Point, IN 46307, United States
| | - Wesley Vernon
- University of Huddersfield, Division of Podiatry and Clinical Sciences, School of Human and Health Sciences, Ramsden Building, Queensgate, Huddersfield, England HD1 3DH, United Kingdom
| | - Ivan Birch
- Sheffield Teaching Hospitals NHS Foundation Trust, Central Health Clinic, 1 Mulberry Street, Sheffield, S1 2PJ, United Kingdom.
| |
Collapse
|
3
|
Abstract
The body of research that examines the perception of biological motion is extensive and explores the factors that are perceived from biological motion and how this information is processed. This research demonstrates that individuals are able to use relative (temporal and spatial) information from a person's movement to recognize factors, including gender, age, deception, emotion, intention, and action. The research also demonstrates that movement presents idiosyncratic properties that allow individual discrimination, thus providing the basis for significant exploration in the domain of biometrics and social signal processing. Medical forensics, safety garments, and victim selection domains also have provided a history of research on the perception of biological motion applications; however, a number of additional domains present opportunities for application that have not been explored in depth. Therefore, the purpose of this paper is to present an overview of the current applications of biological motion-based research and to propose a number of areas where biological motion research, specific to recognition, could be applied in the future.
Collapse
|
4
|
Sandau M, Heimbürger RV, Jensen KE, Moeslund TB, Aanaes H, Alkjaer T, Simonsen EB. Reliable Gait Recognition Using 3D Reconstructions and Random Forests - An Anthropometric Approach. J Forensic Sci 2016; 61:637-48. [PMID: 27122399 DOI: 10.1111/1556-4029.13015] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2014] [Revised: 03/10/2015] [Accepted: 06/07/2015] [Indexed: 11/30/2022]
Abstract
Photogrammetric measurements of bodily dimensions and analysis of gait patterns in CCTV are important tools in forensic investigations but accurate extraction of the measurements are challenging. This study tested whether manual annotation of the joint centers on 3D reconstructions could provide reliable recognition. Sixteen participants performed normal walking where 3D reconstructions were obtained continually. Segment lengths and kinematics from the extremities were manually extracted by eight expert observers. The results showed that all the participants were recognized, assuming the same expert annotated the data. Recognition based on data annotated by different experts was less reliable achieving 72.6% correct recognitions as some parameters were heavily affected by interobserver variability. This study verified that 3D reconstructions are feasible for forensic gait analysis as an improved alternative to conventional CCTV. However, further studies are needed to account for the use of different clothing, field conditions, etc.
Collapse
Affiliation(s)
- Martin Sandau
- Department of Neuroscience and Pharmacology, University of Copenhagen, Blegdamsvej 3, DK-2200, Copenhagen, Denmark.,The Danish Institute of Fire and Security Technology, Jernholmen 12, DK-2650, Hvidovre, Denmark
| | - Rikke V Heimbürger
- Department of Forensic Medicine, University of Copenhagen, Frederik V's Vej 11, DK-2100, Copenhagen, Denmark
| | - Karl E Jensen
- Department of Radiology, Rigshospitalet, Blegdamsvej 9, DK-2100, Copenhagen, Denmark
| | - Thomas B Moeslund
- Department of Architecture, Design and Media Technology, Aalborg University, Rendsburggade 14, DK-9000, Aalborg, Denmark
| | - Henrik Aanaes
- Department of Informatics and Mathematics, Technical University of Denmark, Anker Engelunds Vej 1, DK-2800, Kongens Lyngby, Denmark
| | - Tine Alkjaer
- Department of Neuroscience and Pharmacology, University of Copenhagen, Blegdamsvej 3, DK-2200, Copenhagen, Denmark
| | - Erik B Simonsen
- Department of Neuroscience and Pharmacology, University of Copenhagen, Blegdamsvej 3, DK-2200, Copenhagen, Denmark
| |
Collapse
|
5
|
|
6
|
Díaz-Pereira MP, Gómez-Conde I, Escalona M, Olivieri DN. Automatic recognition and scoring of olympic rhythmic gymnastic movements. Hum Mov Sci 2014; 34:63-80. [PMID: 24502991 DOI: 10.1016/j.humov.2014.01.001] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2013] [Revised: 01/08/2014] [Accepted: 01/08/2014] [Indexed: 11/16/2022]
Abstract
We describe a conceptually simple algorithm for assigning judgement scores to rhythmic gymnastic movements, which could improve scoring objectivity and reduce judgemental bias during competitions. Our method, implemented as a real-time computer vision software, takes a video shot or a live performance video stream as input and extracts detailed velocity field information from body movements, transforming them into specialized spatio-temporal image templates. The collection of such images over time, when projected into a velocity covariance eigenspace, trace out unique but similar trajectories for a particular gymnastic movement type. By comparing separate executions of the same atomic gymnastic routine, our method assigns a quality judgement score that is related to the distance between the respective spatio-temporal trajectories. For several standard gymnastic movements, the method accurately assigns scores that are comparable to those assigned by expert judges. We also describe our rhythmic gymnastic video shot database, which we have made freely available to the human movement research community. The database can be obtained at http://www.milegroup.net/apps/gymdb/.
Collapse
Affiliation(s)
- M Pino Díaz-Pereira
- Department of Special Teaching, Area of Physical Education and Sports, University of Vigo, Ourense 32004, Spain.
| | - Iván Gómez-Conde
- Department of Computer Science, University of Vigo, Ourense 32004, Spain
| | - Merly Escalona
- Department of Computer Science, University of Vigo, Ourense 32004, Spain
| | - David N Olivieri
- Department of Computer Science, University of Vigo, Ourense 32004, Spain.
| |
Collapse
|
7
|
Quantification and recognition of parkinsonian gait from monocular video imaging using kernel-based principal component analysis. Biomed Eng Online 2011; 10:99. [PMID: 22074315 PMCID: PMC3354347 DOI: 10.1186/1475-925x-10-99] [Citation(s) in RCA: 24] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2011] [Accepted: 11/10/2011] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND The computer-aided identification of specific gait patterns is an important issue in the assessment of Parkinson's disease (PD). In this study, a computer vision-based gait analysis approach is developed to assist the clinical assessments of PD with kernel-based principal component analysis (KPCA). METHOD Twelve PD patients and twelve healthy adults with no neurological history or motor disorders within the past six months were recruited and separated according to their "Non-PD", "Drug-On", and "Drug-Off" states. The participants were asked to wear light-colored clothing and perform three walking trials through a corridor decorated with a navy curtain at their natural pace. The participants' gait performance during the steady-state walking period was captured by a digital camera for gait analysis. The collected walking image frames were then transformed into binary silhouettes for noise reduction and compression. Using the developed KPCA-based method, the features within the binary silhouettes can be extracted to quantitatively determine the gait cycle time, stride length, walking velocity, and cadence. RESULTS AND DISCUSSION The KPCA-based method uses a feature-extraction approach, which was verified to be more effective than traditional image area and principal component analysis (PCA) approaches in classifying "Non-PD" controls and "Drug-Off/On" PD patients. Encouragingly, this method has a high accuracy rate, 80.51%, for recognizing different gaits. Quantitative gait parameters are obtained, and the power spectrums of the patients' gaits are analyzed. We show that that the slow and irregular actions of PD patients during walking tend to transfer some of the power from the main lobe frequency to a lower frequency band. Our results indicate the feasibility of using gait performance to evaluate the motor function of patients with PD. CONCLUSION This KPCA-based method requires only a digital camera and a decorated corridor setup. The ease of use and installation of the current method provides clinicians and researchers a low cost solution to monitor the progression of and the treatment to PD. In summary, the proposed method provides an alternative to perform gait analysis for patients with PD.
Collapse
|
8
|
|
9
|
|
10
|
Goffredo M, Bouchrika I, Carter JN, Nixon MS. Self-Calibrating View-Invariant Gait Biometrics. ACTA ACUST UNITED AC 2010; 40:997-1008. [DOI: 10.1109/tsmcb.2009.2031091] [Citation(s) in RCA: 157] [Impact Index Per Article: 11.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
11
|
|
12
|
|
13
|
Tamil EM, Noor MH, Razak Z, Noor NM, Tamil AM. A Review on Feature Extraction & Classification Techniques for Biosignal Processing (Part V: Gait Signal). IFMBE PROCEEDINGS 2008:122-124. [DOI: 10.1007/978-3-540-69139-6_34] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/01/2023]
|
14
|
Zhou X, Bhanu B. Integrating face and gait for human recognition at a distance in video. ACTA ACUST UNITED AC 2007; 37:1119-37. [PMID: 17926696 DOI: 10.1109/tsmcb.2006.889612] [Citation(s) in RCA: 82] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
This paper introduces a new video-based recognition method to recognize noncooperating individuals at a distance in video who expose side views to the camera. Information from two biometrics sources, side face and gait, is utilized and integrated for recognition. For side face, an enhanced side-face image (ESFI), a higher resolution image compared with the image directly obtained from a single video frame, is constructed, which integrates face information from multiple video frames. For gait, the gait energy image (GEI), a spatio-temporal compact representation of gait in video, is used to characterize human-walking properties. The features of face and gait are obtained separately using the principal component analysis and multiple discriminant analysis combined method from ESFI and GEI, respectively. They are then integrated at the match score level by using different fusion strategies. The approach is tested on a database of video sequences, corresponding to 45 people, which are collected over seven months. The different fusion methods are compared and analyzed. The experimental results show that: 1) the idea of constructing ESFI from multiple frames is promising for human recognition in video, and better face features are extracted from ESFI compared to those from the original side-face images (OSFIs); 2) the synchronization of face and gait is not necessary for face template ESFI and gait template GEI; the synthetic match scores combine information from them; and 3) an integrated information from side face and gait is effective for human recognition in video.
Collapse
Affiliation(s)
- Xiaoli Zhou
- Center for Research in Intelligent Systems, University of California, Riverside, CA 92521, USA
| | | |
Collapse
|
15
|
Han J, Bhanu B. Individual recognition using gait energy image. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2006; 28:316-22. [PMID: 16468626 DOI: 10.1109/tpami.2006.38] [Citation(s) in RCA: 192] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
In this paper, we propose a new spatio-temporal gait representation, called Gait Energy Image (GEI), to characterize human walking properties for individual recognition by gait. To address the problem of the lack of training templates, we also propose a novel approach for human recognition by combining statistical gait features from real and synthetic templates. We directly compute the real templates from training silhouette sequences, while we generate the synthetic templates from training sequences by simulating silhouette distortion. We use a statistical approach for learning effective features from real and synthetic templates. We compare the proposed GEI-based gait recognition approach with other gait recognition approaches on USF HumanID Database. Experimental results show that the proposed GEI is an effective and efficient gait representation for individual recognition, and the proposed approach achieves highly competitive performance with respect to the published gait recognition approaches.
Collapse
Affiliation(s)
- Ju Han
- Center for Research in Intelligent Systems, University of California, Riverside, 900 University Avenue, Riverside, CA 92521, USA.
| | | |
Collapse
|
16
|
|
17
|
|
18
|
Kale A, Sundaresan A, Rajagopalan AN, Cuntoor NP, Roy-Chowdhury AK, Krüger V, Chellappa R. Identification of humans using gait. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2004; 13:1163-1173. [PMID: 15449579 DOI: 10.1109/tip.2004.832865] [Citation(s) in RCA: 75] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Abstract
We propose a view-based approach to recognize humans from their gait. Two different image features have been considered: the width of the outer contour of the binarized silhouette of the walking person and the entire binary silhouette itself. To obtain the observation vector from the image features, we employ two different methods. In the first method, referred to as the indirect approach, the high-dimensional image feature is transformed to a lower dimensional space by generating what we call the frame to exemplar (FED) distance. The FED vector captures both structural and dynamic traits of each individual. For compact and effective gait representation and recognition, the gait information in the FED vector sequences is captured in a hidden Markov model (HMM). In the second method, referred to as the direct approach, we work with the feature vector directly (as opposed to computing the FED) and train an HMM. We estimate the HMM parameters (specifically the observation probability B) based on the distance between the exemplars and the image features. In this way, we avoid learning high-dimensional probability density functions. The statistical nature of the HMM lends overall robustness to representation and recognition. The performance of the methods is illustrated using several databases.
Collapse
Affiliation(s)
- Amit Kale
- Department of Computer Science, University of Kentucky, Lexington, KY 40506, USA
| | | | | | | | | | | | | |
Collapse
|
19
|
|
20
|
Mowbray SD, Nixon MS. Automatic Gait Recognition via Fourier Descriptors of Deformable Objects. LECTURE NOTES IN COMPUTER SCIENCE 2003. [DOI: 10.1007/3-540-44887-x_67] [Citation(s) in RCA: 53] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
21
|
|
22
|
|
23
|
Huang PS. Automatic gait recognition via statistical approaches for extended template features. IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS. PART B, CYBERNETICS : A PUBLICATION OF THE IEEE SYSTEMS, MAN, AND CYBERNETICS SOCIETY 2001; 31:818-824. [PMID: 18244847 DOI: 10.1109/3477.956044] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
A gait recognition system using extended template features is presented. A proposed statistical approach is applied for feature extraction from spatial and temporal templates. This method can be used to reduce data dimensionality and to optimize the class separability of different gait sequences simultaneously. Dimensionality reduction is achieved by template extraction followed by principal component analysis. Gait recognition is achieved in the canonical space using a measure of accumulated distance as the metric. By incorporating spatial and temporal information into an extended feature, gait recognition becomes more robust and accurate than using spatial or temporal features alone.
Collapse
Affiliation(s)
- P S Huang
- Dept. of Electr. Eng., Chung Cheng Inst. of Technol., Taoyuan
| |
Collapse
|