1
|
Mogan JN, Lee CP, Lim KM, Ali M, Alqahtani A. Gait-CNN-ViT: Multi-Model Gait Recognition with Convolutional Neural Networks and Vision Transformer. SENSORS (BASEL, SWITZERLAND) 2023; 23:3809. [PMID: 37112147 PMCID: PMC10143319 DOI: 10.3390/s23083809] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Revised: 03/08/2023] [Accepted: 03/13/2023] [Indexed: 06/19/2023]
Abstract
Gait recognition, the task of identifying an individual based on their unique walking style, can be difficult because walking styles can be influenced by external factors such as clothing, viewing angle, and carrying conditions. To address these challenges, this paper proposes a multi-model gait recognition system that integrates Convolutional Neural Networks (CNNs) and Vision Transformer. The first step in the process is to obtain a gait energy image, which is achieved by applying an averaging technique to a gait cycle. The gait energy image is then fed into three different models, DenseNet-201, VGG-16, and a Vision Transformer. These models are pre-trained and fine-tuned to encode the salient gait features that are specific to an individual's walking style. Each model provides prediction scores for the classes based on the encoded features, and these scores are then summed and averaged to produce the final class label. The performance of this multi-model gait recognition system was evaluated on three datasets, CASIA-B, OU-ISIR dataset D, and OU-ISIR Large Population dataset. The experimental results showed substantial improvement compared to existing methods on all three datasets. The integration of CNNs and ViT allows the system to learn both the pre-defined and distinct features, providing a robust solution for gait recognition even under the influence of covariates.
Collapse
Affiliation(s)
- Jashila Nair Mogan
- Faculty of Information Science and Technology, Multimedia University, Melaka 75450, Malaysia
| | - Chin Poo Lee
- Faculty of Information Science and Technology, Multimedia University, Melaka 75450, Malaysia
| | - Kian Ming Lim
- Faculty of Information Science and Technology, Multimedia University, Melaka 75450, Malaysia
| | - Mohammed Ali
- Department of Computer Science, King Khalid University, Abha 61421, Saudi Arabia
| | - Ali Alqahtani
- Department of Computer Science, King Khalid University, Abha 61421, Saudi Arabia
- Center for Artificial Intelligence (CAI), King Khalid University, Abha 61421, Saudi Arabia
| |
Collapse
|
6
|
Liu Y, He X, Wang R, Teng Q, Hu R, Qing L, Wang Z, He X, Yin B, Mou Y, Du Y, Li X, Wang H, Liu X, Zhou L, Deng L, Xu Z, Xiao C, Ge M, Sun X, Jiang J, Chen J, Lin X, Xia L, Gong H, Yu H, Dong B. Application of Machine Vision in Classifying Gait Frailty Among Older Adults. Front Aging Neurosci 2021; 13:757823. [PMID: 34867286 PMCID: PMC8637841 DOI: 10.3389/fnagi.2021.757823] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Accepted: 10/18/2021] [Indexed: 02/05/2023] Open
Abstract
Background: Frail older adults have an increased risk of adverse health outcomes and premature death. They also exhibit altered gait characteristics in comparison with healthy individuals. Methods: In this study, we created a Fried's frailty phenotype (FFP) labelled casual walking video set of older adults based on the West China Health and Aging Trend study. A series of hyperparameters in machine vision models were evaluated for body key point extraction (AlphaPose), silhouette segmentation (Pose2Seg, DPose2Seg, and Mask R-CNN), gait feature extraction (Gaitset, LGaitset, and DGaitset), and feature classification (AlexNet and VGG16), and were highly optimised during analysis of gait sequences of the current dataset. Results: The area under the curve (AUC) of the receiver operating characteristic (ROC) at the physical frailty state identification task for AlexNet was 0.851 (0.827-0.8747) and 0.901 (0.878-0.920) in macro and micro, respectively, and was 0.855 (0.834-0.877) and 0.905 (0.886-0.925) for VGG16 in macro and micro, respectively. Furthermore, this study presents the machine vision method equipped with better predictive performance globally than age and grip strength, as well as than 4-m-walking-time in healthy and pre-frailty classifying. Conclusion: The gait analysis method in this article is unreported and provides promising original tool for frailty and pre-frailty screening with the characteristics of convenience, objectivity, rapidity, and non-contact. These methods can be extended to any gait-related disease identification processes, as well as in-home health monitoring.
Collapse
Affiliation(s)
- Yixin Liu
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Xiaohai He
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Renjie Wang
- Department of Rehabilitation Medicine, West China Hospital, Sichuan University, Chengdu, China
| | - Qizhi Teng
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Rui Hu
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Linbo Qing
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Zhengyong Wang
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Xuan He
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Biao Yin
- College of Electronics and Information Engineering, Sichuan University, Chengdu, China
| | - Yi Mou
- Geroscience and Chronic Disease Department, The 8th Municipal Hospital for the People, Chengdu, China
| | - Yanping Du
- Department of Rehabilitation Medicine, West China Hospital, Sichuan University, Chengdu, China
| | - Xinyi Li
- Medical Examination Center, Aviation Industry Corporation of China 363 Hospital, Chengdu, China
| | - Hui Wang
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Xiaolei Liu
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Lixing Zhou
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Linghui Deng
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Ziqi Xu
- West China School of Basic Medical Sciences and Forensic Medicine, Sichuan University, Chengdu, China
| | - Chun Xiao
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Meiling Ge
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Xuelian Sun
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| | - Junshan Jiang
- Medical College, Jiangsu University, Zhenjiang, China
| | - Jiaoyang Chen
- Public Health Department, Chengdu Medical College, Chengdu, China
| | - Xinyi Lin
- Public Health Department, Chengdu Medical College, Chengdu, China
| | - Ling Xia
- Public Health Department, Chengdu Medical College, Chengdu, China
| | - Haoran Gong
- West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China
| | - Haopeng Yu
- West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, China
- Med-X Center for Informatics, Sichuan University, Chengdu, China
| | - Birong Dong
- National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, China
- Geriatric Health Care and Medical Research Center, Sichuan University, Chengdu, China
- Department of Geriatrics, West China Hospital, Sichuan University, Chengdu, China
| |
Collapse
|