1
|
Lim J, Luo C, Lee S, Song YE, Jung H. Action Recognition of Taekwondo Unit Actions Using Action Images Constructed with Time-Warped Motion Profiles. SENSORS (BASEL, SWITZERLAND) 2024; 24:2595. [PMID: 38676211 PMCID: PMC11055144 DOI: 10.3390/s24082595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 04/05/2024] [Accepted: 04/15/2024] [Indexed: 04/28/2024]
Abstract
Taekwondo has evolved from a traditional martial art into an official Olympic sport. This study introduces a novel action recognition model tailored for Taekwondo unit actions, utilizing joint-motion data acquired via wearable inertial measurement unit (IMU) sensors. The utilization of IMU sensor-measured motion data facilitates the capture of the intricate and rapid movements characteristic of Taekwondo techniques. The model, underpinned by a conventional convolutional neural network (CNN)-based image classification framework, synthesizes action images to represent individual Taekwondo unit actions. These action images are generated by mapping joint-motion profiles onto the RGB color space, thus encapsulating the motion dynamics of a single unit action within a solitary image. To further refine the representation of rapid movements within these images, a time-warping technique was applied, adjusting motion profiles in relation to the velocity of the action. The effectiveness of the proposed model was assessed using a dataset compiled from 40 Taekwondo experts, yielding remarkable outcomes: an accuracy of 0.998, a precision of 0.983, a recall of 0.982, and an F1 score of 0.982. These results underscore this time-warping technique's contribution to enhancing feature representation, as well as the proposed method's scalability and effectiveness in recognizing Taekwondo unit actions.
Collapse
Affiliation(s)
- Junghwan Lim
- Department of Motion, Torooc Co., Ltd., Seoul 04585, Republic of Korea;
| | - Chenglong Luo
- Department of Mechanical Engineering, Konkuk University, Seoul 05029, Republic of Korea;
| | - Seunghun Lee
- School of Mechanical and Aerospace Engineering, Seoul National University, Seoul 08826, Republic of Korea;
| | - Young Eun Song
- Department of Autonomous Mobility, Korea University, Sejong 30019, Republic of Korea
| | - Hoeryong Jung
- Department of Mechanical Engineering, Konkuk University, Seoul 05029, Republic of Korea;
| |
Collapse
|
2
|
Kwon MP, Hullfish TJ, Humbyrd CJ, Boakye LAT, Baxter JR. Wearable sensor and machine learning estimate tendon load and walking speed during immobilizing boot ambulation. Sci Rep 2023; 13:18086. [PMID: 37872320 PMCID: PMC10593749 DOI: 10.1038/s41598-023-45375-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Accepted: 10/18/2023] [Indexed: 10/25/2023] Open
Abstract
The purpose of this study is to develop a wearable paradigm to accurately monitor Achilles tendon loading and walking speed using wearable sensors that reduce subject burden. Ten healthy adults walked in an immobilizing boot under various heel wedge conditions (30°, 5°, 0°) and walking speeds. Three-dimensional motion capture, ground reaction force, and 6-axis inertial measurement unit (IMU) signals were collected. We used a Least Absolute Shrinkage and Selection Operator (LASSO) regression to predict peak Achilles tendon load and walking speed. The effects of altering sensor parameters were also explored. Walking speed models (mean absolute percentage error (MAPE): 8.81 ± 4.29%) outperformed tendon load models (MAPE: 34.93 ± 26.3%). Models trained with subject-specific data performed better than models trained without subject-specific data. Removing the gyroscope, decreasing the sampling frequency, and using combinations of sensors did not change the usability of the models, having inconsequential effects on model performance. We developed a simple monitoring paradigm that uses LASSO regression and wearable sensors to accurately predict (MAPE ≤ 12.6%) Achilles tendon loading and walking speed while ambulating in an immobilizing boot. This paradigm provides a clinically implementable strategy to longitudinally monitor patient loading and activity while recovering from Achilles tendon injuries.
Collapse
Affiliation(s)
- Michelle P Kwon
- Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA, USA
| | - Todd J Hullfish
- Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA, USA
| | - Casey Jo Humbyrd
- Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA, USA
| | - Lorraine A T Boakye
- Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA, USA
| | - Josh R Baxter
- Department of Orthopaedic Surgery, University of Pennsylvania, Philadelphia, PA, USA.
| |
Collapse
|
3
|
Kwon MP, Hullfish TJ, Humbyrd CJ, Boakye LA, Baxter JR. Wearable sensor and machine learning accurately estimate tendon load and walking speed during immobilizing boot ambulation. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.06.03.23290612. [PMID: 37333069 PMCID: PMC10274996 DOI: 10.1101/2023.06.03.23290612] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/20/2023]
Abstract
Achilles tendon injuries are treated with progressive weight bearing to promote tendon healing and restore function. Patient rehabilitation progression are typically studied in controlled, lab settings and do not represent the long-term loading experienced during daily living. The purpose of this study is to develop a wearable paradigm to accurately monitor Achilles tendon loading and walking speed using low-cost sensors that reduce subject burden. Ten healthy adults walked in an immobilizing boot under various heel wedge conditions (30°, 5°, 0°) and walking speeds. Three-dimensional motion capture, ground reaction force, and 6-axis inertial measurement unit (IMU) signals were collected per trial. We used Least Absolute Shrinkage and Selection Operator (LASSO) regression to predict peak Achilles tendon load and walking speed. The effects of using only accelerometer data, different sampling frequency, and multiple sensors to train the model were also explored. Walking speed models outperformed (mean absolute percentage error (MAPE): 8.41 ± 4.08%) tendon load models (MAPE: 33.93 ± 23.9%). Models trained with subject-specific data performed significantly better than generalized models. For example, our personalized model that was trained with only subject-specific data predicted tendon load with a 11.5 ± 4.41% MAPE and walking speed with a 4.50 ± 0.91% MAPE. Removing gyroscope channels, decreasing sampling frequency, and using combinations of sensors had inconsequential effects on models performance (changes in MAPE < 6.09%). We developed a simple monitoring paradigm that uses LASSO regression and wearable sensors to accurately predict Achilles tendon loading and walking speed while ambulating in an immobilizing boot. This paradigm provides a clinically implementable strategy to longitudinally monitor patient loading and activity while recovering from Achilles tendon injuries.
Collapse
|
4
|
Kwon H, Clifford GD, Genias I, Bernhard D, Esper CD, Factor SA, McKay JL. An Explainable Spatial-Temporal Graphical Convolutional Network to Score Freezing of Gait in Parkinsonian Patients. SENSORS (BASEL, SWITZERLAND) 2023; 23:1766. [PMID: 36850363 PMCID: PMC9968199 DOI: 10.3390/s23041766] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Revised: 01/27/2023] [Accepted: 02/01/2023] [Indexed: 06/18/2023]
Abstract
Freezing of gait (FOG) is a poorly understood heterogeneous gait disorder seen in patients with parkinsonism which contributes to significant morbidity and social isolation. FOG is currently measured with scales that are typically performed by movement disorders specialists (i.e., MDS-UPDRS), or through patient completed questionnaires (N-FOG-Q) both of which are inadequate in addressing the heterogeneous nature of the disorder and are unsuitable for use in clinical trials The purpose of this study was to devise a method to measure FOG objectively, hence improving our ability to identify it and accurately evaluate new therapies. A major innovation of our study is that it is the first study of its kind that uses the largest sample size (>30 h, N = 57) in order to apply explainable, multi-task deep learning models for quantifying FOG over the course of the medication cycle and at varying levels of parkinsonism severity. We trained interpretable deep learning models with multi-task learning to simultaneously score FOG (cross-validated F1 score 97.6%), identify medication state (OFF vs. ON levodopa; cross-validated F1 score 96.8%), and measure total PD severity (MDS-UPDRS-III score prediction error ≤ 2.7 points) using kinematic data of a well-characterized sample of N = 57 patients during levodopa challenge tests. The proposed model was able to explain how kinematic movements are associated with each FOG severity level that were highly consistent with the features, in which movement disorders specialists are trained to identify as characteristics of freezing. Overall, we demonstrate that deep learning models' capability to capture complex movement patterns in kinematic data can automatically and objectively score FOG with high accuracy. These models have the potential to discover novel kinematic biomarkers for FOG that can be used for hypothesis generation and potentially as clinical trial outcome measures.
Collapse
Affiliation(s)
- Hyeokhyen Kwon
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA 30322, USA
| | - Gari D. Clifford
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA 30322, USA
| | - Imari Genias
- Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA 30322, USA
| | - Doug Bernhard
- Jean and Paul Amos Parkinson’s Disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA 30322, USA
| | - Christine D. Esper
- Jean and Paul Amos Parkinson’s Disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA 30322, USA
| | - Stewart A. Factor
- Jean and Paul Amos Parkinson’s Disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA 30322, USA
| | - J. Lucas McKay
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA 30322, USA
- Jean and Paul Amos Parkinson’s Disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA 30322, USA
| |
Collapse
|
5
|
Kwon H, Clifford GD, Genias I, Bernhard D, Esper CD, Factor SA, McKay JL. An explainable spatial-temporal graphical convolutional network to score freezing of gait in parkinsonian patients. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.01.13.23284535. [PMID: 36711809 PMCID: PMC9882551 DOI: 10.1101/2023.01.13.23284535] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
Abstract
Freezing of gait (FOG) is a poorly understood heterogeneous gait disorder seen in patients with parkinsonism which contributes to significant morbidity and social isolation. FOG is currently measured with scales that are typically performed by movement disorders specialists (ie. MDS-UPDRS), or through patient completed questionnaires (N-FOG-Q) both of which are inadequate in addressing the heterogeneous nature of the disorder and are unsuitable for use in clinical trials The purpose of this study was to devise a method to measure FOG objectively, hence improving our ability to identify it and accurately evaluate new therapies. We trained interpretable deep learning models with multi-task learning to simultaneously score FOG (cross-validated F1 score 97.6%), identify medication state (OFF vs. ON levodopa; cross-validated F1 score 96.8%), and measure total PD severity (MDS-UPDRS-III score prediction error ≤ 2.7 points) using kinematic data of a well-characterized sample of N=57 patients during levodopa challenge tests. The proposed model was able to identify kinematic features associated with each FOG severity level that were highly consistent with the features that movement disorders specialists are trained to identify as characteristic of freezing. In this work, we demonstrate that deep learning models' capability to capture complex movement patterns in kinematic data can automatically and objectively score FOG with high accuracy. These models have the potential to discover novel kinematic biomarkers for FOG that can be used for hypothesis generation and potentially as clinical trial outcome measures.
Collapse
Affiliation(s)
- Hyeokhyen Kwon
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, USA
| | - Gari D. Clifford
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, USA
| | - Imari Genias
- Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, USA
| | - Doug Bernhard
- Jean and Paul Amos Parkinson’s disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA, USA
| | - Christine D. Esper
- Jean and Paul Amos Parkinson’s disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA, USA
| | - Stewart A. Factor
- Jean and Paul Amos Parkinson’s disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA, USA
| | - J. Lucas McKay
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, USA
- Jean and Paul Amos Parkinson’s disease and Movement Disorders Program, Department of Neurology, School of Medicine, Emory University, Atlanta, GA, USA
| |
Collapse
|
6
|
Thapa K, Seo Y, Yang SH, Kim K. Semi-Supervised Adversarial Auto-Encoder to Expedite Human Activity Recognition. SENSORS (BASEL, SWITZERLAND) 2023; 23:683. [PMID: 36679478 PMCID: PMC9863227 DOI: 10.3390/s23020683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Revised: 12/26/2022] [Accepted: 12/29/2022] [Indexed: 06/17/2023]
Abstract
The study of human activity recognition concentrates on classifying human activities and the inference of human behavior using modern sensing technology. However, the issue of domain adaptation for inertial sensing-based human activity recognition (HAR) is still burdensome. The existing requirement of labeled training data for adapting such classifiers to every new person, device, or on-body location is a significant barrier to the widespread adoption of HAR-based applications, making this a challenge of high practical importance. We propose the semi-supervised HAR method to improve reconstruction and generation. It executes proper adaptation with unlabeled data without changes to a pre-trained HAR classifier. Our approach decouples VAE with adversarial learning to ensure robust classifier operation, without newly labeled training data, under changes to the individual activity and the on-body sensor position. Our proposed framework shows the empirical results using the publicly available benchmark dataset compared to state-of-art baselines, achieving competitive improvement for handling new and unlabeled activity. The result demonstrates SAA has achieved a 5% improvement in classification score compared to the existing HAR platform.
Collapse
Affiliation(s)
- Keshav Thapa
- Department of Rehabilitation Medical Engineering, Daegu Haany University, Gyeongsan-si 38610, Republic of Korea
| | - Yousung Seo
- Department of Geriatric Rehabilitation, Daegu Haany University, Gyeongsan-si 38610, Republic of Korea
| | - Sung-Hyun Yang
- Department of Electronic Engineering, Kwangwoon University, Seoul 01897, Republic of Korea
| | - Kyong Kim
- Department of Rehabilitation Medical Engineering, Daegu Haany University, Gyeongsan-si 38610, Republic of Korea
| |
Collapse
|
7
|
Javeed M, Shorfuzzaman M, Alsufyani N, Chelloug SA, Jalal A, Park J. Physical human locomotion prediction using manifold regularization. PeerJ Comput Sci 2022; 8:e1105. [PMID: 36262158 PMCID: PMC9575869 DOI: 10.7717/peerj-cs.1105] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2022] [Accepted: 08/24/2022] [Indexed: 06/16/2023]
Abstract
Human locomotion is an imperative topic to be conversed among researchers. Predicting the human motion using multiple techniques and algorithms has always been a motivating subject matter. For this, different methods have shown the ability of recognizing simple motion patterns. However, predicting the dynamics for complex locomotion patterns is still immature. Therefore, this article proposes unique methods including the calibration-based filter algorithm and kinematic-static patterns identification for predicting those complex activities from fused signals. Different types of signals are extracted from benchmarked datasets and pre-processed using a novel calibration-based filter for inertial signals along with a Bessel filter for physiological signals. Next, sliding overlapped windows are utilized to get motion patterns defined over time. Then, polynomial probability distribution is suggested to decide the motion patterns natures. For features extraction based kinematic-static patterns, time and probability domain features are extracted over physical action dataset (PAD) and growing old together validation (GOTOV) dataset. Further, the features are optimized using quadratic discriminant analysis and orthogonal fuzzy neighborhood discriminant analysis techniques. Manifold regularization algorithms have also been applied to assess the performance of proposed prediction system. For the physical action dataset, we achieved an accuracy rate of 82.50% for patterned signals. While, the GOTOV dataset, we achieved an accuracy rate of 81.90%. As a result, the proposed system outdid when compared to the other state-of-the-art models in literature.
Collapse
Affiliation(s)
- Madiha Javeed
- Department of Computer Science, Air University, Islamabad, ICT, Pakistan
| | - Mohammad Shorfuzzaman
- Department of Computer Science, College of Computers and Information Technology, Taif University, Taif, Saudi Arabia
| | - Nawal Alsufyani
- Department of Computer Science, College of Computers and Information Technology, Taif University, Taif, Saudi Arabia
| | - Samia Allaoua Chelloug
- Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia
| | - Ahmad Jalal
- Department of Computer Science, Air University, Islamabad, ICT, Pakistan
| | - Jeongmin Park
- Department of Computer Engineering, Tech University of Korea, Sangidaehak-ro, Gyeonggi-do, South Korea
| |
Collapse
|