Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Action Recognition of Taekwondo Unit Actions Using Action Images Constructed with Time-Warped Motion Profiles

Taekwondo has evolved from a traditional martial art into an official Olympic sport. This study introduces a novel action recognition model tailored for Taekwondo unit actions, utilizing joint-motion data acquired via wearable inertial measurement unit (IMU) sensors. The utilization of IMU sensor-measured motion data facilitates the capture of the intricate and rapid movements characteristic of Taekwondo techniques. The model, underpinned by a conventional convolutional neural network (CNN)-based image classification framework, synthesizes action images to represent individual Taekwondo unit actions. These action images are generated by mapping joint-motion profiles onto the RGB color space, thus encapsulating the motion dynamics of a single unit action within a solitary image. To further refine the representation of rapid movements within these images, a time-warping technique was applied, adjusting motion profiles in relation to the velocity of the action. The effectiveness of the proposed model was assessed using a dataset compiled from 40 Taekwondo experts, yielding remarkable outcomes: an accuracy of 0.998, a precision of 0.983, a recall of 0.982, and an F1 score of 0.982. These results underscore this time-warping technique's contribution to enhancing feature representation, as well as the proposed method's scalability and effectiveness in recognizing Taekwondo unit actions.

Wearable sensor and machine learning estimate tendon load and walking speed during immobilizing boot ambulation

The purpose of this study is to develop a wearable paradigm to accurately monitor Achilles tendon loading and walking speed using wearable sensors that reduce subject burden. Ten healthy adults walked in an immobilizing boot under various heel wedge conditions (30°, 5°, 0°) and walking speeds. Three-dimensional motion capture, ground reaction force, and 6-axis inertial measurement unit (IMU) signals were collected. We used a Least Absolute Shrinkage and Selection Operator (LASSO) regression to predict peak Achilles tendon load and walking speed. The effects of altering sensor parameters were also explored. Walking speed models (mean absolute percentage error (MAPE): 8.81 ± 4.29%) outperformed tendon load models (MAPE: 34.93 ± 26.3%). Models trained with subject-specific data performed better than models trained without subject-specific data. Removing the gyroscope, decreasing the sampling frequency, and using combinations of sensors did not change the usability of the models, having inconsequential effects on model performance. We developed a simple monitoring paradigm that uses LASSO regression and wearable sensors to accurately predict (MAPE ≤ 12.6%) Achilles tendon loading and walking speed while ambulating in an immobilizing boot. This paradigm provides a clinically implementable strategy to longitudinally monitor patient loading and activity while recovering from Achilles tendon injuries.

Wearable sensor and machine learning accurately estimate tendon load and walking speed during immobilizing boot ambulation

Achilles tendon injuries are treated with progressive weight bearing to promote tendon healing and restore function. Patient rehabilitation progression are typically studied in controlled, lab settings and do not represent the long-term loading experienced during daily living. The purpose of this study is to develop a wearable paradigm to accurately monitor Achilles tendon loading and walking speed using low-cost sensors that reduce subject burden. Ten healthy adults walked in an immobilizing boot under various heel wedge conditions (30°, 5°, 0°) and walking speeds. Three-dimensional motion capture, ground reaction force, and 6-axis inertial measurement unit (IMU) signals were collected per trial. We used Least Absolute Shrinkage and Selection Operator (LASSO) regression to predict peak Achilles tendon load and walking speed. The effects of using only accelerometer data, different sampling frequency, and multiple sensors to train the model were also explored. Walking speed models outperformed (mean absolute percentage error (MAPE): 8.41 ± 4.08%) tendon load models (MAPE: 33.93 ± 23.9%). Models trained with subject-specific data performed significantly better than generalized models. For example, our personalized model that was trained with only subject-specific data predicted tendon load with a 11.5 ± 4.41% MAPE and walking speed with a 4.50 ± 0.91% MAPE. Removing gyroscope channels, decreasing sampling frequency, and using combinations of sensors had inconsequential effects on models performance (changes in MAPE < 6.09%). We developed a simple monitoring paradigm that uses LASSO regression and wearable sensors to accurately predict Achilles tendon loading and walking speed while ambulating in an immobilizing boot. This paradigm provides a clinically implementable strategy to longitudinally monitor patient loading and activity while recovering from Achilles tendon injuries.

An Explainable Spatial-Temporal Graphical Convolutional Network to Score Freezing of Gait in Parkinsonian Patients

Freezing of gait (FOG) is a poorly understood heterogeneous gait disorder seen in patients with parkinsonism which contributes to significant morbidity and social isolation. FOG is currently measured with scales that are typically performed by movement disorders specialists (i.e., MDS-UPDRS), or through patient completed questionnaires (N-FOG-Q) both of which are inadequate in addressing the heterogeneous nature of the disorder and are unsuitable for use in clinical trials The purpose of this study was to devise a method to measure FOG objectively, hence improving our ability to identify it and accurately evaluate new therapies. A major innovation of our study is that it is the first study of its kind that uses the largest sample size (>30 h, N = 57) in order to apply explainable, multi-task deep learning models for quantifying FOG over the course of the medication cycle and at varying levels of parkinsonism severity. We trained interpretable deep learning models with multi-task learning to simultaneously score FOG (cross-validated F1 score 97.6%), identify medication state (OFF vs. ON levodopa; cross-validated F1 score 96.8%), and measure total PD severity (MDS-UPDRS-III score prediction error ≤ 2.7 points) using kinematic data of a well-characterized sample of N = 57 patients during levodopa challenge tests. The proposed model was able to explain how kinematic movements are associated with each FOG severity level that were highly consistent with the features, in which movement disorders specialists are trained to identify as characteristics of freezing. Overall, we demonstrate that deep learning models' capability to capture complex movement patterns in kinematic data can automatically and objectively score FOG with high accuracy. These models have the potential to discover novel kinematic biomarkers for FOG that can be used for hypothesis generation and potentially as clinical trial outcome measures.

An explainable spatial-temporal graphical convolutional network to score freezing of gait in parkinsonian patients

Freezing of gait (FOG) is a poorly understood heterogeneous gait disorder seen in patients with parkinsonism which contributes to significant morbidity and social isolation. FOG is currently measured with scales that are typically performed by movement disorders specialists (ie. MDS-UPDRS), or through patient completed questionnaires (N-FOG-Q) both of which are inadequate in addressing the heterogeneous nature of the disorder and are unsuitable for use in clinical trials The purpose of this study was to devise a method to measure FOG objectively, hence improving our ability to identify it and accurately evaluate new therapies. We trained interpretable deep learning models with multi-task learning to simultaneously score FOG (cross-validated F1 score 97.6%), identify medication state (OFF vs. ON levodopa; cross-validated F1 score 96.8%), and measure total PD severity (MDS-UPDRS-III score prediction error ≤ 2.7 points) using kinematic data of a well-characterized sample of N=57 patients during levodopa challenge tests. The proposed model was able to identify kinematic features associated with each FOG severity level that were highly consistent with the features that movement disorders specialists are trained to identify as characteristic of freezing. In this work, we demonstrate that deep learning models' capability to capture complex movement patterns in kinematic data can automatically and objectively score FOG with high accuracy. These models have the potential to discover novel kinematic biomarkers for FOG that can be used for hypothesis generation and potentially as clinical trial outcome measures.

Semi-Supervised Adversarial Auto-Encoder to Expedite Human Activity Recognition

The study of human activity recognition concentrates on classifying human activities and the inference of human behavior using modern sensing technology. However, the issue of domain adaptation for inertial sensing-based human activity recognition (HAR) is still burdensome. The existing requirement of labeled training data for adapting such classifiers to every new person, device, or on-body location is a significant barrier to the widespread adoption of HAR-based applications, making this a challenge of high practical importance. We propose the semi-supervised HAR method to improve reconstruction and generation. It executes proper adaptation with unlabeled data without changes to a pre-trained HAR classifier. Our approach decouples VAE with adversarial learning to ensure robust classifier operation, without newly labeled training data, under changes to the individual activity and the on-body sensor position. Our proposed framework shows the empirical results using the publicly available benchmark dataset compared to state-of-art baselines, achieving competitive improvement for handling new and unlabeled activity. The result demonstrates SAA has achieved a 5% improvement in classification score compared to the existing HAR platform.

Physical human locomotion prediction using manifold regularization

Human locomotion is an imperative topic to be conversed among researchers. Predicting the human motion using multiple techniques and algorithms has always been a motivating subject matter. For this, different methods have shown the ability of recognizing simple motion patterns. However, predicting the dynamics for complex locomotion patterns is still immature. Therefore, this article proposes unique methods including the calibration-based filter algorithm and kinematic-static patterns identification for predicting those complex activities from fused signals. Different types of signals are extracted from benchmarked datasets and pre-processed using a novel calibration-based filter for inertial signals along with a Bessel filter for physiological signals. Next, sliding overlapped windows are utilized to get motion patterns defined over time. Then, polynomial probability distribution is suggested to decide the motion patterns natures. For features extraction based kinematic-static patterns, time and probability domain features are extracted over physical action dataset (PAD) and growing old together validation (GOTOV) dataset. Further, the features are optimized using quadratic discriminant analysis and orthogonal fuzzy neighborhood discriminant analysis techniques. Manifold regularization algorithms have also been applied to assess the performance of proposed prediction system. For the physical action dataset, we achieved an accuracy rate of 82.50% for patterned signals. While, the GOTOV dataset, we achieved an accuracy rate of 81.90%. As a result, the proposed system outdid when compared to the other state-of-the-art models in literature.