Estimation of kinematics from inertial measurement units using a combined deep learning and optimization framework

Application of Machine Learning Methods to Investigate Joint Load in Agility on the Football Field: Creating the Model, Part I

Laboratory studies have limitations in screening for anterior cruciate ligament (ACL) injury risk due to their lack of ecological validity. Machine learning (ML) methods coupled with wearable sensors are state-of-art approaches for joint load estimation outside the laboratory in athletic tasks. The aim of this study was to investigate ML approaches in predicting knee joint loading during sport-specific agility tasks. We explored the possibility of predicting high and low knee abduction moments (KAMs) from kinematic data collected in a laboratory setting through wearable sensors and of predicting the actual KAM from kinematics. Xsens MVN Analyze and Vicon motion analysis, together with Bertec force plates, were used. Talented female football (soccer) players (n = 32, age 14.8 ± 1.0 y, height 167.9 ± 5.1 cm, mass 57.5 ± 8.0 kg) performed unanticipated sidestep cutting movements (number of trials analyzed = 1105). According to the findings of this technical note, classification models that aim to identify the players exhibiting high or low KAM are preferable to the ones that aim to predict the actual peak KAM magnitude. The possibility of classifying high versus low KAMs during agility with good approximation (AUC 0.81-0.85) represents a step towards testing in an ecologically valid environment.

Human motion data expansion from arbitrary sparse sensors with shallow recurrent decoders

Advances in deep learning and sparse sensing have emerged as powerful tools for monitoring human motion in natural environments. We develop a deep learning architecture, constructed from a shallow recurrent decoder network, that expands human motion data by mapping a limited (sparse) number of sensors to a comprehensive (dense) configuration, thereby inferring the motion of unmonitored body segments. Even with a single sensor, we reconstruct the comprehensive set of time series measurements, which are important for tracking and informing movement-related health and performance outcomes. Notably, this mapping leverages sensor time histories to inform the transformation from sparse to dense sensor configurations. We apply this mapping architecture to a variety of datasets, including controlled movement tasks, gait pattern exploration, and free-moving environments. Additionally, this mapping can be subject-specific (based on an individual's unique data for deployment at home and in the community) or group-based (where data from a large group are used to learn a general movement model and predict outcomes for unknown subjects). By expanding our datasets to unmeasured or unavailable quantities, this work can impact clinical trials, robotic/device control, and human performance by improving the accuracy and availability of digital biomarker estimates.

Hip contact forces can be predicted with a neural network using only synthesised key points and electromyography in people with hip osteoarthritis

OBJECTIVE

To develop and validate a neural network to estimate hip contact forces (HCF), and lower body kinematics and kinetics during walking in individuals with hip osteoarthritis (OA) using synthesised anatomical key points and electromyography. To assess the capability of the neural network to detect directional changes in HCF resulting from prescribed gait modifications.

DESIGN

A calibrated electromyography-informed neuromusculoskeletal model was used to compute lower body joint angles, moments, and HCF for 17 participants with mild-to-moderate hip OA. Anatomical key points (e.g., joint centres) were synthesised from marker trajectories and augmented with bias and noise expected from computer vision-based pose estimation systems. Temporal convolutional and long short-term memory neural networks (NN) were trained using leave-one-subject-out validation to predict neuromusculoskeletal modelling outputs from the synthesised key points and measured electromyography data from 5 hip-spanning muscles.

RESULTS

HCF was predicted with an average error of 13.4 ± 7.1% of peak force. Joint angles and moments were predicted with an average root-mean-square-error of 5.3 degrees and 0.10 Nm/kg, respectively. The NN could detect changes in peak HCF that occur due to gait modifications with good agreement with neuromusculoskeletal modelling (r2 = 0.72) and a minimum detectable change of 9.5%.

CONCLUSION

The developed neural network predicted HCF and lower body joint angles and moments in individuals with hip OA using noisy synthesised key point locations with acceptable errors. Changes in HCF magnitude due to gait modifications were predicted with high accuracy. These findings have important implications for implementation of load-modification based gait retraining interventions for people with hip OA in a natural environment (i.e., home, clinic).

The effect of time normalization and biomechanical signal processing techniques of ground reaction force curves on deep-learning model performance

Time-series data are common in biomechanical studies. These data often undergo pre-processing steps such as time normalization or filtering prior to use in further analyses, including deep-learning classification. In this context, it remains unclear how these preprocessing steps affect deep-learning model performance. Thus, the aim of this study is to assess the effect of time-normalization and filtering on the performance of deep-learning classification models. We also investigated the effect of amplitude scaling. Using a public dataset (Gutenburg Gait Database, a ground reaction force database of level overground walking at self-selected walking speed involving 350 healthy individuals), we trained convolutional neural network (CNN) and long short-term memory (LSTM) models to predict binary sex (male, female) using three-dimensional ground-reaction forces to which we applied different processing approaches: zero padding, interpolation to 100% of signal, filtering, and scaling (min-max, body mass). The results show that transformations resulted in differences in model performances. Highest performance was obtained using unfiltered data, zero-padding, and min-max amplitude scaling (F1-score of 91 and 87% for CNN and LSTM, respectively). Not filtering data and using min-max scaling generally improve performance for both model architectures. For interpolation, results are not consistent across model architectures. This study suggests that processing steps must be considered in applications where deep-learning classification performance is relevant.

Validity and reliability of inertial measurement units used to measure motion of the lumbar spine: A systematic review of individuals with and without low back pain

Low back pain (LBP) is a leading cause of disability, resulting in aberrant movement. This movement is difficult to measure accurately in clinical practice and gold standard methods, such as optoelectronic systems involve the use of expensive laboratory equipment. Inertial measurement units (IMU) offer an alternative method of quantifying movement that is accessible in most environments. However, there is no consensus around the validity and reliability of IMUs for quantifying lumbar spine movements compared with gold standard measures. The aim of this systematic review was to establish concurrent validity and repeated measures reliability of using IMUs for the measurement of lumbar spine movements in individuals with and without LBP. A systematic search of electronic databases, incorporating PRISMA guidelines was completed, limited to the English language. 503 studies were identified where 15 studies met the inclusion criteria. Overall, 305 individuals were included, and 109 of these individuals had LBP. Weighted synthesis of the results demonstrated root mean squared differences of <2.4° compared to the gold standard and intraclass correlations >0.84 for lumbar spine movements. IMUs offer clinicians and researchers valid and reliable measurement of motion in the lumbar spine, comparable to laboratory methods, such as optoelectronic motion capture for individuals with and without LBP.

Integrating an LSTM framework for predicting ankle joint biomechanics during gait using inertial sensors

The ankle joint plays a crucial role in gait, facilitating the articulation of the lower limb, maintaining foot-ground contact, balancing the body, and transmitting the center of gravity. This study aimed to implement long short-term memory (LSTM) networks for predicting ankle joint angles, torques, and contact forces using inertial measurement unit (IMU) sensors. Twenty-five healthy participants were recruited. Two IMU sensors were attached to the foot dorsum and the vertical axis of the distal anteromedial tibia in the right lower limb to record acceleration and angular velocity during running. We proposed a LSTM-MLP (multilayer perceptron) model for training time-series data from IMU sensors and predicting ankle joint biomechanics. The model underwent validation and testing using a custom nested k-fold cross-validation process. The average values of the coefficient of determination (R2), mean absolute error (MAE), and mean squared error (MSE) for ankle dorsiflexion joint and moment, subtalar inversion joint and moment, and ankle joint contact forces were 0.89 ± 0.04, 0.75 ± 1.04, and 2.96 ± 4.96 for walking, and 0.87 ± 0.07, 0.88 ± 1.26, and 4.1 ± 7.17 for running, respectively. This study demonstrates that IMU sensors, combined with LSTM neural networks, are invaluable tools for evaluating ankle joint biomechanics in lower limb pathological diagnosis and rehabilitation, offering a cost-effective and versatile alternative to traditional experimental settings.

Sagittal plane knee kinematics can be measured during activities of daily living following total knee arthroplasty with two IMU

Knee function is rarely measured objectively during functional tasks following total knee arthroplasty. Inertial measurement units (IMU) can measure knee kinematics and range of motion (ROM) during dynamic activities and offer an easy-to-use system for knee function assessment post total knee arthroplasty. However, IMU must be validated against gold standard three-dimensional optical motion capture systems (OMC) across a range of tasks if they are to see widespread uptake. We computed knee rotations and ROM from commercial IMU sensor measurements during walking, squatting, sit-to-stand, stair ascent, and stair descent in 21 patients one-year post total knee arthroplasty using two methods: direct computation using segment orientations (r_IMU), and an IMU-driven iCloud-based interactive lower limb model (m_IMU). This cross-sectional study compared computed knee angles and ROM to a gold-standard OMC and inverse kinematics method using Pearson's correlation coefficient (R) and root-mean-square-differences (RMSD). The r_IMU and m_IMU methods estimated sagittal plane knee angles with excellent correlation (>0.95) compared to OMC for walking, squatting, sit-to-stand, and stair-ascent, and very good correlation (>0.90) for stair descent. For squatting, sit-to-stand, and walking, the mean RMSD for r_IMU and m_IMU compared to OMC were <4 degrees, < 5 degrees, and <6 degrees, respectively but higher for stair ascent and descent (~12 degrees). Frontal and transverse plane knee kinematics estimated using r_IMU and m_IMU showed poor to moderate correlation compared to OMC. There were no differences in ROM measurements during squatting, sit-to-stand, and walking across the two methods. Thus, IMUs can measure sagittal plane knee angles and ROM with high accuracy for a variety of tasks and may be a useful in-clinic tool for objective assessment of knee function following total knee arthroplasty.

Self-Supervised Learning Improves Accuracy and Data Efficiency for IMU-Based Ground Reaction Force Estimation

Objective

Recent deep learning techniques hold promise to enable IMU-driven kinetic assessment; however, they require large extents of ground reaction force (GRF) data to serve as labels for supervised model training. We thus propose using existing self-supervised learning (SSL) techniques to leverage large IMU datasets to pre-train deep learning models, which can improve the accuracy and data efficiency of IMU-based GRF estimation.

Methods

We performed SSL by masking a random portion of the input IMU data and training a transformer model to reconstruct the masked portion. We systematically compared a series of masking ratios across three pre-training datasets that included real IMU data, synthetic IMU data, or a combination of the two. Finally, we built models that used pre-training and labeled data to estimate GRF during three prediction tasks: overground walking, treadmill walking, and drop landing.

Results

When using the same amount of labeled data, SSL pre-training significantly improved the accuracy of 3-axis GRF estimation during walking compared to baseline models trained by conventional supervised learning. Fine-tuning SSL model with 1-10% of walking data yielded comparable accuracy to training baseline model with 100% of walking data. The optimal masking ratio for SSL is 6.25-12.5%.

Conclusion

SSL leveraged large real and synthetic IMU datasets to increase the accuracy and data efficiency of deep-learning-based GRF estimation, reducing the need for labeled data.

Significance

This work, with its open-source code and models, may unlock broader use cases of IMU-driven kinetic assessment by mitigating the scarcity of GRF measurements in practical applications.

Markerless Motion Tracking With Noisy Video and IMU Data

OBJECTIVE

Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home.

METHODS

We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities.

RESULTS

On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean ± std root-mean-squared errors by 7.6 ± 5.4 ° and 5.9 ± 3.3 °, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0 ± 5.3 ° for sensor-misplacement errors of up to 30.0 ± 13.7 ° and by up to 7.4 ± 5.5 ° for joint-center-estimation errors of up to 101.1 ± 11.2 mm, across joints.

CONCLUSION

These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.

Fusion of video and inertial sensing data via dynamic optimization of a biomechanical model

Inertial sensing and computer vision are promising alternatives to traditional optical motion tracking, but until now these data sources have been explored either in isolation or fused via unconstrained optimization, which may not take full advantage of their complementary strengths. By adding physiological plausibility and dynamical robustness to a proposed solution, biomechanical modeling may enable better fusion than unconstrained optimization. To test this hypothesis, we fused video and inertial sensing data via dynamic optimization with a nine degree-of-freedom model and investigated when this approach outperforms video-only, inertial-sensing-only, and unconstrained-fusion methods. We used both experimental and synthetic data that mimicked different ranges of video and inertial measurement unit (IMU) data noise. Fusion with a dynamically constrained model significantly improved estimation of lower-extremity kinematics over the video-only approach and estimation of joint centers over the IMU-only approach. It consistently outperformed single-modality approaches across different noise profiles. When the quality of video data was high and that of inertial data was low, dynamically constrained fusion improved estimation of joint kinematics and joint centers over unconstrained fusion, while unconstrained fusion was advantageous in the opposite scenario. These findings indicate that complementary modalities and techniques can improve motion tracking by clinically meaningful margins and that data quality and computational complexity must be considered when selecting the most appropriate method for a particular application.

Lower body kinematics estimation during walking using an accelerometer

Measuring and predicting accurate joint angles are important to developing analytical tools to gauge users' progress. Such measurement is usually performed in laboratory settings, which is difficult and expensive. So, the aim of this study was continuous estimation of lower limb joint angles during walking using an accelerometer and random forest (RF). Thus, 73 subjects (26 women and 47 men) voluntarily participated in this study. The subjects walked at the slow, moderate, and fast speeds on a walkway, which was covered with 10 Vicon camera. Acceleration was used as input for a RF to estimate ankle, knee, and hip angles (in transverse, frontal, and sagittal planes). Pearson correlation coefficient (r) and Mean Square Error (MSE) were computed between the experimental and estimated data. Paired statistical parametric mapping (SPM) t-test was used to compare the experimental and estimated data throughout gait cycle. The results of this study showed that the MSE of joint angles between the experimental and estimated data ranged from 0.04 to 24.29 and r > 0.91. Moreover, the findings of SPM indicated that there was no significant difference between the experimental and estimated data of ankle, knee, and hip angles in all three planes throughout gait cycle. The results of our research developed a more accessible, portable procedure to quantifying lower limb joint angles by an accelerometer and RF. So, such wearable-based joint angles have the potential to be used in outside-laboratory settings to measure walking kinematics.

Machine-Learning-Based Methodology for Estimation of Shoulder Load in Wheelchair-Related Activities Using Wearables

There is a high prevalence of shoulder problems in manual wheelchair users (MWUs) with a spinal cord injury. How shoulder load relates to shoulder problems remains unclear. This study aimed to develop a machine-learning-based methodology to estimate the shoulder load in wheelchair-related activities of daily living using wearable sensors. Ten able-bodied participants equipped with five inertial measurement units (IMU) on their thorax, right arm, and wheelchair performed activities exemplary of daily life of MWUs. Electromyography (EMG) was recorded from the long head of the biceps and medial part of the deltoid. A neural network was trained to predict the shoulder load based on IMU and EMG data. Different cross-validation strategies, sensor setups, and model architectures were examined. The predicted shoulder load was compared to the shoulder load determined with musculoskeletal modeling. A subject-specific biLSTM model trained on a sparse sensor setup yielded the most promising results (mean correlation coefficient = 0.74 ± 0.14, relative root-mean-squared error = 8.93% ± 2.49%). The shoulder-load profiles had a mean similarity of 0.84 ± 0.10 over all activities. This study demonstrates the feasibility of using wearable sensors and neural networks to estimate the shoulder load in wheelchair-related activities of daily living.

Wearables for Running Gait Analysis: A Systematic Review

BACKGROUND

Running gait assessment has traditionally been performed using subjective observation or expensive laboratory-based objective technologies, such as three-dimensional motion capture or force plates. However, recent developments in wearable devices allow for continuous monitoring and analysis of running mechanics in any environment. Objective measurement of running gait is an important (clinical) tool for injury assessment and provides measures that can be used to enhance performance.

OBJECTIVES

We aimed to systematically review the available literature investigating how wearable technology is being used for running gait analysis in adults.

METHODS

A systematic search of the literature was conducted in the following scientific databases: PubMed, Scopus, Web of Science and SPORTDiscus. Information was extracted from each included article regarding the type of study, participants, protocol, wearable device(s), main outcomes/measures, analysis and key findings.

RESULTS

A total of 131 articles were reviewed: 56 investigated the validity of wearable technology, 22 examined the reliability and 77 focused on applied use. Most studies used inertial measurement units (n = 62) [i.e. a combination of accelerometers, gyroscopes and magnetometers in a single unit] or solely accelerometers (n = 40), with one using gyroscopes alone and 31 using pressure sensors. On average, studies used one wearable device to examine running gait. Wearable locations were distributed among the shank, shoe and waist. The mean number of participants was 26 (± 27), with an average age of 28.3 (± 7.0) years. Most studies took place indoors (n = 93), using a treadmill (n = 62), with the main aims seeking to identify running gait outcomes or investigate the effects of injury, fatigue, intrinsic factors (e.g. age, sex, morphology) or footwear on running gait outcomes. Generally, wearables were found to be valid and reliable tools for assessing running gait compared to reference standards.

CONCLUSIONS

This comprehensive review highlighted that most studies that have examined running gait using wearable sensors have done so with young adult recreational runners, using one inertial measurement unit sensor, with participants running on a treadmill and reporting outcomes of ground contact time, stride length, stride frequency and tibial acceleration. Future studies are required to obtain consensus regarding terminology, protocols for testing validity and the reliability of devices and suitability of gait outcomes.

CLINICAL TRIAL REGISTRATION

CRD42021235527.

Evaluation of Error-State Kalman Filter Method for Estimating Human Lower-Limb Kinematics during Various Walking Gaits

Inertial measurement units (IMUs) offer an attractive way to study human lower-limb kinematics without traditional laboratory constraints. We present an error-state Kalman filter method to estimate 3D joint angles, joint angle ranges of motion, stride length, and step width using data from an array of seven body-worn IMUs. Importantly, this paper contributes a novel joint axis measurement correction that reduces joint angle drift errors without assumptions of strict hinge-like joint behaviors of the hip and knee. We evaluate the method compared to two optical motion capture methods on twenty human subjects performing six different types of walking gait consisting of forward walking (at three speeds), backward walking, and lateral walking (left and right). For all gaits, RMS differences in joint angle estimates generally remain below 5 degrees for all three ankle joint angles and for flexion/extension and abduction/adduction of the hips and knees when compared to estimates from reflective markers on the IMUs. Additionally, mean RMS differences in estimated stride length and step width remain below 0.13 m for all gait types, except stride length during slow walking. This study confirms the method's potential for non-laboratory based gait analysis, motivating further evaluation with IMU-only measurements and pathological gaits.

Synthesising 2D Video from 3D Motion Data for Machine Learning Applications

To increase the utility of legacy, gold-standard, three-dimensional (3D) motion capture datasets for computer vision-based machine learning applications, this study proposed and validated a method to synthesise two-dimensional (2D) video image frames from historic 3D motion data. We applied the video-based human pose estimation model OpenPose to real (in situ) and synthesised 2D videos and compared anatomical landmark keypoint outputs, with trivial observed differences (2.11−3.49 mm). We further demonstrated the utility of the method in a downstream machine learning use-case in which we trained and then tested the validity of an artificial neural network (ANN) to estimate ground reaction forces (GRFs) using synthesised and real 2D videos. Training an ANN to estimate GRFs using eight OpenPose keypoints derived from synthesised 2D videos resulted in accurate waveform GRF estimations (r > 0.9; nRMSE < 14%). When compared with using the smaller number of real videos only, accuracy was improved by adding the synthetic views and enlarging the dataset. The results highlight the utility of the developed approach to enlarge small 2D video datasets, or to create 2D video images to accompany 3D motion capture datasets to make them accessible for machine learning applications.

Predicting Coordination Variability of Selected Lower Extremity Couplings during a Cutting Movement: An Investigation of Deep Neural Networks with the LSTM Structure

There are still few portable methods for monitoring lower limb joint coordination during the cutting movements (CM). This study aims to obtain the relevant motion biomechanical parameters of the lower limb joints at 90°, 135°, and 180° CM by collecting IMU data of the human lower limbs, and utilizing the Long Short-Term Memory (LSTM) deep neural-network framework to predict the coordination variability of selected lower extremity couplings at the three CM directions. There was a significant (p < 0.001) difference between the three couplings during the swing, especially at 90° vs the other directions. At 135° and 180°, t13-he coordination variability of couplings was significantly greater than at 90° (p < 0.001). It is important to note that the coordination variability of Hip rotation/Knee flexion-extension was significantly higher at 90° than at 180° (p < 0.001). By the LSTM, the CM coordination variability for 90° (CMC = 0.99063, RMSE = 0.02358), 135° (CMC = 0.99018, RMSE = 0.02465) and 180° (CMC = 0.99485, RMSE = 0.01771) were accurately predicted. The predictive model could be used as a reliable tool for predicting the coordination variability of different CM directions in patients or athletes and real-world open scenarios using inertial sensors.

Feasibility of Markerless Motion Capture for Three-Dimensional Gait Assessment in Community Settings

Three-dimensional (3D) kinematic analysis of gait holds potential as a digital biomarker to identify neuropathologies, monitor disease progression, and provide a high-resolution outcome measure to monitor neurorehabilitation efficacy by characterizing the mechanisms underlying gait impairments. There is a need for 3D motion capture technologies accessible to community, clinical, and rehabilitation settings. Image-based markerless motion capture (MLMC) using neural network-based deep learning algorithms shows promise as an accessible technology in these settings. In this study, we assessed the feasibility of implementing 3D MLMC technology outside the traditional laboratory environment to evaluate its potential as a tool for outcomes assessment in neurorehabilitation. A sample population of 166 individuals aged 9-87 years (mean 43.7, S.D. 20.4) of varied health history were evaluated at six different locations in the community over a 3-month period. Participants walked overground at self-selected (SS) and fastest comfortable (FC) speeds. Feasibility measures considered the expansion, implementation, and practicality of this MLMC system. A subset of the sample population (46 individuals) walked over a pressure-sensitive walkway (PSW) concurrently with MLMC to assess agreement of the spatiotemporal gait parameters measured between the two systems. Twelve spatiotemporal parameters were compared using mean differences, Bland-Altman analysis, and intraclass correlation coefficients for agreement (ICC2,1) and consistency (ICC3,1). All measures showed good to excellent agreement between MLMC and the PSW system with cadence, speed, step length, step time, stride length, and stride time showing strong similarity. Furthermore, this information can inform the development of rehabilitation strategies targeting gait dysfunction. These first experiments provide evidence for feasibility of using MLMC in community and clinical practice environments to acquire robust 3D kinematic data from a diverse population. This foundational work enables future investigation with MLMC especially its use as a digital biomarker of disease progression and rehabilitation outcome.

Recent Machine Learning Progress in Lower Limb Running Biomechanics With Wearable Technology: A Systematic Review

With the emergence of wearable technology and machine learning approaches, gait monitoring in real-time is attracting interest from the sports biomechanics community. This study presents a systematic review of machine learning approaches in running biomechanics using wearable sensors. Electronic databases were retrieved in PubMed, Web of Science, SPORTDiscus, Scopus, IEEE Xplore, and ScienceDirect. A total of 4,068 articles were identified via electronic databases. Twenty-four articles that met the eligibility criteria after article screening were included in this systematic review. The range of quality scores of the included studies is from 0.78 to 1.00, with 40% of articles recruiting participant numbers between 20 and 50. The number of inertial measurement unit (IMU) placed on the lower limbs varied from 1 to 5, mainly in the pelvis, thigh, distal tibia, and foot. Deep learning algorithms occupied 57% of total machine learning approaches. Convolutional neural networks (CNN) were the most frequently used deep learning algorithm. However, the validation process for machine learning models was lacking in some studies and should be given more attention in future research. The deep learning model combining multiple CNN and recurrent neural networks (RNN) was observed to extract different running features from the wearable sensors and presents a growing trend in running biomechanics.

DeepBBWAE-Net: A CNN-RNN Based Deep SuperLearner For Estimating Lower Extremity Sagittal Plane Joint Kinematics Using Shoe-Mounted IMU Sensors In Daily Living

Measurement of human body movement is an essential step in biomechanical analysis. The current standard for human motion capture systems uses infrared cameras to track reflective markers placed on the subject. While these systems can accurately track joint kinematics, the analyses are spatially limited to the lab environment. Though Inertial Measurement Unit (IMU) can eliminate the spatial limitations of the motion capture system, those systems are impractical for use in daily living due to the need for many sensors, typically one per body segment. Due to the need for practical and accurate estimation of joint kinematics, this study implements a reduced number of IMU sensors and employs machine learning algorithm to map sensor data to joint angles. Our developed algorithm estimates hip, knee, and ankle angles in the sagittal plane using two shoe-mounted IMU sensors in different practical walking conditions: treadmill, level overground, stair, and slope conditions. Specifically, we proposed five deep learning networks that use combinations of Convolutional Neural Networks (CNN) and Gated Recurrent Unit (GRU) based Recurrent Neural Networks (RNN) as base learners for our framework. Using those five baseline models, we proposed a novel framework, DeepBBWAE-Net, that implements ensemble techniques such as bagging, boosting, and weighted averaging to improve kinematic predictions. DeepBBWAE-Net predicts joint kinematics for the three joint angles under all the walking conditions with a Root Mean Square Error (RMSE) 6.93-29.0% lower than base models individually. This is the first study that uses a reduced number of IMU sensors to estimate kinematics in multiple walking environments.

OpenSense: An open-source toolbox for inertial-measurement-unit-based measurement of lower extremity kinematics over long durations

Background

The ability to measure joint kinematics in natural environments over long durations using inertial measurement units (IMUs) could enable at-home monitoring and personalized treatment of neurological and musculoskeletal disorders. However, drift, or the accumulation of error over time, inhibits the accurate measurement of movement over long durations. We sought to develop an open-source workflow to estimate lower extremity joint kinematics from IMU data that was accurate and capable of assessing and mitigating drift.

Methods

We computed IMU-based estimates of kinematics using sensor fusion and an inverse kinematics approach with a constrained biomechanical model. We measured kinematics for 11 subjects as they performed two 10-min trials: walking and a repeated sequence of varied lower-extremity movements. To validate the approach, we compared the joint angles computed with IMU orientations to the joint angles computed from optical motion capture using root mean square (RMS) difference and Pearson correlations, and estimated drift using a linear regression on each subject’s RMS differences over time.

Results

IMU-based kinematic estimates agreed with optical motion capture; median RMS differences over all subjects and all minutes were between 3 and 6 degrees for all joint angles except hip rotation and correlation coefficients were moderate to strong (r = 0.60–0.87). We observed minimal drift in the RMS differences over 10 min; the average slopes of the linear fits to these data were near zero (− 0.14–0.17 deg/min).

Conclusions

Our workflow produced joint kinematics consistent with those estimated by optical motion capture, and could mitigate kinematic drift even in the trials of continuous walking without rest, which may obviate the need for explicit sensor recalibration (e.g. sitting or standing still for a few seconds or zero-velocity updates) used in current drift-mitigation approaches when studying similar activities. This could enable long-duration measurements, bringing the field one step closer to estimating kinematics in natural environments.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12984-022-01001-x.

Population and Age-Based Cardiorespiratory Fitness Level Investigation and Automatic Prediction

Maximal oxygen consumption (VO₂max) reflects aerobic capacity and is crucial for assessing cardiorespiratory fitness and physical activity level. The purpose of this study was to classify and predict the population-based cardiorespiratory fitness based on anthropometric parameters, workload, and steady-state heart rate (HR) of the submaximal exercise test. Five hundred and seventeen participants were recruited into this study. This study initially classified aerobic capacity followed by VO₂max predicted using an ordinary least squares regression model with measured VO₂max from a submaximal cycle test as ground truth. Furthermore, we predicted VO₂max in the age ranges 21–40 and above 40. For the support vector classification model, the test accuracy was 75%. The ordinary least squares regression model showed the coefficient of determination (R²) between measured and predicted VO₂max was 0.83, mean absolute error (MAE) and root mean square error (RMSE) were 3.12 and 4.24 ml/kg/min, respectively. R² in the age 21–40 and above 40 groups were 0.85 and 0.75, respectively. In conclusion, this study provides a practical protocol for estimating cardiorespiratory fitness of an individual in large populations. An applicable submaximal test for population-based cohorts could evaluate physical activity levels and provide exercise recommendations.

Predicting Knee Joint Kinematics from Wearable Sensor Data in People with Knee Osteoarthritis and Clinical Considerations for Future Machine Learning Models

Deep learning models developed to predict knee joint kinematics are usually trained on inertial measurement unit (IMU) data from healthy people and only for the activity of walking. Yet, people with knee osteoarthritis have difficulties with other activities and there are a lack of studies using IMU training data from this population. Our objective was to conduct a proof-of-concept study to determine the feasibility of using IMU training data from people with knee osteoarthritis performing multiple clinically important activities to predict knee joint sagittal plane kinematics using a deep learning approach. We trained a bidirectional long short-term memory model on IMU data from 17 participants with knee osteoarthritis to estimate knee joint flexion kinematics for phases of walking, transitioning to and from a chair, and negotiating stairs. We tested two models, a double-leg model (four IMUs) and a single-leg model (two IMUs). The single-leg model demonstrated less prediction error compared to the double-leg model. Across the different activity phases, RMSE (SD) ranged from 7.04° (2.6) to 11.78° (6.04), MAE (SD) from 5.99° (2.34) to 10.37° (5.44), and Pearson’s R from 0.85 to 0.99 using leave-one-subject-out cross-validation. This study demonstrates the feasibility of using IMU training data from people who have knee osteoarthritis for the prediction of kinematics for multiple clinically relevant activities.

American society of biomechanics early career achievement award 2020: Toward portable and modular biomechanics labs: How video and IMU fusion will change gait analysis

The field of biomechanics is at a turning point, with marker-based motion capture set to be replaced by portable and inexpensive hardware, rapidly improving markerless tracking algorithms, and open datasets that will turn these new technologies into field-wide team projects. Despite progress, several challenges inhibit both inertial and vision-based motion tracking from reaching the high accuracies that many biomechanics applications require. Their complementary strengths, however, could be harnessed toward better solutions than those offered by either modality alone. The drift from inertial measurement units (IMUs) could be corrected by video data, while occlusions in videos could be corrected by inertial data. To expedite progress in this direction, we have collected the CMU Panoptic Dataset 2.0, which contains 86 subjects captured with 140 VGA cameras, 31 HD cameras, and 15 IMUs, performing on average 6.5 min of activities, including range of motion activities and tasks of daily living. To estimate ground-truth kinematics, we imposed simultaneous consistency with the video and IMU data. Three-dimensional joint centers were first computed by geometrically triangulating proposals from a convolutional neural network applied to each video independently. A statistical meshed model parametrized in terms of body shape and pose was then fit through a top-down optimization approach that enforced consistency with both the video-based joint centers and IMU data. As proof of concept, we used this dataset to benchmark pose estimation from a sparse set of sensors, showing that incorporation of complementary modalities is a promising frontier that can be further strengthened through physics-informed frameworks.

Validity and Sensitivity of an Inertial Measurement Unit-Driven Biomechanical Model of Motor Variability for Gait

Motor variability in gait is frequently linked to fall risk, yet field-based biomechanical joint evaluations are scarce. We evaluated the validity and sensitivity of an inertial measurement unit (IMU)-driven biomechanical model of joint angle variability for gait. Fourteen healthy young adults completed seven-minute trials of treadmill gait at several speeds and arm swing amplitudes. Trunk, pelvis, and lower-limb joint kinematics were estimated by IMU- and optoelectronic-based models using OpenSim. We calculated range of motion (ROM), magnitude of variability (meanSD), local dynamic stability (λmax), persistence of ROM fluctuations (DFAα), and regularity (SaEn) of each angle over 200 continuous strides, and evaluated model accuracy (RMSD: root mean square difference), consistency (ICC2,1: intraclass correlation), biases, limits of agreement, and sensitivity to within-participant gait responses (effects of speed and swing). RMSDs of joint angles were 1.7-9.2° (pooled mean of 4.8°), excluding ankle inversion. ICCs were mostly good to excellent in the primary plane of motion for ROM and in all planes for meanSD and λmax, but were poor to moderate for DFAα and SaEn. Modelled speed and swing responses for ROM, meanSD, and λmax were similar. Results suggest that the IMU-driven model is valid and sensitive for field-based assessments of joint angle time series, ROM in the primary plane of motion, magnitude of variability, and local dynamic stability.

The Use of Synthetic IMU Signals in the Training of Deep Learning Models Significantly Improves the Accuracy of Joint Kinematic Predictions

Gait analysis based on inertial sensors has become an effective method of quantifying movement mechanics, such as joint kinematics and kinetics. Machine learning techniques are used to reliably predict joint mechanics directly from streams of IMU signals for various activities. These data-driven models require comprehensive and representative training datasets to be generalizable across the movement variability seen in the population at large. Bottlenecks in model development frequently occur due to the lack of sufficient training data and the significant time and resources necessary to acquire these datasets. Reliable methods to generate synthetic biomechanical training data could streamline model development and potentially improve model performance. In this study, we developed a methodology to generate synthetic kinematics and the associated predicted IMU signals using open source musculoskeletal modeling software. These synthetic data were used to train neural networks to predict three degree-of-freedom joint rotations at the hip and knee during gait either in lieu of or along with previously measured experimental gait data. The accuracy of the models’ kinematic predictions was assessed using experimentally measured IMU signals and gait kinematics. Models trained using the synthetic data out-performed models using only the experimental data in five of the six rotational degrees of freedom at the hip and knee. On average, root mean square errors in joint angle predictions were improved by 38% at the hip (synthetic data RMSE: 2.3°, measured data RMSE: 4.5°) and 11% at the knee (synthetic data RMSE: 2.9°, measured data RMSE: 3.3°), when models trained solely on synthetic data were compared to measured data. When models were trained on both measured and synthetic data, root mean square errors were reduced by 54% at the hip (measured + synthetic data RMSE: 1.9°) and 45% at the knee (measured + synthetic data RMSE: 1.7°), compared to measured data alone. These findings enable future model development for different activities of clinical significance without the burden of generating large quantities of gait lab data for model training, streamlining model development, and ultimately improving model performance.

A Comparison of Three Neural Network Approaches for Estimating Joint Angles and Moments from Inertial Measurement Units

The application of artificial intelligence techniques to wearable sensor data may facilitate accurate analysis outside of controlled laboratory settings—the holy grail for gait clinicians and sports scientists looking to bridge the lab to field divide. Using these techniques, parameters that are difficult to directly measure in-the-wild, may be predicted using surrogate lower resolution inputs. One example is the prediction of joint kinematics and kinetics based on inputs from inertial measurement unit (IMU) sensors. Despite increased research, there is a paucity of information examining the most suitable artificial neural network (ANN) for predicting gait kinematics and kinetics from IMUs. This paper compares the performance of three commonly employed ANNs used to predict gait kinematics and kinetics: multilayer perceptron (MLP); long short-term memory (LSTM); and convolutional neural networks (CNN). Overall high correlations between ground truth and predicted kinematic and kinetic data were found across all investigated ANNs. However, the optimal ANN should be based on the prediction task and the intended use-case application. For the prediction of joint angles, CNNs appear favourable, however these ANNs do not show an advantage over an MLP network for the prediction of joint moments. If real-time joint angle and joint moment prediction is desirable an LSTM network should be utilised.