The accuracy of several pose estimation methods for 3D joint centre localisation

Video-Based Kinematic Analysis of Movement Quality in a Phase 3 Clinical Trial of Troriluzole in Adults with Spinocerebellar Ataxia: A Post Hoc Analysis

INTRODUCTION

Traditional methods for assessing movement quality rely on subjective standardized scales and clinical expertise. This limitation creates challenges for assessing patients with spinocerebellar ataxia (SCA), in whom changes in mobility can be subtle and varied. We hypothesized that a machine learning analytic system might complement traditional clinician-rated measures of gait. Our objective was to use a video-based assessment of gait dispersion to compare the effects of troriluzole with placebo on gait quality in adults with SCA.

METHODS

Participants with SCA underwent gait assessment in a phase 3, double-blind, placebo-controlled trial of troriluzole (NCT03701399). Videos were processed through a deep learning pose extraction algorithm, followed by the estimation of a novel gait stability measure, the Pose Dispersion Index, quantifying the frame-by-frame symmetry, balance, and stability during natural and tandem walk tasks. The effects of troriluzole treatment were assessed in mixed linear models, participant-level grouping, and treatment group-by-visit week interaction adjusted for age, sex, baseline modified Functional Scale for the Assessment and Rating of Ataxia (f-SARA), and time since diagnosis.

RESULTS

From 218 randomized participants, 67 and 56 participants had interpretable videos of a tandem and natural walk attempt, respectively. At Week 48, individuals assigned to troriluzole exhibited significant (p = 0.010) improvement in tandem walk Pose Dispersion Index versus placebo {adjusted interaction coefficient: 0.584 [95% confidence interval (CI) 0.137 to 1.031]}. A similar, nonsignificant trend was observed in the natural walk assessment [coefficient: 1.198 (95% CI - 1.067 to 3.462)]. Further, lower baseline Pose Dispersion Index during the natural walk was significantly (p = 0.041) associated with a higher risk of subsequent falls [adjusted Poisson coefficient: - 0.356 [95% CI - 0.697 to - 0.014)].

CONCLUSION

Using this novel approach, troriluzole-treated subjects demonstrated improvement in gait as compared to placebo for the tandem walk. Machine learning applied to video-captured gait parameters can complement clinician-reported motor assessment in adults with SCA. The Pose Dispersion Index may enhance assessment in future research. TRIAL REGISTRATION-CLINICALTRIALS.

GOV IDENTIFIER

NCT03701399.

Smartphone videos-driven musculoskeletal multibody dynamics modelling workflow to estimate the lower limb joint contact forces and ground reaction forces

The estimation of joint contact forces in musculoskeletal multibody dynamics models typically requires the use of expensive and time-consuming technologies, such as reflective marker-based motion capture (Mocap) system. In this study, we aim to propose a more accessible and cost-effective solution that utilizes the dual smartphone videos (SPV)-driven musculoskeletal multibody dynamics modeling workflow to estimate the lower limb mechanics. Twelve participants were recruited to collect marker trajectory data, force plate data, and motion videos during walking and running. The smartphone videos were initially analyzed using the OpenCap platform to identify key joint points and anatomical markers. The markers were used as inputs for the musculoskeletal multibody dynamics model to calculate the lower limb joint kinematics, joint contact forces, and ground reaction forces, which were then evaluated by the Mocap-based workflow. The root mean square error (RMSE), mean absolute deviation (MAD), and Pearson correlation coefficient (ρ) were adopted to evaluate the results. Excellent or strong Pearson correlations were observed in most lower limb joint angles (ρ = 0.74 ~ 0.94). The averaged MADs and RMSEs for the joint angles were 1.93 ~ 6.56° and 2.14 ~ 7.08°, respectively. Excellent or strong Pearson correlations were observed in most lower limb joint contact forces and ground reaction forces (ρ = 0.78 ~ 0.92). The averaged MADs and RMSEs for the joint lower limb joint contact forces were 0.18 ~ 1.07 bodyweight (BW) and 0.28 ~ 1.32 BW, respectively. Overall, the proposed smartphone video-driven musculoskeletal multibody dynamics simulation workflow demonstrated reliable accuracy in predicting lower limb mechanics and ground reaction forces, which has the potential to expedite gait dynamics analysis in a clinical setting.

A leg model based on anatomical landmarks to study 3D joint kinematics of walking in Drosophila melanogaster

Walking is the most common form of how animals move on land. The model organism Drosophila melanogaster has become increasingly popular for studying how the nervous system controls behavior in general and walking in particular. Despite recent advances in tracking and modeling leg movements of walking Drosophila in 3D, there are still gaps in knowledge about the biomechanics of leg joints due to the tiny size of fruit flies. For instance, the natural alignment of joint rotational axes was largely neglected in previous kinematic analyses. In this study, we therefore present a detailed kinematic leg model in which not only the segment lengths but also the main rotational axes of the joints were derived from anatomical landmarks, namely, the joint condyles. Our model with natural oblique joint axes is able to adapt to the 3D leg postures of straight and forward walking fruit flies with high accuracy. When we compared our model to an orthogonalized version, we observed that our model showed a smaller error as well as differences in the used range of motion (ROM), highlighting the advantages of modeling natural rotational axes alignment for the study of joint kinematics. We further found that the kinematic profiles of front, middle, and hind legs differed in the number of required degrees of freedom as well as their contributions to stepping, time courses of joint angles, and ROM. Our findings provide deeper insights into the joint kinematics of walking in Drosophila, and, additionally, will help to develop dynamical, musculoskeletal, and neuromechanical simulations.

Validation and application of computer vision algorithms for video-based tremor analysis

Tremor is one of the most common neurological symptoms. Its clinical and neurobiological complexity necessitates novel approaches for granular phenotyping. Instrumented neurophysiological analyses have proven useful, but are highly resource-intensive and lack broad accessibility. In contrast, bedside scores are simple to administer, but lack the granularity to capture subtle but relevant tremor features. We utilise the open-source computer vision pose tracking algorithm Mediapipe to track hands in clinical video recordings and use the resulting time series to compute canonical tremor features. This approach is compared to marker-based 3D motion capture, wrist-worn accelerometry, clinical scoring and a second, specifically trained tremor-specific algorithm in two independent clinical cohorts. These cohorts consisted of 66 patients diagnosed with essential tremor, assessed in different task conditions and states of deep brain stimulation therapy. We find that Mediapipe-derived tremor metrics exhibit high convergent clinical validity to scores (Spearman's ρ = 0.55-0.86, p≤ .01) as well as an accuracy of up to 2.60 mm (95% CI [-3.13, 8.23]) and ≤0.21 Hz (95% CI [-0.05, 0.46]) for tremor amplitude and frequency measurements, matching gold-standard equipment. Mediapipe, but not the disease-specific algorithm, was capable of analysing videos involving complex configurational changes of the hands. Moreover, it enabled the extraction of tremor features with diagnostic and prognostic relevance, a dimension which conventional tremor scores were unable to provide. Collectively, this demonstrates that current computer vision algorithms can be transformed into an accurate and highly accessible tool for video-based tremor analysis, yielding comparable results to gold standard tremor recordings.

Prediction of Achilles Tendon Force During Common Motor Tasks From Markerless Video

Remodeling of the Achilles tendon (AT) is partly driven by its mechanical environment. AT force can be estimated with neuromusculoskeletal (NMSK) modeling; however, the complex experimental setup required to perform the analyses confines use to the laboratory. We developed task-specific long short-term memory (LSTM) neural networks that employ markerless video data to predict the AT force during walking, running, countermovement jump, single-leg landing, and single-leg heel rise. The task-specific LSTM models were trained on pose estimation keypoints and corresponding AT force data from 16 subjects, calculated via an established NMSK modeling pipeline, and cross-validated using a leave-one-subject-out approach. As proof-of-concept, new motion data of one participant was collected with two smartphones and used to predict AT forces. The task-specific LSTM models predicted the time-series AT force using synthesized pose estimation data with root mean square error (RMSE) ≤ 526 N, normalized RMSE (nRMSE) ≤ 0.21 , R 2 ≥ 0.81 . Walking task resulted the most accurate with RMSE = 189±62 N; nRMSE = 0.11±0.03 , R 2 = 0.92±0.04 . AT force predicted with smartphones video data was physiologically plausible, agreeing in timing and magnitude with established force profiles. This study demonstrated the feasibility of using low-cost solutions to deploy complex biomechanical analyses outside the laboratory.

Inclusion of a skeletal model partly improves the reliability of lower limb joint angles derived from a markerless depth camera

A single depth camera provides a fast and easy approach to performing biomechanical assessments in a clinical setting; however, there are currently no established methods to reliably determine joint angles from these devices. The primary aim of this study was to compare joint angles as well as the between-day reliability of direct kinematics to model-constrained inverse kinematics recorded using a single markerless depth camera during a range of clinical and athletic movement assessments.A secondary aim was to determine the minimum number of trials required to maximize reliability. Eighteen healthy participants attended two testing sessions one week apart. Tasks included treadmill walking, treadmill running, single-leg squats, single-leg countermovement jumps, bilateral countermovement jumps, and drop vertical jumps. Keypoint data were processed using direct kinematics as well as in OpenSim using a full-body musculoskeletal model and inverse kinematics. Kinematic methods were compared using statistical parametric mapping and between-day reliability was calculated using intraclass correlation coefficients, mean absolute error, and minimal detectable change. Keypoint-derived inverse kinematics resulted in significantly smaller hip flexion (range = -9 to -2°), hip abduction (range = -3 to -2°), knee flexion (range = -5° to -2°), and greater dorsiflexion angles (range = 6-15°) than direct kinematics. Both markerless kinematic methods had high between-day reliability (inverse kinematics ICC 95 %CI = 0.83-0.90; direct kinematics ICC 95 %CI = 0.80-0.93). For certain tasks and joints, keypoint-derived inverse kinematics resulted in greater reliability (up to 0.47 ICC) and smaller minimal detectable changes (up to 13°) than direct kinematics. Performing 2-4 trials was sufficient to maximize reliability for most tasks. A single markerless depth camera can reliably measure lower limb joint angles, and skeletal model-constrained inverse kinematics improves lower limb joint angle reliability for certain tasks and joints.

Hip contact forces can be predicted with a neural network using only synthesised key points and electromyography in people with hip osteoarthritis

OBJECTIVE

To develop and validate a neural network to estimate hip contact forces (HCF), and lower body kinematics and kinetics during walking in individuals with hip osteoarthritis (OA) using synthesised anatomical key points and electromyography. To assess the capability of the neural network to detect directional changes in HCF resulting from prescribed gait modifications.

DESIGN

A calibrated electromyography-informed neuromusculoskeletal model was used to compute lower body joint angles, moments, and HCF for 17 participants with mild-to-moderate hip OA. Anatomical key points (e.g., joint centres) were synthesised from marker trajectories and augmented with bias and noise expected from computer vision-based pose estimation systems. Temporal convolutional and long short-term memory neural networks (NN) were trained using leave-one-subject-out validation to predict neuromusculoskeletal modelling outputs from the synthesised key points and measured electromyography data from 5 hip-spanning muscles.

RESULTS

HCF was predicted with an average error of 13.4 ± 7.1% of peak force. Joint angles and moments were predicted with an average root-mean-square-error of 5.3 degrees and 0.10 Nm/kg, respectively. The NN could detect changes in peak HCF that occur due to gait modifications with good agreement with neuromusculoskeletal modelling (r2 = 0.72) and a minimum detectable change of 9.5%.

CONCLUSION

The developed neural network predicted HCF and lower body joint angles and moments in individuals with hip OA using noisy synthesised key point locations with acceptable errors. Changes in HCF magnitude due to gait modifications were predicted with high accuracy. These findings have important implications for implementation of load-modification based gait retraining interventions for people with hip OA in a natural environment (i.e., home, clinic).

Biomechanical Posture Analysis in Healthy Adults with Machine Learning: Applicability and Reliability

Posture analysis is important in musculoskeletal disorder prevention but relies on subjective assessment. This study investigates the applicability and reliability of a machine learning (ML) pose estimation model for the human posture assessment, while also exploring the underlying structure of the data through principal component and cluster analyses. A cohort of 200 healthy individuals with a mean age of 24.4 ± 4.2 years was photographed from the frontal, dorsal, and lateral views. We used Student's t-test and Cohen's effect size (d) to identify gender-specific postural differences and used the Intraclass Correlation Coefficient (ICC) to assess the reliability of this method. Our findings demonstrate distinct sex differences in shoulder adduction angle (men: 16.1° ± 1.9°, women: 14.1° ± 1.5°, d = 1.14) and hip adduction angle (men: 9.9° ± 2.2°, women: 6.7° ± 1.5°, d = 1.67), with no significant differences in horizontal inclinations. ICC analysis, with the highest value of 0.95, confirms the reliability of the approach. Principal component and clustering analyses revealed potential new patterns in postural analysis such as significant differences in shoulder-hip distance, highlighting the potential of unsupervised ML for objective posture analysis, offering a promising non-invasive method for rapid, reliable screening in physical therapy, ergonomics, and sports.

Agreement between a markerless and a marker-based motion capture systems for balance related quantities

Balance studies usually focus on quantities describing the global body motion. Assessing such quantities using classical marker-based approach can be tedious and modify the participant's behaviour. The recent development of markerless motion capture methods could bypass the issues related to the use of markers. This work compared dynamic balance related quantities obtained with markers and videos. Sixteen young healthy participants performed four different motor tasks: walking at self-selected speed, balance loss, walking on a narrow beam and countermovement jumps. Their movements were recorded simultaneously by marker-based and markerless motion capture systems. Videos were processed using a commercial markerless pose estimation software, Theia3D. The centre of mass position (CoM) was computed, and the associated extrapolated centre of mass position (XCoM) and whole-body angular momentum (WBAM) were derived. Bland-Altman analysis was performed and root mean square difference (RMSD) and coefficient of correlation were computed to compare the results obtained with marker-based and markerless methods. Bias remained of the magnitude of a few mm for CoM and XCoM positions, and RMSD of CoM and XCoM was around 1 cm. RMSD of the WBAM was less than 10 % of the total amplitude in any direction, and bias was less than 1 %. Results suggest that outcomes of balance studies will be similar whether marker-based or markerless motion capture system are used. Nevertheless, one should be careful when assessing dynamic movements such as jumping, as they displayed the biggest differences (both bias and RMSD), although it is unclear whether these differences are due to errors in markerless or marker-based motion capture system.

Evaluation of 3D Markerless Motion Capture System Accuracy during Skate Skiing on a Treadmill

In this study, we developed a deep learning-based 3D markerless motion capture system for skate skiing on a treadmill and evaluated its accuracy against marker-based motion capture during G1 and G3 skating techniques. Participants performed roller skiing trials on a skiing treadmill. Trials were recorded with two synchronized video cameras (100 Hz). We then trained a custom model using DeepLabCut, and the skiing movements were analyzed using both DeepLabCut-based markerless motion capture and marker-based motion capture systems. We statistically compared joint centers and joint vector angles between the methods. The results demonstrated a high level of agreement for joint vector angles, with mean differences ranging from -2.47° to 3.69°. For joint center positions and toe placements, mean differences ranged from 24.0 to 40.8 mm. This level of accuracy suggests that our markerless approach could be useful as a skiing coaching tool. The method presents interesting opportunities for capturing and extracting value from large amounts of data without the need for markers attached to the skier and expensive cameras.

Effective evaluation of HGcnMLP method for markerless 3D pose estimation of musculoskeletal diseases patients based on smartphone monocular video

Markerless pose estimation based on computer vision provides a simpler and cheaper alternative to human motion capture, with great potential for clinical diagnosis and remote rehabilitation assessment. Currently, the markerless 3D pose estimation is mainly based on multi-view technology, while the more promising single-view technology has defects such as low accuracy and reliability, which seriously limits clinical application. This study proposes a high-resolution graph convolutional multilayer perception (HGcnMLP) human 3D pose estimation framework for smartphone monocular videos and estimates 15 healthy adults and 12 patients with musculoskeletal disorders (sarcopenia and osteoarthritis) gait spatiotemporal, knee angle, and center-of-mass (COM) velocity parameters, etc., and compared with the VICON gold standard system. The results show that most of the calculated parameters have excellent reliability (VICON, ICC (2, k): 0.853-0.982; Phone, ICC (2, k): 0.839-0.975) and validity (Pearson r: 0.808-0.978, p< 0.05). In addition, the proposed system can better evaluate human gait balance ability, and the K-means++ clustering algorithm can successfully distinguish patients into different recovery level groups. This study verifies the potential of a single smartphone video for 3D human pose estimation for rehabilitation auxiliary diagnosis and balance level recognition, and is an effective attempt at the clinical application of emerging computer vision technology. In the future, it is hoped that the corresponding smartphone program will be developed to provide a low-cost, effective, and simple new tool for remote monitoring and rehabilitation assessment of patients.

Feasibility of OpenPose markerless motion analysis in a real athletics competition

This study tested the performance of OpenPose on footage collected by two cameras at 200 Hz from a real-life competitive setting by comparing it with manually analyzed data in SIMI motion. The same take-off recording from the men's Long Jump finals at the 2017 World Athletics Championships was used for both approaches (markerless and manual) to reconstruct the 3D coordinates from each of the camera's 2D coordinates. Joint angle and Centre of Mass (COM) variables during the final step and take-off phase of the jump were determined. Coefficients of Multiple Determinations (CMD) for joint angle waveforms showed large variation between athletes with the knee angle values typically being higher (take-off leg: 0.727 ± 0.242; swing leg: 0.729 ± 0.190) than those for hip (take-off leg: 0.388 ± 0.193; swing leg: 0.370 ± 0.227) and ankle angle (take-off leg: 0.247 ± 0.172; swing leg: 0.155 ± 0.228). COM data also showed considerable variation between athletes and parameters, with position (0.600 ± 0.322) and projection angle (0.658 ± 0.273) waveforms generally showing better agreement than COM velocity (0.217 ± 0.241). Agreement for discrete data was generally poor with high random error for joint kinematics and COM parameters at take-off and an average ICC across variables of 0.17. The poor agreement statistics and a range of unrealistic values returned by the pose estimation underline that OpenPose is not suitable for in-competition performance analysis in events such as the long jump, something that manual analysis still achieves with high levels of accuracy and reliability.

Walking speeds are lower for short distance and turning locomotion: Experiments and modeling in low-cost prosthesis users

Preferred walking speed is a widely-used performance measure for people with mobility issues, but is usually measured in straight line walking for fixed distances or durations, and without explicitly accounting for turning. However, daily walking involves walking for bouts of different distances and walking with turning, with prior studies showing that short bouts with at most 10 steps could be 40% of all bouts and turning steps could be 8-50% of all steps. Here, we studied walking in a straight line for short distances (4 m to 23 m) and walking in circles (1 m to 3 m turning radii) in people with transtibial amputation or transfemoral amputation using a passive ankle-foot prosthesis (Jaipur Foot). We found that the study participants' preferred walking speeds are lower for shorter straight-line walking distances and lower for circles of smaller radii, which is analogous to earlier results in subjects without amputation. Using inverse optimization, we estimated the cost of changing speeds and turning such that the observed preferred walking speeds in our experiments minimizes the total cost of walking. The inferred costs of changing speeds and turning were larger for subjects with amputation compared to subjects without amputation in a previous study, specifically, being 4x to 8x larger for the turning cost and being highest for subjects with transfemoral amputation. Such high costs inferred by inverse optimization could potentially include non-energetic costs such as due to joint or interfacial stress or stability concerns, as inverse optimization cannot distinguish such terms from true metabolic cost. These experimental findings and models capturing the experimental trends could inform prosthesis design and rehabilitation therapy to better assist changing speeds and turning tasks. Further, measuring the preferred speed for a range of distances and radii could be a more comprehensive subject-specific measure of walking performance than commonly used straight line walking metrics.

Quantification of soft tissue artifacts using CT registration and subject-specific multibody modeling

The potential use of gait analysis for quantitative preoperative planning in total hip arthroplasty (THA) has previously been demonstrated. However, the joint kinematic data measured through this process tend to be unreliable for surgical planning due to distortions caused by soft tissue artifacts (STAs). In this study, we developed a novel motion capture framework by combining computed tomography (CT)-based postural calibration and subject-specific multibody dynamics modeling to prevent the effect of STAs in measuring hip kinematics. Three subjects with femoroacetabular impingement syndrome were recruited, and CT data for each patient were collected by attaching marker clusters near the hip. A subject-specific multibody hip joint model was developed based on reconstructed CT data. Spring-dashpot network calculations were performed to minimize the distance between the anatomical landmark and its corresponding infrared reflective marker. The STAs of the thigh was described as six degrees of freedom viscoelastic bushing elements, and their parameter values were identified via smooth orthogonal decomposition. Least squares optimization was used to modify the pelvic rotations to compensate for the rigid components of STAs. The results showed that CT-assisted motion tracking enabled the successful identification of STA influences in gait and squat positions. Furthermore, STA effects were found to alter maximal pelvis tilt and hip rotations during a squat. Compared to other techniques, such as dual fluoroscopic imaging, the adopted framework does not require additional medical imaging for patients undergoing robot-assisted THA surgery and is thus a practical way of evaluating hip joint kinematics for preoperative surgical planning.

Examination of 2D frontal and sagittal markerless motion capture: Implications for markerless applications

This study examined if occluded joint locations, obtained from 2D markerless motion capture (single camera view), produced 2D joint angles with reduced agreement compared to visible joints, and if 2D frontal plane joint angles were usable for practical applications. Fifteen healthy participants performed over-ground walking whilst recorded by fifteen marker-based cameras and two machine vision cameras (frontal and sagittal plane). Repeated measures Bland-Altman analysis illustrated that markerless standard deviation of bias and limits of agreement for the occluded-side hip and knee joint angles in the sagittal plane were double that of the camera-side (visible) hip and knee. Camera-side sagittal plane knee and hip angles were near or within marker-based error values previously observed. While frontal plane limits of agreement accounted for 35-46% of total range of motion at the hip and knee, Bland-Altman bias and limits of agreement (-4.6-1.6 ± 3.7-4.2˚) were actually similar to previously reported marker-based error values. This was not true for the ankle, where the limits of agreement (± 12˚) were still too high for practical applications. Our results add to previous literature, highlighting shortcomings of current pose estimation algorithms and labelled datasets. As such, this paper finishes by reviewing methods for creating anatomically accurate markerless training data using marker-based motion capture data.

Prediction of Knee Joint Compartmental Loading Maxima Utilizing Simple Subject Characteristics and Neural Networks

Joint loading may affect the development of osteoarthritis, but patient-specific load estimation requires cumbersome motion laboratory equipment. This reliance could be eliminated using artificial neural networks (ANNs) to predict loading from simple input predictors. We used subject-specific musculoskeletal simulations to estimate knee joint contact forces for 290 subjects during over 5000 stance phases of walking and then extracted compartmental and total joint loading maxima from the first and second peaks of the stance phase. We then trained ANN models to predict the loading maxima from predictors that can be measured without motion laboratory equipment (subject mass, height, age, gender, knee abduction-adduction angle, and walking speed). When compared to the target data, our trained models had NRMSEs (RMSEs normalized to the mean of the response variable) between 0.14 and 0.42 and Pearson correlation coefficients between 0.42 and 0.84. The loading maxima were predicted most accurately using the models trained with all predictors. We demonstrated that prediction of knee joint loading maxima may be possible without laboratory-measured motion capture data. This is a promising step in facilitating knee joint loading predictions in simple environments, such as a physician's appointment. In future, the rapid measurement and analysis setup could be utilized to guide patients in rehabilitation to slow development of joint disorders, such as osteoarthritis.

A comparison of three-dimensional kinematics between markerless and marker-based motion capture in overground gait

Vision-based methods using RGB inputs for human pose estimation have grown in recent years but have undergone limited testing in clinical and biomechanics research areas like gait analysis. The purpose of the present study was to compare lower extremity kinematics during overground gait between a traditional marker-based approach and a commercial multi-view markerless system in a sample of subjects including young adults, older adults, and adults diagnosed with Parkinson's disease. A convenience sample of 35 adults between the age of 18-85 years were included in this study, yielding a total of 114 trials and 228 gait cycles that were compared between systems. A total of 30 time normalized waveforms, including three-dimensional joint centers, segment angles, and joint angles were compared between systems using root mean-squared error (RMSE), range of motion difference (ΔROM), Pearson correlation coefficients (r), and interclass correlation coefficients (ICC). RMSEs for joint center positions were less than 28 mm in all joints with correlations indicating good to excellent agreement. RMSEs for segment and joint angles were in range of previous results, with highest agreement between systems in the sagittal plane. ΔROM differences were within reference values that characterize clinical groups like Parkinson's disease, stroke, or knee osteoarthritis. Further improvements in pelvis tracking, markerless keypoint model definitions, and standardization of comparison study protocols are needed. Nevertheless, markerless solutions seem promising toward unrestricted motion analysis in biomechanics research and clinical settings.

Optimizing Trajectories and Inverse Kinematics for Biomechanical Analysis of Markerless Motion Capture Data

Markerless motion capture using computer vision and human pose estimation (HPE) has the potential to expand access to precise movement analysis. This could greatly benefit rehabilitation by enabling more accurate tracking of outcomes and providing more sensitive tools for research. There are numerous steps between obtaining videos to extracting accurate biomechanical results and limited research to guide many critical design decisions in these pipelines. In this work, we analyze several of these steps including the algorithm used to detect keypoints and the keypoint set, the approach to reconstructing trajectories for biomechanical inverse kinematics and optimizing the IK process. Several features we find important are: 1) using a recent algorithm trained on many datasets that produces a dense set of biomechanically-motivated keypoints, 2) using an implicit representation to reconstruct smooth, anatomically constrained marker trajectories for IK, 3) iteratively optimizing the biomechanical model to match the dense markers, 4) appropriate regularization of the IK process. Our pipeline makes it easy to obtain accurate biomechanical estimates of movement in a rehabilitation hospital.

Prediction of gait kinetics using Markerless-driven musculoskeletal modeling

Video-based motion analysis systems are emerging in the biomechanics research community, yet there is limited exploration of kinetics prediction using RGB-markerless kinematics and musculoskeletal modeling. This project aimed to provide ground reaction force (GRF) and ground reaction moment (GRM) predictions during over-ground gait by introducing RGB-markerless kinematics into a musculoskeletal modeling framework. Full-body markerless kinematic inputs and musculoskeletal modeling were used to obtain GRF and GRM predictions which were compared to measured force plate values. The markerless-driven predictions yielded average root mean-squared error (RMSE) in the stance phase of 0.035 ± 0.009 N∙BW-1, 0.070 ± 0.014 N∙BW-1, and 0.155 ± 0.041 N∙BW-1 in the mediolateral (ML), anteroposterior (AP), and vertical (V) GRFs. This was accompanied by moderate to high correlations and interclass correlation coefficients (ICC) indicating moderate to good agreement between measured and predicted values (95% Confidence Inervals: ML = [0.479, 0.717], AP = [0.714, 0.856], V = [0.803, 0.905]). For ground reaction moments (GRM), average RMSE was 0.029 ± 0.013 Nm∙BWH-1, 0.014 ± 0.005 Nm∙BWH-1, and 0.005 ± 0.002 Nm∙BWH-1 in the sagittal, frontal, and transverse planes. Pearson correlations and ICCs indicated poor agreement between systems for GRMs (95% Confidence Intervals: Sagittal = [0.314, 0.608], Frontal = [0.006, 0.373], Transverse = [0.269, 0.570]). Currently, RMSE is larger than target thresholds set from studies using Kinect, inertial, or marker-based kinematic drivers; but methodological considerations highlighted in this work may help guide follow-up iterations. At this point, further use in research or clinical practice is cautioned until methodological considerations are addressed, although results are promising at this point.

Comparison of kinematics between Theia markerless and conventional marker-based gait analysis in clinical patients

BACKGROUND

Markerless motion capture systems have the potential to make clinical gait analysis more efficient and convenient. Theia3D is a commercially available markerless system that may serve as an alternative to traditional gait analysis for clinical gait laboratories.

RESEARCH QUESTION

What is the concurrent validity of markerless gait analysis using Theia3D compared to traditional marker-based gait analysis in pediatric clinical gait patients?

METHODS

Thirty-six patients (20 male, age 2-25 years) with a range of diagnoses underwent clinical gait analysis with data being captured concurrently by a traditional marker-based motion capture system (Vicon Nexus) and a commercial markerless system (Theia3D). Multiple left strides were averaged for each subject, and the difference in kinematics (Theia - Vicon) was calculated over the gait cycle and evaluated using root mean square difference (RMSD), mean difference, and RMSD after subtracting the mean value across the gait cycle (RMSD_offset). Sub-analysis was performed for 25 patients with foot deformities, 9 wearing ankle-foot orthoses, and 6 walking with assistance (cane, crutches, walker, or handheld).

RESULTS

Kinematics showed similar patterns between the marker-based and markerless systems. RMSD was < 6° except for pelvic tilt, hip flexion, ankle inversion, foot progression, and transverse plane rotation of the hip, knee, and ankle. These measures mainly differed due to an offset between the curves. After adjusting for offsets, all RMSD_offset were < 6°. RMSD was larger for patients with foot deformities, wearing orthoses, or using assistive devices, but all RMSD_offset were still < 8°. In some cases, however, the markerless system had greater trial-to-trial variability, showed a larger knee varus "bump" in swing, or failed to track the subject.

SIGNIFICANCE

This study provides preliminary evidence of concurrent validity of Theia3D for pediatric patients with abnormal gait. However, some questions remain regarding identification of the knee axis and for patients with foot deformity or assistive devices.

Monocular 3D Human Pose Markerless Systems for Gait Assessment

Gait analysis plays an important role in the fields of healthcare and sports sciences. Conventional gait analysis relies on costly equipment such as optical motion capture cameras and wearable sensors, some of which require trained assessors for data collection and processing. With the recent developments in computer vision and deep neural networks, using monocular RGB cameras for 3D human pose estimation has shown tremendous promise as a cost-effective and efficient solution for clinical gait analysis. In this paper, a markerless human pose technique is developed using motion captured by a consumer monocular camera (800 × 600 pixels and 30 FPS) for clinical gait analysis. The experimental results have shown that the proposed post-processing algorithm significantly improved the original human pose detection model (BlazePose)'s prediction performance compared to the gold-standard gait signals by 10.7% using the MoVi dataset. In addition, the predicted T² score has an excellent correlation with ground truth (r = 0.99 and y = 0.94x + 0.01 regression line), which supports that our approach can be a potential alternative to the conventional marker-based solution to assist the clinical gait assessment.

Computer-assisted approaches for measuring, segmenting, and analyzing functional upper extremity movement: a narrative review of the current state, limitations, and future directions

The analysis of functional upper extremity (UE) movement kinematics has implications across domains such as rehabilitation and evaluating job-related skills. Using movement kinematics to quantify movement quality and skill is a promising area of research but is currently not being used widely due to issues associated with cost and the need for further methodological validation. Recent developments by computationally-oriented research communities have resulted in potentially useful methods for evaluating UE function that may make kinematic analyses easier to perform, generally more accessible, and provide more objective information about movement quality, the importance of which has been highlighted during the COVID-19 pandemic. This narrative review provides an interdisciplinary perspective on the current state of computer-assisted methods for analyzing UE kinematics with a specific focus on how to make kinematic analyses more accessible to domain experts. We find that a variety of methods exist to more easily measure and segment functional UE movement, with a subset of those methods being validated for specific applications. Future directions include developing more robust methods for measurement and segmentation, validating these methods in conjunction with proposed kinematic outcome measures, and studying how to integrate kinematic analyses into domain expert workflows in a way that improves outcomes.

Towards Single Camera Human 3D-Kinematics

Markerless estimation of 3D Kinematics has the great potential to clinically diagnose and monitor movement disorders without referrals to expensive motion capture labs; however, current approaches are limited by performing multiple de-coupled steps to estimate the kinematics of a person from videos. Most current techniques work in a multi-step approach by first detecting the pose of the body and then fitting a musculoskeletal model to the data for accurate kinematic estimation. Errors in training data of the pose detection algorithms, model scaling, as well the requirement of multiple cameras limit the use of these techniques in a clinical setting. Our goal is to pave the way toward fast, easily applicable and accurate 3D kinematic estimation. To this end, we propose a novel approach for direct 3D human kinematic estimation D3KE from videos using deep neural networks. Our experiments demonstrate that the proposed end-to-end training is robust and outperforms 2D and 3D markerless motion capture based kinematic estimation pipelines in terms of joint angles error by a large margin (35% from 5.44 to 3.54 degrees). We show that D3KE is superior to the multi-step approach and can run at video framerate speeds. This technology shows the potential for clinical analysis from mobile devices in the future.

Estimating Ground Reaction Forces from Two-Dimensional Pose Data: A Biomechanics-Based Comparison of AlphaPose, BlazePose, and OpenPose

The adoption of computer vision pose estimation approaches, used to identify keypoint locations which are intended to reflect the necessary anatomical landmarks relied upon by biomechanists for musculoskeletal modelling, has gained increasing traction in recent years. This uptake has been further accelerated by keypoint use as inputs into machine learning models used to estimate biomechanical parameters such as ground reaction forces (GRFs) in the absence of instrumentation required for direct measurement. This study first aimed to investigate the keypoint detection rate of three open-source pose estimation models (AlphaPose, BlazePose, and OpenPose) across varying movements, camera views, and trial lengths. Second, this study aimed to assess the suitability and interchangeability of keypoints detected by each pose estimation model when used as inputs into machine learning models for the estimation of GRFs. The keypoint detection rate of BlazePose was distinctly lower than that of AlphaPose and OpenPose. All pose estimation models achieved a high keypoint detection rate at the centre of an image frame and a lower detection rate in the true sagittal plane camera field of view, compared with slightly anteriorly or posteriorly located quasi-sagittal plane camera views. The three-dimensional ground reaction force, instantaneous loading rate, and peak force for running could be estimated using the keypoints of all three pose estimation models. However, only AlphaPose and OpenPose keypoints could be used interchangeably with a machine learning model trained to estimate GRFs based on AlphaPose keypoints resulting in a high estimation accuracy when OpenPose keypoints were used as inputs and vice versa. The findings of this study highlight the need for further evaluation of computer vision-based pose estimation models for application in biomechanical human modelling, and the limitations of machine learning-based GRF estimation models that rely on 2D keypoints. This is of particular relevance given that machine learning models informing athlete monitoring guidelines are being developed for application related to athlete well-being.

Automatic extraction of upper-limb kinematic activity using deep learning-based markerless tracking during deep brain stimulation implantation for Parkinson's disease: A proof of concept study

Optimal placement of deep brain stimulation (DBS) therapy for treating movement disorders routinely relies on intraoperative motor testing for target determination. However, in current practice, motor testing relies on subjective interpretation and correlation of motor and neural information. Recent advances in computer vision could improve assessment accuracy. We describe our application of deep learning-based computer vision to conduct markerless tracking for measuring motor behaviors of patients undergoing DBS surgery for the treatment of Parkinson's disease. Video recordings were acquired during intraoperative kinematic testing (N = 5 patients), as part of standard of care for accurate implantation of the DBS electrode. Kinematic data were extracted from videos post-hoc using the Python-based computer vision suite DeepLabCut. Both manual and automated (80.00% accuracy) approaches were used to extract kinematic episodes from threshold derived kinematic fluctuations. Active motor epochs were compressed by modeling upper limb deflections with a parabolic fit. A semi-supervised classification model, support vector machine (SVM), trained on the parameters defined by the parabolic fit reliably predicted movement type. Across all cases, tracking was well calibrated (i.e., reprojection pixel errors 0.016-0.041; accuracies >95%). SVM predicted classification demonstrated high accuracy (85.70%) including for two common upper limb movements, arm chain pulls (92.30%) and hand clenches (76.20%), with accuracy validated using a leave-one-out process for each patient. These results demonstrate successful capture and categorization of motor behaviors critical for assessing the optimal brain target for DBS surgery. Conventional motor testing procedures have proven informative and contributory to targeting but have largely remained subjective and inaccessible to non-Western and rural DBS centers with limited resources. This approach could automate the process and improve accuracy for neuro-motor mapping, to improve surgical targeting, optimize DBS therapy, provide accessible avenues for neuro-motor mapping and DBS implantation, and advance our understanding of the function of different brain areas.

Accuracy of a markerless motion capture system in estimating upper extremity kinematics during boxing

Kinematic analysis of the upper extremity can be useful to assess the performance and skill levels of athletes during combat sports such as boxing. Although marker-based approach is widely used to obtain kinematic data, it is not suitable for “in the field” activities, i.e., when performed outside the laboratory environment. Markerless video-based systems along with deep learning-based pose estimation algorithms show great potential for estimating skeletal kinematics. However, applicability of these systems in assessing upper-limb kinematics remains unexplored in highly dynamic activities. This study aimed to assess kinematics of the upper limb estimated with a markerless motion capture system (2D video cameras along with commercially available pose estimation software Theia3D) compared to those measured with marker-based system during “in the field” boxing. A total of three elite boxers equipped with retroreflective markers were instructed to perform specific sequences of shadow boxing trials. Their movements were simultaneously recorded with 12 optoelectronic and 10 video cameras, providing synchronized data to be processed further for comparison. Comparative assessment showed higher differences in 3D joint center positions at the elbow (more than 3 cm) compared to the shoulder and wrist (&lt;2.5 cm). In the case of joint angles, relatively weaker agreement was observed along internal/external rotation. The shoulder joint revealed better performance across all the joints. Segment velocities displayed good-to-excellent agreement across all the segments. Overall, segment velocities exhibited better performance compared to joint angles. The findings indicate that, given the practicality of markerless motion capture system, it can be a promising alternative to analyze sports-performance.

A semi-automatic toolbox for markerless effective semantic feature extraction

VisionTool is an open-source python toolbox for semantic features extraction, capable to provide accurate features detectors for different applications, including motion analysis, markerless pose estimation, face recognition and biological cell tracking. VisionTool leverages transfer-learning with a large variety of deep neural networks allowing high-accuracy features detection with few training data. The toolbox offers a friendly graphical user interface, efficiently guiding the user through the entire process of features extraction. To facilitate broad usage and scientific community contribution, the code and a user guide are available at https://github.com/Malga-Vision/VisionTool.git.

Alzheimer's Disease Detection Using Comprehensive Analysis of Timed Up and Go Test via Kinect V.2 Camera and Machine Learning

Alzheimer's disease (AD) is a progressive neurodegenerative disease affecting cognitive and functional abilities. However, many patients presume lower cognitive or functional abilities because of aging and do not undergo clinical assessments until the symptoms become too advanced. Developing a low-cost and easy-to-use AD detection tool, which can be used in any clinical or non-clinical setting, can enable widespread AD assessments and diagnosis. This paper investigated the feasibility of developing such a tool to detect AD vs. healthy control (HC) from a simple balance and walking assessment called the Timed Up and Go (TUG) test. We collected joint position data of 47 HC and 38 AD subjects as they performed TUG in front of a Kinect V.2 camera. Our signal processing and statistical analyses provided a comprehensive analysis of balance and gait with 12 significant features for discriminating AD from HC after adjusting for age and the Geriatric Depression Scale. Using these features and a support vector machine classifier, our model classified the two groups with an average accuracy of 97.75% and an F-score of 97.67% for five-fold cross-validation and 98.68% and 98.67% for leave-one-subject out cross-validation. These results demonstrate the potential of our approach as a new quantitative complementary tool for detecting AD among older adults. Our work is novel as it presents the first application of Kinect V.2 camera and machine learning to provide a comprehensive and quantitative analysis of the TUG test to detect AD patients from HC. This study supports the feasibility of developing a low-cost and convenient AD assessment tool that can be used during routine checkups or even at home; however, future investigations could confirm its clinical diagnostic value in a larger cohort.

Using a Video Device and a Deep Learning-Based Pose Estimator to Assess Gait Impairment in Neurodegenerative Related Disorders: A Pilot Study

As the world’s population is living longer, age-related neurodegenerative diseases are becoming a more significant global issue. Neurodegenerative diseases cause worsening motor, cognitive and autonomic dysfunction over time and reduce functional abilities required for daily living. Compromised motor performance is one of the first and most evident manifestations. In the case of Parkinson’s disease, these impairments are currently evaluated by experts through the use of rating scales. Although this method is widely used by experts worldwide, it includes subjective and error-prone motor examinations that also fail in the characterization of symptoms’ fluctuations. The aim of this study is to evaluate whether artificial intelligence techniques can be used to objectively assess gait impairment in subjects with Parkinson’s disease. This paper presents the results of a cohort of ten subjects, five with a Parkinson’s disease diagnosis at different degrees of severity. We experimentally demonstrate good effectiveness of the proposed system in extracting the main features concerning people’s gait during the standard tests that clinicians use to assess the burden of disease. This system can offer neurologists, through accurate and objective data, a second opinion or a suggestion to reconsider score assignment. Thanks to its simplicity, tactful and non-intrusive approach and clinical-grade accuracy, it can be adopted on an ongoing basis even in environments where people usually live and work.

Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy

Two-dimensional deep-learning pose estimation algorithms can suffer from biases in joint pose localizations, which are reflected in triangulated coordinates, and then in 3D joint angle estimation. Pose2Sim, our robust markerless kinematics workflow, comes with a physically consistent OpenSim skeletal model, meant to mitigate these errors. Its accuracy was concurrently validated against a reference marker-based method. Lower-limb joint angles were estimated over three tasks (walking, running, and cycling) performed multiple times by one participant. When averaged over all joint angles, the coefficient of multiple correlation (CMC) remained above 0.9 in the sagittal plane, except for the hip in running, which suffered from a systematic 15° offset (CMC = 0.65), and for the ankle in cycling, which was partially occluded (CMC = 0.75). When averaged over all joint angles and all degrees of freedom, mean errors were 3.0°, 4.1°, and 4.0°, in walking, running, and cycling, respectively; and range of motion errors were 2.7°, 2.3°, and 4.3°, respectively. Given the magnitude of error traditionally reported in joint angles computed from a marker-based optoelectronic system, Pose2Sim is deemed accurate enough for the analysis of lower-body kinematics in walking, cycling, and running.

Backward Double Integration is a Valid Method to Calculate Maximal and Sub-Maximal Jump Height

The backward double integration method uses one force plate and could calculate jump height for countermovement jumping, squat jumping and drop jumping by analysing the landing phase instead of the push-off phase. This study compared the accuracy and variability of the forward double integration (FDI), backwards double integration (BDI) and Flight Time + Constant (FT+C) methods, against the marker-based rigid-body modelling method. It was hypothesised that the jump height calculated using the BDI method would be equivalent to the FDI method, while the FT+C method would have reduced accuracy and increased variability during sub-maximal jumping compared to maximal jumping. Twenty-four volunteers performed five maximal and five sub-maximal countermovement jumps, while force plate and motion capture data were collected. The BDI method calculated equivalent mean jump heights compared to the FDI method, with only slightly higher variability (2-3 mm), and therefore can be used in situations where FDI cannot be employed. The FT+C method was able to account for reduced heel-lift distance, despite employing an anthropometrically scaled heel-lift constant. However, across both sub-maximal and maximal jumping, it had increased variability (1.1 cm) compared to FDI and BDI and should not be used when alternate methods are available.

Markerless vs. Marker-Based Gait Analysis: A Proof of Concept Study

The analysis of human gait is an important tool in medicine and rehabilitation to evaluate the effects and the progression of neurological diseases resulting in neuromotor disorders. In these fields, the gold standard techniques adopted to perform gait analysis rely on motion capture systems and markers. However, these systems present drawbacks: they are expensive, time consuming and they can affect the naturalness of the motion. For these reasons, in the last few years, considerable effort has been spent to study and implement markerless systems based on videography for gait analysis. Unfortunately, only few studies quantitatively compare the differences between markerless and marker-based systems in 3D settings. This work presented a new RGB video-based markerless system leveraging computer vision and deep learning to perform 3D gait analysis. These results were compared with those obtained by a marker-based motion capture system. To this end, we acquired simultaneously with the two systems a multimodal dataset of 16 people repeatedly walking in an indoor environment. With the two methods we obtained similar spatio-temporal parameters. The joint angles were comparable, except for a slight underestimation of the maximum flexion for ankle and knee angles. Taking together these results highlighted the possibility to adopt markerless technique for gait analysis.

Applications and limitations of current markerless motion capture methods for clinical gait biomechanics

BACKGROUND

Markerless motion capture has the potential to perform movement analysis with reduced data collection and processing time compared to marker-based methods. This technology is now starting to be applied for clinical and rehabilitation applications and therefore it is crucial that users of these systems understand both their potential and limitations. This literature review aims to provide a comprehensive overview of the current state of markerless motion capture for both single camera and multi-camera systems. Additionally, this review explores how practical applications of markerless technology are being used in clinical and rehabilitation settings, and examines the future challenges and directions markerless research must explore to facilitate full integration of this technology within clinical biomechanics.

METHODOLOGY

A scoping review is needed to examine this emerging broad body of literature and determine where gaps in knowledge exist, this is key to developing motion capture methods that are cost effective and practically relevant to clinicians, coaches and researchers around the world. Literature searches were performed to examine studies that report accuracy of markerless motion capture methods, explore current practical applications of markerless motion capture methods in clinical biomechanics and identify gaps in our knowledge that are relevant to future developments in this area.

RESULTS

Markerless methods increase motion capture data versatility, enabling datasets to be re-analyzed using updated pose estimation algorithms and may even provide clinicians with the capability to collect data while patients are wearing normal clothing. While markerless temporospatial measures generally appear to be equivalent to marker-based motion capture, joint center locations and joint angles are not yet sufficiently accurate for clinical applications. Pose estimation algorithms are approaching similar error rates of marker-based motion capture, however, without comparison to a gold standard, such as bi-planar videoradiography, the true accuracy of markerless systems remains unknown.

CONCLUSIONS

Current open-source pose estimation algorithms were never designed for biomechanical applications, therefore, datasets on which they have been trained are inconsistently and inaccurately labelled. Improvements to labelling of open-source training data, as well as assessment of markerless accuracy against gold standard methods will be vital next steps in the development of this technology.

A SWOT Analysis of Portable and Low-Cost Markerless Motion Capture Systems to Assess Lower-Limb Musculoskeletal Kinematics in Sport

Markerless motion capture systems are promising for the assessment of movement in more real world research and clinical settings. While the technology has come a long way in the last 20 years, it is important for researchers and clinicians to understand the capacities and considerations for implementing these types of systems. The current review provides a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis related to the successful adoption of markerless motion capture technology for the assessment of lower-limb musculoskeletal kinematics in sport medicine and performance settings. 31 articles met the a priori inclusion criteria of this analysis. Findings from the analysis indicate that the improving accuracy of these systems via the refinement of machine learning algorithms, combined with their cost efficacy and the enhanced ecological validity outweighs the current weaknesses and threats. Further, the analysis makes clear that there is a need for multidisciplinary collaboration between sport scientists and computer vision scientists to develop accurate clinical and research applications that are specific to sport. While work remains to be done for broad application, markerless motion capture technology is currently on a positive trajectory and the data from this analysis provide an efficient roadmap toward widespread adoption.