Comparison of a Deep Learning-Based Pose Estimation System to Marker-Based and Kinect Systems in Exergaming for Balance Training

Evaluation of 3D Markerless Motion Capture System Accuracy during Skate Skiing on a Treadmill

In this study, we developed a deep learning-based 3D markerless motion capture system for skate skiing on a treadmill and evaluated its accuracy against marker-based motion capture during G1 and G3 skating techniques. Participants performed roller skiing trials on a skiing treadmill. Trials were recorded with two synchronized video cameras (100 Hz). We then trained a custom model using DeepLabCut, and the skiing movements were analyzed using both DeepLabCut-based markerless motion capture and marker-based motion capture systems. We statistically compared joint centers and joint vector angles between the methods. The results demonstrated a high level of agreement for joint vector angles, with mean differences ranging from -2.47° to 3.69°. For joint center positions and toe placements, mean differences ranged from 24.0 to 40.8 mm. This level of accuracy suggests that our markerless approach could be useful as a skiing coaching tool. The method presents interesting opportunities for capturing and extracting value from large amounts of data without the need for markers attached to the skier and expensive cameras.

A comparison of machine learning methods for quantifying self-grooming behavior in mice

Background

As machine learning technology continues to advance and the need for standardized behavioral quantification grows, commercial and open-source automated behavioral analysis tools are gaining prominence in behavioral neuroscience. We present a comparative analysis of three behavioral analysis pipelines-DeepLabCut (DLC) and Simple Behavioral Analysis (SimBA), HomeCageScan (HCS), and manual scoring-in measuring repetitive self-grooming among mice.

Methods

Grooming behavior of mice was recorded at baseline and after water spray or restraint treatments. Videos were processed and analyzed in parallel using 3 methods (DLC/SimBA, HCS, and manual scoring), quantifying both total number of grooming bouts and total grooming duration.

Results

Both treatment conditions (water spray and restraint) resulted in significant elevation in both total grooming duration and number of grooming bouts. HCS measures of grooming duration were significantly elevated relative to those derived from manual scoring: specifically, HCS tended to overestimate duration at low levels of grooming. DLC/SimBA duration measurements were not significantly different than those derived from manual scoring. However, both SimBA and HCS measures of the number of grooming bouts were significantly different than those derived from manual scoring; the magnitude and direction of the difference depended on treatment condition.

Conclusion

DLC/SimBA provides a high-throughput pipeline for quantifying grooming duration that correlates well with manual scoring. However, grooming bout data derived from both DLC/SimBA and HCS did not reliably estimate measures obtained via manual scoring.

Optical Measurements of the Skin Surface to Infer Bilateral Distinctions in Myofascial Tissue Stiffness

About half the U.S. adult population suffers from chronic neuromusculoskeletal pain. While its evaluation and treatment are widely addressed by therapies using soft tissue manipulation (STM), their efficacy is based upon clinician judgment. Robust biomarkers are needed to quantify the effects of STM on patient outcomes. Among noninvasive methods to quantify the mechanics of myofascial tissue, most are limited to small (<10 mm2), localized regions of interest. In contrast, we develop an approach to optically simultaneously measure a larger (~100 cm2) field of deformation at the skin surface. Biomarkers based on skin lateral mobility are derived to infer distinctions in myofascial tissue stiffness. In specific, three cameras track ink speckles whose fields of deformation and stretch are resolved with digital image correlation. Their ability to differentiate bilateral distinctions of the cervicothoracic region is evaluated with four participants, as a licensed clinician performs STM. The results indicate that the optically derived surface biomarkers can differentiate bilateral differences in skin mobility, with trend directions within a participant similar to measurements with an instrumented force probe. These findings preliminarily suggest skin surface measurements are capable of inferring underlying myofascial tissue stiffness, although further confirmation will require a larger, more diverse group of participants.

MTGEA: A Multimodal Two-Stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment

Because of societal changes, human activity recognition, part of home care systems, has become increasingly important. Camera-based recognition is mainstream but has privacy concerns and is less accurate under dim lighting. In contrast, radar sensors do not record sensitive information, avoid the invasion of privacy, and work in poor lighting. However, the collected data are often sparse. To address this issue, we propose a novel Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA), which improves recognition accuracy through accurate skeletal features from Kinect models. We first collected two datasets using the mmWave radar and Kinect v4 sensors. Then, we used zero-padding, Gaussian Noise (GN), and Agglomerative Hierarchical Clustering (AHC) to increase the number of collected point clouds to 25 per frame to match the skeleton data. Second, we used Spatial Temporal Graph Convolutional Network (ST-GCN) architecture to acquire multimodal representations in the spatio-temporal domain focusing on skeletal features. Finally, we implemented an attention mechanism aligning the two multimodal features to capture the correlation between point clouds and skeleton data. The resulting model was evaluated empirically on human activity data and shown to improve human activity recognition with radar data only. All datasets and codes are available in our GitHub.

Effect of Flat Running Shoes on Hip Kinematics in Male Recreational Runners

Patellofemoral joint pain and iliotibial band syndrome are very common running-related injuries. Excessive contralateral pelvic drop, hip adduction, and hip internal rotation have been suggested to be associated with the two injuries. The purpose of this repeated measures and the cross-sectional study was to investigate the effect of flat running shoes on these kinematic variables compared with that of conventional running shoes with a 10 mm drop. Eighteen male recreational runners were recruited to run in flat shoes and conventional shoes with a 10 mm drop, in random order. Impact force data and lower extremity kinematics were synchronously obtained using two Kistler force plates and eight motion infrared cameras, whereas differences in the impact force and hip kinematics were compared using statistical parametric mapping. Regarding hip kinematics, the hip flexion (p = 0.004) and adduction angles (p = 0.004) decreased significantly at 30-70% and 62-85% of the stance phase, respectively, while wearing flat running shoes; the contralateral pelvic drop angle (p = 0.001) decreased significantly at 31-75% of the stance phase while wearing flat running shoes. The knee internal rotation angle (p = 0.035) decreased significantly at 8-17% of the stance phase while wearing flat running shoes compared with conventional running shoes. Given that these kinematic variables are associated with patellofemoral joint pain and iliotibial band syndrome, flat running shoes may have potential benefits for the prevention or treatment of knee injuries.

Automatic extraction of upper-limb kinematic activity using deep learning-based markerless tracking during deep brain stimulation implantation for Parkinson's disease: A proof of concept study

Optimal placement of deep brain stimulation (DBS) therapy for treating movement disorders routinely relies on intraoperative motor testing for target determination. However, in current practice, motor testing relies on subjective interpretation and correlation of motor and neural information. Recent advances in computer vision could improve assessment accuracy. We describe our application of deep learning-based computer vision to conduct markerless tracking for measuring motor behaviors of patients undergoing DBS surgery for the treatment of Parkinson's disease. Video recordings were acquired during intraoperative kinematic testing (N = 5 patients), as part of standard of care for accurate implantation of the DBS electrode. Kinematic data were extracted from videos post-hoc using the Python-based computer vision suite DeepLabCut. Both manual and automated (80.00% accuracy) approaches were used to extract kinematic episodes from threshold derived kinematic fluctuations. Active motor epochs were compressed by modeling upper limb deflections with a parabolic fit. A semi-supervised classification model, support vector machine (SVM), trained on the parameters defined by the parabolic fit reliably predicted movement type. Across all cases, tracking was well calibrated (i.e., reprojection pixel errors 0.016-0.041; accuracies >95%). SVM predicted classification demonstrated high accuracy (85.70%) including for two common upper limb movements, arm chain pulls (92.30%) and hand clenches (76.20%), with accuracy validated using a leave-one-out process for each patient. These results demonstrate successful capture and categorization of motor behaviors critical for assessing the optimal brain target for DBS surgery. Conventional motor testing procedures have proven informative and contributory to targeting but have largely remained subjective and inaccessible to non-Western and rural DBS centers with limited resources. This approach could automate the process and improve accuracy for neuro-motor mapping, to improve surgical targeting, optimize DBS therapy, provide accessible avenues for neuro-motor mapping and DBS implantation, and advance our understanding of the function of different brain areas.

Programs Using Stimulation-Regulating Technologies to Promote Physical Activity in People With Intellectual and Multiple Disabilities: Scoping Review

Background

People with intellectual and multiple disabilities tend to engage in very low levels of physical activity.

Objective

This review paper aims to provide a comprehensive picture of intervention programs using stimulation-regulating technologies to promote forms of physical activity in people with intellectual and multiple disabilities.

Methods

Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist, a scoping review was conducted to identify and provide a synthesis of eligible studies published in English between 2010 and 2021. Studies were identified by searching PubMed, Web of Science, PsycINFO, ERIC, and CINAHL as well as by using Google Scholar and manual searches. Studies were included if they involved individuals with intellectual or multiple disabilities, used stimulation-regulating technology systems to help participants engage in physical activity, and reported data on the impact of the intervention.

Results

A total of 42 studies met the inclusion criteria. These studies were divided into 2 groups based on whether they pursued the increase in physical activity through technology-aided delivery of brief periods of preferred stimulation contingent on specific responses or the use of video games (exergames) and related auditory and visual stimulation. Subsequently, a narrative synthesis of the studies was provided.

Conclusions

The evidence reported by the 2 groups of studies is encouraging. However, further research is needed to compare the overall applicability and impact of the intervention strategies proposed by these groups of studies.

Applications and limitations of current markerless motion capture methods for clinical gait biomechanics

BACKGROUND

Markerless motion capture has the potential to perform movement analysis with reduced data collection and processing time compared to marker-based methods. This technology is now starting to be applied for clinical and rehabilitation applications and therefore it is crucial that users of these systems understand both their potential and limitations. This literature review aims to provide a comprehensive overview of the current state of markerless motion capture for both single camera and multi-camera systems. Additionally, this review explores how practical applications of markerless technology are being used in clinical and rehabilitation settings, and examines the future challenges and directions markerless research must explore to facilitate full integration of this technology within clinical biomechanics.

METHODOLOGY

A scoping review is needed to examine this emerging broad body of literature and determine where gaps in knowledge exist, this is key to developing motion capture methods that are cost effective and practically relevant to clinicians, coaches and researchers around the world. Literature searches were performed to examine studies that report accuracy of markerless motion capture methods, explore current practical applications of markerless motion capture methods in clinical biomechanics and identify gaps in our knowledge that are relevant to future developments in this area.

RESULTS

Markerless methods increase motion capture data versatility, enabling datasets to be re-analyzed using updated pose estimation algorithms and may even provide clinicians with the capability to collect data while patients are wearing normal clothing. While markerless temporospatial measures generally appear to be equivalent to marker-based motion capture, joint center locations and joint angles are not yet sufficiently accurate for clinical applications. Pose estimation algorithms are approaching similar error rates of marker-based motion capture, however, without comparison to a gold standard, such as bi-planar videoradiography, the true accuracy of markerless systems remains unknown.

CONCLUSIONS

Current open-source pose estimation algorithms were never designed for biomechanical applications, therefore, datasets on which they have been trained are inconsistently and inaccurately labelled. Improvements to labelling of open-source training data, as well as assessment of markerless accuracy against gold standard methods will be vital next steps in the development of this technology.

A New Look at Infant Problem-Solving: Using DeepLabCut to Investigate Exploratory Problem-Solving Approaches

When confronted with novel problems, problem-solvers must decide whether to copy a modeled solution or to explore their own unique solutions. While past work has established that infants can learn to solve problems both through their own exploration and through imitation, little work has explored the factors that influence which of these approaches infants select to solve a given problem. Moreover, past work has treated imitation and exploration as qualitatively distinct, although these two possibilities may exist along a continuum. Here, we apply a program novel to developmental psychology (DeepLabCut) to archival data (Lucca et al., 2020) to investigate the influence of the effort and success of an adult's modeled solution, and infants' firsthand experience with failure, on infants' imitative versus exploratory problem-solving approaches. Our results reveal that tendencies toward exploration are relatively immune to the information from the adult model, but that exploration generally increased in response to firsthand experience with failure. In addition, we found that increases in maximum force and decreases in trying time were associated with greater exploration, and that exploration subsequently predicted problem-solving success on a new iteration of the task. Thus, our results demonstrate that infants increase exploration in response to failure and that exploration may operate in a larger motivational framework with force, trying time, and expectations of task success.

A Validation of Supervised Deep Learning for Gait Analysis in the Cat

Gait analysis in cats and other animals is generally performed with custom-made or commercially developed software to track reflective markers placed on bony landmarks. This often involves costly motion tracking systems. However, deep learning, and in particular DeepLabCut^TM (DLC), allows motion tracking without requiring placing reflective markers or an expensive system. The purpose of this study was to validate the accuracy of DLC for gait analysis in the adult cat by comparing results obtained with DLC and a custom-made software (Expresso) that has been used in several cat studies. Four intact adult cats performed tied-belt (both belts at same speed) and split-belt (belts operating at different speeds) locomotion at different speeds and left-right speed differences on a split-belt treadmill. We calculated several kinematic variables, such as step/stride lengths and joint angles from the estimates made by the two software and assessed the agreement between the two measurements using intraclass correlation coefficient or Lin's concordance correlation coefficient as well as Pearson's correlation coefficients. The results showed that DLC is at least as precise as Expresso with good to excellent agreement for all variables. Indeed, all 12 variables showed an agreement above 0.75, considered good, while nine showed an agreement above 0.9, considered excellent. Therefore, deep learning, specifically DLC, is valid for measuring kinematic variables during locomotion in cats, without requiring reflective markers and using a relatively low-cost system.

Closed-loop automated reaching apparatus (CLARA) for interrogating complex motor behaviors

Objective.Closed-loop neuromodulation technology is a rapidly expanding category of therapeutics for a broad range of indications. Development of these innovative neurological devices requires high-throughput systems for closed-loop stimulation of model organisms, while monitoring physiological signals and complex, naturalistic behaviors. To address this need, we developed CLARA, a closed-loop automated reaching apparatus.Approach.Using breakthroughs in computer vision, CLARA integrates fully-automated, markerless kinematic tracking of multiple features to classify animal behavior and precisely deliver neural stimulation based on behavioral outcomes. CLARA is compatible with advanced neurophysiological tools, enabling the testing of neurostimulation devices and identification of novel neurological biomarkers.Results.The CLARA system tracks unconstrained skilled reach behavior in 3D at 150 Hz without physical markers. The system fully automates trial initiation and pellet delivery and is capable of accurately delivering stimulation in response to trial outcome with short latency. Kinematic data from the CLARA system provided novel insights into the dynamics of reach consistency over the course of learning, suggesting that learning selectively improves reach failures but does not alter the kinematics of successful reaches. Additionally, using the closed-loop capabilities of CLARA, we demonstrate that vagus nerve stimulation (VNS) improves skilled reach performance and increases reach trajectory consistency in healthy animals.Significance.The CLARA system is the first mouse behavior apparatus that uses markerless pose tracking to provide real-time closed-loop stimulation in response to the outcome of an unconstrained motor task. Additionally, we demonstrate that the CLARA system was essential for our investigating the role of closed-loop VNS stimulation on motor performance in healthy animals. This approach has high translational relevance for developing neurostimulation technology based on complex human behavior.

Detection of Postural Control in Young and Elderly Adults Using Deep and Machine Learning Methods with Joint-Node Plots

Postural control decreases with aging. Thus, an efficient and accurate method of detecting postural control is needed. We enrolled 35 elderly adults (aged 82.06 ± 8.74 years) and 20 healthy young adults (aged 21.60 ± 0.60 years) who performed standing tasks for 40 s, performed six times. The coordinates of 15 joint nodes were captured using a Kinect device (30 Hz). We plotted joint positions into a single 2D figure (named a joint–node plot, JNP) once per second for up to 40 s. A total of 15 methods combining deep and machine learning for postural control classification were investigated. The accuracy, sensitivity, specificity, positive predicted value (PPV), negative predicted value (NPV), and kappa values of the selected methods were assessed. The highest PPV, NPV, accuracy, sensitivity, specificity, and kappa values were higher than 0.9 in validation testing. The presented method using JNPs demonstrated strong performance in detecting the postural control ability of young and elderly adults.