A Deep Neural Network-based method for estimation of 3D lifting motions

Detecting the symptoms of Parkinson's disease with non-standard video

BACKGROUND

Neurodegenerative diseases, such as Parkinson's disease (PD), necessitate frequent clinical visits and monitoring to identify changes in motor symptoms and provide appropriate care. By applying machine learning techniques to video data, automated video analysis has emerged as a promising approach to track and analyze motor symptoms, which could facilitate more timely intervention. However, existing solutions often rely on specialized equipment and recording procedures, which limits their usability in unstructured settings like the home. In this study, we developed a method to detect PD symptoms from unstructured videos of clinical assessments, without the need for specialized equipment or recording procedures.

METHODS

Twenty-eight individuals with Parkinson's disease completed a video-recorded motor examination that included the finger-to-nose and hand pronation-supination tasks. Clinical staff provided ground truth scores for the level of Parkinsonian symptoms present. For each video, we used a pre-existing model called PIXIE to measure the location of several joints on the person's body and quantify how they were moving. Features derived from the joint angles and trajectories, designed to be robust to recording angle, were then used to train two types of machine-learning classifiers (random forests and support vector machines) to detect the presence of PD symptoms.

RESULTS

The support vector machine trained on the finger-to-nose task had an F1 score of 0.93 while the random forest trained on the same task yielded an F1 score of 0.85. The support vector machine and random forest trained on the hand pronation-supination task had F1 scores of 0.20 and 0.33, respectively.

CONCLUSION

These results demonstrate the feasibility of developing video analysis tools to track motor symptoms across variable perspectives. These tools do not work equally well for all tasks, however. This technology has the potential to overcome barriers to access for many individuals with degenerative neurological diseases like PD, providing them with a more convenient and timely method to monitor symptom progression, without requiring a structured video recording procedure. Ultimately, more frequent and objective home assessments of motor function could enable more precise telehealth optimization of interventions to improve clinical outcomes inside and outside of the clinic.

Prediction of knee biomechanics with different tibial component malrotations after total knee arthroplasty: conventional machine learning vs. deep learning

The precise alignment of tibiofemoral components in total knee arthroplasty is a crucial factor in enhancing the longevity and functionality of the knee. However, it is a substantial challenge to quickly predict the biomechanical response to malrotation of tibiofemoral components after total knee arthroplasty using musculoskeletal multibody dynamics models. The objective of the present study was to conduct a comparative analysis between a deep learning method and four conventional machine learning methods for predicting knee biomechanics with different tibial component malrotation during a walking gait after total knee arthroplasty. First, the knee contact forces and kinematics with different tibial component malrotation in the range of ±5° in the three directions of anterior/posterior slope, internal/external rotation, and varus/valgus rotation during a walking gait after total knee arthroplasty were calculated based on the developed musculoskeletal multibody dynamics model. Subsequently, deep learning and four conventional machine learning methods were developed using the above 343 sets of biomechanical data as the dataset. Finally, the results predicted by the deep learning method were compared to the results predicted by four conventional machine learning methods. The findings indicated that the deep learning method was more accurate than four conventional machine learning methods in predicting knee contact forces and kinematics with different tibial component malrotation during a walking gait after total knee arthroplasty. The deep learning method developed in this study enabled quickly determine the biomechanical response with different tibial component malrotation during a walking gait after total knee arthroplasty. The proposed method offered surgeons and surgical robots the ability to establish a calibration safety zone, which was essential for achieving precise alignment in both preoperative surgical planning and intraoperative robotic-assisted surgical navigation.

Soft computing applications in the field of human factors and ergonomics: A review of the past decade of research

The main objectives of this study were to 1) review the literature on the applications of soft computing concepts to the field of human factors and ergonomics (HFE) between 2013 and 2022 and 2) highlight future developments and trends. Multiple soft computing methods and techniques have been investigated for their ability to address various applications in HFE effectively. These techniques include fuzzy logic, artificial neural networks, genetic algorithms, and their combinations. Applications of these methods in HFE have been highlighted in one hundred and four articles selected from 406 papers. The results of this study help address the challenges of complexity, vagueness, and imprecision in human factors and ergonomics research through the application of soft computing methodologies.

Motion Capture Technologies for Ergonomics: A Systematic Literature Review

Muscular skeletal disorder is a difficult challenge faced by the working population. Motion capture (MoCap) is used for recording the movement of people for clinical, ergonomic and rehabilitation solutions. However, knowledge barriers about these MoCap systems have made them difficult to use for many people. Despite this, no state-of-the-art literature review on MoCap systems for human clinical, rehabilitation and ergonomic analysis has been conducted. A medical diagnosis using AI applies machine learning algorithms and motion capture technologies to analyze patient data, enhancing diagnostic accuracy, enabling early disease detection and facilitating personalized treatment plans. It revolutionizes healthcare by harnessing the power of data-driven insights for improved patient outcomes and efficient clinical decision-making. The current review aimed to investigate: (i) the most used MoCap systems for clinical use, ergonomics and rehabilitation, (ii) their application and (iii) the target population. We used preferred reporting items for systematic reviews and meta-analysis guidelines for the review. Google Scholar, PubMed, Scopus and Web of Science were used to search for relevant published articles. The articles obtained were scrutinized by reading the abstracts and titles to determine their inclusion eligibility. Accordingly, articles with insufficient or irrelevant information were excluded from the screening. The search included studies published between 2013 and 2023 (including additional criteria). A total of 40 articles were eligible for review. The selected articles were further categorized in terms of the types of MoCap used, their application and the domain of the experiments. This review will serve as a guide for researchers and organizational management.

Assessment of a novel deep learning-based marker-less motion capture system for gait study

BACKGROUND

Marker-less systems based on digital video cameras and deep learning for gait analysis could have a deep impact in clinical routine. A recently developed system has shown promising results in terms of joint center position but has not been yet evaluated in terms of gait outcomes.

RESEARCH QUESTION

How does this novel marker-less system compare to a marker-based reference system in terms of clinically relevant gait parameters?

METHODS

The deep learning method behind the developed marker-less system was trained on a dedicated dataset consisting of forty-one asymptomatic and pathological subjects each performing ten walking trials. The system could estimate the three-dimensional position of seventeen joint centers or keypoints (e.g., neck, shoulders, hip, knee, and ankles). We evaluated the marker-less system against a marker-based system in terms of differences in joint position (Euclidean distance), detection of gait events (e.g., heel strike and toe-off), spatiotemporal parameters (e.g., step length, time), kinematic parameters (e.g., hip and knee extension-flexion), and inter-trial reliability for kinematic parameters.

RESULTS

The marker-less system was able to estimate the three-dimensional position of joint centers with a mean difference of 13.1 mm (SD = 10.2 mm). 99% of the estimated gait events were estimated within 10 ms of the corresponding reference values. Estimated spatiotemporal parameters showed zero bias. The mean and standard deviation of the differences of the estimated kinematic parameters varied by parameter (for example, the mean and standard deviation for knee extension flexion angle were -3.0° and 2.7°). Inter-trial reliability of the measured parameters was similar to that of the marker-based references.

SIGNIFICANCE

The developed marker-less system can measure the spatiotemporal parameters within the range of the minimum detectable changes obtained using the marker-based reference system. Moreover, except for hip extension flexion, the system showed promising results in terms of several kinematic parameters.

A Review of Finite Element Models of Ligaments in the Foot and Considerations for Practical Application

PURPOSE

Finite element (FE) modeling has been used as a research tool for investigating underlying ligaments biomechanics and orthopedic applications. However, FE models of the ligament in the foot have been developed with various configurations, mainly due to their complex 3D geometry, material properties, and boundary conditions. Therefore, the purpose of this review was to summarize the current state of finite element modeling approaches that have been used in the ?eld of ligament biomechanics, to discuss their applicability to foot ligament modeling in a practical setting, and also to acknowledge current limitations and challenges.

METHODS

A comprehensive literature search was performed. Each article was analyzed in terms of the methods used for: (a) ligament geometry, (b) material property, (c) boundary and loading condition related to its application, and (d) model verification and validation.

RESULTS

Of the reviewed studies, 80% of the studies used simplified representations of ligament geometry, the non-linear mechanical behavior of ligaments was taken into account in only 19.2% of the studies, 33% of included studies did not include any kind of validation of the FE model.

CONCLUSION

Further refinement in the functional modeling of ligaments, the micro-structure level characteristics, nonlinearity, and time-dependent response, may be warranted to ensure the predictive ability of the models.

Deep learning approach to estimate foot pressure distribution in walking with application for a cost-effective insole system

Background

Foot pressure distribution can be used as a quantitative parameter for evaluating anatomical deformity of the foot and for diagnosing and treating pathological gait, falling, and pressure sores in diabetes. The objective of this study was to propose a deep learning model that could predict pressure distribution of the whole foot based on information obtained from a small number of pressure sensors in an insole.

Methods

Twenty young and twenty older adults walked a straight pathway at a preferred speed with a Pedar-X system in anti-skid socks. A long short-term memory (LSTM) model was used to predict foot pressure distribution. Pressure values of nine major sensors and the remaining 90 sensors in a Pedar-X system were used as input and output for the model, respectively. The performance of the proposed LSTM structure was compared with that of a traditionally used adaptive neuro-fuzzy interference system (ANFIS). A low-cost insole system consisting of a small number of pressure sensors was fabricated. A gait experiment was additionally performed with five young and five older adults, excluding subjects who were used to construct models. The Pedar-X system placed parallelly on top of the insole prototype developed in this study was in anti-skid socks. Sensor values from a low-cost insole prototype were used as input of the LSTM model. The accuracy of the model was evaluated by applying a leave-one-out cross-validation.

Results

Correlation coefficient and relative root mean square error (RMSE) of the LSTM model were 0.98 (0.92 ~ 0.99) and 7.9 ± 2.3%, respectively, higher than those of the ANFIS model. Additionally, the usefulness of the proposed LSTM model for fabricating a low-cost insole prototype with a small number of sensors was confirmed, showing a correlation coefficient of 0.63 to 0.97 and a relative RMSE of 12.7 ± 7.4%.

Conclusions

This model can be used as an algorithm to develop a low-cost portable smart insole system to monitor age-related physiological and anatomical alterations in foot. This model has the potential to evaluate clinical rehabilitation status of patients with pathological gait, falling, and various foot pathologies when more data of patients with various diseases are accumulated for training.

Load Asymmetry Angle Estimation Using Multiple view Videos

A robust computer vision-based approach is developed to estimate the load asymmetry angle defined in the revised NIOSH lifting equation (RNLE). The angle of asymmetry enables the computation of a recommended weight limit for repetitive lifting operations in a workplace to prevent lower back injuries. An open-source package OpenPose is applied to estimate the 2D locations of skeletal joints of the worker from two synchronous videos. Combining these joint location estimates, a computer vision correspondence and depth estimation method is developed to estimate the 3D coordinates of skeletal joints during lifting. The angle of asymmetry is then deduced from a subset of these 3D positions. Error analysis reveals unreliable angle estimates due to occlusions of upper limbs. A robust angle estimation method that mitigates this challenge is developed. We propose a method to flag unreliable angle estimates based on the average confidence level of 2D joint estimates provided by OpenPose. An optimal threshold is derived that balances the percentage variance reduction of the estimation error and the percentage of angle estimates flagged. Tested with 360 lifting instances in a NIOSH-provided dataset, the standard deviation of angle estimation error is reduced from 10.13° to 4.99°. To realize this error variance reduction, 34% of estimated angles are flagged and require further validation.

Applying deep neural networks and inertial measurement unit in recognizing irregular walking differences in the real world

Falling injuries pose serious health risks to people of all ages, and knowing the extent of exposure to irregular surfaces will increase the ability to measure fall risk. Current gait analysis methods require overly complicated instrumentation and have not been tested for external factors such as walking surfaces that are encountered in the real-world, thus the results are difficult to extrapolate to real-world situations. Artificial intelligence approaches (in particular deep learning networks of varied architectures) to analyze data collected from wearable sensors were used to identify irregular surface exposure in a real-world setting. Thirty young adults wore six Inertial Measurement Unit (IMU) sensors placed on their body (right wrist, trunks at the L5/S1 level, left and right thigh, left and right shank) while walking over eight different surfaces commonly encountered in the living community as well as occupational settings. Three variations of deep learning models were trained to solve this walking surface recognition problem: 1) convolution neural network (CNN); 2) long short term memory (LSTM) network and 3) LSTM structure with an extra global pooling layer (Global-LSTM) which learns the coordination between different data streams (e.g. different channels of the same sensor as well as different sensors). Results indicated that all three deep learning models can recognize walking surfaces with above 0.90 accuracy, with the Global-LSTM yielding the best performance at 0.92 accuracy. In terms of individual sensors, the right thigh based Global-LSTM model reported the highest accuracy (0.90 accuracy). Results from this study provide further evidence that deep learning and wearable sensors can be utilized to recognize irregular walking surfaces induced motion alteration and applied to prevent falling injuries.

Simple benchmarking method for determining the accuracy of depth cameras in body landmark location estimation: Static upright posture as a measurement example

To evaluate the postures in ergonomics applications, studies have proposed the use of low-cost, marker-less, and portable depth camera-based motion tracking systems (DCMTSs) as a potential alternative to conventional marker-based motion tracking systems (MMTSs). However, a simple but systematic method for examining the estimation errors of various DCMTSs is lacking. This paper proposes a benchmarking method for assessing the estimation accuracy of depth cameras for full-body landmark location estimation. A novel alignment board was fabricated to align the coordinate systems of the DCMTSs and MMTSs. The data from an MMTS were used as a reference to quantify the error of using a DCMTS to identify target locations in a 3-D space. To demonstrate the proposed method, the full-body landmark location tracking errors were evaluated for a static upright posture using two different DCMTSs. For each landmark, we compared each DCMTS (Kinect system and RealSense system) with an MMTS by calculating the Euclidean distances between symmetrical landmarks. The evaluation trials were performed twice. The agreement between the tracking errors of the two evaluation trials was assessed using intraclass correlation coefficient (ICC). The results indicate that the proposed method can effectively assess the tracking performance of DCMTSs. The average errors (standard deviation) for the Kinect system and RealSense system were 2.80 (1.03) cm and 5.14 (1.49) cm, respectively. The highest average error values were observed in the depth orientation for both DCMTSs. The proposed method achieved high reliability with ICCs of 0.97 and 0.92 for the Kinect system and RealSense system, respectively.

Machine learning for rapid estimation of lower extremity muscle and joint loading during activities of daily living

Joint contact and muscle forces estimated with musculoskeletal modeling techniques offer useful metrics describing movement quality that benefit multiple research and clinical applications. The expensive processing of laboratory data associated with generating these outputs presents challenges to researchers and clinicians, including significant time and expertise requirements that limit the number of subjects typically evaluated. The objective of the current study was to develop and compare machine learning techniques for rapid, data-driven estimation of musculoskeletal metrics from derived gait lab data. OpenSim estimates of patient joint and muscle forces during activities of daily living were simulated using laboratory data from 70 total knee replacement patients and used to develop 4 different machine learning algorithms. Trained machine learning models predicted both trend and magnitude of estimated joint contact (mean correlation coefficients ranging from 0.93 to 0.94 during gait) and muscle forces (mean correlation coefficients ranging from 0.83 to 0.91 during gait) based on anthropometrics, ground reaction forces, and joint angle data. Patient mechanics were accurately predicted by recurrent neural networks, even after removing dependence on key subsets of predictor features. The ability to quickly estimate patient mechanics from derived measurements of movement has the potential to broaden the impact of musculoskeletal modeling by enabling faster assessment in both clinical and research settings.

Back loading estimation during team handling: Is the use of only motion data sufficient?

Analyzing back loading during team manual handling tasks requires the measurement of external contacts and is thus limited to standardized tasks. This paper evaluates the possibility of estimating L5/S1 joint moments based solely on motion data. Ten subjects constituted five two-person teams and handling tasks were analyzed with four different box configurations. Three prediction methods for estimating L5/S1 joint moments were evaluated by comparing them to a gold standard using force platforms: one used only motion data, another used motion data and the traction/compression force applied to the box and one used motion data and the ground reaction forces of one team member. The three prediction methods were based on a contact model with an optimization-based method. Using only motion data did not allow an accurate estimate due to the traction/compression force applied by each team member, which affected L5/S1 joint moments. Back loading can be estimated using motion data and the measurement of the traction/compression force with relatively small errors, comparable to the uncertainty levels reported in other studies. The traction/compression force can be obtained directly with a force measurement unit built into the object to be moved or indirectly by using force platforms on which one of the two handlers stands during the handling task. The use of the proposed prediction methods allows team manual handling tasks to be analyzed in various realistic contexts, with team members who have different anthropometric measurements and with different box characteristics.

Estimation of the ground reaction forces from a single video camera based on the spring-like center of mass dynamics of human walking

In clinical studies, the ground reaction forces (GRFs) during walking have found being highly useful. Therefore, the force sensing shoes with small sensors and estimation methods based on kinematics from motion capture systems or inertial measurement units were proposed. Recent studies demonstrated methods of extracting GRFs from whole-body joint kinematics, which requires a significant computational load. In this study, we propose a vertical and anterior-posterior GRFs estimation method using a single camera based on the dynamic relationship between the center of mass (CoM) and the GRFs in terms of spring mechanics. The estimation method consisted of two steps: the extraction of the vertical CoM from the video clip and the conversion of the CoM information into GRFs using a walking model. From the image of the greater trochanter that is positioned near the pelvic joint, the vertical CoM was extracted. This was done after removing the artifacts by pelvic rotation and postural change of lower limbs. The parameters of a compliant bipedal walking model were tuned to best match the CoM trajectory coupled with GRFs by spring mechanics. A video camera was used to record the walking trials of five healthy young participants from the side. The walking trials was conducted at three different speeds on the instrumented treadmill; each lasted one minute long. The GRF prediction errors were approximately 9-11%, with the best matching trials found to be at a self-selected gait speed. The prediction of anterior-posterior GRF components showed a more consistent match than the vertical GRF. The results demonstrated the possibility of marker-less kinetics prediction from video images incorporating the mechanical characteristics of the CoM.

MOPED25: A multimodal dataset of full-body pose and motion in occupational tasks

In recent years, there has been a trend of using images and deep neural network-based computer vision algorithms to perform postural evaluation in workplace safety and ergonomics community. The performance of the computer vision algorithms, however, heavily relies on the generalizability of the posture dataset that was used for algorithm training. Current open-access posture datasets from the computer vision community mainly focus on the pose and motion of daily activities and lack the context in workplaces. In this study, a new posture dataset named, MOPED25 (Multimodal Occupational Posture Dataset with 25 tasks) is presented. This dataset includes full-body kinematics data and the synchronized videos of 11 participants, performing commonly seen tasks at workplaces. All the data has been made publicly available online. This dataset can serve as a benchmark for developing more robust computer vision algorithms for postural evaluation at workplaces.

A computer vision approach for classifying isometric grip force exertion levels

Exposure to high and/or repetitive force exertions can lead to musculoskeletal injuries. However, measuring worker force exertion levels is challenging, and existing techniques can be intrusive, interfere with human-machine interface, and/or limited by subjectivity. In this work, computer vision techniques are developed to detect isometric grip exertions using facial videos and wearable photoplethysmogram. Eighteen participants (19-24 years) performed isometric grip exertions at varying levels of maximum voluntary contraction. Novel features that predict forces were identified and extracted from video and photoplethysmogram data. Two experiments with two (High/Low) and three (0%MVC/50%MVC/100%MVC) labels were performed to classify exertions. The Deep Neural Network classifier performed the best with 96% and 87% accuracy for two- and three-level classifications, respectively. This approach was robust to leave subjects out during cross-validation (86% accuracy when 3-subjects were left out) and robust to noise (i.e. 89% accuracy for correctly classifying talking activities as low force exertions). Practitioner summary: Forceful exertions are contributing factors to musculoskeletal injuries, yet it remains difficult to measure in work environments. This paper presents an approach to estimate force exertion levels, which is less distracting to workers, easier to implement by practitioners, and could potentially be used in a wide variety of workplaces. Abbreviations: MSD: musculoskeletal disorders; ACGIH: American Conference of Governmental Industrial Hygienists; HAL: hand activity level; MVC: maximum voluntary contraction; PPG: photoplethysmogram; DNN: deep neural networks; LOSO: leave-one-subject-out; ROC: receiver operating characteristic; AUC: area under curve.

Motion-based prediction of external forces and moments and back loading during manual material handling tasks

This paper evaluates a method for motion-based prediction of external forces and moments on manual material handling (MMH) tasks. From a set of hypothesized contact points between the subject and the environment (ground and load), external forces were calculated as the minimal forces at each contact point while ensuring the dynamics equilibrium. Ground reaction forces and moments (GRF&M) and load contact forces and moments (LCF&M) were computed from motion data alone. With an inverse dynamics method, the predicted data were then used to compute kinetic variables such as back loading. On a cohort of 65 subjects performing MMH tasks, the mean correlation coefficients between predicted and experimentally measured GRF for the vertical, antero-posterior and medio-lateral components were 0.91 (0.08), 0.95 (0.03) and 0.94 (0.08), respectively. The associated RMSE were 0.51 N/kg, 0.22 N/kg and 0.19 N/kg. The correlation coefficient between L5/S1 joint moments computed from predicted and measured data was 0.95 with a RMSE of 14 Nm for the flexion/extension component. In conclusion, this method allows the assessment of MMH tasks without force platforms, which increases the ecological aspect of the tasks studied and enables performance of dynamic analyses in real settings outside the laboratory.

Machine learning approach to predict center of pressure trajectories in a complete gait cycle: a feedforward neural network vs. LSTM network

Center of pressure (COP) trajectories of human can maintain regulation of forward progression and stability of lateral sway during walking. The insole pressure system can only detect COP trajectories of each foot during single stance. In this study, we developed artificial neural network models that could present COP trajectories in an integrated coordinate system during a complete gait cycle using pressure information of the insole system. A feed forward artificial neural network (FFANN) and a long short-term memory (LSTM) model were developed. For FFANN, among 198 pressure sensors from Pedar-X insoles, proper input variables were selected using sequential forward selection to reduce input dimension. The LSTM model used all 198 signals as inputs because of its self-learning characteristic. As results of cross-validation, the FFANN model showed correlation coefficients of 0.98-0.99 and 0.93-0.95 in anterior/posterior and medial/lateral directions, respectively. For the LSTM model, correlation coefficients were similar to those of FFANN. However, the relative root mean square error (12.5%) of the FFANN model was higher than that (9.8%) of the LSTM model in medial/lateral direction (p = 0.03). This study can be used for quantitative evaluation of clinical diagnosis and rehabilitation status for patient with various diseases through further training using varied databases. Graphical abstract Architectures of neural networks developed in this study (a feed forward artificial neural network; b LSTM network).