From deep learning to transfer learning for the prediction of skeletal muscle forces

Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A Survey

The inference of 3D motion and dynamics of the human musculoskeletal system has traditionally been solved using physics-based methods that exploit physical parameters to provide realistic simulations. Yet, such methods suffer from computational complexity and reduced stability, hindering their use in computer graphics applications that require real-time performance. With the recent explosion of data capture (mocap, video) machine learning (ML) has started to become popular as it is able to create surrogate models harnessing the huge amount of data stemming from various sources, minimizing computational time (instead of resource usage), and most importantly, approximate real-time solutions. The main purpose of this paper is to provide a review and classification of the most recent works regarding motion prediction, motion synthesis as well as musculoskeletal dynamics estimation problems using ML techniques, in order to offer sufficient insight into the state-of-the-art and draw new research directions. While the study of motion may appear distinct to musculoskeletal dynamics, these application domains provide jointly the link for more natural computer graphics character animation, since ML-based musculoskeletal dynamics estimation enables modeling of more long-term, temporally evolving, ergonomic effects, while offering automated and fast solutions. Overall, our review offers an in-depth presentation and classification of ML applications in human motion analysis, unlike previous survey articles focusing on specific aspects of motion prediction.

MMG-Based Knee Dynamic Extension Force Estimation Using Cross-Talk and IGWO-LSTM

Mechanomyography (MMG) is an important muscle physiological activity signal that can reflect the amount of motor units recruited as well as the contraction frequency. As a result, MMG can be utilized to estimate the force produced by skeletal muscle. However, cross-talk and time-series correlation severely affect MMG signal recognition in the real world. These restrict the accuracy of dynamic muscle force estimation and their interaction ability in wearable devices. To address these issues, a hypothesis that the accuracy of knee dynamic extension force estimation can be improved by using MMG signals from a single muscle with less cross-talk is first proposed. The hypothesis is then confirmed using the estimation results from different muscle signal feature combinations. Finally, a novel model (improved grey wolf optimizer optimized long short-term memory networks, i.e., IGWO-LSTM) is proposed for further improving the performance of knee dynamic extension force estimation. The experimental results demonstrate that MMG signals from a single muscle with less cross-talk have a superior ability to estimate dynamic knee extension force. In addition, the proposed IGWO-LSTM provides the best performance metrics in comparison to other state-of-the-art models. Our research is expected to not only improve the understanding of the mechanisms of quadriceps contraction but also enhance the flexibility and interaction capabilities of future rehabilitation and assistive devices.

Prediction of medial knee contact force using multisource fusion recurrent neural network and transfer learning

Estimation of knee contact force (KCF) during gait provides essential information to evaluate knee joint function. Machine learning has been employed to estimate KCF because of the advantages of low computational cost and real-time. However, the existing machine learning models do not adequately consider gait-related data's temporal-dependent, multidimensional, and highly heterogeneous nature. This study is aimed at developing a multisource fusion recurrent neural network to predict the medial condyle KCF. First, a multisource fusion long short-term memory (MF-LSTM) model was established. Then, we developed a transfer learning strategy based on the MF-LSTM model for subject-specific medial KCF prediction. Four subjects with instrumented tibial prostheses were obtained from the literature. The results showed that the MF-LSTM model could predict medial KCF to a certain high level of accuracy (the mean of ρ = 0.970). The transfer learning model improved the prediction accuracy (the mean of ρ = 0.987). This study shows that the MF-LSTM model is a powerful and accurate computational tool for medial KCF prediction. Introducing transfer learning techniques could further improve the prediction performance for the target subject. This coupling strategy can help clinicians accurately estimate and track joint contact forces in real time.

Estimation of Muscle Forces of Lower Limbs Based on CNN-LSTM Neural Network and Wearable Sensor System

Estimation of vivo muscle forces during human motion is important for understanding human motion control mechanisms and joint mechanics. This paper combined the advantages of the convolutional neural network (CNN) and long-short-term memory (LSTM) and proposed a novel muscle force estimation method based on CNN-LSTM. A wearable sensor system was also developed to collect the angles and angular velocities of the hip, knee, and ankle joints in the sagittal plane during walking, and the collected kinematic data were used as the input for the neural network model. In this paper, the muscle forces calculated using OpenSim based on the Static Optimization (SO) method were used as the standard value to train the neural network model. Four lower limb muscles of the left leg, including gluteus maximus (GM), rectus femoris (RF), gastrocnemius (GAST), and soleus (SOL), were selected as the studying objects in this paper. The experiment results showed that compared to the standard CNN and the standard LSTM, the CNN-LSTM performed better in muscle forces estimation under slow (1.2 m/s), medium (1.5 m/s), and fast walking speeds (1.8 m/s). The average correlation coefficients between true and estimated values of four muscle forces under slow, medium, and fast walking speeds were 0.9801, 0.9829, and 0.9809, respectively. The average correlation coefficients had smaller fluctuations under different walking speeds, which indicated that the model had good robustness. The external testing experiment showed that the CNN-LSTM also had good generalization. The model performed well when the estimated object was not included in the training sample. This article proposed a convenient method for estimating muscle forces, which could provide theoretical assistance for the quantitative analysis of human motion and muscle injury. The method has established the relationship between joint kinematic signals and muscle forces during walking based on a neural network model; compared to the SO method to calculate muscle forces in OpenSim, it is more convenient and efficient in clinical analysis or engineering applications.

Efficient Generation of Pretraining Samples for Developing a Deep Learning Brain Injury Model via Transfer Learning

The large amount of training samples required to develop a deep learning brain injury model demands enormous computational resources. Here, we study how a transformer neural network (TNN) of high accuracy can be used to efficiently generate pretraining samples for a convolutional neural network (CNN) brain injury model to reduce computational cost. The samples use synthetic impacts emulating real-world events or augmented impacts generated from limited measured impacts. First, we verify that the TNN remains highly accurate for the two impact types (N = 100 each; [Formula: see text] of 0.948-0.967 with root mean squared error, RMSE, ~ 0.01, for voxelized peak strains). The TNN-estimated samples (1000-5000 for each data type) are then used to pretrain a CNN, which is further finetuned using directly simulated training samples (250-5000). An independent measured impact dataset considered of complete capture of impact event is used to assess estimation accuracy (N = 191). We find that pretraining can significantly improve CNN accuracy via transfer learning compared to a baseline CNN without pretraining. It is most effective when the finetuning dataset is relatively small (e.g., 2000-4000 pretraining synthetic or augmented samples improves success rate from 0.72 to 0.81 with 500 finetuning samples). When finetuning samples reach 3000 or more, no obvious improvement occurs from pretraining. These results support using the TNN to rapidly generate pretraining samples to facilitate a more efficient training strategy for future deep learning brain models, by limiting the number of costly direct simulations from an alternative baseline model. This study could contribute to a wider adoption of deep learning brain injury models for large-scale predictive modeling and ultimately, enhancing safety protocols and protective equipment.

Convolutional LSTM: a deep learning approach to predict shoulder joint reaction forces

We developed a Convolutional LSTM (ConvLSTM) network to predict shoulder joint reaction forces using 3D shoulder kinematics data containing 30 different shoulder activities from eight human subjects. We considered simulation outcomes from the AnyBody musculoskeletal model as the baseline force dataset to validate ConvLSTM model predictions. Results showed a good correlation (>80% accuracy, r ≥ 0.82) between ConvLSTM predicted and AnyBody estimated force values, the generalization of the developed model for novel task type (p-value = 0.07 ∼ 0.33), and a better prediction accuracy for the ConvLSTM model than conventional CNN and LSTM models.

Prediction on the medial knee contact force in patients with knee valgus using transfer learning approaches: Application to rehabilitation gaits

The Knee contact force (KCF) is a key factor in evaluating knee joint function of patients with knee osteoarthritis. In vivo measurement of KCF based on the instrumented implants is limited due to the ethical issues and technical complexities. Machine learning can be used to predict tibiofemoral compartment contact forces. However, anthropometric differences between individuals make the accurate predictions challenging. The purpose of this study was to develop transfer learning models to predict the medial KCF of patients with knee valgus in rehabilitation gaits. Four subjects with instrumented tibial prostheses were considered, including one with knee valgus and three with normal knee joint alignment. Two transfer learning models were proposed: a fine-tuning model and an adaptive model. In particular, a synchronization method for extracting experimental data in a complete gait cycle was developed, since different types of experimental data have different sampling frequencies. The transfer learning models were pre-trained by the experiment data of patients with normal knee joint alignment, and re-trained by the data of the patient with knee valgus. Predictions of the transfer learning models and traditional machine learning model were validated against the in vivo measurements. The proposed transfer learning models were tested within two levels: the single subject (Level 1) and multiple subjects (Level 2). The results show that the two transfer learning models could more accurately predict the medial KCF of patients with knee valgus than the traditional machine learning model. The performance of the fine-tuning model is better than that of the adaptive model. Compared with the traditional machine learning and inverse dynamics analysis, transfer learning represents a much easier and more accurate method. It can be introduced to help clinicians validate and adjust the rehabilitation gait for specific patients.

Recurrent neural network to predict hyperelastic constitutive behaviors of the skeletal muscle

Hyperelastic constitutive laws have been commonly used to model the passive behavior of the human skeletal muscle. Despite many efforts, the use of accurate finite element formulations of hyperelastic constitutive laws is still time-consuming for a real-time medical simulation system. The objective of the present study was to develop a deep learning model to predict the hyperelastic constitutive behaviors of the skeletal muscle toward a fast estimation of the muscle tissue stress.A finite element (FE) model of the right psoas muscle was developed. Neo-Hookean and Mooney-Rivlin laws were used. A tensile test was performed with an applied body force. A learning database was built from this model using an automatic probabilistic generation process. A long-short term memory (LSTM) neural network was implemented to predict the stress evolution of the skeletal muscle tissue. A hyperparameter tuning process was conducted. Root mean square error (RMSE) and associated relative error was quantified to evaluate the precision of the predictive capacity of the developed deep learning model. Pearson correlation coefficients (R) was also computed.The nodal displacements and the maximal stresses range from 70 to 227 mm and from 2.79 to 5.61 MPa for Neo-Hookean and Monney-Rivlin laws, respectively. Regarding the LSTM predictions, the RMSE ranges from 224.3 ± 3.9 Pa (8%) to 227.5 [Formula: see text] 5.7 Pa (4%) for Neo-Hookean and Monney-Rivlin laws, respectively. Pearson correlation coefficients (R) of 0.78 [Formula: see text] 0.02 and 0.77 [Formula: see text] 0.02 were obtained for Neo-Hookean and Monney-Rivlin laws, respectively.The present study showed that, for the first time, the use of a deep learning model can reproduce the time-series behaviors of the complex FE formulations for skeletal muscle modeling. In particular, the use of a LSTM neural network leads to a fast and accurate surrogate model for the in silico prediction of the hyperelastic constitutive behaviors of the skeletal muscle. As perspectives, the developed deep learning model will be integrated into a real-time medical simulation of the skeletal muscle for prosthetic socket design and childbirth simulator.

Lower-limb Joint Torque Prediction using LSTM Neural Networks and Transfer Learning

Estimation of joint torque during movement provides important information in several settings, such as effect of athletes' training or of a medical intervention, or analysis of the remaining muscle strength in a wearer of an assistive device. The ability to estimate joint torque during daily activities using wearable sensors is increasingly relevant in such settings. In this study, lower limb joint torques during ten daily activities were predicted by long short-term memory (LSTM) neural networks and transfer learning. LSTM models were trained with muscle electromyography signals and lower limb joint angles. Hip flexion/extension, hip abduction/adduction, knee flexion/extension and ankle dorsiflexion/plantarflexion torques were predicted. The LSTM models' performance in predicting torque was investigated in both intra-subject and inter-subject scenarios. Each scenario was further divided into intra-task and inter-task tests. We observed that LSTM models could predict lower limb joint torques during various activities accurately with relatively low error (root mean square error ≤ 0.14 Nm/kg, normalized root mean square error ≤8.7%) either through a uniform model or through ten separate models in intra-subject tests. Furthermore, a transfer learning technique was adopted in the inter-task and inter-subject tests to further improve the generalizability of LSTM models by pre-training a model on multiple subjects and/or tasks and transferring the learned knowledge to a target task/subject. Particularly in the inter-subject tests, we could predict joint torques accurately in several movements after training from only a few movements from new subjects.

Deep Learning-Based Myoelectric Potential Estimation Method for Wheelchair Operation

Wheelchair sports are recognized as an international sport, and research and support are being promoted to increase the competitiveness of wheelchair sports. For example, an electromyogram can observe muscle activity. However, it is generally used under controlled conditions due to the complexity of preparing the measurement equipment and the movement restrictions imposed by cables and measurement equipment. It is difficult to perform measurements in actual competition environments. Therefore, in this study, we developed a method to estimate myoelectric potential that can be used in competitive environments and does not limit physical movement. We developed a deep learning model that outputs surface myoelectric potentials by inputting camera images of wheelchair movements and the measured values of inertial sensors installed on wheelchairs. For seven subjects, we estimated the myoelectric potential during chair work, which is important in wheelchair sports. As a result of creating an in-subject model and comparing the estimated myoelectric potential with the myoelectric potential measured by an electromyogram, we confirmed a correlation (correlation coefficient 0.5 or greater at a significance level of 0.1%). Since this method can estimate the myoelectric potential without limiting the movement of the body, it is considered that it can be applied to the performance evaluation of wheelchair sports.

Transductive Learning Models for Accurate Ambulatory Gait Analysis in Elderly Residents of Assisted Living Facilities

Instrumented footwear represents a promising technology for spatiotemporal gait analysis in out-of-the-lab conditions. However, moderate accuracy impacts this technology's ability to capture subtle, but clinically meaningful, changes in gait patterns that may indicate adverse outcomes or underlying neurological conditions. This limitation hampers the use of instrumented footwear to aid functional assessments and clinical decision making. This paper introduces new transductive-learning inference models that substantially reduce measurement errors relative to conventional data processing techniques, without requiring subject-specific labelled data. The proposed models use subject-optimized input features and hyperparameters to adjust the spatiotemporal gait metrics (i.e., stride time, length, and velocity, swing time, and double support time) obtained with conventional techniques, resulting in computationally simpler models compared to end-to-end machine learning approaches. Model validity and reliability were evaluated against a gold-standard electronic walkway during a clinical gait performance test (6-minute walk test) administered to N=95 senior residents of assisted living facilities with diverse levels of gait and balance impairments. Average reductions in absolute errors relative to conventional techniques were -42.0% and -33.5% for spatial and gait-phase parameters, respectively, indicating the potential of transductive learning models for improving the accuracy of instrumented footwear for ambulatory gait analysis.

InverseMuscleNET: Alternative Machine Learning Solution to Static Optimization and Inverse Muscle Modeling

InverseMuscleNET, a machine learning model, is proposed as an alternative to static optimization for resolving the redundancy issue in inverse muscle models. A recurrent neural network (RNN) was optimally configured, trained, and tested to estimate the pattern of muscle activation signals. Five biomechanical variables (joint angle, joint velocity, joint acceleration, joint torque, and activation torque) were used as inputs to the RNN. A set of surface electromyography (EMG) signals, experimentally measured around the shoulder joint for flexion/extension, were used to train and validate the RNN model. The obtained machine learning model yields a normalized regression in the range of 88-91% between experimental data and estimated muscle activation. A sequential backward selection algorithm was used as a sensitivity analysis to discover the less dominant inputs. The order of most essential signals to least dominant ones was as follows: joint angle, activation torque, joint torque, joint velocity, and joint acceleration. The RNN model required 0.06 s of the previous biomechanical input signals and 0.01 s of the predicted feedback EMG signals, demonstrating the dynamic temporal relationships of the muscle activation profiles. The proposed approach permits a fast and direct estimation ability instead of iterative solutions for the inverse muscle model. It raises the possibility of integrating such a model in a real-time device for functional rehabilitation and sports evaluation devices with real-time estimation and tracking. This method provides clinicians with a means of estimating EMG activity without an invasive electrode setup.

Utility of Inter-subject Transfer Learning for Wearable-Sensor-Based Joint Torque Prediction Models

Generalizability between individuals and groups is often a significant hurdle in model development for human subjects research. In the domain of wearable-sensor-controlled exoskeleton devices, the ability to generalize models across subjects or fine-tune more general models to individual subjects is key to enabling widespread adoption of these technologies. Transfer learning techniques applied to machine learning models afford the ability to apply and investigate the viability and utility such knowledge-transfer scenarios. This paper investigates the utility of single- and multi-subject based parameter transfer on LSTM models trained for "sensor-to-joint torque" prediction tasks, with regards to task performance and computational resources required for network training. We find that parameter transfer between both single- and multi-subject models provide useful knowledge transfer, with varying results across specific "source" and "target" subject pairings. This could be leveraged to lower model training time or computational cost in compute-constrained environments or, with further study to understand causal factors of the observed variance in performance across source and target pairings, to minimize data collection and model retraining requirements to select and personalize a generic model for personalized wearable-sensor-based joint torque prediction technologies.

Instantaneous Brain Strain Estimation for Automotive Head Impacts via Deep Learning

Efficient brain strain estimation is critical for routine application of a head injury model. Lately, a convolutional neural network (CNN) has been successfully developed to estimate spatially detailed brain strains instantly and accurately in contact sports. Here, we extend its application to automotive head impacts, where impact profiles are typically more complex with longer durations. Head impact kinematics (N=458) from two public databases were used to generate augmented impacts (N=2694). They were simulated using the anisotropic Worcester Head Injury Model (WHIM) V1.0, which provided baseline elementwise peak maximum principal strain (MPS). For each augmented impact, rotational velocity (v_rot) and the corresponding rotational acceleration (a_rot) profiles were concatenated as static images to serve as CNN input. Three training strategies were evaluated: 1) "baseline", using random initial weights; 2) "transfer learning", using weight transfer from a previous CNN model trained on head impacts drawn from contact sports; and 3) "combined training", combining previous training data from contact sports (N=5661) for training. The combined training achieved the best performances. For peak MPS, the CNN achieved a coefficient of determination (R²) of 0.932 and root mean squared error (RMSE) of 0.031 for the real-world testing dataset. It also achieved a success rate of 60.5% and 94.8% for elementwise MPS, where the linear regression slope, k, and correlation coefficient, r, between estimated and simulated MPS did not deviate from 1.0 (when identical) by more than 0.1 and 0.2, respectively. Cumulative strain damage measure (CSDM) from the CNN estimation was also highly accurate compared to those from direct simulation across a range of thresholds (R² of 0.899-0.943 with RMSE of 0.054-0.069). Finally, the CNN achieved an average k and r of 0.98±0.12 and 0.90±0.07, respectively, for six reconstructed car crash impacts drawn from two other sources independent of the training dataset. Importantly, the CNN is able to efficiently estimate elementwise MPS with sufficient accuracy while conventional kinematic injury metrics cannot. Therefore, the CNN has the potential to supersede current kinematic injury metrics that can only approximate a global peak MPS or CSDM. The CNN technique developed here may offer enhanced utility in the design and development of head protective countermeasures, including in the automotive industry. This is the first study aimed at instantly estimating spatially detailed brain strains for automotive head impacts, which employs >8.8 thousand impact simulations generated from ~1.5 years of nonstop computations on a high-performance computing platform.

Real-time Estimation of the Strength Capacity of the Upper Limb for Physical Human-Robot Collaboration

In physical Human-Robot Collaboration (pHRC), having an estimate of the operator's strength capacity can help implement control strategies. Currently, the trend is to integrate devices that can measure physiological signals. This is not always a viable option, especially for highly dynamic tasks. For pHRC tasks, the physical interaction point usually occurs at the operator's hand. Therefore, a musculo-skeletal model was used to have a real-time estimation of the strength capacity of the operator's upper limb. First, the model has been simplified to reduce the complexity of the problem. The model was used to obtain offline estimations of the strength capacity, that were then curve-fitted to enable real-time estimation. An experiment was carried out to compare the results of the approximated model with human data. Results suggest that this method for estimating the strength capacity of the upper limb is a viable solution for real-time applications.

Enhanced facial expression recognition using 3D point sets and geometric deep learning

Facial expression recognition plays an essential role in human conversation and human-computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.

Machine learning for rapid estimation of lower extremity muscle and joint loading during activities of daily living

Joint contact and muscle forces estimated with musculoskeletal modeling techniques offer useful metrics describing movement quality that benefit multiple research and clinical applications. The expensive processing of laboratory data associated with generating these outputs presents challenges to researchers and clinicians, including significant time and expertise requirements that limit the number of subjects typically evaluated. The objective of the current study was to develop and compare machine learning techniques for rapid, data-driven estimation of musculoskeletal metrics from derived gait lab data. OpenSim estimates of patient joint and muscle forces during activities of daily living were simulated using laboratory data from 70 total knee replacement patients and used to develop 4 different machine learning algorithms. Trained machine learning models predicted both trend and magnitude of estimated joint contact (mean correlation coefficients ranging from 0.93 to 0.94 during gait) and muscle forces (mean correlation coefficients ranging from 0.83 to 0.91 during gait) based on anthropometrics, ground reaction forces, and joint angle data. Patient mechanics were accurately predicted by recurrent neural networks, even after removing dependence on key subsets of predictor features. The ability to quickly estimate patient mechanics from derived measurements of movement has the potential to broaden the impact of musculoskeletal modeling by enabling faster assessment in both clinical and research settings.

sEMG-Based Neural Network Prediction Model Selection of Gesture Fatigue and Dataset Optimization

The fatigue energy consumption of independent gestures can be obtained by calculating the power spectrum of surface electromyography (sEMG) signals. The existing research studies focus on the fatigue of independent gestures, while the research studies on integrated gestures are few. However, the actual gesture operation mode is usually integrated by multiple independent gestures, so the fatigue degree of integrated gestures can be predicted by training neural network of independent gestures. Three natural gestures including browsing information, playing games, and typing are divided into nine independent gestures in this paper, and the predicted model is established and trained by calculating the energy consumption of independent gestures. The artificial neural networks (ANNs) including backpropagation (BP) neural network, recurrent neural network (RNN), and long short-term memory (LSTM) are used to predict the fatigue of gesture. The support vector machine (SVM) is used to assist verification. Mean square error (MSE), root mean square error (RMSE), and mean absolute error (MAE) are utilized to evaluate the optimal prediction model. Furthermore, the different datasets of the processed sEMG signal and its decomposed wavelet coefficients are trained, respectively, and the changes of error functions of them are compared. The experimental results show that LSTM model is more suitable for gesture fatigue prediction. The processed sEMG signals are appropriate for using as the training set the fatigue degree of one-handed gesture. It is better to use wavelet decomposition coefficients as datasets to predict the high-dimensional sEMG signals of two-handed gestures. The experimental results can be applied to predict the fatigue degree of complex human-machine interactive gestures, help to avoid unreasonable gestures, and improve the user's interactive experience.

Multi-Segmentation Parallel CNN Model for Estimating Assembly Torque Using Surface Electromyography Signals

The precise application of tightening torque is one of the important measures to ensure accurate bolt connection and improvement in product assembly quality. Currently, due to the limited assembly space and efficiency, a wrench without the function of torque measurement is still an extensively used assembly tool. Therefore, wrench torque monitoring is one of the urgent problems that needs to be solved. This study proposes a multi-segmentation parallel convolution neural network (MSP-CNN) model for estimating assembly torque using surface electromyography (sEMG) signals, which is a method of torque monitoring through classification methods. The MSP-CNN model contains two independent CNN models with different or offset torque granularities, and their outputs are fused to obtain a finer classification granularity, thus improving the accuracy of torque estimation. First, a bolt tightening test bench is established to collect sEMG signals and tightening torque signals generated when the operator tightens various bolts using a wrench. Second, the sEMG and torque signals are preprocessed to generate the sEMG signal graphs. The range of the torque transducer is divided into several equal subdivision ranges according to different or offset granularities, and each subdivision range is used as a torque label for each torque signal. Then, the training set, verification set, and test set are established for torque monitoring to train the MSP-CNN model. The effects of different signal preprocessing methods, torque subdivision granularities, and pooling methods on the recognition accuracy and torque monitoring accuracy of a single CNN network are compared experimentally. The results show that compared to maximum pooling, average pooling can improve the accuracy of CNN torque classification and recognition. Moreover, the MSP-CNN model can improve the accuracy of torque monitoring as well as solve the problems of non-convergence and slow convergence of independent CNN network models.

Multi-view deep learning for rigid gas permeable lens base curve fitting based on Pentacam images

Many studies in the rigid gas permeable (RGP) lens fitting field have focused on providing the best fit for patients with irregular astigmatism, a challenging issue. Despite the ease and accuracy of fitting in the current fitting methods, no studies have provided a high-pace solution with the final best fit to assist experts. This work presents a deep learning solution for identifying features in Pentacam four refractive maps and RGP base curve identification. An authentic dataset of 247 samples of Pentacam four refractive maps was gathered, providing a multi-view image of the corneal structure. Scratch-based convolutional neural network (CNN) architectures and well-known CNN architectures such as AlexNet, GoogLeNet, and ResNet have been used to extract features and transfer learning. Features are aggregated through a fusion technique. Based on a comparison of means square error (MSE) of normalized labels, the multi-view scratch-based CNN provided R-squared of 0.849, 0.846, 0.835, and 0.834 followed by GoogLeNet, comparable with current methods. Transfer learning outperforms various scratch-based CNN models, through which proper specifications some scratch-based models were able to increase coefficient of determinations. CNNs on multi-view Pentacam images have enabled fast detection of the RGP lens base curve, higher patient satisfaction, and reduced chair time. Graphical abstract The Pentacam four refractive maps is learned by the proposed scratch-based and transfer learning-based CNN methodology. The deep network-based solutions enable identification of rigid gas permeable lens for patients with irregular astigmatism.

Random Forest enhancement using improved Artificial Fish Swarm for the medial knee contact force prediction

Knee contact force (KCF) is an important factor to evaluate the knee joint function for the patients with knee joint impairment. However, the KCF measurement based on the instrumented prosthetic implants or inverse dynamics analysis is limited due to the invasive, expensive price and time consumption. In this work, we propose a KCF prediction method by integrating the Artificial Fish Swarm and the Random Forest algorithm. First, we train a Random Forest to learn the nonlinear relation between gait parameters (input) and contact pressures (output) based on a dataset of three patients instrumented with knee replacement. Then, we use the improved artificial fish group algorithm to optimize the main parameters of the Random Forest based KCF prediction model. The extensive experiments verify that our method can predict the medial knee contact force both before and after the intervention of gait patterns, and the performance outperforms the classical multi-body dynamics analysis and artificial neural network model.

Machine learning approach to predict center of pressure trajectories in a complete gait cycle: a feedforward neural network vs. LSTM network

Center of pressure (COP) trajectories of human can maintain regulation of forward progression and stability of lateral sway during walking. The insole pressure system can only detect COP trajectories of each foot during single stance. In this study, we developed artificial neural network models that could present COP trajectories in an integrated coordinate system during a complete gait cycle using pressure information of the insole system. A feed forward artificial neural network (FFANN) and a long short-term memory (LSTM) model were developed. For FFANN, among 198 pressure sensors from Pedar-X insoles, proper input variables were selected using sequential forward selection to reduce input dimension. The LSTM model used all 198 signals as inputs because of its self-learning characteristic. As results of cross-validation, the FFANN model showed correlation coefficients of 0.98-0.99 and 0.93-0.95 in anterior/posterior and medial/lateral directions, respectively. For the LSTM model, correlation coefficients were similar to those of FFANN. However, the relative root mean square error (12.5%) of the FFANN model was higher than that (9.8%) of the LSTM model in medial/lateral direction (p = 0.03). This study can be used for quantitative evaluation of clinical diagnosis and rehabilitation status for patient with various diseases through further training using varied databases. Graphical abstract Architectures of neural networks developed in this study (a feed forward artificial neural network; b LSTM network).

Single Inertial Sensor-Based Neural Networks to Estimate COM-COP Inclination Angle During Walking

A biomechanical understanding of gait stability is needed to reduce falling risk. As a typical parameter, the COM-COP (center of mass-center of pressure) inclination angle (IA) could provide valuable insight into postural control and balance recovery ability. In this study, an artificial neural network (ANN) model was developed to estimate COM-COP IA based on signals using an inertial sensor. Also, we evaluated how different types of ANN and the cutoff frequency of the low-pass filter applied to input signals could affect the accuracy of the model. An inertial measurement unit (IMU) including an accelerometer, gyroscope, and magnetometer sensors was fabricated as a prototype. The COM-COP IA was calculated using a 3D motion analysis system including force plates. In order to predict the COM-COP IA, a feed-forward ANN and long-short term memory (LSTM) network was developed. As a result, the feed-forward ANN showed a relative root-mean-square error (rRMSE) of 15% while the LSTM showed an improved accuracy of 9% rRMSE. Additionally, the LSTM displayed a stable accuracy regardless of the cutoff frequency of the filter applied to the input signals. This study showed that estimating the COM-COP IA was possible with a cheap inertial sensor system. Furthermore, the neural network models in this study can be implemented in systems to monitor the balancing ability of the elderly or patients with impaired balancing ability.