A Framework to Automate Assessment of Upper-Limb Motor Function Impairment: A Feasibility Study

Clinical validation of automated depth camera-based measurement of the Fugl-Meyer assessment for upper extremity

OBJECTIVE

Depth camera-based measurement has demonstrated efficacy in automated assessment of upper limb Fugl-Meyer Assessment for paralysis rehabilitation. However, there is a lack of adequately sized studies to provide clinical support. Thus, we developed an automated system utilizing depth camera and machine learning, and assessed its feasibility and validity in a clinical setting.

DESIGN

Validation and feasibility study of a measurement instrument based on single cross-sectional data.

SETTING

Rehabilitation unit in a general hospital.

PARTICIPANTS

Ninety-five patients with hemiparesis admitted for inpatient rehabilitation unit (2021-2023).

MAIN MEASURES

Scores for each item, excluding those related to reflexes, were computed utilizing machine learning models trained on participant videos and readouts from force test devices, while the remaining reflex scores were derived through regression algorithms. Concurrent criterion validity was evaluated using sensitivity, specificity, percent agreement and Cohen's Kappa coefficient for ordinal scores of individual items, as well as correlations and intraclass correlation coefficients for total scores. Video-based manual assessment was also conducted and compared to the automated tools.

RESULT

The majority of patients completed the assessment without therapist intervention. The automated scoring models demonstrated superior validity compared to video-based manual assessment across most items. The total scores derived from the automated assessment exhibited a high coefficient of 0.960. However, the validity of force test items utilizing force sensing resistors was relatively low.

CONCLUSION

The integration of depth camera technology and machine learning models for automated Fugl-Meyer Assessment demonstrated acceptable validity and feasibility, suggesting its potential as a valuable tool in rehabilitation assessment.

Smartwatch-based functional assessment for upper extremity impairment after musculoskeletal injuries: A pilot study

Introduction

Wearable sensors are increasingly applied to rehabilitation for arm movement analysis. However, simple and clinically relevant applications are scarce.

Objectives

To investigate the feasibility of single smart watch-based parameters for functional assessment in upper limb rehabilitation for musculoskeletal injuries using a commercial smart watch.

Method

Ten patients with unilateral shoulder pain and range-of-motion limitations were enrolled. They wore Galaxy Watch® and performed three sets of upper extremity tasks consisting of gross activities-of-daily-living tasks, Wolf Motor Function Test (WMFT), and Upper Extremity Functional Index (UEFI), and the acceleration and angular velocities were acquired. The motion segment size (MSS), representing motion smoothness from a clinical perspective, and various sensor-based parameters were extracted. The correlation between the parameters and clinical outcome measures were analyzed. The percent relative range (PRR) of the significant parameters was also calculated.

Results

For overhead and behind body activity task set, mean MSS for elbow flexion/extension axis significantly correlated with WMFT score (R = 0.784, p = .012). For planar tasks, mean MSS for the forearm supination/pronation (R = 0.815, p = .007) and shoulder rotation (R = 0.870, p = .002) axes significantly correlated with WMFT score. For forearm and fine movement task set, mean MSS of the elbow flexion/extension angle showed significant correlation with WMFT (R = 0.880, p < .001) and UEFI (R = 0.718, p = .019). The total performance time (R = -0.741, p = .014) also showed significant correlation with WMFT score. The PRR for mean MSS in forearm supination (71.5%, planar tasks) and mean MSS in x-direction (49.8%, forearm and fine motor movements) were similar to the PRR of WMFT (58.5%), suggesting sufficient variation range across different degree of impairments.

Conclusion

The commercial smart watch-based parameters showed consistent potential for use in clinical functional assessments.

Detection and Assessment of Point-to-Point Movements During Functional Activities Using Deep Learning and Kinematic Analyses of the Stroke-Affected Wrist

Stoke is a leading cause of long-term disability, including upper-limb hemiparesis. Frequent, unobtrusive assessment of naturalistic motor performance could enable clinicians to better assess rehabilitation effectiveness and monitor patients' recovery trajectories. We therefore propose and validate a two-phase data analytic pipeline to estimate upper-limb impairment based on the naturalistic performance of activities of daily living (ADLs). Eighteen stroke survivors were equipped with an inertial sensor on the stroke-affected wrist and performed up to four ADLs in a naturalistic manner. Continuous inertial time series were segmented into sliding windows, and a machine-learned model identified windows containing instances of point-to-point (P2P) movements. Using kinematic features extracted from the detected windows, a subsequent model was used to estimate upper-limb motor impairment, as measured by the Fugl-Meyer Assessment (FMA). Both models were evaluated using leave-one-subject-out cross-validation. The P2P movement detection model had an area under the precision-recall curve of 0.72. FMA estimates had a normalized root mean square error of 18.8% with R2=0.72. These promising results support the potential to develop seamless, ecologically valid measures of real-world motor performance.

A novel scoring approach for the Wolf Motor Function Test in stroke survivors using motion-sensing technology and machine learning: A preliminary study

BACKGROUND AND OBJECTIVE

Human-administered clinical scales, such as the Functional Ability Scale of the Wolf Motor Function Test (WMFT-FAS), are widely utilized to evaluate upper-limb motor function in stroke survivors. However, these scales are generally subjective and labor-intensive. To end this, we proposed a novel scoring approach for the motor function assessment.

METHODS

The proposed novel scoring approach mainly contained one Microsoft Kinect v2, one customized motion tracking system, and one customized intelligent scoring system. Specifically, the Kinect v2 was used to capture stroke survivors' functional movements, the motion tracking system was developed for recording the gathered movement data, and the intelligent scoring system (kernel: feed-forward neural network, FFNN) was developed to evaluate movement quality and provide corresponding WMFT-FAS scores. Several methods have been applied to enhance the approach's usability, such as singular spectrum analysis and multi-ReliefF method.

RESULTS

Sixteen stroke survivors and ten healthy subjects were recruited for validation. Inspiring results of the proposed approach were achieved when compared with the clinical scores provided by a physiotherapist: 0.924 ± 0.027 for accuracy, 0.875 ± 0.063 for F1-score, 0.915 ± 0.051 for sensitivity, 0.969 ± 0.013 for specificity, 0.952 ± 0.038 for AUC, 0.098 ± 0.037 for mean absolute error, and 0.214 ± 0.078 for root mean squared error.

CONCLUSIONS

The results indicate that the proposed novel scoring approach can provide objective and accurate assessment scores, which can help physiotherapists make individualized treatment decisions.

D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation

This paper tackles the challenge of automatically assessing physical rehabilitation exercises for patients who perform the exercises without clinician supervision. The objective is to provide a quality score to ensure correct performance and achieve desired results. To achieve this goal, a new graph-based model, the Dense Spatio-Temporal Graph Conv-GRU Network with Transformer, is introduced. This model combines a modified version of STGCN and transformer architectures for efficient handling of spatio-temporal data. The key idea is to consider skeleton data respecting its non-linear structure as a graph and detecting joints playing the main role in each rehabilitation exercise. Dense connections and GRU mechanisms are used to rapidly process large 3D skeleton inputs and effectively model temporal dynamics. The transformer encoder's attention mechanism focuses on relevant parts of the input sequence, making it useful for evaluating rehabilitation exercises. The evaluation of our proposed approach on the KIMORE and UI-PRMD datasets highlighted its potential, surpassing state-of-the-art methods in terms of accuracy and computational time. This resulted in faster and more accurate learning and assessment of rehabilitation exercises. Additionally, our model provides valuable feedback through qualitative illustrations, effectively highlighting the significance of joints in specific exercises.

aBnormal motION capture In aCute Stroke (BIONICS): A Low-Cost Tele-Evaluation Tool for Automated Assessment of Upper Extremity Function in Stroke Patients

BACKGROUND

The incidence of stroke and stroke-related hemiparesis has been steadily increasing and is projected to become a serious social, financial, and physical burden on the aging population. Limited access to outpatient rehabilitation for these stroke survivors further deepens the healthcare issue and estranges the stroke patient demographic in rural areas. However, new advances in motion detection deep learning enable the use of handheld smartphone cameras for body tracking, offering unparalleled levels of accessibility.

METHODS

In this study we want to develop an automated method for evaluation of a shortened variant of the Fugl-Meyer assessment, the standard stroke rehabilitation scale describing upper extremity motor function. We pair this technology with a series of machine learning models, including different neural network structures and an eXtreme Gradient Boosting model, to score 16 of 33 (49%) Fugl-Meyer item activities.

RESULTS

In this observational study, 45 acute stroke patients completed at least 1 recorded Fugl-Meyer assessment for the training of the auto-scorers, which yielded average accuracies ranging from 78.1% to 82.7% item-wise.

CONCLUSION

In this study, an automated method was developed for the evaluation of a shortened variant of the Fugl-Meyer assessment, the standard stroke rehabilitation scale describing upper extremity motor function. This novel method is demonstrated with potential to conduct telehealth rehabilitation evaluations and assessments with accuracy and availability.

Exercise repetition rate measured with simple sensors at home can be used to estimate Upper Extremity Fugl-Meyer score after stroke

Introduction

It would be valuable if home-based rehabilitation training technologies could automatically assess arm impairment after stroke. Here, we tested whether a simple measure-the repetition rate (or "rep rate") when performing specific exercises as measured with simple sensors-can be used to estimate Upper Extremity Fugl-Meyer (UEFM) score.

Methods

41 individuals with arm impairment after stroke performed 12 sensor-guided exercises under therapist supervision using a commercial sensor system comprised of two pucks that use force and motion sensing to measure the start and end of each exercise repetition. 14 of these participants then used the system at home for three weeks.

Results

Using linear regression, UEFM score was well estimated using the rep rate of one forward-reaching exercise from the set of 12 exercises (r2 = 0.75); this exercise required participants to alternately tap pucks spaced about 20 cm apart (one proximal, one distal) on a table in front of them. UEFM score was even better predicted using an exponential model and forward-reaching rep rate (Leave One Out Cross Validation (LOOCV) r2 = 0.83). We also tested the ability of a nonlinear, multivariate model (a regression tree) to predict UEFM, but such a model did not improve prediction (LOOCV r2 = 0.72). However, the optimal decision tree also used the forward-reaching task along with a pinch grip task to subdivide more and less impaired patients in a way consistent with clinical intuition. At home, rep rate for the forward-reaching exercise well predicted UEFM score using an exponential model (LOOCV r2 = 0.69), but only after we re-estimated coefficients using the home data.

Discussion

These results show how a simple measure-exercise rep rate measured with simple sensors-can be used to infer an arm impairment score and suggest that prediction models should be tuned separately for the clinic and home environments.

Automatic Post-Stroke Severity Assessment Using Novel Unsupervised Consensus Learning for Wearable and Camera-Based Sensor Datasets

Stroke survivors often suffer from movement impairments that significantly affect their daily activities. The advancements in sensor technology and IoT have provided opportunities to automate the assessment and rehabilitation process for stroke survivors. This paper aims to provide a smart post-stroke severity assessment using AI-driven models. With the absence of labelled data and expert assessment, there is a research gap in providing virtual assessment, especially for unlabeled data. Inspired by the advances in consensus learning, in this paper, we propose a consensus clustering algorithm, PSA-NMF, that combines various clusterings into one united clustering, i.e., cluster consensus, to produce more stable and robust results compared to individual clustering. This paper is the first to investigate severity level using unsupervised learning and trunk displacement features in the frequency domain for post-stroke smart assessment. Two different methods of data collection from the U-limb datasets-the camera-based method (Vicon) and wearable sensor-based technology (Xsens)-were used. The trunk displacement method labelled each cluster based on the compensatory movements that stroke survivors employed for their daily activities. The proposed method uses the position and acceleration data in the frequency domain. Experimental results have demonstrated that the proposed clustering method that uses the post-stroke assessment approach increased the evaluation metrics such as accuracy and F-score. These findings can lead to a more effective and automated stroke rehabilitation process that is suitable for clinical settings, thus improving the quality of life for stroke survivors.

Attempt to Make the Upper-Limb Item of Objective Fugl-Meyer Assessment Using 9-Axis Motion Sensors

The Fugl-Meyer Assessment (FMA) has been used as a functional assessment of upper-limb function in stroke patients. This study aimed to create a more objective and standardized evaluation based on an FMA of the upper-limb items. A total of 30 first-ever stroke patients (65.3 ± 10.3 years old) and 15 healthy participants (35.4 ± 13.4 years old) admitted to Itami Kousei Neurosurgical Hospital were included. A nine-axis motion sensor was attached to the participants, and the joint angles of 17 upper-limb items (excluding fingers) and 23 FMA upper-limb items (excluding reflexes and fingers) were measured. From the measurement results, we analyzed the time-series data of each movement and obtained the correlation between the joint angles of each part. Discriminant analysis showed that 17 and 6 items had a concordance rate of ≥80% (80.0~95.6%) and <80% (64.4~75.6%), respectively. In the multiple regression analysis of continuous variables of FMA, a good regression model was obtained to predict the FMA with three to five joint angles. The discriminant analysis for 17 evaluation items suggests the possibility of roughly calculating FMA scores from joint angles.

Data-Driven Quantitation of Movement Abnormality after Stroke

Stroke commonly affects the ability of the upper extremities (UEs) to move normally. In clinical settings, identifying and measuring movement abnormality is challenging due to the imprecision and impracticality of available assessments. These challenges interfere with therapeutic tracking, communication, and treatment. We thus sought to develop an approach that blends precision and pragmatism, combining high-dimensional motion capture with out-of-distribution (OOD) detection. We used an array of wearable inertial measurement units to capture upper body motion in healthy and chronic stroke subjects performing a semi-structured, unconstrained 3D tabletop task. After data were labeled by human coders, we trained two deep learning models exclusively on healthy subject data to classify elemental movements (functional primitives). We tested these healthy subject-trained models on previously unseen healthy and stroke motion data. We found that model confidence, indexed by prediction probabilities, was generally high for healthy test data but significantly dropped when encountering OOD stroke data. Prediction probabilities worsened with more severe motor impairment categories and were directly correlated with individual impairment scores. Data inputs from the paretic UE, rather than trunk, most strongly influenced model confidence. We demonstrate for the first time that using OOD detection with high-dimensional motion data can reveal clinically meaningful movement abnormality in subjects with chronic stroke.

Survey of Movement Reproduction in Immersive Virtual Rehabilitation

Virtual reality (VR) has emerged as a powerful tool for rehabilitation. Many effective VR applications have been developed to support motor rehabilitation of people affected by motor issues. Movement reproduction, which transfers users' movements from the physical world to the virtual environment, is commonly used in VR rehabilitation applications. Three major components are required for movement reproduction in VR: (1) movement input, (2) movement representation, and (3) movement modulation. Until now, movement reproduction in virtual rehabilitation has not yet been systematically studied. This article aims to provide a state-of-the-art review on this subject by focusing on existing literature on immersive motor rehabilitation using VR. In this review, we provided in-depth discussions on the rehabilitation goals and outcomes, technology issues behind virtual rehabilitation, and user experience regarding movement reproduction. Similarly, we present good practices and highlight challenges and opportunities that can form constructive suggestions for the design and development of fit-for-purpose VR rehabilitation applications and can help frame future research directions for this emerging area that combines VR and health.

Assessment of movement disorders using wearable sensors during upper limb tasks: A scoping review

Background: Studies aiming to objectively quantify movement disorders during upper limb tasks using wearable sensors have recently increased, but there is a wide variety in described measurement and analyzing methods, hampering standardization of methods in research and clinics. Therefore, the primary objective of this review was to provide an overview of sensor set-up and type, included tasks, sensor features and methods used to quantify movement disorders during upper limb tasks in multiple pathological populations. The secondary objective was to identify the most sensitive sensor features for the detection and quantification of movement disorders on the one hand and to describe the clinical application of the proposed methods on the other hand. Methods: A literature search using Scopus, Web of Science, and PubMed was performed. Articles needed to meet following criteria: 1) participants were adults/children with a neurological disease, 2) (at least) one sensor was placed on the upper limb for evaluation of movement disorders during upper limb tasks, 3) comparisons between: groups with/without movement disorders, sensor features before/after intervention, or sensor features with a clinical scale for assessment of the movement disorder. 4) Outcome measures included sensor features from acceleration/angular velocity signals. Results: A total of 101 articles were included, of which 56 researched Parkinson's Disease. Wrist(s), hand(s) and index finger(s) were the most popular sensor locations. Most frequent tasks were: finger tapping, wrist pro/supination, keeping the arms extended in front of the body and finger-to-nose. Most frequently calculated sensor features were mean, standard deviation, root-mean-square, ranges, skewness, kurtosis/entropy of acceleration and/or angular velocity, in combination with dominant frequencies/power of acceleration signals. Examples of clinical applications were automatization of a clinical scale or discrimination between a patient/control group or different patient groups. Conclusion: Current overview can support clinicians and researchers in selecting the most sensitive pathology-dependent sensor features and methodologies for detection and quantification of upper limb movement disorders and objective evaluations of treatment effects. Insights from Parkinson's Disease studies can accelerate the development of wearable sensors protocols in the remaining pathologies, provided that there is sufficient attention for the standardisation of protocols, tasks, feasibility and data analysis methods.

Autonomous Exercise Generator for Upper Extremity Rehabilitation: A Fuzzy-Logic-Based Approach

In this paper, an autonomous exercise generation system based of fuzzy logic approach is presented. This work attempts to close a gap in the design of a completely autonomous robotic rehabilitation system that can recommend exercises to patients based on their data, such as shoulder range of motion (ROM) and muscle strength, from a pre-set library of exercises. The input parameters are fed into a system that uses Mamdani-style fuzzy logic rules to process them. In medical applications, the rationale behind decision making is a sophisticated process that involves a certain amount of uncertainty and ambiguity. In this instance, a fuzzy-logic-based system emerges as a viable option for dealing with the uncertainty. The system's rules have been reviewed by a therapist to ensure that it adheres to the relevant healthcare standards. Moreover, the system has been tested with a series of test data and the results obtained ensures the proposed idea's feasibility.

Quantitative Evaluation System of Wrist Motor Function for Stroke Patients Based on Force Feedback

Motor function evaluation is a significant part of post-stroke rehabilitation protocols, and the evaluation of wrist motor function helps provide patients with individualized rehabilitation training programs. However, traditional assessment is coarsely graded, lacks quantitative analysis, and relies heavily on clinical experience. In order to objectively quantify wrist motor dysfunction in stroke patients, a novel quantitative evaluation system based on force feedback and machine learning algorithm was proposed. Sensors embedded in the force-feedback robot record the kinematic and movement data of the subject, and the rehabilitation doctor used an evaluation scale to score the wrist function of the subject. The quantitative evaluation models of wrist motion function based on random forest (RF), support vector machine regression (SVR), k-nearest neighbor (KNN), and back propagation neural network (BPNN) were established, respectively. To verify the effectiveness of the proposed quantitative evaluation system, 25 stroke patients and 10 healthy volunteers were recruited in this study. Experimental results show that the evaluation accuracy of the four models is all above 88%. The accuracy of BPNN model is 94.26%, and the Pearson correlation coefficient between model prediction and clinician scores is 0.964, indicating that the BPNN model can accurately evaluate the wrist motor function for stroke patients. In addition, there was a significant correlation between the prediction score of the quantitative assessment system and the physician scale score (p < 0.05). The proposed system enables quantitative and refined assessment of wrist motor function in stroke patients and has the feasibility of helping rehabilitation physicians in evaluating patients’ motor function clinically.

Kinect v2-Assisted Semi-Automated Method to Assess Upper Limb Motor Performance in Children

The interruption of rehabilitation activities caused by the COVID-19 lockdown has significant health negative consequences for the population with physical disabilities. Thus, measuring the range of motion (ROM) using remotely taken photographs, which are then sent to specialists for formal assessment, has been recommended. Currently, low-cost Kinect motion capture sensors with a natural user interface are the most feasible implementations for upper limb motion analysis. An active range of motion (AROM) measuring system based on a Kinect v2 sensor for upper limb motion analysis using Fugl-Meyer Assessment (FMA) scoring is described in this paper. Two test groups of children, each having eighteen participants, were analyzed in the experimental stage, where upper limbs’ AROM and motor performance were assessed using FMA. Participants in the control group (mean age of 7.83 ± 2.54 years) had no cognitive impairment or upper limb musculoskeletal problems. The study test group comprised children aged 8.28 ± 2.32 years with spastic hemiparesis. A total of 30 samples of elbow flexion and 30 samples of shoulder abduction of both limbs for each participant were analyzed using the Kinect v2 sensor at 30 Hz. In both upper limbs, no significant differences (p < 0.05) in the measured angles and FMA assessments were observed between those obtained using the described Kinect v2-based system and those obtained directly using a universal goniometer. The measurement error achieved by the proposed system was less than ±1° compared to the specialist’s measurements. According to the obtained results, the developed measuring system is a good alternative and an effective tool for FMA assessment of AROM and motor performance of upper limbs, while avoiding direct contact in both healthy children and children with spastic hemiparesis.

Quantitative Evaluation System of Upper Limb Motor Function of Stroke Patients Based on Desktop Rehabilitation Robot

Rehabilitation training and movement evaluation after stroke have become a research hotspot as stroke has become a very common and harmful disease. However, traditional rehabilitation training and evaluation are mainly conducted under the guidance of rehabilitation doctors. The evaluation process is time-consuming and the evaluation results are greatly influenced by doctors. In this study, a desktop upper limb rehabilitation robot was designed and a quantitative evaluation system of upper limb motor function for stroke patients was proposed. The kinematics and dynamics data of stroke patients during active training were collected by sensors. Combined with the scores of patients' upper limb motor function by rehabilitation doctors using the Wolf Motor Function Test (WMFT) scale, three different quantitative evaluation models of upper limb motor function based on Back Propagation Neural Network (BPNN), K-Nearest Neighbors (KNN), and Support Vector Regression (SVR) algorithms were established. To verify the effectiveness of the quantitative evaluation system, 10 healthy subjects and 21 stroke patients were recruited for experiments. The experimental results show that the BPNN model has the best evaluation performance among the three quantitative evaluation models. The scoring accuracy of the BPNN model reached up to 87.1%. Moreover, there was a significant correlation between the models' scores and the doctors' scores. The proposed system can help doctors to quantitatively evaluate the upper limb motor function of stroke patients and accurately master the rehabilitation progress of patients.

Design of a Data Glove for Assessment of Hand Performance Using Supervised Machine Learning

The large number of poststroke recovery patients poses a burden on rehabilitation centers, hospitals, and physiotherapists. The advent of rehabilitation robotics and automated assessment systems can ease this burden by assisting in the rehabilitation of patients with a high level of recovery. This assistance will enable medical professionals to either better provide for patients with severe injuries or treat more patients. It also translates into financial assistance as well in the long run. This paper demonstrated an automated assessment system for in-home rehabilitation utilizing a data glove, a mobile application, and machine learning algorithms. The system can be used by poststroke patients with a high level of recovery to assess their performance. Furthermore, this assessment can be sent to a medical professional for supervision. Additionally, a comparison between two machine learning classifiers was performed on their assessment of physical exercises. The proposed system has an accuracy of 85% (±5.1%) with careful feature and classifier selection.

Proof-of-Concept of a Sensor-Based Evaluation Method for Better Sensitivity of Upper-Extremity Motor Function Assessment

In rehabilitation, the Fugl-Meyer assessment (FMA) is a typical clinical instrument to assess upper-extremity motor function of stroke patients, but it cannot measure fine changes of motor function (both in recovery and deterioration) due to its limited sensitivity. This paper introduces a sensor-based automated FMA system that addresses this limitation with a continuous rating algorithm. The system consists of a depth sensor (Kinect V2) and an algorithm to rate the continuous FM scale based on fuzzy inference. Using a binary logic based classification method developed from a linguistic scoring guideline of FMA, we designed fuzzy input/output variables, fuzzy rules, membership functions, and a defuzzification method for several representative FMA tests. A pilot trial with nine stroke patients was performed to test the feasibility of the proposed approach. The continuous FM scale from the proposed algorithm exhibited a high correlation with the clinician rated scores and the results showed the possibility of more sensitive upper-extremity motor function assessment.

Kinematic Evaluation via Inertial Measurement Unit Associated with Upper Extremity Motor Function in Subacute Stroke: A Cross-Sectional Study

Kinematic evaluation via portable sensor system has been increasingly applied in neurological sciences and clinical practice. However, conventional kinematic evaluation rarely extends the context beyond the motor impairment level. In addition, kinematic tasks with numerous items could be complex and time consuming that pose a burden to test applications and data processing. The study aimed to explore the correlation of finger-to-nose task (FNT) kinematics via Inertial Measurement Unit with upper limb motor function in subacute stroke. In this study, six FNT kinematic variables were used to measure movement time, smoothness, and velocity in 37 participants with subacute stroke. Upper limb motor function was evaluated with the Fugl-Meyer Assessment for Upper Extremity (FMA-UE), Action Research Arm Test (ARAT), and modified Barthel Index (MBI). As a result, mean velocity, peak velocity, and the number of movement units were associated with the clinical assessments. The multivariable linear regression models could estimate 55%, 51%, and 32% of variance in FMA-UE, ARAT, and MBI, respectively. In addition, age, gender, type of stroke, and paretic side had no significant effects on these associations. Results show that FNT kinematic variables measured via Inertial Measurement Unit are associated with upper extremity motor function in individuals with subacute stroke. The objective kinematic evaluation may be suitable for predicting clinical measures of motor impairment and capacity to understand upper extremity motor recovery and clinical decision making after stroke. This trial is registered with ChiCTR1900026656.

A Data-Driven Investigation on Surface Electromyography Based Clinical Assessment in Chronic Stroke

Background: Surface electromyography (sEMG) based robot-assisted rehabilitation systems have been adopted for chronic stroke survivors to regain upper limb motor function. However, the evaluation of rehabilitation effects during robot-assisted intervention relies on traditional manual assessments. This study aimed to develop a novel sEMG data-driven model for automated assessment. Method: A data-driven model based on a three-layer backpropagation neural network (BPNN) was constructed to map sEMG data to two widely used clinical scales, i.e., the Fugl-Meyer Assessment (FMA) and the Modified Ashworth Scale (MAS). Twenty-nine stroke participants were recruited in a 20-session sEMG-driven robot-assisted upper limb rehabilitation, which consisted of hand reaching and withdrawing tasks. The sEMG signals from four muscles in the paretic upper limbs, i.e., biceps brachii (BIC), triceps brachii (TRI), flexor digitorum (FD), and extensor digitorum (ED), were recorded before and after the intervention. Meanwhile, the corresponding clinical scales of FMA and MAS were measured manually by a blinded assessor. The sEMG features including Mean Absolute Value (MAV), Zero Crossing (ZC), Slope Sign Change (SSC), Root Mean Square (RMS), and Wavelength (WL) were adopted as the inputs to the data-driven model. The mapped clinical scores from the data-driven model were compared with the manual scores by Pearson correlation. Results: The BPNN, with 15 nodes in the hidden layer and sEMG features, i.e., MAV, ZC, SSC, and RMS, as the inputs to the model, was established to achieve the best mapping performance with significant correlations (r > 0.9, P < 0.001), according to the FMA. Significant correlations were also obtained between the mapped and manual FMA subscores, i.e., FMA-wrist/hand and FMA-shoulder/elbow, before and after the intervention (r > 0.9, P < 0.001). Significant correlations (P < 0.001) between the mapped and manual scores of MASs were achieved, with the correlation coefficients r = 0.91 at the fingers, 0.88 at the wrist, and 0.91 at the elbow after the intervention. Conclusion: An sEMG data-driven BPNN model was successfully developed. It could evaluate upper limb motor functions in chronic stroke and have potential application in automated assessment in post-stroke rehabilitation, once validated with large sample sizes. Clinical Trial Registration: www.ClinicalTrials.gov, identifier: NCT02117089.

Quantification of Motor Function Post-Stroke Using Novel Combination of Wearable Inertial and Mechanomyographic Sensors

Subjective clinical rating scales represent the gold-standard for diagnosis of motor function following stroke. In practice however, they suffer from well-recognized limitations including assessor variance, low inter-rater reliability and low resolution. Automated systems have been proposed for empirical quantification but have not significantly impacted clinical practice. We address translational challenges in this arena through: (1) implementation of a novel sensor suite combining inertial measurement and mechanomyography (MMG) to quantify hand and wrist motor function; and (2) introduction of a new range of signal features extracted from the suite to supplement predicted clinical scores. The wearable sensors, signal features, and machine learning algorithms have been combined to produce classified ratings from the Fugl-Meyer clinical assessment rating scale. Furthermore, we have designed the system to augment clinical rating with several sensor-derived supplementary features encompassing critical aspects of motor dysfunction (e.g. joint angle, muscle activity, etc.). Performance is validated through a large-scale study on a post-stroke cohort of 64 patients. Fugl-Meyer Assessment tasks were classified with 75% accuracy for gross motor tasks and 62% for hand/wrist motor tasks. Of greater import, supplementary features demonstrated concurrent validity with Fugl-Meyer ratings, evidencing their utility as new measures of motor function suited to automated assessment. Finally, the supplementary features also provide continuous measures of sub-components of motor function, offering the potential to complement low accuracy but well-validated clinical rating scales when high-quality motor outcome measures are required. We believe this work provides a basis for widespread clinical adoption of inertial-MMG sensor use for post-stroke clinical motor assessment.

A review of the evolution of scientific literature on technology-assisted approaches using RGB-D sensors for musculoskeletal health monitoring

The human musculoskeletal (MSK) system (also known as the locomotor system) provides strength and assistance to perform functional tasks and daily life activities. The MSK health monitoring plays a vital role in maintaining the body mobility and quality of life. Manual approaches for musculoskeletal health monitoring are subjective and require a clinician's intervention. The evolution in motion tracking technology enables us to capture the fine details of body movements. The research community has proposed various approaches to help clinicians in diagnosis and monitor treatment sessions. This paper succinctly reviews the evolution of technology-assisted approaches for musculoskeletal health monitoring, using motion capture sensors. To streamline the search through the literature database, the PICOS framework and PRISMA method have been incorporated. The present study reviews methods to transform motion capture data into kinematics variables and factors that affect the tracking performance of RGB-D sensors. Furthermore, widely utilized time-series filters for skeletal data denoising and smoothing for kinematics analysis, stochastic models for movement modeling, rule-based and template-based approaches for rehabilitation exercises assessment, and telerehabilitation sessions for remote health monitoring are explored. This article analyzes skeletal tracking methods by providing advantages and drawbacks of the state of the art rehabilitation sessions assessment, skeletal joint kinematics analysis, and MSK Telerehabilitation approaches. It also discusses the possible future research avenues to improve musculoskeletal disorder diagnosis and treatment monitoring. Our review signifies that RGB-D sensor-based approaches are inexpensive and portable for disorder diagnosis and treatment monitoring. It can also be a viable option for clinicians to provide contactless healthcare access to patients in the current scenario of the COVID-19 pandemic.

Quantification of the Therapist's Gentle Pull for Pinch Strength Testing Based on FMA and MMT: An Experimental Study with Healthy Subjects

Static pinch strength against a therapist’s gentle pull is evaluated using the pincer grasp component of the Fugl Meyer Assessment (FMA) to assess pinch impairment after stroke. In the pincer grasp component, therapists applied a gentle pull to distinguish between a score of 1 (moderate pinch impairment) and a score of 2 (no pinch impairment). The gentle pull is described as a resistance equivalent to a manual muscle test (MMT) score 4/5. The accepted use of “gentle” as a qualitative description for the pull results is a non-standardized subjective interpretation. The goal of this paper was to determine the quantitative value of the gentle pull applied by the therapists as in their clinical practice using a pinch–pull gripping system. The FMA protocol was used to standardize the body and fingers positions of three occupational therapists who were then instructed to apply a gentle pull of 4/5 MMT using their thumb and index fingers (in a tip-to-tip pinch). The results show that the therapists exerted a mean gentle pull (4/5 MMT score) of 6.34 ± 0.98 N with high reliability and acceptable repeatability. In investigating the ability of healthy subjects to resist the gentle pull, 50 adult male volunteers were instructed to pinch the pincer object and resist a dynamic loading exerted by the pinch–pull gripping system as much as possible to the moment the pincer object slips away. The results show that all subjects were able to exert a pulling force higher than the quantitative value of the gentle pull.

Automated classification of movement quality using the Microsoft Kinect V2 sensor

Practitioners commonly perform movement quality assessment through qualitative assessment protocols, which can be time-intensive and prone to inter-rater measurement bias. The advent of portable and inexpensive marker-less motion capture systems can improve assessment through objective joint kinematic analysis. The current study aimed to evaluate various machine learning models that used kinematic features from Kinect position data to classify a performer's Movement Competency Screen (MCS) score. A Kinect V2 sensor collected position data from 31 physically active males as they performed bilateral squat, forward lunge, and single-leg squat; and the movement quality was rated according to the MCS criteria. Features were extracted and selected from domain knowledge-based kinematic variables as model input. Multiclass logistic regression (MLR) was then performed to translate joint kinematics into MCS score. Performance indicators were calculated after a 10-fold cross validation of each model developed from Kinect-based kinematic variables. The analyses revealed that the models' sensitivity, specificity, and accuracy ranged from 0.66 to 0.89, 0.58 to 0.86, and 0.74 to 0.85, respectively. In conclusion, the Kinect-based automated movement quality assessment is a suitable, novel, and practical approach to movement quality assessment.

A Framework for User Adaptation and Profiling for Social Robotics in Rehabilitation

Physical rehabilitation therapies for children present a challenge, and its success-the improvement of the patient's condition-depends on many factors, such as the patient's attitude and motivation, the correct execution of the exercises prescribed by the specialist or his progressive recovery during the therapy. With the aim to increase the benefits of these therapies, social humanoid robots with a friendly aspect represent a promising tool not only to boost the interaction with the pediatric patient, but also to assist physicians in their work. To achieve both goals, it is essential to monitor in detail the patient's condition, trying to generate user profile models which enhance the feedback with both the system and the specialist. This paper describes how the project NAOTherapist-a robotic architecture for rehabilitation with social robots-has been upgraded in order to include a monitoring system able to generate user profile models through the interaction with the patient, performing user-adapted therapies. Furthermore, the system has been improved by integrating a machine learning algorithm which recognizes the pose adopted by the patient and by adding a clinical reports generation system based on the QUEST metric.

A review of computational approaches for evaluation of rehabilitation exercises

Recent advances in data analytics and computer-aided diagnostics stimulate the vision of patient-centric precision healthcare, where treatment plans are customized based on the health records and needs of every patient. In physical rehabilitation, the progress in machine learning and the advent of affordable and reliable motion capture sensors have been conducive to the development of approaches for automated assessment of patient performance and progress toward functional recovery. The presented study reviews computational approaches for evaluating patient performance in rehabilitation programs using motion capture systems. Such approaches will play an important role in supplementing traditional rehabilitation assessment performed by trained clinicians, and in assisting patients participating in home-based rehabilitation. The reviewed computational methods for exercise evaluation are grouped into three main categories: discrete movement score, rule-based, and template-based approaches. The review places an emphasis on the application of machine learning methods for movement evaluation in rehabilitation. Related work in the literature on data representation, feature engineering, movement segmentation, and scoring functions is presented. The study also reviews existing sensors for capturing rehabilitation movements and provides an informative listing of pertinent benchmark datasets. The significance of this paper is in being the first to provide a comprehensive review of computational methods for evaluation of patient performance in rehabilitation programs.

Quantitative Assessment of Upper-Limb Motor Function for Post-Stroke Rehabilitation Based on Motor Synergy Analysis and Multi-Modality Fusion

Functional assessment is an essential part of rehabilitation protocols after stroke. Conventionally, the assessment process relies heavily on clinical experience and lacks quantitative analysis. In order to objectively quantify the upper-limb motor impairments in patients with post-stroke hemiparesis, this study proposes a novel assessment approach based on motor synergy quantification and multi-modality fusion. Fifteen post-stroke hemiparetic patients and fifteen age-matched healthy persons participated in this study. During different goal-directed tasks, kinematic data and surface electromyography(sEMG) signals were synchronously collected from these participants, and then motor features extracted from each modal data could be fed into the respective local classifiers. In addition, kinematic synergies and muscle synergies were quantified by principal component analysis (PCA) and k weighted angular similarity ( k WAS) algorithm to provide in-depth analysis of the coactivated features responsible for observable movement impairments. By integrating the outputs of local classifiers and the quantification results of motor synergies, ensemble classifiers can be created to generate quantitative assessment for different modalities separately. In order to further exploit the complementarity between the evaluation results at kinematic and muscular levels, a multi-modal fusion scheme was developed to comprehensively analyze the upper-limb motor function and generate a probability-based function score. Under the proposed assessment framework, three types of machine learning methods were employed to search the optimal performance of each classifier. Experimental results demonstrated that the classification accuracy was respectively improved by 4.86% and 2.78% when the analysis of kinematic and muscle synergies was embedded in the assessment system, and could be further enhanced to 96.06% by fusing the characteristics derived from different modalities. Furthermore, the assessment result of multi-modality fusion framework exhibited a significant correlation with the score of standard clinical tests ( R = - 0.87, P = 1.98e - 5 ). These promising results show the feasibility of applying the proposed method to clinical assessments for post-stroke hemiparetic patients.

Estimating Upper-Limb Impairment Level in Stroke Survivors Using Wearable Inertial Sensors and a Minimally-Burdensome Motor Task

Upper-limb paresis is the most common motor impairment post stroke. Current solutions to automate the assessment of upper-limb impairment impose a number of critical burdens on patients and their caregivers that preclude frequent assessment. In this work, we propose an approach to estimate upper-limb impairment in stroke survivors using two wearable inertial sensors, on the wrist and the sternum, and a minimally-burdensome motor task. Twenty-three stroke survivors with no, mild, or moderate upper-limb impairment performed two repetitions of one-to-two minute-long continuous, random (i.e., patternless), voluntary upper-limb movements spanning the entire range of motion. The three-dimensional time-series of upper-limb movements were segmented into a series of one-dimensional submovements by employing a unique movement decomposition technique. An unsupervised clustering algorithm and a supervised regression model were used to estimate Fugl-Meyer Assessment (FMA) scores based on features extracted from these submovements. Our regression model estimated FMA scores with a normalized root mean square error of 18.2% ( r²=0.70 ) and needed as little as one minute of movement data to yield reasonable estimation performance. These results support the possibility of frequently monitoring stroke survivors' rehabilitation outcomes, ultimately enabling the development of individually-tailored rehabilitation programs.

Cellphone-Based Automated Fugl-Meyer Assessment to Evaluate Upper Extremity Motor Function After Stroke

The Fugl-Meyer Assessment (FMA) is a widely used evaluation tool for assessing upper extremity motor function during stroke rehabilitation. However, the FMA is a repetitive, time-consuming task that currently must be performed by therapists in a hospital or clinic. We thus propose an alternative automated approach in which patients perform FMA movements while holding a cellphone at the hand and receive automated FMA scores. In the proposed system, features are extracted from cellphone movement data and decision trees are used to automatically score FMA test items. Ten stroke patients with upper extremity dysfunction participated in a validation experiment to compare automated FMA scores with traditional FMA scores from a trained therapist. Results showed that FMA scores from the cellphone-based automated system were highly correlated with FMA scores from the trained therapist (r² = 0.97), and that the average accuracy for individual FMA test items was 85%. These results demonstrate that such a portable, automated FMA system could potentially be used to assess upper extremity function during stroke rehabilitation to remove the repetitive, time-consuming burden from therapists and could potentially be performed in clinic or home-based settings.

Development of a biological signal-based evaluator for robot-assisted upper-limb rehabilitation: a pilot study

Bio-signal based assessment for upper-limb functions is an attractive technology for rehabilitation. In this work, an upper-limb function evaluator is developed based on biological signals, which could be used for selecting different robotic training protocols. Interaction force (IF) and participation level (PL, processed surface electromyography (sEMG) signals) are used as the key bio-signal inputs for the evaluator. Accordingly, a robot-based standardized performance testing (SPT) is developed to measure these key bio-signal data. Moreover, fuzzy logic is used to regulate biological signals, and a rules-based selector is then developed to select different training protocols. To the authors' knowledge, studies focused on biological signal-based evaluator for selecting robotic training protocols, especially for robot-based bilateral rehabilitation, has not yet been reported in literature. The implementation of SPT and fuzzy logic to measure and process key bio-signal data with a rehabilitation robot system is the first of its kind. Five healthy participants were then recruited to test the performance of the SPT, fuzzy logic and evaluator in three different conditions (tasks). The results show: (1) the developed SPT has an ability to measure precise bio-signal data from participants; (2) the utilized fuzzy logic has an ability to process the measured data with the accuracy of 86.7% and 100% for the IF and PL respectively; and (3) the proposed evaluator has an ability to distinguish the intensity of biological signals and thus to select different robotic training protocols. The results from the proposed evaluator, and biological signals measured from healthy people could also be used to standardize the criteria to assess the results of stroke patients later.

Capturing Upper Limb Gross Motor Categories Using the Kinect® Sensor

IMPORTANCE

Along with growth in telerehabilitation, a concurrent need has arisen for standardized methods of tele-evaluation.

OBJECTIVE

To examine the feasibility of using the Kinect sensor in an objective, computerized clinical assessment of upper limb motor categories.

DESIGN

We developed a computerized Mallet classification using the Kinect sensor. Accuracy of computer scoring was assessed on the basis of reference scores determined collaboratively by multiple evaluators from reviewing video recording of movements. In addition, using the reference score, we assessed the accuracy of the typical clinical procedure in which scores were determined immediately on the basis of visual observation. The accuracy of the computer scores was compared with that of the typical clinical procedure.

SETTING

Research laboratory.

PARTICIPANTS

Seven patients with stroke and 10 healthy adult participants. Healthy participants intentionally achieved predetermined scores.

OUTCOMES AND MEASURES

Accuracy of the computer scores in comparison with accuracy of the typical clinical procedure (immediate visual assessment).

RESULTS

The computerized assessment placed participants' upper limb movements in motor categories as accurately as did typical clinical procedures.

CONCLUSIONS AND RELEVANCE

Computerized clinical assessment using the Kinect sensor promises to facilitate tele-evaluation and complement telehealth applications.

WHAT THIS ARTICLE ADDS

Computerized clinical assessment can enable patients to conduct evaluations remotely in their homes without therapists present.

Automatic Outcome in Manual Dexterity Assessment Using Colour Segmentation and Nearest Neighbour Classifier

Objective assessment of motor function is an important component to evaluating the effectiveness of a rehabilitation process. Such assessments are carried out by clinicians using traditional tests and scales. The Box and Blocks Test (BBT) is one such scale, focusing on manual dexterity evaluation. The score is the maximum number of cubes that a person is able to displace during a time window. In a previous paper, an automated version of the Box and Blocks Test using a Microsoft Kinect sensor was presented, and referred to as the Automated Box and Blocks Test (ABBT). In this paper, the feasibility of ABBT as an automated tool for manual dexterity assessment is discussed. An algorithm, based on image segmentation in CIELab colour space and the Nearest Neighbour (NN) rule, was developed to improve the reliability of automatic cube counting. A pilot study was conducted to assess the hand motor function in people with Parkinson’s disease (PD). Three functional assessments were carried out. The success rate in automatic cube counting was studied by comparing the manual (BBT) and the automatic (ABBT) methods. The additional information provided by the ABBT was analysed to discuss its clinical significance. The results show a high correlation between manual (BBT) and automatic (ABBT) scoring. The lowest average success rate in cube counting for ABBT was 92%. Additionally, the ABBT acquires extra information from the cubes’ displacement, such as the average velocity and the time instants in which the cube was detected. The analysis of this information can be related to indicators of health status (coordination and dexterity). The results showed that the ABBT is a useful tool for automating the assessment of unilateral gross manual dexterity, and provides additional information about the user’s performance.

Objective motor assessment for personalized rehabilitation of upper extremity in brain injury patients

BACKGROUND

The increasing number of patients with acquired brain injury and the current subjectivity of the conventional upper extremity (UE) assessment tests require new objective assessment techniques.

OBJECTIVE

This research proposes a novel objective motor assessment (OMA) methodology based on the Fugl-Meyer assessment (FMA). The goals are to automatically calculate the objective scores (OSs) of FMA-UE movements (as well as a global OS) and to interpret the estimated OSs.

METHODS

Fifteen patients participated in the study. The OMA algorithm was designed to detect small-scale variations in UE movements. The OSs for 14 FMA-UE movements and the global OSs were automatically calculated using the algorithm and evaluated by 2 therapists. The interpretation of the global OSs was performed at 3 levels: by item, movement and globally.

RESULTS

The global OSs calculated by our algorithm had a significant correlation with the therapists' scores (0.783 and 0.938, p <  0.01). All OSs for each movement were correlated with the scores given by the therapists. The correlation coefficient can reach values as high as 0.981 (p <  0.01).

CONCLUSIONS

We provide a new objective assessment tool for therapists to help them improve the diagnostic accuracy and to achieve a more personalized and potentially effective physical rehabilitation of brain injury patients.

A Review of Robotics in Neurorehabilitation: Towards an Automated Process for Upper Limb

Robot-mediated neurorehabilitation is a growing field that seeks to incorporate advances in robotics combined with neuroscience and rehabilitation to define new methods for treating problems related with neurological diseases. In this paper, a systematic literature review is conducted to identify the contribution of robotics for upper limb neurorehabilitation, highlighting its relation with the rehabilitation cycle, and to clarify the prospective research directions in the development of more autonomous rehabilitation processes. With this aim, first, a study and definition of a general rehabilitation process are made, and then, it is particularized for the case of neurorehabilitation, identifying the components involved in the cycle and their degree of interaction between them. Next, this generic process is compared with the current literature in robotics focused on upper limb treatment, analyzing which components of this rehabilitation cycle are being investigated. Finally, the challenges and opportunities to obtain more autonomous rehabilitation processes are discussed. In addition, based on this study, a series of technical requirements that should be taken into account when designing and implementing autonomous robotic systems for rehabilitation is presented and discussed.

A Preliminary Study on Precision Image Guidance for Electrode Placement in an EEG Study

Conventional methods for positioning electroencephalography electrodes according to the international 10/20 system are based on the manual identification of the principal 10/20 landmarks via visual inspection and palpation, inducing intersession variations in their determined locations due to structural ambiguity or poor visibility. To address the variation issue, we propose an image guidance system for precision electrode placement. Following the electrode placement according to the 10/20 system, affixed electrodes are laser-scanned together with the facial surface. For subsequent procedures, the laser scan is conducted likewise after positioning the electrodes in an arbitrary manner, and following the measurement of fiducial electrode locations, frame matching is performed to determine a transformation from the coordinate frame of the position tracker to that of the laser-scanned image. Finally, by registering the intra-procedural scan of the facial surface to the reference scan, the current tracking data of the electrodes can be visualized relative to the reference goal positions without manually measuring the four principal landmarks for each trial. The experimental results confirmed that use of the electrode navigation system significantly improved the electrode placement precision compared to the conventional 10/20 system (p < 0.005). The proposed system showed the possibility of precise image-guided electrode placement as an alternative to the conventional manual 10/20 system.

Motor Function Evaluation of Hemiplegic Upper-Extremities Using Data Fusion from Wearable Inertial and Surface EMG Sensors

Quantitative evaluation of motor function is of great demand for monitoring clinical outcome of applied interventions and further guiding the establishment of therapeutic protocol. This study proposes a novel framework for evaluating upper limb motor function based on data fusion from inertial measurement units (IMUs) and surface electromyography (EMG) sensors. With wearable sensors worn on the tested upper limbs, subjects were asked to perform eleven straightforward, specifically designed canonical upper-limb functional tasks. A series of machine learning algorithms were applied to the recorded motion data to produce evaluation indicators, which is able to reflect the level of upper-limb motor function abnormality. Sixteen healthy subjects and eighteen stroke subjects with substantial hemiparesis were recruited in the experiment. The combined IMU and EMG data yielded superior performance over the IMU data alone and the EMG data alone, in terms of decreased normal data variation rate (NDVR) and improved determination coefficient (DC) from a regression analysis between the derived indicator and routine clinical assessment score. Three common unsupervised learning algorithms achieved comparable performance with NDVR around 10% and strong DC around 0.85. By contrast, the use of a supervised algorithm was able to dramatically decrease the NDVR to 6.55%. With the proposed framework, all the produced indicators demonstrated high agreement with the routine clinical assessment scale, indicating their capability of assessing upper-limb motor functions. This study offers a feasible solution to motor function assessment in an objective and quantitative manner, especially suitable for home and community use.

Quaternion-Based Gesture Recognition Using Wireless Wearable Motion Capture Sensors

This work presents the development and implementation of a unified multi-sensor human motion capture and gesture recognition system that can distinguish between and classify six different gestures. Data was collected from eleven participants using a subset of five wireless motion sensors (inertial measurement units) attached to their arms and upper body from a complete motion capture system. We compare Support Vector Machines and Artificial Neural Networks on the same dataset under two different scenarios and evaluate the results. Our study indicates that near perfect classification accuracies are achievable for small gestures and that the speed of classification is sufficient to allow interactivity. However, such accuracies are more difficult to obtain when a participant does not participate in training, indicating that more work needs to be done in this area to create a system that can be used by the general population.

Fusion of Haptic and Gesture Sensors for Rehabilitation of Bimanual Coordination and Dexterous Manipulation

Disabilities after neural injury, such as stroke, bring tremendous burden to patients, families and society. Besides the conventional constrained-induced training with a paretic arm, bilateral rehabilitation training involves both the ipsilateral and contralateral sides of the neural injury, fitting well with the fact that both arms are needed in common activities of daily living (ADLs), and can promote good functional recovery. In this work, the fusion of a gesture sensor and a haptic sensor with force feedback capabilities has enabled a bilateral rehabilitation training therapy. The Leap Motion gesture sensor detects the motion of the healthy hand, and the omega.7 device can detect and assist the paretic hand, according to the designed cooperative task paradigm, as much as needed, with active force feedback to accomplish the manipulation task. A virtual scenario has been built up, and the motion and force data facilitate instantaneous visual and audio feedback, as well as further analysis of the functional capabilities of the patient. This task-oriented bimanual training paradigm recruits the sensory, motor and cognitive aspects of the patient into one loop, encourages the active involvement of the patients into rehabilitation training, strengthens the cooperation of both the healthy and impaired hands, challenges the dexterous manipulation capability of the paretic hand, suits easy of use at home or centralized institutions and, thus, promises effective potentials for rehabilitation training.