A comparative analysis of three non-invasive human-machine interfaces for the disabled

A Multimodal Bracelet to Acquire Muscular Activity and Gyroscopic Data to Study Sensor Fusion for Intent Detection

Researchers have attempted to control robotic hands and prostheses through biosignals but could not match the human hand. Surface electromyography records electrical muscle activity using non-invasive electrodes and has been the primary method in most studies. While surface electromyography-based hand motion decoding shows promise, it has not yet met the requirements for reliable use. Combining different sensing modalities has been shown to improve hand gesture classification accuracy. This work introduces a multimodal bracelet that integrates a 24-channel force myography system with six commercial surface electromyography sensors, each containing a six-axis inertial measurement unit. The device's functionality was tested by acquiring muscular activity with the proposed device from five participants performing five different gestures in a random order. A random forest model was then used to classify the performed gestures from the acquired signal. The results confirmed the device's functionality, making it suitable to study sensor fusion for intent detection in future studies. The results showed that combining all modalities yielded the highest classification accuracies across all participants, reaching 92.3±2.6% on average, effectively reducing misclassifications by 37% and 22% compared to using surface electromyography and force myography individually as input signals, respectively. This demonstrates the potential benefits of sensor fusion for more robust and accurate hand gesture classification and paves the way for advanced control of robotic and prosthetic hands.

A Wearable Force Myography-Based Armband for Recognition of Upper Limb Gestures

Force myography (FMG) represents a promising alternative to surface electromyography (EMG) in the context of controlling bio-robotic hands. In this study, we built upon our prior research by introducing a novel wearable armband based on FMG technology, which integrates force-sensitive resistor (FSR) sensors housed in newly designed casings. We evaluated the sensors' characteristics, including their load-voltage relationship and signal stability during the execution of gestures over time. Two sensor arrangements were evaluated: arrangement A, featuring sensors spaced at 4.5 cm intervals, and arrangement B, with sensors distributed evenly along the forearm. The data collection involved six participants, including three individuals with trans-radial amputations, who performed nine upper limb gestures. The prediction performance was assessed using support vector machines (SVMs) and k-nearest neighbor (KNN) algorithms for both sensor arrangments. The results revealed that the developed sensor exhibited non-linear behavior, and its sensitivity varied with the applied force. Notably, arrangement B outperformed arrangement A in classifying the nine gestures, with an average accuracy of 95.4 ± 2.1% compared to arrangement A's 91.3 ± 2.3%. The utilization of the arrangement B armband led to a substantial increase in the average prediction accuracy, demonstrating an improvement of up to 4.5%.

Feasibility of force myography for the direct control of an assistive robotic hand orthosis in non-impaired individuals

BACKGROUND

Assistive robotic hand orthoses can support people with sensorimotor hand impairment in many activities of daily living and therefore help to regain independence. However, in order for the users to fully benefit from the functionalities of such devices, a safe and reliable way to detect their movement intention for device control is crucial. Gesture recognition based on force myography measuring volumetric changes in the muscles during contraction has been previously shown to be a viable and easy to implement strategy to control hand prostheses. Whether this approach could be efficiently applied to intuitively control an assistive robotic hand orthosis remains to be investigated.

METHODS

In this work, we assessed the feasibility of using force myography measured from the forearm to control a robotic hand orthosis worn on the hand ipsilateral to the measurement site. In ten neurologically-intact participants wearing a robotic hand orthosis, we collected data for four gestures trained in nine arm configurations, i.e., seven static positions and two dynamic movements, corresponding to typical activities of daily living conditions. In an offline analysis, we determined classification accuracies for two binary classifiers (one for opening and one for closing) and further assessed the impact of individual training arm configurations on the overall performance.

RESULTS

We achieved an overall classification accuracy of 92.9% (averaged over two binary classifiers, individual accuracies 95.5% and 90.3%, respectively) but found a large variation in performance between participants, ranging from 75.4 up to 100%. Averaged inference times per sample were measured below 0.15 ms. Further, we found that the number of training arm configurations could be reduced from nine to six without notably decreasing classification performance.

CONCLUSION

The results of this work support the general feasibility of using force myography as an intuitive intention detection strategy for a robotic hand orthosis. Further, the findings also generated valuable insights into challenges and potential ways to overcome them in view of applying such technologies for assisting people with sensorimotor hand impairment during activities of daily living.

Assessment of Low-Density Force Myography Armband for Classification of Upper Limb Gestures

Using force myography (FMG) to monitor volumetric changes in limb muscles is a promising and effective alternative for controlling bio-robotic prosthetic devices. In recent years, there has been a focus on developing new methods to improve the performance of FMG technology in the control of bio-robotic devices. This study aimed to design and evaluate a novel low-density FMG (LD-FMG) armband for controlling upper limb prostheses. The study investigated the number of sensors and sampling rate for the newly developed LD-FMG band. The performance of the band was evaluated by detecting nine gestures of the hand, wrist, and forearm at varying elbow and shoulder positions. Six subjects, including both fit and amputated individuals, participated in this study and completed two experimental protocols: static and dynamic. The static protocol measured volumetric changes in forearm muscles at the fixed elbow and shoulder positions. In contrast, the dynamic protocol included continuous motion of the elbow and shoulder joints. The results showed that the number of sensors significantly impacts gesture prediction accuracy, with the best accuracy achieved on the 7-sensor FMG band arrangement. Compared to the number of sensors, the sampling rate had a lower influence on prediction accuracy. Additionally, variations in limb position greatly affect the classification accuracy of gestures. The static protocol shows an accuracy above 90% when considering nine gestures. Among dynamic results, shoulder movement shows the least classification error compared to elbow and elbow-shoulder (ES) movements.

Unobtrusive, natural support control of an adaptive industrial exoskeleton using force myography

Repetitive or tiring tasks and movements during manual work can lead to serious musculoskeletal disorders and, consequently, to monetary damage for both the worker and the employer. Among the most common of these tasks is overhead working while operating a heavy tool, such as drilling, painting, and decorating. In such scenarios, it is desirable to provide adaptive support in order to take some of the load off the shoulder joint as needed. However, even to this day, hardly any viable approaches have been tested, which could enable the user to control such assistive devices naturally and in real time. Here, we present and assess the adaptive Paexo Shoulder exoskeleton, an unobtrusive device explicitly designed for this kind of industrial scenario, which can provide a variable amount of support to the shoulders and arms of a user engaged in overhead work. The adaptive Paexo Shoulder exoskeleton is controlled through machine learning applied to force myography. The controller is able to determine the lifted mass and provide the required support in real time. Twelve subjects joined a user study comparing the Paexo driven through this adaptive control to the Paexo locked in a fixed level of support. The results showed that the machine learning algorithm can successfully adapt the level of assistance to the lifted mass. Specifically, adaptive assistance can sensibly reduce the muscle activity's sensitivity to the lifted mass, with an observed relative reduction of up to 31% of the muscular activity observed when lifting 2 kg normalized by the baseline when lifting no mass.

A force-based human machine interface to drive a motorized upper limb exoskeleton. a pilot study

Muscular dystrophy is a strongly invalidating disease that causes the progressive loss of motor skills. The use of assistive devices, especially those in support of the upper limb, can increase the ability to perform daily-life activities and foster a partial recovery of the lost motor functionalities. However, for the use of these devices to be truly effective and accepted by patients, their activation must coincide with the user's intention to move. This work describes a new human-machine interface based on the integration of a six-axis force sensor to drive an upper limb motorized exoskeleton. This novel system can detect the patient's intention to move and produce displacements of the robotic device that are of magnitude and direction consistent with the user's wishes. The integration of the force-sensor interface in the BRIDGE/EMPATIA exoskeletal system was successful, and tests performed on both healthy and dystrophic subjects showed promising results, especially for the execution of planar movements.

A Way of Bionic Control Based on EI, EMG, and FMG Signals

Creating highly functional prosthetic, orthotic, and rehabilitation devices is a socially relevant scientific and engineering task. Currently, certain constraints hamper the development of such devices. The primary constraint is the lack of an intuitive and reliable control interface working between the organism and the actuator. The critical point in developing these devices and systems is determining the type and parameters of movements based on control signals recorded on an extremity. In the study, we investigate the simultaneous acquisition of electric impedance (EI), electromyography (EMG), and force myography (FMG) signals during basic wrist movements: grasping, flexion/extension, and rotation. For investigation, a laboratory instrumentation and software test setup were made for registering signals and collecting data. The analysis of the acquired signals revealed that the EI signals in conjunction with the analysis of EMG and FMG signals could potentially be highly informative in anthropomorphic control systems. The study results confirm that the comprehensive real-time analysis of EI, EMG, and FMG signals potentially allows implementing the method of anthropomorphic and proportional control with an acceptable delay.

Investigation on the Sampling Frequency and Channel Number for Force Myography Based Hand Gesture Recognition

Force myography (FMG) is a method that uses pressure sensors to measure muscle contraction indirectly. Compared with the conventional approach utilizing myoelectric signals in hand gesture recognition, it is a valuable substitute. To achieve the aim of gesture recognition at minimum cost, it is necessary to study the minimum sampling frequency and the minimal number of channels. For purpose of investigating the effect of sampling frequency and the number of channels on the accuracy of gesture recognition, a hardware system that has 16 channels has been designed for capturing forearm FMG signals with a maximum sampling frequency of 1 kHz. Using this acquisition equipment, a force myography database containing 10 subjects' data has been created. In this paper, gesture accuracies under different sampling frequencies and channel's number are obtained. Under 1 kHz sampling rate and 16 channels, four of five tested classifiers reach an accuracy up to about 99%. Other experimental results indicate that: (1) the sampling frequency of the FMG signal can be as low as 5 Hz for the recognition of static movements; (2) the reduction of channel number has a large impact on the accuracy, and the suggested channel number for gesture recognition is eight; and (3) the distribution of the sensors on the forearm would affect the recognition accuracy, and it is possible to improve the accuracy via optimizing the sensor position.

A Multi-DoF Prosthetic Hand Finger Joint Controller for Wearable sEMG Sensors by Nonlinear Autoregressive Exogenous Model

The loss of mobility function and sensory information from the arm, hand, and fingertips hampers the activities of daily living (ADL) of patients. A modern bionic prosthetic hand can compensate for the lost functions and realize multiple degree of freedom (DoF) movements. However, the commercially available prosthetic hands usually have limited DoFs due to limited sensors and lack of stable classification algorithms. This study aimed to propose a controller for finger joint angle estimation by surface electromyography (sEMG). The sEMG data used for training were gathered with the Myo armband, which is a commercial EMG sensor. Two features in the time domain were extracted and fed into a nonlinear autoregressive model with exogenous inputs (NARX). The NARX model was trained with pre-selected parameters using the Levenberg-Marquardt algorithm. Comparing with the targets, the regression correlation coefficient (R) of the model outputs was more than 0.982 over all test subjects, and the mean square error was less than 10.02 for a signal range in arbitrary units equal to [0, 255]. The study also demonstrated that the proposed model could be used in daily life movements with good accuracy and generalization abilities.

Improved hand prostheses control for transradial amputees based on hybrid of voice recognition and electromyography

The control of prostheses and their complexities is one of the greatest challenges limiting wide amputees' use of upper limb prostheses. The main challenges include the difficulty of extracting signals for controlling the prostheses, limited number of degrees of freedom (DoF), and cost-prohibitive for complex controlling systems. In this study, a real-time hybrid control system, based on electromyography (EMG) and voice commands (VC) is designed to render the prosthesis more dexterous with the ability to accomplish amputee's daily activities proficiently. The voice and EMG systems were combined in three proposed hybrid strategies, each strategy had different number of movements depending on the combination protocol between voice and EMG control systems. Furthermore, the designed control system might serve a large number of amputees with different amputation levels, and since it has a reasonable cost and be easy to use. The performance of the proposed control system, based on hybrid strategies, was tested by intact-limbed and amputee participants for controlling the HANDi hand. The results showed that the proposed hybrid control system was robust, feasible, with an accuracy of 94%, 98%, and 99% for Strategies 1, 2, and 3, respectively. It was possible to specify the grip force applied to the prosthetic hand within three gripping forces. The amputees participated in this study preferred combination Strategy 3 where the voice and EMG are working concurrently, with an accuracy of 99%.

A comparative study of motion detection with FMG and sEMG methods for assistive applications

Introduction

While surface-electromyography (sEMG) has been widely used in limb motion detection for the control of exoskeleton, there is an increasing interest to use forcemyography (FMG) method to detect motion. In this paper, we review the applications of two types of motion detection methods. Their performances were experimentally compared in day-to-day classification of forearm motions. The objective is to select a detection method suitable for motion assistance on a daily basis.

Methods

Comparisons of motion detection with FMG and sEMG were carried out considering classification accuracy (CA), repeatability and training scheme. For both methods, classification of motions was achieved through feed-forward neural network. Repeatability was evaluated on the basis of change in CA between days and also training schemes.

Results

The experiments shows that day-to-day CA with FMG can reach 84.9%, compared with a CA of 77.8% with sEMG, when the classifiers were trained only on the first day. Moreover, the CA with FMG can reach to 86.5%, comparable to CA of 84.1% with sEMG, if classifiers were trained daily.

Conclusions

Results suggest that data recorded from FMG is more repeatable in day-to-day testing and therefore FMG-based methods can be more useful than sEMG-based methods for motion detection in applications where exoskeletons are used as needed on a daily basis.

Effective Multi-Mode Grasping Assistance Control of a Soft Hand Exoskeleton Using Force Myography

Human intention detection is fundamental to the control of robotic devices in order to assist humans according to their needs. This paper presents a novel approach for detecting hand motion intention, i.e., rest, open, close, and grasp, and grasping force estimation using force myography (FMG). The output is further used to control a soft hand exoskeleton called an SEM Glove. In this method, two sensor bands constructed using force sensing resistor (FSR) sensors are utilized to detect hand motion states and muscle activities. Upon placing both bands on an arm, the sensors can measure normal forces caused by muscle contraction/relaxation. Afterwards, the sensor data is processed, and hand motions are identified through a threshold-based classification method. The developed method has been tested on human subjects for object-grasping tasks. The results show that the developed method can detect hand motions accurately and to provide assistance w.r.t to the task requirement.

Action interference in simultaneous and proportional myocontrol: comparing force- and electromyography

Myocontrol, that is, control of a prosthesis via muscle signals, is still a surprisingly hard problem. Recent research indicates that surface electromyography (sEMG), the traditional technique used to detect a subject's intent, could proficiently be replaced, or conjoined with, other techniques (multi-modal myocontrol), with the aim to improve both on dexterity and reliability. Objective. In this paper we present an online assessment of multi-modal sEMG and force myography (FMG) targeted at hand and wrist myocontrol. Approach. Twenty sEMG and FMG sensors in total were used to enforce simultaneous and proportional control of hand opening/closing, wrist pronation/supination and wrist flexion/extension of 12 intact subjects. Main results and Significance. We found that FMG yields in general a better performance than sEMG, and that the main drawback of the sEMG array we used is not the inability to perform a desired action, but rather action interference, that is, the undesired concurrent activation of another action. FMG, on the other hand, causes less interference.

Integrated Optical Fiber Force Myography Sensor as Pervasive Predictor of Hand Postures

Force myography (FMG) is an appealing alternative to traditional electromyography in biomedical applications, mainly due to its simpler signal pattern and immunity to electrical interference. Most FMG sensors, however, send data to a computer for further processing, which reduces the user mobility and, thus, the chances for practical application. In this sense, this work proposes to remodel a typical optical fiber FMG sensor with smaller portable components. Moreover, all data acquisition and processing routines were migrated to a Raspberry Pi 3 Model B microprocessor, ensuring the comfort of use and portability. The sensor was successfully demonstrated for 2 input channels and 9 postures classification with an average precision and accuracy of ~99.5% and ~99.8%, respectively, using a feedforward artificial neural network of 2 hidden layers and a competitive output layer.

Comparative Analysis of Wearable A-Mode Ultrasound and sEMG for Muscle-Computer Interface

OBJECTIVE

While surface electromyography (sEMG) is still dominant in the field of muscle-computer interface, ultrasound (US) sensing has been regarded as a promising alternative to sEMG, owing to its ability to precisely monitor muscle deformations. Among different US modalities, A-mode US is more compact and cost-effective for wearable applications against its cumbersome B-mode counterpart. In this article, we conduct a comprehensive comparison of wearable A-mode US and sEMG on gesture recognition and isometric muscle contraction force estimation.

METHODS

We experimented with eight types of gesture, with a range of 0-60% maximum voluntary contraction for each motion.

RESULTS

Results show that A-mode US outperforms sEMG on gesture recognition accuracy, robustness, and discrete force estimation accuracy, while sEMG is superior to US on continuous force estimation accuracy and ease of use in force estimation. Moreover, an extended online experiment demonstrates that the complementary advantages of US and sEMG on gesture recognition and continuous force estimation can be combined for the achievement of multi-class proportional gesture control.

SIGNIFICANCE

This article demonstrates the potential of A-mode US in automated gesture recognition, and the prospect of sEMG/US fusion for proportional gesture interaction.

Estimation of User-Applied Isometric Force/Torque Using Upper Extremity Force Myography

Hand force estimation is critical for applications that involve physical human-machine interactions for force monitoring and machine control. Force Myography (FMG) is a potential technique to be used for estimating hand force/torque. The FMG signals reflect the volumetric changes in the arm muscles due to muscle contraction or expansion. This paper investigates the feasibility of employing force-sensing resistors (FSRs) worn on the arm to measure the FMG signals for isometric force/torque estimation. Nine participants were recruited in this study and were asked to exert isometric force along three perpendicular axes, torque about the same three axes, and force and torque simultaneously. During the tests, the isometric force and torque were measured using a 6-degree-of-freedom (DoF) (i.e., force in three axes and torque around the same axes) load cell for ground truth labels whereas the FMG signals were recorded using a total number of 60 FSRs, which were embedded into four bands worn on the different locations of the arm. A two-stage regression strategy was employed to enhance the performance of the FMG bands, where three regression algorithms including general regression neural network (GRNN), support vector regression (SVR), and random forest regression (RF) models were employed, respectively, in the first stage and GRNN was used in the second stage. Two cases were considered to explore the performance of the FMG bands in estimating: (1) 3-DoF force and 3-DoF torque at once and (2) 6-DoF force and torque. In addition, the impact of sensor placement and the spatial coverage of FMG measurements were studied. This preliminary investigation demonstrates promising potential of FMG to estimate multi-DoF isometric force/torque. Specifically, R² accuracies of 0.83 for the 3-DoF force, 0.84 for 3-DoF torque, and 0.77 for the combination of force and torque (6-DoF) regressions were obtained using the four bands on the arm in cross-trial evaluation.

Tactile Signatures and Hand Motion Intent Recognition for Wearable Assistive Devices

Within the field of robotics and autonomous systems where there is a human in the loop, intent recognition plays an important role. This is especially true for wearable assistive devices used for rehabilitation, particularly post-stroke recovery. This paper reports results on the use of tactile patterns to detect weak muscle contractions in the forearm while at the same time associating these patterns with the muscle synergies during different grips. To investigate this concept, a series of experiments with healthy participants were carried out using a tactile arm brace (TAB) on the forearm while performing four different types of grip. The expected force patterns were established by analysing the muscle synergies of the four grip types and the forearm physiology. The results showed that the tactile signatures of the forearm recorded on the TAB align with the anticipated force patterns. Furthermore, a linear separability of the data across all four grip types was identified. Using the TAB data, machine learning algorithms achieved a 99% classification accuracy. The TAB results were highly comparable to a similar commercial intent recognition system based on a surface electromyography (sEMG) sensing.

A Review of Force Myography Research and Development

Information about limb movements can be used for monitoring physical activities or for human-machine-interface applications. In recent years, a technique called Force Myography (FMG) has gained ever-increasing traction among researchers to extract such information. FMG uses force sensors to register the variation of muscle stiffness patterns around a limb during different movements. Using machine learning algorithms, researchers are able to predict many different limb activities. This review paper presents state-of-art research and development on FMG technology in the past 20 years. It summarizes the research progress in both the hardware design and the signal processing techniques. It also discusses the challenges that need to be solved before FMG can be used in an everyday scenario. This paper aims to provide new insight into FMG technology and contribute to its advancement.

Metal oxide semiconductor nanomembrane-based soft unnoticeable multifunctional electronics for wearable human-machine interfaces

Wearable human-machine interfaces (HMIs) are an important class of devices that enable human and machine interaction and teaming. Recent advances in electronics, materials, and mechanical designs have offered avenues toward wearable HMI devices. However, existing wearable HMI devices are uncomfortable to use and restrict the human body's motion, show slow response times, or are challenging to realize with multiple functions. Here, we report sol-gel-on-polymer-processed indium zinc oxide semiconductor nanomembrane-based ultrathin stretchable electronics with advantages of multifunctionality, simple manufacturing, imperceptible wearing, and robust interfacing. Multifunctional wearable HMI devices range from resistive random-access memory for data storage to field-effect transistors for interfacing and switching circuits, to various sensors for health and body motion sensing, and to microheaters for temperature delivery. The HMI devices can be not only seamlessly worn by humans but also implemented as prosthetic skin for robotics, which offer intelligent feedback, resulting in a closed-loop HMI system.

Adolescents Environmental Emotion Perception by Integrating EEG and Eye Movements

Giving a robot the ability to perceive emotion in its environment can improve human-robot interaction (HRI), thereby facilitating more human-like communication. To achieve emotion recognition in different built environments for adolescents, we propose a multi-modal emotion intensity perception method using an integration of electroencephalography (EEG) and eye movement information. Specifically, we first develop a new stimulus video selection method based on computation of normalized arousal and valence scores according to subjective feedback from participants. Then, we establish a valence perception sub-model and an arousal sub-model by collecting and analyzing emotional EEG and eye movement signals, respectively. We employ this dual recognition method to perceive emotional intensities synchronously in two dimensions. In the laboratory environment, the best recognition accuracies of the modality fusion for the arousal and valence dimensions are 72.8 and 69.3%. The experimental results validate the feasibility of the proposed multi-modal emotion recognition method for environment emotion intensity perception. This promising tool not only achieves more accurate emotion perception for HRI systems but also provides an alternative approach to quantitatively assess environmental psychology.

An Investigation on the Sampling Frequency of the Upper-Limb Force Myographic Signals

Force myography (FMG) is an emerging method to register muscle activity of a limb using force sensors for human-machine interface and movement monitoring applications. Despite its newly gained popularity among researchers, many of its fundamental characteristics remain to be investigated. The aim of this study is to identify the minimum sampling frequency needed for recording upper-limb FMG signals without sacrificing signal integrity. Twelve healthy volunteers participated in an experiment in which they were instructed to perform rapid hand actions with FMG signals being recorded from the wrist and the bulk region of the forearm. The FMG signals were sampled at 1 kHz with a 16-bit resolution data acquisition device. We downsampled the signals with frequencies ranging from 1 Hz to 500 Hz to examine the discrepancies between the original signals and the downsampled ones. Based on the results, we suggest that FMG signals from the forearm and wrist should be collected with minimum sampling frequencies of 54 Hz and 58 Hz for deciphering isometric actions, and 70 Hz and 84 Hz for deciphering dynamic actions. This fundamental work provides insight into minimum requirements for sampling FMG signals such that the data content of such signals is not compromised.

Does force myography recorded at the wrist correlate to resistance load levels during bicep curls?

Resistance strength training is a proven method to improve bone density and muscle strength. A solution capable of automatically detecting the resistance force level exerted by a user from a wrist-based device can offer great convenience to the trainee and hence facilitate a better training outcome. In this short communication, we present our investigation aimed at exploring if force myographic (FMG) signals recorded at the wrist can predict the relative resistance levels that are associated with different weights. Specifically, we investigated the Spearman's correlations between the wrist FMG signal features and the dumbbell weights during a bicep curl exercise. 10 volunteers were recruited to perform a total of 100 curl actions, which included both the hammer and regular curls while the wrist FMG signals were being recorded. Three sets of weights ranging from 0.2 lb to 8 lb were used. For the hammer curls, a median correlation coefficient of 0.92 with an interquartile range (IQR) of 0.03 was obtained. For the regular curls, a 0.94 median correlation with a 0.02 IQR was obtained. We also used the data from the first 36 curls to generate a classifier model and applied it onto the rest of the data. An averaged validation accuracy of 88% was obtained. The results of this study showed the potential use of wrist FMG signal to detect different levels of the load during exercises; such information could potentially be used as feedback in fitness, sports, and rehabilitation activities.

Towards Zero Re-Training for Long-Term Hand Gesture Recognition via Ultrasound Sensing

While myoelectric pattern recognition is a prevailing way for gesture recognition, the inherent nonstationarity of electromyography signals hinders its long-term application. This study aims to prove a hypothesis that morphological information of muscle contraction detected by ultrasound image is potentially suitable for long-term use. A set of ultrasound-based algorithms are proposed to realize robust hand gesture recognition over multiple days, with user training only at the first day. A markerless calibration algorithm is first presented to position the ultrasound probe during donning and doffing; an algorithm combining speeded-up robust features and bag-of-features model being immune to ultrasound probe shift and rotation is then introduced; a self-enhancing classification method is next adopted to update classification model automatically by incorporating useful knowledge from testing data; finally the performance of long-term hand gesture recognition with zero re-training is validated by a six-day experiment of six healthy subjects, whose outcomes strongly support the hypothesis with about 94% of gesture recognition accuracy for each testing day. This study confirms the feasibility of adoption of ultrasound sensing for long-term musculature related applications.

Ultrasound-Based Sensing Models for Finger Motion Classification

Motions of the fingers are complex since hand grasping and manipulation are conducted by spatial and temporal coordination of forearm muscles and tendons. The dominant methods based on surface electromyography (sEMG) could not offer satisfactory solutions for finger motion classification due to its inherent nature of measuring the electrical activity of motor units at the skin's surface. In order to recognize morphological changes of forearm muscles for accurate hand motion prediction, ultrasound imaging is employed to investigate the feasibility of detecting mechanical deformation of deep muscle compartments in potential clinical applications. In this study, finger motion classification has been represented as subproblems: recognizing the discrete finger motions and predicting the continuous finger angles. Predefined 14 finger motions are presented in both sEMG signals and ultrasound images and captured simultaneously. Linear discriminant analysis classifier shows the ultrasound has better average accuracy (95.88%) than the sEMG (90.14%). On the other hand, the study of predicting the metacarpophalangeal (MCP) joint angle of each finger in nonperiod movements also confirms that classification method based on ultrasound achieves better results (average correlation 0.89 $\pm$ 0.07 and NRMSE 0.15 $\pm$ 0.05) than sEMG (0.81 $\pm$ 0.09 and 0.19 $\pm$ 0.05). The research outcomes evidently demonstrate that the ultrasound can be a feasible solution for muscle-driven machine interface, such as accurate finger motion control of prostheses and wearable robotic devices.

Improving the Robustness of Electromyogram-Pattern Recognition for Prosthetic Control by a Postprocessing Strategy

Electromyogram (EMG) contains rich information for motion decoding. As one of its major applications, EMG-pattern recognition (PR)-based control of prostheses has been proposed and investigated in the field of rehabilitation robotics for decades. These prostheses can offer a higher level of dexterity compared to the commercially available ones. However, limited progress has been made toward clinical application of EMG-PR-based prostheses, due to their unsatisfactory robustness against various interferences during daily use. These interferences may lead to misclassifications of motion intentions, which damage the control performance of EMG-PR-based prostheses. A number of studies have applied methods that undergo a postprocessing stage to determine the current motion outputs, based on previous outputs or other information, which have proved effective in reducing erroneous outputs. In this study, we proposed a postprocessing strategy that locks the outputs during the constant contraction to block out occasional misclassifications, upon detecting the motion onset using a threshold. The strategy was investigated using three different motion onset detectors, namely mean absolute value, Teager–Kaiser energy operator, or mechanomyogram (MMG). Our results indicate that the proposed strategy could suppress erroneous outputs, during rest and constant contractions in particular. In addition, with MMG as the motion onset detector, the strategy was found to produce the most significant improvement in the performance, reducing the total errors up to around 50% (from 22.9 to 11.5%) in comparison to the original classification output in the online test, and it is the most robust against threshold value changes. We speculate that motion onset detectors that are both smooth and responsive would further enhance the efficacy of the proposed postprocessing strategy, which would facilitate the clinical application of EMG-PR-based prosthetic control.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.

Multi-modal myocontrol: Testing combined force- and electromyography

Myocontrol, that is control of prostheses using bodily signals, has proved in the decades to be a surprisingly hard problem for the scientific community of assistive and rehabilitation robotics. In particular, traditional surface electromyography (sEMG) seems to be no longer enough to guarantee dexterity (i.e., control over several degrees of freedom) and, most importantly, reliability. Multi-modal myocontrol is concerned with the idea of using novel signal gathering techniques as a replacement of, or alongside, sEMG, to provide high-density and diverse signals to improve dexterity and make the control more reliable. In this paper we present an offline and online assessment of multi-modal sEMG and force myography (FMG) targeted at hand and wrist myocontrol. A total number of twenty sEMG and FMG sensors were used simultaneously, in several combined configurations, to predict opening/closing of the hand and activation of two degrees of freedom of the wrist of ten intact subjects. The analysis was targeted at determining the optimal sensor combination and control parameters; the experimental results indicate that sEMG sensors alone perform worst, yielding a nRMSE of 9.1%, while mixing FMG and sEMG or using FMG only reduces the nRMSE to 5.2-6.6%. To validate these results, we engaged the subject with median performance in an online goal-reaching task. Analysis of this further experiment reveals that the online behaviour is similar to the offline one.

Counting Grasping Action Using Force Myography: An Exploratory Study With Healthy Individuals

Background

Functional arm movements generally require grasping an object. The possibility of detecting and counting the action of grasping is believed to be of importance for individual with motor function deficits of the arm, as it could be an indication of the number of the functional arm movements performed by the individuals during rehabilitation. In this exploratory work, the feasibility of using armbands recording radial displacements of forearm muscles and tendons (ie, force myography, FMG) to estimate hand grasping with healthy individuals was investigated. In contrast to previous studies, this exploratory study investigates the feasibility of (1) detecting grasping when the participants move their arms, which could introduce large artifacts to the point of potentially preventing the practical use of the proposed technology, and (2) counting grasping during arm-reaching tasks.

Objective

The aim of this study was to determine the usefulness of FMG in the detection of functional arm movements. The use of FMG straps placed on the forearm is proposed for counting the number of grasping actions in the presence of arm movements.

Methods

Ten healthy volunteers participated in this study to perform a pick-and-place exercise after providing informed consent. FMG signals were simultaneously collected using 2 FMG straps worn on their wrist and at the midposition of their forearm, respectively. Raw FMG signals and 3 additional FMG features (ie, root mean square, wavelength, and window symmetry) were extracted and fed into a linear discriminant analysis classifier to predict grasping states. The transition from nongrasping to grasping states was detected during the process of counting the number of grasping actions.

Results

The median accuracy for detecting grasping events using FMG recorded from the wrist was 95%, and the corresponding interquartile range (IQR) was 5%. For forearm FMG classification, the median accuracy was 92%, and the corresponding IQR was 3%. The difference between the 2 median accuracies was statistically significant (P<.001) when using a paired 2-tailed sign test. The median percentage error for counting grasping events when FMG was recorded from the wrist was 1%, and the corresponding IQR was 2%. The median percentage error for FMG recorded from the forearm was 2%, and the corresponding IQR was also 2%. While the median percentage error for the wrist was lower than that of the forearm, the difference between the 2 was not statistically significant based on a paired 2-tailed sign test (P=.29).

Conclusions

This study reports that grasping can reliably be counted using an unobtrusive and simple FMG strap even in the presence of arm movements. Such a result supports the foundation for future research evaluating the feasibility of monitoring hand grasping during unsupervised ADL, leading to further investigations with individuals with motor function deficits of the arm.

Hybrid EEG-fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control

In this paper, a hybrid electroencephalography–functional near-infrared spectroscopy (EEG–fNIRS) scheme to decode eight active brain commands from the frontal brain region for brain–computer interface is presented. A total of eight commands are decoded by fNIRS, as positioned on the prefrontal cortex, and by EEG, around the frontal, parietal, and visual cortices. Mental arithmetic, mental counting, mental rotation, and word formation tasks are decoded with fNIRS, in which the selected features for classification and command generation are the peak, minimum, and mean ΔHbO values within a 2-s moving window. In the case of EEG, two eyeblinks, three eyeblinks, and eye movement in the up/down and left/right directions are used for four-command generation. The features in this case are the number of peaks and the mean of the EEG signal during 1 s window. We tested the generated commands on a quadcopter in an open space. An average accuracy of 75.6% was achieved with fNIRS for four-command decoding and 86% with EEG for another four-command decoding. The testing results show the possibility of controlling a quadcopter online and in real-time using eight commands from the prefrontal and frontal cortices via the proposed hybrid EEG–fNIRS interface.

Exploration of Force Myography and surface Electromyography in hand gesture classification

Whereas pressure sensors increasingly have received attention as a non-invasive interface for hand gesture recognition, their performance has not been comprehensively evaluated. This work examined the performance of hand gesture classification using Force Myography (FMG) and surface Electromyography (sEMG) technologies by performing 3 sets of 48 hand gestures using a prototyped FMG band and an array of commercial sEMG sensors worn both on the wrist and forearm simultaneously. The results show that the FMG band achieved classification accuracies as good as the high quality, commercially available, sEMG system on both wrist and forearm positions; specifically, by only using 8 Force Sensitive Resisters (FSRs), the FMG band achieved accuracies of 91.2% and 83.5% in classifying the 48 hand gestures in cross-validation and cross-trial evaluations, which were higher than those of sEMG (84.6% and 79.1%). By using all 16 FSRs on the band, our device achieved high accuracies of 96.7% and 89.4% in cross-validation and cross-trial evaluations.

Assessment of a Wearable Force- and Electromyography Device and Comparison of the Related Signals for Myocontrol

In the frame of assistive robotics, multi-finger prosthetic hand/wrists have recently appeared, offering an increasing level of dexterity; however, in practice their control is limited to a few hand grips and still unreliable, with the effect that pattern recognition has not yet appeared in the clinical environment. According to the scientific community, one of the keys to improve the situation is multi-modal sensing, i.e., using diverse sensor modalities to interpret the subject's intent and improve the reliability and safety of the control system in daily life activities. In this work, we first describe and test a novel wireless, wearable force- and electromyography device; through an experiment conducted on ten intact subjects, we then compare the obtained signals both qualitatively and quantitatively, highlighting their advantages and disadvantages. Our results indicate that force-myography yields signals which are more stable across time during whenever a pattern is held, than those obtained by electromyography. We speculate that fusion of the two modalities might be advantageous to improve the reliability of myocontrol in the near future.

Stable force-myographic control of a prosthetic hand using incremental learning

Force myography has been proposed as an appealing alternative to electromyography for control of upper limb prosthesis. A limitation of this technique is the non-stationary nature of the recorded force data. Force patterns vary under influence of various factors such as change in orientation and position of the prosthesis. We hereby propose an incremental learning method to overcome this limitation. We use an online sequential extreme learning machine where occasional updates allow continual adaptation to signal changes. The applicability and effectiveness of this approach is demonstrated for predicting the hand status from forearm muscle forces at various arm positions. The results show that incremental updates are indeed effective to maintain a stable level of performance, achieving an average classification accuracy of 98.75% for two subjects.

Proportional estimation of finger movements from high-density surface electromyography

BACKGROUND

The importance to restore the hand function following an injury/disease of the nervous system led to the development of novel rehabilitation interventions. Surface electromyography can be used to create a user-driven control of a rehabilitation robot, in which the subject needs to engage actively, by using spared voluntary activation to trigger the assistance of the robot.

METHODS

The study investigated methods for the selective estimation of individual finger movements from high-density surface electromyographic signals (HD-sEMG) with minimal interference between movements of other fingers. Regression was evaluated in online and offline control tests with nine healthy subjects (per test) using a linear discriminant analysis classifier (LDA), a common spatial patterns proportional estimator (CSP-PE), and a thresholding (THR) algorithm. In all tests, the subjects performed an isometric force tracking task guided by a moving visual marker indicating the contraction type (flexion/extension), desired activation level and the finger that should be moved. The outcome measures were mean square error (nMSE) between the reference and generated trajectories normalized to the peak-to-peak value of the reference, the classification accuracy (CA), the mean amplitude of the false activations (MAFA) and, in the offline tests only, the Pearson correlation coefficient (PCORR).

RESULTS

The offline tests demonstrated that, for the reduced number of electrodes (≤24), the CSP-PE outperformed the LDA with higher precision of proportional estimation and less crosstalk between the movement classes (e.g., 8 electrodes, median MAFA ~ 0.6 vs. 1.1 %, median nMSE ~ 4.3 vs. 5.5 %). The LDA and the CSP-PE performed similarly in the online tests (median nMSE < 3.6 %, median MAFA < 0.7 %), but the CSP-PE provided a more stable performance across the tested conditions (less improvement between different sessions). Furthermore, THR, exploiting topographical information about the single finger activity from HD-sEMG, provided in many cases a regression accuracy similar to that of the pattern recognition techniques, but the performance was not consistent across subjects and fingers.

CONCLUSIONS

The CSP-PE is a method of choice for selective individual finger control with the limited number of electrodes (<24), whereas for the higher resolution of the recording, either method (CPS-PA or LDA) can be used with a similar performance. Despite the abundance of detection points, the simple THR showed to be significantly worse compared to both pattern recognition/regression methods. Nevertheless, THR is a simple method to apply (no training), and it could still give satisfactory performance in some subjects and/or simpler scenarios (e.g., control of selected fingers). These conclusions are important for guiding future developments towards the clinical application of the methods for individual finger control in rehabilitation robotics.

Optical Myography: Detecting Finger Movements by Looking at the Forearm

One of the crucial problems found in the scientific community of assistive/rehabilitation robotics nowadays is that of automatically detecting what a disabled subject (for instance, a hand amputee) wants to do, exactly when she wants to do it, and strictly for the time she wants to do it. This problem, commonly called “intent detection,” has traditionally been tackled using surface electromyography, a technique which suffers from a number of drawbacks, including the changes in the signal induced by sweat and muscle fatigue. With the advent of realistic, physically plausible augmented- and virtual-reality environments for rehabilitation, this approach does not suffice anymore. In this paper, we explore a novel method to solve the problem, which we call Optical Myography (OMG). The idea is to visually inspect the human forearm (or stump) to reconstruct what fingers are moving and to what extent. In a psychophysical experiment involving ten intact subjects, we used visual fiducial markers (AprilTags) and a standard web camera to visualize the deformations of the surface of the forearm, which then were mapped to the intended finger motions. As ground truth, a visual stimulus was used, avoiding the need for finger sensors (force/position sensors, datagloves, etc.). Two machine-learning approaches, a linear and a non-linear one, were comparatively tested in settings of increasing realism. The results indicate an average error in the range of 0.05–0.22 (root mean square error normalized over the signal range), in line with similar results obtained with more mature techniques such as electromyography. If further successfully tested in the large, this approach could lead to vision-based intent detection of amputees, with the main application of letting such disabled persons dexterously and reliably interact in an augmented-/virtual-reality setup.

Force Myography to Control Robotic Upper Extremity Prostheses: A Feasibility Study

Advancement in assistive technology has led to the commercial availability of multi-dexterous robotic prostheses for the upper extremity. The relatively low performance of the currently used techniques to detect the intention of the user to control such advanced robotic prostheses, however, limits their use. This article explores the use of force myography (FMG) as a potential alternative to the well-established surface electromyography. Specifically, the use of FMG to control different grips of a commercially available robotic hand, Bebionic3, is investigated. Four male transradially amputated subjects participated in the study, and a protocol was developed to assess the prediction accuracy of 11 grips. Different combinations of grips were examined, ranging from 6 up to 11 grips. The results indicate that it is possible to classify six primary grips important in activities of daily living using FMG with an accuracy of above 70% in the residual limb. Additional strategies to increase classification accuracy, such as using the available modes on the Bebionic3, allowed results to improve up to 88.83 and 89.00% for opposed thumb and non-opposed thumb modes, respectively.