Virtual Reality Provides an Effective Platform for Functional Evaluations of Closed-Loop Neuromyoelectric Control

Current status and clinical perspectives of extended reality for myoelectric prostheses: review

Training with "Extended Reality" or X-Reality (XR) systems can undoubtedly enhance the control of the myoelectric prostheses. However, there is no consensus on which factors improve the efficiency of skill transfer from virtual training to actual prosthesis abilities. This review examines the current status and clinical applications of XR in the field of myoelectric prosthesis training and analyses possible influences on skill migration. We have conducted a thorough search on databases in the field of prostheses using keywords such as extended reality, virtual reality and serious gaming. Our scoping review encompassed relevant applications, control methods, performance evaluation and assessment metrics. Our findings indicate that the implementation of XR technology for myoelectric rehabilitative training on prostheses provides considerable benefits. Additionally, there are numerous standardised methods available for evaluating training effectiveness. Recently, there has been a surge in the number of XR-based training tools for myoelectric prostheses, with an emphasis on user engagement and virtual training evaluation. Insufficient attention has been paid to significant limitations in the behaviour, functionality, and usage patterns of XR and myoelectric prostheses, potentially obstructing the transfer of skills and prospects for clinical application. Improvements are recommended in four critical areas: activities of daily living, training strategies, feedback, and the alignment of the virtual environment with the physical devices.

A Review on the Usability, Flexibility, Affinity, and Affordability of Virtual Technology for Rehabilitation Training of Upper Limb Amputees

(1) Background: Prosthetic rehabilitation is essential for upper limb amputees to regain their ability to work. However, the abandonment rate of prosthetics is higher than 50% due to the high cost of rehabilitation. Virtual technology shows potential for improving the availability and cost-effectiveness of prosthetic rehabilitation. This article systematically reviews the application of virtual technology for the prosthetic rehabilitation of upper limb amputees. (2) Methods: We followed PRISMA review guidance, STROBE, and CASP to evaluate the included articles. Finally, 17 articles were screened from 22,609 articles. (3) Results: This study reviews the possible benefits of using virtual technology from four aspects: usability, flexibility, psychological affinity, and long-term affordability. Three significant challenges are also discussed: realism, closed-loop control, and multi-modality integration. (4) Conclusions: Virtual technology allows for flexible and configurable control rehabilitation, both during hospital admissions and after discharge, at a relatively low cost. The technology shows promise in addressing the critical barrier of current prosthetic training issues, potentially improving the practical availability of prosthesis techniques for upper limb amputees.

Automated calibration of somatosensory stimulation using reinforcement learning

BACKGROUND

The identification of the electrical stimulation parameters for neuromodulation is a subject-specific and time-consuming procedure that presently mostly relies on the expertise of the user (e.g., clinician, experimenter, bioengineer). Since the parameters of stimulation change over time (due to displacement of electrodes, skin status, etc.), patients undergo recurrent, long calibration sessions, along with visits to the clinics, which are inefficient and expensive. To address this issue, we developed an automatized calibration system based on reinforcement learning (RL) allowing for accurate and efficient identification of the peripheral nerve stimulation parameters for somatosensory neuroprostheses.

METHODS

We developed an RL algorithm to automatically select neurostimulation parameters for restoring sensory feedback with transcutaneous electrical nerve stimulation (TENS). First, the algorithm was trained offline on a dataset comprising 49 subjects. Then, the neurostimulation was then integrated with a graphical user interface (GUI) to create an intuitive AI-based mapping platform enabling the user to autonomously perform the sensation characterization procedure. We assessed the algorithm against the performance of both experienced and naïve and of a brute force algorithm (BFA), on 15 nerves from five subjects. Then, we validated the AI-based platform on six neuropathic nerves affected by distal sensory loss.

RESULTS

Our automatized approach demonstrated the ability to find the optimal values of neurostimulation achieving reliable and comfortable elicited sensations. When compared to alternatives, RL outperformed the naïve and BFA, significantly decreasing the time for mapping and the number of delivered stimulation trains, while improving the overall quality. Furthermore, the RL algorithm showed performance comparable to trained experimenters. Finally, we exploited it successfully for eliciting sensory feedback in neuropathic patients.

CONCLUSIONS

Our findings demonstrated that the AI-based platform based on a RL algorithm can automatically and efficiently calibrate parameters for somatosensory nerve stimulation. This holds promise to avoid experts' employment in similar scenarios, thanks to the merging between AI and neurotech. Our RL algorithm has the potential to be used in other neuromodulation fields requiring a mapping process of the stimulation parameters.

TRIAL REGISTRATION

ClinicalTrial.gov (Identifier: NCT04217005).

Closed-loop stimulation of lateral cervical spinal cord in upper-limb amputees to enable sensory discrimination: a case study

Modern myoelectric prosthetic hands have multiple independently controllable degrees of freedom, but require constant visual attention to use effectively. Somatosensory feedback provides information not available through vision alone and is essential for fine motor control of our limbs. Similarly, stimulation of the nervous system can potentially provide artificial somatosensory feedback to reduce the reliance on visual cues to efficiently operate prosthetic devices. We have shown previously that epidural stimulation of the lateral cervical spinal cord can evoke tactile sensations perceived as emanating from the missing arm and hand in people with upper-limb amputation. In this case study, two subjects with upper-limb amputation used this somatotopically-matched tactile feedback to discriminate object size and compliance while controlling a prosthetic hand. With less than 30 min of practice each day, both subjects were able to use artificial somatosensory feedback to perform a subset of the discrimination tasks at a success level well above chance. Subject 1 was consistently more adept at determining object size (74% accuracy; chance: 33%) while Subject 2 achieved a higher accuracy level in determining object compliance (60% accuracy; chance 33%). In each subject, discrimination of the other object property was only slightly above or at chance level suggesting that the task design and stimulation encoding scheme are important determinants of which object property could be reliably identified. Our observations suggest that changes in the intensity of artificial somatosensory feedback provided via spinal cord stimulation can be readily used to infer information about object properties with minimal training.

Embodiment of a virtual prosthesis through training using an EMG-based human-machine interface: Case series

Therapeutic strategies capable of inducing and enhancing prosthesis embodiment are a key point for better adaptation to and acceptance of prosthetic limbs. In this study, we developed a training protocol using an EMG-based human-machine interface (HMI) that was applied in the preprosthetic rehabilitation phase of people with amputation. This is a case series with the objective of evaluating the induction and enhancement of the embodiment of a virtual prosthesis. Six men and a woman with unilateral transfemoral traumatic amputation without previous use of prostheses participated in the study. Participants performed a training protocol with the EMG-based HMI, composed of six sessions held twice a week, each lasting 30 mins. This system consisted of myoelectric control of the movements of a virtual prosthesis immersed in a 3D virtual environment. Additionally, vibrotactile stimuli were provided on the participant’s back corresponding to the movements performed. Embodiment was investigated from the following set of measurements: skin conductance response (affective measurement), crossmodal congruency effect (spatial perception measurement), ability to control the virtual prosthesis (motor measurement), and reports before and after the training. The increase in the skin conductance response in conditions where the virtual prosthesis was threatened, recalibration of the peripersonal space perception identified by the crossmodal congruency effect, ability to control the virtual prosthesis, and participant reports consistently showed the induction and enhancement of virtual prosthesis embodiment. Therefore, this protocol using EMG-based HMI was shown to be a viable option to achieve and enhance the embodiment of a virtual prosthetic limb.

On the use of virtual reality for individuals with upper limb loss: a systematic scoping review

INTRODUCTION

Virtual reality has recently become a popular application for rehabilitation and motor control research. This technology has emerged as a valid addition to conventional therapy and promises a successful rehabilitation. This study describes recent research related to the use of virtual reality applications in the rehabilitation of individuals with upper limb loss and to see whether this technology has enough proof of its applicability.

EVIDENCE ACQUISITION

Searches were conducted with the Web of Science, Google Scholar, IEEE Xplore, and PubMed databases from inception up to September 2020. Articles that employed virtual reality in the rehabilitation of individual with upper limb loss were included in the research if it is written in English, the keyword exists in the title and abstract; it uses visual feedback in nonimmersive, semi-immersive, or fully immersive virtual environments. Data extraction was carried out by two independent researchers. The study was drafted using the Preferred Reporting Items for Systematic Reviews and Meta-analysis Protocols (PRISMA).

EVIDENCE SYNTHESIS

A total of 38 articles met the inclusion criteria. Most studies were published between 2010 and 2020. Thirty-nine percent of the studies (N.=15), originates from North America; 55% of the studies (N.=21), were publicly funded; 61% of the studies (N.=24), was without disclosure of conflict of interest; 82% of the studies (N.=31), were cited in other studies. All the studies were published in journals and conference proceedings. Sixty-six percent of the studies (N.=25) has come out with positive outcome. The design studies were mostly case reports, case series, and poorly designed cohort studies that made up 55% (N.=21) of all the studies cited here.

CONCLUSIONS

The research conducted on the use of virtual reality in individual with upper limb loss rehabilitation is of very low quality. The improvements to the research protocol are much needed. It is not necessary to develop new devices, but rather to assess existing devices with well-conducted randomized controlled trials.

Virtual/Augmented Reality for Rehabilitation Applications Using Electromyography as Control/Biofeedback: Systematic Literature Review

Virtual reality (VR) and augmented reality (AR) are engaging interfaces that can be of benefit for rehabilitation therapy. However, they are still not widely used, and the use of surface electromyography (sEMG) signals is not established for them. Our goal is to explore whether there is a standardized protocol towards therapeutic applications since there are not many methodological reviews that focus on sEMG control/feedback. A systematic literature review using the PRISMA (preferred reporting items for systematic reviews and meta-analyses) methodology is conducted. A Boolean search in databases was performed applying inclusion/exclusion criteria; articles older than 5 years and repeated were excluded. A total of 393 articles were selected for screening, of which 66.15% were excluded, 131 records were eligible, 69.46% use neither VR/AR interfaces nor sEMG control; 40 articles remained. Categories are, application: neurological motor rehabilitation (70%), prosthesis training (30%); processing algorithm: artificial intelligence (40%), direct control (20%); hardware: Myo Armband (22.5%), Delsys (10%), proprietary (17.5%); VR/AR interface: training scene model (25%), videogame (47.5%), first-person (20%). Finally, applications are focused on motor neurorehabilitation after stroke/amputation; however, there is no consensus regarding signal processing or classification criteria. Future work should deal with proposing guidelines to standardize these technologies for their adoption in clinical practice.

Exploring Biomimetic Stiffness Modulation and Wearable Finger Haptics for Improving Myoelectric Control of Virtual Hand

The embodiment of virtual hand (VH) by the user is generally deemed to be important for virtual reality (VR) based hand rehabilitation applications, which may help to engage the user and promote motor skill relearning. In particular, it requires that the VH should produce task-dependent interaction behaviors from rigid to soft. While such a capability is inherent to humans via hand stiffness regulation and haptic interactions, yet it have not been successfully imitated by VH in existing studies. In this paper, we present a work which integrates biomimetic stiffness regulation and wearable finger force feedback in VR scenarios involving myoelectric control of VH. On one hand, the biomimetic stiffness modulation intuitively enables VH to imitate the stiffness profile of the user's hand in real time. On the other hand, the wearable finger force-feedback device elicits a natural and realistic sensation of external force on the fingertip, which provides the user a proper understanding of the environment for enhancing his/her stiffness regulation. The benefits of the proposed integrated system were evaluated with eight healthy subjects that performed two tasks with opposite stiffness requirements. The achieved performance is compared with reduced versions of the integrated system, where either biomimetic impedance control or wearable force feedback is excluded. The results suggest that the proposed integrated system enables the stiffness of VH to be adaptively regulated by the user through the perception of interaction torques and vision, resulting in task-dependent behaviors from rigid to soft for VH.

A Multi-User Transradial Functional-Test Socket for Validation of New Myoelectric Prosthetic Control Strategies

The validation of myoelectric prosthetic control strategies for individuals experiencing upper-limb loss is hindered by the time and cost affiliated with traditional custom-fabricated sockets. Consequently, researchers often rely upon virtual reality or robotic arms to validate novel control strategies, which limits end-user involvement. Prosthetists fabricate diagnostic check sockets to assess and refine socket fit, but these clinical techniques are not readily available to researchers and are not intended to assess functionality for control strategies. Here we present a multi-user, low-cost, transradial, functional-test socket for short-term research use that can be custom-fit and donned rapidly, used in conjunction with various electromyography configurations, and adapted for use with various residual limbs and terminal devices. In this study, participants with upper-limb amputation completed functional tasks in physical and virtual environments both with and without the socket, and they reported on their perceived comfort level over time. The functional-test socket was fabricated prior to participants' arrival, iteratively fitted by the researchers within 10 mins, and donned in under 1 min (excluding electrode placement, which will vary for different use cases). It accommodated multiple individuals and terminal devices and had a total cost of materials under $10 USD. Across all participants, the socket did not significantly impede functional task performance or reduce the electromyography signal-to-noise ratio. The socket was rated as comfortable enough for at least 2 h of use, though it was expectedly perceived as less comfortable than a clinically-prescribed daily-use socket. The development of this multi-user, transradial, functional-test socket constitutes an important step toward increased end-user participation in advanced myoelectric prosthetic research. The socket design has been open-sourced and is available for other researchers.

Virtual Reality for Evaluating Prosthetic Hand Control Strategies: A Preliminary Report

Improving prosthetic hand functionality is critical in reducing abandonment rates and improving the amputee's quality of life. Techniques such as joint force estimation and gesture recognition using myoelectric signals could enable more realistic control of the prosthetic hand. To accelerate the translation of these advanced control strategies from lab to clinic, We created a virtual prosthetic control environment that enables rich user interactions and dexterity evaluation. The virtual environment is made of two parts, namely the Unity scene for rendering and user interaction, and a Python back-end to support accurate physics simulation and communication with control algorithms. By utilizing the built-in tracking capabilities of a virtual reality headset, the user can visualize and manipulate a virtual hand without additional motion tracking setups. In the virtual environment, we demonstrate actuation of the prosthetic hand through decoded EMG signal streaming, hand tracking, and the use of a VR controller. By providing a flexible platform to investigate different control modalities, we believe that our virtual environment will allow for faster experimentation and further progress in clinical translation.

Sensory feedback for limb prostheses in amputees

Commercial prosthetic devices currently do not provide natural sensory information on the interaction with objects or movements. The subsequent disadvantages include unphysiological walking with a prosthetic leg and difficulty in controlling the force exerted with a prosthetic hand, thus creating health issues. Restoring natural sensory feedback from the prosthesis to amputees is an unmet clinical need. An optimal device should be able to elicit natural sensations of touch or proprioception, by delivering the complex signals to the nervous system that would be produced by skin, muscles and joints receptors. This Review covers the various neurotechnological approaches that have been proposed for the development of the optimal sensory feedback restoration device for arm and leg amputees.

Brain-machine and muscle-machine bio-sensing methods for gesture intent acquisition in upper-limb prosthesis control: a review

This paper presents a review of a number of bio-sensing methods for gesture intent signal acquisition in control tasks for upper-limb prosthesis. The paper specifically provides a breakdown of the control task in myoelectric prosthesis, and in addition, highlights and describes the importance of the acquisition of a high-quality bio-signal. The paper also describes commonly used invasive and non-invasive brain and muscle machine interfaces such as electroencephalography, electrocorticography, electroneurography, surface electromyography, sonomyography, mechanomyography, near infra-red, force sensitive resistance/pressure, and magnetoencephalography. Each modality is reviewed based on its operating principle and limitations in gesture recognition, followed by respective advantages and disadvantages. Also described within this paper, are multimodal sensing approaches, which involve data fusion of information from various sensing modalities for an enhanced neuromuscular bio-sensing source. Using a semi-systematic review methodology, we are able to derive a novel tabular approach towards contrasting the various strengths and weaknesses of the reviewed bio-sensing methods towards gesture recognition in a prosthesis interface. This would allow for a streamlined method of down selection of an appropriate bio-sensor given specific prosthesis design criteria and requirements. The paper concludes by highlighting a number of research areas that require more work for strides to be made towards improving and enhancing the connection between man and machine as it concerns upper-limb prosthesis. Such areas include classifier augmentation for gesture recognition, filtering techniques for sensor disturbance rejection, feeling of tactile sensations with an artificial limb.

Discriminability of multiple cutaneous and proprioceptive hand percepts evoked by intraneural stimulation with Utah slanted electrode arrays in human amputees

BACKGROUND

Electrical stimulation of residual afferent nerve fibers can evoke sensations from a missing limb after amputation, and bionic arms endowed with artificial sensory feedback have been shown to confer functional and psychological benefits. Here we explore the extent to which artificial sensations can be discriminated based on location, quality, and intensity.

METHODS

We implanted Utah Slanted Electrode Arrays (USEAs) in the arm nerves of three transradial amputees and delivered electrical stimulation via different electrodes and frequencies to produce sensations on the missing hand with various locations, qualities, and intensities. Participants performed blind discrimination trials to discriminate among these artificial sensations.

RESULTS

Participants successfully discriminated cutaneous and proprioceptive sensations ranging in location, quality and intensity. Performance was significantly greater than chance for all discrimination tasks, including discrimination among up to ten different cutaneous location-intensity combinations (15/30 successes, p < 0.0001) and seven different proprioceptive location-intensity combinations (21/40 successes, p < 0.0001). Variations in the site of stimulation within the nerve, via electrode selection, enabled discrimination among up to five locations and qualities (35/35 successes, p < 0.0001). Variations in the stimulation frequency enabled discrimination among four different intensities at the same location (13/20 successes, p < 0.0005). One participant also discriminated among individual stimulation of two different USEA electrodes, simultaneous stimulation on both electrodes, and interleaved stimulation on both electrodes (20/24 successes, p < 0.0001).

CONCLUSION

Electrode location, stimulation frequency, and stimulation pattern can be modulated to evoke functionally discriminable sensations with a range of locations, qualities, and intensities. This rich source of artificial sensory feedback may enhance functional performance and embodiment of bionic arms endowed with a sense of touch.

Long-term performance of Utah slanted electrode arrays and intramuscular electromyographic leads implanted chronically in human arm nerves and muscles

OBJECTIVE

We explore the long-term performance and stability of seven percutaneous Utah Slanted Electrode Arrays (USEAs) and intramuscular recording leads (iEMGs) implanted chronically in the residual arm nerves and muscles of three human participants as a means to permanently restore sensorimotor function after transradial amputations.

APPROACH

We quantify the number of functional recording and functional stimulating electrodes over time. We also calculate the signal-to-noise ratio (SNR) of USEA and iEMG recordings and quantify the stimulation current necessary to evoke detectable sensory percepts. Furthermore, we quantify the consistency of the sensory modality, receptive field location, and receptive field size of USEA-evoked percepts.

MAIN RESULTS

In the most recent subject, involving USEAs with technical improvements, neural recordings persisted for 502 d (entire implant duration) and the number of functional recording electrodes for one USEA increased over time. However, for six out of seven USEAs across the three participants, the number of functional recording electrodes decreased within the first 2 months after implantation. The SNR of neural recordings and electromyographic recordings stayed relatively consistent over time. Sensory percepts were consistently evoked over the span of 14 months, were not significantly different in size, and highlighted the nerves' fascicular organization. The percentage of percepts with consistent modality or consistent receptive field location between sessions (∼1 month apart) varied between 0%-86.2% and 9.1%-100%, respectively. Stimulation thresholds and electrode impedances increased initially but then remained relatively stable over time.

SIGNIFICANCE

This work demonstrates improved performance of USEAs, and provides a basis for comparing the longevity and stability of USEAs to that of other neural interfaces. USEAs provide a rich repertoire of neural recordings and sensory percepts. Although their performance still generally declines over time, functionality can persist long-term. Future work should leverage the results presented here to further improve USEA design or to develop adaptive algorithms that can maintain a high level of performance.

Portable Take-Home System Enables Proportional Control and High-Resolution Data Logging With a Multi-Degree-of-Freedom Bionic Arm

This paper describes a portable, prosthetic control system and the first at-home use of a multi-degree-of-freedom, proportionally controlled bionic arm. The system uses a modified Kalman filter to provide 6 degree-of-freedom, real-time, proportional control. We describe (a) how the system trains motor control algorithms for use with an advanced bionic arm, and (b) the system's ability to record an unprecedented and comprehensive dataset of EMG, hand positions and force sensor values. Intact participants and a transradial amputee used the system to perform activities-of-daily-living, including bi-manual tasks, in the lab and at home. This technology enables at-home dexterous bionic arm use, and provides a high-temporal resolution description of daily use—essential information to determine clinical relevance and improve future research for advanced bionic arms.

Intuitive neuromyoelectric control of a dexterous bionic arm using a modified Kalman filter

BACKGROUND

Multi-articulate prostheses are capable of performing dexterous hand movements. However, clinically available control strategies fail to provide users with intuitive, independent and proportional control over multiple degrees of freedom (DOFs) in real-time.

NEW METHOD

We detail the use of a modified Kalman filter (MKF) to provide intuitive, independent and proportional control over six-DOF prostheses such as the DEKA "LUKE" arm. Input features include neural firing rates recorded from Utah Slanted Electrode Arrays and mean absolute value of intramuscular electromyographic (EMG) recordings. Ad-hoc modifications include thresholds and non-unity gains on the output of a Kalman filter.

RESULTS

We demonstrate that both neural and EMG data can be combined effectively. We also highlight that modifications can be optimized to significantly improve performance relative to an unmodified Kalman filter. Thresholds significantly reduced unintended movement and promoted more independent control of the different DOFs. Gains were significantly greater than one and served to ease movement initiation. Optimal modifications can be determined quickly offline and translate to functional improvements online. Using a portable take-home system, participants performed various activities of daily living.

COMPARISON WITH EXISTING METHODS

In contrast to pattern recognition, the MKF allows users to continuously modulate their force output, which is critical for fine dexterity. The MKF is also computationally efficient and can be trained in less than five minutes.

CONCLUSIONS

The MKF can be used to explore the functional and psychological benefits associated with long-term, at-home control of dexterous prosthetic hands.

A Modular Transradial Bypass Socket for Surface Myoelectric Prosthetic Control in Non-Amputees

Bypass sockets allow researchers to perform tests of prosthetic systems from the prosthetic user's perspective. We designed a modular upper-limb bypass socket with 3D-printed components that can be easily modified for use with a variety of terminal devices. Our bypass socket preserves access to forearm musculature and the hand, which are necessary for surface electromyography and to provide substituted sensory feedback. Our bypass socket allows a sufficient range of motion to complete tasks in the frontal working area, as measured on non-amputee participants. We examined the performance of non-amputee participants using the bypass socket on the original and modified Box and Block Tests. Participants moved 11.3 ± 2.7 and 11.7 ± 2.4 blocks in the original and modified Box and Block Tests (mean ± SD), respectively, within the range of reported scores using amputee participants. Range of motion for users wearing the bypass socket meets or exceeds most reported range of motion requirements for activities of daily living. The bypass socket was originally designed with a freely rotating wrist; we found that adding elastic resistance to user wrist rotation while wearing the bypass socket had no significant effect on motor decode performance. We have open-sourced the design files and an assembly manual for the bypass socket. We anticipate that the bypass socket will be a useful tool to evaluate and develop sensorized myoelectric prosthesis technology.