Biomimetic rehabilitation engineering: the importance of somatosensory feedback for brain-machine interfaces

Improved Spatial Knowledge Acquisition through Sensory Augmentation

Sensory augmentation provides novel opportunities to broaden our knowledge of human perception through external sensors that record and transmit information beyond natural perception. To assess whether such augmented senses affect the acquisition of spatial knowledge during navigation, we trained a group of 27 participants for six weeks with an augmented sense for cardinal directions called the feelSpace belt. Then, we recruited a control group that did not receive the augmented sense and the corresponding training. All 53 participants first explored the Westbrook virtual reality environment for two and a half hours spread over five sessions before assessing their spatial knowledge in four immersive virtual reality tasks measuring cardinal, route, and survey knowledge. We found that the belt group acquired significantly more accurate cardinal and survey knowledge, which was measured in pointing accuracy, distance, and rotation estimates. Interestingly, the augmented sense also positively affected route knowledge, although to a lesser degree. Finally, the belt group reported a significant increase in the use of spatial strategies after training, while the groups' ratings were comparable at baseline. The results suggest that six weeks of training with the feelSpace belt led to improved survey and route knowledge acquisition. Moreover, the findings of our study could inform the development of assistive technologies for individuals with visual or navigational impairments, which may lead to enhanced navigation skills and quality of life.

LIVE-streaming 3D images: A neuroscience approach to full-body illusions

Inspired by recent technological advances in the gaming industry, we used capture cards to create and LIVE-stream high quality 3D-images. With this novel technique, we developed a real-life stereoscopic 3D full-body illusion paradigm (3D projection). Unlike previous versions of the full-body illusion that rely upon unwieldy head-mounted displays, this paradigm enables the unobstructed investigation of such illusions with neuroscience methods (e.g., transcranial direct current stimulation, transcranial magnetic stimulation, electroencephalography, and near-infrared spectroscopy) and examination of their neural underpinnings. This paper has three aims: (i) to provide a step-by-step guide on how to implement 3D LIVE-streaming, (ii) to explain how this can be used to create a full-body illusion paradigm; and (iii) to present evidence that documents the effectiveness of our methods (de Boer et al., 2020), including suggestions for potential applications. Particularly significant is the fact that 3D LIVE-streaming is not GPU-intensive and can easily be applied to any device or screen that can display 3D images (e.g., TV, tablet, mobile phone). Therefore, these methods also have potential future clinical and commercial benefits. 3D LIVE-streaming could be used to enhance future clinical observations or educational tools, or potentially guide medical interventions with real-time high-quality 3D images. Alternatively, our methods can be used in future rehabilitation programs to aid recovery from nervous system injury (e.g., spinal cord injury, brain damage, limb loss) or in therapies aimed at alleviating psychosis symptoms. Finally, 3D LIVE-streaming could set a new standard for immersive online gaming as well as augmenting online and mobile experiences (e.g., video chat, social sharing/events).

A causal role for the right angular gyrus in self-location mediated perspective taking

Recent theories suggest that self-consciousness, in its most elementary form, is functionally disconnected from the phenomenal body. Patients with psychosis frequently misattribute their thoughts and actions to external sources; and in certain out-of-body experiences, lucid states, and dreams body-ownership is absent but self-identification is preserved. To explain these unusual experiences, we hypothesized that self-identification depends on inferring self-location at the right angular gyrus (i.e., perspective-taking). This process relates to the discrimination of self-produced signals (endogenous attention) from environmental stimulation (exogenous attention). Therefore, when this mechanism fails, this causes altered sensations and perceptions. We combined a Full-body Illusion paradigm with brain stimulation (HD-tDCS) and found a clear causal association between right angular gyrus activation and alterations in self-location (perspective-taking). Anodal versus sham HD-tDCS resulted in: a more profound out-of-body shift (with reduced sense of agency); and a weakened ability to discriminate self from other perspectives. We conclude that self-identification is mediated in the brain by inferring self-location (i.e., perspective-taking). Self-identification can be decoupled from the bodily self, explaining phenomena associated with disembodiment. These findings present novel insights into the relationship between mind and body, and may offer important future directions for treating psychosis symptoms and rehabilitation programs to aid in the recovery from a nervous system injury. The brain's ability to locate itself might be the key mechanism for self-identification and distinguishing self from other signals (i.e., perspective-taking).

UnReal? Investigating the Sense of Reality and Psychotic Symptoms with Virtual Reality

Distortions of reality, such as hallucinations, are common symptoms of many psychiatric conditions. Accordingly, sense of reality (SoR), the ability to discriminate between true and false perceptions, is a central criterion in the assessment of neurological and psychiatric health. Despite the critical role of the SoR in daily life, little is known about how this is formed in the mind. Here, we propose a novel theoretical and methodological framework to study the SoR and its relation to psychotic symptoms. In two experiments, we employed a specialized immersive virtual reality (VR) environment allowing for well-controlled manipulations of visual reality. We first tested the impact of manipulating visual reality on objective perceptual thresholds (just noticeable differences). In a second experiment, we tested how these manipulations affected subjective judgments of reality. The results revealed that the objective perceptual thresholds were robust and replicable, demonstrating that SoR is a stable psychometric property that can be measured experimentally. Furthermore, reality alterations reduced subjective reality judgments across all manipulated visual aspects. Finally, reduced sensitivity to changes in visual reality was related to self-reported prodromal psychotic symptoms. These results provide evidence for the relevance of SoR in the assessment of psychosis and other mental disorders in which reality is distorted.

Consolidation and retention of motor skill after motor imagery training

Complex motor tasks are learned through training which results in lasting improvement in sensorimotor performance and accuracy. Learning a motor skill is commonly attained via physical execution. However, research has shown that cognitive training, such as motor imagery (MI), effectively facilitates skill learning. Neurophysiological findings suggest that learning-induced plasticity in the human motor cortex, subserving consolidation and retention of motor skills, is stronger after movement execution (ME) than after MI training. Here, we designed an experimental task able to test for the fast and slow learning phases and for retention of motor skills for both MI and ME. We hypothesize that differences between MI and ME training would emerge in terms of reduced consolidation and retention of motor skills. Twenty-four young healthy subjects were divided into two groups, performing MI or ME training. Participants wore sensor-engineered gloves and their sensorimotor performance was assessed over a period of 15 days with 4-days training. We analysed the touch duration (TD), the inter-tapping interval (ITI), movement rate and accuracy. Results showed that (i) during the first phase of acquisition of motor skills, sensorimotor performance improved similarly in MI and ME groups; (ii) during the second learning phase movement rate increased more in ME than MI group and this difference was mainly driven by differences in the duration of TD; (iii) consolidation deficits with MI training reflected in impaired retention of the acquired skills, as TD and ITI were larger and movement rate was lower in the MI group with respect to the ME, till to 10 days after the last training session. Explicit component of motor learning, accuracy, was maintained in retention phase in both groups. Following our hypothesis, our findings show that MI training is as effective as ME within the first learning phase, but consolidation and retention of motor skills are less effective following MI training. This study highlights MI limitations and suggests option to enhance MI, as by providing an external sensory feedback.

Visuo-motor and interoceptive influences on peripersonal space representation following spinal cord injury

Peripersonal space (PPS) representation is modulated by information coming from the body. In paraplegic individuals, whose lower limb sensory-motor functions are impaired or completely lost, the representation of PPS around the feet is reduced. However, passive motion can have short-term restorative effects. What remains unclear is the mechanisms underlying this recovery, in particular with regard to the contribution of visual and motor feedback and of interoception. Using virtual reality technology, we dissociated the motor and visual feedback during passive motion in paraplegics with complete and incomplete lesions and in healthy controls. The results show that in the case of paraplegics, the presence of motor feedback was necessary for the recovery of PPS representation, both when the motor feedback was congruent and when it was incongruent with the visual feedback. In contrast, visuo-motor incongruence led to an inhibition of PPS representation in the control group. There were no differences in sympathetic responses between the three groups. Nevertheless, in individuals with incomplete lesions, greater interoceptive sensitivity was associated with a better representation of PPS around the feet in the visuo-motor incongruent conditions. These results shed new light on the modulation of PPS representation, and demonstrate the importance of residual motor feedback and its integration with other bodily information in maintaining space representation.

Embedded Bio-Mimetic System for Functional Electrical Stimulation Controlled by Event-Driven sEMG

The analysis of the surface ElectroMyoGraphic (sEMG) signal for controlling the Functional Electrical Stimulation (FES) therapy is being widely accepted as an active rehabilitation technique for the restoration of neuro-muscular disorders. Portability and real-time functionalities are major concerns, and, among others, two correlated challenges are the development of an embedded system and the implementation of lightweight signal processing approaches. In this respect, the event-driven nature of the Average Threshold Crossing (ATC) technique, considering its high correlation with the muscle force and the sparsity of its representation, could be an optimal solution. In this paper we present an embedded ATC-FES control system equipped with a multi-platform software featuring an easy-to-use Graphical User Interface (GUI). The system has been first characterized and validated by analyzing CPU and memory usage in different operating conditions, as well as measuring the system latency (fulfilling the real-time requirements with a 140 ms FES definition process). We also confirmed system effectiveness, testing it on 11 healthy subjects: The similarity between the voluntary movement and the stimulate one has been evaluated, computing the cross-correlation coefficient between the angular signals acquired during the limbs motion. We obtained high correlation values of 0.87 ± 0.07 and 0.93 ± 0.02 for the elbow flexion and knee extension exercises, respectively, proving good stimulation application in real therapy-scenarios.

A survey on what Australians with upper limb difference want in a prosthesis: justification for using soft robotics and additive manufacturing for customized prosthetic hands

Purpose: Upper limb prostheses are part of a rapidly changing market place. Despite development in device design, surveys report low levels of uptake and dissatisfaction with current prosthetic design. In this study, we present the results of a survey conducted with people with upper limb difference in Australia on their use of current prostheses and preferences in a prosthetic in order to inform future prosthetic hand design.Methods: An online survey was conducted on upper limb amputees, with 27 respondents that completed the survey. The survey was a mixture of open-ended questions, ranking design features and quantitative questions on problems experienced and desired attributes of future prosthesis designs.Results: Common key issues and concerns were isolated in the survey related to the weight, manipulation and dexterity, aesthetics, sensory feedback and financial cost; each of which could be addressed by additive manufacturing and soft robotics techniques.Conclusions: The adaptability of additive manufacturing and soft robotics to the highlighted concerns of participants shows that further research into these techniques is a feasible method to improve patient satisfaction and acceptance in prosthetic hands.Implications for rehabilitationEven with recent developments and advances in prosthetic design, the needs and desires of prosthetic users are not being met with current products.The desires and needs of those with upper limb difference are diverse.Using additive manufacturing to produce prosthetics allows for mass customization of prosthetics to meet these diverse needs while reducing costs.A soft robotic approach to prosthetics can help meet the desires of reducing weight and costs, while maintaining functionality.

Restoration of motor control and proprioceptive and cutaneous sensation in humans with prior upper-limb amputation via multiple Utah Slanted Electrode Arrays (USEAs) implanted in residual peripheral arm nerves

Background

Despite advances in sophisticated robotic hands, intuitive control of and sensory feedback from these prostheses has been limited to only 3-degrees-of-freedom (DOF) with 2 sensory percepts in closed-loop control. A Utah Slanted Electrode Array (USEA) has been used in the past to provide up to 81 sensory percepts for human amputees. Here, we report on the advanced capabilities of multiple USEAs implanted in the residual peripheral arm nerves of human amputees for restoring control of 5 DOF and sensation of up to 131 proprioceptive and cutaneous hand sensory percepts. We also demonstrate that USEA-restored sensory percepts provide a useful source of feedback during closed-loop virtual prosthetic hand control.

Methods

Two 100-channel USEAs were implanted for 4–5 weeks, one each in the median and ulnar arm nerves of two human subjects with prior long-duration upper-arm amputations. Intended finger and wrist positions were decoded from neuronal firing patterns via a modified Kalman filter, allowing subjects to control many movements of a virtual prosthetic hand. Additionally, USEA microstimulation was used to evoke numerous sensory percepts spanning the phantom hand. Closed-loop control was achieved by stimulating via an electrode of the ulnar-nerve USEA while recording and decoding movement via the median-nerve USEA.

Results

Subjects controlled up to 12 degrees-of-freedom during informal, ‘freeform’ online movement decode sessions, and experienced up to 131 USEA-evoked proprioceptive and cutaneous sensations spanning the phantom hand. Independent control was achieved for a 5-DOF real-time decode that included flexion/extension of the thumb, index, middle, and ring fingers, and the wrist. Proportional control was achieved for a 4-DOF real-time decode. One subject used a USEA-evoked hand sensation as feedback to complete a 1-DOF closed-loop virtual-hand movement task. There were no observed long-term functional deficits due to the USEA implants.

Conclusions

Implantation of high-channel-count USEAs enables multi-degree-of-freedom control of virtual prosthetic hand movement and restoration of a rich selection of both proprioceptive and cutaneous sensory percepts spanning the hand during the short 4–5 week post-implant period. Future USEA use in longer-term implants and in closed-loop may enable restoration of many of the capabilities of an intact hand while contributing to a meaningful embodiment of the prosthesis.

Electronic supplementary material

The online version of this article (10.1186/s12984-017-0320-4) contains supplementary material, which is available to authorized users.

State-Based Delay Representation and Its Transfer from a Game of Pong to Reaching and Tracking

To accurately estimate the state of the body, the nervous system needs to account for delays between signals from different sensory modalities. To investigate how such delays may be represented in the sensorimotor system, we asked human participants to play a virtual pong game in which the movement of the virtual paddle was delayed with respect to their hand movement. We tested the representation of this new mapping between the hand and the delayed paddle by examining transfer of adaptation to blind reaching and blind tracking tasks. These blind tasks enabled to capture the representation in feedforward mechanisms of movement control. A Time Representation of the delay is an estimation of the actual time lag between hand and paddle movements. A State Representation is a representation of delay using current state variables: the distance between the paddle and the ball originating from the delay may be considered as a spatial shift; the low sensitivity in the response of the paddle may be interpreted as a minifying gain; and the lag may be attributed to a mechanical resistance that influences paddle's movement. We found that the effects of prolonged exposure to the delayed feedback transferred to blind reaching and tracking tasks and caused participants to exhibit hypermetric movements. These results, together with simulations of our representation models, suggest that delay is not represented based on time, but rather as a spatial gain change in visuomotor mapping.

Cortical and subcortical mechanisms of brain-machine interfaces

Technical advances in the field of Brain-Machine Interfaces (BMIs) enable users to control a variety of external devices such as robotic arms, wheelchairs, virtual entities and communication systems through the decoding of brain signals in real time. Most BMI systems sample activity from restricted brain regions, typically the motor and premotor cortex, with limited spatial resolution. Despite the growing number of applications, the cortical and subcortical systems involved in BMI control are currently unknown at the whole-brain level. Here, we provide a comprehensive and detailed report of the areas active during on-line BMI control. We recorded functional magnetic resonance imaging (fMRI) data while participants controlled an EEG-based BMI inside the scanner. We identified the regions activated during BMI control and how they overlap with those involved in motor imagery (without any BMI control). In addition, we investigated which regions reflect the subjective sense of controlling a BMI, the sense of agency for BMI-actions. Our data revealed an extended cortical-subcortical network involved in operating a motor-imagery BMI. This includes not only sensorimotor regions but also the posterior parietal cortex, the insula and the lateral occipital cortex. Interestingly, the basal ganglia and the anterior cingulate cortex were involved in the subjective sense of controlling the BMI. These results inform basic neuroscience by showing that the mechanisms of BMI control extend beyond sensorimotor cortices. This knowledge may be useful for the development of BMIs that offer a more natural and embodied feeling of control for the user. Hum Brain Mapp 38:2971-2989, 2017. © 2017 Wiley Periodicals, Inc.

Differential neural encoding of sensorimotor and visual body representations

Sensorimotor processing specifically impacts mental body representations. In particular, deteriorated somatosensory input (as after complete spinal cord injury) increases the relative weight of visual aspects of body parts’ representations, leading to aberrancies in how images of body parts are mentally manipulated (e.g. mental rotation). This suggests that a sensorimotor or visual reference frame, respectively, can be relatively dominant in local (hands) versus global (full-body) bodily representations. On this basis, we hypothesized that the recruitment of a specific reference frame could be reflected in the activation of sensorimotor versus visual brain networks. To this aim, we directly compared the brain activity associated with mental rotation of hands versus full-bodies. Mental rotation of hands recruited more strongly the supplementary motor area, premotor cortex, and secondary somatosensory cortex. Conversely, mental rotation of full-bodies determined stronger activity in temporo-occipital regions, including the functionally-localized extrastriate body area. These results support that (1) sensorimotor and visual frames of reference are used to represent the body, (2) two distinct brain networks encode local or global bodily representations, and (3) the extrastriate body area is a multimodal region involved in body processing both at the perceptual and representational level.