Augmenting sensorimotor control using "goal-aware" vibrotactile stimulation during reaching and manipulation behaviors

Utility and Usability of Two Forms of Supplemental Vibrotactile Kinesthetic Feedback for Enhancing Movement Accuracy and Efficiency in Goal-Directed Reaching

Recent advances in wearable sensors and computing have made possible the development of novel sensory augmentation technologies that promise to enhance human motor performance and quality of life in a wide range of applications. We compared the objective utility and subjective user experience for two biologically inspired ways to encode movement-related information into supplemental feedback for the real-time control of goal-directed reaching in healthy, neurologically intact adults. One encoding scheme mimicked visual feedback encoding by converting real-time hand position in a Cartesian frame of reference into supplemental kinesthetic feedback provided by a vibrotactile display attached to the non-moving arm and hand. The other approach mimicked proprioceptive encoding by providing real-time arm joint angle information via the vibrotactile display. We found that both encoding schemes had objective utility in that after a brief training period, both forms of supplemental feedback promoted improved reach accuracy in the absence of concurrent visual feedback over performance levels achieved using proprioception alone. Cartesian encoding promoted greater reductions in target capture errors in the absence of visual feedback (Cartesian: 59% improvement; Joint Angle: 21% improvement). Accuracy gains promoted by both encoding schemes came at a cost in terms of temporal efficiency; target capture times were considerably longer (1.5 s longer) when reaching with supplemental kinesthetic feedback than without. Furthermore, neither encoding scheme yielded movements that were particularly smooth, although movements made with joint angle encoding were smoother than movements with Cartesian encoding. Participant responses on user experience surveys indicate that both encoding schemes were motivating and that both yielded passable user satisfaction scores. However, only Cartesian endpoint encoding was found to have passable usability; participants felt more competent using Cartesian encoding than joint angle encoding. These results are expected to inform future efforts to develop wearable technology to enhance the accuracy and efficiency of goal-directed actions using continuous supplemental kinesthetic feedback.

Extended training improves the accuracy and efficiency of goal-directed reaching guided by supplemental kinesthetic vibrotactile feedback

Prior studies have shown that the accuracy and efficiency of reaching can be improved using novel sensory interfaces to apply task-specific vibrotactile feedback (VTF) during movement. However, those studies have typically evaluated performance after less than 1 h of training using VTF. Here, we tested the effects of extended training using a specific form of vibrotactile cues-supplemental kinesthetic VTF-on the accuracy and temporal efficiency of goal-directed reaching. Healthy young adults performed planar reaching with VTF encoding of the moving hand's instantaneous position, applied to the non-moving arm. We compared target capture errors and movement times before, during, and after approximately 10 h (20 sessions) of training on the VTF-guided reaching task. Initial performance of VTF-guided reaching showed that people were able to use supplemental VTF to improve reaching accuracy. Performance improvements were retained from one training session to the next. After 20 sessions of training, the accuracy and temporal efficiency of VTF-guided reaching were equivalent to or better than reaches performed with only proprioception. However, hand paths during VTF-guided reaching exhibited a persistent strategy where movements were decomposed into discrete sub-movements along the cardinal axes of the VTF display. We also used a dual-task condition to assess the extent to which performance gains in VTF-guided reaching resist dual-task interference. Dual-tasking capability improved over the 20 sessions, such that the primary VTF-guided reaching and a secondary choice reaction time task were performed with increasing concurrency. Thus, VTF-guided reaching is a learnable skill in young adults, who can achieve levels of accuracy and temporal efficiency equaling or exceeding those observed during movements guided only by proprioception. Future studies are warranted to explore learnability in older adults and patients with proprioceptive deficits, who might benefit from using wearable sensory augmentation technologies to enhance control of arm movements.

Vibrotactile Perception for Sensorimotor Augmentation: Perceptual Discrimination of Vibrotactile Stimuli Induced by Low-Cost Eccentric Rotating Mass Motors at Different Body Locations in Young, Middle-Aged, and Older Adults

Sensory augmentation technologies are being developed to convey useful supplemental sensory cues to people in comfortable, unobtrusive ways for the purpose of improving the ongoing control of volitional movement. Low-cost vibration motors are strong contenders for providing supplemental cues intended to enhance or augment closed-loop feedback control of limb movements in patients with proprioceptive deficits, but who still retain the ability to generate movement. However, it remains unclear what form such cues should take and where on the body they may be applied to enhance the perception-cognition-action cycle implicit in closed-loop feedback control. As a step toward addressing this knowledge gap, we used low-cost, wearable technology to examine the perceptual acuity of vibrotactile stimulus intensity discrimination at several candidate sites on the body in a sample of participants spanning a wide age range. We also sought to determine the extent to which the acuity of vibrotactile discrimination can improve over several days of discrimination training. Healthy adults performed a series of 2-alternative forced choice experiments that quantified capability to perceive small differences in the intensity of stimuli provided by low-cost eccentric rotating mass vibration motors fixed at various body locations. In one set of experiments, we found that the acuity of intensity discrimination was poorer in older participants than in middle-aged and younger participants, and that stimuli applied to the torso were systematically harder to discriminate than stimuli applied to the forearm, knee, or shoulders, which all had similar acuities. In another set of experiments, we found that older adults could improve intensity discrimination over the course of 3 days of practice on that task such that their final performance did not differ significantly from that of younger adults. These findings may be useful for future development of wearable technologies intended to improve the control of movements through the application of supplemental vibrotactile cues.

Closed-loop Control using Electrotactile Feedback Encoded in Frequency and Pulse Width

Sensory substitution by electrotactile stimulation has been widely investigated for improving the functionality of human-machine interfaces. Few studies, however, have objectively compared different ways in which such systems can be implemented. In this study, we compare encoding of a feedback variable in stimulation pulse width or stimulation frequency during a closed-loop control task. Specifically, participants were asked to track a predefined pseudorandom trajectory using a joystick with electrotactile feedback as the only indication of the tracking error. Each participant performed eight 90 s trials per encoding scheme. Tracking performance using frequency modulation enabled lower tracking error (RMSE: Frequency modulation: 0.27 ± 0.03; Pulse width modulation: 0.31 ± 0.05; p < 0.05) and a higher correlation with the target trajectory (Frequency modulation: 83.4 ± 4.1%; Pulse width modulation: 79.8 ± 5.2%; p < 0.05). There was no significant improvement in performance over the eight trials. Furthermore, frequency-domain analysis revealed that frequency modulation was characterized with a higher gain at lower error frequencies. In summary, the results indicate that encoding of feedback variables in the frequency of pulses enables better control than pulse width modulation in closed-loop dynamic tasks.

The effect of tactile augmentation on manipulation and grip force control during force-field adaptation

Background

When exposed to a novel dynamic perturbation, participants adapt by changing their movements’ dynamics. This adaptation is achieved by constructing an internal representation of the perturbation, which allows for applying forces that compensate for the novel external conditions. To form an internal representation, the sensorimotor system gathers and integrates sensory inputs, including kinesthetic and tactile information about the external load. The relative contribution of the kinesthetic and tactile information in force-field adaptation is poorly understood.

Methods

In this study, we set out to establish the effect of augmented tactile information on adaptation to force-field. Two groups of participants received a velocity-dependent tangential skin deformation from a custom-built skin-stretch device together with a velocity-dependent force-field from a kinesthetic haptic device. One group experienced a skin deformation in the same direction of the force, and the other in the opposite direction. A third group received only the velocity-dependent force-field.

Results

We found that adding a skin deformation did not affect the kinematics of the movement during adaptation. However, participants who received skin deformation in the opposite direction adapted their manipulation forces faster and to a greater extent than those who received skin deformation in the same direction of the force. In addition, we found that skin deformation in the same direction to the force-field caused an increase in the applied grip-force per amount of load force, both in response and in anticipation of the stretch, compared to the other two groups.

Conclusions

Augmented tactile information affects the internal representations for the control of manipulation and grip forces, and these internal representations are likely updated via distinct mechanisms. We discuss the implications of these results for assistive and rehabilitation devices.

Spatial and temporal influences on discrimination of vibrotactile stimuli on the arm

Body-machine interfaces (BMIs) provide a non-invasive way to control devices. Vibrotactile stimulation has been used by BMIs to provide performance feedback to the user, thereby reducing visual demands. To advance the goal of developing a compact, multivariate vibrotactile display for BMIs, we performed two psychophysical experiments to determine the acuity of vibrotactile perception across the arm. The first experiment assessed vibration intensity discrimination of sequentially presented stimuli within four dermatomes of the arm (C5, C7, C8, and T1) and on the ulnar head. The second experiment compared vibration intensity discrimination when pairs of vibrotactile stimuli were presented simultaneously vs. sequentially within and across dermatomes. The first experiment found a small but statistically significant difference between dermatomes C7 and T1, but discrimination thresholds at the other three locations did not differ. Thus, while all tested dermatomes of the arm and hand could serve as viable sites of vibrotactile stimulation for a practical BMI, ideal implementations should account for small differences in perceptual acuity across dermatomes. The second experiment found that sequential delivery of vibrotactile stimuli resulted in better intensity discrimination than simultaneous delivery, independent of whether the pairs were located within the same dermatome or across dermatomes. Taken together, our results suggest that the arm may be a viable site to transfer multivariate information via vibrotactile feedback for body-machine interfaces. However, user training may be needed to overcome the perceptual disadvantage of simultaneous vs. sequentially presented stimuli.

Robotic interfaces for cognitive psychology and embodiment research: A research roadmap

Advanced human-machine interfaces render robotic devices applicable to study and enhance human cognition. This turns robots into formidable neuroscientific tools to study processes such as the adaptation between a human operator and the operated robotic device and how this adaptation modulates human embodiment and embodied cognition. We analyze bidirectional human-machine interface (bHMI) technologies for transparent information transfer between a human and a robot via efferent and afferent channels. Even if such interfaces have a tremendous positive impact on feedback loops and embodiment, advanced bHMIs face immense technological challenges. We critically discuss existing technical approaches, mainly focusing on haptics, and suggest extensions thereof, which include other aspects of touch. Moreover, we point out other potential constraints such as limited functionality, semi-autonomy, intent-detection, and feedback methods. From this, we develop a research roadmap to guide understanding and development of bidirectional human-machine interfaces that enable robotic experiments to empirically study the human mind and embodiment. We conclude the integration of dexterous control and multisensory feedback to be a promising roadmap towards future robotic interfaces, especially regarding applications in the cognitive sciences. This article is categorized under: Computer Science > Robotics Psychology > Motor Skill and Performance Neuroscience > Plasticity.

Effect of Dual Tasking on Vibrotactile Feedback Guided Reaching - a Pilot Study

Vibrotactile feedback (VTF) has been proposed as a non-invasive way to augment impaired or lost kinesthetic feedback in certain patient populations, thereby enhancing the real-time control of purposeful limb movements and quality of life. We used a dual tasking scenario to investigate the effects of cognitive load and short-term VTF training on VTF-guided reaching. Participants grasped the handle of a planar manipulandum with one hand and received VTF of its motion via a vibrotactile display attached to the non-moving arm. We asked participants to simultaneously perform VTF-guided reaching and a choice reaction time task both before and after training with VTF-guided reaching. Participants readily used VTF to guide goal-directed hand movements in the absence of visual feedback in the dual-task setting, even prior to training. This capability came at the cost of increased movement completion time. Short-term training on VTF-guided reaching induced significant improvements in target capture errors. Pre- and post-training comparisons of dual-task performance found training-related improvements in VTF-guided reach accuracy were resistant to dual-task interference. We found no training-related improvements in movement completion time or button press performance. These results indicate that VTF can be used to complete goal-directed reaches in a dual task situation, and that a single short bout of training sufficed for participants to begin the transition between the cognitive and associative phases of learning for the integration of VTF into the planning and ongoing control of reaching movements.

Elucidating Sensorimotor Control Principles with Myoelectric Musculoskeletal Models

There is an old saying that you must walk a mile in someone's shoes to truly understand them. This mini-review will synthesize and discuss recent research that attempts to make humans "walk a mile" in an artificial musculoskeletal system to gain insight into the principles governing human movement control. In this approach, electromyography (EMG) is used to sample human motor commands; these commands serve as inputs to mathematical models of muscular dynamics, which in turn act on a model of skeletal dynamics to produce a simulated motor action in real-time (i.e., the model's state is updated fast enough produce smooth motion without noticeable transitions; Manal et al., 2002). In this mini-review, these are termed myoelectric musculoskeletal models (MMMs). After a brief overview of typical MMM design and operation principles, the review will highlight how MMMs have been used for understanding human sensorimotor control and learning by evoking apparent alterations in a user's biomechanics, neural control, and sensory feedback experiences.

Supplemental vibrotactile feedback control of stabilization and reaching actions of the arm using limb state and position error encodings

Background

Deficits of kinesthesia (limb position and movement sensation) commonly limit sensorimotor function and its recovery after neuromotor injury. Sensory substitution technologies providing synthetic kinesthetic feedback might re-establish or enhance closed-loop control of goal-directed behaviors in people with impaired kinesthesia.

Methods

As a first step toward this goal, we evaluated the ability of unimpaired people to use vibrotactile sensory substitution to enhance stabilization and reaching tasks. Through two experiments, we compared the objective and subjective utility of two forms of supplemental feedback – limb state information or hand position error – to eliminate hand position drift, which develops naturally during stabilization tasks after removing visual feedback.

Results

Experiment 1 optimized the encoding of limb state feedback; the best form included hand position and velocity information, but was weighted much more heavily toward position feedback. Upon comparing optimal limb state feedback vs. hand position error feedback in Experiment 2, we found both encoding schemes capable of enhancing stabilization and reach performance in the absence of vision. However, error encoding yielded superior outcomes - objective and subjective - due to the additional task-relevant information it contains.

Conclusions

The results of this study have established the immediate utility and relative merits of two forms of vibrotactile kinesthetic feedback in enhancing stabilization and reaching actions performed with the arm and hand in neurotypical people. These findings can guide future development of vibrotactile sensory substitution technologies for improving sensorimotor function after neuromotor injury in survivors who retain motor capacity, but lack proprioceptive integrity in their more affected arm.

Electronic supplementary material

The online version of this article (doi:10.1186/s12984-017-0248-8) contains supplementary material, which is available to authorized users.