Integration of visual and auditory information for hand actions: preliminary evidence for the contribution of natural sounds to grasping

Influence of the Somatic Rubber Hand Illusion on Maximum Grip Aperture

The classic rubber hand illusion (RHI), based on visual, proprioceptive, and tactile feedback, can affect actions. However, it is not known whether these effects still occur if the paradigm is administered without visual feedback. In this study, we used the somatic RHI to test in thirty-two healthy individuals whether the incorporation of the rubber hand based on proprioceptive and tactile information only is sufficient to generate changes in actions. We measured maximum grip aperture (GA) changes towards a target and associated brain activations within the dorsal stream before and after the somatic RHI. Behavioural and neuroimaging data do not support an effect on maximum GA when the RHI is based on proprioceptive and tactile information only.

Forward optic flow is prioritised in visual awareness independently of walking direction

When two different images are presented separately to each eye, one experiences smooth transitions between them-a phenomenon called binocular rivalry. Previous studies have shown that exposure to signals from other senses can enhance the access of stimulation-congruent images to conscious perception. However, despite our ability to infer perceptual consequences from bodily movements, evidence that action can have an analogous influence on visual awareness is scarce and mainly limited to hand movements. Here, we investigated whether one's direction of locomotion affects perceptual access to optic flow patterns during binocular rivalry. Participants walked forwards and backwards on a treadmill while viewing highly-realistic visualisations of self-motion in a virtual environment. We hypothesised that visualisations congruent with walking direction would predominate in visual awareness over incongruent ones, and that this effect would increase with the precision of one's active proprioception. These predictions were not confirmed: optic flow consistent with forward locomotion was prioritised in visual awareness independently of walking direction and proprioceptive abilities. Our findings suggest the limited role of kinaesthetic-proprioceptive information in disambiguating visually perceived direction of self-motion and indicate that vision might be tuned to the (expanding) optic flow patterns prevalent in everyday life.

Reach-to-Grasp: A Multisensory Experience

The reach-to-grasp movement is ordinarily performed in everyday living activities and it represents a key behavior that allows humans to interact with their environment. Remarkably, it serves as an experimental test case for probing the multisensory architecture of goal-oriented actions. This review focuses on experimental evidence that enhances or modifies how we might conceptualize the "multisensory" substrates of prehension. We will review evidence suggesting that how reach-to-grasp movements are planned and executed is influenced by information coming from different sensory modalities such as vision, proprioception, audition, taste, and olfaction. The review closes with some considerations about the predominant role of the multisensory constituents in shaping prehensile behavior and how this might be important for future research developments, especially in the rehabilitative domain.

Multisensory action effects facilitate the performance of motor sequences

Research has shown that contingent, distinct action effects have a beneficial influence on motor sequence performance. Previous studies showed the beneficial influence of task-irrelevant action effects from one modality (auditory) on motor sequence performance, compared with no task-irrelevant action effects. The present study investigated the influence of task-irrelevant action effects on motor sequence performance from a multiple-modality perspective. We compared motor sequence performances of participants who received different task-irrelevant action effects in an auditory, visual, or audiovisual condition. In the auditory condition, key presses produced tones of a C-major scale that mapped to keys from left to right in ascending order. In the visual condition, key presses produced rectangles in different locations on the screen that mapped to keys from left to right in ascending order. In the audiovisual condition, both tone and rectangle effects were produced simultaneously by key presses. There were advantages for the audiovisual group in motor sequence initiation and execution. The results implied that, compared with unimodal action effects, action effects from multiple sensory modalities can prime an action faster and strengthen associations between successive actions, leading to faster motor sequence performance.

Effects of auditory feedback on movements with two-segment sequence and eye-hand coordination: Using a short auditory contact cue

During sequential reaches to multiple targets, eye and hand movements are highly coordinated, and the gaze is anchored to each target until the reaching hand makes contact to each of them. Such contact events are monitored by multimodal (visual, proprioceptive) sensory systems, and one function of the gaze anchoring to each target is verification of successful target contact (reach completion). The present study focused on this verification function and examined how planning and control of eye and hand movements during two-segment eye-hand movements are affected by augmented auditory feedback of reach completion. Young adults made a reach to the first target with a saccade, and then made another saccade to the second target in blocked trials. An auditory target-contact cue condition delivered four short sounds during the initial reach, and the last sound was synchronized with target contact, whereas a control condition lacked the last target-contact sound. The results showed that saccadic reaction time increased with the target-contact cue, especially when the reaching accuracy demand was high. The reach also became slower with lower peak velocity and longer time to peak velocity with that cue, suggesting that the limb-motor system lower the preplanned speed of the reach in a top-down fashion for a better preparation toward reach completion. However, no auditory effects were found for the timing of gaze shift to the second target. These results were different from those seen in previous studies, indicating that the effects of the additional auditory contact feedback differ depending on behavioral tasks and cue characteristics.

The Influence of Auditory Cues on Bodily and Movement Perception

The sounds that result from our movement and that mark the outcome of our actions typically convey useful information concerning the state of our body and its movement, as well as providing pertinent information about the stimuli with which we are interacting. Here we review the rapidly growing literature investigating the influence of non-veridical auditory cues (i.e., inaccurate in terms of their context, timing, and/or spectral distribution) on multisensory body and action perception, and on motor behavior. Inaccurate auditory cues provide a unique opportunity to study cross-modal processes: the ability to detect the impact of each sense when they provide a slightly different message is greater. Additionally, given that similar cross-modal processes likely occur regardless of the accuracy or inaccuracy of sensory input, studying incongruent interactions are likely to also help us predict interactions between congruent inputs. The available research convincingly demonstrates that perceptions of the body, of movement, and of surface contact features (e.g., roughness) are influenced by the addition of non-veridical auditory cues. Moreover, auditory cues impact both motor behavior and emotional valence, the latter showing that sounds that are highly incongruent with the performed movement induce feelings of unpleasantness (perhaps associated with lower processing fluency). Such findings are relevant to the design of auditory cues associated with product interaction, and the use of auditory cues in sport performance and therapeutic situations given the impact on motor behavior.

Attentional disengagement from negative natural sounds for high-anxious individuals

BACKGROUND AND OBJECTIVES

Previous studies have not consistently concluded whether high-anxious persons exhibit attentional bias towards negative natural auditory stimuli. The present study explores whether auditory negative stimuli could induce attentional bias to negative sounds in real life and investigates the exact nature of these biases using an emotional spatial cueing task.

DESIGN

Experimental study with a mixed factorial design.

METHOD

We created two groups according to the state-trait anxiety scale, namely high and low trait anxiety. Participants (N = 68 undergraduate students) were required to respond to an auditory target after receiving a negative (aversive sounds from natural life) or neutral auditory stimuli.

RESULTS

A 2 (Validity: valid/invalid)   × 2 (Cue Valence: negative/neutral) × 2 (Anxiety Group: LA/HA) repeated-measures ANOVA on reaction times revealed that participants with high trait anxiety exhibited slower reaction times in invalid trials following negative cues than following neutral cues. Higher levels of trait anxiety were associated with more difficult attentional disengagement from negative auditory information.

CONCLUSIONS

The results demonstrate that impaired attentional disengagement was one of the mechanisms by which high-anxious participants exhibited auditory attentional bias to natural negative information.

Effects of auditory feedback on movements with two-segment sequence and eye-hand coordination

The present study investigated the effect of auditory feedback on planning and control of two-segment reaching movements and eye-hand coordination. In particular, it was examined whether additional auditory information indicating the progression of the initial reach (i.e., passing the midway and contacting the target) affects the performance of that reach and gaze shift to the second target at the transition between two segments. Young adults performed a rapid two-segment reaching task, in which both the first and second segments had two target sizes. One out of three auditory feedback conditions included the reach-progression information: a continuous tone was delivered at a consistent timing during the initial reach from the midway to the target contact. Conversely, the other two were control conditions: a continuous tone was delivered at a random timing in one condition or not delivered in the other. The results showed that the initial reach became more accurate with the auditory reach-progression cue compared to without any auditory cue. When that cue was available, movement time of the initial reach was decreased, which was accompanied by an increased peak velocity and a decreased time to peak velocity. These findings suggest that the auditory reach-progression feedback enhanced the preplanned control of the initial reach. Deceleration time of that reach was also decreased with auditory feedback, but it was observed regardless of whether the sound contained the reach-progression information. At the transition between the two segments, the onset latencies of both the gaze shift and reach to the second target became shorter with the auditory reach-progression cue, the effect of which was pronounced when the initial reach had a higher terminal accuracy constraint. This suggests that the reach-progression cue enhanced verification of the termination of initial reach, thereby facilitating the initiation of eye and hand movements to the second target. Taken together, the additional auditory information of reach-progression enhances the planning and control of multi-segment reaches and eye-hand coordination at the segment transition.

To Hear or Not to Hear: Sound Availability Modulates Sensory-Motor Integration

When we walk in place with our eyes closed after a few minutes of walking on a treadmill, we experience an unintentional forward body displacement (drift), called the sensory-motor aftereffect. Initially, this effect was thought to be due to the mismatch experienced during treadmill walking between the visual (absence of optic flow signaling body steadiness) and proprioceptive (muscle spindles firing signaling body displacement) information. Recently, the persistence of this effect has been shown even in the absence of vision, suggesting that other information, such as the sound of steps, could play a role. To test this hypothesis, six cochlear-implanted individuals were recruited and their forward drift was measured before (Control phase) and after (Post Exercise phase) walking on a treadmill while having their cochlear system turned on and turned off. The relevance in testing cochlear-implanted individuals was that when their system is turned off, they perceive total silence, even eliminating the sounds normally obtained from bone conduction. Results showed the absence of the aftereffect when the system was turned off, underlining the fundamental role played by sounds in the control of action and breaking new ground in the use of interactive sound feedback in motor learning and motor development.

Priming Gestures with Sounds

We report a series of experiments about a little-studied type of compatibility effect between a stimulus and a response: the priming of manual gestures via sounds associated with these gestures. The goal was to investigate the plasticity of the gesture-sound associations mediating this type of priming. Five experiments used a primed choice-reaction task. Participants were cued by a stimulus to perform response gestures that produced response sounds; those sounds were also used as primes before the response cues. We compared arbitrary associations between gestures and sounds (key lifts and pure tones) created during the experiment (i.e. no pre-existing knowledge) with ecological associations corresponding to the structure of the world (tapping gestures and sounds, scraping gestures and sounds) learned through the entire life of the participant (thus existing prior to the experiment). Two results were found. First, the priming effect exists for ecological as well as arbitrary associations between gestures and sounds. Second, the priming effect is greatly reduced for ecologically existing associations and is eliminated for arbitrary associations when the response gesture stops producing the associated sounds. These results provide evidence that auditory-motor priming is mainly created by rapid learning of the association between sounds and the gestures that produce them. Auditory-motor priming is therefore mediated by short-term associations between gestures and sounds that can be readily reconfigured regardless of prior knowledge.

Absence of modulatory action on haptic height perception with musical pitch

Although acoustic frequency is not a spatial property of physical objects, in common language, pitch, i.e., the psychological correlated of frequency, is often labeled spatially (i.e., “high in pitch” or “low in pitch”). Pitch-height is known to modulate (and interact with) the response of participants when they are asked to judge spatial properties of non-auditory stimuli (e.g., visual) in a variety of behavioral tasks. In the current study we investigated whether the modulatory action of pitch-height extended to the haptic estimation of height of a virtual step. We implemented a HW/SW setup which is able to render virtual 3D objects (stair-steps) haptically through a PHANTOM device, and to provide real-time continuous auditory feedback depending on the user interaction with the object. The haptic exploration was associated with a sinusoidal tone whose pitch varied as a function of the interaction point's height within (i) a narrower and (ii) a wider pitch range, or (iii) a random pitch variation acting as a control audio condition. Explorations were also performed with no sound (haptic only). Participants were instructed to explore the virtual step freely, and to communicate height estimation by opening their thumb and index finger to mimic the step riser height, or verbally by reporting the height in centimeters of the step riser. We analyzed the role of musical expertise by dividing participants into non-musicians and musicians. Results showed no effects of musical pitch on high-realistic haptic feedback. Overall there is no difference between the two groups in the proposed multimodal conditions. Additionally, we observed a different haptic response distribution between musicians and non-musicians when estimations of the auditory conditions are matched with estimations in the no sound condition.

Use of early phase online vision for grip configuration is modulated according to movement duration in prehension

Our previous study (Hum Mov Sci 25:349-371, 2006) investigated whether and how online vision in the early phase of movement influences the control of reach-to-grasp movements (movement duration: approximately 1000 ms). We used liquid-crystal shutter goggles to manipulate the duration of available online vision during the movement and specified that online vision during the early phase influences grasping movements. The current study examined the effect of online early phase vision on the grip configuration according to the movement duration and compared it between two different movement durations (approximately 500 and 1000 ms). We found that non-availability of early phase online vision affected the grip configuration (i.e., inducing a larger peak grip aperture) even in the shorter movement duration. The influential period for online vision for grasping control shifts to an earlier time when movement time is shorter (i.e., from approximately 214 to 106 ms after movement onset), indicating a flexible mechanism for grip configuration according to the movement duration and the available online vision.

Dorsal and ventral streams across sensory modalities

In this review, we describe the current models of dorsal and ventral streams in vision, audition and touch. Available theories take their first steps from the model of Milner and Goodale, which was developed to explain how human actions can be efficiently carried out using visual information. Since then, similar concepts have also been applied to other sensory modalities. We propose that advances in the knowledge of brain functioning can be achieved through models explaining action and perception patterns independently from sensory modalities.

Might as well jump: sound affects muscle activation in skateboarding

The aim of the study is to reveal the role of sound in action anticipation and performance, and to test whether the level of precision in action planning and execution is related to the level of sensorimotor skills and experience that listeners possess about a specific action. Individuals ranging from 18 to 75 years of age - some of them without any skills in skateboarding and others experts in this sport - were compared in their ability to anticipate and simulate a skateboarding jump by listening to the sound it produces. Only skaters were able to modulate the forces underfoot and to apply muscle synergies that closely resembled the ones that a skater would use if actually jumping on a skateboard. More importantly we showed that only skaters were able to plan the action by activating anticipatory postural adjustments about 200 ms after the jump event. We conclude that expert patterns are guided by auditory events that trigger proper anticipations of the corresponding patterns of movements.

Word-induced postural changes reflect a tight interaction between motor and lexico-semantic representations

A tight coupling between lexico-semantic access and motor control has been established on the basis of neuropsychological, neurophysiological, and behavioral evidence. For example, sensory and motor cortices have been shown to be active when subjects listen to words denoting bodily actions. Kinematic analyses of subjects' motor actions during the processing of linguistic stimuli provide further insights into the nature and time-course of this relationship. However, such studies have largely focused on individual body parts, in particular the upper limbs, thus neglecting the effect of language processing on lower or whole body representations. The present study bridges this gap by evaluating the interaction between linguistic processing and whole-body postural control during quiet standing. The results reveal a systematic influence of passive listening to action verbs, but not mental-state verbs, on measures of postural control, pointing to a clear and specific neural link between words conveying action concepts and whole-body motor functions.

Evidence for a basic level in a taxonomy of everyday action sounds

We searched for evidence that the auditory organization of categories of sounds produced by actions includes a privileged or "basic" level of description. The sound events consisted of single objects (or substances) undergoing simple actions. Performance on sound events was measured in two ways: sounds were directly verified as belonging to a category, or sounds were used to create lexical priming. The category verification experiment measured the accuracy and reaction time to brief excerpts of these sounds. The lexical priming experiment measured reaction time benefits and costs caused by the presentation of these sounds prior to a lexical decision. The level of description of a sound varied in how specifically it described the physical properties of the action producing the sound. Both identification and priming effects were superior when a label described the specific interaction causing the sound (e.g. trickling) in comparison to the following: (1) more general descriptions (e.g. pour, liquid: trickling is a specific manner of pouring liquid), (2) more detailed descriptions using adverbs to provide detail regarding the manner of the action (e.g. trickling evenly). These results are consistent with neuroimaging studies showing that auditory representations of sounds produced by actions familiar to the listener activate motor representations of the gestures involved in sound production.

When a bang makes you run away: spatial avoidance of threatening environmental sounds

Environmental sounds can be powerful alarm signals. Hence, attentional orienting towards their location might occur extremely rapidly. Here, we used the beep probe task to investigate attentional biases to negative, positive and taboo sounds. While both left-presented negative and taboo sounds elicited attentional avoidance, taboo but not negative sounds triggered Inhibition of Return. Moreover, taboo sounds slowed participants' responses, whatever the sound and beep locations. Positive sounds had no effect. Interestingly, although spatial effects specific to taboo sounds were related to their disgusting nature, their non-spatial effects were linked to their shocking/surprising trait. This is the first evidence of emotional sounds' influence on spatial attentional orienting and of the involved emotional dimensions.