Action-related auditory ERP attenuation is not modulated by action effect relevance

Event-related potentials (ERPs) elicited by self-induced sounds are often smaller than ERPs elicited by identical, but externally generated sounds. This action-related auditory ERP attenuation is more pronounced when self-induced sounds are intermixed with similar sounds generated by an external source. The current study explored whether attentional factors contributed to this phenomenon. Participants performed tone-eliciting actions, while the action-tone contingency and the set of additional action effects (tactile only, tactile and visual) were manipulated in a blocked manner. Previous action-tone contingence-effects were replicated, but the addition of other sensory action consequences did not influence the magnitude of auditory ERP attenuation. This suggests that the amount of attention allocated to concurrent non-auditory action effects does not substantially affect the magnitude of action-related auditory ERP attenuation and is on a par with the assumption that action-related auditory ERP attenuation might be related to the process of distinguishing self-induced stimuli from externally generated ones.

Temporal persistence of after-effects in the n-1 replication task

In line with the theory of event coding, many studies on tool use show that perceived visual and haptic information interacts with action execution. In two experiments, we investigated the temporal persistence of after-effects within an event file, and after-effects in temporally overlapping event files with the n-1 replication task. Each trial consisted of two phases: In phase 1, participants moved a cursor with a pen on a covered tablet while a gain varied the relation between hand and cursor amplitude (Experiment 1). In phase 2, participants replicated the hand amplitude of phase 1 of the previous trial without visual feedback. Any systematic over- and undershoot would be indicative for after-effects. When the cursor amplitude varied and the hand amplitude remained constant, we did not find any after-effects but adjustment of the internal model. For varying hand amplitudes, after-effects appeared in terms of a contrast and assimilation effect between temporally overlapping event files and within an event file, respectively. In Experiment 2, we confirmed that the observed pattern of over- and undershoots fully reflect assimilation/contrast due to perception-action interaction. The findings extend the current view on the temporal stability of short-term binding in sensorimotor transformation tasks: In the n-1 replication task, after-effects appeared only in trials with varying hand amplitudes. We replicated the contrast effect and assimilation effect, and the assimilation effect persisted for up to approximately 20 s.

A condition that produces sensory recalibration and abolishes multisensory integration

We examined the influence of extended exposure to a visuomotor rotation, which induces both motor adaptation and sensory recalibration, on (partial) multisensory integration in a cursor-control task. Participants adapted to a 30° (adaptation condition) or 0° (control condition) visuomotor rotation by making center-out movements to remembered targets. In subsequent test trials of sensory integration, they made center-out movements with variable visuomotor rotations and judged the position of hand or cursor at the end of these movements. Test trials were randomly embedded among twice the number of maintenance trials with 30° or 0° rotation. The biases of perceived hand (or cursor) position toward the cursor (or hand) position were measured. We found motor adaptation together with proprioceptive and visual recalibrations in the adaptation condition. Unexpectedly, multisensory integration was absent in both the adaptation and control condition. The absence stemmed from the extensive experience of constant visuomotor rotations of 30° or 0°, which probably produced highly precise predictions of the visual consequences of hand movements. The frequently confirmed predictions then dominated the estimate of the visual movement consequences, leaving no influence of the actual visuomotor rotations in the minority of test trials. Conversely, multisensory integration was present for sensed hand positions when these were indirectly assessed from movement characteristics, indicating that the relative weighting of discrepant estimates of hand position was different for motor control. The existence of a condition that abolishes multisensory integration while keeping sensory recalibration suggests that mechanisms that reduce sensory discrepancies (partly) differ between integration and recalibration.

Effects of Hand and Hemispace on Multisensory Integration of Hand Position and Visual Feedback

The brain generally integrates a multitude of sensory signals to form a unified percept. Even in cursor control tasks, such as reaching while looking at rotated visual feedback on a monitor, visual information on cursor position and proprioceptive information on hand position are partially integrated (sensory coupling), resulting in mutual biases of the perceived positions of cursor and hand. Previous studies showed that the strength of sensory coupling (sum of the mutual biases) depends on the experience of kinematic correlations between hand movements and cursor motions, whereas the asymmetry of sensory coupling (difference between the biases) depends on the relative reliabilities (inverse of variability) of hand-position and cursor-position estimates (reliability rule). Furthermore, the precision of movement control and perception of hand position are known to differ between hands (left, right) and workspaces (ipsilateral, contralateral), and so does the experience of kinematic correlations from daily life activities. Thus, in the present study, we tested whether strength and asymmetry of sensory coupling for the endpoints of reaches in a cursor control task differ between the right and left hand and between ipsilateral and contralateral hemispace. No differences were found in the strength of sensory coupling between hands or between hemispaces. However, asymmetry of sensory coupling was less in ipsilateral than in contralateral hemispace: in ipsilateral hemispace, the bias of the perceived hand position was reduced, which was accompanied by a smaller variability of the estimates. The variability of position estimates of the dominant right hand was also less than for the non-dominant left hand, but this difference was not accompanied by a difference in the asymmetry of sensory coupling – a violation of the reliability rule, probably due a stronger influence of visual information on right-hand movements. According to these results, the long-term effects of the experienced kinematic correlation between hand movements and cursor motions on the strength of sensory coupling are generic and not specific for hemispaces or hands, whereas the effects of relative reliabilities on the asymmetry of sensory coupling are specific for hemispaces but not for hands.

Sensory integration of movements and their visual effects is not enhanced by spatial proximity

Spatial proximity enhances the sensory integration of exafferent position information, likely because it indicates whether the information comes from a single physical source. Does spatial proximity also affect the integration of position information regarding an action (here a hand movement) with that of its visual effect (here a cursor motion), that is, when the sensory information comes from physically distinct objects? In this study, participants made out-and-back hand movements whereby the outward movements were accompanied by corresponding cursor motions on a monitor. Their subsequent judgments of hand or cursor movement endpoints are typically biased toward each other, consistent with an underlying optimal integration mechanism. To study the effect of spatial proximity, we presented the hand and cursor either in orthogonal planes (horizontal and frontal, respectively) or we aligned them in the horizontal plane. We did not find the expected enhanced integration strength in the latter spatial condition. As a secondary question we asked whether spatial transformations required for the position judgments (i.e., horizontal to frontal or vice versa) could be the origin of previously observed suboptimal variances of the integrated hand and cursor position judgments. We found, however, that the suboptimality persisted when spatial transformations were omitted (i.e., with the hand and cursor in the same plane). Our findings thus clearly show that the integration of actions with their visual effects is, at least for cursor control, independent of spatial proximity.

Kinematic cross-correlation induces sensory integration across separate objects

In a basic cursor-control task, the perceived positions of the hand and the cursor are biased towards each other. We recently found that this phenomenon conforms to the reliability-based weighting mechanism of optimal multisensory integration. This indicates that optimal integration is not restricted to sensory signals originating from a single source, as is the prevailing view, but that it also applies to separate objects that are connected by a kinematic relation (i.e. hand and cursor). In the current study, we examined which aspects of the kinematic relation are crucial for eliciting the sensory integration: (i) the cross-correlation between kinematic variables of the hand and cursor trajectories, and/or (ii) an internal model of the hand-cursor kinematic transformation. Participants made out-and-back movements from the centre of a semicircular workspace to its boundary, after which they judged the position where either their hand or the cursor hit the boundary. We analysed the position biases and found that the integration was strong in a condition with high kinematic correlations (a straight hand trajectory was mapped to a straight cursor trajectory), that it was significantly reduced for reduced kinematic correlations (a straight hand trajectory was transformed into a curved cursor trajectory) and that it was not affected by the inability to acquire an internal model of the kinematic transformation (i.e. by the trial-to-trial variability of the cursor curvature). These findings support the idea that correlations play a crucial role in multisensory integration irrespective of the number of sensory sources involved.

Dissociating explicit and implicit measures of sensed hand position in tool use: Effect of relative frequency of judging different objects

In a cursor-control task, the sensed positions of cursor and hand are biased toward each other. We previously found different characteristics of implicit and explicit measures of the bias of sensed hand position toward the position of the cursor, suggesting the existence of distinct neural representations. Here we further explored differences between the two types of measure by varying the proportions of trials with explicit hand-position (H) and cursor-position (C) judgments (C20:H80, C50:H50, and C80:H20). In each trial, participants made a reaching movement to a remembered target, with the visual feedback being rotated randomly, and subsequently they judged the hand or the cursor position. Both the explicitly and implicitly measured biases of sensed hand position were stronger with a low proportion (C80:H20) than with a high proportion (C20:H80) of hand-position judgments, suggesting that both measures place more weight on the sensory modality relevant for the more frequent judgment. With balanced proportions of such judgments (C50:H50), the explicitly assessed biases were similar to those observed with a high proportion of cursor-position judgments (C80:H20), whereas the implicitly assessed biases were similar to those observed with a high proportion of hand-position judgments (C20:H80). Because strong weights of cursor-position or hand-position information may be difficult to increase further but are easy to reduce, the findings suggest that the implicit measure of the bias of sensed hand position places a relatively stronger weight on proprioceptive hand-position information, which is increased no further by a high proportion of hand-position judgments. Conversely, the explicit measure places a relatively stronger weight on visual cursor-position information.

Contrasting effects of adaptation to a visuomotor rotation on explicit and implicit measures of sensory coupling

We previously investigated sensory coupling of the sensed positions of cursor and hand in a cursor-control task and found differential characteristics of implicit and explicit measures of the bias of sensed hand position toward the position of the cursor. The present study further tested whether adaptation to a visuomotor rotation differentially affects these two measures. Participants made center-out reaching movements to remembered targets while looking at a rotated feedback cursor. After sets of practice trials with constant (adaptation condition) or random (control condition) visuomotor rotations, test trials served to assess sensory coupling. In these trials, participants judged the position of the hand at the end of the center-out movement, and the deviation of these judgments from the physical hand positions served as explicit measure of the bias of sensed hand position toward the position of the cursor, whereas the implicit measure was based on the direction of the return movement. The results showed that inter-individual variability of explicitly assessed biases of sensed hand position toward the cursor position was less in the adaptation condition than in the control condition. Conversely, no such changes were observed for the implicit measure of the bias of sensed hand position, revealing contrasting effects of adaptation on the explicit and implicit measures. These results suggest that biases of explicitly sensed hand position reflect sensory coupling of neural representations that are altered by visuomotor adaptation. In contrast, biases of implicitly sensed hand position reflect sensory coupling of neural representations that are unaffected by adaptation.

Perceptual attraction in tool use: evidence for a reliability-based weighting mechanism

Humans are well able to operate tools whereby their hand movement is linked, via a kinematic transformation, to a spatially distant object moving in a separate plane of motion. An everyday example is controlling a cursor on a computer monitor. Despite these separate reference frames, the perceived positions of the hand and the object were found to be biased toward each other. We propose that this perceptual attraction is based on the principles by which the brain integrates redundant sensory information of single objects or events, known as optimal multisensory integration. That is, 1) sensory information about the hand and the tool are weighted according to their relative reliability (i.e., inverse variances), and 2) the unisensory reliabilities sum up in the integrated estimate. We assessed whether perceptual attraction is consistent with optimal multisensory integration model predictions. We used a cursor-control tool-use task in which we manipulated the relative reliability of the unisensory hand and cursor position estimates. The perceptual biases shifted according to these relative reliabilities, with an additional bias due to contextual factors that were present in experiment 1 but not in experiment 2 The biased position judgments' variances were, however, systematically larger than the predicted optimal variances. Our findings suggest that the perceptual attraction in tool use results from a reliability-based weighting mechanism similar to optimal multisensory integration, but that certain boundary conditions for optimality might not be satisfied.NEW & NOTEWORTHY Kinematic tool use is associated with a perceptual attraction between the spatially separated hand and the effective part of the tool. We provide a formal account for this phenomenon, thereby showing that the process behind it is similar to optimal integration of sensory information relating to single objects.

Visual target distance, but not visual cursor path length produces shifts in motor behavior

When using tools effects in body space and distant space often do not correspond. Findings so far demonstrated that in this case visual feedback has more impact on action control than proprioceptive feedback. The present study varies the dimensional overlap between visual and proprioceptive action effects and investigates its impact on aftereffects in motor responses. In two experiments participants perform linear hand movements on a covered digitizer tablet to produce ∩-shaped cursor trajectories on the display. The shape of hand motion and cursor motion (linear vs. curved) is dissimilar and therefore does not overlap. In one condition the length of hand amplitude and visual target distance is similar and constant while the length of the cursor path is dissimilar and varies. In another condition the length of the hand amplitude varies while the lengths of visual target distance (similar or dissimilar) and cursor path (dissimilar) are constant. First, we found that aftereffects depended on the relation between hand path length and visual target distance, and not on the relation between hand and cursor path length. Second, increasing contextual interference did not reveal larger aftereffects. Finally, data exploration demonstrated a considerable benefit from gain repetitions across trials when compared to gain switches. In conclusion, dimensional overlap between visual and proprioceptive action effects modulates human information processing in visually controlled actions. However, adjustment of the internal model seems to occur very fast for this kind of simple linear transformation, so that the impact of prior visual feedback is fleeting.

Intra- and intermodal integration of discrepant visual and proprioceptive action effects

Integration of discrepant visual and proprioceptive action effects puts high demands on the human information processing system. The present study aimed to examine the integration mechanisms for the motor (Exp. 1) and visual modality (Exp. 2). According to theories of common coding, we assumed that visual as well as proprioceptive information is represented within the same cognitive domain and is therefore likely to affect each other (multisensory cross talk). Thus, apart from the often-confirmed visual dominance in multisensory integration, we asked about intra- and intermodal recall of either proprioceptive or visual information and whether there were any differences between the motor and visual modality. In a replication paradigm, we perturbed the relation between hand movements and cursor movements. The task required the (intra- vs. intermodal) replication of an initially performed (seen) hand (cursor) movement in a subsequent motor (visual) replication phase. First, mechanisms of integration were found to be dependent on the output modality. Visual action effects interfered the motor modality, but proprioceptive action effects did not have any effects on the visual modality. Second, however, intermodal integration was more susceptible to interference, and this was found to be independent from the output modality. Third, for the motor modality, the locus of perturbation (perturbation of cursor amplitude or perturbation of hand amplitude) was irrelevant, but for the visual modality, perturbation of hand amplitudes reduced the cross talk. Tool use is one field of application of these kinds of results, since the optimized integration of conflicting action effects is a precondition for using tools successfully.

Concurrent adaptation to opposite visual distortions: impairment and cue

The present study compared single and dual adaptation to visuomotor rotations in different cueing conditions. Participants adapted either to a constant rotation or to opposing rotations (dual adaptation) applied in an alternating order. In Experiment 1, visual and corresponding postural cues were provided to indicate different rotation directions. In Experiment 2, either a visual or a postural cue was available. In all cueing conditions, substantial dual adaptation was observed, although it was attenuated in comparison to single adaptation. Analysis of switching costs determined as the performance difference between the last trial before and the first trial after the change of rotation direction suggested substantial advantage of the visual cue compared to the postural cue, which was in line with previous findings demonstrating the dominance of visual sense in movement representation and control.