Dissociating explicit and implicit measures of sensed hand position in tool use: Effect of relative frequency of judging different objects

Different time scales of common-cause evidence shape multisensory integration, recalibration and motor adaptation

Perceptual coherence in the face of discrepant multisensory signals is achieved via the processes of multisensory integration, recalibration and sometimes motor adaptation. These supposedly operate on different time scales, with integration reducing immediate sensory discrepancies and recalibration and motor adaptation reflecting the cumulative influence of their recent history. Importantly, whether discrepant signals are bound during perception is guided by the brains' inference of whether they originate from a common cause. When combined, these two notions lead to the hypothesis that the time scales on which integration and recalibration (or motor adaptation) operate are associated with different time scales of evidence about a common cause underlying two signals. We tested this prediction in a well-established visuo-motor paradigm, in which human participants performed visually guided hand movements. The kinematic correlation between hand and cursor movements indicates their common origin, which allowed us to manipulate the common-cause evidence by titrating this correlation. Specifically, we dissociated hand and cursor signals during individual movements while preserving their correlation across the series of movement endpoints. Following our hypothesis, this manipulation reduced integration compared with a condition in which visual and proprioceptive signals were perfectly correlated. In contrast, recalibration and motor adaption were not affected by this manipulation. This supports the notion that multisensory integration and recalibration deal with sensory discrepancies on different time scales guided by common-cause evidence: Integration is prompted by local common-cause evidence and reduces immediate discrepancies, whereas recalibration and motor adaptation are prompted by global common-cause evidence and reduce persistent discrepancies.

Short-term effects of visuomotor discrepancies on multisensory integration, proprioceptive recalibration, and motor adaptation

Information about the position of our hand is provided by multisensory signals that are often not perfectly aligned. Discrepancies between the seen and felt hand position or its movement trajectory engage the processes of 1) multisensory integration, 2) sensory recalibration, and 3) motor adaptation, which adjust perception and behavioral responses to apparently discrepant signals. To foster our understanding of the coemergence of these three processes, we probed their short-term dependence on multisensory discrepancies in a visuomotor task that has served as a model for multisensory perception and motor control previously. We found that the well-established integration of discrepant visual and proprioceptive signals is tied to the immediate discrepancy and independent of the outcome of the integration of discrepant signals in immediately preceding trials. However, the strength of integration was context dependent, being stronger in an experiment featuring stimuli that covered a smaller range of visuomotor discrepancies (±15°) compared with one covering a larger range (±30°). Both sensory recalibration and motor adaptation for nonrepeated movement directions were absent after two bimodal trials with same or opposite visuomotor discrepancies. Hence our results suggest that short-term sensory recalibration and motor adaptation are not an obligatory consequence of the integration of preceding discrepant multisensory signals.NEW & NOTEWORTHY The functional relation between multisensory integration and recalibration remains debated. We here refute the notion that they coemerge in an obligatory manner and support the hypothesis that they serve distinct goals of perception.

Exploring the time window for causal inference and the multisensory integration of actions and their visual effects

Successful computer use requires the operator to link the movement of the cursor to that of his or her hand. Previous studies suggest that the brain establishes this perceptual link through multisensory integration, whereby the causality evidence that drives the integration is provided by the correlated hand and cursor movement trajectories. Here, we explored the temporal window during which this causality evidence is effective. We used a basic cursor-control task, in which participants performed out-and-back reaching movements with their hand on a digitizer tablet. A corresponding cursor movement could be shown on a monitor, yet slightly rotated by an angle that varied from trial to trial. Upon completion of the backward movement, participants judged the endpoint of the outward hand or cursor movement. The mutually biased judgements that typically result reflect the integration of the proprioceptive information on hand endpoint with the visual information on cursor endpoint. We here manipulated the time period during which the cursor was visible, thereby selectively providing causality evidence either before or after sensory information regarding the to-be-judged movement endpoint was available. Specifically, the cursor was visible either during the outward or backward hand movement (conditions Out and Back, respectively). Our data revealed reduced integration in the condition Back compared with the condition Out, suggesting that causality evidence available before the to-be-judged movement endpoint is more powerful than later evidence in determining how strongly the brain integrates the endpoint information. This finding further suggests that sensory integration is not delayed until a judgement is requested.

A condition that produces sensory recalibration and abolishes multisensory integration

We examined the influence of extended exposure to a visuomotor rotation, which induces both motor adaptation and sensory recalibration, on (partial) multisensory integration in a cursor-control task. Participants adapted to a 30° (adaptation condition) or 0° (control condition) visuomotor rotation by making center-out movements to remembered targets. In subsequent test trials of sensory integration, they made center-out movements with variable visuomotor rotations and judged the position of hand or cursor at the end of these movements. Test trials were randomly embedded among twice the number of maintenance trials with 30° or 0° rotation. The biases of perceived hand (or cursor) position toward the cursor (or hand) position were measured. We found motor adaptation together with proprioceptive and visual recalibrations in the adaptation condition. Unexpectedly, multisensory integration was absent in both the adaptation and control condition. The absence stemmed from the extensive experience of constant visuomotor rotations of 30° or 0°, which probably produced highly precise predictions of the visual consequences of hand movements. The frequently confirmed predictions then dominated the estimate of the visual movement consequences, leaving no influence of the actual visuomotor rotations in the minority of test trials. Conversely, multisensory integration was present for sensed hand positions when these were indirectly assessed from movement characteristics, indicating that the relative weighting of discrepant estimates of hand position was different for motor control. The existence of a condition that abolishes multisensory integration while keeping sensory recalibration suggests that mechanisms that reduce sensory discrepancies (partly) differ between integration and recalibration.

Visual and proprioceptive recalibrations after exposure to a visuomotor rotation

Adaptation to a visuomotor rotation in a cursor-control task is accompanied by proprioceptive recalibration, whereas the existence of visual recalibration is uncertain and has even been doubted. In the present study, we tested both visual and proprioceptive recalibration; proprioceptive recalibration was not only assessed by means of psychophysical judgments of the perceived position of the hand, but also by an indirect procedure based on movement characteristics. Participants adapted to a gradually introduced visuomotor rotation of 30° by making center-out movements to remembered targets. In subsequent test trials, they made center-out movements without visual feedback or observed center-out motions of a cursor without moving the hand. In each test trial, they judged the endpoint of hand or cursor by matching the position of the hand or of a visual marker, respectively, moving along a semicircular path. This path ran through all possible endpoints of the center-out movements. We observed proprioceptive recalibration of 7.3° (3.1° with the indirect procedure) and a smaller, but significant, visual recalibration of 1.3°. Total recalibration of 8.6° was about half as strong as motor adaptation, the adaptive shift of the movement direction. The evidence of both proprioceptive and visual recalibration was obtained with a judgment procedure that suggests that recalibration is restricted to the type of movement performed during exposure to a visuomotor rotation. Consequently, identical physical positions of the hand can be perceived differently depending on how they have been reached, and similarly identical positions of a cursor on a monitor can be perceived differently.

Effects of Hand and Hemispace on Multisensory Integration of Hand Position and Visual Feedback

The brain generally integrates a multitude of sensory signals to form a unified percept. Even in cursor control tasks, such as reaching while looking at rotated visual feedback on a monitor, visual information on cursor position and proprioceptive information on hand position are partially integrated (sensory coupling), resulting in mutual biases of the perceived positions of cursor and hand. Previous studies showed that the strength of sensory coupling (sum of the mutual biases) depends on the experience of kinematic correlations between hand movements and cursor motions, whereas the asymmetry of sensory coupling (difference between the biases) depends on the relative reliabilities (inverse of variability) of hand-position and cursor-position estimates (reliability rule). Furthermore, the precision of movement control and perception of hand position are known to differ between hands (left, right) and workspaces (ipsilateral, contralateral), and so does the experience of kinematic correlations from daily life activities. Thus, in the present study, we tested whether strength and asymmetry of sensory coupling for the endpoints of reaches in a cursor control task differ between the right and left hand and between ipsilateral and contralateral hemispace. No differences were found in the strength of sensory coupling between hands or between hemispaces. However, asymmetry of sensory coupling was less in ipsilateral than in contralateral hemispace: in ipsilateral hemispace, the bias of the perceived hand position was reduced, which was accompanied by a smaller variability of the estimates. The variability of position estimates of the dominant right hand was also less than for the non-dominant left hand, but this difference was not accompanied by a difference in the asymmetry of sensory coupling – a violation of the reliability rule, probably due a stronger influence of visual information on right-hand movements. According to these results, the long-term effects of the experienced kinematic correlation between hand movements and cursor motions on the strength of sensory coupling are generic and not specific for hemispaces or hands, whereas the effects of relative reliabilities on the asymmetry of sensory coupling are specific for hemispaces but not for hands.

Explicit knowledge of sensory non-redundancy can reduce the strength of multisensory integration

The brain integrates incoming sensory signals to a degree that depends on the signals' redundancy. Redundancy-which is commonly high when signals originate from a common physical object or event-is estimated by the brain from the signals' spatial and/or temporal correspondence. Here we tested whether verbally instructed knowledge of non-redundancy can also be used to reduce the strength of the sensory integration. We used a cursor-control task in which cursor motions in the frontoparallel plane were controlled by hand movements in the horizontal plane, yet with a small and randomly varying visuomotor rotation that created spatial discrepancies between hand and cursor positions. Consistent with previous studies, we found mutual biases in the hand and cursor position judgments, indicating partial sensory integration. The integration was reduced in strength, but not eliminated, after participants were verbally informed about the non-redundancy (i.e., the spatial discrepancies) in the hand and cursor positions. Comparisons with model predictions excluded confounding bottom-up effects of the non-redundancy instruction. Our findings thus show that participants have top-down control over the degree to which they integrate sensory information. Additionally, we found that the magnitude of this top-down modulatory capability is a reliable individual trait. A comparison between participants with and without video-gaming experience tentatively suggested a relation between top-down modulation of integration strength and attentional control.

Optimal integration of actions and their visual effects is based on both online and prior causality evidence

The brain needs to identify redundant sensory signals in order to integrate them optimally. The identification process, referred to as causal inference, depends on the spatial and temporal correspondence of the incoming sensory signals ('online sensory causality evidence') as well as on prior expectations regarding their causal relation. We here examine whether the same causal inference process underlies spatial integration of actions and their visual consequences. We used a basic cursor-control task for which online sensory causality evidence is provided by the correlated hand and cursor movements, and prior expectations are formed by everyday experience of such correlated movements. Participants made out-and-back movements and subsequently judged the hand or cursor movement endpoints. In one condition, we omitted the online sensory causality evidence by showing the cursor only at the movement endpoint. The integration strength was lower than in conditions where the cursor was visible during the outward movement, but a substantial level of integration persisted. These findings support the hypothesis that the binding of actions and their visual consequences is based on the general mechanism of optimal integration, and they specifically show that such binding can occur even if it is previous experience only that identifies the action consequence.

Contrasting effects of adaptation to a visuomotor rotation on explicit and implicit measures of sensory coupling

We previously investigated sensory coupling of the sensed positions of cursor and hand in a cursor-control task and found differential characteristics of implicit and explicit measures of the bias of sensed hand position toward the position of the cursor. The present study further tested whether adaptation to a visuomotor rotation differentially affects these two measures. Participants made center-out reaching movements to remembered targets while looking at a rotated feedback cursor. After sets of practice trials with constant (adaptation condition) or random (control condition) visuomotor rotations, test trials served to assess sensory coupling. In these trials, participants judged the position of the hand at the end of the center-out movement, and the deviation of these judgments from the physical hand positions served as explicit measure of the bias of sensed hand position toward the position of the cursor, whereas the implicit measure was based on the direction of the return movement. The results showed that inter-individual variability of explicitly assessed biases of sensed hand position toward the cursor position was less in the adaptation condition than in the control condition. Conversely, no such changes were observed for the implicit measure of the bias of sensed hand position, revealing contrasting effects of adaptation on the explicit and implicit measures. These results suggest that biases of explicitly sensed hand position reflect sensory coupling of neural representations that are altered by visuomotor adaptation. In contrast, biases of implicitly sensed hand position reflect sensory coupling of neural representations that are unaffected by adaptation.