Keeping the world at hand: rapid visuomotor processing for hand–object interactions

Peripersonal space: why so last-second?

A vast range of neurophysiological, neuropsychological and behavioural results in monkeys and humans have shown that the immediate surroundings of the body, also known as peripersonal space (PPS), are processed in a unique way. Three roles have been ascribed to PPS mechanisms: to react to threats, to avoid obstacles and to act on objects. However, in many circumstances, one does not wait for objects or agents to enter PPS to plan these behaviours. Typically, one has more chances to survive if one starts running away from the lion when one sees it in the distance than if it is a few steps away. PPS makes sense in shortsighted creatures but we are not such creatures. The crucial question is thus twofold: (i) why are these adaptive processes triggered only at the last second or even milliseconds? And (ii) what is their exact contribution, especially for defensive and navigational behaviours? Here, we propose that PPS mechanisms correspond to a plan B, useful in unpredictable situations or when other anticipatory mechanisms have failed. Furthermore, we argue that there are energetic, cognitive and behavioural costs to PPS mechanisms, which explain why this plan B is triggered only at the last second. This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Always expect the unexpected: eye position modulates visual cortex excitability in a stimulus-free environment

Stimuli that potentially require a rapid defensive or avoidance action can appear from the periphery at any time in natural environments. de Wit et al. (Cortex 127: 120-130, 2020) recently reported novel evidence suggestive of a fundamental neural mechanism that allows organisms to effectively deal with such situations. In the absence of any task, motor cortex excitability was found to be greater whenever gaze was directed away from either hand. If modulation of cortical excitability as a function of gaze location is a fundamental principle of brain organization, then one would expect its operation to be present outside of motor cortex, including brain regions involved in perception. To test this hypothesis, we applied single-pulse transcranial magnetic stimulation (TMS) to the right lateral occipital lobe while participants directed their eyes to the left, straight-ahead, or to the right, and reported the presence or absence of a phosphene. No external stimuli were presented. Cortical excitability as reflected by the proportion of trials on which phosphenes were elicited from stimulation of the right visual cortex was greater with eyes deviated to the right as compared with the left. In conjunction with our previous findings of change in motor cortex excitability when gaze and effector are not aligned, this eye position-driven change in visual cortex excitability presumably serves to facilitate the detection of stimuli and subsequent readiness to act in nonfoveated regions of space. The existence of this brain-wide mechanism has clear adaptive value given the unpredictable nature of natural environments in which human beings are situated and have evolved.NEW & NOTEWORTHY For many complex tasks, humans focus attention on the site relevant to the task at hand. Humans evolved and live in dangerous environments, however, in which threats arise from outside the attended site; this fact necessitates a process by which the periphery is monitored. Using single-pulse transcranial magnetic stimulation (TMS), we demonstrated for the first time that eye position modulates visual cortex excitability. We argue that this underlies at least in part what we term "surveillance attention."

Body and peripersonal space representations in chronic stroke patients with upper limb motor deficits

The continuous stream of multisensory information between the brain and the body during body–environment interactions is crucial to maintain the updated representation of the perceived dimensions of body parts (metric body representation) and the space around the body (the peripersonal space). Such flow of multisensory signals is often limited by upper limb sensorimotor deficits after stroke. This would suggest the presence of systematic distortions of metric body representation and peripersonal space in chronic patients with persistent sensorimotor deficits. We assessed metric body representation and peripersonal space representation in 60 chronic stroke patients with unilateral upper limb motor deficits, in comparison with age-matched healthy controls. We also administered a questionnaire capturing explicit feelings towards the affected limb. These novel measures were analysed with respect to patients’ clinical profiles and brain lesions to investigate the neural and functional origin of putative deficits. Stroke patients showed distortions in metric body representation of the affected limb, characterized by an underestimation of the arm length and an alteration of the arm global shape. A descriptive lesion analysis (subtraction analysis) suggests that these distortions may be more frequently associated with lesions involving the superior corona radiata and the superior frontal gyrus. Peripersonal space representation was also altered, with reduced multisensory facilitation for stimuli presented around the affected limb. These deficits were more common in patients reporting pain during motion. Explorative lesion analyses (subtraction analysis, disconnection maps) suggest that the peripersonal space distortions would be more frequently associated with lesions involving the parietal operculum and white matter frontoparietal connections. Moreover, patients reported altered feelings towards the affected limb, which were associated with right brain damage, proprioceptive deficits and a lower cognitive profile. These results reveal implicit and explicit distortions involving metric body representation, peripersonal space representation and the perception of the affected limb in chronic stroke patients. These findings might have important clinical implications for the longitudinal monitoring and the treatments of often-neglected deficits in body perception and representation.

Precision control for a flexible body representation

Adaptive body representation requires the continuous integration of multisensory inputs within a flexible 'body model' in the brain. The present review evaluates the idea that this flexibility is augmented by the contextual modulation of sensory processing 'top-down'; which can be described as precision control within predictive coding formulations of Bayesian inference. Specifically, I focus on the proposal that an attenuation of proprioception may facilitate the integration of conflicting visual and proprioceptive bodily cues. Firstly, I review empirical work suggesting that the processing of visual vs proprioceptive body position information can be contextualised 'top-down'; for instance, by adopting specific attentional task sets. Building up on this, I review research showing a similar contextualisation of visual vs proprioceptive information processing in the rubber hand illusion and in visuomotor adaptation. Together, the reviewed literature suggests that proprioception, despite its indisputable importance for body perception and action control, can be attenuated top-down (through precision control) to facilitate the contextual adaptation of the brain's body model to novel visual feedback.

Associative learning in peripersonal space: fear responses are acquired in hand-centered coordinates

Space coding affects perception of stimuli associated to negative valence: threatening stimuli presented within the peripersonal space (PPS) speed up behavioral responses compared with nonthreatening events. However, it remains unclear whether the association between stimuli and their negative valence is acquired in a body part-centered reference system, a main feature of the PPS coding. Here we test the hypothesis that associative learning takes place in hand-centered coordinates and can therefore remap according to hand displacement. In two experiments, we used a Pavlovian fear-learning paradigm to associate a visual stimulus [light circle, the conditioned stimulus (CS)] with an aversive stimulus (electrocutaneous shock) applied on the right hand only when the CS was displayed close (CS+) but when not far from it (CS-). Measuring the skin conductance response (SCR), we observed successful fear conditioning, with increased anticipatory fear responses associated with CS+. Crucially, experiment I showed a remapping of these responses following hand displacement, with a generalization to both types of CS. Experiment II corroborated and further extended our findings by ruling out the novelty of the experimental context as a driving factor of such modulations. Indeed, fear responses were present only for stimuli within the PPS but not for new stimuli displayed outside the PPS. By revealing a hand-centered (re)mapping of the conditioning effect, these findings indicate that associative learning can arise in hand-centered coordinates. They further suggest that the threatening valence of an object also depends on its basic spatial relationship with our body.NEW & NOTEWORTHY Associative fear learning takes place in hand-centered coordinates. Using a Pavlovian fear-learning paradigm, we show that the anticipatory skin conductance response indicating the association between the negative value and an initially neutral stimulus is acquired and then remapped in space when the stimulated body part moves to a different position. These results demonstrate the relationship between the representation of peripersonal space and the encoding of threatening stimuli. Hypotheses concerning the underlying neural network are discussed.

Allocation of attention in 3D space is adaptively modulated by relative position of target and distractor stimuli

Allocation of attention across different depth planes is a prerequisite for visual selection in a three-dimensional environment. Previous research showed that participants successfully used stereoscopic depth information to focus their attention. This, however, does not mean that salient information from other depth planes is completely neglected. The present study investigated the question of whether competing visual information is differentially processed when displayed in a single depth plane or across two different depth planes. Moreover, it was of interest whether potential effects were further modulated by the items' relative spatial position (near or far). In three experiments participants performed a variant of the additional singleton paradigm. Targets were defined by stereoscopic depth information and as such appeared either in a near or far depth plane. Distractor stimuli were displayed in the same or in the opposed depth plane. The results consistently showed that visual selection was slower when target and distractor coincided within the same depth plane. There was no general advantage for targets presented in near or far depth planes. However, differential effects of target depth plane and the target-distractor relation were observed. Selection of near targets was more affected by distractors within the same depth plane while far targets were identified more slowly when the amount of information in closer depth planes increased. While attentional resources could not be exclusively centered to a distinct depth plane, the allocation of attention might be organized along an egocentric gradient through space and varies with the organization of the visual surrounding.

Cortical beta oscillations reflect the contextual gating of visual action feedback

•

We decouple seen and felt hand postures during action via virtual reality.

•

Vision of the hand is either task-relevant or a distractor.

•

Task-relevance of vision is reflected by in- or decreases of occipital beta power.

•

DCM suggests underlying changes in cortical (visual) excitability.

•

Occipital beta may indicate the contextual gating of visual action feedback.

In sensorimotor integration, the brain needs to decide how its predictions should accommodate novel evidence by ‘gating’ sensory data depending on the current context. Here, we examined the oscillatory correlates of this process by recording magnetoencephalography (MEG) data during a new task requiring action under intersensory conflict. We used virtual reality to decouple visual (virtual) and proprioceptive (real) hand postures during a task in which the phase of grasping movements tracked a target (in either modality). Thus, we rendered visual information either task-relevant or a (to-be-ignored) distractor. Under visuo-proprioceptive incongruence, occipital beta power decreased (relative to congruence) when vision was task-relevant but increased when it had to be ignored. Dynamic causal modeling (DCM) revealed that this interaction was best explained by diametrical, task-dependent changes in visual gain. These results suggest a crucial role for beta oscillations in the contextual gating (i.e., gain or precision control) of visual vs proprioceptive action feedback, depending on current behavioral demands.

Ever-ready for action: Spatial effects on motor system excitability

Modulation of excitability in the motor system can be observed before overt movements but also in response to covert invitations to act. We asked whether such changes can be induced in the absence of even covert motor instructions, namely, as a function of the location of the hand with reference to the body. Participants received single-pulse TMS over the motor cortex while they placed their contralateral hand (right hand in Experiment 1, left hand in Experiment 2) to the right or left of their body midline, and looked either at or away from their hand. In both experiments, greater excitability was observed when gaze was directed to the right. This finding is interpreted as a consequence of left brain lateralization of motor attention. Contrary to our expectations, we furthermore consistently observed greater excitability when gaze was directed away from the hand. To account for this finding, we introduce the concept of "surveillance attention" which, we speculate, modulates cortical gain, and thereby cortical excitability. Its function is to increase readiness to act in non-foveated regions of space. Such a process confers an advantage in environments, like those in which humans evolved, in which threatening stimuli may appear unexpectedly, and at any time.

Faster recognition of graspable targets defined by orientation in a visual search task

Peri-hand space is the area surrounding the hand. Objects within this space may be subject to increased visuospatial perception, increased attentional prioritization, and slower attentional disengagement compared to more distal objects. This may result from kinesthetic and visual feedback about the location of the hand that projects from the reach and grasp networks of the dorsal visual stream back to occipital visual areas, which in turn, refines cortical visual processing that can subsequently guide skilled motor actions. Thus, we hypothesized that visual stimuli that afford action, which are known to potentiate activity in the dorsal visual stream, would be associated with greater alterations in visual processing when presented near the hand. To test this, participants held their right hand near or far from a touchscreen that presented a visual array containing a single target object that differed from 11 distractor objects by orientation only. The target objects and their accompanying distractors either strongly afforded grasping or did not. Participants identified the target among the distractors by reaching out and touching it with their left index finger while eye-tracking was used to measure visual search times, target recognition times, and search accuracy. The results failed to support the theory of enhanced visual processing of graspable objects near the hand as participants were faster at recognizing graspable compared to non-graspable targets, regardless of the position of the right hand. The results are discussed in relation to the idea that, in addition to potentiating appropriate motor responses, object affordances may also potentiate early visual processes necessary for object recognition.

Hand blink reflex in virtual reality: The role of vision and proprioception in modulating defensive responses

Our research focused on the role of vision and proprioception in modulating a defensive reflex (hand blink reflex, HBR) whose magnitude is enhanced when the threatened hand is inside the peripersonal space of the face. We capitalized on virtual reality, which allows dissociating vision and proprioception by presenting a virtual limb in congruent/incongruent positions with respect to the participants' limb. In experiment 1, participants placed their own stimulated hand in far/near positions with respect to their face (postural manipulation task), while observing a virtual empty scenario. Vision was not informative, but the HBR was significantly enhanced in near compared with far position, suggesting that proprioception is sufficient for the HBR modulation to occur. In experiment 2, participants did not perform the postural manipulation but they (passively) observed the avatar's virtual limb performing it. Proprioceptive signals were not informative, but the HBR was significantly enhanced when the observed virtual limb was near to the face, suggesting that visual information plays a role in modulating the HBR. In experiment 3, both participants and avatar performed the postural manipulation, either congruently (both of them far/near) or incongruently (one of them far, the other near). The HBR modulation was present only in congruent conditions. In incongruent conditions, the conflict between vision and proprioception confounded the system, abolishing the difference between far and near positions. Taken together, these findings promote the view that observing a virtual limb modulates the HBR, providing also new evidence on the role of vision and proprioception in modulating this subcortical reflex.

This is for you: Social modulations of proximal vs. distal space in collaborative interaction

Human spatial representations are shaped by affordances for action offered by the environment. A prototypical example is the organization of space into peripersonal (within reach) and extrapersonal (outside reach) regions, mirrored by proximal (this/here) and distal (that/there) linguistic expressions. The peri-/extrapersonal distinction has been widely investigated in individual contexts, but little is known about how spatial representations are modulated by interaction with other people. Is near/far coding of space dynamically adapted to the position of a partner when space, objects, and action goals are shared? Over two preregistered experiments based on a novel interactive paradigm, we show that, in individual and social contexts involving no direct collaboration, linguistic coding of locations as proximal or distal depends on their distance from the speaker's hand. In contrast, in the context of collaborative interactions involving turn-taking and role reversal, proximal space is shifted towards the partner, and linguistic coding of near space ('this' / 'here') is remapped onto the partner's action space.

Action Planning Modulates Peripersonal Space

Peripersonal space is a multisensory representation relying on the processing of tactile and visual stimuli presented on and close to different body parts. The most studied peripersonal space representation is perihand space (PHS), a highly plastic representation modulated following tool use and by the rapid approach of visual objects. Given these properties, PHS may serve different sensorimotor functions, including guidance of voluntary actions such as object grasping. Strong support for this hypothesis would derive from evidence that PHS plastic changes occur before the upcoming movement rather than after its initiation, yet to date, such evidence is scant. Here, we tested whether action-dependent modulation of PHS, behaviorally assessed via visuotactile perception, may occur before an overt movement as early as the action planning phase. To do so, we probed tactile and visuotactile perception at different time points before and during the grasping action. Results showed that visuotactile perception was more strongly affected during the planning phase (250 msec after vision of the target) than during a similarly static but earlier phase (50 msec after vision of the target). Visuotactile interaction was also enhanced at the onset of hand movement, and it further increased during subsequent phases of hand movement. Such a visuotactile interaction featured interference effects during all phases from action planning onward as well as a facilitation effect at the movement onset. These findings reveal that planning to grab an object strengthens the multisensory interaction of visual information from the target and somatosensory information from the hand. Such early updating of the visuotactile interaction reflects multisensory processes supporting motor planning of actions.

Aim and Plausibility of Action Chains Remap Peripersonal Space

Successful interaction with objects in the peripersonal space requires that the information relative to current and upcoming positions of our body is continuously monitored and updated with respect to the location of target objects. Voluntary actions, for example, are known to induce an anticipatory remapping of the peri-hand space (PHS, i.e., the space near the acting hand) during the very early stages of the action chain: planning and initiating an object grasp increase the interference exerted by visual stimuli coming from the object on touches delivered to the grasping hand, thus allowing for hand-object position monitoring and guidance. Voluntarily grasping an object, though, is rarely performed in isolation. Grasping a candy, for example, is most typically followed by concatenated secondary action steps (bringing the candy to the mouth and swallowing it) that represent the agent’s ultimate intention (to eat the candy). However, whether and when complex action chains remap the PHS remains unknown, just as whether remapping is conditional to goal achievability (e.g., candy-mouth fit). Here we asked these questions by assessing changes in visuo-tactile interference on the acting hand while participants had to grasp an object serving as a support for an elongated candy, and bring it toward their mouth. Depending on its orientation, the candy could potentially enter the participants’ mouth (plausible goal), or not (implausible goal). We observed increased visuo-tactile interference at relatively late stages of the action chain, after the object had been grasped, and only when the action goal was plausible. These findings suggest that multisensory interactions during action execution depend upon the final aim and plausibility of complex goal-directed actions, and extend our knowledge about the role of peripersonal space in guiding goal-directed voluntary actions.

Movement of environmental threats modifies the relevance of the defensive eye-blink in a spatially-tuned manner

Subcortical reflexive motor responses are under continuous cortical control to produce the most effective behaviour. For example, the excitability of brainstem circuitry subserving the defensive hand-blink reflex (HBR), a response elicited by intense somatosensory stimuli to the wrist, depends on a number of properties of the eliciting stimulus. These include face-hand proximity, which has allowed the description of an HBR response field around the face (commonly referred to as a defensive peripersonal space, DPPS), as well as stimulus movement and probability of stimulus occurrence. However, the effect of stimulus-independent movements of objects in the environment has not been explored. Here we used virtual reality to test whether and how the HBR-derived DPPS is affected by the presence and movement of threatening objects in the environment. In two experiments conducted on 40 healthy volunteers, we observed that threatening arrows flying towards the participant result in DPPS expansion, an effect directionally-tuned towards the source of the arrows. These results indicate that the excitability of brainstem circuitry subserving the HBR is continuously adjusted, taking into account the movement of environmental objects. Such adjustments fit in a framework where the relevance of defensive actions is continually evaluated, to maximise their survival value.

When social and action spaces diverge: A study in children with typical development and autism

The space around the body has been defined as action space ( peripersonal space) and a social space ( interpersonal space). Within the current debate about the characteristics of these spaces, here we investigated the functional properties and plasticity of action and social space in developmental age. To these aims, children with typical development and autism spectrum disorders were submitted to Reaching- and Comfort-distance tasks, to assess peripersonal and interpersonal space, respectively. Participants approached a person (confederate) or an object and stopped when they thought they could reach the stimulus (Reaching-distance task), or they felt comfortable with stimulus’ proximity (Comfort-distance task). Both tasks were performed before and after a cooperative tool-use training, in which participant and confederate actively cooperated to reach tokens by using either a long (Experiment 1) or a short (Experiment 2) tool. Results showed that in both groups, peripersonal space extended following long-tool-use but not short-tool-use training. Conversely, in typical development, but not in autism spectrum disorders children, interpersonal space toward confederate reduced following the cooperative tool-use training. These findings reveal that action and social spaces are functionally dissociable both in typical and atypical development, and that action but not social space regulation is intact in children with autism.

An Action Field Theory of Peripersonal Space

Predominant conceptual frameworks often describe peripersonal space (PPS) as a single, distance-based, in-or-out zone within which stimuli elicit enhanced neural and behavioural responses. Here we argue that this intuitive framework is contradicted by neurophysiological and behavioural data. First, PPS-related measures are not binary, but graded with proximity. Second, they are strongly influenced by factors other than proximity, such as walking, tool use, stimulus valence, and social cues. Third, many different PPS-related responses exist, and each can be used to describe a different space. Here, we reconceptualise PPS as a set of graded fields describing behavioural relevance of actions aiming to create or avoid contact between objects and the body. This reconceptualisation incorporates PPS into mainstream theories of action selection and behaviour.

Mind the Depth: Visual Perception of Shapes Is Better in Peripersonal Space

Closer objects are invariably perceived as bigger than farther ones and are therefore easier to detect and discriminate. This is so deeply grounded in our daily experience that no question has been raised as to whether the advantage for near objects depends on other features (e.g., depth itself). In a series of five experiments (N = 114), we exploited immersive virtual environments and visual illusions (i.e., Ponzo) to probe humans’ perceptual abilities in depth and, specifically, in the space closely surrounding our body, termed peripersonal space. We reversed the natural distance scaling of size in favor of the farther object, which thus appeared bigger, to demonstrate a persistent shape-discrimination advantage for close objects. Psychophysical modeling further suggested a sigmoidal trend for this benefit, mirroring that found for multisensory estimates of peripersonal space. We argue that depth is a fundamental, yet overlooked, dimension of human perception and that future studies in vision and perception should be depth aware.

Magnifying the View of the Hand Changes Its Cortical Representation. A Transcranial Magnetic Stimulation Study

Changes in the perceived size of a body part using magnifying lenses influence tactile perception and pain. We investigated whether the visual magnification of one's hand also influences the motor system, as indexed by transcranial magnetic stimulation (TMS)-induced motor evoked potentials (MEPs). In Experiment 1, MEPs were measured while participants gazed at their hand with and without magnification of the hand. MEPs were significantly larger when participants gazed at a magnified image of their hand. In Experiment 2, we demonstrated that this effect is specific to the hand that is visually magnified. TMS of the left motor cortex did not induce an increase of MEPs when participants looked at their magnified left hand. Experiment 3 was performed to determine if magnification altered the topography of the cortical representation of the hand. To that end, a 3 × 5 grid centered on the cortical hot spot (cortical location at which a motor threshold is obtained with the lowest level of stimulation) was overlaid on the participant's MRI image, and all 15 sites in the grid were stimulated with and without magnification of the hand. We confirmed the increase in the MEPs at the hot spot with magnification and demonstrated that MEPs significantly increased with magnification at sites up to 16.5 mm from the cortical hot spot. In Experiment 4, we used paired-pulse TMS to measure short-interval intracortical inhibition and intracortical facilitation. Magnification was associated with an increase in short-interval intracortical inhibition. These experiments demonstrate that the visual magnification of one's hand induces changes in motor cortex excitability and generates a rapid remapping of the cortical representation of the hand that may, at least in part, be mediated by changes in short-interval intracortical inhibition.

Posterior parietal cortex evaluates visuoproprioceptive congruence based on brief visual information

To represent one’s upper limbs for action, the brain relies on a combined position estimate based on visual and proprioceptive information. Monkey neurophysiology and human brain imaging suggest that the underlying operations are implemented in a network of fronto-parietal and occipitotemporal cortical areas. Recently, a potential hierarchical arrangement of these areas has been proposed, emphasizing the posterior parietal cortex (PPC) in early multisensory comparison and integration. Here, we used functional magnetic resonance imaging (fMRI) and a virtual reality-based setup to briefly (0.5 s) present healthy human participants photorealistic virtual hands, of matching or nonmatching anatomical side, or objects at the same or a different location than their real hidden left or right hand. The inferior parietal lobe (IPL) of the left PPC showed a significant preference for congruent visuoproprioceptive hand position information. Moreover, the left body part-selective extrastriate body area (EBA; functionally localized) significantly increased its coupling with the left IPL during visuoproprioceptive congruence vs. incongruence. Our results suggest that the PPC implements early visuoproprioceptive comparison and integration processes, likely relying on information exchange with the EBA.

Grasp preparation modulates early visual processing of size and detection of local/global stimulus features

Preparing to grasp objects facilitates visual processing of object location, orientation and size, compared to preparing actions such as pointing. This influence of action on perception reflects mechanisms of selection in visual perception tuned to current action goals, such that action relevant sensory information is prioritized relative to less relevant information. In three experiments, rather than varying movement type (grasp vs point), the magnitude of a prepared movement (power vs precision grasps) was manipulated while visual processing of object size, as well as local/global target detection was measured. Early event-related potentials (ERP) elicited by task-irrelevant visual probes were enhanced for larger probes during power grasp preparation and smaller probes during precision grasp preparation. Local targets were detected faster following precision, relative to power grasp cues. The results demonstrate a direct influence of grasp preparation on sensory processing of size and suggest that the hierarchical dimension of objects may be a relevant perceptual feature for grasp programming. To our knowledge, this is the first evidence that preparing different magnitudes of the same basic action has systematic effects on visual processing.

Effector-based attention systems

Visual processing is known to be enhanced at the end point of eye movements. Feedback within the oculomotor system has been shown to drive these alterations in visual processing. However, we do not simply view the world; we also reach out and interact using our hands. Consequently, it is not surprising that visual processing has also been shown to be altered in near-hand space. A growing body of work documents a myriad of alterations in near-hand visual processing, with little consensus on the neural underpinnings of the effect of the hand. Since movement of the eyes and hands is governed by parallel frontoparietal networks and since within the oculomotor system feedback from these motor control regions has been shown to drive enhanced visual processing at saccade end points, it is plausible that a similar feedback mechanism is at play in near-hand improvements in visual processing. Here, we compare and contrast oculomotor-driven and hand-driven changes in visual processing and provide support for the hypothesis that feedback within the reaching and grasping systems enhances visual processing near the hand in a novel way.

Approaching threatening stimuli cause an expansion of defensive peripersonal space

When sudden environmental stimuli signaling threat occur in the portion of space surrounding the body (defensive peripersonal space), defensive responses are enhanced. Recently Bisio et al. (Bisio A, Garbarini F, Biggio M, Fossataro C, Ruggeri P, Bove M. J Neurosci 37: 2415-2424, 2017) showed that a marker of defensive peripersonal space, the defensive hand-blink reflex, is modulated by the motion of the eliciting threatening stimulus. These results can be parsimoniously explained by the continuous monitoring of environmental threats, resulting in an expansion of defensive peripersonal space when threatening stimuli approach.

Neuronal correlates of continuous manual tracking under varying visual movement feedback in a virtual reality environment

To accurately guide one's actions online, the brain predicts sensory action feedback ahead of time based on internal models, which can be updated by sensory prediction errors. The underlying operations can be experimentally investigated in sensorimotor adaptation tasks, in which moving under perturbed sensory action feedback requires internal model updates. Here we altered healthy participants' visual hand movement feedback in a virtual reality setup, while assessing brain activity with functional magnetic resonance imaging (fMRI). Participants tracked a continually moving virtual target object with a photorealistic, three-dimensional (3D) virtual hand controlled online via a data glove. During the continuous tracking task, the virtual hand's movements (i.e., visual movement feedback) were repeatedly periodically delayed, which participants had to compensate for to maintain accurate tracking. This realistic task design allowed us to simultaneously investigate processes likely operating at several levels of the brain's motor control hierarchy. FMRI revealed that the length of visual feedback delay was parametrically reflected by activity in the inferior parietal cortex and posterior temporal cortex. Unpredicted changes in visuomotor mapping (at transitions from synchronous to delayed visual feedback periods or vice versa) activated biological motion-sensitive regions in the lateral occipitotemporal cortex (LOTC). Activity in the posterior parietal cortex (PPC), focused on the contralateral anterior intraparietal sulcus (aIPS), correlated with tracking error, whereby this correlation was stronger in participants with higher tracking performance. Our results are in line with recent proposals of a wide-spread cortical motor control hierarchy, where temporoparietal regions seem to evaluate visuomotor congruence and thus possibly ground a self-attribution of movements, the LOTC likely processes early visual prediction errors, and the aIPS computes action goal errors and possibly corresponding motor corrections.

A moving observer in a three-dimensional world

For many tasks such as retrieving a previously viewed object, an observer must form a representation of the world at one location and use it at another. A world-based three-dimensional reconstruction of the scene built up from visual information would fulfil this requirement, something computer vision now achieves with great speed and accuracy. However, I argue that it is neither easy nor necessary for the brain to do this. I discuss biologically plausible alternatives, including the possibility of avoiding three-dimensional coordinate frames such as ego-centric and world-based representations. For example, the distance, slant and local shape of surfaces dictate the propensity of visual features to move in the image with respect to one another as the observer's perspective changes (through movement or binocular viewing). Such propensities can be stored without the need for three-dimensional reference frames. The problem of representing a stable scene in the face of continual head and eye movements is an appropriate starting place for understanding the goal of three-dimensional vision, more so, I argue, than the case of a static binocular observer.This article is part of the themed issue 'Vision in our three-dimensional world'.

Multisensory simultaneity judgment and proximity to the body

The integration of information across different sensory modalities is known to be dependent upon the statistical characteristics of the stimuli to be combined. For example, the spatial and temporal proximity of stimuli are important determinants with stimuli that are close in space and time being more likely to be bound. These multisensory interactions occur not only for singular points in space/time, but over “windows” of space and time that likely relate to the ecological statistics of real-world stimuli. Relatedly, human psychophysical work has demonstrated that individuals are highly prone to judge multisensory stimuli as co-occurring over a wide range of time—a so-called simultaneity window (SW). Similarly, there exists a spatial representation of peripersonal space (PPS) surrounding the body in which stimuli related to the body and to external events occurring near the body are highly likely to be jointly processed. In the current study, we sought to examine the interaction between these temporal and spatial dimensions of multisensory representation by measuring the SW for audiovisual stimuli through proximal–distal space (i.e., PPS and extrapersonal space). Results demonstrate that the audiovisual SWs within PPS are larger than outside PPS. In addition, we suggest that this effect is likely due to an automatic and additional computation of these multisensory events in a body-centered reference frame. We discuss the current findings in terms of the spatiotemporal constraints of multisensory interactions and the implication of distinct reference frames on this process.

Tool-use: An open window into body representation and its plasticity

Over the last decades, scientists have questioned the origin of the exquisite human mastery of tools. Seminal studies in monkeys, healthy participants and brain-damaged patients have primarily focused on the plastic changes that tool-use induces on spatial representations. More recently, we focused on the modifications tool-use must exert on the sensorimotor system and highlighted plastic changes at the level of the body representation used by the brain to control our movements, i.e., the Body Schema. Evidence is emerging for tool-use to affect also more visually and conceptually based representations of the body, such as the Body Image. Here we offer a critical review of the way different tool-use paradigms have been, and should be, used to try disentangling the critical features that are responsible for tool incorporation into different body representations. We will conclude that tool-use may offer a very valuable means to investigate high-order body representations and their plasticity.

Depth: the Forgotten Dimension in Multisensory Research

The last quarter of a century has seen a dramatic rise of interest in the spatial constraints on multisensory integration. However, until recently, the majority of this research has investigated integration in the space directly in front of the observer. The space around us, however, extends in three spatial dimensions in the front and to the rear beyond such a limited area. The question to be addressed in this review concerns whether multisensory integration operates according to the same rules throughout the whole of three-dimensional space. The results reviewed here not only show that the space around us seems to be divided into distinct functional regions, but they also suggest that multisensory interactions are modulated by the region of space in which stimuli happen to be presented. We highlight a number of key limitations with previous research in this area, including: (1) The focus on only a very narrow region of two-dimensional space in front of the observer; (2) the use of static stimuli in most research; (3) the study of observers who themselves have been mostly static; and (4) the study of isolated observers. All of these factors may change the way in which the senses interact at any given distance, as can the emotional state/personality of the observer. In summarizing these salient issues, we hope to encourage researchers to consider these factors in their own research in order to gain a better understanding of the spatial constraints on multisensory integration as they affect us in our everyday life.

Predictive coding accounts of shared representations in parieto-insular networks

The discovery of mirror neurons in the ventral premotor cortex (area F5) and inferior parietal cortex (area PFG) in the macaque monkey brain has provided the physiological evidence for direct matching of the intrinsic motor representations of the self and the visual image of the actions of others. The existence of mirror neurons implies that the brain has mechanisms reflecting shared self and other action representations. This may further imply that the neural basis self-body representations may also incorporate components that are shared with other-body representations. It is likely that such a mechanism is also involved in predicting other's touch sensations and emotions. However, the neural basis of shared body representations has remained unclear. Here, we propose a neural basis of body representation of the self and of others in both human and non-human primates. We review a series of behavioral and physiological findings which together paint a picture that the systems underlying such shared representations require integration of conscious exteroception and interoception subserved by a cortical sensory-motor network involving parieto-inner perisylvian circuits (the ventral intraparietal area [VIP]/inferior parietal area [PFG]-secondary somatosensory cortex [SII]/posterior insular cortex [pIC]/anterior insular cortex [aIC]). Based on these findings, we propose a computational mechanism of the shared body representation in the predictive coding (PC) framework. Our mechanism proposes that processes emerging from generative models embedded in these specific neuronal circuits play a pivotal role in distinguishing a self-specific body representation from a shared one. The model successfully accounts for normal and abnormal shared body phenomena such as mirror-touch synesthesia and somatoparaphrenia. In addition, it generates a set of testable experimental predictions.

Multisensory Representation of the Space Near the Hand

When interacting with objects and other people, the brain needs to locate our limbs and the relevant visual information surrounding them. Studies on monkeys showed that information from different sensory modalities converge at the single cell level within a set of interconnected multisensory frontoparietal areas. It is largely accepted that this network allows for multisensory processing of the space surrounding the body (peripersonal space), whose function has been linked to the sensory guidance of appetitive and defensive movements, and localization of the limbs in space. In the current review, we consider multidisciplinary findings about the processing of the space near the hands in humans and offer a convergent view of its functions and underlying neural mechanisms. We will suggest that evolution has provided the brain with a clever tool for representing visual information around the hand, which takes the hand itself as a reference for the coding of surrounding visual space. We will contend that the hand-centered representation of space, known as perihand space, is a multisensory-motor interface that allows interaction with the objects and other persons around us.

Global and local processing near the left and right hands

Visual targets can be processed more quickly and reliably when a hand is placed near the target. Both unimodal and bimodal representations of hands are largely lateralized to the contralateral hemisphere, and since each hemisphere demonstrates specialized cognitive processing, it is possible that targets appearing near the left hand may be processed differently than targets appearing near the right hand. The purpose of this study was to determine whether visual processing near the left and right hands interacts with hemispheric specialization. We presented hierarchical-letter stimuli (e.g., small characters used as local elements to compose large characters at the global level) near the left or right hands separately and instructed participants to discriminate the presence of target letters (X and O) from non-target letters (T and U) at either the global or local levels as quickly as possible. Targets appeared at either the global or local level of the display, at both levels, or were absent from the display; participants made foot-press responses. When discriminating target presence at the global level, participants responded more quickly to stimuli presented near the left hand than near either the right hand or in the no-hand condition. Hand presence did not influence target discrimination at the local level. Our interpretation is that left-hand presence may help participants discriminate global information, a right hemisphere (RH) process, and that the left hand may influence visual processing in a way that is distinct from the right hand.

A brief review of the role of training in near-tool effects

Research suggests that, like near-hand effects, visual targets appearing near the tip of a hand-held real or virtual tool are treated differently than other targets. This paper reviews neurological and behavioral evidence relevant to near-tool effects and describes how the effect varies with the functional properties of the tool and the knowledge of the participant. In particular, the paper proposes that motor knowledge plays a key role in the appearance of near-tool effects.

A self-organizing model of the visual development of hand-centred representations

We show how hand-centred visual representations could develop in the primate posterior parietal and premotor cortices during visually guided learning in a self-organizing neural network model. The model incorporates trace learning in the feed-forward synaptic connections between successive neuronal layers. Trace learning encourages neurons to learn to respond to input images that tend to occur close together in time. We assume that sequences of eye movements are performed around individual scenes containing a fixed hand-object configuration. Trace learning will then encourage individual cells to learn to respond to particular hand-object configurations across different retinal locations. The plausibility of this hypothesis is demonstrated in computer simulations.

That's near my hand! Parietal and premotor coding of hand-centered space contributes to localization and self-attribution of the hand

The ability to identify and localize our own limbs is crucial for survival. Indeed, the majority of our interactions with objects occur within the space surrounding the hands. In non-human primates, neurons in the posterior parietal and premotor cortices dynamically represent the space near the upper limbs in hand-centered coordinates. Neuronal populations selective for the space near the hand also exist in humans. It is unclear whether these remap the peri-hand representation as the arm is moved in space. Furthermore, no combined neuronal and behavioral data are available about the possible involvement of peri-hand neurons in the perception of the upper limbs in any species. We used fMRI adaptation to demonstrate dynamic hand-centered encoding of space by reporting response suppression in human premotor and posterior parietal cortices to repeated presentations of an object near the hand for different arm postures. Furthermore, we show that such spatial representation is related to changes in body perception, being remapped onto a prosthetic hand if perceived as one's own during an illusion. Interestingly, our results further suggest that peri-hand space remapping in the premotor cortex is most tightly linked to the subjective feeling of ownership of the seen limb, whereas remapping in the posterior parietal cortex closely reflects changes in the position sense of the arm. These findings identify the neural bases for dynamic hand-centered encoding of peripersonal space in humans and provide hitherto missing evidence for the link between the peri-hand representation of space and the perceived self-attribution and position of the upper limb.