An Eye in the Palm of Your Hand: Alterations in Visual Processing Near the Hand, a Mini-Review

Rethinking Vision and Action

Action is an important arbitrator as to whether an individual or a species will survive. Yet, action has not been well integrated into the study of psychology. Action or motor behavior is a field apart. This is traditional science with its need for specialization. The sequence in a typical laboratory experiment of see → decide → act provides the rationale for broad disciplinary categorizations. With renewed interest in action itself, surprising and exciting anomalous findings at odds with this simplified caricature have emerged. They reveal a much more intimate coupling of vision and action, which we describe. In turn, this prompts us to identify and dwell on three pertinent theories deserving of greater notice.

A Survey of Multifingered Robotic Manipulation: Biological Results, Structural Evolvements, and Learning Methods

Multifingered robotic hands (usually referred to as dexterous hands) are designed to achieve human-level or human-like manipulations for robots or as prostheses for the disabled. The research dates back 30 years ago, yet, there remain great challenges to effectively design and control them due to their high dimensionality of configuration, frequently switched interaction modes, and various task generalization requirements. This article aims to give a brief overview of multifingered robotic manipulation from three aspects: a) the biological results, b) the structural evolvements, and c) the learning methods, and discuss potential future directions. First, we investigate the structure and principle of hand-centered visual sensing, tactile sensing, and motor control and related behavioral results. Then, we review several typical multifingered dexterous hands from task scenarios, actuation mechanisms, and in-hand sensors points. Third, we report the recent progress of various learning-based multifingered manipulation methods, including but not limited to reinforcement learning, imitation learning, and other sub-class methods. The article concludes with open issues and our thoughts on future directions.

Efference copy in kinesthetic perception: a copy of what is it?

A number of notions in the fields of motor control and kinesthetic perception have been used without clear definitions. In this review, we consider definitions for efference copy, percept, and sense of effort based on recent studies within the physical approach, which assumes that the neural control of movement is based on principles of parametric control and involves defining time-varying profiles of spatial referent coordinates for the effectors. The apparent redundancy in both motor and perceptual processes is reconsidered based on the principle of abundance. Abundance of efferent and afferent signals is viewed as the means of stabilizing both salient action characteristics and salient percepts formalized as stable manifolds in high-dimensional spaces of relevant elemental variables. This theoretical scheme has led recently to a number of novel predictions and findings. These include, in particular, lower accuracy in perception of variables produced by elements involved in a multielement task compared with the same elements in single-element tasks, dissociation between motor and perceptual effects of muscle coactivation, force illusions induced by muscle vibration, and errors in perception of unintentional drifts in performance. Taken together, these results suggest that participation of efferent signals in perception frequently involves distorted copies of actual neural commands, particularly those to antagonist muscles. Sense of effort is associated with such distorted efferent signals. Distortions in efference copy happen spontaneously and can also be caused by changes in sensory signals, e.g., those produced by muscle vibration.

Rodin has it! The role of hands in improving the selectivity of attention

We report a new discovery on the role of hands in guiding attention, using the classic Stroop effect as our assay. We show that the Stroop effect diminishes, hence selective attention improves, when observers hold their chin, emulating Rodin's famous sculpture, "The Thinker." In two experiments we show that the Rodin posture improves the selectivity of attention as efficiently as holding the hands nearby the visual stimulus (the near-hands effect). Because spatial proximity to the displayed stimulus is neither present nor intended, the presence of the Rodin effect implies that attentional prioritization by the hands is not limited to the space between the hands.

Deploying attention to the target location of a pointing action modulates audiovisual processes at nontarget locations

The current study examined how the deployment of spatial attention at the onset of a pointing movement influenced audiovisual crossmodal interactions at the target of the pointing action and at nontarget locations. These interactions were quantified by measuring the susceptibility to the fission (i.e., reporting two visual flashes under one flash and two auditory beep pairings) and fusion (i.e., reporting one flash under two flashes and one beep pairing) audiovisual illusions. At movement onset, unimodal, or auditory and visual bimodal stimuli were either presented at the target of the pointing action or in an adjacent, nontarget location. In Experiment 1, perceptual accuracy within the unimodal and bimodal conditions was lower in the nontarget relative to the target condition. The fission illusion was uninfluenced by target condition. However, the fusion illusion was more likely to be reported at the target relative to the nontarget location. In Experiment 2, the stimuli from Experiment 1 were further presented at a location near where the eyes were fixated (i.e., congruent condition), where the hand was aiming (i.e., target), or in a location where neither the eyes were fixated nor the hand was aiming. The results yielded the greatest susceptibility to the fusion illusion when the visual location and movement end points were congruent relative to when either movement or fixation was incongruent. Although attention may facilitate the processing of unisensory and multisensory cues in general, attention might have the strongest influence on the audiovisual integration mechanisms that underlie the sound-induced fusion illusion.

Faster recognition of graspable targets defined by orientation in a visual search task

Peri-hand space is the area surrounding the hand. Objects within this space may be subject to increased visuospatial perception, increased attentional prioritization, and slower attentional disengagement compared to more distal objects. This may result from kinesthetic and visual feedback about the location of the hand that projects from the reach and grasp networks of the dorsal visual stream back to occipital visual areas, which in turn, refines cortical visual processing that can subsequently guide skilled motor actions. Thus, we hypothesized that visual stimuli that afford action, which are known to potentiate activity in the dorsal visual stream, would be associated with greater alterations in visual processing when presented near the hand. To test this, participants held their right hand near or far from a touchscreen that presented a visual array containing a single target object that differed from 11 distractor objects by orientation only. The target objects and their accompanying distractors either strongly afforded grasping or did not. Participants identified the target among the distractors by reaching out and touching it with their left index finger while eye-tracking was used to measure visual search times, target recognition times, and search accuracy. The results failed to support the theory of enhanced visual processing of graspable objects near the hand as participants were faster at recognizing graspable compared to non-graspable targets, regardless of the position of the right hand. The results are discussed in relation to the idea that, in addition to potentiating appropriate motor responses, object affordances may also potentiate early visual processes necessary for object recognition.

Tuning characteristics of low-frequency EEG to positions and velocities in visuomotor and oculomotor tracking tasks

Movement decoders exploit the tuning of neural activity to various movement parameters with the ultimate goal of controlling end-effector action. Invasive approaches, typically relying on spiking activity, have demonstrated feasibility. Results of recent functional neuroimaging studies suggest that information about movement parameters is even accessible non-invasively in the form of low-frequency brain signals. However, their spatiotemporal tuning characteristics to single movement parameters are still unclear. Here, we extend the current understanding of low-frequency electroencephalography (EEG) tuning to position and velocity signals. We recorded EEG from 15 healthy participants while they performed visuomotor and oculomotor pursuit tracking tasks. Linear decoders, fitted to EEG signals in the frequency range of the tracking movements, predicted positions and velocities with moderate correlations (0.2–0.4; above chance level) in both tasks. Predictive activity in terms of decoder patterns was significant in superior parietal and parieto-occipital areas in both tasks. By contrasting the two tracking tasks, we found that predictive activity in contralateral primary sensorimotor and premotor areas exhibited significantly larger tuning to end-effector velocity when the visuomotor tracking task was performed.

Independent Effects of Eye and Hand Movements on Visual Working Memory

Both eye and hand movements have been shown to selectively interfere with visual working memory. We investigated working memory in the context of simultaneous eye-hand movements to approach the question whether the eye and the hand movement systems independently interact with visual working memory. Participants memorized several locations and performed eye, hand, or simultaneous eye-hand movements during the maintenance interval. Subsequently, we tested spatial working memory at the eye or the hand motor goal, and at action-irrelevant locations. We found that for single eye and single hand movements, memory at the eye or hand target was significantly improved compared to action-irrelevant locations. Remarkably, when an eye and a hand movement were prepared in parallel, but to distinct locations, memory at both motor targets was enhanced-with no tradeoff between the two separate action goals. This suggests that eye and hand movements independently enhance visual working memory at their goal locations, resulting in an overall working memory performance that is higher than that expected when recruiting only one effector.

Effect of vibration during visual-inertial integration on human heading perception during eccentric gaze

Heading direction is determined from visual and inertial cues. Visual headings use retinal coordinates while inertial headings use body coordinates. Thus during eccentric gaze the same heading may be perceived differently by visual and inertial modalities. Stimulus weights depend on the relative reliability of these stimuli, but previous work suggests that the inertial heading may be given more weight than predicted. These experiments only varied the visual stimulus reliability, and it is unclear what occurs with variation in inertial reliability. Five human subjects completed a heading discrimination task using 2s of translation with a peak velocity of 16cm/s. Eye position was ±25° left/right with visual, inertial, or combined motion. The visual motion coherence was 50%. Inertial stimuli included 6 Hz vertical vibration with 0, 0.10, 0.15, or 0.20cm amplitude. Subjects reported perceived heading relative to the midline. With an inertial heading, perception was biased 3.6° towards the gaze direction. Visual headings biased perception 9.6° opposite gaze. The inertial threshold without vibration was 4.8° which increased significantly to 8.8° with vibration but the amplitude of vibration did not influence reliability. With visual-inertial headings, empirical stimulus weights were calculated from the bias and compared with the optimal weight calculated from the threshold. In 2 subjects empirical weights were near optimal while in the remaining 3 subjects the inertial stimuli were weighted greater than optimal predictions. On average the inertial stimulus was weighted greater than predicted. These results indicate multisensory integration may not be a function of stimulus reliability when inertial stimulus reliability is varied.

Three-dimensional binocular eye-hand coordination in normal vision and with simulated visual impairment

Sensorimotor coupling in healthy humans is demonstrated by the higher accuracy of visually tracking intrinsically-rather than extrinsically-generated hand movements in the fronto-parallel plane. It is unknown whether this coupling also facilitates vergence eye movements for tracking objects in depth, or can overcome symmetric or asymmetric binocular visual impairments. Human observers were therefore asked to track with their gaze a target moving horizontally or in depth. The movement of the target was either directly controlled by the observer's hand or followed hand movements executed by the observer in a previous trial. Visual impairments were simulated by blurring stimuli independently in each eye. Accuracy was higher for self-generated movements in all conditions, demonstrating that motor signals are employed by the oculomotor system to improve the accuracy of vergence as well as horizontal eye movements. Asymmetric monocular blur affected horizontal tracking less than symmetric binocular blur, but impaired tracking in depth as much as binocular blur. There was a critical blur level up to which pursuit and vergence eye movements maintained tracking accuracy independent of blur level. Hand-eye coordination may therefore help compensate for functional deficits associated with eye disease and may be employed to augment visual impairment rehabilitation.

Effect of eye position during human visual-vestibular integration of heading perception

Visual and inertial stimuli provide heading discrimination cues. Integration of these multisensory stimuli has been demonstrated to depend on their relative reliability. However, the reference frame of visual stimuli is eye centered while inertia is head centered, and it remains unclear how these are reconciled with combined stimuli. Seven human subjects completed a heading discrimination task consisting of a 2-s translation with a peak velocity of 16 cm/s. Eye position was varied between 0° and ±25° left/right. Experiments were done with inertial motion, visual motion, or a combined visual-inertial motion. Visual motion coherence varied between 35% and 100%. Subjects reported whether their perceived heading was left or right of the midline in a forced-choice task. With the inertial stimulus the eye position had an effect such that the point of subjective equality (PSE) shifted 4.6 ± 2.4° in the gaze direction. With the visual stimulus the PSE shift was 10.2 ± 2.2° opposite the gaze direction, consistent with retinotopic coordinates. Thus with eccentric eye positions the perceived inertial and visual headings were offset ~15°. During the visual-inertial conditions the PSE varied consistently with the relative reliability of these stimuli such that at low visual coherence the PSE was similar to that of the inertial stimulus and at high coherence it was closer to the visual stimulus. On average, the inertial stimulus was weighted near Bayesian ideal predictions, but there was significant deviation from ideal in individual subjects. These findings support visual and inertial cue integration occurring in independent coordinate systems.NEW & NOTEWORTHY In multiple cortical areas visual heading is represented in retinotopic coordinates while inertial heading is in body coordinates. It remains unclear whether multisensory integration occurs in a common coordinate system. The experiments address this using a multisensory integration task with eccentric gaze positions making the effect of coordinate systems clear. The results indicate that the coordinate systems remain separate to the perceptual level and that during the multisensory task the perception depends on relative stimulus reliability.