Hand placement near the visual stimulus improves orientation selectivity in V2 neurons

Action-specific feature processing in the human cortex: An fMRI study

Sensorimotor integration involves feedforward and reentrant processing of sensory input. Grasp-related motor activity precedes and is thought to influence visual object processing. Yet, while the importance of reentrant feedback is well established in perception, the top-down modulations for action and the neural circuits involved in this process have received less attention. Do action-specific intentions influence the processing of visual information in the human cortex? Using a cue-separation fMRI paradigm, we found that action-specific instruction processing (manual alignment vs. grasp) became apparent only after the visual presentation of oriented stimuli, and occurred as early as in the primary visual cortex and extended to the dorsal visual stream, motor and premotor areas. Further, dorsal stream area aIPS, known to be involved in object manipulation, and the primary visual cortex showed task-related functional connectivity with frontal, parietal and temporal areas, consistent with the idea that reentrant feedback from dorsal and ventral visual stream areas modifies visual inputs to prepare for action. Importantly, both the task-dependent modulations and connections were linked specifically to the object presentation phase of the task, suggesting a role in processing the action goal. Our results show that intended manual actions have an early, pervasive, and differential influence on the cortical processing of vision.

Rethinking Vision and Action

Action is an important arbitrator as to whether an individual or a species will survive. Yet, action has not been well integrated into the study of psychology. Action or motor behavior is a field apart. This is traditional science with its need for specialization. The sequence in a typical laboratory experiment of see → decide → act provides the rationale for broad disciplinary categorizations. With renewed interest in action itself, surprising and exciting anomalous findings at odds with this simplified caricature have emerged. They reveal a much more intimate coupling of vision and action, which we describe. In turn, this prompts us to identify and dwell on three pertinent theories deserving of greater notice.

A Survey of Multifingered Robotic Manipulation: Biological Results, Structural Evolvements, and Learning Methods

Multifingered robotic hands (usually referred to as dexterous hands) are designed to achieve human-level or human-like manipulations for robots or as prostheses for the disabled. The research dates back 30 years ago, yet, there remain great challenges to effectively design and control them due to their high dimensionality of configuration, frequently switched interaction modes, and various task generalization requirements. This article aims to give a brief overview of multifingered robotic manipulation from three aspects: a) the biological results, b) the structural evolvements, and c) the learning methods, and discuss potential future directions. First, we investigate the structure and principle of hand-centered visual sensing, tactile sensing, and motor control and related behavioral results. Then, we review several typical multifingered dexterous hands from task scenarios, actuation mechanisms, and in-hand sensors points. Third, we report the recent progress of various learning-based multifingered manipulation methods, including but not limited to reinforcement learning, imitation learning, and other sub-class methods. The article concludes with open issues and our thoughts on future directions.

Direction-selective modulation of visual motion rivalry by collocated tactile motion

Early models of multisensory integration posited that cross-modal signals only converged in higher-order association cortices and that vision automatically dominates. However, recent studies have challenged this view. In this study, the significance of the alignment of motion axes and spatial alignment across visual and tactile stimuli, as well as the effect of hand visibility on visuo-tactile interactions were examined. Using binocular rivalry, opposed motions were presented to each eye and participants were required to track the perceived visual direction. A tactile motion that was either a leftward or rightward sweep across the fingerpad was intermittently presented. Results showed that tactile effects on visual percepts were dependent on the alignment of motion axes: rivalry between up/down visual motions was not modulated at all by left/right tactile motion. On the other hand, visual percepts could be altered by tactile motion signals when both modalities shared a common axis of motion: a tactile stimulus could maintain the dominance duration of a congruent visual stimulus and shorten its suppression period. The effects were also conditional on the spatial alignment of the visual and tactile stimuli, being eliminated when the tactile device was displaced 15 cm away to the right of the visual stimulus. In contrast, visibility of the hand touching the tactile stimulus facilitated congruent switches relative to a visual-only baseline but did not present a significant advantage overall. In sum, these results show a low-level sensory interaction that is conditional on visual and tactile stimuli sharing a common motion axis and location in space.

Variability of neuronal responses in the posterior superior temporal sulcus predicts choice behavior during social interactions

Recent studies have shown that neural activity in a well-defined patch in the posterior superior temporal sulcus (the "gaze-following patch," GFP) of the primate brain is strongly modulated when the other's gaze attracts the observer's attention to locations/objects, the other is looking at. Changes of the mean discharge rate of neurons in the monkey GFP indicate that they are involved in two distinct computations: the allocation of spatial attention guided by the other's gaze vector and the suppression of gaze following if inappropriate in a given situation. Here, we asked if and how the discharge variability of neurons in the GFP is related to the task and if it carries information on behavioral performance. To this end, we calculated the Fano factor as a measure of across-trial discharge variability as a function of time. Our results show that all neurons exhibiting a task-related discharge-rate modulation also exhibit a stimulus onset-dependent drop in the Fano factor. Furthermore, the amplitude of the Fano factor reduction is modulated by task condition and the neuron's selectivity in this regard. We found that these effects are directly related to the monkeys' behavioral performance in that the Fano factor is predictive about upcoming correct or wrong decisions. Our results indicate that neuronal discharge variability as gauged by the Fano factor, hitherto primarily studied in the context of visual perception or motor control, is an informative measure also in studies of the neural underpinnings of complex social behavior.NEW & NOTEWORTHY Quenching of neural variability following stimulus onset is a widely accepted phenomenon. However, the relevance of quenching for the shaping of complex social behaviors remains to be explored. Here, we show that task selective neurons in the GFP exhibit a higher degree of variability quenching than their neighboring unselective neurons. Furthermore, we demonstrate that behavioral errors are not only associated with lower firing rates but also less variability quenching, suggesting that both facilitate optimal performance.

Keeping it real: Looking beyond capacity limits in visual cognition

Research within visual cognition has made tremendous strides in uncovering the basic operating characteristics of the visual system by reducing the complexity of natural vision to artificial but well-controlled experimental tasks and stimuli. This reductionist approach has for example been used to assess the basic limitations of visual attention, visual working memory (VWM) capacity, and the fidelity of visual long-term memory (VLTM). The assessment of these limits is usually made in a pure sense, irrespective of goals, actions, and priors. While it is important to map out the bottlenecks our visual system faces, we focus here on selected examples of how such limitations can be overcome. Recent findings suggest that during more natural tasks, capacity may be higher than reductionist research suggests and that separable systems subserve different actions, such as reaching and looking, which might provide important insights about how pure attentional or memory limitations could be circumvented. We also review evidence suggesting that the closer we get to naturalistic behavior, the more we encounter implicit learning mechanisms that operate "for free" and "on the fly." These mechanisms provide a surprisingly rich visual experience, which can support capacity-limited systems. We speculate whether natural tasks may yield different estimates of the limitations of VWM, VLTM, and attention, and propose that capacity measurements should also pass the real-world test within naturalistic frameworks. Our review highlights various approaches for this and suggests that our understanding of visual cognition will benefit from incorporating the complexities of real-world cognition in experimental approaches.

Deploying attention to the target location of a pointing action modulates audiovisual processes at nontarget locations

The current study examined how the deployment of spatial attention at the onset of a pointing movement influenced audiovisual crossmodal interactions at the target of the pointing action and at nontarget locations. These interactions were quantified by measuring the susceptibility to the fission (i.e., reporting two visual flashes under one flash and two auditory beep pairings) and fusion (i.e., reporting one flash under two flashes and one beep pairing) audiovisual illusions. At movement onset, unimodal, or auditory and visual bimodal stimuli were either presented at the target of the pointing action or in an adjacent, nontarget location. In Experiment 1, perceptual accuracy within the unimodal and bimodal conditions was lower in the nontarget relative to the target condition. The fission illusion was uninfluenced by target condition. However, the fusion illusion was more likely to be reported at the target relative to the nontarget location. In Experiment 2, the stimuli from Experiment 1 were further presented at a location near where the eyes were fixated (i.e., congruent condition), where the hand was aiming (i.e., target), or in a location where neither the eyes were fixated nor the hand was aiming. The results yielded the greatest susceptibility to the fusion illusion when the visual location and movement end points were congruent relative to when either movement or fixation was incongruent. Although attention may facilitate the processing of unisensory and multisensory cues in general, attention might have the strongest influence on the audiovisual integration mechanisms that underlie the sound-induced fusion illusion.

Faster recognition of graspable targets defined by orientation in a visual search task

Peri-hand space is the area surrounding the hand. Objects within this space may be subject to increased visuospatial perception, increased attentional prioritization, and slower attentional disengagement compared to more distal objects. This may result from kinesthetic and visual feedback about the location of the hand that projects from the reach and grasp networks of the dorsal visual stream back to occipital visual areas, which in turn, refines cortical visual processing that can subsequently guide skilled motor actions. Thus, we hypothesized that visual stimuli that afford action, which are known to potentiate activity in the dorsal visual stream, would be associated with greater alterations in visual processing when presented near the hand. To test this, participants held their right hand near or far from a touchscreen that presented a visual array containing a single target object that differed from 11 distractor objects by orientation only. The target objects and their accompanying distractors either strongly afforded grasping or did not. Participants identified the target among the distractors by reaching out and touching it with their left index finger while eye-tracking was used to measure visual search times, target recognition times, and search accuracy. The results failed to support the theory of enhanced visual processing of graspable objects near the hand as participants were faster at recognizing graspable compared to non-graspable targets, regardless of the position of the right hand. The results are discussed in relation to the idea that, in addition to potentiating appropriate motor responses, object affordances may also potentiate early visual processes necessary for object recognition.

Action Fluency Facilitates Perceptual Discrimination

Perception and action interact in nearly every moment of daily life. Previous studies have demonstrated not only that perceptual input shapes action but also that various factors associated with action-including individual abilities and biomechanical costs-influence perceptual decisions. However, it is unknown how action fluency affects the sensitivity of early-stage visual perception, such as orientation. To address this question, we used a dual-task paradigm: Participants prepared an action (e.g., grasping), while concurrently performing an orientation-change-detection task. We demonstrated that as actions became more fluent (e.g., as grasping errors decreased), perceptual-discrimination performance also improved. Importantly, we found that grasping training prior to discrimination enhanced subsequent perceptual sensitivity, supporting the notion of a reciprocal relation between perception and action.

Feedback inhibition derived from the posterior parietal cortex regulates the neural properties of the mouse visual cortex

Feedback regulation from the higher association areas is thought to control the primary sensory cortex, contribute to the cortical processing of sensory information, and work for higher cognitive functions such as multimodal integration and attentional control. However, little is known about the underlying neural mechanisms. Here, we show that the posterior parietal cortex (PPC) persistently inhibits the activity of the primary visual cortex (V1) in mice. Activation of the PPC causes the suppression of visual responses in V1 and induces the short-term depression, which is specific to visual stimuli. In contrast, pharmacological inactivation of the PPC or disconnection of cortical pathways from the PPC to V1 results in an effect of transient enhancement of visual responses in V1. Two-photon calcium imaging demonstrated that the cortical disconnection caused V1 excitatory neurons an enhancement of visual responses and a reduction of orientation selectivity index (OSI). These results show that the PPC regulates the response properties of V1 excitatory neurons. Our findings reveal one of the functions of the PPC, which may contribute to higher brain functions in mice.

The Fragility of the Near-Hand Effect

Recent literature has demonstrated that hand position can affect visual processing, a set of phenomena termed Near Hand Effects (NHEs). Across four studies we looked for single-hand NHEs on a large screen when participants were asked to discriminate stimuli based on size, colour, and orientation (Study 1), to detect stimuli after a manipulation of hand shaping (Study 2), to detect stimuli after the introduction of a peripheral cue (Study 3), and finally to detect stimuli after a manipulation of screen orientation (Study 4). Each study failed to find a NHE. Further examination of the pooled data using a Bayesian analysis also failed to reveal positive evidence for faster responses or larger cueing effects near a hand. These findings suggest that at least some NHEs may be surprisingly fragile, which dovetails with the recent proposition that NHEs may not form a unitary set of phenomena (Gozli & Deng, 2018). The implication is that visual processing may be less sensitive to hand position across measurement techniques than previously thought, and points to a need for well-powered, methodologically rigorous studies on this topic in the future.

Touch the table before the target: contact with an underlying surface may assist the development of precise visually controlled reach and grasp movements in human infants

Multiple motor channel theory posits that skilled hand movements arise from the coordinated activation of separable neural circuits in parietofrontal cortex, each of which produces a distinct movement and responds to different sensory inputs. Prehension, the act of reaching to grasp an object, consists of at least two movements: a reach movement that transports the hand to a target location and a grasp movement that shapes and closes the hand for target acquisition. During early development, discrete pre-reach and pre-grasp movements are refined based on proprioceptive and tactile feedback, but are gradually coordinated together into a singular hand preshaping movement under feedforward visual control. The neural and behavioural factors that enable this transition are currently unknown. In an attempt to identify such factors, the present descriptive study used frame-by-frame video analysis to examine 9-, 12-, and 15-month-old infants, along with sighted and unsighted adults, as they reached to grasp small ring-shaped pieces of cereal (Cheerios) resting on a table. Compared to sighted adults, infants and unsighted adults were more likely to make initial contact with the underlying table before they contacted the target. The way in which they did so was also similar in that they generally contacted the table with the tip of the thumb and/or pinky finger, a relatively open hand, and poor reach accuracy. Despite this, infants were similar to sighted adults in that they tended to use a pincer digit, defined as the tip of the thumb or index finger, to subsequently contact the target. Only in infants was this ability related to their having made prior contact with the underlying table. The results are discussed in relation to the idea that initial contact with an underlying table or surface may assist infants in learning to use feedforward visual control to direct their digits towards a precise visual target.

Three-dimensional binocular eye-hand coordination in normal vision and with simulated visual impairment

Sensorimotor coupling in healthy humans is demonstrated by the higher accuracy of visually tracking intrinsically-rather than extrinsically-generated hand movements in the fronto-parallel plane. It is unknown whether this coupling also facilitates vergence eye movements for tracking objects in depth, or can overcome symmetric or asymmetric binocular visual impairments. Human observers were therefore asked to track with their gaze a target moving horizontally or in depth. The movement of the target was either directly controlled by the observer's hand or followed hand movements executed by the observer in a previous trial. Visual impairments were simulated by blurring stimuli independently in each eye. Accuracy was higher for self-generated movements in all conditions, demonstrating that motor signals are employed by the oculomotor system to improve the accuracy of vergence as well as horizontal eye movements. Asymmetric monocular blur affected horizontal tracking less than symmetric binocular blur, but impaired tracking in depth as much as binocular blur. There was a critical blur level up to which pursuit and vergence eye movements maintained tracking accuracy independent of blur level. Hand-eye coordination may therefore help compensate for functional deficits associated with eye disease and may be employed to augment visual impairment rehabilitation.

Effector-based attention systems

Visual processing is known to be enhanced at the end point of eye movements. Feedback within the oculomotor system has been shown to drive these alterations in visual processing. However, we do not simply view the world; we also reach out and interact using our hands. Consequently, it is not surprising that visual processing has also been shown to be altered in near-hand space. A growing body of work documents a myriad of alterations in near-hand visual processing, with little consensus on the neural underpinnings of the effect of the hand. Since movement of the eyes and hands is governed by parallel frontoparietal networks and since within the oculomotor system feedback from these motor control regions has been shown to drive enhanced visual processing at saccade end points, it is plausible that a similar feedback mechanism is at play in near-hand improvements in visual processing. Here, we compare and contrast oculomotor-driven and hand-driven changes in visual processing and provide support for the hypothesis that feedback within the reaching and grasping systems enhances visual processing near the hand in a novel way.

An Eye in the Palm of Your Hand: Alterations in Visual Processing Near the Hand, a Mini-Review

Feedback within the oculomotor system improves visual processing at eye movement end points, also termed a visual grasp. We do not just view the world around us however, we also reach out and grab things with our hands. A growing body of literature suggests that visual processing in near-hand space is altered. The control systems for moving either the eyes or the hands rely on parallel networks of fronto-parietal regions, which have feedback connections to visual areas. Since the oculomotor system effects on visual processing occur through feedback, both through the motor plan and the motor efference copy, a parallel system where reaching and/or grasping motor-related activity also affects visual processing is likely. Areas in the posterior parietal cortex, for example, receive proprioceptive and visual information used to guide actions, as well as motor efference signals. This trio of information channels is all that would be necessary to produce spatial allocation of reach-related visual attention. We review evidence from behavioral and neurophysiological studies that support the hypothesis that feedback from the reaching and/or grasping motor control networks affects visual processing while noting ways in which it differs from that seen within the oculomotor system. We also suggest that object affordances may represent the neural mechanism through which certain object features are selected for preferential processing when stimuli are near the hand. Finally, we summarize the two effector-based feedback systems and discuss how having separate but parallel effector systems allows for efficient decoupling of eye and hand movements.