Depth-cue integration in grasp programming: no evidence for a binocular specialism

Impact of focus cue presentation on perceived realism of 3-D scene structure: Implications for scene perception and for display technology

Stereoscopic imagery often aims to evoke three-dimensional (3-D) percepts that are accurate and realistic-looking. The "gap" between 3-D imagery and real scenes is small, but focus cues typically remain incorrect because images are displayed on a single focal plane. Research has concentrated on the resulting vergence-accommodation conflicts. Yet, incorrect focus cues may also affect the appearance of 3-D imagery. We investigated whether incorrect focus cues reduce perceived realism of 3-D structure ("depth realism"). Experiment 1 used a multiple-focal-planes display to compare depth realism with correct focus cues vs. conventional stereo presentation. The stimuli were random-dot stereograms, which isolated the role of focus cues. Depth realism was consistently lower with incorrect focus cues, providing proof-of-principle evidence that they contribute to perceptual realism. Experiments 2 and 3 examined whether focus cues play a similar role with realistic objects, presented with an almost complete set of visual cues using a high-resolution, high-dynamic-range multiple-focal-planes display. We also examined the efficacy of approximating correct focus cues via gaze-contingent depth-of-field rendering. Improvements in depth realism with correct focus cues were less clear in more realistic scenes, indicating that the role of focus cues in depth realism depends on scene content. Rendering-based approaches, if anything, reduced depth realism, which we attribute to their inability to present higher-order aspects of blur correctly. Our findings suggest future general 3-D display solutions may need to present focus cues correctly to maximise perceptual realism.

Virtual Reality for Vision Science

Virtual reality (VR) allows us to create visual stimuli that are both immersive and reactive. VR provides many new opportunities in vision science. In particular, it allows us to present wide field-of-view, immersive visual stimuli; for observers to actively explore the environments that we create; and for us to understand how visual information is used in the control of behaviour. In contrast with traditional psychophysical experiments, VR provides much greater flexibility in creating environments and tasks that are more closely aligned with our everyday experience. These benefits of VR are of particular value in developing our theories of the behavioural goals of the visual system and explaining how visual information is processed to achieve these goals. The use of VR in vision science presents a number of technical challenges, relating to how the available software and hardware limit our ability to accurately specify the visual information that defines our virtual environments and the interpretation of data gathered in experiments with a freely moving observer in a responsive environment.

Binocular Information Improves the Reliability and Consistency of Pictorial Relief

Binocular disparity is an important cue to three-dimensional shape. We assessed the contribution of this cue to the reliability and consistency of depth in stereoscopic photographs of natural scenes. Observers viewed photographs of cluttered scenes while adjusting a gauge figure to indicate the apparent three-dimensional orientation of the surfaces of objects. The gauge figure was positioned on the surfaces of objects at multiple points in the scene, and settings were made under monocular and binocular, stereoscopic viewing. Settings were used to create a depth relief map, indicating the apparent three-dimensional structure of the scene. We found that binocular cues increased the magnitude of apparent depth, the reliability of settings across repeated measures, and the consistency of perceived depth across participants. These results show that binocular cues make an important contribution to the precise and accurate perception of depth in natural scenes that contain multiple pictorial cues.

Binocular Viewing Facilitates Size Constancy for Grasping and Manual Estimation

A prerequisite for efficient prehension is the ability to estimate an object's distance and size. While most studies demonstrate that binocular viewing is associated with a more efficient grasp programming and execution compared to monocular viewing, the factors contributing to this advantage are not fully understood. Here, we examined how binocular vision facilitates grasp scaling using two tasks: prehension and manual size estimation. Participants (n = 30) were asked to either reach and grasp an object or to provide an estimate of an object's size using their thumb and index finger. The objects were cylinders with a diameter of 0.5, 1.0, or 1.5 cm placed at three distances along the midline (40, 42, or 44 cm). Results from a linear regression analysis relating grip aperture to object size revealed that grip scaling during monocular viewing was reduced similarly for both grasping and estimation tasks. Additional analysis revealed that participants adopted a larger safety margin for grasping during monocular compared to binocular viewing, suggesting that monocular depth cues do not provide sufficient information about an object's properties, which consequently leads to a less efficient grasp execution.

Size and shape constancy in consumer virtual reality

With the increase in popularity of consumer virtual reality headsets, for research and other applications, it is important to understand the accuracy of 3D perception in VR. We investigated the perceptual accuracy of near-field virtual distances using a size and shape constancy task, in two commercially available devices. Participants wore either the HTC Vive or the Oculus Rift and adjusted the size of a virtual stimulus to match the geometric qualities (size and depth) of a physical stimulus they were able to refer to haptically. The judgments participants made allowed for an indirect measure of their perception of the egocentric, virtual distance to the stimuli. The data show under-constancy and are consistent with research from carefully calibrated psychophysical techniques. There was no difference in the degree of constancy found in the two headsets. We conclude that consumer virtual reality headsets provide a sufficiently high degree of accuracy in distance perception, to allow them to be used confidently in future experimental vision science, and other research applications in psychology.

Perceptual uncertainty and action consequences independently affect hand movements in a virtual environment

When we use virtual and augmented reality (VR/AR) environments to investigate behaviour or train motor skills, we expect that the insights or skills acquired in VR/AR transfer to real-world settings. Motor behaviour is strongly influenced by perceptual uncertainty and the expected consequences of actions. VR/AR differ in both of these aspects from natural environments. Perceptual information in VR/AR is less reliable than in natural environments, and the knowledge of acting in a virtual environment might modulate our expectations of action consequences. Using mirror reflections to create a virtual environment free of perceptual artefacts, we show that hand movements in an obstacle avoidance task systematically differed between real and virtual obstacles and that these behavioural differences occurred independent of the quality of the available perceptual information. This suggests that even when perceptual correspondence between natural and virtual environments is achieved, action correspondence does not necessarily follow due to the disparity in the expected consequences of actions in the two environments.

Some binocular advantages for planning reach, but not grasp, components of prehension

Proficient (fast, accurate, precise) hand actions for reaching-to-grasp 3D objects are known to benefit significantly from the use of binocular vision compared to one eye alone. We examined whether these binocular advantages derive from increased reliability in encoding the goal object’s properties for feedforward planning of prehension movements or from enhanced feedback mediating their online control. Adult participants reached for, precision grasped and lifted cylindrical table-top objects (two sizes, 2 distances) using binocular vision or only their dominant/sighting eye or their non-dominant eye to program and fully execute their movements or using each of the three viewing conditions only to plan their reach-to-grasp during a 1 s preview, with vision occluded just before movement onset. Various kinematic measures of reaching and grasping proficiency, including corrective error rates, were quantified and compared by view, feedback and object type. Some significant benefits of binocular over monocular vision when they were just available for pre-movement planning were retained for the reach regardless of target distance, including higher peak velocities, straighter paths and shorter low velocity approach times, although these latter were contaminated by more velocity corrections and by poorer coordination with object contact. By contrast, virtually all binocular advantages for grasping, including improvements in peak grip aperture scaling, the accuracy and precision of digit placements at object contact and shorter grip application times preceding the lift, were eliminated with no feedback available, outcomes that were influenced by the object’s size. We argue that vergence cues can improve the reliability of binocular internal representations of object distance for the feedforward programming of hand transport, whereas the major benefits of binocular vision for enhancing grasping performance derive exclusively from its continuous presence online.

A margin for error in grasping: hand pre-shaping takes into account task-dependent changes in the probability of errors

Ideal grasping movements should maintain an appropriate probability of success, while controlling movement-related costs, in the presence of varying visual (and motor) uncertainty. It is often assumed that the probability of errors is managed by adjusting a margin for error in hand opening (e.g., opening the hand wider with increased visual uncertainty). This idea is intuitive, but non-trivial. It implies not only that the brain can estimate the amount of uncertainty, but also that it can compute how different possible alterations to the movement will affect the probability of errors—which we term the ‘probability landscape’. Previous work suggests the amount of uncertainty is factored into grasping movements. Our aim was to determine whether grasping movements are also sensitive to the probability landscape. Subjects completed three different grasping tasks, with naturally different probability landscapes, such that appropriate margin-for-error responses to increased uncertainty were qualitatively different (opening the hand wider, the same amount, or less wide). We increased visual uncertainty by blurring vision, and by covering one eye. Movements were performed without visual feedback to isolate uncertainty in the brain’s initial estimate of object properties. Changes to hand opening in response to increased visual uncertainty closely resembled those predicted by the margin-for-error account, suggesting that grasping is sensitive to the probability landscape associated with different tasks. Our findings therefore support the intuitive idea that grasping movements employ a true margin-for-error mechanism, which exerts active control over the probability of errors across changing circumstances.

Multiple distance cues do not prevent systematic biases in reach to grasp movements

The perceived distance of objects is biased depending on the distance from the observer at which objects are presented, such that the egocentric distance tends to be overestimated for closer objects, but underestimated for objects further away. This leads to the perceived depth of an object (i.e., the perceived distance from the front to the back of the object) also being biased, decreasing with object distance. Several studies have found the same pattern of biases in grasping tasks. However, in most of those studies, object distance and depth were solely specified by ocular vergence and binocular disparities. Here we asked whether grasping objects viewed from above would eliminate distance-dependent depth biases, since this vantage point introduces additional information about the object's distance, given by the vertical gaze angle, and its depth, given by contour information. Participants grasped objects presented at different distances (1) at eye-height and (2) 130 mm below eye-height, along their depth axes. In both cases, grip aperture was systematically biased by the object distance along most of the trajectory. The same bias was found whether the objects were seen in isolation or above a ground plane to provide additional depth cues. In two additional experiments, we verified that a consistent bias occurs in a perceptual task. These findings suggest that grasping actions are not immune to biases typically found in perceptual tasks, even when additional cues are available. However, online visual control can counteract these biases when direct vision of both digits and final contact points is available.

Viewing geometry determines the contribution of binocular vision to the online control of grasping

Binocular vision is often assumed to make a specific, critical contribution to online visual control of grasping by providing precise information about the separation between digits and object. This account overlooks the ‘viewing geometry’ typically encountered in grasping, however. Separation of hand and object is rarely aligned precisely with the line of sight (the visual depth dimension), and analysis of the raw signals suggests that, for most other viewing angles, binocular feedback is less precise than monocular feedback. Thus, online grasp control relying selectively on binocular feedback would not be robust to natural changes in viewing geometry. Alternatively, sensory integration theory suggests that different signals contribute according to their relative precision, in which case the role of binocular feedback should depend on viewing geometry, rather than being ‘hard-wired’. We manipulated viewing geometry, and assessed the role of binocular feedback by measuring the effects on grasping of occluding one eye at movement onset. Loss of binocular feedback resulted in a significantly less extended final slow-movement phase when hand and object were separated primarily in the frontoparallel plane (where binocular information is relatively imprecise), compared to when they were separated primarily along the line of sight (where binocular information is relatively precise). Consistent with sensory integration theory, this suggests the role of binocular (and monocular) vision in online grasp control is not a fixed, ‘architectural’ property of the visuo-motor system, but arises instead from the interaction of viewer and situation, allowing robust online control across natural variations in viewing geometry.

Dressmakers show enhanced stereoscopic vision

The ability to estimate the distance of objects from one’s self and from each other is fundamental to a variety of behaviours from grasping objects to navigating. The main cue to distance, stereopsis, relies on the slight offsets between the images derived from our left and right eyes, also termed disparities. Here we ask whether the precision of stereopsis varies with professional experience with precise manual tasks. We measured stereo-acuities of dressmakers and non-dressmakers for both absolute and relative disparities. We used a stereoscope and a computerized test removing monocular cues. We also measured vergence noise and bias using the Nonius line technique. We demonstrate that dressmakers’ stereoscopic acuities are better than those of non-dressmakers, for both absolute and relative disparities. In contrast, vergence noise and bias were comparable in the two groups. Two non-exclusive mechanisms may be at the source of the group difference we document: (i) self-selection or the fact that stereo-vision is functionally important to become a dressmaker, and (ii) plasticity, or the fact that training on demanding stereovision tasks improves stereo-acuity.

Magnitude, precision, and realism of depth perception in stereoscopic vision

Our perception of depth is substantially enhanced by the fact that we have binocular vision. This provides us with more precise and accurate estimates of depth and an improved qualitative appreciation of the three-dimensional (3D) shapes and positions of objects. We assessed the link between these quantitative and qualitative aspects of 3D vision. Specifically, we wished to determine whether the realism of apparent depth from binocular cues is associated with the magnitude or precision of perceived depth and the degree of binocular fusion. We presented participants with stereograms containing randomly positioned circles and measured how the magnitude, realism, and precision of depth perception varied with the size of the disparities presented. We found that as the size of the disparity increased, the magnitude of perceived depth increased, while the precision with which observers could make depth discrimination judgments decreased. Beyond an initial increase, depth realism decreased with increasing disparity magnitude. This decrease occurred well below the disparity limit required to ensure comfortable viewing.

Two Visual Pathways in Primates Based on Sampling of Space: Exploitation and Exploration of Visual Information

Evidence is strong that the visual pathway is segregated into two distinct streams—ventral and dorsal. Two proposals theorize that the pathways are segregated in function: The ventral stream processes information about object identity, whereas the dorsal stream, according to one model, processes information about either object location, and according to another, is responsible in executing movements under visual control. The models are influential; however recent experimental evidence challenges them, e.g., the ventral stream is not solely responsible for object recognition; conversely, its function is not strictly limited to object vision; the dorsal stream is not responsible by itself for spatial vision or visuomotor control; conversely, its function extends beyond vision or visuomotor control. In their place, we suggest a robust dichotomy consisting of a ventral stream selectively sampling high-resolution/focal spaces, and a dorsal stream sampling nearly all of space with reduced foveal bias. The proposal hews closely to the theme of embodied cognition: Function arises as a consequence of an extant sensory underpinning. A continuous, not sharp, segregation based on function emerges, and carries with it an undercurrent of an exploitation-exploration dichotomy. Under this interpretation, cells of the ventral stream, which individually have more punctate receptive fields that generally include the fovea or parafovea, provide detailed information about object shapes and features and lead to the systematic exploitation of said information; cells of the dorsal stream, which individually have large receptive fields, contribute to visuospatial perception, provide information about the presence/absence of salient objects and their locations for novel exploration and subsequent exploitation by the ventral stream or, under certain conditions, the dorsal stream. We leverage the dichotomy to unify neuropsychological cases under a common umbrella, account for the increased prevalence of multisensory integration in the dorsal stream under a Bayesian framework, predict conditions under which object recognition utilizes the ventral or dorsal stream, and explain why cells of the dorsal stream drive sensorimotor control and motion processing and have poorer feature selectivity. Finally, the model speculates on a dynamic interaction between the two streams that underscores a unified, seamless perception. Existing theories are subsumed under our proposal.

Binocular advantage for prehension movements performed in visually enriched environments requiring visual search

The purpose of this study was to examine the role of binocular vision during a prehension task performed in a visually enriched environment where the target object was surrounded by distractors/obstacles. Fifteen adults reached and grasped for a cylindrical peg while eye movements and upper limb kinematics were recorded. The complexity of the visual environment was manipulated by varying the number of distractors and by varying the saliency of the target. Gaze behavior (i.e., the latency of the primary gaze shift and frequency of gaze shifts prior to reach initiation) was comparable between viewing conditions. In contrast, a binocular advantage was evident in performance accuracy. Specifically, participants picked up the wrong object twice as often during monocular viewing when the complexity of the environment increased. Reach performance was more efficient during binocular viewing, which was demonstrated by shorter reach reaction time and overall movement time. Reaching movements during the approach phase had higher peak velocity during binocular viewing. During monocular viewing reach trajectories exhibited a direction bias during the acceleration phase, which was leftward during left eye viewing and rightward during right eye viewing. This bias can be explained by the presence of esophoria in the covered eye. The grasping interval was also extended by ~20% during monocular viewing; however, the duration of the return phase after the target was picked up was comparable across viewing conditions. In conclusion, binocular vision provides important input for planning and execution of prehension movements in visually enriched environments. Binocular advantage was evident, regardless of set size or target saliency, indicating that adults plan their movements more cautiously during monocular viewing, even in relatively simple environments with a highly salient target. Nevertheless, in visually-normal adults monocular input provides sufficient information to engage in online control to correct the initial errors in movement planning.

Visual-haptic integration with pliers and tongs: signal "weights" take account of changes in haptic sensitivity caused by different tools

When we hold an object while looking at it, estimates from visual and haptic cues to size are combined in a statistically optimal fashion, whereby the “weight” given to each signal reflects their relative reliabilities. This allows object properties to be estimated more precisely than would otherwise be possible. Tools such as pliers and tongs systematically perturb the mapping between object size and the hand opening. This could complicate visual-haptic integration because it may alter the reliability of the haptic signal, thereby disrupting the determination of appropriate signal weights. To investigate this we first measured the reliability of haptic size estimates made with virtual pliers-like tools (created using a stereoscopic display and force-feedback robots) with different “gains” between hand opening and object size. Haptic reliability in tool use was straightforwardly determined by a combination of sensitivity to changes in hand opening and the effects of tool geometry. The precise pattern of sensitivity to hand opening, which violated Weber's law, meant that haptic reliability changed with tool gain. We then examined whether the visuo-motor system accounts for these reliability changes. We measured the weight given to visual and haptic stimuli when both were available, again with different tool gains, by measuring the perceived size of stimuli in which visual and haptic sizes were varied independently. The weight given to each sensory cue changed with tool gain in a manner that closely resembled the predictions of optimal sensory integration. The results are consistent with the idea that different tool geometries are modeled by the brain, allowing it to calculate not only the distal properties of objects felt with tools, but also the certainty with which those properties are known. These findings highlight the flexibility of human sensory integration and tool-use, and potentially provide an approach for optimizing the design of visual-haptic devices.

Binocular vision enhances a rapidly evolving affordance priming effect: behavioural and TMS evidence

Extensive research has suggested that simply viewing an object can automatically prime compatible actions for object manipulation, known as affordances. Here we explored the generation of covert motor plans afforded by real objects with precision ('pinchable') or whole-hand/power ('graspable') grip significance under different types of vision. In Experiment 1, participants viewed real object primes either monocularly or binocularly and responded to orthogonal auditory stimuli by making precision or power grips. Pinchable primes facilitated congruent precision grip responses relative to incongruent power grips, and vice versa for graspable primes, but only in the binocular vision condition. To examine the temporal evolution of the binocular affordance effect, participants in Experiment 2 always viewed the objects binocularly but made no responses, instead receiving a transcranial magnetic stimulation pulse over their primary motor cortex at three different times (150, 300, 450ms) after prime onset. Motor evoked potentials (MEPs) recorded from a pinching muscle were selectively increased when subjects were primed with a pinchable object, whereas MEPs from a muscle associated with power grips were increased when viewing graspable stimuli. This interaction was obtained both 300 and 450ms (but not 150ms) after the visual onset of the prime, characterising for the first time the rapid development of binocular grip-specific affordances predicted by functional accounts of the affordance effect.

The optic chiasm: a turning point in the evolution of eye/hand coordination

The primate visual system has a uniquely high proportion of ipsilateral retinal projections, retinal ganglial cells that do not cross the midline in the optic chiasm. The general assumption is that this developed due to the selective advantage of accurate depth perception through stereopsis. Here, the hypothesis that the need for accurate eye-forelimb coordination substantially influenced the evolution of the primate visual system is presented. Evolutionary processes may change the direction of retinal ganglial cells. Crossing, or non-crossing, in the optic chiasm determines which hemisphere receives visual feedback in reaching tasks. Each hemisphere receives little tactile and proprioceptive information about the ipsilateral hand. The eye-forelimb hypothesis proposes that abundant ipsilateral retinal projections developed in the primate brain to synthesize, in a single hemisphere, visual, tactile, proprioceptive, and motor information about a given hand, and that this improved eye-hand coordination and optimized the size of the brain. If accurate eye-hand coordination was a major factor in the evolution of stereopsis, stereopsis is likely to be highly developed for activity in the area where the hands most often operate.The primate visual system is ideally suited for tasks within arm's length and in the inferior visual field, where most manual activity takes place. Altering of ocular dominance in reaching tasks, reduced cross-modal cuing effects when arms are crossed, response of neurons in the primary motor cortex to viewed actions of a hand, multimodal neuron response to tactile as well as visual events, and extensive use of multimodal sensory information in reaching maneuvers support the premise that benefits of accurate limb control influenced the evolution of the primate visual system. The eye-forelimb hypothesis implies that evolutionary change toward hemidecussation in the optic chiasm provided parsimonious neural pathways in animals developing frontal vision and visually guided forelimbs, and also suggests a new perspective on vision convergence in prey and predatory animals.

The binocular advantage in visuomotor tasks involving tools

We compared performance on three manual-dexterity tasks under monocular and binocular viewing. The tasks were the standard Morrisby Fine Dexterity Test, using forceps to manipulate the items, a modified version of the Morrisby test using fingers, and a "buzz-wire" task in which subjects had to guide a wire hoop around a 3D track without bringing the hoop into contact with the track. In all three tasks, performance was better for binocular viewing. The extent of the binocular advantage in individuals did not correlate significantly with their stereoacuity measured on the Randot test. However, the extent of the binocular advantage depended strongly on the task. It was weak when fingers were used on the Morrisby task, stronger with forceps, and extremely strong on the buzz-wire task (fivefold increase in error rate with monocular viewing). We suggest that the 3D buzz-wire game is particularly suitable for assessing binocularly based dexterity.

A right hemisphere dominance for bimanual grasps

To find points on the surface of an object that ensure a stable grasp, it would be most effective to employ one area in one cortical hemisphere. But grasping the object with both hands requires control through both hemispheres. To better understand the control mechanisms underlying this "bimanual grasping", here we examined how the two hemispheres coordinate their control processes for bimanual grasping depending on visual field. We asked if bimanual grasping involves both visual fields equally or one more than the other. To test this, participants fixated either to the left or right of an object and then grasped or pushed it off a pedestal. We found that when participants grasped the object in the right visual field, maximum grip aperture (MGA) was larger and more variable, and participants were slower to react and to show MGA compared to when they grasped the object in the left visual field. In contrast, when participants pushed the object we observed no comparable visual field effects. These results suggest that grasping with both hands, specifically the computation of grasp points on the object, predominantly involves the right hemisphere. Our study provides new insights into the interactions of the two hemispheres for grasping.