Adaptation to the reversal of binocular depth cues: effects of wearing left-right reversing spectacles on stereoscopic depth perception

How small changes to one eye's retinal image can transform the perceived shape of a very familiar object

Vision can provide useful cues about the geometric properties of an object, like its size, distance, pose, and shape. But how the brain merges these properties into a complete sensory representation of a three-dimensional object is poorly understood. To address this gap, we investigated a visual illusion in which humans misperceive the shape of an object due to a small change in one eye's retinal image. We first show that this illusion affects percepts of a highly familiar object under completely natural viewing conditions. Specifically, people perceived their own rectangular mobile phone to have a trapezoidal shape. We then investigate the perceptual underpinnings of this illusion by asking people to report both the perceived shape and pose of controlled stimuli. Our results suggest that the shape illusion results from distorted cues to object pose. In addition to yielding insights into object perception, this work informs our understanding of how the brain combines information from multiple visual cues in natural settings. The shape illusion can occur when people wear everyday prescription spectacles; thus, these findings also provide insight into the cue combination challenges that some spectacle wearers experience on a regular basis.

Stereopsis provides a constant feed to visual shape representation

The contribution of stereopsis in human visual shape perception was examined using stimuli with either null, normal, or reversed binocular disparity in an old/new object recognition task. The highest levels of recognition performance were observed with null and normal binocular disparity displays, which did not differ. However, reversed disparity led to significantly worse performance than either of the other display conditions. This indicates that stereopsis provides a continuous input to the mechanisms involved in shape perception.

Shape judgments in natural scenes: Convexity biases versus stereopsis

Determining the relief of upcoming terrain is critical to locomotion over rough or uneven ground. Given the significant contribution of stereopsis to perceived surface shape, it should play a crucial role in determining the shape of ground surfaces. The aim of this series of experiments was to evaluate the relative contribution of monocular and binocular depth cues to judgments of ground relief. To accomplish this goal, we simulated a depth discrimination task using naturalistic imagery. Stimuli consisted of a stereoscopically rendered grassy terrain with a central mound or a dip with varying height. We measured thresholds for discrimination of the direction of the depth offset. To determine the relationship between relief discrimination and measures of stereopsis, we used two stereoacuity tasks performed under the same viewing conditions. To assess the impact of ambiguous two-dimensional shading cues on depth judgments in our terrain task, we manipulated the intensity of the shading (low and high). Our results show that observers reliably discriminated ground reliefs as small as 20 cm at a viewing distance of 9.1 m. As the shading was intensified, a large proportion of observers (30%) exhibited a strong convexity bias, even when stereopsis indicated a concave depression. This finding suggests that there are significant individual differences in the reliance on assumptions of surface curvature that must be considered in experimental conditions. In impoverished viewing environments with limiting depth cues, these convexity biases could persist in judgments of ground relief, especially when shading cues are highly salient.

Coming of Age in Science: Just Look?

With Professor Patrick Cavanagh, I started the Harvard Vision Sciences Laboratory in 1990. Blessed with the largesse of a wealthy university, we occupied a very large common space. Here, students pursued their own projects in a uniquely cooperative and exciting scientific environment. The times were just right in the emerging and expanding field of vision science. With good thesis projects under their belt, most of the students went on to successful careers. However, my own coming of age in science did not have such promising start. It only started well into my thirties when I joined the Smith Kettlewell Eye Research Institute in San Francisco. Providentially, it was there that I had the rare and unique opportunity to work closely and essentially only with peers (not students). Through these intense collaborations, I found my way as a scientist. Most of this account describes these formative years. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

The stereoscopic advantage for vection persists despite reversed disparity

Research has shown that consistent stereoscopic information improves the vection (i.e. illusions of self-motion) induced in stationary observers. This study investigates the effects of placing stereoscopic information into direct conflict with monocular motion signals by swapping the observer's left and right eye views to reverse disparity. Experiments compared the vection induced by stereo-consistent, stereo-reversed and flat-stereo patterns of: (1) same-size optic flow, which contained monocular motion perspective information about self-motion, and (2) changing-size optic flow, which provided additional monocular information about motion-in-depth based on local changes in object image sizes. As expected, consistent stereoscopic information improved the vection-in-depth induced by both changing-size and same-size patterns of optic flow. Unexpectedly, stereo-reversed patterns of same-size optic flow also induced stronger vection-in-depth than flat-stereo patterns of same-size optic flow. The effects of stereo-consistent and stereo-reversed information on vection strength were found to correlate reliably with their effects on perceived motion-in-depth and motion after-effect durations, but not with their effects on perceived scene depth. This suggests that stereo-consistent and stereo-reversed advantages for vection were both due to effects on perceived motion-in-depth. The current findings clearly demonstrate that stereoscopic information does not need to be consistent with monocular motion signals in order to improve vection. When taken together with past findings, they suggest that stereoscopic information only needs to be dynamic (as opposed to static) in order to improve vection-in-depth.

Stereoscopic 3D geometric distortions analyzed from the viewer's point of view

Stereoscopic 3D (S3D) geometric distortions can be introduced by mismatches among image capture, display, and viewing configurations. In previous work of S3D geometric models, geometric distortions have been analyzed from a third-person perspective based on the binocular depth cue (i.e., binocular disparity). A third-person perspective is different from what the viewer sees since monocular depth cues (e.g., linear perspective, occlusion, and shadows) from different perspectives are different. However, depth perception in a 3D space involves both monocular and binocular depth cues. Geometric distortions that are solely predicted by the binocular depth cue cannot describe what a viewer really perceives. In this paper, we combine geometric models and retinal disparity models to analyze geometric distortions from the viewer's perspective where both monocular and binocular depth cues are considered. Results show that binocular and monocular depth-cue conflicts in a geometrically distorted S3D space. Moreover, user-initiated head translations averting from the optimal viewing position in conventional S3D displays can also introduce geometric distortions, which are inconsistent with our natural 3D viewing condition. The inconsistency of depth cues in a dynamic scene may be a source of visually induced motions sickness.

Tomographic optical imaging of cortical responses after crossing nerve transfer in mice

To understand the neural mechanisms underlying the therapeutic effects of crossing nerve transfer for brachial plexus injuries in human patients, we investigated the cortical responses after crossing nerve transfer in mice using conventional and tomographic optical imaging. The distal cut ends of the left median and ulnar nerves were connected to the central cut ends of the right median and ulnar nerves with a sciatic nerve graft at 8 weeks of age. Eight weeks after the operation, the responses in the primary somatosensory cortex (S1) elicited by vibratory stimulation applied to the left forepaw were visualized based on activity-dependent flavoprotein fluorescence changes. In untreated mice, the cortical responses to left forepaw stimulation were mainly observed in the right S1. In mice with nerve crossing transfer, cortical responses to left forepaw stimulation were observed in the left S1 together with clear cortical responses in the right S1. We expected that the right S1 responses in the untreated mice were produced by thalamic inputs to layer IV, whereas those in the operated mice were mediated by callosal inputs from the left S1 to layer II/III of the right S1. To confirm this hypothesis, we performed tomographic imaging of flavoprotein fluorescence responses by macroconfocal microscopy. Flavoprotein fluorescence responses in layer IV were dominant compared to those in layer II/III in untreated mice. In contrast, responses in layer II/III were dominant compared to those in layer IV in operated mice. The peak latency of the cortical responses in the operated mice was longer than that in the untreated mice. These results confirmed our expectation that drastic reorganization in the cortical circuits was induced after crossing nerve transfer in mice.

The Nature and Timing of Tele-Pseudoscopic Experiences

Interchanging the left and right eye views of a scene (pseudoscopic viewing) has been reported to produce vivid stereoscopic effects under certain conditions. In two separate field studies, we examined the experiences of 124 observers (76 in Study 1 and 48 in Study 2) while pseudoscopically viewing a distant natural outdoor scene. We found large individual differences in both the nature and the timing of their pseudoscopic experiences. While some observers failed to notice anything unusual about the pseudoscopic scene, most experienced multiple pseudoscopic phenomena, including apparent scene depth reversals, apparent object shape reversals, apparent size and flatness changes, apparent reversals of border ownership, and even complex illusory foreground surfaces. When multiple effects were experienced, patterns of co-occurrence suggested possible causal relationships between apparent scene depth reversals and several other pseudoscopic phenomena. The latency for experiencing pseudoscopic phenomena was found to correlate significantly with observer visual acuity, but not stereoacuity, in both studies.

Stereo and Shading Contribute Independently to Shape Convexity-Concavity Discrimination

The present study examined the joint contribution of shading and stereopsis to the perception of shape convexity–concavity. The stimuli were the images of a synthetic convex 3-D shape seen from viewpoints leading to ambiguity as to its convexity. Illumination came from either above or below, and from either the right or the left, and stimuli were presented dichoptically with normal binocular disparity, reversed disparity, or no disparity. Participants responded “convex” more often when the lighting came from above than from below. Also, participants responded that the shape was convex more often with normal than with zero disparity, and more often with zero disparity than with reversed stereopsis. The effects of lighting direction and display mode were additive—that is, they did not interact. This indicates that shading and stereopsis contribute independently to shape perception.

Restoration of contralateral representation in the mouse somatosensory cortex after crossing nerve transfer

Avulsion of spinal nerve roots in the brachial plexus (BP) can be repaired by crossing nerve transfer via a nerve graft to connect injured nerve ends to the BP contralateral to the lesioned side. Sensory recovery in these patients suggests that the contralateral primary somatosensory cortex (S1) is activated by afferent inputs that bypassed to the contralateral BP. To confirm this hypothesis, the present study visualized cortical activity after crossing nerve transfer in mice through the use of transcranial flavoprotein fluorescence imaging. In naïve mice, vibratory stimuli applied to the forepaw elicited localized fluorescence responses in the S1 contralateral to the stimulated side, with almost no activity in the ipsilateral S1. Four weeks after crossing nerve transfer, forepaw stimulation in the injured and repaired side resulted in cortical responses only in the S1 ipsilateral to the stimulated side. At eight weeks after crossing nerve transfer, forepaw stimulation resulted in S1 cortical responses of both hemispheres. These cortical responses were abolished by cutting the nerve graft used for repair. Exposure of the ipsilateral S1 to blue laser light suppressed cortical responses in the ipsilateral S1, as well as in the contralateral S1, suggesting that ipsilateral responses propagated to the contralateral S1 via cortico-cortical pathways. Direct high-frequency stimulation of the ipsilateral S1 in combination with forepaw stimulation acutely induced S1 bilateral cortical representation of the forepaw area in naïve mice. Cortical responses in the contralateral S1 after crossing nerve transfer were reduced in cortex-restricted heterotypic GluN1 (NMDAR1) knockout mice. Functional bilateral cortical representation was not clearly observed in genetically manipulated mice with impaired cortico-cortical pathways between S1 of both hemispheres. Taken together, these findings strongly suggest that activity-dependent potentiation of cortico-cortical pathways has a critical role for sensory recovery in patients after crossing nerve transfer.

Consciousness, plasticity, and connectomics: the role of intersubjectivity in human cognition

Consciousness is typically construed as being explainable purely in terms of either private, raw feels or higher-order, reflective representations. In contrast to this false dichotomy, we propose a new view of consciousness as an interactive, plastic phenomenon open to sociocultural influence. We take up our account of consciousness from the observation of radical cortical neuroplasticity in human development. Accordingly, we draw upon recent research on macroscopic neural networks, including the "default mode," to illustrate cases in which an individual's particular "connectome" is shaped by encultured social practices that depend upon and influence phenomenal and reflective consciousness. On our account, the dynamically interacting connectivity of these networks bring about important individual differences in conscious experience and determine what is "present" in consciousness. Further, we argue that the organization of the brain into discrete anti-correlated networks supports the phenomenological distinction of prereflective and reflective consciousness, but we emphasize that this finding must be interpreted in light of the dynamic, category-resistant nature of consciousness. Our account motivates philosophical and empirical hypotheses regarding the appropriate time-scale and function of neuroplastic adaptation, the relation of high and low-frequency neural activity to consciousness and cognitive plasticity, and the role of ritual social practices in neural development and cognitive function.

Rapid Learning by Walking Observers Wearing a Reversing or Inverting Prism

We measured the initial rapid learning of walking observers who wore an up–down inverting or left–right reversing prism. This prism-walking version of the ‘mirror-drawing’ experiment revealed that the learning curve as a function of the trial number was the same as that typically acquired from a traditional mirror-drawing experiment. We suggest that the initial short-term learning process involved in prism walking is similar to that in mirror drawing and is related to the high-level decision-making process involved in visuo-motor planning of actions with feedback from transformed vision.

Long-lasting aftereffect of a single prism adaptation: shifts in vision and proprioception are independent

After a single adaptation session to prisms with gradually incremented shift magnitude, the prism adaptation aftereffect was measured by open loop mid-sagittal pointing (O) to a visual target without visual feedback. This aftereffect corresponded to the summation of the shift in proprioception, measured by straight ahead pointing without vision (S), and the visual straight ahead judgement (V), measured by verbal stopping of an LED moving from two opposite directions. However, the measurement of the aftereffects made over a period of 7 days revealed significantly different decay curves in V, O and S. Surprisingly the S shift was still present up to 7 days after the training, while V had returned to the original level by 2 h, which was the first measurement after subjects returned to a normal visual environment. O had returned to pre-test level after 1 day. After 3 days Wilkinson's (J Exp Psychol 89:250-257, 1971) additive hypothesis (O=S-V) no longer fit the data. Rather "O=Pl-V", where Pl (Pr) is the shift in proprioception measured by passive lateral arm movements from left (right), fitted better during the whole 7 days of aftereffect in our study. Therefore, the aftereffect of our strong prism adaptation revealed, firstly, that classical open loop pointing consisted of aftereffect shifts equal to the summation of the shifts in the two passively measurable aftereffect components, vision (V) and proprioception (Pl), rather than with active straight ahead pointing (S). Secondly, the decay of the shift in visual perception and in passively measurable proprioception is independent. The former decays fast, and the latter decays slowly with two separate waves. Thirdly, we suggest that the use of visual perception-dependent spatial codes for visual-manual transformation and the vision-independent internal egocentric reference frame are mutually exclusive. We proposed a model to explain these possible mechanisms.

Manual control of the visual stimulus reduces the flash-lag effect

We investigated how observers' control of the stimulus change affects temporal aspects of visual perception. We compared the flash-lag effects for motion (Experiment 1) and for luminance (Experiment 2) under several conditions that differed in the degree of the observers' control of change in a stimulus. The flash-lag effect was salient if the observers passively viewed the automatic change in the stimulus. However, if the observers controlled the stimulus change by a computer-mouse, the flash-lag effect was significantly reduced. In Experiment 3, we examined how observers' control of the stimulus movement by a mouse affects the reaction time for the shape change in the moving stimulus and flash. Results showed that the control reduced the reaction time for both moving stimulus and flash. These results suggest that observers' manual control of the stimulus reduces the flash-lag effect in terms of facilitation in visual processing.

Disparity-Tuning Characteristics of Neuronal Responses to Dynamic Random-Dot Stereograms in Macaque Visual Area V4

Stereo processing begins in the striate cortex and involves several extrastriate visual areas. We quantitatively analyzed the disparity-tuning characteristics of neurons in area V4 of awake, fixating monkeys. Approximately half of the analyzed V4 cells were tuned for horizontal binocular disparities embedded in dynamic random-dot stereograms (RDSs). Their response preferences were strongly biased for crossed disparities. To characterize the disparity-tuning profile, we fitted a Gabor function to the disparity-tuning data. The distribution of V4 cells showed a single dense cluster in a joint parameter space of the center and the phase parameters of the fitted Gabor function; most V4 neurons were maximally sensitive to fine stereoscopic depth increments near zero disparity. Comparing single-cell responses with background multiunit responses at the same sites showed that disparity-sensitive cells were clustered within V4 and that nearby cells possessed similar preferred disparities. Consistent with a recent report by Hegdé and Van Essen, the disparity tuning for an RDS drastically differed from that for a solid-figure stereogram (SFS). Disparity-tuning curves were generally broader for SFSs than for RDSs, and there was no correlation between the fitted Gabor functions' amplitudes, widths, or peaks for the two types of stereograms. The differences were partially attributable to shifts in the monocular images of an SFS. Our results suggest that the representation of stereoscopic depth in V4 is suited for detecting fine structural features protruding from a background. The representation is not generic and differs when the stimulus is broad-band noise or a solid figure.

Modification of depth and distance perception caused by long-term wearing of left-right reversing spectacles

For 35 to 39 days, four observers wore continuously left-right reversing spectacles which pseudoscopically reverse the order of binocular disparity and direction of convergence. In three tests, we investigated how the visual system copes with the transformation of depth and distance information due to the reversing spectacles. In stereogram observation, after a few days of wearing the spectacles. the observers sometimes perceived a depth order which was opposite to the depth order that they had perceived in the pre-spectacle-wearing period. Monocular depth cues contributed more to depth perception in the spectacle-wearing period than they did in the pre-spectacle-wearing period. While the perceived distance significantly decreased during the spectacle-wearing period, we found no evidence of adaptive change in distance perception. The results indicate that the visual system adapts itself to the transformed situation by not only changing the processing of disparity but also by changing the relative efficiency of each cue in determining apparent depth.

Global plasticity in adult visual cortex following reversal of visual input

The reversal or displacement of the retinal image by prism spectacles leads to extreme disruption of visually guided behaviour, but after an extended period of visual transformation normal behaviour is gradually restored. It is unclear whether this adaptation involves a change in visual perception, the learning of new motor responses, a modification of the sensori-motor control system or a proprioceptive change in the perceived positions of the body parts. Here I describe the effects of visual field reversal on neuronal activity in the monkey visual cortex. After a few months of wearing reversing spectacles, some cells in the primary visual cortex began to respond to stimuli presented not only in the contralateral visual field but also in the ipsilateral field. These cells were not selective for orientation or direction of motion, but responded well to a light flash. This result suggests that adaptation to visual field reversal is mediated, at least in part, by a large-scale functional reorganization at an early stage in the visual processing pathway.

How is motion disparity integrated with binocular disparity in depth perception?

Two experiments presented motion disparity conflicting with binocular disparity to examine how these cues determined apparent depth order (convex, concave) and depth magnitude. In each experiment, 8 subjects estimated the depth order and depth magnitude. The first experiment showed the following. (1) The visual system used one of these cues exclusively in selecting a depth order for each display. (2) The visual system integrated the depth magnitude information from these cues by a weighted additive fashion if it selected the binocular disparity in depth order perception and if the depth magnitude specified by motion disparity was small relative to that specified by binocular disparity. (3) The visual system ignored the depth magnitude information of binocular disparity if it selected the motion disparity in depth order perception. The second experiment showed that these three points were consistent whether the subject's head movement or object movement generated motion disparity.

How is depth perception affected by long-term wearing of left-right reversing spectacles?

The plasticity of binocular depth perception was investigated. Six subjects wore left-right reversing spectacles continuously for 10 or 11 days. On looking through the spectacles, the relation between the direction of physical depth (convex or concave) and the direction of binocular disparity (crossed or uncrossed) was reversed, but other depth cues did not change. When subjects observed stereograms through a haploscope and were asked to judge the direction of perceived depth, the directional relation between perceived depth and disparity was reversed both in the two line-contoured stereograms and in the random-dot stereogram in the middle of the wearing period, but the normal relation often returned late in the wearing period. When subjects observed two objects while wearing the spectacles and were asked which appeared the nearer, veridical depth perception increased as the wearing-time passed. These results indicate that the visual transformation reversing the direction of binocular disparity causes changes both in binocular stereopsis and in processes integrating different depth cues.

A stereoscopic view of visual processing streams

Recent anatomical and physiological studies of the visual pathway suggest the existence of at least three parallel processing streams in the lateral geniculate/primary cortex structure--a magno/interblob stream for motion and transient information; a parvo/interblob stream for high spatial frequency, static information; and a parvo/blob stream for chromatic and low spatial frequency information. How does this functional typology relate to the processing for stereoscopic depth? Human stereopsis may be viewed as consisting of three distinct types of disparity processing: coarse, local stereopsis suitable for stereomovement processing by the magno/interblob stream; fine, global stereopsis suitable for the processing of complex random-dot stereograms by the parvo/interblob stream; and simple, protostereopsis for processing size differences between the two eyes by the parvo/blob stream. Extensive psychophysical evidence supports the identification of these three disparity processes with the three processing streams.

Changes in oculocentric visual direction induced by the recalibration of saccades

The association between perceived visual direction and retinal location has been considered to be specified at birth and unmodifiable. We found that small changes occurred in the visual direction associated with a peripheral retinal location when the amplitudes of saccadic eye movements to a target imaged at that location were experimentally increased or decreased. This result indicates that at least limited plasticity exists in the perceptual representation of retinal location.

Pseudoscopic viewing: transfer and persistence of reversed depth relations from the viewing of photographs to the real scene

After viewing stereo pictures pseudoscopically (that is, with the pictures of the stereo-pair presented to the opposite eyes) part of the real scene depicted was perceived as inside out from the standpoint of depth during normal binocular vision. Experiments were designed to investigate this form of transfer but, since serious consequences can arise when an inside-out percept persists in normal binocular vision, it was intended that the experiments would be restricted to transfer of a less extreme kind. The scene chosen had not developed persistence during early trials but, surprisingly, one subject taking part in these experiments did experience some transfer from pseudoscopic direct viewing to normal binocular vision.