The role of uncertainty in the systematic spatial mislocalization of moving objects

Tuning curves vs. population responses, and perceptual consequences of receptive-field remapping

Sensory processing is often studied by examining how a given neuron responds to a parameterized set of stimuli (tuning curve) or how a given stimulus evokes responses from a parameterized set of neurons (population response). Although tuning curves and the corresponding population responses contain the same information, they can have different properties. These differences are known to be important because the perception of a stimulus should be decoded from its population response, not from any single tuning curve. The differences are less studied in the spatial domain where a cell's spatial tuning curve is simply its receptive field (RF) profile. Here, we focus on evaluating the common belief that perrisaccadic forward and convergent RF shifts lead to forward (translational) and convergent (compressive) perceptual mislocalization, respectively, and investigate the effects of three related factors: decoders' awareness of RF shifts, changes of cells' covering density near attentional locus (the saccade target), and attentional response modulation. We find that RF shifts alone produce either no shift or an opposite shift of the population responses depending on whether or not decoders are aware of the RF shifts. Thus, forward RF shifts do not predict forward mislocalization. However, convergent RF shifts change cells' covering density for aware decoders (but not for unaware decoders) which may predict convergent mislocalization. Finally, attentional modulation adds a convergent component to population responses for stimuli near the target. We simulate the combined effects of these factors and discuss the results with extant mislocalization data. We speculate that perisaccadic mislocalization might be the flash-lag effect unrelated to perisaccadic RF remapping but to resolve the issue, one has to address the question of whether or not perceptual decoders are aware of RF shifts.

The implicit sense of agency is not a perceptual effect but is a judgment effect

The sense of agency (SoA) is characterized as the sense of being the causal agent of one's own actions, and it is measured in two forms: explicit and implicit. In the explicit SoA experiments, the participants explicitly report whether they have a sense of control over their actions or whether they or somebody else is the causal agent of seen actions; the implicit SoA experiments study how do participants' agentive or voluntary actions modify perceptual processes (like time, vision, tactility, and audition) without directly asking the participants to explicitly think about their causal agency or sense of control. However, recent implicit SoA literature reported contradictory findings of the relationship between implicit SoA reports and agency states. Thus, I argue that the purported implicit SoA reports are not agency-driven perceptual effects per se but are judgment effects, by showing that (a) the typical operationalizations in implicit SoA domain lead to perceptual uncertainty on the part of the participants, (b) under uncertainty, participants' implicit SoA reports are due to heuristic judgments which are independent of agency states, and (c) under perceptual certainty, the typical implicit SoA reports might not have occurred at all. Thus, I conclude that the instances of implicit SoA are judgments (or response biases)-under uncertainty-rather than perceptual effects.

Motion-in-depth effects on interceptive timing errors in an immersive environment

We often need to interact with targets that move along arbitrary trajectories in the 3D scene. In these situations, information of parameters like speed, time-to-contact, or motion direction is required to solve a broad class of timing tasks (e.g., shooting, or interception). There is a large body of literature addressing how we estimate different parameters when objects move both in the fronto-parallel plane and in depth. However, we do not know to which extent the timing of interceptive actions is affected when motion-in-depth (MID) is involved. Unlike previous studies that have looked at the timing of interceptive actions using constant distances and fronto-parallel motion, we here use immersive virtual reality to look at how differences in the above-mentioned variables influence timing errors in a shooting task performed in a 3D environment. Participants had to shoot at targets that moved following different angles of approach with respect to the observer when those reached designated shooting locations. We recorded the shooting time, the temporal and spatial errors and the head's position and orientation in two conditions that differed in the interval between the shot and the interception of the target's path. Results show a consistent change in the temporal error across approaching angles: the larger the angle, the earlier the error. Interestingly, we also found different error patterns within a given angle that depended on whether participants tracked the whole target's trajectory or only its end-point. These differences had larger impact when the target moved in depth and are consistent with underestimating motion-in-depth in the periphery. We conclude that the strategy participants use to track the target's trajectory interacts with MID and affects timing performance.

A dynamic noise background reveals perceptual motion extrapolation: The twinkle-goes illusion

We find that on a dynamic noise background, the perceived disappearance location of a moving object is shifted in the direction of motion. This “twinkle-goes” illusion does not require luminance- or chromaticity-based confusability of the object with the background, or on the amount of background motion energy in the same direction as the object motion. This suggests that the illusion is enabled by the dynamic noise masking the offset transients that otherwise accompany an object's disappearance. While these results are consistent with an anticipatory process that pre-activates positions ahead of the object's current position, additional findings suggest an alternative account: a continuation of attentional tracking after the object disappears. First, the shift increased with speed until over 1.2 revolutions per second (rps), nearing the attentional tracking limit. Second, the shift was greatly reduced when attention was divided between two moving objects. Finally, the illusion was associated with a delay in simple reaction time to the disappearance of the object. We propose that in the absence of offset transients, attentional tracking keeps moving for several tens of milliseconds after the target disappearance, and this causes one to hallucinate a moving object at the position of attention.

The haptic and the visual flash-lag effect and the role of flash characteristics

When a short flash occurs in spatial alignment with a moving object, the moving object is seen ahead the stationary one. Similar to this visual "flash-lag effect" (FLE) it has been recently observed for the haptic sense that participants judge a moving hand to be ahead a stationary hand when judged at the moment of a short vibration ("haptic flash") that is applied when the two hands are spatially aligned. We further investigated the haptic FLE. First, we compared participants' performance in two isosensory visual or haptic conditions, in which moving object and flash were presented only in a single modality (visual: sphere and short color change, haptic: hand and vibration), and two bisensory conditions, in which the moving object was presented in both modalities (hand aligned with visible sphere), but the flash was presented only visually or only haptically. The experiment aimed to disentangle contributions of the flash's and the objects' modalities to the FLEs in haptics versus vision. We observed a FLE when the flash was visually displayed, both when the moving object was visual and visuo-haptic. Because the position of a visual flash, but not of an analogue haptic flash, is misjudged relative to a same visuo-haptic moving object, the difference between visual and haptic conditions can be fully attributed to characteristics of the flash. The second experiment confirmed that a haptic FLE can be observed depending on flash characteristics: the FLE increases with decreasing intensity of the flash (slightly modulated by flash duration), which had been previously observed for vision. These findings underline the high relevance of flash characteristics in different senses, and thus fit well with the temporal-sampling framework, where the flash triggers a high-level, supra-modal process of position judgement, the time point of which further depends on the processing time of the flash.

Perisaccadic visual perception

Primates use frequent, rapid eye movements to sample their visual environment. This is a fruitful strategy to make the best use of the highly sensitive foveal part of the retina, but it requires neural mechanisms to bind the rapidly changing visual input into a single, stable percept. Studies investigating these neural mechanisms have typically assumed that perisaccadic perception in nonhuman primates matches that of humans. We tested this assumption by performing identical experiments in human and nonhuman primates. Our data confirm that perisaccadic visual perception of macaques and humans is qualitatively similar. Specifically, we found a reduction in detectability and mislocalization of targets presented at the time of saccades. We also found substantial differences between human and nonhuman primates. Notably, in nonhuman primates, localization that requires knowledge of eye position was less precise, nonhuman primates detected fewer perisaccadic stimuli, and perisaccadic compression was not towards the saccade target. The qualitative similarities between species support the view that the nonhuman primate is ideally suited to study aspects of brain function—such as those relying on foveal vision—that are uniquely developed in primates. The quantitative differences, however, demonstrate the need for a reassessment of the models purportedly linking neural response changes at the time of saccades with the behavioral phenomena of perisaccadic reduction of detectability and mislocalization.

The buzz-lag effect

In the flash-lag illusion, a brief visual flash and a moving object presented at the same location appear to be offset with the flash trailing the moving object. A considerable amount of studies investigated the visual flash-lag effect, and flash-lag-like effects have also been observed in audition, and cross-modally between vision and audition. In the present study, we investigate whether a similar effect can also be observed when using only haptic stimuli. A fast vibration (or buzz, lasting less than 20 ms) was applied to the moving finger of the observers and employed as a "haptic flash." Participants performed a two-alternative forced-choice (2AFC) task where they had to judge whether the moving finger was located to the right or to the left of the stationary finger at the time of the buzz. We used two different movement velocities (Slow and Fast conditions). We found that the moving finger was systematically misperceived to be ahead of the stationary finger when the two were physically aligned. This result can be interpreted as a purely haptic analogue of the flash-lag effect, which we refer to as "buzz-lag effect." The buzz-lag effect can be well accounted for by the temporal-sampling explanation of flash-lag-like effects.

The Role of Temporal Information in Perisaccadic Mislocalization

In dynamic environments, it is crucial to accurately consider the timing of information. For instance, during saccades the eyes rotate so fast that even small temporal errors in relating retinal stimulation by flashed stimuli to extra-retinal information about the eyes' orientations will give rise to substantial errors in where the stimuli are judged to be. If spatial localization involves judging the eyes' orientations at the estimated time of the flash, we should be able to manipulate the pattern of mislocalization by altering the estimated time of the flash. We reasoned that if we presented a relevant flash within a short rapid sequence of irrelevant flashes, participants' estimates of when the relevant flash was presented might be shifted towards the centre of the sequence. In a first experiment, we presented five bars at different positions around the time of a saccade. Four of the bars were black. Either the second or the fourth bar in the sequence was red. The task was to localize the red bar. We found that when the red bar was presented second in the sequence, it was judged to be further in the direction of the saccade than when it was presented fourth in the sequence. Could this be because the red bar was processed faster when more black bars preceded it? In a second experiment, a red bar was either presented alone or followed by two black bars. When two black bars followed it, it was judged to be further in the direction of the saccade. We conclude that the spatial localization of flashed stimuli involves judging the eye orientation at the estimated time of the flash.

Does perisaccadic compression require foveal vision?

People make systematic errors when localizing a stimulus that is presented briefly near the time of a saccade. These errors have been interpreted as compression towards the position that is fixated after the saccade. Normally, fixating a position means that its image falls on the fovea. Macular degeneration (MD) damages the central retina, obliterating foveal vision. Many people with MD adopt a new retinal locus for fixation, called the preferred retinal locus (PRL). If the compression of space during the saccade is a special characteristic of the fovea, possibly due to the high density of cones that is found in the fovea, one might expect people lacking central vision to show no compression of space around the time of a saccade. If the compression of space during the saccade is related to the position that is fixated after the saccade, one would expect compression towards the PRL, despite the lack of a high density of cones in this area. We found that a person with MD showed a clear compression towards her PRL. We conclude that perisaccadic compression is related to the position that is fixated after the saccade rather than to the high density of receptors in the fovea.

Truncation of the flash-lag effect by a fixed spatial landmark

The flash-lag effect is a visual illusion where a moving image is perceived to be advanced in its spatial location relative to a flashed image. Multiple studies have shown that the flash-lag effect can be enhanced by increasing the uncertainty of the moving and/or flashed images. However, little is known about the effect of task-irrelevant visual objects on the flash-lag effect. We were interested to see whether a task-irrelevant spatial landmark might reduce uncertainty and hence reduce the flash-lag effect. We placed a fixed bar between moving and flashed bars while measuring the flash-lag effect in six participants. For most participants, the fixed bar substantially truncated the flash-lag effect. The effect was maximal when the fixed bar was aligned with the flashed bar and decreased when the fixed bar was positioned more peripherally. A second experiment with two participants used a smaller fixed bar; the smaller bar had less truncation effect in one participant, while the other participant showed similar truncation regardless of the fixed bar size. Our results support models that place the locus of the flash-lag effect in higher-order brain areas, e.g., the parietal lobe.

The role of differential delays in integrating transient visual and proprioceptive information

Many actions involve limb movements toward a target. Visual and proprioceptive estimates are available online, and by optimally combining (Ernst and Banks, 2002) both modalities during the movement, the system can increase the precision of the hand estimate. The notion that both sensory modalities are integrated is also motivated by the intuition that we do not consciously perceive any discrepancy between the felt and seen hand's positions. This coherence as a result of integration does not necessarily imply realignment between the two modalities (Smeets et al., 2006). For example, the two estimates (visual and proprioceptive) might be different without either of them (e.g., proprioception) ever being adjusted after recovering the other (e.g., vision). The implication that the felt and seen positions might be different has a temporal analog. Because the actual feedback from the hand at a given instantaneous position reaches brain areas at different times for proprioception and vision (shorter for proprioception), the corresponding instantaneous unisensory position estimates will be different, with the proprioceptive one being ahead of the visual one. Based on the assumption that the system integrates optimally and online the available evidence from both senses, we introduce a temporal mechanism that explains the reported overestimation of hand positions when vision is occluded for active and passive movements (Gritsenko et al., 2007) without the need to resort to initial feedforward estimates (Wolpert et al., 1995). We set up hypotheses to test the validity of the model, and we contrast simulation-based predictions with empirical data.

Spatiotemporal integration for tactile localization during arm movements: a probabilistic approach

It has been shown that people make systematic errors in the localization of a brief tactile stimulus that is delivered to the index finger while they are making an arm movement. Here we modeled these spatial errors with a probabilistic approach, assuming that they follow from temporal uncertainty about the occurrence of the stimulus. In the model, this temporal uncertainty converts into a spatial likelihood about the external stimulus location, depending on arm velocity. We tested the prediction of the model that the localization errors depend on arm velocity. Participants (n = 8) were instructed to localize a tactile stimulus that was presented to their index finger while they were making either slow- or fast-targeted arm movements. Our results confirm the model's prediction that participants make larger localization errors when making faster arm movements. The model, which was used to fit the errors for both slow and fast arm movements simultaneously, accounted very well for all the characteristics of these data with temporal uncertainty in stimulus processing as the only free parameter. We conclude that spatial errors in dynamic tactile perception stem from the temporal precision with which tactile inputs are processed.

Do the flash-lag effect and representational momentum involve similar extrapolations?

In the flash-lag effect (FLE) and in representational momentum (RM), the represented position of a moving target is displaced in the direction of motion. Effects of numerous variables on the FLE and on RM are briefly considered. In many cases, variables appear to have the same effect on the FLE and on RM, and this is consistent with a hypothesis that displacements in the FLE and in RM result from overlapping or similar mechanisms. In other cases, variables initially appear to have different effects on the FLE and on RM, but accounts reconciling those apparent differences with a hypothesis of overlapping or similar mechanisms are suggested. Given that RM is simpler and accounts for a wider range of findings (i.e., RM involves a single stimulus rather than the relationship between two stimuli, RM accounts for displacement in absolute position of a single stimulus and for differences in relative position of two stimuli), it is suggested that (at least some cases of) the FLE might be a special case of RM in which the position of the target is assessed relative to the position of another stimulus (i.e., the flashed object) rather than relative to the actual position of the target.

The perceived position of moving objects: transcranial magnetic stimulation of area MT+ reduces the flash-lag effect

How does the visual system assign the perceived position of a moving object? This question is surprisingly complex, since sluggish responses of photoreceptors and transmission delays along the visual pathway mean that visual cortex does not have immediate information about a moving object's position. In the flash-lag effect (FLE), a moving object is perceived ahead of an aligned flash. Psychophysical work on this illusion has inspired models for visual localization of moving objects. However, little is known about the underlying neural mechanisms. Here, we investigated the role of neural activity in areas MT+ and V1/V2 in localizing moving objects. Using short trains of repetitive Transcranial Magnetic Stimulation (TMS) or single pulses at different time points, we measured the influence of TMS on the perceived location of a moving object. We found that TMS delivered to MT+ significantly reduced the FLE; single pulse timings revealed a broad temporal tuning with maximum effect for TMS pulses, 200 ms after the flash. Stimulation of V1/V2 did not significantly influence perceived position. Our results demonstrate that area MT+ contributes to the perceptual localization of moving objects and is involved in the integration of position information over a long time window.

Testing the limits of optimal integration of visual and proprioceptive information of path trajectory

Many studies provide evidence that information from different modalities is integrated following the maximum likelihood estimation model (MLE). For instance, we recently found that visual and proprioceptive path trajectories are optimally combined (Reuschel et al. in Exp Brain Res 201:853-862, 2010). However, other studies have failed to reveal optimal integration of such dynamic information. In the present study, we aim to generalize our previous findings to different parts of the workspace (central, ipsilateral, or contralateral) and to different types of judgments (relative vs. absolute). Participants made relative judgments by judging whether an angular path was acute or obtuse, or they made absolute judgments by judging whether a one-segmented straight path was directed to left or right. Trajectories were presented in the visual, proprioceptive, or combined visual-proprioceptive modality. We measured the bias and the variance of these estimates and predicted both parameters using the MLE. In accordance with the MLE model, participants linearly combined and weighted the unimodal angular path information by their reliabilities irrespective of the side of workspace. However, the precision of bimodal estimates was not greater than that for unimodal estimates, which is inconsistent with the MLE. For the absolute judgment task, participants' estimates were highly accurate and did not differ across modalities. Thus, we were unable to test whether the bimodal percept resulted as a weighted average of the visual and proprioceptive input. Additionally, participants were not more precise in the bimodal compared with the unimodal conditions, which is inconsistent with the MLE. Current findings suggest that optimal integration of visual and proprioceptive information of path trajectory only applies in some conditions.

Compensation for changing motor uncertainty

When movement outcome differs consistently from the intended movement, errors are used to correct subsequent movements (e.g., adaptation to displacing prisms or force fields) by updating an internal model of motor and/or sensory systems. Here, we examine changes to an internal model of the motor system under changes in the variance structure of movement errors lacking an overall bias. We introduced a horizontal visuomotor perturbation to change the statistical distribution of movement errors anisotropically, while monetary gains/losses were awarded based on movement outcomes. We derive predictions for simulated movement planners, each differing in its internal model of the motor system. We find that humans optimally respond to the overall change in error magnitude, but ignore the anisotropy of the error distribution. Through comparison with simulated movement planners, we found that aimpoints corresponded quantitatively to an ideal movement planner that updates a strictly isotropic (circular) internal model of the error distribution. Aimpoints were planned in a manner that ignored the direction-dependence of error magnitudes, despite the continuous availability of unambiguous information regarding the anisotropic distribution of actual motor errors.

Motion extrapolation into the blind spot

The flash-lag effect, in which a moving object is perceived ahead of a colocalized flash, has led to keen empirical and theoretical debates. To test the proposal that a predictive mechanism overcomes neural delays in vision by shifting objects spatially, we asked observers to judge the final position of a bar moving into the retinal blind spot. The bar was perceived to disappear in positions well inside the unstimulated area. Given that photoreceptors are absent in the blind spot, the perceived shift must be based on the history of the moving object. Such predictive overshoots are suppressed when a moving object disappears abruptly from the retina, triggering retinal transient signals. No such transient-driven suppression occurs when the object disappears by virtue of moving into the blind spot. The extrapolated position of the moving bar revealed in this manner provides converging support for visual prediction.

Temporal information can influence spatial localization

To localize objects relative to ourselves, we need to combine various sensory and motor signals. When these signals change abruptly, as information about eye orientation does during saccades, small differences in latency between the signals could introduce localization errors. We examine whether independent temporal information can influence such errors. We asked participants to follow a randomly jumping dot with their eyes and to point at flashes that occurred near the time they made saccades. Such flashes are mislocalized. We presented a tone at different times relative to the flash. We found that the flash was mislocalized as if it had occurred closer in time to the tone. This demonstrates that temporal information is taken into consideration when combining sensory information streams for localization.

Effects of texture and shape on perceived time to passage: knowing "what" influences judging "when"

In the present study, we examined whether it is easier to judge when an object will pass one's head if the object's surface is textured. There are three reasons to suspect that this might be so: First, the additional (local) optic flow may help one judge the rate of expansion and the angular velocity more reliably. Second, local deformations related to the change in angle between the object and the observer could help track the object's position along its path. Third, more reliable judgments of the object's shape could help separate global expansioncaused by changes in distance from expansion due to changes in the angle between the object and the observer. We can distinguish among these three reasons by comparing performance for textured and uniform spheres and disks. Moving objects were displayed for 0.5-0.7 sec. Subjects had to decide whether the object would pass them before or after a beep that was presented 1 sec after the object started moving. Subjects were not more precise with textured objects. When the disk rotated in order to compensate for the orientation-related contraction that its image would otherwise undergo during its motion, it appeared to arrive later, despite the fact that this strategy increases the global rate of expansion. We argue that this is because the expected deformation of the object's image during its motion is considered when time to passage is judged. Therefore, the most important role for texture in everyday judgments of time to passage is probably that it helps one judge the object's shape and thereby estimate how its image will deform as it moves.

The perceived position of a moving object is not the result of position integration

The flash-lag effect is a robust visual illusion in which a flash appears to spatially lag a continuously moving stimulus, even though both stimuli are actually precisely aligned. Some research has been done to test how visual information has been integrated over time. The position integration model suggests motion integration is a form of interpolation of past positions, and predicts that we cannot perceive the reversal point at its actual position on the trajectory of a moving object which reverses abruptly. In current research, we demonstrate that subjects could perceive the reversal point accurately while the psychometric function measured by a flash does not pass through the actual turning point. These results do not support the position integration model. We propose that the flash-lag effect is more likely to be a temporal illusion.

Perisaccadic compression correlates with saccadic peak velocity: differential association of eye movement dynamics with perceptual mislocalization patterns

Objects flashed around the onset of a saccadic eye movement are grossly mislocalized. Perisaccadic mislocalization has been related to a spatiotemporal misalignment of an extraretinal eye position signal with the corresponding saccade. Two phenomena have been observed: a systematic shift of perceived positions in saccade direction and an additional compression toward the saccade target. At present, it is unclear whether these two components of mislocalization are mediated by distinct mechanisms and how extraretinal signals may contribute to either of them. Moreover, the pattern and strength of perisaccadic mislocalization varies considerably across studies and even between subjects tested under identical conditions. Here, we investigated whether interindividual differences in saccade parameters are related to differences in mislocalization. We found that the individual strength of perceptual compression selectively correlates with the peak velocity of corresponding saccades. Other saccade parameters did not correlate with compression. No correlation was found between the shift component of perisaccadic mislocalization and any saccade parameter. This dissociation suggests that shift and compression components are, at least partially, mediated by distinct mechanisms. Because neuronal activity in the superior colliculus and downstream oculomotor areas has been shown to correlate with saccadic peak velocity, our findings support the notion that a reafferent extraretinal signal associated with saccadic motor commands may contribute to perisaccadic compression of perceived positions.