Time gaps in mental imagery introduced by competing saccadic tasks

Internal coupling: Eye behavior coupled to visual imagery

Our eyes do not only respond to visual perception but also to internal cognition involving visual imagery, which can be referred to as internal coupling. This review synthesizes evidence on internal coupling across diverse domains including episodic memory and simulation, visuospatial memory, numerical cognition, object movement, body movement, and brightness imagery. In each domain, eye movements consistently reflect distinct aspects of mental imagery typically akin to those seen in corresponding visual experiences. Several findings further suggest that internal coupling may not only coincide with but also supports internal cognition as evidenced by improved cognitive performance. Available theoretical accounts suggest that internal coupling may serve at least two functional roles in visual imagery: facilitating memory reconstruction and indicating shifts in internal attention. Moreover, recent insights into the neurobiology of internal coupling highlight substantially shared neural pathways in externally and internally directed cognition. The review concludes by identifying open questions and promising avenues for future research such as exploring moderating roles of context and individual differences in internal coupling.

Prediction of time to contact under perceptual and contextual uncertainties

Accurately estimating time to contact (TTC) is crucial for successful interactions with moving objects, yet it is challenging under conditions of sensory and contextual uncertainty, such as occlusion. In this study, participants engaged in a prediction motion task, monitoring a target that moved rightward and an occluder. The participants' task was to press a key when they predicted the target would be aligned with the occluder's right edge. We manipulated sensory uncertainty by varying the visible and occluded periods of the target, thereby modulating the time available to integrate sensory information and the duration over which motion must be extrapolated. Additionally, contextual uncertainty was manipulated by having a predictable and unpredictable condition, meaning the occluder either reliably indicated where the moving target would disappear or provided no such indication. Results showed differences in accuracy between the predictable and unpredictable occluder conditions, with different eye movement patterns in each case. Importantly, the ratio of the time the target was visible, which allows for the integration of sensory information, to the occlusion time, which determines perceptual uncertainty, was a key factor in determining performance. This ratio is central to our proposed model, which provides a robust framework for understanding and predicting human performance in dynamic environments with varying degrees of uncertainty.

The influence of time structure on prediction motion in visual and auditory modalities

Usually people can estimate the correct position of a moving object even when it temporarily moves behind an occlusion. Studies have been performed on this type of occluded motion with prediction motion (PM) tasks in the laboratory. Previous publications have emphasized that people could use mental imagery or apply an oculomotor system to estimate the arrival of a moving stimulus at the target place. Nevertheless, these two ways cannot account for the performance difference under a different set of conditions. Our study tested the role of time structure in a time-to-collision (TTC) task using visual and auditory modalities. In the visual condition, the moving red bar travelled from left to right and was invisible during the entire course but flashed at the initial and the occluded points. The auditory condition and visual condition were alike, except that the flashes in the visual condition were changed to clicks at the initial and the occluded points. The results illustrated that participants' performance was better in the equal time structure condition. The comparison between the two sense modalities demonstrated a similar tendency, which suggested there could be common cognitive processes between visual and auditory modalities when participants took advantage of temporal cues to judge TTC.

Temporal context affects the perceived time of visual events

We investigated whether the moment at which an event is perceived depends on its temporal context. Participants learned a mapping between time and space by watching the hand of a clock rotating a full revolution in a fixed duration. Then the hand was removed, and a target disc was flashed within a fixed-interval duration. Participants were to indicate where the hand would have been at the time of the target. In three separate experiments, we estimated the disruption from a distractor disc that was presented before or after the target disc, with a variable time between them. The target was either revealed at the end of the trial or cued beforehand, and in the latter case, was cued by either color or temporal order. We found an attraction to the presentation time of the distractor when both events were attended equally (target revealed at the end). When the target was cued beforehand, the reported time was under- or overestimated, depending on whether the nature of distractor had to be decoded (precued by color) or not (precued by order). In summary, the perceived time of an event is always affected by other events in temporal proximity, but the nature of this effect depends on how each event is attended.

Intuitive physics of gravitational motion as shown by perceptual judgment and prediction-motion tasks

In Experiment 1, we explored participants' perceptual knowledge of vertical fall by presenting them with virtually simulated polystyrene or wooden spheres falling to the ground from about two meters high. Participants rated the perceived naturalness of the motion. Besides the implied mass of the sphere, we manipulated the motion pattern (i.e., uniform acceleration vs. uniform velocity), and the magnitude of acceleration or velocity. Results show that relatively low values of acceleration or velocity were judged as natural for the polystyrene sphere, whereas relatively high values of acceleration or velocity were judged as natural for the wooden sphere. In Experiment 2, the same stimuli of Experiment 1 were used, but the sphere disappeared behind an invisible occluder at some point of its trajectory. Participants were asked to predict the time-to-contact (TTC) of the sphere with the ground by pressing a key at the exact time of impact of the lower edge of the sphere with the floor of the room. Results show that the estimated TTC for the simulated wooden sphere was slightly but consistently smaller than the estimated TTC for the simulated polystyrene sphere. The influence of the implied mass on participants' responses might be the manifestation of two processes, namely an explicit 'heavy-fast, light-slow' heuristic, and/or an implicit, automatic association between mass and falling speed.

Motion prediction at low contrast

Accurate motion prediction is fundamental for survival. How does this reconcile with the well-known speed underestimation of low-contrast stimuli? Here we asked whether this contrast-dependent perceptual bias is retained in motion prediction under two different saccadic planning conditions: making a saccade to an occluded moving target, and real-time gaze interaction with multiple moving targets. In a first experiment, observers made a saccade to the mentally extrapolated position of a moving target (imagery condition). In a second experiment, observers had to prevent collisions among multiple moving targets by glancing at them through a gaze-contingent display or by hitting them with the touchpad cursor (interaction condition). In both experiments, target contrast was manipulated. We found that, whereas saccades to the imagined moving target were systematically biased by contrast, the gaze interaction performance, as measured by missed collisions, was generally unaffected - even though low-contrast targets looked slower. Interceptive actions increased at low contrast, but only when the gaze was used for interaction. Thus, perceptual speed underestimation transfers to saccades made to imagined low-contrast targets, without however necessarily being detrimental to effective performance when real-time interaction with multiple targets is required. This differential effect of stimulus contrast suggests that in complex dynamic conditions saccades are rather tolerant to visual speed biases.

Contribution of Visuospatial and Motion-Tracking to Invisible Motion

People experience an object's motion even when it is occluded. We investigate the processing of invisible motion in three experiments. Observers saw a moving circle passing behind an invisible, irregular hendecagonal polygon and had to respond as quickly as possible when the target had “just reappeared” from behind the occluder. Without explicit cues allowing the end of each of the eight hidden trajectories to be predicted (length ranging between 4.7 and 5 deg), we found as expected, if visuospatial attention was involved, anticipation errors, providing that information on pre-occluder motion was available. This indicates that the observers, rather than simply responding when they saw the target, tended to anticipate its reappearance (Experiment 1). The new finding is that, with a fixation mark indicating the center of the invisible trajectory, a linear relationship between the physical and judged occlusion duration is found, but not without it (Experiment 2) or with a fixation mark varying in position from trial to trial (Experiment 3). We interpret the role of central fixation in the differences in distinguishing trajectories smaller than 0.3 deg, by suggesting that it reflects spatiotemporal computation and motion-tracking. These two mechanisms allow visual imagery to form of the point symmetrical to that of the disappearance, with respect to fixation, and then for the occluded moving target to be tracked up to this point.

Covert enaction at work: Recording the continuous movements of visuospatial attention to visible or imagined targets by means of Steady-State Visual Evoked Potentials (SSVEPs)

Whereas overt visuospatial attention is customarily measured with eye tracking, covert attention is assessed by various methods. Here we exploited Steady-State Visual Evoked Potentials (SSVEPs) - the oscillatory responses of the visual cortex to incoming flickering stimuli - to record the movements of covert visuospatial attention in a way operatively similar to eye tracking (attention tracking), which allowed us to compare motion observation and motion extrapolation with and without eye movements. Observers fixated a central dot and covertly tracked a target oscillating horizontally and sinusoidally. In the background, the left and the right halves of the screen flickered at two different frequencies, generating two SSVEPs in occipital regions whose size varied reciprocally as observers attended to the moving target. The two signals were combined into a single quantity that was modulated at the target frequency in a quasi-sinusoidal way, often clearly visible in single trials. The modulation continued almost unchanged when the target was switched off and observers mentally extrapolated its motion in imagery, and also when observers pointed their finger at the moving target during covert tracking, or imagined doing so. The amplitude of modulation during covert tracking was ∼25-30% of that measured when observers followed the target with their eyes. We used 4 electrodes in parieto-occipital areas, but similar results were achieved with a single electrode in Oz. In a second experiment we tested ramp and step motion. During overt tracking, SSVEPs were remarkably accurate, showing both saccadic-like and smooth pursuit-like modulations of cortical responsiveness, although during covert tracking the modulation deteriorated. Covert tracking was better with sinusoidal motion than ramp motion, and better with moving targets than stationary ones. The clear modulation of cortical responsiveness recorded during both overt and covert tracking, identical for motion observation and motion extrapolation, suggests to include covert attention movements in enactive theories of mental imagery.

Probing the involvement of the earliest levels of cortical processing in motion extrapolation with rapid forms of visual motion priming and adaptation

In this study, we investigated the effect of brief motion priming and adaptation, occurring at the earliest levels of the cortical visual stream, on time-to-contact (TTC) estimation of a target passing behind an occluder. By using different exposure times of directional motion presented in the occluder area prior to the target's disappearance behind it, our aim was to modulate (prime or adapt) the extrapolated motion of the invisible target, thus producing different TTC estimates. Our results showed that longer (yet subsecond) exposures to motion in the same direction as the target produced late TTC estimates, whereas shorter exposures produced shorter TTC estimates, indicating that rapid forms of motion adaptation and motion priming affect extrapolated motion. Our findings suggest that motion extrapolation might occur at the earliest levels of cortical processing of motion, at which these rapid mechanisms of priming and adaptation take place.

Saccade reaction time asymmetries during task-switching in pursuit tracking

We investigate how smooth pursuit eye movements affect the latencies of task-switching saccades. Participants had to alternate their foveal vision between a continuous pursuit task in the display center and a discrete object discrimination task in the periphery. The pursuit task was either carried out by following the target with the eyes only (ocular) or by steering an on-screen cursor with a joystick (oculomanual). We measured participants' saccadic reaction times (SRTs) when foveal vision was shifted from the pursuit task to the discrimination task and back to the pursuit task. Our results show asymmetries in SRTs depending on the movement direction of the pursuit target: SRTs were generally shorter in the direction of pursuit. Specifically, SRTs from the pursuit target were shorter when the discrimination object appeared in the motion direction. SRTs to pursuit were shorter when the pursuit target moved away from the current fixation location. This result was independent of the type of smooth pursuit behavior that was performed by participants (ocular/oculomanual). The effects are discussed in regard to asymmetries in attention and processes that suppress saccades at the onset of pursuit.

Dissociating oculomotor contributions to spatial and feature-based selection

Saccades not only deliver the high-resolution retinal image requisite for visual perception, but processing stages associated with saccade target selection affect visual perception even before the eye movement starts. These presaccadic effects are thought to arise from two visual selection mechanisms: spatial selection that enhances processing of the saccade target location and feature-based selection that enhances processing of the saccade target features. By measuring oculomotor performance and perceptual discrimination, we determined which selection mechanisms are associated with saccade preparation. We observed both feature-based and space-based selection during saccade preparation but found that feature-based selection was neither related to saccade initiation nor was it affected by simultaneously observed redistribution of spatial selection. We conclude that oculomotor selection biases visual selection only in a spatial, feature-unspecific manner.

Illusory speed is retained in memory during invisible motion

The brain can retain speed information in early visual short-term memory in an astonishingly precise manner. We investigated whether this (early) visual memory system is active during the extrapolation of occluded motion and whether it reflects speed misperception due to contrast and size. Experiments 1A and 2A showed that reducing target contrast or increasing its size led to an illusory speed underestimation. Experiments 1B, 2B, and 3 showed that this illusory phenomenon is reflected in the memory of speed during occluded motion, independent of the range of visible speeds, of the length of the visible trajectory or the invisible trajectory, and of the type of task. These results suggest that illusory speed is retained in memory during invisible motion.

Do common systems control eye movements and motion extrapolation?

People are able to judge the current position of occluded moving objects. This operation is known as motion extrapolation. It has previously been suggested that motion extrapolation is independent of the oculomotor system. Here we revisited this question by measuring eye position while participants completed two types of motion extrapolation task. In one task, a moving visual target travelled rightwards, disappeared, then reappeared further along its trajectory. Participants discriminated correct reappearance times from incorrect (too early or too late) with a two-alternative forced-choice button press. In the second task, the target travelled rightwards behind a visible, rectangular occluder, and participants pressed a button at the time when they judged it should reappear. In both tasks, performance was significantly different under fixation as compared to free eye movement conditions. When eye movements were permitted, eye movements during occlusion were related to participants' judgements. Finally, even when participants were required to fixate, small changes in eye position around fixation (<2°) were influenced by occluded target motion. These results all indicate that overlapping systems control eye movements and judgements on motion extrapolation tasks. This has implications for understanding the mechanism underlying motion extrapolation.