Oculomotor measures reveal the temporal dynamics of preparing for search

Microsaccades Track Location-Based Object Rehearsal in Visual Working Memory

Besides controlling eye movements, the brain's oculomotor system has been implicated in the control of covert spatial attention and the rehearsal of spatial information in working memory. We investigated whether the oculomotor system also contributes to rehearsing visual objects in working memory when object location is never asked about. To address this, we tracked the incidental use of locations for mnemonic rehearsal via directional biases in microsaccades while participants maintained two visual objects (colored oriented gratings) in working memory. By varying the stimulus configuration (horizontal, diagonal, and vertical) at encoding, we could quantify whether microsaccades were more aligned with the configurational axis of the memory contents, as opposed to the orthogonal axis. Experiment 1 revealed that microsaccades continued to be biased along the axis of the memory content several seconds into the working memory delay. In Experiment 2, we confirmed that this directional microsaccade bias was specific to memory demands, ruling out lingering effects from passive and attentive encoding of the same visual objects in the same configurations. Thus, by studying microsaccade directions, we uncover oculomotor-driven rehearsal of visual objects in working memory through their associated locations.

Do We Prepare for What We Predict? How Target Expectations Affect Preparatory Attentional Templates and Target Selection in Visual Search

Visual search is guided by representations of target-defining features (attentional templates) that are activated in a preparatory fashion. Here, we investigated whether these template activation processes are modulated by probabilistic expectations about upcoming search targets. We tracked template activation while observers prepared to search for one or two possible color-defined targets by measuring N2pc components (markers of attentional capture) to task-irrelevant color probes flashed every 200 msec during the interval between search displays. These probes elicit N2pcs only if the corresponding color template is active at the time when the probe appears. Probe N2pcs emerged from about 600 msec before search display onset. They did not differ between one-color and two-color search, indicating that two color templates can be activated concurrently. Critically, probe N2pcs measured during two-color search were identical for probes matching an expected or unexpected color (target color probability: 80% vs. 20%), or one of two equally likely colors. This strongly suggests that probabilistic target color expectations had no impact on search preparation. In marked contrast, subsequent target selection processes were strongly affected by these expectations. We discuss possible explanations for this clear dissociation in the effects of expectations on preparatory search template activation and search target selection, respectively.

Microsaccade rate activity during the preparation of pro- and antisaccades

Microsaccades belong to the category of fixational micromovements and may be crucial for image stability on the retina. Eye movement paradigms typically require fixational control, but this does not eliminate all oculomotor activity. The antisaccade task requires a planned eye movement in the direction opposite of an onset, allowing separation of planning and execution. We build on previous studies of microsaccades in the antisaccade task using a combination of fixed and mixed pro- and antisaccade blocks. We hypothesized that microsaccade rates may be reduced prior to the execution of antisaccades as compared with regular saccades (prosaccades). In two experiments, we measured microsaccades in four conditions across three trial blocks: one block each of fixed prosaccade and antisaccade trials, and a mixed block where both saccade types were randomized. We anticipated that microsaccade rates would be higher prior to antisaccades than prosaccades due to the need to preemptively suppress reflexive saccades during antisaccade generation. In Experiment 1, with monocular eye tracking, there was an interaction between the effects of saccade and block type on microsaccade rates, suggesting lower rates on antisaccade trials, but only within mixed blocks. In Experiment 2, eye tracking was binocular, revealing suppressed microsaccade rates on antisaccade trials. A cluster permutation analysis of the microsaccade rate over the course of a trial did not reveal any particular critical time for this difference in microsaccade rates. Our findings suggest that microsaccade rates reflect the degree of suppression of the oculomotor system during the antisaccade task.

Microsaccades transiently lateralise EEG alpha activity

The lateralisation of 8-12 Hz alpha activity is a canonical signature of human spatial cognition that is typically studied under strict fixation requirements. Yet, even during attempted fixation, the brain produces small involuntary eye movements known as microsaccades. Here we report how spontaneous microsaccades - made in the absence of incentives to look elsewhere - can themselves drive transient lateralisation of EEG alpha power according to microsaccade direction. This transient lateralisation of posterior alpha power occurs similarly following start and return microsaccades and is, at least for start microsaccades, driven by increased alpha power ipsilateral to microsaccade direction. This reveals new links between spontaneous microsaccades and human electrophysiological brain activity. It highlights how microsaccades are an important factor to consider in studies relating alpha activity - including spontaneous fluctuations in alpha activity - to spatial cognition, such as studies on visual attention, anticipation, and working memory.

Eye Movements as a Window into Decision-Making

For over 100 years, eye movements have been studied and used as indicators of human sensory and cognitive functions. This review evaluates how eye movements contribute to our understanding of the processes that underlie decision-making. Eye movement metrics signify the visual and task contexts in which information is accumulated and weighed. They indicate the efficiency with which we evaluate the instructions for decision tasks, the timing and duration of decision formation, the expected reward associated with a decision, the accuracy of the decision outcome, and our ability to predict and feel confident about a decision. Because of their continuous nature, eye movements provide an exciting opportunity to probe decision processes noninvasively in real time. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Shielding working-memory representations from temporally predictable external interference

Protecting working-memory content from distracting external sensory inputs and intervening tasks is an ubiquitous demand in daily life. Here, we ask whether and how temporal expectations about external events can help mitigate effects of such interference during working-memory retention. We manipulated the temporal predictability of interfering items that occurred during the retention period of a visual working-memory task and report that temporal expectations reduce the detrimental influence of external interference on subsequent memory performance. Moreover, to determine if the protective effects of temporal expectations rely on distractor suppression or involve shielding of internal representations, we compared effects after irrelevant distractors that could be ignored vs. interrupters that required a response. Whereas distractor suppression may be sufficient to confer protection from predictable distractors, any benefits after interruption are likely to involve memory shielding. We found similar benefits of temporal expectations after both types of interference. We conclude that temporal expectations may play an important role in safeguarding behaviour based on working memory - acting through mechanisms that include the shielding of internal content from external interference.

Evaluating cyclist biometrics to develop urban transportation safety metrics

The transportation safety paradigm for urban transportation - particularly safety for those walking and cycling - relies on counting crashes to parameterize safety. These objective measures of safety are spatially static and reflective of past events: they can be enriched by including the human response to risk at diverse infrastructure designs. This perceived risk has been well captured qualitatively in the transportation safety literature; in the following study, we seek to develop a quantitative methodology that captures perceived risk as a continuous measure of human biometrics. Building on diverse safety-critical fields, we hypothesize that the perception of safety can be measured proactively with traveler biometrics, including eye and head movements, such that high readings of biometric indicators correlate with less safe areas. We collect biometric data from cyclists traversing an urban corridor with a protected, yet not continuously, cycle lane. By isolating and correlating peaks in cyclist biometric measures with infrastructure design, we develop a set of continuous variables - lateral head movements, gaze velocity, and off-mean gaze distance, both independently and as a vector - that allow for the evaluation of urban infrastructure based on perceived risk. The results reflect that higher biometric readings correspond to less safe (i.e., unprotected) areas, indicating that perceived risk can be measured proactively with biometric data.

Differential effects of aging and cognitive decline on visual exploration behavior in the elderly

Visual exploration disturbance has been examined in the elderly, mainly from the perspective of associations with cognitive function. However, it remains unknown whether this is a consequence of cognitive decline, age-related changes independent of cognitive decline, or both. In this study, 15 healthy elderly individuals were evaluated using two figure-matching tasks representing visual information processing (clock-matching and inverted clock-matching tasks). Cognitive functions were evaluated for each subject using the Mini-Mental State Examination (MMSE). Infrared eye-movement assessments were used to analyze eye movements during task performance. Behavioral analyses showed that age was associated with longer reaction time, while MMSE score was associated with higher accuracy on the inverted clock-matching task. Analyses of eye-movement parameters showed that MMSE score was negatively associated with a parameter indicating difficulty in the efficiency of visual exploration planning, while age was tended to be positively associated with the sum of saccade times in each trial, both predominantly on the inverted clock-matching task. Our approaches highlighted that age and cognitive decline are separately associated with eye-movement characteristics: cognitive decline is associated with difficulty in visual exploration planning, particularly in situations that require substantial visual working memory resources, whereas aging may be associated with oculomotor dysfunction.

The warning stimulus as retrieval cue: The role of associative memory in temporal preparation

In a warned reaction time task, the warning stimulus (S1) initiates a process of temporal preparation, which promotes a speeded response to the impending target stimulus (S2). According to the multiple trace theory of temporal preparation (MTP), participants learn the timing of S2 by storing a memory trace on each trial, which contains a temporal profile of the events on that trial. On each new trial, S1 serves as a retrieval cue that implicitly and associatively activates memory traces created on earlier trials, which jointly drive temporal preparation for S2. The idea that S1 assumes this role as a retrieval cue was tested across eight experiments, in which two different S1s were associated with two different distributions of S1-S2 intervals: one with predominantly short and one with predominantly long intervals. Experiments differed regarding the S1 features that made up a pair, ranging from highly distinct (e.g., tone and flash) to more similar (e.g., red and green flash) and verbal (i.e., "short" vs "long"). Exclusively for pairs of highly distinct S1s, the results showed that the S1 cue modified temporal preparation, even in participants who showed no awareness of the contingency. This cueing effect persisted in a subsequent transfer phase, in which the contingency between S1 and the timing of S2 was broken - a fact participants were informed of in advance. Together, these findings support the role of S1 as an implicit retrieval cue, consistent with MTP.

Microsaccadic Eye Movements but not Pupillary Dilation Response Characterizes the Crossmodal Freezing Effect

In typical spatial orienting tasks, the perception of crossmodal (e.g., audiovisual) stimuli evokes greater pupil dilation and microsaccade inhibition than unisensory stimuli (e.g., visual). The characteristic pupil dilation and microsaccade inhibition has been observed in response to "salient" events/stimuli. Although the "saliency" account is appealing in the spatial domain, whether this occurs in the temporal context remains largely unknown. Here, in a brief temporal scale (within 1 s) and with the working mechanism of involuntary temporal attention, we investigated how eye metric characteristics reflect the temporal dynamics of perceptual organization, with and without multisensory integration. We adopted the crossmodal freezing paradigm using the classical Ternus apparent motion. Results showed that synchronous beeps biased the perceptual report for group motion and triggered the prolonged sound-induced oculomotor inhibition (OMI), whereas the sound-induced OMI was not obvious in a crossmodal task-free scenario (visual localization without audiovisual integration). A general pupil dilation response was observed in the presence of sounds in both visual Ternus motion categorization and visual localization tasks. This study provides the first empirical account of crossmodal integration by capturing microsaccades within a brief temporal scale; OMI but not pupillary dilation response characterizes task-specific audiovisual integration (shown by the crossmodal freezing effect).

Revisit once more the sensory storage account of visual working memory

Recent work has highlighted the role of early visual areas in visual working memory (VWM) storage and put forward a sensory storage account of VWM. Using a distractor interference paradigm, however, we previolsy showed that the contribution of early visual areas to VWM storage may not be essential. Instead, higher cortical regions such as the posterior parietal cortex may play a more significant role in VWM storage. This is consistent with reviews of other available behavioral, neuroimaging and neurophysiology results. Recently, a number of studies brought forward new evidence regarding this debate. Here I review these new pieces of evidence in detail and show that there is still no strong and definitive evidence supporting an essential role of the early visual areas in VWM storage. Instead, converging evidence suggests that early visual areas may contribute to the decision stage of a VWM task by facilitating target and probe comparison. Aside from further clarifying this debate, it is also important to note that whether or not VWM storage uses a sensory code depends on how it is defined, and that behavioral interactions between VWM and perception tasks do not necessarily support the involvement of sensory regions in VWM storage.

Temporal Expectations Prepare Visual Working Memory for Behavior

Working memory enables us to retain past sensations in service of anticipated task demands. How we prepare for anticipated task demands during working memory retention remains poorly understood. Here, we focused on the role of time—asking how temporal expectations help prepare for ensuing memory-guided behavior. We manipulated the expected probe time in a delayed change-detection task and report that temporal expectation can have a profound influence on memory-guided behavioral performance. EEG measurements corroborated the utilization of temporal expectations: demonstrating the involvement of a classic EEG signature of temporal expectation—the contingent negative variation—in the context of working memory. We also report the influence of temporal expectations on 2 EEG signatures associated with visual working memory—the lateralization of 8- to 12-Hz alpha activity, and the contralateral delay activity. We observed a dissociation between these signatures, whereby alpha lateralization (but not the contralateral delay activity) adapted to the time of expected memory utilization. These data show how temporal expectations prepare visual working memory for behavior and shed new light on the electrophysiological markers of both temporal expectation and working memory.

Oculomotor freezing reflects tactile temporal expectation and aids tactile perception

The oculomotor system keeps the eyes steady in expectation of visual events. Here, recording microsaccades while people performed a tactile, frequency discrimination task enabled us to test whether the oculomotor system shows an analogous preparatory response for unrelated tactile events. We manipulated the temporal predictability of tactile targets using tactile cues, which preceded the target by either constant (high predictability) or variable (low predictability) time intervals. We find that microsaccades are inhibited prior to tactile targets and more so for constant than variable intervals, revealing a tight crossmodal link between tactile temporal expectation and oculomotor action. These findings portray oculomotor freezing as a marker of crossmodal temporal expectation. Moreover, microsaccades occurring around the tactile target presentation are associated with reduced task performance, suggesting that oculomotor freezing mitigates potential detrimental, concomitant effects of microsaccades and revealing a crossmodal coupling between tactile perception and oculomotor action.

Visual working memory and action: Functional links and bi-directional influences

Working memory bridges perception to action over extended delays, enabling flexible goal-directed behaviour. To date, studies of visual working memory – concerned with detailed visual representations such as shape and colour – have considered visual memory predominantly in the context of visual task demands, such as visual identification and search. Another key purpose of visual working memory is to directly inform and guide upcoming actions. Taking this as a starting point, I review emerging evidence for the pervasive bi-directional links between visual working memory and (planned) action, and discuss these links from the perspective of their common goal of enabling flexible and precise behaviour.

Preparatory Template Activation during Search for Alternating Targets

Visual search is guided by representations of target-defining features (attentional templates). We tracked the time course of template activation processes during the preparation for search in a task where the identity of color-defined search targets switched across successive trials (ABAB). Task-irrelevant color probes that matched either the upcoming relevant target color or the previous now-irrelevant target color were presented every 200 msec during the interval between search displays. N2pc components (markers of attentional capture) were measured for both types of probes at each time point. A reliable probe N2pc indicates that the corresponding color template is active at the time when the probe appears. N2pcs of equal size emerged from 1000 msec before search display onset for both relevant-color and irrelevant-color probes, demonstrating that both color templates were activated concurrently. Evidence for color-selective attentional control was found only immediately before the arrival of the search display, where N2pcs were larger for relevant-color probes. These results reveal important limitations in the executive control of search preparation in tasks where two targets alternate across trials. Although the identity of the upcoming target is fully predictable, both task-relevant and task-irrelevant target templates are coactivated. Knowledge about target identity selectively biases these template activation processes in a temporally discrete fashion, guided by temporal expectations about when the target template will become relevant.

Oscillatory Control over Representational States in Working Memory

In the visual world, attention is guided by perceptual goals activated in visual working memory (VWM). However, planning multiple-task sequences also requires VWM to store representations for future goals. These future goals need to be prevented from interfering with the current perceptual task. Recent findings have implicated neural oscillations as a control mechanism serving the implementation and switching of different states of prioritization of VWM representations. We review recent evidence that posterior alpha-band oscillations underlie the flexible activation and deactivation of VWM representations and that frontal delta-to-theta-band oscillations play a role in the executive control of this process. That is, frontal delta-to-theta appears to orchestrate posterior alpha through long-range oscillatory networks to flexibly set up and change VWM states during multitask sequences.

Evidence for confounding eye movements under attempted fixation and active viewing in cognitive neuroscience

Eye movements can have serious confounding effects in cognitive neuroscience experiments. Therefore, participants are commonly asked to fixate. Regardless, participants will make so-called fixational eye movements under attempted fixation, which are thought to be necessary to prevent perceptual fading. Neural changes related to these eye movements could potentially explain previously reported neural decoding and neuroimaging results under attempted fixation. In previous work, under attempted fixation and passive viewing, we found no evidence for systematic eye movements. Here, however, we show that participants' eye movements are systematic under attempted fixation when active viewing is demanded by the task. Since eye movements directly affect early visual cortex activity, commonly used for neural decoding, our findings imply alternative explanations for previously reported results in neural decoding.

Directing Voluntary Temporal Attention Increases Fixational Stability

Our visual input is constantly changing, but not all moments are equally relevant. Visual temporal attention, the prioritization of visual information at specific points in time, increases perceptual sensitivity at behaviorally relevant times. The dynamic processes underlying this increase are unclear. During fixation, humans make small eye movements called microsaccades, and inhibiting microsaccades improves perception of brief stimuli. Here, we investigated whether temporal attention changes the pattern of microsaccades in anticipation of brief stimuli. Human observers (female and male) judged stimuli presented within a short sequence. Observers were given either an informative precue to attend to one of the stimuli, which was likely to be probed, or an uninformative (neutral) precue. We found strong microsaccadic inhibition before the stimulus sequence, likely due to its predictable onset. Critically, this anticipatory inhibition was stronger when the first target in the sequence (T1) was precued (task-relevant) than when the precue was uninformative. Moreover, the timing of the last microsaccade before T1 and the first microsaccade after T1 shifted such that both occurred earlier when T1 was precued than when the precue was uninformative. Finally, the timing of the nearest pre- and post-T1 microsaccades affected task performance. Directing voluntary temporal attention therefore affects microsaccades, helping to stabilize fixation at the most relevant moments over and above the effect of predictability. Just as saccading to a relevant stimulus can be an overt correlate of the allocation of spatial attention, precisely timed gaze stabilization can be an overt correlate of the allocation of temporal attention.SIGNIFICANCE STATEMENT We pay attention at moments in time when a relevant event is likely to occur. Such temporal attention improves our visual perception, but how it does so is not well understood. Here, we discovered a new behavioral correlate of voluntary, or goal-directed, temporal attention. We found that the pattern of small fixational eye movements called microsaccades changes around behaviorally relevant moments in a way that stabilizes the position of the eyes. Microsaccades during a brief visual stimulus can impair perception of that stimulus. Therefore, such fixation stabilization may contribute to the improvement of visual perception at attended times. This link suggests that, in addition to cortical areas, subcortical areas mediating eye movements may be recruited with temporal attention.

The Time Course of Target Template Activation Processes during Preparation for Visual Search

Search for target objects in visual scenes is guided by mental representations of target features (attentional templates). However, it is unknown when such templates are activated during each search episode and whether this can be controlled by temporal expectations. We used electrophysiological measures to track search template activation processes in real time. In three experiments, female and male humans searched for a color-defined target object in search displays where targets were accompanied by distractors in different nontarget colors. Brief task-irrelevant color singleton probes that matched the target template were flashed rapidly (every 200 ms) throughout each block. Probes presented at times when the target template is active should capture attention, whereas probes presented at other times should not. To assess this, N2pc components were measured as markers of attentional capture, separately for probes at each successive temporal position between two search displays. Results demonstrated that search templates were active from ∼1000 ms before the arrival of the next search display, and were deactivated after each search episode, even when the preceding search display did not contain a target object. Templates were activated later when the predictable interval between search displays was increased. Results demonstrate that search templates are not continuously active but are transiently activated during the preparation for each new search episode. These activation states are regulated in a top-down fashion by temporal expectations about when an attentional template will become task-relevant.SIGNIFICANCE STATEMENT It is often assumed that observers prepare for a visual search task by activating mental representations of search target objects (attentional templates). However, the time course of such template activation processes is completely unknown. By using a new sequential probe presentation technique and electrophysiological measures of attentional processing, we demonstrate that target templates are rapidly activated and deactivated before and after each successive search display, and that these template activation states are tuned to observers' temporal expectations. These results provide novel insights into the temporal dynamics of cognitive control processes in visual attention. They show that attentional templates for visual search are preparatory states that are activated in a transient fashion before each new search episode.

Pupil Dilation Reflects Task Relevance Prior to Search

When observers search for a specific target, it is assumed that they activate a representation of the task relevant object in visual working memory (VWM). This representation – often referred to as the template – guides attention towards matching visual input. In two experiments we tested whether the pupil response can be used to differentiate stimuli that match the task-relevant template from irrelevant input. Observers memorized a target color to be searched for in a multi-color visual search display, presented after a delay period. In Experiment 1, one color appeared at the start of the trial, which was then automatically the search template. In Experiments 2, two colors were presented, and a retro-cue indicated which of these was relevant for the upcoming search task. Crucially, before the search display appeared, we briefly presented one colored probe stimulus. The probe could match either the relevant-template color, the non-cued color (irrelevant), or be a new color not presented in the trial. We measured the pupil response to the probe as a signature of task relevance. Experiment 1 showed significantly smaller pupil size in response to probes matching the search template than for irrelevant colors. Experiment 2 replicated the template matching effect and allowed us to rule out that it was solely due to repetition priming. Taken together, we show that the pupil responds selectively to participants’ target template prior to search.