Selective visual processing across competition episodes: a theory of task-driven visual attention and working memory

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

A bias in transsaccadic perception of spatial frequency changes

Visual processing differs between the foveal and peripheral visual field. These differences can lead to different appearances of objects in the periphery and the fovea, posing a challenge to perception across saccades. Differences in the appearance of visual features between the peripheral and foveal visual field may bias change discrimination across saccades. Previously it has been reported that spatial frequency (SF) appears higher in the periphery compared to the fovea (Davis et al., 1987). In this study, we investigated the visual appearance of SF before and after a saccade and the discrimination of SF changes during saccades. In addition, we tested the contributions of pre- and postsaccadic information to change discrimination performance. In the first experiment, we found no differences in the appearance of SF before and after a saccade. However, participants showed a clear bias to report SF increases. Interestingly, a 200-ms postsaccadic blank improved the precision of the responses but did not affect the bias. In the second experiment, participants showed lower thresholds for SF increases than for decreases, suggesting that the bias in the first experiment was not just a response bias. Finally, we asked participants to discriminate the SF of stimuli presented before a saccade. Thresholds in the presaccadic discrimination task were lower than in the change discrimination task, suggesting that transsaccadic change discrimination is not merely limited by presaccadic discrimination in the periphery. The change direction bias might stem from more effective masking or overwriting of the presaccadic stimulus by the postsaccadic low SF stimulus.

Photonic neuromorphic architecture for tens-of-task lifelong learning

Scalable, high-capacity, and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls, hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light, photonic computing has been progressively applied in high-efficient neuromorphic systems. Here, we innovate a reconfigurable lifelong-learning optical neural network (L2ONN), for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsity and parallelism in massive photonic connections, L2ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field, while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time, L2ONN avoided the catastrophic forgetting issue of photonic computing, owning versatile skills on challenging tens-of-tasks (vision classification, voice recognition, medical diagnosis, etc.) with a single model. Particularly, L2ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks, and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme, permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.

Maintaining eye fixation relieves pressure of cognitive action control

Cognitive control enables humans to behave guided by their current goals and intentions. Cognitive control in one task generally suffers when humans try to engage in another task on top. However, we discovered an additional task that supports conflict resolution. In two experiments, participants performed a spatial cognitive control task. For different blocks of trials, they either received no instruction regarding eye movements or were asked to maintain the eyes fixated on a stimulus. The additional eye fixation task did not reduce task performance, but selectively ameliorated the adverse effects of cognitive conflicts on reaction times (Experiment 1). Likewise, in urgent situations, the additional task reduced performance impairments due to stimulus-driven processing overpowering cognitive control (Experiment 2). These findings suggest that maintaining eye fixation locks attentional resources that would otherwise induce spatial cognitive conflicts. This reveals an attentional disinhibition that boosts goal-directed action by relieving pressure from cognitive control.

The role of color in transsaccadic object correspondence

With each saccade, visual information is disrupted, and the visual system is tasked with establishing object correspondence between the presaccadic and postsaccadic representations of the saccade target. There is substantial evidence that the visual system consults spatiotemporal continuity when determining object correspondence across saccades. The evidence for surface feature continuity, however, is mixed. Surface features that are integral to the saccade target object's identity (e.g., shape and contrast polarity) are informative of object continuity, but features that may only imply the state of the object (e.g., orientation) are ignored. The present study tested whether color information is consulted to determine transsaccadic object continuity. We used two variations of the intrasaccadic target displacement task. In Experiments 1 and 2, participants reported the direction of the target displacement. In Experiments 3 and 4, they instead reported whether they detected any target movement. In all experiments, we manipulated the saccade target's continuity by removing it briefly (i.e., blanking) and by changing its color. We found that large color changes can disrupt stability and increase sensitivity to displacements for both direction and movement reports, although not as strongly as long blank durations (250 ms). Interestingly, even smaller color changes, but not blanking, reduced response biases. These results indicate that disrupting surface feature continuity may impact the process of transsaccadic object correspondence more strongly than spatiotemporal disruptions by both increasing the sensitivity and decreasing the response bias.

Does attentional suppression occur at the level of perception or decision-making? Evidence from Gaspelin et al.'s (2015) probe letter task

Visual attention is often inadvertently captured by salient stimuli. It was suggested that it is possible to prevent attentional capture in some search tasks by suppressing salient stimuli below baseline. Evidence for attentional suppression comes from a probe task that was interleaved with the main search task. In the probe task of Gaspelin et al. (Psychol Sci 26(11):1740-1750, 2015. https://doi.org/10.1177/0956797615597913 ), letters were shown on the stimuli of the search display and participants had to identify as many letters as possible. Performance was found to be worse for letters shown on the distractor compared to non-salient non-target stimuli, suggesting that distractor processing was suppressed below baseline. However, it is unclear whether suppression occurred at the level of perception or decision-making because participants may have reported letters on the distractor less frequently than letters on nontargets. This decision-level bias may have degraded performance for letters on distractor compared to nontarget stimuli without changing perception. After replicating the original findings, we conducted two experiments where we avoided report bias by cueing only a single letter for report. We found that the difference between distractor and nontarget stimuli was strongly reduced, suggesting that decision-level processes contribute to attentional suppression. In contrast, the difference between target and non-target stimuli was unchanged, suggesting that it reflected perceptual-level enhancement of the target stimuli.

Warning signals only support the first action in a sequence

Acting upon target stimuli from the environment becomes faster when the targets are preceded by a warning (alerting) cue. Accordingly, alerting is often used to support action in safety-critical contexts (e.g., honking to alert others of a traffic situation). Crucially, however, the benefits of alerting for action have been established using laboratory tasks assessing only simple choice reactions. Real-world actions are considerably more complex and mainly consist of sensorimotor sequences of several sub-actions. Therefore, it is still unknown if the benefits of alerting for action transfer from simple choice reactions to such sensorimotor sequences. Here, we investigated how alerting affected performance in a sequential action task derived from the Trail-Making-Test, a well-established neuropsychological test of cognitive action control (Experiment 1). In addition to this task, participants performed a classic alerting paradigm including a simple choice reaction task (Experiment 2). Results showed that alerting sped up responding in both tasks, but in the sequential action task, this benefit was restricted to the first action of a sequence. This was the case, even when multiple actions were performed within a short time (Experiment 3), ruling out that the restriction of alerting to the first action was due to its short-lived nature. Taken together, these findings reveal the existence of an interface between phasic alertness and action control that supports the next action.

Biased Competition between Targets and Distractors Reduces Attentional Suppression: Evidence from the Positivity Posterior Contralateral and Distractor Positivity

The biased competition account claims that competition between two stimuli increases when they are close together compared with when they are far apart. The reason is that nearby stimuli are more likely to be represented in the same receptive fields, requiring top-down or bottom-up biases to resolve the ambiguity. Consistent with biased competition, previous research showed that an index of attentional enhancement, the N2pc component, was attenuated when two targets were close together. In contrast, it is unclear whether distractor processing would also be attenuated when the distractor is close to the target. To answer this question, we used the additional singleton paradigm where a target is sometimes accompanied by a more salient, but entirely irrelevant, distractor. In the conditions of interest, the distance between the target and the distractor was systematically manipulated whereas the eccentricity to central fixation was always the same. The results showed that two indices of attentional suppression, the positivity posterior contralateral and distractor positivity components, were attenuated when the distractor was close to the target. Consistent with biased competition, attentional suppression of distractors was inhibited when the distance between target and distractor was short. The reduced attentional suppression of distractors with nearby targets may contribute to the increased behavioral interference with close distractors.

Advances in the application of a computational Theory of Visual Attention (TVA): Moving towards more naturalistic stimuli and game-like tasks

The theory of visual attention, “TVA”, is an influential and formal theory of attentional selection. It is widely applied in clinical assessment of attention and fundamental attention research. However, most TVA-based research is based on accuracy data from letter report experiments performed in controlled laboratory environments. While such basic approaches to questions regarding attentional selection are undoubtedly useful, recent technological advances have enabled the use of increasingly sophisticated experimental paradigms involving more realistic scenarios. Notably, these studies have in many cases resulted in different estimates of capacity limits than those found in studies using traditional TVA-based assessment. Here we review recent developments in TVA-based assessment of attention that goes beyond the use of letter report experiments and experiments performed in controlled laboratory environments. We show that TVA can be used with other tasks and new stimuli, that TVA-based parameter estimation can be embedded into complex scenarios, such as games that can be used to investigate particular problems regarding visual attention, and how TVA-based simulations of “visual foraging” can elucidate attentional control in more naturalistic tasks. We also discuss how these developments may inform future advances of TVA.

Statistical learning in visual search reflects distractor rarity, not only attentional suppression

In visual search tasks, salient distractors may capture attention involuntarily, but interference can be reduced when the salient distractor appears more frequently on one out of several possible positions. The reduction was attributed to attentional suppression of the high-probability position. However, all previous studies on this topic compared performance on the high-probability position to the remaining positions, which had a low probability of containing the distractor. Therefore, it is not clear whether the difference resulted from reduced interference on the high-probability position or from increased interference on the low-probability positions. To decide between these alternatives, we compared high-probability and low-probability with equal-probability positions. Consistent with attentional suppression, interference was reduced on the high-probability position compared with equal-probability positions. However, there was also an increase in interference on low-probability positions compared with equal-probability positions. The increase is in line with previous reports of boosted interference when distractors are rare. Our results show that the experimental design used in previous research is insufficient to separate effects of attentional suppression and those of distractor rarity.

The role of temporal cortex in the control of attention

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

Working memory and active sampling of the environment: Medial temporal contributions

Working memory (WM) refers to the ability to maintain and actively process information-either derived from perception or long-term memory (LTM)-for intelligent thought and action. This chapter focuses on the contributions of the temporal lobe, particularly medial temporal lobe (MTL) to WM. First, neuropsychological evidence for the involvement of MTL in WM maintenance is reviewed, arguing for a crucial role in the case of retaining complex relational bindings between memorized features. Next, MTL contributions at the level of neural mechanisms are covered-with a focus on WM encoding and maintenance, including interactions with ventral temporal cortex. Among WM use processes, we focus on active sampling of environmental information, a key input source to capacity-limited WM. MTL contributions to the bidirectional relationship between active sampling and memory are highlighted-WM control of active sampling and sampling as a way of selecting input to WM. Memory-based sampling studies relying on scene and object inspection, visual-based exploration behavior (e.g., vicarious behavior), and memory-guided visual search are reviewed. The conclusion is that MTL serves an important function in the selection of information from perception and transfer from LTM to capacity-limited WM.

Capacity limitations in template-guided multiple color search

Visual selection of target objects relies on representations of their known features in visual working memory. These representations are referred to as attentional templates. We asked how the capacity of visual working memory relates to the maximal number of attentional templates that can simultaneously guide visual selection. To measure the number of active attentional templates, we used the contingent capture paradigm where cues matching the attentional template have larger effects than cues in a non-matching color. We found larger cueing effects for matching than non-matching cues in one-, two-, and also three-color searches, suggesting that participants can establish up to three attentional templates. However, scrutiny of matching cue trials showed that with three attentional templates, larger cueing effects only occurred when the matching cue had the same color as the actual target. When the matching cue had a possible target color that was different from the actual target color, cueing effects were similar to non-matching cue colors. We assume that processing of a matching cue activates one of the three templates, which inhibits the remaining templates to the level of non-matching colors. With two colors, the inhibition from the activated template is less complete because the initial template activation is higher. Overall, only a maximum of two attentional templates can operate successfully in the contingent capture paradigm. The capacity of template-guided search is therefore far below the capacity of visual working memory.

To look or not to look: dissociating presaccadic and covert spatial attention

Attention is a central neural process that enables selective and efficient processing of visual information. Individuals can attend to specific visual information either overtly, by making an eye movement to an object of interest, or covertly, without moving their eyes. We review behavioral, neuropsychological, neurophysiological, and computational evidence of presaccadic attentional modulations that occur while preparing saccadic eye movements, and highlight their differences from those of covert spatial endogenous (voluntary) and exogenous (involuntary) attention. We discuss recent studies and experimental procedures on how these different types of attention impact visual performance, alter appearance, differentially modulate the featural representation of basic visual dimensions (orientation and spatial frequency), engage different neural computations, and recruit partially distinct neural substrates. We conclude that presaccadic attention and covert attention are dissociable.

TVA in the wild: Applying the theory of visual attention to game-like and less controlled experiments

As a formal theory, Bundesen’s theory of visual attention (TVA) enables the estimation of several theoretically meaningful parameters involved in attentional selection and visual encoding. As of yet, TVA has almost exclusively been used in restricted empirical scenarios such as whole and partial report and with strictly controlled stimulus material. We present a series of experiments in which we test whether the advantages of TVA can be exploited in more realistic scenarios with varying degree of stimulus control. This includes brief experimental sessions conducted on different mobile devices, computer games, and a driving simulator. Overall, six experiments demonstrate that the TVA parameters for processing capacity and attentional weight can be measured with sufficient precision in less controlled scenarios and that the results do not deviate strongly from typical laboratory results, although some systematic differences were found.

Allocation of resources in working memory: Theoretical and empirical implications for visual search

Recently, working memory (WM) has been conceptualized as a limited resource, distributed flexibly and strategically between an unlimited number of representations. In addition to improving the precision of representations in WM, the allocation of resources may also shape how these representations act as attentional templates to guide visual search. Here, we reviewed recent evidence in favor of this assumption and proposed three main principles that govern the relationship between WM resources and template-guided visual search. First, the allocation of resources to an attentional template has an effect on visual search, as it may improve the guidance of visual attention, facilitate target recognition, and/or protect the attentional template against interference. Second, the allocation of the largest amount of resources to a representation in WM is not sufficient to give this representation the status of attentional template and thus, the ability to guide visual search. Third, the representation obtaining the status of attentional template, whether at encoding or during maintenance, receives an amount of WM resources proportional to its relevance for visual search. Thus defined, the resource hypothesis of visual search constitutes a parsimonious and powerful framework, which provides new perspectives on previous debates and complements existing models of template-guided visual search.

The time course of salience: not entirely caused by salience

Visual salience is a key component of attentional selection, the process that guards the scarce resources needed for conscious recognition and perception. In previous works, we proposed a measure of visual salience based on a formal theory of visual selection. However, the strength of visual salience depends on the time course as well as local physical contrasts. Evidence from multiple experimental designs in the literature suggests that the strength of salience rises initially and declines after approximately 150 ms. The present article amends the theory-based salience measure beyond local physical contrasts to the time course of salience. It does so through a first experiment which reveals that—contrary to expectations—salience is not reduced during the first 150 ms after onset. Instead, the overall visual processing capacity is severely reduced, which corresponds to a reduced processing speed of all stimuli in the visual field. A second experiment confirms this conclusion by replicating the result. We argue that the slower stimulus processing may have been overlooked previously because the attentional selection mechanism had not yet been modeled in studies on the time course of salience.

Attentional Templates Are Sharpened through Differential Signal Enhancement, Not Differential Allocation of Attention

In visual search, the internal representation of the target feature is referred to as the attentional template. The attentional template can be broad or precise depending on the task requirements. In singleton search, the attentional template is broad because the target is the only colored element in the display. In feature search, a precise attentional template is required because the target is in a specific color in an array of varied colors. To measure the precision of the attentional template, we used a cue-target paradigm where cueing benefits decrease when the cue color differs from the target color. Consistent with broad and precise attentional templates, the decrease of cueing effects was stronger in feature than in singleton search. Measurements of ERPs showed that the N2pc elicited by the cue decreased with increasing color difference, suggesting that attention was more strongly captured by cues that were similar to the target. However, the cue-elicited N2pc did not differ between feature and singleton search, making it unlikely to reflect the mechanism underlying attentional template precision. Furthermore, there was no evidence for attentional suppression as there was no cue-elicited P_D, even in conditions where the cueing benefit turned into a same-location cost. However, an index of signal enhancement, the contralateral positivity, reflected attention template precision. In general, there was sensory enhancement of the stimulus appearing at the cued location in the search display. With broad attentional templates, any stimulus at the cued location was enhanced, whereas enhancement was restricted to target-matching colors with precise attentional templates.

Statistical regularities cause attentional suppression with target-matching distractors

Visual search may be disrupted by the presentation of salient, but irrelevant stimuli. To reduce the impact of salient distractors, attention may suppress their processing below baseline level. While there are many studies on the attentional suppression of distractors with features distinct from the target (e.g., a color distractor with a shape target), there is little and inconsistent evidence for attentional suppression with distractors sharing the target feature. In this study, distractor and target were temporally separated in a cue–target paradigm, where the cue was shown briefly before the target display. With target-matching cues, RTs were shorter when the cue appeared at the target location (valid cues) compared with when it appeared at a nontarget location (invalid cues). To induce attentional suppression, we presented the cue more frequently at one out of four possible target positions. We found that invalid cues appearing at the high-frequency cue position produced less interference than invalid cues appearing at a low-frequency cue position. Crucially, target processing was also impaired at the high-frequency cue position, providing strong evidence for attentional suppression of the cued location. Overall, attentional suppression of the frequent distractor location could be established through feature-based attention, suggesting that feature-based attention may guide attentional suppression just as it guides attentional enhancement.

Oculomotor capture by search-irrelevant features in visual working memory: on the crucial role of target-distractor similarity

When searching for varying targets in the environment, a target template has to be maintained in visual working memory (VWM). Recently, we showed that search-irrelevant features of a VWM template bias attention in an object-based manner, so that objects sharing such features with a VWM template capture the eyes involuntarily. Here, we investigated whether target-distractor similarity modulates capture strength. Participants saccaded to a target accompanied by a distractor. A single feature (e.g., shape) defined the target in each trial indicated by a cue, and the cue also varied in one irrelevant feature (e.g., color). The distractor matched the cue's irrelevant feature in half of the trials. Nine experiments showed that target-distractor similarity consistently influenced the degree of oculomotor capture. High target-distractor dissimilarity in the search-relevant feature reduced capture by the irrelevant feature (Experiments 1, 3, 6, 7). However, capture was reduced by high target-distractor similarity in the search-irrelevant feature (Experiments 1, 4, 5, 8). Strong oculomotor capture was observed if target-distractor similarity was reasonably low in the relevant and high in the irrelevant feature, irrespective of whether color or shape were relevant (Experiments 2 and 5). These findings argue for involuntary and object-based, top-down control by VWM templates, whereas its manifestation in oculomotor capture depends crucially on target-distractor similarity in relevant and irrelevant feature dimensions of the search object.

Spatiotopic and saccade-specific transsaccadic memory for object detail

The content and nature of transsaccadic memory are still a matter of debate. Brief postsaccadic target blanking was demonstrated to recover transsaccadic memory and defeat saccadic suppression of displacement. We examined whether blanking would also support transsaccadic transfer of detailed form information. Observers saccaded to a peripheral, checkerboard-like stimulus and reported whether an intrasaccadic change had occurred in its upper or lower half. On half of the trials, the stimulus was blanked for 200 ms with saccade onset. In a fixation condition, observers kept fixation but the stimulus was displaced from periphery to fixation, mimicking the retinal events of the saccade condition. Results show that stimulus blanking improves transsaccadic change detection, with performance being far superior to the retinally equivalent fixation condition. Our findings argue in favor of a remapped memory trace that can be accessed only in the blanking condition, when not being overwritten by the salient postsaccadic stimulus.

The Binding Problem after an eye movement

Spatial attention is thought to be the "glue" that binds features together (e.g., Treisman & Gelade, 1980, Psychology, 12[1], 97-136)-but attention is dynamic, constantly moving across multiple goals and locations. For example, when a person moves her eyes, visual inputs that are coded relative to the eyes (retinotopic) must be rapidly updated to maintain stable world-centered (spatiotopic) representations. Here, we examined how dynamic updating of spatial attention after a saccadic eye movement affects object-feature binding. Immediately after a saccade, participants were simultaneously presented with four colored and oriented bars (one at a precued spatiotopic target location) and instructed to reproduce both the color and orientation of the target item. Object-feature binding was assessed by applying probabilistic mixture models to the joint distribution of feature errors: feature reports for the target item could be correlated (and thus bound together) or independent. We found that compared with holding attention without an eye movement, attentional updating after an eye movement produced more independent errors, including illusory conjunctions, in which one feature of the item at the spatiotopic target location was misbound with the other feature of the item at the initial retinotopic location. These findings suggest that even when only one spatiotopic location is task relevant, spatial attention-and thus object-feature binding-is malleable across and after eye movements, heightening the challenge that eye movements pose for the binding problem and for visual stability.

Prioritization in visual working memory enhances memory retention and speeds up processing in a comparison task

Visual working memory retains visual information for controlling behavior. We studied how information in visual working memory is prioritized for being used. In two experiments, participants memorized the stimuli of a memory display for a brief interval, followed by a retro-cue. The retro-cue was either valid, indicating which stimulus from the memory display was relevant (i.e., had priority) in the upcoming comparison with a probe, or was neutral (uninformative). Next, the probe was presented, terminated by a mask, and participants reported whether it matched a stimulus from the memory display. The presentation duration of the probe was varied. Assessing performance as a function of presentation duration allowed to disentangle two components of working memory: memory retention and the speed of processing the probe for the memory-based comparison. Compared with neutral retro-cues, valid retro-cues improved retention and at the same time accelerated processing of the probe. These findings show for the first time that prioritization in working memory impacts on distinct mechanisms: retrospectively, it supports memory retention, and prospectively, it enhances perceptual processing in upcoming comparison tasks.

The intersection between the oculomotor and hippocampal memory systems: empirical developments and clinical implications

Decades of cognitive neuroscience research has shown that where we look is intimately connected to what we remember. In this article, we review findings from human and nonhuman animals, using behavioral, neuropsychological, neuroimaging, and computational modeling methods, to show that the oculomotor and hippocampal memory systems interact in a reciprocal manner, on a moment-to-moment basis, mediated by a vast structural and functional network. Visual exploration serves to efficiently gather information from the environment for the purpose of creating new memories, updating existing memories, and reconstructing the rich, vivid details from memory. Conversely, memory increases the efficiency of visual exploration. We call for models of oculomotor control to consider the influence of the hippocampal memory system on the cognitive control of eye movements, and for models of hippocampal and broader medial temporal lobe function to consider the influence of the oculomotor system on the development and expression of memory. We describe eye movement-based applications for the detection of neurodegeneration and delivery of therapeutic interventions for mental health disorders for which the hippocampus is implicated and memory dysfunctions are at the forefront.

Exogeneous Spatial Cueing beyond the Near Periphery: Cueing Effects in a Discrimination Paradigm at Large Eccentricities

Although visual attention is one of the most thoroughly investigated topics in experimental psychology and vision science, most of this research tends to be restricted to the near periphery. Eccentricities used in attention studies usually do not exceed 20° to 30°, but most studies even make use of considerably smaller maximum eccentricities. Thus, empirical knowledge about attention beyond this range is sparse, probably due to a previous lack of suitable experimental devices to investigate attention in the far periphery. This is currently changing due to the development of temporal high-resolution projectors and head-mounted displays (HMDs) that allow displaying experimental stimuli at far eccentricities. In the present study, visual attention was investigated beyond the near periphery (15°, 30°, 56° Exp. 1) and (15°, 35°, 56° Exp. 2) in a peripheral Posner cueing paradigm using a discrimination task with placeholders. Interestingly, cueing effects were revealed for the whole range of eccentricities although the inhomogeneity of the visual field and its functional subdivisions might lead one to suspect otherwise.

Transsaccadic integration is dominated by early, independent noise

Humans are able to integrate pre- and postsaccadic percepts of an object across saccades to maintain perceptual stability. Previous studies have used Maximum Likelihood Estimation (MLE) to determine that integration occurs in a near-optimal manner. Here, we compared three different models to investigate the mechanism of integration in more detail: an early noise model, where noise is added to the pre- and postsaccadic signals before integration occurs; a late-noise model, where noise is added to the integrated signal after integration occurs; and a temporal summation model, where integration benefits arise from the longer transsaccadic presentation duration compared to pre- and postsaccadic presentation only. We also measured spatiotemporal aspects of integration to determine whether integration can occur for very brief stimulus durations, across two hemifields, and in spatiotopic and retinotopic coordinates. Pre-, post-, and transsaccadic performance was measured at different stimulus presentation durations, both at the saccade target and a location where the pre- and postsaccadic stimuli were presented in different hemifields across the saccade. Results showed that for both within- and between-hemifields conditions, integration could occur when pre- and postsaccadic stimuli were presented only briefly, and that the pattern of integration followed an early noise model. Whereas integration occurred when the pre- and post-saccadic stimuli were presented in the same spatiotopic coordinates, there was no integration when they were presented in the same retinotopic coordinates. This contrast suggests that transsaccadic integration is limited by early, independent, sensory noise acting separately on pre- and postsaccadic signals.

Object features reinstated from episodic memory guide attentional selection

When observers search for an object in the environment, they compare the incoming sensory information to the attentional template, a representation of the target in visual working memory (VWM). Previous studies have shown that visual search is more efficient when the attentional template is precise. We pursued the hypothesis that the attentional template in VWM is automatically complemented by features from long-term memory, possibly to increase its precision. At the beginning of the experiment, observers learned associations between shape and color. Then, we tested whether selecting one of these shapes was influenced by the previously associated color. To this end, we ran a saccadic selection task consisting of a memory and choice display. In the memory display, the target shape was presented at central fixation and participants were instructed to foveate this shape in the subsequent choice display. In the choice display, the target shape appeared together with a distractor shape at eccentric positions. Importantly, the target shape was colorless (gray) in the memory display so that only shape, but not color was loaded into VWM. However, saccades went more frequently to the target shape when it was shown in the learned color than when this color was shown in the distractor. Thus, the color of the target shape was reinstated from episodic memory to complement the attentional template in VWM.

Task-Irrelevant Features in Visual Working Memory Influence Covert Attention: Evidence from a Partial Report Task

Selecting a target based on a representation in visual working memory (VWM) affords biasing covert attention towards objects with memory-matching features. Recently, we showed that even task-irrelevant features of a VWM template bias attention. Specifically, when participants had to saccade to a cued shape, distractors sharing the cue's search-irrelevant color captured the eyes. While a saccade always aims at one target location, multiple locations can be attended covertly. Here, we investigated whether covert attention is captured similarly as the eyes. In our partial report task, each trial started with a shape-defined search cue, followed by a fixation cross. Next, two colored shapes, each including a letter, appeared left and right from fixation, followed by masks. The letter inside that shape matching the preceding cue had to be reported. In Experiment 1, either target, distractor, both, or no object matched the cue's irrelevant color. Target-letter reports were most frequent in target-match trials and least frequent in distractor-match trials. Irrelevant cue and target color never matched in Experiment 2. Still, participants reported the distractor more often to the target's disadvantage, when cue and distractor color matched. Thus, irrelevant features of a VWM template can influence covert attention in an involuntarily object-based manner when searching for trial-wise varying targets.

Feature prediction across eye movements is location specific and based on retinotopic coordinates

With each saccadic eye movement, internal object representations change their retinal position and spatial resolution. Recently, we suggested that the visual system deals with these saccade-induced changes by predicting visual features across saccades based on transsaccadic associations of peripheral and foveal input (Herwig & Schneider, 2014). Here we tested the specificity of feature prediction by asking (a) whether it is spatially restricted to the previous learning location or the saccade target location, and (b) whether it is based on retinotopic (eye-centered) or spatiotopic (world-centered) coordinates. In a preceding acquisition phase, objects systematically changed their spatial frequency during saccades. In the following test phases of two experiments, participants had to judge the frequency of briefly presented peripheral objects. These objects were presented either at the previous learning location or at new locations and were either the target of a saccadic eye movement or not (Experiment 1). Moreover, objects were presented either in the same or different retinotopic and spatiotopic coordinates (Experiment 2). Spatial frequency perception was biased toward previously associated foveal input indicating transsaccadic learning and feature prediction. Importantly, while this pattern was not bound to the saccade target location, it was seen only at the previous learning location in retinotopic coordinates, suggesting that feature prediction probably affects low- or mid-level perception.

Current directions in visual working memory research: An introduction and emerging insights

Visual working memory (VWM) is a core construct in the cognitive (neuro-)sciences, assumed to serve as a hub for information exchange and thus supporting a multitude of cognitive functions related to processing visual information. Here, we give an introduction into key terms and paradigms and an overview of ongoing debates in the field, to which the articles collected in this Special Issue on 'Current Directions in Visual Working Memory Research' contribute. Our aim is to extract, from this overview, some 'emerging' theoretical insights concerning questions such as the optimal way to examine VWM, which types of mental representations contribute to performance on VWM tasks, and how VWM keeps features from the same object together and apart from features of concurrently maintained objects (the binding problem).

Functions of Memory Across Saccadic Eye Movements

Several times per second, humans make rapid eye movements called saccades which redirect their gaze to sample new regions of external space. Saccades present unique challenges to both perceptual and motor systems. During the movement, the visual input is smeared across the retina and severely degraded. Once completed, the projection of the world onto the retina has undergone a large-scale spatial transformation. The vector of this transformation, and the new orientation of the eye in the external world, is uncertain. Memory for the pre-saccadic visual input is thought to play a central role in compensating for the disruption caused by saccades. Here, we review evidence that memory contributes to (1) detecting and identifying changes in the world that occur during a saccade, (2) bridging the gap in input so that visual processing does not have to start anew, and (3) correcting saccade errors and recalibrating the oculomotor system to ensure accuracy of future saccades. We argue that visual working memory (VWM) is the most likely candidate system to underlie these behaviours and assess the consequences of VWM's strict resource limitations for transsaccadic processing. We conclude that a full understanding of these processes will require progress on broader unsolved problems in psychology and neuroscience, in particular how the brain solves the object correspondence problem, to what extent prior beliefs influence visual perception, and how disparate signals arriving with different delays are integrated.

Saccade latency delays in young apolipoprotein E (APOE) epsilon 4 carriers

The apolipoprotein E (APOE) epsilon 4 isoform has been associated with a significantly greater risk of developing late onset Alzheimer's disease (AD). However, the negative effects of APOE-ε4 allele on cognitive function vary across the lifespan: reduced memory and executive function have been found in older individuals but, paradoxically, young APOE-ε4 carriers perform better on cognitive tests and show higher neural efficiency. This study aimed to assess the association between APOE genotype and saccade latency using a prosaccade and antisaccade task in young individuals (N = 97, age: 17-35 years). Results showed that prosaccade latency was significantly delayed in a group of ε4 carriers in comparison to non-carriers, which was due to a lower rate of signal accumulation rather than a change in the criterion threshold. In contrast, there was no significant genotype difference for antisaccade latency in this young cohort. These results indicate that prosaccade latency may be useful in establishing the APOE behavioural phenotype, which could ultimately assist with distinguishing between normal and pathological aging.

Attentional competition across saccadic eye movements

Human behavior is guided by visual object recognition. For being recognized, objects compete for limited attentional processing resources. The more objects compete, the lower is performance in recognizing each individual object. Here, we ask whether this competition is confined to eye fixations, periods of relatively stable gaze, or whether it extends from one fixation to the next, across saccadic eye movements. Participants made saccades to a peripheral saccade target. After the saccade, a letter was briefly presented within the saccade target and terminated by a mask. Object recognition of the letter was assessed as participants' report. Critically, either no, two, or four additional non-target objects appeared before the saccade. In Experiment 1, presaccadic non-targets were task-irrelevant and had no effects on postsaccadic object recognition. In Experiment 2, presaccadic non-targets were task-relevant and, here, postsaccadic object recognition deteriorated with increasing number of presaccadic non-targets. As suggested by Experiment 3 and a mathematical model, this effect was due to a slowing down but also a delayed start of visual processing after the saccade. Together, our findings show that objects compete for recognition across saccades, but only if they are task-relevant. This reveals an attentional mechanism of task-driven object recognition that is interlaced with active saccade-mediated vision.

Poking Left To Be Right? A Model-Based Analysis of Temporal Order Judged by Mice

The theory of visual attention (TVTVA) provides a formal framework for the assessment of visual attention and related processes. Its center is a mathematical model of visual encoding processes and discretely defined components of attention. Building on this model, TVTVA offers quantitative and process-related explanations for a variety of phenomena in the domain of visual attention. Because the theory relies on very general assumptions which might hold true for other domains of sensory processing, we tested its possible explanatory value for tactile processing in mice. Reanalyzing published data of temporal-order judgments by mice, we show how a TVTVA-based analysis identifies the processes which drive observable behavior and that it comes to conclusions quite different from those of conventional analyses of temporal-order judgments. According to this analysis, despite the same overall capacity dedicated to the task, some mice assume attentional biases toward one side, possibly to optimize their overall performance. We suggest that TVTVA's concepts provide a powerful point of vantage to find explanations for observable behavior where conventional analysis easily leads to dead ends.

Task implementation and top-down control in continuous search

Evidence from continuous search suggests that targets are detected by default, whereas distractors are processed in considerable depth. These observations shed light on task implementation and top-down control. Task implementation builds on forming dynamic distractor models, based on continuous integration of distractor-related information. Top-down control builds on using these models for testing upcoming stimulus information.

Involuntary top-down control by search-irrelevant features: Visual working memory biases attention in an object-based manner

Many everyday tasks involve successive visual-search episodes with changing targets. Converging evidence suggests that these targets are retained in visual working memory (VWM) and bias attention from there. It is unknown whether all or only search-relevant features of a VWM template bias attention during search. Bias signals might be configured exclusively to task-relevant features so that only search-relevant features bias attention. Alternatively, VWM might maintain objects in the form of bound features. Then, all template features will bias attention in an object-based manner, so that biasing effects are ranked by feature relevance. Here, we investigated whether search-irrelevant VWM template features bias attention. Participants had to saccade to a target opposite a distractor. A colored cue depicted the target prior to each search trial. The target was predefined only by its identity, while its color was irrelevant. When target and cue matched not only in identity (search-relevant) but also in color (search-irrelevant), saccades went more often and faster directly to the target than without any color match (Experiment 1). When introducing a cue-distractor color match (Experiment 2), direct target saccades were most likely when target and cue matched in the search-irrelevant color and least likely in case of a cue-distractor color match. When cue and target were never colored the same (Experiment 3), cue-colored distractors still captured the eyes more often than different-colored distractors despite color being search-irrelevant. As participants were informed about the misleading color, the result argues against a strategical and voluntary usage of color. Instead, search-irrelevant features biased attention obligatorily arguing for involuntary top-down control by object-based VWM templates.

Task-Irrelevant Expectation Violations in Sequential Manual Actions: Evidence for a "Check-after-Surprise" Mode of Visual Attention and Eye-Hand Decoupling

When performing sequential manual actions (e.g., cooking), visual information is prioritized according to the task determining where and when to attend, look, and act. In well-practiced sequential actions, long-term memory (LTM)-based expectations specify which action targets might be found where and when. We have previously demonstrated (Foerster and Schneider, 2015b) that violations of such expectations that are task-relevant (e.g., target location change) cause a regression from a memory-based mode of attentional selection to visual search. How might task-irrelevant expectation violations in such well-practiced sequential manual actions modify attentional selection? This question was investigated by a computerized version of the number-connection test. Participants clicked on nine spatially distributed numbered target circles in ascending order while eye movements were recorded as proxy for covert attention. Target’s visual features and locations stayed constant for 65 prechange-trials, allowing practicing the manual action sequence. Consecutively, a task-irrelevant expectation violation occurred and stayed for 20 change-trials. Specifically, action target number 4 appeared in a different font. In 15 reversion-trials, number 4 returned to the original font. During the first task-irrelevant change trial, manual clicking was slower and eye scanpaths were larger and contained more fixations. The additional fixations were mainly checking fixations on the changed target while acting on later targets. Whereas the eyes repeatedly revisited the task-irrelevant change, cursor-paths remained completely unaffected. Effects lasted for 2–3 change trials and did not reappear during reversion. In conclusion, an unexpected task-irrelevant change on a task-defining feature of a well-practiced manual sequence leads to eye-hand decoupling and a “check-after-surprise” mode of attentional selection.

Using the virtual reality device Oculus Rift for neuropsychological assessment of visual processing capabilities

Neuropsychological assessment of human visual processing capabilities strongly depends on visual testing conditions including room lighting, stimuli, and viewing-distance. This limits standardization, threatens reliability, and prevents the assessment of core visual functions such as visual processing speed. Increasingly available virtual reality devices allow to address these problems. One such device is the portable, light-weight, and easy-to-use Oculus Rift. It is head-mounted and covers the entire visual field, thereby shielding and standardizing the visual stimulation. A fundamental prerequisite to use Oculus Rift for neuropsychological assessment is sufficient test-retest reliability. Here, we compare the test-retest reliabilities of Bundesen’s visual processing components (visual processing speed, threshold of conscious perception, capacity of visual working memory) as measured with Oculus Rift and a standard CRT computer screen. Our results show that Oculus Rift allows to measure the processing components as reliably as the standard CRT. This means that Oculus Rift is applicable for standardized and reliable assessment and diagnosis of elementary cognitive functions in laboratory and clinical settings. Oculus Rift thus provides the opportunity to compare visual processing components between individuals and institutions and to establish statistical norm distributions.

Episodic Short-Term Recognition Requires Encoding into Visual Working Memory: Evidence from Probe Recognition after Letter Report

Human vision is organized in discrete processing episodes (e.g., eye fixations or task-steps). Object information must be transmitted across episodes to enable episodic short-term recognition: recognizing whether a current object has been seen in a previous episode. We ask whether episodic short-term recognition presupposes that objects have been encoded into capacity-limited visual working memory (VWM), which retains visual information for report. Alternatively, it could rely on the activation of visual features or categories that occurs before encoding into VWM. We assessed the dependence of episodic short-term recognition on VWM by a new paradigm combining letter report and probe recognition. Participants viewed displays of 10 letters and reported as many as possible after a retention interval (whole report). Next, participants viewed a probe letter and indicated whether it had been one of the 10 letters (probe recognition). In Experiment 1, probe recognition was more accurate for letters that had been encoded into VWM (reported letters) compared with non-encoded letters (non-reported letters). Interestingly, those letters that participants reported in their whole report had been near to one another within the letter displays. This suggests that the encoding into VWM proceeded in a spatially clustered manner. In Experiment 2, participants reported only one of 10 letters (partial report) and probes either referred to this letter, to letters that had been near to it, or far from it. Probe recognition was more accurate for near than for far letters, although none of these letters had to be reported. These findings indicate that episodic short-term recognition is constrained to a small number of simultaneously presented objects that have been encoded into VWM.

A "blanking effect" for surface features: Transsaccadic spatial-frequency discrimination is improved by postsaccadic blanking

Although saccadic eye movements occur frequently—about three or four times a second—humans are astonishingly blind to transsaccadic changes. Locational displacements of the saccade target of up to 2 deg of visual angle, and even large changes of a visual scene, can go unnoticed. For a long time, this insensitivity was ascribed to deficits in transsaccadic memory: Only a coarse, (spatially) imprecise representation would be retained across a saccade. This assumption was contradicted by Deubel's and Schneider's (Behavioral and Brain Sciences 17:259-260, 1994) striking finding that locational discrimination performance across a saccade is greatly improved by inserting a short postsaccadic blank. Surprisingly, the question of whether blanking effects occur also for other forms of transsaccadic changes (i.e., surface-feature changes) has been widely ignored. We tested this question by means of a transsaccadic change in spatial frequency. Postsaccadic blanking facilitated spatial-frequency discrimination, but to a smaller amount than the usual blanking effects obtained with locational displacements. This finding bears important implications for models of visual stability and transsaccadic memory.

Breaking Object Correspondence Across Saccadic Eye Movements Deteriorates Object Recognition

Visual perception is based on information processing during periods of eye fixations that are interrupted by fast saccadic eye movements. The ability to sample and relate information on task-relevant objects across fixations implies that correspondence between presaccadic and postsaccadic objects is established. Postsaccadic object information usually updates and overwrites information on the corresponding presaccadic object. The presaccadic object representation is then lost. In contrast, the presaccadic object is conserved when object correspondence is broken. This helps transsaccadic memory but it may impose attentional costs on object recognition. Therefore, we investigated how breaking object correspondence across the saccade affects postsaccadic object recognition. In Experiment 1, object correspondence was broken by a brief postsaccadic blank screen. Observers made a saccade to a peripheral object which was displaced during the saccade. This object reappeared either immediately after the saccade or after the blank screen. Within the postsaccadic object, a letter was briefly presented (terminated by a mask). Observers reported displacement direction and letter identity in different blocks. Breaking object correspondence by blanking improved displacement identification but deteriorated postsaccadic letter recognition. In Experiment 2, object correspondence was broken by changing the object's contrast-polarity. There were no object displacements and observers only reported letter identity. Again, breaking object correspondence deteriorated postsaccadic letter recognition. These findings identify transsaccadic object correspondence as a key determinant of object recognition across the saccade. This is in line with the recent hypothesis that breaking object correspondence results in separate representations of presaccadic and postsaccadic objects which then compete for limited attentional processing resources (Schneider, 2013). Postsaccadic object recognition is then deteriorated because less resources are available for processing postsaccadic objects.

Population responses in V1 encode different figures by response amplitude

The visual system simultaneously segregates between several objects presented in a visual scene. The neural code for encoding different objects or figures is not well understood. To study this question, we trained two monkeys to discriminate whether two elongated bars are either separate, thus generating two different figures, or connected, thus generating a single figure. Using voltage-sensitive dyes, we imaged at high spatial and temporal resolution V1 population responses evoked by the two bars, while keeping their local attributes similar among the two conditions. In the separate condition, unlike the connected condition, the population response to one bar is enhanced, whereas the response to the other is simultaneously suppressed. The response to the background remained unchanged between the two conditions. This divergent pattern developed ∼200 ms poststimulus onset and could discriminate well between the separate and connected single trials. The stimulus separation saliency and behavioral report were highly correlated with the differential response to the bars. In addition, the proximity and/or the specific location of the connectors seemed to have only a weak effect on this late activity pattern, further supporting the involvement of top-down influences. Additional neural codes were less informative about the separate and connected conditions, with much less consistency and discriminability compared with a response amplitude code. We suggest that V1 is involved in the encoding of each figure by different neuronal response amplitude, which can mediate their segregation and perception.

Task representation in individual and joint settings

This paper outlines a framework for task representation and discusses applications to interference tasks in individual and joint settings. The framework is derived from the Theory of Event Coding (TEC). This theory regards task sets as transient assemblies of event codes in which stimulus and response codes interact and shape each other in particular ways. On the one hand, stimulus and response codes compete with each other within their respective subsets (horizontal interactions). On the other hand, stimulus and response code cooperate with each other (vertical interactions). Code interactions instantiating competition and cooperation apply to two time scales: on-line performance (i.e., doing the task) and off-line implementation (i.e., setting the task). Interference arises when stimulus and response codes overlap in features that are irrelevant for stimulus identification, but relevant for response selection. To resolve this dilemma, the feature profiles of event codes may become restructured in various ways. The framework is applied to three kinds of interference paradigms. Special emphasis is given to joint settings where tasks are shared between two participants. Major conclusions derived from these applications include: (1) Response competition is the chief driver of interference. Likewise, different modes of response competition give rise to different patterns of interference; (2) The type of features in which stimulus and response codes overlap is also a crucial factor. Different types of such features give likewise rise to different patterns of interference; and (3) Task sets for joint settings conflate intraindividual conflicts between responses (what), with interindividual conflicts between responding agents (whom). Features of response codes may, therefore, not only address responses, but also responding agents (both physically and socially).

Competition with and without priority control: linking rivalry to attention through winner-take-all networks with memory

Competition is ubiquitous in perception. For example, items in the visual field compete for processing resources, and attention controls their priority (biased competition). The inevitable ambiguity in the interpretation of sensory signals yields another form of competition: distinct perceptual interpretations compete for access to awareness. Rivalry, where two equally likely percepts compete for dominance, explicates the latter form of competition. Building upon the similarity between attention and rivalry, we propose to model rivalry by a generic competitive circuit that is widely used in the attention literature-a winner-take-all (WTA) network. Specifically, we show that a network of two coupled WTA circuits replicates three common hallmarks of rivalry: the distribution of dominance durations, their dependence on input strength ("Levelt's propositions"), and the effects of stimulus removal (blanking). This model introduces a form of memory by forming discrete states and explains experimental data better than competitive models of rivalry without memory. This result supports the crucial role of memory in rivalry specifically and in competitive processes in general. Our approach unifies the seemingly distinct phenomena of rivalry, memory, and attention in a single model with competition as the common underlying principle.

When circles become triangular: how transsaccadic predictions shape the perception of shape

Human vision is characterized by a consistent pattern of saccadic eye movements. With each saccade, internal object representations change their retinal position and spatial resolution. This raises the question as to how peripheral perception is affected by imminent saccadic eye movements. Here, we suggest that saccades are accompanied by a prediction of their perceptual consequences (i.e., the foveation of the target object). Accordingly, peripheral perception should be biased toward previously associated foveal input. In this study, we first exposed participants to an altered visual stimulation where one object systematically changed its shape during saccades. Subsequently, participants had to judge the shape of briefly presented peripheral saccade targets. The results showed that targets were perceived as less curved for objects that previously changed from more circular in the periphery to more triangular in the fovea. Similarly, shapes were perceived as more curved for objects that previously changed from triangular to circular. Thus, peripheral perception seems to depend not solely on the current input but also on memorized experiences, enabling predictions about the perceptual consequences of saccadic eye movements.