Reading scenes: how scene grammar guides attention and aids perception in real-world environments

How schema knowledge influences memory in older adults: Filling in the gaps, or leading memory astray?

Age-related declines in episodic memory do not affect all types of mnemonic information equally: when to-be-remembered information is in line with one's prior knowledge, or schema-congruent, older adults often show no impairments. There are two major accounts of this effect: One proposes that schemas compensate for memory failures in aging, and the other proposes that schemas instead actively impair older adults' otherwise intact memory for incongruent information. However, the evidence thus far is inconclusive, likely due to methodological constraints in teasing apart these complex underlying dynamics. We developed a paradigm that separately examines the contributions of underlying memory and schema knowledge to a final memory decision, allowing these dynamics to be examined directly. In the present study, healthy older and younger adults first searched for target objects in congruent or incongruent locations within scenes. In a subsequent test, participants indicated where in each scene the target had been located previously, and provided confidence-based recognition memory judgments that indexed underlying memory, in terms of recollection and familiarity, for the background scenes. We found that age-related increases in schema effects on target location spatial recall were predicted and statistically mediated by age-related increases in underlying memory failures, specifically within recollection. We also found that, relative to younger adults, older adults had poorer spatial memory precision within recollected scenes but slightly better precision within familiar scenes-and age increases in schema bias were primarily exhibited within recollected scenes. Interestingly, however, there were also slight age-related increases in schema effects that could not be explained by memory deficits alone, outlining a role for active schema influences as well. Together, these findings support the account that age-related schema effects on memory are compensatory in that they are driven primarily by underlying memory failures, and further suggest that age-related deficits in memory precision may also drive schema effects.

Scene semantics affects allocentric spatial coding for action in naturalistic (virtual) environments

Interacting with objects in our environment requires determining their locations, often with respect to surrounding objects (i.e., allocentrically). According to the scene grammar framework, these usually small, local objects are movable within a scene and represent the lowest level of a scene's hierarchy. How do higher hierarchical levels of scene grammar influence allocentric coding for memory-guided actions? Here, we focused on the effect of large, immovable objects (anchors) on the encoding of local object positions. In a virtual reality study, participants (n = 30) viewed one of four possible scenes (two kitchens or two bathrooms), with two anchors connected by a shelf, onto which were presented three local objects (congruent with one anchor) (Encoding). The scene was re-presented (Test) with 1) local objects missing and 2) one of the anchors shifted (Shift) or not (No shift). Participants, then, saw a floating local object (target), which they grabbed and placed back on the shelf in its remembered position (Response). Eye-tracking data revealed that both local objects and anchors were fixated, with preference for local objects. Additionally, anchors guided allocentric coding of local objects, despite being task-irrelevant. Overall, anchors implicitly influence spatial coding of local object locations for memory-guided actions within naturalistic (virtual) environments.

The Brain's Sensitivity to Real-world Statistical Regularity Does Not Require Full Attention

Predictive coding accounts of perception state that the brain generates perceptual predictions in the service of processing incoming sensory data. These predictions are hypothesized to be afforded by the brain's ability to internalize useful patterns, that is, statistical regularities, from the environment. We have previously argued that the N300 ERP component serves as an index of the brain's use of representations of (real-world) statistical regularities. However, we do not yet know whether overt attention is necessary in order for this process to engage. We addressed this question by presenting stimuli of either high or low real-world statistical regularity in terms of their representativeness (good/bad exemplars of natural scene categories) to participants who either fully attended the stimuli or were distracted by another task (attended/distracted conditions). Replicating past work, N300 responses were larger to bad than to good scene exemplars, and furthermore, we demonstrate minimal impacts of distraction on N300 effects. Thus, it seems that overtly focused attention is not required to maintain the brain's sensitivity to real-world statistical regularity. Furthermore, in an exploratory analysis, we showed that providing additional, artificial regularities, formed by altering the proportions of good and bad exemplars within blocks, further enhanced the N300 effect in both attended and distracted conditions, shedding light on the relationship between statistical regularities learned in the real world and those learned within the context of an experiment.

A Phone in a Basket Looks Like a Knife in a Cup: Role-Filler Independence in Visual Processing

When a piece of fruit is in a bowl, and the bowl is on a table, we appreciate not only the individual objects and their features, but also the relations containment and support, which abstract away from the particular objects involved. Independent representation of roles (e.g., containers vs. supporters) and "fillers" of those roles (e.g., bowls vs. cups, tables vs. chairs) is a core principle of language and higher-level reasoning. But does such role-filler independence also arise in automatic visual processing? Here, we show that it does, by exploring a surprising error that such independence can produce. In four experiments, participants saw a stream of images containing different objects arranged in force-dynamic relations-e.g., a phone contained in a basket, a marker resting on a garbage can, or a knife sitting in a cup. Participants had to respond to a single target image (e.g., a phone in a basket) within a stream of distractors presented under time constraints. Surprisingly, even though participants completed this task quickly and accurately, they false-alarmed more often to images matching the target's relational category than to those that did not-even when those images involved completely different objects. In other words, participants searching for a phone in a basket were more likely to mistakenly respond to a knife in a cup than to a marker on a garbage can. Follow-up experiments ruled out strategic responses and also controlled for various confounding image features. We suggest that visual processing represents relations abstractly, in ways that separate roles from fillers.

Finding the meaning in meaning maps: Quantifying the roles of semantic and non-semantic scene information in guiding visual attention

In real-world vision, people prioritise the most informative scene regions via eye-movements. According to the cognitive guidance theory of visual attention, viewers allocate visual attention to those parts of the scene that are expected to be the most informative. The expected information of a scene region is coded in the semantic distribution of that scene. Meaning maps have been proposed to capture the spatial distribution of local scene semantics in order to test cognitive guidance theories of attention. Notwithstanding the success of meaning maps, the reason for their success has been contested. This has led to at least two possible explanations for the success of meaning maps in predicting visual attention. On the one hand, meaning maps might measure scene semantics. On the other hand, meaning maps might measure scene features, overlapping with, but distinct from, scene semantics. This study aims to disentangle these two sources of information by considering both conceptual information and non-semantic scene entropy simultaneously. We found that both semantic and non-semantic information is captured by meaning maps, but scene entropy accounted for more unique variance in the success of meaning maps than conceptual information. Additionally, some explained variance was unaccounted for by either source of information. Thus, although meaning maps may index some aspect of semantic information, their success seems to be better explained by non-semantic information. We conclude that meaning maps may not yet be a good tool to test cognitive guidance theories of attention in general, since they capture non-semantic aspects of local semantic density and only a small portion of conceptual information. Rather, we suggest that researchers should better define the exact aspect of cognitive guidance theories they wish to test and then use the tool that best captures that desired semantic information. As it stands, the semantic information contained in meaning maps seems too ambiguous to draw strong conclusions about how and when semantic information guides visual attention.

Exploring the Semantic-Inconsistency Effect in Scenes Using a Continuous Measure of Linguistic-Semantic Similarity

Viewers use contextual information to visually explore complex scenes. Object recognition is facilitated by exploiting object-scene relations (which objects are expected in a given scene) and object-object relations (which objects are expected because of the occurrence of other objects). Semantically inconsistent objects deviate from these expectations, so they tend to capture viewers' attention (the semantic-inconsistency effect). Some objects fit the identity of a scene more or less than others, yet semantic inconsistencies have hitherto been operationalized as binary (consistent vs. inconsistent). In an eye-tracking experiment (N = 21 adults), we study the semantic-inconsistency effect in a continuous manner by using the linguistic-semantic similarity of an object to the scene category and to other objects in the scene. We found that both highly consistent and highly inconsistent objects are viewed more than other objects (U-shaped relationship), revealing that the (in)consistency effect is more than a simple binary classification.

Predicting attentional allocation in real-world environments: The need to investigate crossmodal semantic guidance

Real-world environments are multisensory, meaningful, and highly complex. To parse these environments in a highly efficient manner, a subset of this information must be selected both within and across modalities. However, the bulk of attention research has been conducted within sensory modalities, with a particular focus on vision. Visual attention research has made great strides, with over a century of research methodically identifying the underlying mechanisms that allow us to select critical visual information. Spatial attention, attention to features, and object-based attention have all been studied extensively. More recently, research has established semantics (meaning) as a key component to allocating attention in real-world scenes, with the meaning of an item or environment affecting visual attentional selection. However, a full understanding of how semantic information modulates real-world attention requires studying more than vision in isolation. The world provides semantic information across all senses, but with this extra information comes greater complexity. Here, we summarize visual attention (including semantic-based visual attention), crossmodal attention, and argue for the importance of studying crossmodal semantic guidance of attention. This article is categorized under: Psychology > Attention Psychology > Perception and Psychophysics.

Quantifying task-related gaze

Competing theories attempt to explain what guides eye movements when exploring natural scenes: bottom-up image salience and top-down semantic salience. In one study, we apply language-based analyses to quantify the well-known observation that task influences gaze in natural scenes. Subjects viewed ten scenes as if they were performing one of two tasks. We found that the semantic similarity between the task and the labels of objects in the scenes captured the task-dependence of gaze (t(39) = 13.083; p < 0.001). In another study, we examined whether image salience or semantic salience better predicts gaze during a search task, and if viewing strategies are affected by searching for targets of high or low semantic relevance to the scene. Subjects searched 100 scenes for a high- or low-relevance object. We found that image salience becomes a worse predictor of gaze across successive fixations, while semantic salience remains a consistent predictor (X2(1, N=40) = 75.148, p < .001). Furthermore, we found that semantic salience decreased as object relevance decreased (t(39) = 2.304; p = .027). These results suggest that semantic salience is a useful predictor of gaze during task-related scene viewing, and that even in target-absent trials, gaze is modulated by the relevance of a search target to the scene in which it might be located.

Multiple mechanisms of visual prediction as revealed by the timecourse of scene-object facilitation

Not only semantic, but also recently learned arbitrary associations have the potential to facilitate visual processing in everyday life-for example, knowledge of a (moveable) object's location at a specific time may facilitate visual processing of that object. In our prior work, we showed that previewing a scene can facilitate processing of recently associated objects at the level of visual analysis (Smith and Federmeier in Journal of Cognitive Neuroscience, 32(5):783-803, 2020). In the current study, we assess how rapidly this facilitation unfolds by manipulating scene preview duration. We then compare our results to studies using well-learned object-scene associations in a first-pass assessment of whether systems consolidation might speed up high-level visual prediction. In two ERP experiments (N = 60), we had participants study categorically organized novel object-scene pairs in an explicit paired associate learning task. At test, we varied contextual pre-exposure duration, both between (200 vs. 2500 ms) and within subjects (0-2500 ms). We examined the N300, an event-related potential component linked to high-level visual processing of objects and scenes and found that N300 effects of scene congruity increase with longer scene previews, up to approximately 1-2 s. Similar results were obtained for response times and in a separate component-neutral ERP analysis of visual template matching. Our findings contrast with prior evidence that scenes can rapidly facilitate visual processing of commonly associated objects. This raises the possibility that systems consolidation might mediate different kinds of predictive processing with different temporal profiles.

Individual differences in internal models explain idiosyncrasies in scene perception

According to predictive processing theories, vision is facilitated by predictions derived from our internal models of what the world should look like. However, the contents of these models and how they vary across people remains unclear. Here, we use drawing as a behavioral readout of the contents of the internal models in individual participants. Participants were first asked to draw typical versions of scene categories, as descriptors of their internal models. These drawings were converted into standardized 3d renders, which we used as stimuli in subsequent scene categorization experiments. Across two experiments, participants' scene categorization was more accurate for renders tailored to their own drawings compared to renders based on others' drawings or copies of scene photographs, suggesting that scene perception is determined by a match with idiosyncratic internal models. Using a deep neural network to computationally evaluate similarities between scene renders, we further demonstrate that graded similarity to the render based on participants' own typical drawings (and thus to their internal model) predicts categorization performance across a range of candidate scenes. Together, our results showcase the potential of a new method for understanding individual differences - starting from participants' personal expectations about the structure of real-world scenes.

Divergent thinking modulates interactions between episodic memory and schema knowledge: Controlled and spontaneous episodic retrieval processes

The ability to generate novel ideas, known as divergent thinking, depends on both semantic knowledge and episodic memory. Semantic knowledge and episodic memory are known to interact to support memory decisions, but how they may interact to support divergent thinking is unknown. Moreover, it is debated whether divergent thinking relies on spontaneous or controlled retrieval processes. We addressed these questions by examining whether divergent thinking ability relates to interactions between semantic knowledge and different episodic memory processes. Participants completed the alternate uses task of divergent thinking, and completed a memory task in which they searched for target objects in schema-congruent or schema-incongruent locations within scenes. In a subsequent test, participants indicated where in each scene the target object had been located previously (i.e., spatial accuracy test), and provided confidence-based recognition memory judgments that indexed distinct episodic memory processes (i.e., recollection, familiarity, and unconscious memory) for the scenes. We found that higher divergent thinking ability-specifically in terms of the number of ideas generated-was related to (1) more of a benefit from recollection (a controlled process) and unconscious memory (a spontaneous process) on spatial accuracy and (2) beneficial differences in how semantic knowledge was combined with recollection and unconscious memory to influence spatial accuracy. In contrast, there were no effects with respect to familiarity (a spontaneous process). These findings indicate that divergent thinking is related to both controlled and spontaneous memory processes, and suggest that divergent thinking is related to the ability to flexibly combine semantic knowledge with episodic memory.

Reliability of individual differences in distractor suppression driven by statistical learning

A series of recent studies has demonstrated that attentional selection is modulated by statistical regularities, even when they concern task-irrelevant stimuli. Irrelevant distractors presented more frequently at one location interfere less with search than distractors presented elsewhere. To account for this finding, it has been proposed that through statistical learning, the frequent distractor location becomes suppressed relative to the other locations. Learned distractor suppression has mainly been studied at the group level, where individual differences are treated as unexplained error variance. Yet these individual differences may provide important mechanistic insights and could be predictive of cognitive and real-life outcomes. In the current study, we ask whether in an additional singleton task, the standard measures of attentional capture and learned suppression are reliable and stable at the level of the individual. In an online study, we assessed both the within- and between-session reliability of individual-level measures of attentional capture and learned suppression. We show that the measures of attentional capture, but not of distractor suppression, are moderately stable within the same session (i.e., split-half reliability). Test-retest reliability over a 2-month period was found to be moderate for attentional capture but weak or absent for suppression. RT-based measures proved to be superior to accuracy measures. While producing very robust findings at the group level, the predictive validity of these RT-based measures is still limited when it comes to individual-level performance. We discuss the implications for future research drawing on inter-individual variation in the attentional biases that result from statistical learning.

Visual search patterns during exploration of naturalistic scenes are driven by saliency cues in individuals with cerebral visual impairment

We investigated the relative influence of image salience and image semantics during the visual search of naturalistic scenes, comparing performance in individuals with cerebral visual impairment (CVI) and controls with neurotypical development. Participants searched for a prompted target presented as either an image or text cue. Success rate and reaction time were collected, and gaze behavior was recorded with an eye tracker. A receiver operating characteristic (ROC) analysis compared the distribution of individual gaze landings based on predictions of image salience (using Graph-Based Visual Saliency) and image semantics (using Global Vectors for Word Representations combined with Linguistic Analysis of Semantic Salience) models. CVI participants were less likely and were slower in finding the target. Their visual search behavior was also associated with a larger visual search area and greater number of fixations. ROC scores were also lower in CVI compared to controls for both model predictions. Furthermore, search strategies in the CVI group were not affected by cue type, although search times and accuracy showed a significant correlation with verbal IQ scores for text-cued searches. These results suggest that visual search patterns in CVI are driven mainly by image salience and provide further characterization of higher-order processing deficits observed in this population.

Mission impossible? Spatial context relearning following a target relocation event depends on cue predictiveness

Visual search for a target is faster when the spatial layout of distractors is repeatedly encountered, illustrating that statistical learning of contextual invariances facilitates attentional guidance (contextual cueing; Chun & Jiang, 1998, Cognitive Psychology, 36, 28-71). While contextual learning is usually relatively efficient, relocating the target to an unexpected location (within an otherwise unchanged search layout) typically abolishes contextual cueing and the benefits deriving from invariant contexts recover only slowly with extensive training (Zellin et al., 2014, Psychonomic Bulletin & Review, 21(4), 1073-1079). However, a recent study by Peterson et al. (2022, Attention, Perception, & Psychophysics, 84(2), 474-489) in fact reported rather strong adaptation of spatial contextual memories following target position changes, thus contrasting with prior work. Peterson et al. argued that previous studies may have been underpowered to detect a reliable recovery of contextual cueing after the change. However, their experiments also used a specific display design that frequently presented the targets at the same locations, which might reduce the predictability of the contextual cues thereby facilitating its flexible relearning (irrespective of statistical power). The current study was a (high-powered) replication of Peterson et al., taking into account both statistical power and target overlap in context-memory adaptation. We found reliable contextual cueing for the initial target location irrespective of whether the targets shared their location across multiple displays, or not. However, contextual adaptation following a target relocation event occurred only when target locations were shared. This suggests that cue predictability modulates contextual adaptation, over and above a possible (yet negligible) influence of statistical power.

Terms of debate: Consensus definitions to guide the scientific discourse on visual distraction

Hypothesis-driven research rests on clearly articulated scientific theories. The building blocks for communicating these theories are scientific terms. Obviously, communication - and thus, scientific progress - is hampered if the meaning of these terms varies idiosyncratically across (sub)fields and even across individual researchers within the same subfield. We have formed an international group of experts representing various theoretical stances with the goal to homogenize the use of the terms that are most relevant to fundamental research on visual distraction in visual search. Our discussions revealed striking heterogeneity and we had to invest much time and effort to increase our mutual understanding of each other's use of central terms, which turned out to be strongly related to our respective theoretical positions. We present the outcomes of these discussions in a glossary and provide some context in several essays. Specifically, we explicate how central terms are used in the distraction literature and consensually sharpen their definitions in order to enable communication across theoretical standpoints. Where applicable, we also explain how the respective constructs can be measured. We believe that this novel type of adversarial collaboration can serve as a model for other fields of psychological research that strive to build a solid groundwork for theorizing and communicating by establishing a common language. For the field of visual distraction, the present paper should facilitate communication across theoretical standpoints and may serve as an introduction and reference text for newcomers.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

Where's Wanda? The influence of visual imagery vividness on visual search speed measured by means of hidden object pictures

Previous research demonstrated effects of visual imagery on search speed in visual search paradigms. However, these effects were rather small, questioning their ecological validity. Thus, our present study aimed to generalize these effects to more naturalistic material (i.e., a paradigm that allows for top-down strategies in highly complex visual search displays that include overlapping stimuli while simultaneously avoiding possibly confounding search instructions). One hundred and four participants with aphantasia (= absence of voluntary mental imagery) and 104 gender and age-matched controls were asked to find hidden objects in several hidden object pictures with search times recorded. Results showed that people with aphantasia were significantly slower than controls, even when controlling for age and general processing speed. Thus, effects of visual imagery might be strong enough to influence the perception of our real-life surroundings, probably because of the involvement of visual imagery in several top-down strategies.

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Learned associations serve as target proxies during difficult but not easy visual search

The target template contains information in memory that is used to guide attention during visual search and is typically thought of as containing features of the actual target object. However, when targets are hard to find, it is advantageous to use other information in the visual environment that is predictive of the target's location to help guide attention. The purpose of these studies was to test if newly learned associations between face and scene category images lead observers to use scene information as a proxy for the face target. Our results showed that scene information was used as a proxy for the target to guide attention but only when the target face was difficult to discriminate from the distractor face; when the faces were easy to distinguish, attention was no longer guided by the scene unless the scene was presented earlier. The results suggest that attention is flexibly guided by both target features as well as features of objects that are predictive of the target location. The degree to which each contributes to guiding attention depends on the efficiency with which that information can be used to decode the location of the target in the current moment. The results contribute to the view that attentional guidance is highly flexible in its use of information to rapidly locate the target.

Predictive processing of scenes and objects

Real-world visual input consists of rich scenes that are meaningfully composed of multiple objects which interact in complex, but predictable, ways. Despite this complexity, we recognize scenes, and objects within these scenes, from a brief glance at an image. In this review, we synthesize recent behavioral and neural findings that elucidate the mechanisms underlying this impressive ability. First, we review evidence that visual object and scene processing is partly implemented in parallel, allowing for a rapid initial gist of both objects and scenes concurrently. Next, we discuss recent evidence for bidirectional interactions between object and scene processing, with scene information modulating the visual processing of objects, and object information modulating the visual processing of scenes. Finally, we review evidence that objects also combine with each other to form object constellations, modulating the processing of individual objects within the object pathway. Altogether, these findings can be understood by conceptualizing object and scene perception as the outcome of a joint probabilistic inference, in which "best guesses" about objects act as priors for scene perception and vice versa, in order to concurrently optimize visual inference of objects and scenes.

The effect of image fractal properties and its interaction with visual discomfort on gait kinematics

Exposure to images of urban environments affords higher cognitive processing demands than exposure to images of nature scenes; an effect potentially related to differences in low-level image statistics such as fractals. The aim of the current study was to investigate whether the fractal dimensions of an abstract scene affect cognitive processing demands, using gait kinematics as a measure of cognitive demand. Participants (n = 40) were asked to walk towards different types of synthetic images which were parametrically varied in their fractal dimensions. At the end of each walk, participants rated each image for its visual discomfort (n = 20) or for its likability (n = 20) as potential confounding factors. Fractal dimensions were predictors of walking speed. Moreover, the interaction between fractal dimensions and subjective visual discomfort but not liking predicted velocity. Overall, these data suggest that fractal dimensions indeed contribute to environmentally induced cognitive processing demands.

Changing What You Like: Modifying Contour Properties Shifts Aesthetic Valuations of Scenes

To what extent do aesthetic experiences arise from the human ability to perceive and extract meaning from visual features? Ordinary scenes, such as a beach sunset, can elicit a sense of beauty in most observers. Although it appears that aesthetic responses can be shared among humans, little is known about the cognitive mechanisms that underlie this phenomenon. We developed a contour model of aesthetics that assigns values to visual properties in scenes, allowing us to predict aesthetic responses in adults from around the world. Through a series of experiments, we manipulate contours to increase or decrease aesthetic value while preserving scene semantic identity. Contour manipulations directly shift subjective aesthetic judgments. This provides the first experimental evidence for a causal relationship between contour properties and aesthetic valuation. Our findings support the notion that visual regularities underlie the human capacity to derive pleasure from visual information.

Using XR (Extended Reality) for Behavioral, Clinical, and Learning Sciences Requires Updates in Infrastructure and Funding

Extended reality (XR, including augmented and virtual reality) creates a powerful intersection between information technology and cognitive, clinical, and education sciences. XR technology has long captured the public imagination, and its development is the focus of major technology companies. This article demonstrates the potential of XR to (1) deliver behavioral insights, (2) transform clinical treatments, and (3) improve learning and education. However, without appropriate policy, funding, and infrastructural investment, many research institutions will struggle to keep pace with the advances and opportunities of XR. To realize the full potential of XR for basic and translational research, funding should incentivize (1) appropriate training, (2) open software solutions, and (3) collaborations between complementary academic and industry partners. Bolstering the XR research infrastructure with the right investments and incentives is vital for delivering on the potential for transformative discoveries, innovations, and applications.

Influence of prior knowledge on eye movements to scenes as revealed by hidden Markov models

Human visual experience usually provides ample opportunity to accumulate knowledge about events unfolding in the environment. In typical scene perception experiments, however, participants view images that are unrelated to each other and, therefore, they cannot accumulate knowledge relevant to the upcoming visual input. Consequently, the influence of such knowledge on how this input is processed remains underexplored. Here, we investigated this influence in the context of gaze control. We used sequences of static film frames arranged in a way that allowed us to compare eye movements to identical frames between two groups: a group that accumulated prior knowledge relevant to the situations depicted in these frames and a group that did not. We used a machine learning approach based on hidden Markov models fitted to individual scanpaths to demonstrate that the gaze patterns from the two groups differed systematically and, thereby, showed that recently accumulated prior knowledge contributes to gaze control. Next, we leveraged the interpretability of hidden Markov models to characterize these differences. Additionally, we report two unexpected and interesting caveats of our approach. Overall, our results highlight the importance of recently acquired prior knowledge for oculomotor control and the potential of hidden Markov models as a tool for investigating it.

Scene context automatically drives predictions of object transformations

As our viewpoint changes, the whole scene around us rotates coherently. This allows us to predict how one part of a scene (e.g., an object) will change by observing other parts (e.g., the scene background). While human object perception is known to be strongly context-dependent, previous research has largely focused on how scene context can disambiguate fixed object properties, such as identity (e.g., a car is easier to recognize on a road than on a beach). It remains an open question whether object representations are updated dynamically based on the surrounding scene context, for example across changes in viewpoint. Here, we tested whether human observers dynamically and automatically predict the appearance of objects based on the orientation of the background scene. In three behavioral experiments (N = 152), we temporarily occluded objects within scenes that rotated. Upon the objects' reappearance, participants had to perform a perceptual discrimination task, which did not require taking the scene rotation into account. Performance on this orthogonal task strongly depended on whether objects reappeared rotated coherently with the surrounding scene or not. This effect persisted even when a majority of trials violated this real-world contingency between scene and object, showcasing the automaticity of these scene-based predictions. These findings indicate that contextual information plays an important role in predicting object transformations in structured real-world environments.

Viewpoint dependence and scene context effects generalize to depth rotated three-dimensional objects

Viewpoint effects on object recognition interact with object-scene consistency effects. While recognition of objects seen from "noncanonical" viewpoints (e.g., a cup from below) is typically impeded compared to processing of objects seen from canonical viewpoints (e.g., the string-side of a guitar), this effect is reduced by meaningful scene context information. In the present study we investigated if these findings established by using photographic images, generalize to strongly noncanonical orientations of three-dimensional (3D) models of objects. Using 3D models allowed us to probe a broad range of viewpoints and empirically establish viewpoints with very strong noncanonical and canonical orientations. In Experiment 1, we presented 3D models of objects from six different viewpoints (0°, 60°, 120°, 180° 240°, 300°) in color (1a) and grayscaled (1b) in a sequential matching task. Viewpoint had a significant effect on accuracy and response times. Based on the viewpoint effect in Experiments 1a and 1b, we could empirically determine the most canonical and noncanonical viewpoints from our set of viewpoints to use in Experiment 2. In Experiment 2, participants again performed a sequential matching task, however now the objects were paired with scene backgrounds which could be either consistent (e.g., a cup in the kitchen) or inconsistent (e.g., a guitar in the bathroom) to the object. Viewpoint interacted significantly with scene consistency in that object recognition was less affected by viewpoint when consistent scene information was provided, compared to inconsistent information. Our results show that scene context supports object recognition even when using extremely noncanonical orientations of depth rotated 3D objects. This supports the important role object-scene processing plays for object constancy especially under conditions of high uncertainty.

Objects sharpen visual scene representations: evidence from MEG decoding

Real-world scenes consist of objects, defined by local information, and scene background, defined by global information. Although objects and scenes are processed in separate pathways in visual cortex, their processing interacts. Specifically, previous studies have shown that scene context makes blurry objects look sharper, an effect that can be observed as a sharpening of object representations in visual cortex from around 300 ms after stimulus onset. Here, we use MEG to show that objects can also sharpen scene representations, with the same temporal profile. Photographs of indoor (closed) and outdoor (open) scenes were blurred such that they were difficult to categorize on their own but easily disambiguated by the inclusion of an object. Classifiers were trained to distinguish MEG response patterns to intact indoor and outdoor scenes, presented in an independent run, and tested on degraded scenes in the main experiment. Results revealed better decoding of scenes with objects than scenes alone and objects alone from 300 ms after stimulus onset. This effect was strongest over left posterior sensors. These findings show that the influence of objects on scene representations occurs at similar latencies as the influence of scenes on object representations, in line with a common predictive processing mechanism.

Head and body cues guide eye movements and facilitate target search in real-world videos

Static gaze cues presented in central vision result in observer shifts of covert attention and eye movements, and benefits in perceptual performance in the detection of simple targets. Less is known about how dynamic gazer behaviors with head and body motion influence search eye movements and performance in perceptual tasks in real-world scenes. Participants searched for a target person (yes/no task, 50% presence), whereas watching videos of one to three gazers looking at a designated person (50% valid gaze cue, looking at the target). To assess the contributions of different body parts, we digitally erase parts of the gazers in the videos to create three different body parts/whole conditions for gazers: floating heads (only head movements), headless bodies (only lower body movements), and the baseline condition with intact head and body. We show that valid dynamic gaze cues guided participants' eye movements (up to 3 fixations) closer to the target, speeded the time to foveate the target, reduced fixations to the gazers, and improved target detection. The effect of gaze cues in guiding eye movements to the search target was the smallest when the gazer's head was removed from the videos. To assess the inherent information about gaze goal location for each body parts/whole condition, we collected perceptual judgments estimating gaze goals by a separate group of observers with unlimited time. Observers' perceptual judgments showed larger estimate errors when the gazer's head was removed. This suggests that the reduced eye movement guidance from lower body cueing is related to observers' difficulty extracting gaze information without the presence of the head. Together, the study extends previous work by evaluating the impact of dynamic gazer behaviors on search with videos of real-world cluttered scenes.

Independent spatiotemporal effects of spatial attention and background clutter on human object location representations

Spatial attention helps us to efficiently localize objects in cluttered environments. However, the processing stage at which spatial attention modulates object location representations remains unclear. Here we investigated this question identifying processing stages in time and space in an EEG and fMRI experiment respectively. As both object location representations and attentional effects have been shown to depend on the background on which objects appear, we included object background as an experimental factor. During the experiments, human participants viewed images of objects appearing in different locations on blank or cluttered backgrounds while either performing a task on fixation or on the periphery to direct their covert spatial attention away or towards the objects. We used multivariate classification to assess object location information. Consistent across the EEG and fMRI experiment, we show that spatial attention modulated location representations during late processing stages (>150 ms, in middle and high ventral visual stream areas) independent of background condition. Our results clarify the processing stage at which attention modulates object location representations in the ventral visual stream and show that attentional modulation is a cognitive process separate from recurrent processes related to the processing of objects on cluttered backgrounds.

The representational hierarchy in human and artificial visual systems in the presence of object-scene regularities

Human vision is still largely unexplained. Computer vision made impressive progress on this front, but it is still unclear to which extent artificial neural networks approximate human object vision at the behavioral and neural levels. Here, we investigated whether machine object vision mimics the representational hierarchy of human object vision with an experimental design that allows testing within-domain representations for animals and scenes, as well as across-domain representations reflecting their real-world contextual regularities such as animal-scene pairs that often co-occur in the visual environment. We found that DCNNs trained in object recognition acquire representations, in their late processing stage, that closely capture human conceptual judgements about the co-occurrence of animals and their typical scenes. Likewise, the DCNNs representational hierarchy shows surprising similarities with the representational transformations emerging in domain-specific ventrotemporal areas up to domain-general frontoparietal areas. Despite these remarkable similarities, the underlying information processing differs. The ability of neural networks to learn a human-like high-level conceptual representation of object-scene co-occurrence depends upon the amount of object-scene co-occurrence present in the image set thus highlighting the fundamental role of training history. Further, although mid/high-level DCNN layers represent the category division for animals and scenes as observed in VTC, its information content shows reduced domain-specific representational richness. To conclude, by testing within- and between-domain selectivity while manipulating contextual regularities we reveal unknown similarities and differences in the information processing strategies employed by human and artificial visual systems.

Good-enough attentional guidance

Theories of attention posit that attentional guidance operates on information held in a target template within memory. The template is often thought to contain veridical target features, akin to a photograph, and to guide attention to objects that match the exact target features. However, recent evidence suggests that attentional guidance is highly flexible and often guided by non-veridical features, a subset of features, or only associated features. We integrate these findings and propose that attentional guidance maximizes search efficiency based on a 'good-enough' principle to rapidly localize candidate target objects. Candidates are then serially interrogated to make target-match decisions using more precise information. We suggest that good-enough guidance optimizes the speed-accuracy-effort trade-offs inherent in each stage of visual search.

Omit needless words: Sentence length perception

Short sentences improve readability. Short sentences also promote social justice through accessibility and inclusiveness. Despite this, much remains unknown about sentence length perception-an important factor in producing readable writing. Accordingly, we conducted a psychophysical study using procedures from Signal Detection Theory to examine sentence length perception in naive adults. Participants viewed real-world full-page text samples and judged whether a bolded target sentence contained more or fewer than 17 words. The experiment yielded four findings. First, naïve adults perceived sentence length in real-world text samples quickly (median = 300-400 ms) and precisely (median = ~90% correct). Second, flipping real-world text samples upside-down generated no reaction-time cost and nearly no loss in the precision of sentence length perception. This differs from the large inversion effects that characterize other highly practiced, real-world perceptual tasks involving canonically oriented stimuli, most notably face perception and reading. Third, participants significantly underestimated the length of mirror-reversed sentences-but not upside-down, nor standard sentences. This finding parallels participants' familiarity with commonly occurring left-justified right-ragged text, and suggests a novel demonstration of left-lateralized anchoring in scene syntax. Fourth, error patterns demonstrated that participants achieved their high speed, high precision sentence-length judgments by heuristically counting text lines, not by explicitly counting words. This suggests practical advice for writing instructors to offer students. When copy editing, students can quickly and precisely identify their long sentences via a line-counting heuristic, e.g., "a 17-word sentence spans about 1.5 text lines". Students can subsequently improve a long sentence's readability and inclusiveness by omitting needless words.

Active visual search in naturalistic environments reflects individual differences in classic visual search performance

Visual search is a ubiquitous activity in real-world environments. Yet, traditionally, visual search is investigated in tightly controlled paradigms, where head-restricted participants locate a minimalistic target in a cluttered array that is presented on a computer screen. Do traditional visual search tasks predict performance in naturalistic settings, where participants actively explore complex, real-world scenes? Here, we leverage advances in virtual reality technology to test the degree to which classic and naturalistic search are limited by a common factor, set size, and the degree to which individual differences in classic search behavior predict naturalistic search behavior in a large sample of individuals (N = 75). In a naturalistic search task, participants looked for an object within their environment via a combination of head-turns and eye-movements using a head-mounted display. Then, in a classic search task, participants searched for a target within a simple array of colored letters using only eye-movements. In each task, we found that participants' search performance was impacted by increases in set size-the number of items in the visual display. Critically, we observed that participants' efficiency in classic search tasks-the degree to which set size slowed performance-indeed predicted efficiency in real-world scenes. These results demonstrate that classic, computer-based visual search tasks are excellent models of active, real-world search behavior.

The best game in town: The reemergence of the language-of-thought hypothesis across the cognitive sciences

Mental representations remain the central posits of psychology after many decades of scrutiny. However, there is no consensus about the representational format(s) of biological cognition. This paper provides a survey of evidence from computational cognitive psychology, perceptual psychology, developmental psychology, comparative psychology, and social psychology, and concludes that one type of format that routinely crops up is the language-of-thought (LoT). We outline six core properties of LoTs: (i) discrete constituents; (ii) role-filler independence; (iii) predicate-argument structure; (iv) logical operators; (v) inferential promiscuity; and (vi) abstract content. These properties cluster together throughout cognitive science. Bayesian computational modeling, compositional features of object perception, complex infant and animal reasoning, and automatic, intuitive cognition in adults all implicate LoT-like structures. Instead of regarding LoT as a relic of the previous century, researchers in cognitive science and philosophy-of-mind must take seriously the explanatory breadth of LoT-based architectures. We grant that the mind may harbor many formats and architectures, including iconic and associative structures as well as deep-neural-network-like architectures. However, as computational/representational approaches to the mind continue to advance, classical compositional symbolic structures - that is, LoTs - only prove more flexible and well-supported over time.

Hierarchical organization of objects in scenes is reflected in mental representations of objects

The arrangement of objects in scenes follows certain rules ("Scene Grammar"), which we exploit to perceive and interact efficiently with our environment. We have proposed that Scene Grammar is hierarchically organized: scenes are divided into clusters of objects ("phrases", e.g., the sink phrase); within every phrase, one object ("anchor", e.g., the sink) holds strong predictions about identity and position of other objects ("local objects", e.g., a toothbrush). To investigate if this hierarchy is reflected in the mental representations of objects, we collected pairwise similarity judgments for everyday object pictures and for the corresponding words. Similarity judgments were stronger not only for object pairs appearing in the same scene, but also object pairs appearing within the same phrase of the same scene as opposed to appearing in different phrases of the same scene. Besides, object pairs with the same status in the scenes (i.e., being both anchors or both local objects) were judged as more similar than pairs of different status. Comparing effects between pictures and words, we found similar, significant impact of scene hierarchy on the organization of mental representation of objects, independent of stimulus modality. We conclude that the hierarchical structure of visual environment is incorporated into abstract, domain general mental representations of the world.

Context matters during pick-and-place in VR: Impact on search and transport phases

When considering external assistive systems for people with motor impairments, gaze has been shown to be a powerful tool as it is anticipatory to motor actions and is promising for understanding intentions of an individual even before the action. Up until now, the vast majority of studies investigating the coordinated eye and hand movement in a grasping task focused on single objects manipulation without placing them in a meaningful scene. Very little is known about the impact of the scene context on how we manipulate objects in an interactive task. In the present study, it was investigated how the scene context affects human object manipulation in a pick-and-place task in a realistic scenario implemented in VR. During the experiment, participants were instructed to find the target object in a room, pick it up, and transport it to a predefined final location. Thereafter, the impact of the scene context on different stages of the task was examined using head and hand movement, as well as eye tracking. As the main result, the scene context had a significant effect on the search and transport phases, but not on the reach phase of the task. The present work provides insights into the development of potential supporting intention predicting systems, revealing the dynamics of the pick-and-place task behavior once it is realized in a realistic context-rich scenario.

Statistical learning of distractor co-occurrences facilitates visual search

Visual search is facilitated by knowledge of the relationship between the target and the distractors, including both where the target is likely to be among the distractors and how it differs from the distractors. Whether the statistical structure among distractors themselves, unrelated to target properties, facilitates search is less well understood. Here, we assessed the benefit of distractor structure using novel shapes whose relationship to each other was learned implicitly during visual search. Participants searched for target items in arrays of shapes that comprised either four pairs of co-occurring distractor shapes (structured scenes) or eight distractor shapes randomly partitioned into four pairs on each trial (unstructured scenes). Across five online experiments (N = 1,140), we found that after a period of search training, participants were more efficient when searching for targets in structured than unstructured scenes. This structure benefit emerged independently of whether the position of the shapes within each pair was fixed or variable and despite participants having no explicit knowledge of the structured pairs they had seen. These results show that implicitly learned co-occurrence statistics between distractor shapes increases search efficiency. Increased efficiency in the rejection of regularly co-occurring distractors may contribute to the efficiency of visual search in natural scenes, where such regularities are abundant.

Auxiliary Scene-Context Information Provided by Anchor Objects Guides Attention and Locomotion in Natural Search Behavior

Successful adaptive behavior requires efficient attentional and locomotive systems. Previous research has thoroughly investigated how we achieve this efficiency during natural behavior by exploiting prior knowledge related to targets of our actions (e.g., attending to metallic targets when looking for a pot) and to the environmental context (e.g., looking for the pot in the kitchen). Less is known about whether and how individual nontarget components of the environment support natural behavior. In our immersive virtual reality task, 24 adult participants searched for objects in naturalistic scenes in which we manipulated the presence and arrangement of large, static objects that anchor predictions about targets (e.g., the sink provides a prediction for the location of the soap). Our results show that gaze and body movements in this naturalistic setting are strongly guided by these anchors. These findings demonstrate that objects auxiliary to the target are incorporated into the representations guiding attention and locomotion.

Semantic Scene-Object Consistency Modulates N300/400 EEG Components, but Does Not Automatically Facilitate Object Representations

During natural vision, objects rarely appear in isolation, but often within a semantically related scene context. Previous studies reported that semantic consistency between objects and scenes facilitates object perception and that scene-object consistency is reflected in changes in the N300 and N400 components in EEG recordings. Here, we investigate whether these N300/400 differences are indicative of changes in the cortical representation of objects. In two experiments, we recorded EEG signals, while participants viewed semantically consistent or inconsistent objects within a scene; in Experiment 1, these objects were task-irrelevant, while in Experiment 2, they were directly relevant for behavior. In both experiments, we found reliable and comparable N300/400 differences between consistent and inconsistent scene-object combinations. To probe the quality of object representations, we performed multivariate classification analyses, in which we decoded the category of the objects contained in the scene. In Experiment 1, in which the objects were not task-relevant, object category could be decoded from ~100 ms after the object presentation, but no difference in decoding performance was found between consistent and inconsistent objects. In contrast, when the objects were task-relevant in Experiment 2, we found enhanced decoding of semantically consistent, compared with semantically inconsistent, objects. These results show that differences in N300/400 components related to scene-object consistency do not index changes in cortical object representations but rather reflect a generic marker of semantic violations. Furthermore, our findings suggest that facilitatory effects between objects and scenes are task-dependent rather than automatic.

Priming effects in inefficient visual search: Real, but transient

In visual search tasks, responses to targets on one trial can influence responses on the next trial. Most typically, target repetition speeds response while switching to a different target slows response. Such “priming” effects have sometimes been given very significant roles in theories of search (e.g., Theeuwes, Philosophical Transactions of the Royal Society B: Biological Sciences, 368, 1628, 2013). Most work on priming has involved “singleton” or “popout” tasks. In non-popout priming tasks, observers must often perform a task-switching operation because the guiding template for one target (e.g., a red vertical target in a conjunction task) is incompatible with efficient search for the other target (green horizontal, in this example). We examined priming in inefficient search where the priming feature (Color: Experiments 1–3, Shape: Experiments 4–5) was irrelevant to the task of finding a T among Ls. We wished to determine if finding a red T on one trial helped observers to be more efficient if the next T was also red. In all experiments, we found additive priming effects. The reaction time (RT) for the second trial was shorter if the color of the T was repeated. However, there was no interaction with set size. The slope of the RT × Set Size function was not shallower for runs of the same target color, compared to trials where the target color switched. We propose that priming might produce transient guidance of the earliest deployments of attention on the next trial or it might speed decisions about a selected target. Priming does not appear to guide attention over the entire search.

Ten simple rules to study distractor suppression

Distractor suppression refers to the ability to filter out distracting and task-irrelevant information. Distractor suppression is essential for survival and considered a key aspect of selective attention. Despite the recent and rapidly evolving literature on distractor suppression, we still know little about how the brain suppresses distracting information. What limits progress is that we lack mutually agreed upon principles of how to study the neural basis of distractor suppression and its manifestation in behavior. Here, we offer ten simple rules that we believe are fundamental when investigating distractor suppression. We provide guidelines on how to design conclusive experiments on distractor suppression (Rules 1-3), discuss different types of distractor suppression that need to be distinguished (Rules 4-6), and provide an overview of models of distractor suppression and considerations of how to evaluate distractor suppression statistically (Rules 7-10). Together, these rules provide a concise and comprehensive synopsis of promising advances in the field of distractor suppression. Following these rules will propel research on distractor suppression in important ways, not only by highlighting prominent issues to both new and more advanced researchers in the field, but also by facilitating communication between sub-disciplines.

Gaze-cued shifts of attention and microsaccades are sustained for whole bodies but are transient for body parts

Gaze direction is an evolutionarily important mechanism in daily social interactions. It reflects a person’s internal cognitive state, spatial locus of interest, and predicts future actions. Studies have used static head images presented foveally and simple synthetic tasks to find that gaze orients attention and facilitates target detection at the cued location in a sustained manner. Little is known about how people’s natural gaze behavior, including eyes, head, and body movements, jointly orient covert attention, microsaccades, and facilitate performance in more ecological dynamic scenes. Participants completed a target person detection task with videos of real scenes. The videos showed people looking toward (valid cue) or away from a target (invalid cue) location. We digitally manipulated the individuals in the videos directing gaze to create three conditions: whole-intact (head and body movements), floating heads (only head movements), and headless bodies (only body movements). We assessed their impact on participants’ behavioral performance and microsaccades during the task. We show that, in isolation, an individual’s head or body orienting toward the target-person direction led to facilitation in detection that is transient in time (200 ms). In contrast, only the whole-intact condition led to sustained facilitation (500 ms). Furthermore, observers executed microsaccades more frequently towards the cued direction for valid trials, but this bias was sustained in time only with the joint presence of head and body parts. Together, the results differ from previous findings with foveally presented static heads. In more real-world scenarios and tasks, sustained attention requires the presence of the whole-intact body of the individuals dynamically directing their gaze.

Multiple spatial frames for immersive working memory

As we move around, relevant information that disappears from sight can still be held in working memory to serve upcoming behaviour. How we maintain and select visual information as we move through the environment remains poorly understood because most laboratory tasks of working memory rely on removing visual material while participants remain still. We used virtual reality to study visual working memory following self-movement in immersive environments. Directional biases in gaze revealed the recruitment of more than one spatial frame for maintaining and selecting memoranda following self-movement. The findings bring the important realization that multiple spatial frames support working memory in natural behaviour. The results also illustrate how virtual reality can be a critical experimental tool to characterize this core memory system.

Meaning maps detect the removal of local semantic scene content but deep saliency models do not

Meaning mapping uses human raters to estimate different semantic features in scenes, and has been a useful tool in demonstrating the important role semantics play in guiding attention. However, recent work has argued that meaning maps do not capture semantic content, but like deep learning models of scene attention, represent only semantically-neutral image features. In the present study, we directly tested this hypothesis using a diffeomorphic image transformation that is designed to remove the meaning of an image region while preserving its image features. Specifically, we tested whether meaning maps and three state-of-the-art deep learning models were sensitive to the loss of semantic content in this critical diffeomorphed scene region. The results were clear: meaning maps generated by human raters showed a large decrease in the diffeomorphed scene regions, while all three deep saliency models showed a moderate increase in the diffeomorphed scene regions. These results demonstrate that meaning maps reflect local semantic content in scenes while deep saliency models do something else. We conclude the meaning mapping approach is an effective tool for estimating semantic content in scenes.

Ten simple rules to study distractor suppression

Distractor suppression refers to the ability to filter out distracting and task-irrelevant information. Distractor suppression is essential for survival and considered a key aspect of selective attention. Despite the recent and rapidly evolving literature on distractor suppression, we still know little about how the brain suppresses distracting information. What limits progress is that we lack mutually agreed upon principles of how to study the neural basis of distractor suppression and its manifestation in behavior. Here, we offer ten simple rules that we believe are fundamental when investigating distractor suppression. We provide guidelines on how to design conclusive experiments on distractor suppression (Rules 1–3), discuss different types of distractor suppression that need to be distinguished (Rules 4–6), and provide an overview of models of distractor suppression and considerations of how to evaluate distractor suppression statistically (Rules 7–10). Together, these rules provide a concise and comprehensive synopsis of promising advances in the field of distractor suppression. Following these rules will propel research on distractor suppression in important ways, not only by highlighting prominent issues to both new and more advanced researchers in the field, but also by facilitating communication between sub-disciplines.

•

Distractor suppression is the ability to filter out irrelevant information.

•

At present, we know little about how the brain suppresses distraction.

•

We offer ten rules that are fundamental when investigating distractor suppression.

•

Following the rules will propel research and foster interaction between disciplines.

Perception of semantic relations in scenes: A registered report study of attention hold

To what extent the semantic relations present in scenes guide spatial attention automatically remains a matter of debate. Considering that spatial attention can be understood as a sequence of shifts, engagements, and disengagements, semantic relations might affect each stage of this process differently. Therefore, we investigated whether objects that violate semantic rules engage attention for longer than objects that are expected in a given context. The experiment involved a central presentation of a distractor scene that contained a semantically congruent or incongruent object, and a peripheral presentation of a small target letter. We found that incongruent scenes did not delay responses to the peripheral target, which indicates that they did not hold attention for longer than congruent scenes. Therefore, by showing that violations of semantic relations do not engage attention automatically, our study contributes to a better understanding of how attention operates in naturalistic settings.

The spatiotemporal neural dynamics of object location representations in the human brain

To interact with objects in complex environments, we must know what they are and where they are in spite of challenging viewing conditions. Here, we investigated where, how and when representations of object location and category emerge in the human brain when objects appear on cluttered natural scene images using a combination of functional magnetic resonance imaging, electroencephalography and computational models. We found location representations to emerge along the ventral visual stream towards lateral occipital complex, mirrored by gradual emergence in deep neural networks. Time-resolved analysis suggested that computing object location representations involves recurrent processing in high-level visual cortex. Object category representations also emerged gradually along the ventral visual stream, with evidence for recurrent computations. These results resolve the spatiotemporal dynamics of the ventral visual stream that give rise to representations of where and what objects are present in a scene under challenging viewing conditions.

Humans represent the precision and utility of information acquired across fixations

Our environment contains an abundance of objects which humans interact with daily, gathering visual information using sequences of eye-movements to choose which object is best-suited for a particular task. This process is not trivial, and requires a complex strategy where task affordance defines the search strategy, and the estimated precision of the visual information gathered from each object may be used to track perceptual confidence for object selection. This study addresses the fundamental problem of how such visual information is metacognitively represented and used for subsequent behaviour, and reveals a complex interplay between task affordance, visual information gathering, and metacogntive decision making. People fixate higher-utility objects, and most importantly retain metaknowledge about how much information they have gathered about these objects, which is used to guide perceptual report choices. These findings suggest that such metacognitive knowledge is important in situations where decisions are based on information acquired in a temporal sequence.

Semantic object-scene inconsistencies affect eye movements, but not in the way predicted by contextualized meaning maps

Semantic information is important in eye movement control. An important semantic influence on gaze guidance relates to object-scene relationships: objects that are semantically inconsistent with the scene attract more fixations than consistent objects. One interpretation of this effect is that fixations are driven toward inconsistent objects because they are semantically more informative. We tested this explanation using contextualized meaning maps, a method that is based on crowd-sourced ratings to quantify the spatial distribution of context-sensitive “meaning” in images. In Experiment 1, we compared gaze data and contextualized meaning maps for images, in which objects-scene consistency was manipulated. Observers fixated more on inconsistent versus consistent objects. However, contextualized meaning maps did not assign higher meaning to image regions that contained semantic inconsistencies. In Experiment 2, a large number of raters evaluated image-regions, which were deliberately selected for their content and expected meaningfulness. The results suggest that the same scene locations were experienced as slightly less meaningful when they contained inconsistent compared to consistent objects. In summary, we demonstrated that — in the context of our rating task — semantically inconsistent objects are experienced as less meaningful than their consistent counterparts and that contextualized meaning maps do not capture prototypical influences of image meaning on gaze guidance.