Is attention necessary for object identification? Evidence from eye movements during the inspection of real-world scenes

Is a knife the same as a plunger? Comparing the attentional effects of weapons and non-threatening unusual objects in dynamic scenes

Observers' memory for a person's appearance can be compromised by the presence of a weapon, a phenomenon known as the weapon-focus effect (WFE). According to the unusual-item hypothesis, attention shifts from the perpetrator to the weapon because a weapon is an unusual object in many contexts. To test this assumption, we monitored participants' eye movements while they watched a mock-crime video. The video was presented with sound and featured a female perpetrator holding either a weapon, a non-threatening unusual object, or a neutral object. Contrary to the predictions of current theories, there were no significant differences in total viewing times for the three objects. For the perpetrator, total viewing time was reduced when she held the non-threatening unusual object, but not when she held the weapon. However, weapon presence led to an attentional shift from the perpetrator's face toward her body. Detailed time-course analyses revealed that the effects of object type were more pronounced during early scene viewing. Thus, our results do not support the idea of extended attentional shifts from the perpetrator toward the unusual objects, but instead suggest more complex attentional effects. Contrary to previous research, memory for the perpetrator's appearance was not affected by object type. Thus, there was no WFE. An additional online experiment using the same videos and methodology produced a WFE, but this effect disappeared when the videos were presented without sound.

Determining the efficacy of visual inspections at detecting non-biosecurity-compliant goods

Examination of imported commodities by trained inspectors searching for pest organisms is a common practice that phytosanitary regulatory agencies use to mitigate biosecurity risks along trade pathways. To investigate the effects of target size and color on the efficacy of these visual assessments, we affixed square decals to polystyrene models of mandarins. Sample units of 100 model fruit containing up to 10 marked models were examined by inspectors. Six sizes in six shades of brown were tested across two prevalence levels. The experiment consisted of five inspection rounds where 11 inspectors examined 77 sample units within an allocated time. The probability that decals were detected increased with mark size and color contrast. Smaller, low-contrast marks were mainly missed. The prevalence rate did not affect the detectability. Through the experiment, the false-positive rate dropped from 6% to 3%, whereas false-negative rates were constant throughout. Large, dark targets were readily found with a mean recall of >90%, whereas small, pale marks had a mean recall of 9%. Increased experience made inspectors more competent at recognizing decals, reducing the false positive rate. However, constant false-negative rates indicate that experience did not prevent inspectors from overlooking targets they could not perceive.

ObScene database: Semantic congruency norms for 898 pairs of object-scene pictures

Research on the interaction between object and scene processing has a long history in the fields of perception and visual memory. Most databases have established norms for pictures where the object is embedded in the scene. In this study, we provide a diverse and controlled stimulus set comprising real-world pictures of 375 objects (e.g., suitcase), 245 scenes (e.g., airport), and 898 object-scene pairs (e.g., suitcase-airport), with object and scene presented separately. Our goal was twofold. First, to create a database of object and scene pictures, normed for the same variables to have comparable measures for both types of pictures. Second, to acquire normative data for the semantic relationships between objects and scenes presented separately, which offers more flexibility in the use of the pictures and allows disentangling the processing of the object and its context (the scene). Along three experiments, participants evaluated each object or scene picture on name agreement, familiarity, and visual complexity, and rated object-scene pairs on semantic congruency. A total of 125 septuplets of one scene and six objects (three congruent, three incongruent), and 120 triplets of one object and two scenes (in congruent and incongruent pairings) were built. In future studies, these objects and scenes can be used separately or combined, while controlling for their key features. Additionally, as object-scene pairs received semantic congruency ratings along the entire scale, researchers may select among a wide range of congruency values. ObScene is a comprehensive and ecologically valid database, useful for psychology and neuroscience studies of visual object and scene processing.

The role of memory color in visual attention

The expected color of an object influences how it is perceived. For example, a banana in a greyscale photo may appear slightly yellow because bananas are expected to be yellow. This phenomenon is known as the memory color effect (MCE), and the objects with a memory color are called "color-diagnostic." The MCE is theorized to be a top-down influence of color knowledge on visual perception. However, its validity has been questioned because most evidence for the MCE is based on subjective reports. Here a change detection task is used as an objective measure of the effect and the results show that change detection differs for color-diagnostic objects. Specifically, it was predicted and found that unnaturally colored color-diagnostic objects (e.g., a blue banana) would attract attention and thus be discovered more quickly and accurately. In the experiment, two arrays alternated with the target present in one array and absent in the other while all other objects remained unchanged. Participants had to find the target as quickly and accurately as possible. In the experimental condition, the targets were color-diagnostic objects (e.g., a banana) presented in either their natural (yellow) or an unnatural (blue) color. In the control condition, non-color-diagnostic objects (e.g., a mug) were presented with the same colors as the color-diagnostic objects. Unnaturally colored color-diagnostic objects were found more quickly, which suggests that the MCE is a top-down, preattentive process that can influence a nonsubjective visual perceptual task such as change detection.

Predictive processing of scenes and objects

Real-world visual input consists of rich scenes that are meaningfully composed of multiple objects which interact in complex, but predictable, ways. Despite this complexity, we recognize scenes, and objects within these scenes, from a brief glance at an image. In this review, we synthesize recent behavioral and neural findings that elucidate the mechanisms underlying this impressive ability. First, we review evidence that visual object and scene processing is partly implemented in parallel, allowing for a rapid initial gist of both objects and scenes concurrently. Next, we discuss recent evidence for bidirectional interactions between object and scene processing, with scene information modulating the visual processing of objects, and object information modulating the visual processing of scenes. Finally, we review evidence that objects also combine with each other to form object constellations, modulating the processing of individual objects within the object pathway. Altogether, these findings can be understood by conceptualizing object and scene perception as the outcome of a joint probabilistic inference, in which "best guesses" about objects act as priors for scene perception and vice versa, in order to concurrently optimize visual inference of objects and scenes.

Hand Motions Reveal Attentional Status and Subliminal Semantic Processing: A Mouse-Tracking Technique

Theories of embodied cognition suggest that hand motions and cognition are closely interconnected. An emerging technique of tracking how participants move a computer mouse (i.e., the mouse-tracking technique) has shown advantages over the traditional response time measurement to detect implicit cognitive conflicts. Previous research suggests that attention is essential for subliminal processing to take place at a semantic level. However, this assumption is challenged by evidence showing the presence of subliminal semantic processing in the near-absence of attention. The inconsistency of evidence could stem from the insufficient sensitivity in the response time measurement. Therefore, we examined the role of attention in subliminal semantic processing by analyzing participants' hand motions using the mouse-tracking technique. The results suggest that subliminal semantic processing is not only enhanced by attention but also occurs when attention is disrupted, challenging the necessity of facilitated top-down attention for subliminal semantic processing, as claimed by a number of studies. In addition, by manipulating the color of attentional cues, our experiment shows that the cue color per se could influence participants' response patterns. Overall, the current study suggests that attentional status and subliminal semantic processing can be reliably revealed by temporal-spatial features extracted from cursor motion trajectories.

Perception of semantic relations in scenes: A registered report study of attention hold

To what extent the semantic relations present in scenes guide spatial attention automatically remains a matter of debate. Considering that spatial attention can be understood as a sequence of shifts, engagements, and disengagements, semantic relations might affect each stage of this process differently. Therefore, we investigated whether objects that violate semantic rules engage attention for longer than objects that are expected in a given context. The experiment involved a central presentation of a distractor scene that contained a semantically congruent or incongruent object, and a peripheral presentation of a small target letter. We found that incongruent scenes did not delay responses to the peripheral target, which indicates that they did not hold attention for longer than congruent scenes. Therefore, by showing that violations of semantic relations do not engage attention automatically, our study contributes to a better understanding of how attention operates in naturalistic settings.

To Grasp the World at a Glance: The Role of Attention in Visual and Semantic Associative Processing

Associative relations among words, concepts and percepts are the core building blocks of high-level cognition. When viewing the world ‘at a glance’, the associative relations between objects in a scene, or between an object and its visual background, are extracted rapidly. The extent to which such relational processing requires attentional capacity, however, has been heavily disputed over the years. In the present manuscript, I review studies investigating scene–object and object–object associative processing. I then present a series of studies in which I assessed the necessity of spatial attention to various types of visual–semantic relations within a scene. Importantly, in all studies, the spatial and temporal aspects of visual attention were tightly controlled in an attempt to minimize unintentional attention shifts from ‘attended’ to ‘unattended’ regions. Pairs of stimuli—either objects, scenes or a scene and an object—were briefly presented on each trial, while participants were asked to detect a pre-defined target category (e.g., an animal, a nonsense shape). Response times (RTs) to the target detection task were registered when visual attention spanned both stimuli in a pair vs. when attention was focused on only one of two stimuli. Among non-prioritized stimuli that were not defined as to-be-detected targets, findings consistently demonstrated rapid associative processing when stimuli were fully attended, i.e., shorter RTs to associated than unassociated pairs. Focusing attention on a single stimulus only, however, largely impaired this relational processing. Notably, prioritized targets continued to affect performance even when positioned at an unattended location, and their associative relations with the attended items were well processed and analyzed. Our findings portray an important dissociation between unattended task-irrelevant and task-relevant items: while the former require spatial attentional resources in order to be linked to stimuli positioned inside the attentional focus, the latter may influence high-level recognition and associative processes via feature-based attentional mechanisms that are largely independent of spatial attention.

Contribution of Eye-Tracking to Study Cognitive Impairments Among Clinical Populations

In the field of psychology, the merge of decision-theory and neuroscientific methods produces an array of scientifically recognized paradigms. For example, by exploring consumer’s eye-movement behavior, researchers aim to deepen the understanding of how patterns of retinal activation are being meaningfully transformed into visual experiences and connected with specific reactions (e.g., purchase). Notably, eye-movements provide knowledge of one’s homeostatic balance and gatekeep information that shape decisions. Hence, vision science investigates the quality of observed environments determined under various experimental conditions. Moreover, it answers questions on how human process visual stimuli and use gained information for a successful strategy to achieve certain goals. While capturing cognitive states with the support of the eye-trackers progresses at a relatively fast pace in decision-making research, measuring the visual performance of real-life tasks, which require complex cognitive skills, is tentatively translated into clinical experiments. Nevertheless, the potential of the human eye as a highly valuable source of biomarkers has been underlined. In this article, we aim to draw readers attention to decision-making experimental paradigms supported with eye-tracking technology among clinical populations. Such interdisciplinary approach may become an important component that will (i) help in objectively illustrating patient’s models of beliefs and values, (ii) support clinical interventions, and (iii) contribute to health services. It is possible that shortly, eye-movement data from decision-making experiments will grant the scientific community a greater understanding of mechanisms underlining mental states and consumption practices that medical professionals consider as obsessions, disorders or addiction.

Effects of taillight shape on conspicuity of vehicles at night

Taillight shape in a vehicle provides an essential lighting signal that enables the vehicle to be seen from the rear at night, thereby preventing rear-end crashes. This study aims to investigate the effects of taillight shape on vehicle conspicuity, and proposes ergonomic taillight shape solutions to vehicle designers and manufacturers. Two complementary experiments were conducted to examine three types of taillight shapes at three design levels. The first experiment was designed to investigate the detection speed of a driver and the fixation duration and fixation counts on leading vehicles with different taillight shapes, based on an eye-tracking methodology. The second experiment was designed to investigate the dynamic visual searching performance of a trailing driver for leading vehicles with different taillight shapes, based on a visual search task. The experimental results indicated that a long line-shaped taillight (striplight) was the optimal ergonomic solution for enhancing vehicle conspicuity. Vehicles with an enclosed contour-shaped taillight were more salient than those with an open contour-shaped taillight. Moreover, the experience and gender of the driver and the vehicle-observer distance were found to be closely related to vehicle conspicuity, and therefore, must be considered by vehicle designers when applying a specific taillight shape design. This study provides insights into the taillight shape design that not only aid vehicle designers or manufacturers in enhancing vehicle safety but also enable potential vehicle buyers to choose a safe lighting system.

The effect of target salience and size in visual search within naturalistic scenes under degraded vision

We address two questions concerning eye guidance during visual search in naturalistic scenes. First, search has been described as a task in which visual salience is unimportant. Here, we revisit this question by using a letter-in-scene search task that minimizes any confounding effects that may arise from scene guidance. Second, we investigate how important the different regions of the visual field are for different subprocesses of search (target localization, verification). In Experiment 1, we manipulated both the salience (low vs. high) and the size (small vs. large) of the target letter (a "T"), and we implemented a foveal scotoma (radius: 1°) in half of the trials. In Experiment 2, observers searched for high- and low-salience targets either with full vision or with a central or peripheral scotoma (radius: 2.5°). In both experiments, we found main effects of salience with better performance for high-salience targets. In Experiment 1, search was faster for large than for small targets, and high-salience helped more for small targets. When searching with a foveal scotoma, performance was relatively unimpaired regardless of the target's salience and size. In Experiment 2, both visual-field manipulations led to search time costs, but the peripheral scotoma was much more detrimental than the central scotoma. Peripheral vision proved to be important for target localization, and central vision for target verification. Salience affected eye movement guidance to the target in both central and peripheral vision. Collectively, the results lend support for search models that incorporate salience for predicting eye-movement behavior.

The interplay between gaze and consistency in scene viewing: Evidence from visual search by young and older adults

Searching for an object in a complex scene is influenced by high-level factors such as how much the item would be expected in that setting (semantic consistency). There is also evidence that a person gazing at an object directs our attention towards it. However, there has been little previous research that has helped to understand how we integrate top-down cues such as semantic consistency and gaze to direct attention when searching for an object. Also, there are separate lines of evidence to suggest that older adults may be more influenced by semantic factors and less by gaze cues compared to younger counterparts, but this has not been investigated before in an integrated task. In the current study we analysed eye-movements of 34 younger and 30 older adults as they searched for a target object in complex visual scenes. Younger adults were influenced by semantic consistency in their attention to objects, but were more influenced by gaze cues. In contrast, older adults were more guided by semantic consistency in directing their attention, and showed less influence from gaze cues. These age differences in use of high-level cues were apparent early in processing (time to first fixation and probability of immediate fixation) but not in later processing (total time looking at objects and time to make a response). Overall, this pattern of findings indicates that people are influenced by both social cues and prior expectations when processing a complex scene, and the relative importance of these factors depends on age.

You won't believe what this guy is doing with the potato: The ObjAct stimulus-set depicting human actions on congruent and incongruent objects

Perception famously involves both bottom-up and top-down processes. The latter are influenced by our previous knowledge and expectations about the world. In recent years, many studies have focused on the role of expectations in perception in general, and in object processing in particular. Yet studying this question is not an easy feat, requiring-among other things-the creation and validation of appropriate stimuli. Here, we introduce the ObjAct stimulus-set of free-to-use, highly controlled real-life scenes, on which critical objects are pasted. All scenes depict human agents performing an action with an object that is either congruent or incongruent with the action. The focus on human actions yields highly constraining contexts, strengthening congruency effects. The stimuli were analyzed for low-level properties, using the SHINE toolbox to control for luminance and contrast, and using a deep convolutional neural network to mimic V1 processing and potentially discover other low-level factors that might differ between congruent and incongruent scenes. Two online validation studies (N = 500) were also conducted to assess the congruency manipulation and collect additional ratings of our images (e.g., arousal, likeability, visual complexity). We also provide full descriptions of the online sources from which all images were taken, as well as verbal descriptions of their content. Taken together, this extensive validation and characterization procedure makes the ObjAct stimulus-set highly informative and easy to use for future researchers in multiple fields, from object and scene processing, through top-down contextual effects, to the study of actions.

The meaning and structure of scenes

We live in a rich, three dimensional world with complex arrangements of meaningful objects. For decades, however, theories of visual attention and perception have been based on findings generated from lines and color patches. While these theories have been indispensable for our field, the time has come to move on from this rather impoverished view of the world and (at least try to) get closer to the real thing. After all, our visual environment consists of objects that we not only look at, but constantly interact with. Having incorporated the meaning and structure of scenes, i.e. its "grammar", then allows us to easily understand objects and scenes we have never encountered before. Studying this grammar provides us with the fascinating opportunity to gain new insights into the complex workings of attention, perception, and cognition. In this review, I will discuss how the meaning and the complex, yet predictive structure of real-world scenes influence attention allocation, search, and object identification.

Reciprocal semantic predictions drive categorization of scene contexts and objects even when they are separate

Visual categorization improves when object-context associations in scenes are semantically consistent, thus predictable from schemas stored in long-term memory. However, it is unclear whether this is due to differences in early perceptual processing, in matching of memory representations or in later stages of response selection. We tested these three concurrent explanations across five experiments. At each trial, participants had to categorize a scene context and an object briefly presented within the same image (Experiment 1), or separately in simultaneous images (Experiments 2–5). We analyzed unilateral (Experiments 1, 3) and bilateral presentations (Experiments 2, 4, 5), and presentations on the screen’s horizontal midline (Experiments 1–2) and in the upper and lower visual fields (Experiments 3, 4). In all the experiments, we found a semantic consistency advantage for both context categorization and object categorization. This shows that the memory for object-context semantic associations is activated regardless of whether these two scene components are integrated in the same percept. Our study suggests that the facilitation effect of semantic consistency on categorization occurs at the stage of matching the percept with previous knowledge, supporting the object selection account and extending this framework to an object-context reciprocal influence on matching processes (object-context selection account).

Fixation-related Brain Potentials during Semantic Integration of Object–Scene Information

In vision science, a particularly controversial topic is whether and how quickly the semantic information about objects is available outside foveal vision. Here, we aimed at contributing to this debate by coregistering eye movements and EEG while participants viewed photographs of indoor scenes that contained a semantically consistent or inconsistent target object. Linear deconvolution modeling was used to analyze the ERPs evoked by scene onset as well as the fixation-related potentials (FRPs) elicited by the fixation on the target object (t) and by the preceding fixation (t − 1). Object–scene consistency did not influence the probability of immediate target fixation or the ERP evoked by scene onset, which suggests that object–scene semantics was not accessed immediately. However, during the subsequent scene exploration, inconsistent objects were prioritized over consistent objects in extrafoveal vision (i.e., looked at earlier) and were more effortful to process in foveal vision (i.e., looked at longer). In FRPs, we demonstrate a fixation-related N300/N400 effect, whereby inconsistent objects elicit a larger frontocentral negativity than consistent objects. In line with the behavioral findings, this effect was already seen in FRPs aligned to the pretarget fixation t − 1 and persisted throughout fixation t, indicating that the extraction of object semantics can already begin in extrafoveal vision. Taken together, the results emphasize the usefulness of combined EEG/eye movement recordings for understanding the mechanisms of object–scene integration during natural viewing.

Automatic Attention Capture by Threatening, But Not by Semantically Incongruent Natural Scene Images

Visual objects are typically perceived as parts of an entire visual scene, and the scene’s context provides information crucial in the object recognition process. Fundamental insights into the mechanisms of context-object integration have come from research on semantically incongruent objects, which are defined as objects with a very low probability of occurring in a given context. However, the role of attention in processing of the context-object mismatch remains unclear, with some studies providing evidence in favor, but other against an automatic capture of attention by incongruent objects. Therefore, in the present study, 25 subjects completed a dot-probe task, in which pairs of scenes—congruent and incongruent or neutral and threatening—were presented as task-irrelevant distractors. Importantly, threatening scenes are known to robustly capture attention and thus were included in the present study to provide a context for interpretation of results regarding incongruent scenes. Using N2 posterior-contralateral ERP component as a primary measure, we revealed that threatening images indeed capture attention automatically and rapidly, but semantically incongruent scenes do not benefit from an automatic attentional selection. Thus, our results suggest that identification of the context-object mismatch is not preattentive.

Extrafoveal attentional capture by object semantics

There is ongoing debate on whether object meaning can be processed outside foveal vision, making semantics available for attentional guidance. Much of the debate has centred on whether objects that do not fit within an overall scene draw attention, in complex displays that are often difficult to control. Here, we revisited the question by reanalysing data from three experiments that used displays consisting of standalone objects from a carefully controlled stimulus set. Observers searched for a target object, as per auditory instruction. On the critical trials, the displays contained no target but objects that were semantically related to the target, visually related, or unrelated. Analyses using (generalized) linear mixed-effects models showed that, although visually related objects attracted most attention, semantically related objects were also fixated earlier in time than unrelated objects. Moreover, semantic matches affected the very first saccade in the display. The amplitudes of saccades that first entered semantically related objects were larger than 5° on average, confirming that object semantics is available outside foveal vision. Finally, there was no semantic capture of attention for the same objects when observers did not actively look for the target, confirming that it was not stimulus-driven. We discuss the implications for existing models of visual cognition.

Investigation of eye movement pattern parameters of individuals with different fluid intelligence

Eye movement studies are subject of interest in human cognition. Cortical activity and cognitive load impress eye movement influentially. Here, we investigated whether fluid intelligence (FI) has any effect on eye movement pattern in a comparative visual search (CVS) task. FI of individuals was measured using the Cattell test, and participants were divided into three groups: low FI, middle FI, and high FI. Eye movements of individuals were then recorded during the CVS task. Eye movement patterns were extracted and compared statistically among the three groups. Our experiment demonstrated that eye movement patterns were significantly different among the three groups. Pearson correlation coefficients between FI and eye movement parameters were also calculated to assess which of the eye movement parameters were most affected by FI. Our findings illustrate that saccade peak velocity had the greatest positive correlation with FI score and the ratio of total fixation duration to total saccade duration had the greatest negative correlation with FI. Next, we extracted 24 features from eye movement patterns and designed: (1) a classifier to categorize individuals and (2) a regression analysis to predict the FI score of individuals. In the best case examined, the classifier categorized subjects with 68.3% accuracy, and the regression predicted FI of individuals with a 0.54 correlation between observed FI and predicted FI. In our investigation, the results have emphasized that imposed loads on low FI individuals is greater than that of high FI individuals in the cognitive load tasks.

Stuck on semantics: Processing of irrelevant object-scene inconsistencies modulates ongoing gaze behavior

People have an amazing ability to identify objects and scenes with only a glimpse. How automatic is this scene and object identification? Are scene and object semantics-let alone their semantic congruity-processed to a degree that modulates ongoing gaze behavior even if they are irrelevant to the task at hand? Objects that do not fit the semantics of the scene (e.g., a toothbrush in an office) are typically fixated longer and more often than objects that are congruent with the scene context. In this study, we overlaid a letter T onto photographs of indoor scenes and instructed participants to search for it. Some of these background images contained scene-incongruent objects. Despite their lack of relevance to the search, we found that participants spent more time in total looking at semantically incongruent compared to congruent objects in the same position of the scene. Subsequent tests of explicit and implicit memory showed that participants did not remember many of the inconsistent objects and no more of the consistent objects. We argue that when we view natural environments, scene and object relationships are processed obligatorily, such that irrelevant semantic mismatches between scene and object identity can modulate ongoing eye-movement behavior.

The Effect of Consistency on Short-Term Memory for Scenes

Which is more detectable, the change of a consistent or an inconsistent object in a scene? This question has been debated for decades. We noted that the change of objects in scenes might simultaneously be accompanied with gist changes. In the present study we aimed to examine how the alteration of gist, as well as the consistency of the changed objects, modulated change detection. In Experiment 1, we manipulated the semantic content by either keeping or changing the consistency of the scene. Results showed that the changes of consistent and inconsistent scenes were equally detected. More importantly, the changes were more accurately detected when scene consistency changed than when the consistency remained unchanged, regardless of the consistency of the memory scenes. A phase-scrambled version of stimuli was adopted in Experiment 2 to decouple the possible confounding effect of low-level factors. The results of Experiment 2 demonstrated that the effect found in Experiment 1 was indeed due to the change of high-level semantic consistency rather than the change of low-level physical features. Together, the study suggests that the change of consistency plays an important role in scene short-term memory, which might be attributed to the sensitivity to the change of semantic content.

Modulation of scene consistency and task demand on language-driven eye movements for audio-visual integration

Previous psycholinguistic studies have demonstrated that people tend to direct fixations toward the visual object to which spoken input refers during language comprehension. However, it is still unclear how the visual scene, especially the semantic consistency between object and background, affects the word-object mapping process during comprehension. Two visual world paradigm experiments were conducted to investigate how the scene consistency dynamically influenced the language-driven eye movements in a speech comprehension and a scene comprehension task. In each trial, participants listened to a spoken sentence while viewing a picture with two critical objects: one is the mentioned target object (e.g., tiger), which was embedded in either a consistent (e.g., field), inconsistent (e.g., sky) or blank background; the other is an unmentioned non-target object (e.g., eagle), which was always consistent with its background. The results showed that the fixation proportion of the inconsistent target was higher than the consistent target, and the task demand can affect the strength and the direction of the inconsistency effect before and after the target had been mentioned. In summary, the spoken language, scene-based knowledge and task demand were intertwined to determine eye movements during audio-visual integration for comprehension.

Delineating the effect of semantic congruency on episodic memory: the role of integration and relatedness

A fundamental challenge in the study of learning and memory is to understand the role of existing knowledge in the encoding and retrieval of new episodic information. The importance of prior knowledge in memory is demonstrated in the congruency effect-the robust finding wherein participants display better memory for items that are compatible, rather than incompatible, with their pre-existing semantic knowledge. Despite its robustness, the mechanism underlying this effect is not well understood. In four studies, we provide evidence that demonstrates the privileged explanatory power of the elaboration-integration account over alternative hypotheses. Furthermore, we question the implicit assumption that the congruency effect pertains to the truthfulness/sensibility of a subject-predicate proposition, and show that congruency is a function of semantic relatedness between item and context words.

Synchronous contextual irregularities affect early scene processing: replication and extension

Whether contextual regularities facilitate perceptual stages of scene processing is widely debated, and empirical evidence is still inconclusive. Specifically, it was recently suggested that contextual violations affect early processing of a scene only when the incongruent object and the scene are presented a-synchronously, creating expectations. We compared event-related potentials (ERPs) evoked by scenes that depicted a person performing an action using either a congruent or an incongruent object (e.g., a man shaving with a razor or with a fork) when scene and object were presented simultaneously. We also explored the role of attention in contextual processing by using a pre-cue to direct subjects׳ attention towards or away from the congruent/incongruent object. Subjects׳ task was to determine how many hands the person in the picture used in order to perform the action. We replicated our previous findings of frontocentral negativity for incongruent scenes that started ~ 210 ms post stimulus presentation, even earlier than previously found. Surprisingly, this incongruency ERP effect was negatively correlated with the reaction times cost on incongruent scenes. The results did not allow us to draw conclusions about the role of attention in detecting the regularity, due to a weak attention manipulation. By replicating the 200-300 ms incongruity effect with a new group of subjects at even earlier latencies than previously reported, the results strengthen the evidence for contextual processing during this time window even when simultaneous presentation of the scene and object prevent the formation of prior expectations. We discuss possible methodological limitations that may account for previous failures to find this an effect, and conclude that contextual information affects object model selection processes prior to full object identification, with semantic knowledge activation stages unfolding only later on.

Serial and parallel processes in eye movement control: current controversies and future directions

In this editorial for the special issue on serial and parallel processing in reading we explore the background to the current debate concerning whether the word recognition processes in reading are strictly serial-sequential or take place in an overlapping parallel fashion. We consider the history of the controversy and some of the underlying assumptions, together with an analysis of the types of evidence and arguments that have been adduced to both sides of the debate, concluding that both accounts necessarily presuppose some weakening of, or elasticity in, the eye-mind assumption. We then consider future directions, both for reading research and for scene viewing, and wrap up the editorial with a brief overview of the following articles and their conclusions.

The interplay of bottom-up and top-down mechanisms in visual guidance during object naming

An ongoing issue in visual cognition concerns the roles played by low- and high-level information in guiding visual attention, with current research remaining inconclusive about the interaction between the two. In this study, we bring fresh evidence into this long-standing debate by investigating visual saliency and contextual congruency during object naming (Experiment 1), a task in which visual processing interacts with language processing. We then compare the results of this experiment to data of a memorization task using the same stimuli (Experiment 2). In Experiment 1, we find that both saliency and congruency influence visual and naming responses and interact with linguistic factors. In particular, incongruent objects are fixated later and less often than congruent ones. However, saliency is a significant predictor of object naming, with salient objects being named earlier in a trial. Furthermore, the saliency and congruency of a named object interact with the lexical frequency of the associated word and mediate the time-course of fixations at naming. In Experiment 2, we find a similar overall pattern in the eye-movement responses, but only the congruency of the target is a significant predictor, with incongruent targets fixated less often than congruent targets. Crucially, this finding contrasts with claims in the literature that incongruent objects are more informative than congruent objects by deviating from scene context and hence need a longer processing. Overall, this study suggests that different sources of information are interactively used to guide visual attention on the targets to be named and raises new questions for existing theories of visual attention.

Modeling guidance and recognition in categorical search: bridging human and computer object detection

Search is commonly described as a repeating cycle of guidance to target-like objects, followed by the recognition of these objects as targets or distractors. Are these indeed separate processes using different visual features? We addressed this question by comparing observer behavior to that of support vector machine (SVM) models trained on guidance and recognition tasks. Observers searched for a categorically defined teddy bear target in four-object arrays. Target-absent trials consisted of random category distractors rated in their visual similarity to teddy bears. Guidance, quantified as first-fixated objects during search, was strongest for targets, followed by target-similar, medium-similarity, and target-dissimilar distractors. False positive errors to first-fixated distractors also decreased with increasing dissimilarity to the target category. To model guidance, nine teddy bear detectors, using features ranging in biological plausibility, were trained on unblurred bears then tested on blurred versions of the same objects appearing in each search display. Guidance estimates were based on target probabilities obtained from these detectors. To model recognition, nine bear/nonbear classifiers, trained and tested on unblurred objects, were used to classify the object that would be fixated first (based on the detector estimates) as a teddy bear or a distractor. Patterns of categorical guidance and recognition accuracy were modeled almost perfectly by an HMAX model in combination with a color histogram feature. We conclude that guidance and recognition in the context of search are not separate processes mediated by different features, and that what the literature knows as guidance is really recognition performed on blurred objects viewed in the visual periphery.

Image ambiguity and fluency

Ambiguity is often associated with negative affective responses, and enjoying ambiguity seems restricted to only a few situations, such as experiencing art. Nevertheless, theories of judgment formation, especially the “processing fluency account”, suggest that easy-to-process (non-ambiguous) stimuli are processed faster and are therefore preferred to (ambiguous) stimuli, which are hard to process. In a series of six experiments, we investigated these contrasting approaches by manipulating fluency (presentation duration: 10ms, 50ms, 100ms, 500ms, 1000ms) and testing effects of ambiguity (ambiguous versus non-ambiguous pictures of paintings) on classification performance (Part A; speed and accuracy) and aesthetic appreciation (Part B; liking and interest). As indicated by signal detection analyses, classification accuracy increased with presentation duration (Exp. 1a), but we found no effects of ambiguity on classification speed (Exp. 1b). Fifty percent of the participants were able to successfully classify ambiguous content at a presentation duration of 100 ms, and at 500ms even 75% performed above chance level. Ambiguous artworks were found more interesting (in conditions 50ms to 1000ms) and were preferred over non-ambiguous stimuli at 500ms and 1000ms (Exp. 2a - 2c, 3). Importantly, ambiguous images were nonetheless rated significantly harder to process as non-ambiguous images. These results suggest that ambiguity is an essential ingredient in art appreciation even though or maybe because it is harder to process.

Optimal and Preferred Eye Landing Positions in Objects and Scenes

Viewing position effects are commonly observed in reading, but they have only rarely been investigated in object perception or in the realistic context of a natural scene. In two experiments, we explored where people fixate within photorealistic objects and the effects of this landing position on recognition and subsequent eye movements. The results demonstrate an optimal viewing position—objects are processed more quickly when fixation is in the centre of the object. Viewers also prefer to saccade to the centre of objects within a natural scene, even when making a large saccade. A central landing position is associated with an increased likelihood of making a refixation, a result that differs from previous reports and suggests that multiple fixations within objects, within scenes, occur for a range of reasons. These results suggest that eye movements within scenes are systematic and are made with reference to an early parsing of the scene into constituent objects.

Throwing the banana away and keeping the peel: neuroelectric responses to unexpected but physically feasible action endings

Integrative semantic processing and mirror-neuron system activation in response to incongruence detection during action observation have so far been investigated only through paradigms that involve unfeasible action endings, introduction of extraneous objects or synthetic pictures. The ecological validity of the reported effects, particularly modulation of the N400 event-related potential (ERP), therefore remains questionable. To address this shortcoming, we investigated the ERPs evoked by physically feasible and naturalistic but unexpected action endings. We hypothesized that in comparison to existing work our more realistic scenarios would elicit slower processing, due to engagement of deeper analysis as opposed to fast detection based on automatic processes. ERPs were recorded in 14 healthy participants in response to 60 sequences, each containing three pictures, the last of which could be an expected or unexpected action ending. The N400 was larger for the first image compared to the others, and the LPP was enhanced for unexpected endings. Source localization implicated anterior fronto-temporal and temporo-parietal junction regions in generation of these potentials. Our findings challenge the view that the N400 universally indices verbal and action-perception incongruence detection, and point to situation-contingent effects which, in the case of subtle violations, are reflected in later activity components, speculatively reflecting recourse to the "mentalizing" system.

Semantic consistency versus perceptual salience in visual scenes: findings from change detection

In a one-shot change detection task, we investigated the relationship between semantic properties (high consistency, i.e., diagnosticity, versus inconsistency with regard to gist) and perceptual properties (high versus low salience) of objects in guiding attention in visual scenes and in constructing scene representations. To produce the change an object was added or deleted in either the right or left half of coloured drawings of daily-life events. Diagnostic object deletions were more accurately detected than inconsistent ones, indicating rapid inclusion into early scene representation for the most predictable objects. Detection was faster and more accurate for high salience than for low salience changes. An advantage was found for diagnostic object changes in the high salience condition, although it was limited to additions when considering response speed. For inconsistent objects of high salience, deletions were detected faster than additions. These findings may indicate that objects are primarily selected on a perceptual basis with subsequent and supplementary effect of semantic consistency, in the sense of facilitation due to object diagnosticity or lengthening of processing time due to inconsistency.

The attraction of visual attention to texts in real-world scenes

When we look at real-world scenes, attention seems disproportionately attracted by texts that are embedded in these scenes, for instance, on signs or billboards. The present study was aimed at verifying the existence of this bias and investigating its underlying factors. For this purpose, data from a previous experiment were reanalyzed and four new experiments measuring eye movements during the viewing of real-world scenes were conducted. By pairing text objects with matching control objects and regions, the following main results were obtained: (a) Greater fixation probability and shorter minimum fixation distance of texts confirmed the higher attractiveness of texts; (b) the locations where texts are typically placed contribute partially to this effect; (c) specific visual features of texts, rather than typically salient features (e.g., color, orientation, and contrast), are the main attractors of attention; (d) the meaningfulness of texts does not add to their attentional capture; and (e) the attraction of attention depends to some extent on the observer's familiarity with the writing system and language of a given text.

Object-scene inconsistencies do not capture gaze: evidence from the flash-preview moving-window paradigm

In the present study, we investigated the influence of object-scene relationships on eye movement control during scene viewing. We specifically tested whether an object that is inconsistent with its scene context is able to capture gaze from the visual periphery. In four experiments, we presented rendered images of naturalistic scenes and compared baseline consistent objects with semantically, syntactically, or both semantically and syntactically inconsistent objects within those scenes. To disentangle the effects of extrafoveal and foveal object-scene processing on eye movement control, we used the flash-preview moving-window paradigm: A short scene preview was followed by an object search or free viewing of the scene, during which visual input was available only via a small gaze-contingent window. This method maximized extrafoveal processing during the preview but limited scene analysis to near-foveal regions during later stages of scene viewing. Across all experiments, there was no indication of an attraction of gaze toward object-scene inconsistencies. Rather than capturing gaze, the semantic inconsistency of an object weakened contextual guidance, resulting in impeded search performance and inefficient eye movement control. We conclude that inconsistent objects do not capture gaze from an initial glimpse of a scene.

Computational versus psychophysical bottom-up image saliency: a comparative evaluation study

The predictions of 13 computational bottom-up saliency models and a newly introduced Multiscale Contrast Conspicuity(MCC) metric are compared with human visual conspicuity measurements. The agreement between human visual conspicuity estimates and model saliency predictions is quantified through their rank order correlation. The maximum of the computational saliency value over the target support area correlates most strongly with visual conspicuity for 12 of the 13 models. A simple multiscale contrast model and the MCC metric both yield the largest correlation with human visual target conspicuity (>0:84). Local image saliency largely determines human visual inspection and interpretation of static and dynamic scenes. Computational saliency models therefore have a wide range of important applications, like adaptive content delivery, region-of-interest-based image compression, video summarization, progressive image transmission, image segmentation, image quality assessment, object recognition, and content-aware image scaling. However, current bottom-up saliency models do not incorporate important visual effects like crowding and lateral interaction. Additional knowledge about the exact nature of the interactions between the mechanisms mediating human visual saliency is required to develop these models further. The MCC metric and its associated psychophysical saliency measurement procedure are useful tools to systematically investigate the relative contribution of different feature dimensions to overall visual target saliency.

Predictive coding as a model of the V1 saliency map hypothesis

The predictive coding/biased competition (PC/BC) model is a specific implementation of the predictive coding theory that has previously been shown to provide a detailed account of the response properties of orientation tuned cells in primary visual cortex (V1). Here it is shown that the same model can successfully simulate psychophysical data relating to the saliency of unique items in search arrays, of contours embedded in random texture, and of borders between textured regions. This model thus provides a possible implementation of the hypothesis that V1 generates a bottom-up saliency map. However, PC/BC is very different from previous models of visual salience, in that it proposes that saliency results from the failure of an internal model of simple elementary image components to accurately predict the visual input. Saliency can therefore be interpreted as a mechanism by which prediction errors attract attention in an attempt to improve the accuracy of the brain's internal representation of the world.