Looking, language, and memory: bridging research from the visual world and visual search paradigms

Exploring the Semantic-Inconsistency Effect in Scenes Using a Continuous Measure of Linguistic-Semantic Similarity

Viewers use contextual information to visually explore complex scenes. Object recognition is facilitated by exploiting object-scene relations (which objects are expected in a given scene) and object-object relations (which objects are expected because of the occurrence of other objects). Semantically inconsistent objects deviate from these expectations, so they tend to capture viewers' attention (the semantic-inconsistency effect). Some objects fit the identity of a scene more or less than others, yet semantic inconsistencies have hitherto been operationalized as binary (consistent vs. inconsistent). In an eye-tracking experiment (N = 21 adults), we study the semantic-inconsistency effect in a continuous manner by using the linguistic-semantic similarity of an object to the scene category and to other objects in the scene. We found that both highly consistent and highly inconsistent objects are viewed more than other objects (U-shaped relationship), revealing that the (in)consistency effect is more than a simple binary classification.

Processing reduced speech in the L1 and L2: a combined eye-tracking and ERP study

We examined the cognitive processes underlying the comprehension of reduced word pronunciation variants in natives and advanced learners of French. In a passive listening visual world task, participants heard sentences containing either a reduced or a full form and saw pictures representing the target word, a phonological competitor and two neutral distractors. After each sentence they saw a picture and had to decide whether it matched the content of that sentence. Eye movements and EEG were recorded simultaneously. Because the two recordings offer complementary information about cognitive processes, we developed methods for analysing the signals in combination. We found a stronger effect of reduction on phonetic processing and semantic integration in learners than in natives, but the effects are different from the N100/N400 and P600 effects found in previous research. Time-locking EEG signals on fixation moments in the eye movements offers a window onto the time course of semantic integration.

Sound-meaning associations allow listeners to infer the meaning of foreign language words

An attribute of human language is the seemingly arbitrary association between a word's form and meaning. We provide evidence that the meaning of foreign words can be partially deduced from phonological form. Monolingual English speakers listened to 45 antonym word pairs in nine foreign languages and judged which English words corresponded to these words' respective meanings. Despite no proficiency in the foreign language tested, participants' accuracy was higher than chance in each language. Words that shared meaning across languages were more likely to share phonological form. Accuracy in judging meaning from form was associated with participants' verbal working memory and with how consistently phonological and semantic features of words covaried across unrelated languages. A follow-up study with native Spanish speakers replicated the results. We conclude that sound maps to meaning in natural languages with some regularity, and sensitivity to form-meaning mappings indexes broader cognitive functions.

A network model of referent identification by toddlers in a visual world task

We present a neural network model of referent identification in a visual world task. Inputs are visual representations of item pairs unfolding with sequences of phonemes identifying the target item. The model is trained to output the semantic representation of the target and to suppress the distractor. The training set uses a 200-word lexicon typically known by toddlers. The phonological, visual, and semantic representations are derived from real corpora. Successful performance requires correct association between labels and visual and semantic representations, as well as correct location identification. The model reproduces experimental evidence that phonological, perceptual, and categorical relationships modulate item preferences. The model provides an account of how language can drive visual attention in the inter-modal preferential looking task.

Looking for a word or for its meaning? The impact of induction tasks on adolescents' visual search for verbal material

An eye-tracking experiment was conducted to examine whether the pre-activation of different word-processing pathways by means of semantic versus perceptual induction tasks could modify the way adults and 11- to 15-year-old adolescents searched for single target words within displays of nine words. The presence within the search displays of words either looking like the target word or semantically related to the target word was manipulated. The quality of participants' lexical representations was evaluated through three word-identification and vocabulary tests. Performing a semantic induction task rather than a perceptual one on the target word before searching for it increased search times by 15% in all age groups, reflecting an increase in both the number and duration of gazes directed to non-target words. Moreover, performing the semantic induction task increased the impact of distractor words that were semantically related to the target word on search efficiency. Participants' search efficiency increased with age because of a progressive increase in the quality of adolescents' lexical representations, which allowed participants to more quickly reject the distractors on which they fixated. Indeed, lexical quality scores explained 43% of the variance in search times independently of participants' age. In the simple visual search task used in this study, fostering semantic word processing through the semantic induction task slowed down visual search. However, the literature suggests that semantic induction tasks could, in contrast, help people find information more easily in more complex verbal environments where the meaning of words must be accessed to find task-relevant information.

Recognizing non-native spoken words in background noise increases interference from the native language

Listeners frequently recognize spoken words in the presence of background noise. Previous research has shown that noise reduces phoneme intelligibility and hampers spoken-word recognition - especially for non-native listeners. In the present study, we investigated how noise influences lexical competition in both the non-native and the native language, reflecting the degree to which both languages are co-activated. We recorded the eye movements of native Dutch participants as they listened to English sentences containing a target word while looking at displays containing four objects. On target-present trials, the visual referent depicting the target word was present, along with three unrelated distractors. On target-absent trials, the target object (e.g., wizard) was absent. Instead, the display contained an English competitor, overlapping with the English target in phonological onset (e.g., window), a Dutch competitor, overlapping with the English target in phonological onset (e.g., wimpel, pennant), and two unrelated distractors. Half of the sentences was masked by speech-shaped noise; the other half was presented in quiet. Compared to speech in quiet, noise delayed fixations to the target objects on target-present trials. For target-absent trials, we observed that the likelihood for fixation biases towards the English and Dutch onset competitors (over the unrelated distractors) was larger in noise than in quiet. Our data thus show that the presence of background noise increases lexical competition in the task-relevant non-native (English) and in the task-irrelevant native (Dutch) language. The latter reflects stronger interference of one's native language during non-native spoken-word recognition under adverse conditions.

Decreased gray matter volume in the right middle temporal gyrus associated with cognitive dysfunction in preeclampsia superimposed on chronic hypertension

Introduction

The effects of preeclampsia superimposed on chronic hypertension (CHTN-PE) on the structure and function of the human brain are mostly unknown. The purpose of this study was to examine altered gray matter volume (GMV) and its correlation with cognitive function in pregnant healthy women, healthy non-pregnant individuals, and CHTN-PE patients.

Methods

Twenty-five CHTN-PE patients, thirty-five pregnant healthy controls (PHC) and thirty-five non-pregnant healthy controls (NPHC) were included in this study and underwent cognitive assessment testing. A voxel-based morphometry (VBM) approach was applied to investigate variations in brain GMV among the three groups. Pearson's correlations between mean GMV and the Stroop color-word test (SCWT) scores were calculated.

Results

Compared with the NPHC group, the PHC and CHTN-PE groups showed significantly decreased GMV in a cluster of the right middle temporal gyrus (MTG), and the GMV decrease was more significant in the CHTN-PE group. There were significant differences in the Montreal Cognitive Assessment (MoCA) and Stroop word scores among the three groups. Notably, the mean GMV values in the right MTG cluster were not only significantly negatively correlated with Stroop word and Stroop color scores but also significantly distinguished CHTN-PE patients from the NPHC and PHC groups in receiver operating characteristic curve analysis.

Discussion

Pregnancy may cause a decrease in local GMV in the right MTG, and the GMV decrease is more significant in CHTN-PE patients. The right MTG affects multiple cognitive functions, and combined with the SCWT scores, it may explain the decline in speech motor function and cognitive flexibility in CHTN-PE patients.

Not all is forgotten: Children's associative matrices for features of a word learning episode

Word learning studies traditionally examine the narrow link between words and objects, indifferent to the rich contextual information surrounding objects. This research examined whether children attend to this contextual information and construct an associative matrix of the words, objects, people, and environmental context during word learning. In Experiment 1, preschool-aged children (age: 3;2-5;11 years) were presented with novel words and objects in an animated storybook. Results revealed that children constructed associations beyond words and objects. Specifically, children attended to and had the strongest associations for features of the environmental context but failed to learn word-object associations. Experiment 2 demonstrated that children (age: 3;0-5;8 years) leveraged strong associations for the person and environmental context to support word-object mapping. This work demonstrates that children are especially sensitive to the word learning context and use associative matrices to support word mapping. Indeed, this research suggests associative matrices of the environment may be foundational for children's vocabulary development.

Anticipatory Processing in a Verb-Initial Mayan Language: Eye-Tracking Evidence During Sentence Comprehension in Tseltal

We present a visual world eye-tracking study on Tseltal (a Mayan language) and investigate whether verbal information can be used to anticipate an upcoming referent. Basic word order in transitive sentences in Tseltal is Verb-Object-Subject (VOS). The verb is usually encountered first, making argument structure and syntactic information available at the outset, which should facilitate anticipation of the post-verbal arguments. Tseltal speakers listened to verb-initial sentences with either an object-predictive verb (e.g., "eat") or a general verb (e.g., "look for") (e.g., "Ya slo'/sle ta stukel on te kereme," Is eating/is looking (for) by himself the avocado the boy/ "The boy is eating/is looking (for) an avocado by himself") while seeing a visual display showing one potential referent (e.g., avocado) and three distractors (e.g., bag, toy car, coffee grinder). We manipulated verb type (predictive vs. general) and recorded participants' eye movements while they listened and inspected the visual scene. Participants' fixations to the target referent were analyzed using multilevel logistic regression models. Shortly after hearing the predictive verb, participants fixated the target object before it was mentioned. In contrast, when the verb was general, fixations to the target only started to increase once the object was heard. Our results suggest that Tseltal hearers pre-activate semantic features of the grammatical object prior to its linguistic expression. This provides evidence from a verb-initial language for online incremental semantic interpretation and anticipatory processing during language comprehension. These processes are comparable to the ones identified in subject-initial languages, which is consistent with the notion that different languages follow similar universal processing principles.

Individual differences in attentional control predict working memory capacity in adults who stutter

PURPOSE

Prior research has suggested that people who stutter exhibit differences in some working memory tasks, particularly when more phonologically complex stimuli are used. This study aimed to further specify working memory differences in adults who stutter by not only accounting for linguistic demands of the stimuli but also individual differences in attentional control and experimental influences, such as concomitant processing requirements.

METHOD

This study included 40 adults who stutter and 42 adults who do not stutter who completed the Attention Network Test (ANT; Fan et al., 2002) and three complex span working memory tasks: the Operation Span (OSPAN), Rotation Span, and Symmetry Span (Draheim et al., 2018; Foster et al., 2015; Unsworth et al., 2005, 2009). All complex span tasks were dual-tasks and varied in linguistic content in task stimuli.

RESULTS

Working memory capacities demonstrated by adults who stutter paralleled the hierarchy of linguistic content across the three complex span tasks, with statistically significant between-group differences in working memory capacity apparent in the task with the highest linguistic demand (i.e., OSPAN). Individual differences in attentional control in adults who stutter also significantly predicted working memory capacity on the OSPAN.

DISCUSSION

Findings from this study extend existing working memory research in stuttering by showing that: (1) significant working memory differences are present between adults who stutter and adults who do not stutter even using relatively simple linguistic stimuli in dual-task working memory conditions; (2) adults who stutter with stronger executive control of attention demonstrate working memory capacity more comparable to adults who do not stutter on the OSPAN compared to adults who stutter with lower executive control of attention.

Preserved Extra-Foveal Processing of Object Semantics in Alzheimer's Disease

Alzheimer's disease (AD) patients underperform on a range of tasks requiring semantic processing, but it is unclear whether this impairment is due to a generalised loss of semantic knowledge or to issues in accessing and selecting such information from memory. The objective of this eye-tracking visual search study was to determine whether semantic expectancy mechanisms known to support object recognition in healthy adults are preserved in AD patients. Furthermore, as AD patients are often reported to be impaired in accessing information in extra-foveal vision, we investigated whether that was also the case in our study. Twenty AD patients and 20 age-matched controls searched for a target object among an array of distractors presented extra-foveally. The distractors were either semantically related or unrelated to the target (e.g., a car in an array with other vehicles or kitchen items). Results showed that semantically related objects were detected with more difficulty than semantically unrelated objects by both groups, but more markedly by the AD group. Participants looked earlier and for longer at the critical objects when these were semantically unrelated to the distractors. Our findings show that AD patients can process the semantics of objects and access it in extra-foveal vision. This suggests that their impairments in semantic processing may reflect difficulties in accessing semantic information rather than a generalised loss of semantic memory.

Language is activated by visual input regardless of memory demands or capacity

In the present study, we provide compelling evidence that viewing objects automatically activates linguistic labels and that this activation is not due to task-specific memory demands. In two experiments, eye-movements of English speakers were tracked while they identified a visual target among an array of four images, including a phonological competitor (e.g., flower-flag). Experiment 1 manipulated the capacity to subvocally rehearse the target label by imposing linguistic, spatial, or no working memory load. Experiment 2 manipulated the need to encode target objects by presenting target images either before or concurrently with the search display. While the timing and magnitude of competitor activation varied across conditions, we observed consistent evidence of language activation regardless of the capacity or need to maintain object labels in memory. We propose that language activation is automatic and not contingent upon working memory capacity or demands, and conclude that objects' labels influence visual search.

The Processing of the Second Syllable in Recognizing Chinese Disyllabic Spoken Words: Evidence From Eye Tracking

In the current study, two experiments were conducted to investigate the processing of the second syllable (which was considered as the rhyme at the word level) during Chinese disyllabic spoken word recognition using a printed-word paradigm. In Experiment 1, participants heard a spoken target word and were simultaneously presented with a visual display of four printed words: a target word, a phonological competitor, and two unrelated distractors. The phonological competitors were manipulated to share either full phonemic overlap of the second syllable with targets (the syllabic overlap condition; e.g., , xiao3zhuan4, "calligraphy" vs. , gong1zhuan4, "revolution") or the initial phonemic overlap of the second syllable (the sub-syllabic overlap condition; e.g., , yuan2zhu4, "cylinder" vs. , gong1zhuan4, "revolution") with targets. Participants were asked to select the target words and their eye movements were simultaneously recorded. The results did not show any phonological competition effect in either the syllabic overlap condition or the sub-syllabic overlap condition. In Experiment 2, to maximize the likelihood of observing the phonological competition effect, a target-absent version of the printed-word paradigm was adopted, in which target words were removed from the visual display. The results of Experiment 2 showed significant phonological competition effects in both conditions, i.e., more fixations were made to the phonological competitors than to the distractors. Moreover, the phonological competition effect was found to be larger in the syllabic overlap condition than in the sub-syllabic overlap condition. These findings shed light on the effect of the second syllable competition at the word level during spoken word recognition and, more importantly, showed that the initial phonemes of the second syllable at the syllabic level are also accessed during Chinese disyllabic spoken word recognition.

Memory after visual search: Overlapping phonology, shared meaning, and bilingual experience influence what we remember

How we remember the things that we see can be shaped by our prior experiences. Here, we examine how linguistic and sensory experiences interact to influence visual memory. Objects in a visual search that shared phonology (cat-cast) or semantics (dog-fox) with a target were later remembered better than unrelated items. Phonological overlap had a greater influence on memory when targets were cued by spoken words, while semantic overlap had a greater effect when targets were cued by characteristic sounds. The influence of overlap on memory varied as a function of individual differences in language experience -- greater bilingual experience was associated with decreased impact of overlap on memory. We conclude that phonological and semantic features of objects influence memory differently depending on individual differences in language experience, guiding not only what we initially look at, but also what we later remember.

I see what you mean: Semantic but not lexical factors modulate image processing in bilingual adults

Bilinguals frequently juggle competing representations from their two languages when they interact with their environment (i.e., nonselective activation). As a result, both first (L1) and second language (L2) communication may be impeded when words share orthographic form but not meaning (i.e., interlingual homographs; e.g., CRANE, a machine in English, a skull in French). Similarly, bilinguals' reduced exposure to each known language makes bilingual lexical processing more vulnerable to larger frequency effects. While much is known about processes within the language system, less is known about how the bilingual language system interacts with the visual system, specifically in the context of image processing. We investigated this by testing whether commonly observed semantic (homograph interference) and lexical (frequency) effects extend to a visual word-image matching task. We tested 48 bilinguals, who were asked to determine whether an image corresponded to a written word that was presented immediately beforehand. By modulating the complexity of visual referents and the semantic (Analysis 1) or lexical (Analysis 2) complexity of word cues, we simultaneously burdened the visual and language systems. The results showed that both semantic and lexical factors modulated response accuracy and correct reaction time on the word-image matching task. Crucially, we observed an interaction between the image factor (visual complexity) with the semantic (homograph status) but not the lexical factor (word frequency). We conclude that it is possible for the language and image processing systems to interact, although the extent to which this occurs depends on the degree of linguistic processing involved.

Information stored in memory affects abductive reasoning

Abductive reasoning describes the process of deriving an explanation from given observations. The theory of abductive reasoning (TAR; Johnson and Krems, Cognitive Science 25:903-939, 2001) assumes that when information is presented sequentially, new information is integrated into a mental representation, a situation model, the central data structure on which all reasoning processes are based. Because working memory capacity is limited, the question arises how reasoning might change with the amount of information that has to be processed in memory. Thus, we conducted an experiment (N = 34) in which we manipulated whether previous observation information and previously found explanations had to be retrieved from memory or were still visually present. Our results provide evidence that people experience differences in task difficulty when more information has to be retrieved from memory. This is also evident in changes in the mental representation as reflected by eye tracking measures. However, no differences are found between groups in the reasoning outcome. These findings suggest that individuals construct their situation model from both information in memory as well as external memory stores. The complexity of the model depends on the task: when memory demands are high, only relevant information is included. With this compensation strategy, people are able to achieve similar reasoning outcomes even when faced with tasks that are more difficult. This implies that people are able to adapt their strategy to the task in order to keep their reasoning successful.

From simple agents to information sources: Readers' differential processing of story characters as a function of story consistency

The study examined how readers integrate information from and about multiple information sources into a memory representation. In two experiments, college students read brief news reports containing two critical statements, each attributed to a source character. In half of the texts, the statements were consistent with each other, in the other half they were discrepant. Each story also featured a non-source character (who made no statement). The hypothesis was that discrepant statements, as compared to consistent statements, would promote distinct attention and memory only for the source characters. Experiment 1 used short interviews to assess participants' ability to recognize the source of one of the statements after reading. Experiment 2 used eye-tracking to collect data during reading and during a source-content recognition task after reading. As predicted, discrepancies only enhanced memory of, and attention to source-related segments of the texts. Discrepancies also enhanced the link between the two source characters in memory as opposed to the non-source character, as indicated by the participants' justifications (Experiment 1) and their visual inspection of the recognition items (Experiment 2). The results are interpreted within current theories of text comprehension and document literacy.

Towards Understanding the Task Dependency of Embodied Language Processing: The Influence of Colour During Language-Vision Interactions

A main challenge for theories of embodied cognition is to understand the task dependency of embodied language processing. One possibility is that perceptual representations (e.g., typical colour of objects mentioned in spoken sentences) are not activated routinely but the influence of perceptual representation emerges only when context strongly supports their involvement in language. To explore this question, we tested the effects of colour representations during language processing in three visual-world eye-tracking experiments. On critical trials, participants listened to sentence-embedded words associated with a prototypical colour (e.g., ‘…spinach…’) while they inspected a visual display with four printed words (Experiment 1), coloured or greyscale line drawings (Experiment 2) and a ‘blank screen’ after a preview of coloured or greyscale line drawings (Experiment 3). Visual context always presented a word/object (e.g., frog) associated with the same prototypical colour (e.g. green) as the spoken target word and three distractors. When hearing spinach participants did not prefer the written word frog compared to other distractor words (Experiment 1). In Experiment 2, colour competitors attracted more overt attention compared to average distractors, but only for the coloured condition and not for greyscale trials. Finally, when the display was removed at the onset of the sentence, and in contrast to the previous blank-screen experiments with semantic competitors, there was no evidence of colour competition in the eye-tracking record (Experiment 3). These results fit best with the notion that the main role of perceptual representations in language processing is to contextualize language in the immediate environment.

When It's Harder to Ignorar than to Ignore: Evidence of Greater Attentional Capture from a Non-Dominant Language

AIMS AND OBJECTIVES

Imagine you're driving and you become so distracted by the radio that you miss your turn. Which is more likely to have caught your attention, a broadcast in your native tongue or one in your second language? The present study explores the effect of language proficiency on our ability to inhibit irrelevant phonological information.

METHODOLOGY

Participants were asked to identify which of two drawings changed color while ignoring irrelevant words in either their native language, English, or a less proficient language, Spanish. The drawings appeared on screen for either 200 or 2000ms prior to word-onset, which was followed 200ms later by a color-change. On critical trials, the irrelevant word shared phonological features with the label of the non-target drawing. Trials were blocked by preview time and language.

DATA AND ANALYSIS

Reaction time data from 19 bilinguals were analyzed utilizing generalized linear mixed-effects models, with fixed effects of Competition (competitor vs. control), and Language (English vs. Spanish) and random effects for Subject and Item within each preview window.

FINDINGS/CONCLUSIONS

No interference was observed when participants heard their native tongue in either preview condition. However, participants in the long-preview condition were significantly slower to respond when there was phonological competition in their less proficient language, despite the fact that the task required no language processing.

ORIGINALITY

Past work has indicated that languages are processed more automatically and cause greater interference as proficiency increases. We propose that though higher-proficiency languages may receive greater activation overall, lower-proficiency languages may be more likely to exogenously capture attention due to both relatively greater salience, and relatively less control.

SIGNIFICANCE

The present findings have implications for how we understand the dynamic relationship between language proficiency, activation, and inhibition, suggesting that the salience of the less familiar influences our ability to ignore irrelevant information.

Spatial language difficulties reflect the structure of intact spatial representation: Evidence from high-functioning autism

Previous studies have shown that the basic properties of the visual representation of space are reflected in spatial language. This close relationship between linguistic and non-linguistic spatial systems has been observed both in typical development and in some developmental disorders. Here we provide novel evidence for structural parallels along with a degree of autonomy between these two systems among individuals with Autism Spectrum Disorder, a developmental disorder with uneven cognitive and linguistic profiles. In four experiments, we investigated language and memory for locations organized around an axis-based reference system. Crucially, we also recorded participants' eye movements during the tasks in order to provide new insights into the online processes underlying spatial thinking. Twenty-three intellectually high-functioning individuals with autism (HFA) and 23 typically developing controls (TD), all native speakers of Norwegian matched on chronological age and cognitive abilities, participated in the studies. The results revealed a well-preserved axial reference system in HFA and weakness in the representation of direction within the axis, which was especially evident in spatial language. Performance on the non-linguistic tasks did not differ between HFA and control participants, and we observed clear structural parallels between spatial language and spatial representation in both groups. However, there were some subtle differences in the use of spatial language in HFA compared to TD, suggesting that despite the structural parallels, some aspects of spatial language in HFA deviated from the typical pattern. These findings provide novel insights into the prominence of the axial reference systems in non-linguistic spatial representations and spatial language, as well as the possibility that the two systems are, to some degree, autonomous.

Verb-mediated anticipatory eye movements in people with Down syndrome

BACKGROUND

Children and adults with neurotypical development employ linguistic information to predict and anticipate information. Individuals with Down syndrome (DS) have weaknesses in language production and the domain of grammar but relative strengths in language comprehension and the domain of semantics. What is not clear is the extent to which they can use linguistic information, as it unfolds in real time, to anticipate upcoming information correctly.

AIMS

To investigate whether children and young people with DS employ verb information to predict and anticipate upcoming linguistic information.

METHODS & PROCEDURES

A preferential looking task was performed, using an eye-tracker, with children and teenagers with DS and a typically developing (TD) control group matched by sex and mental age (average = 5.48 years). In each of 10 trials, two images were presented, a target and a distractor, while participants heard a phrase that contained a semantically informative verb (e.g., 'eat') or an uninformative verb (e.g., 'see').

OUTCOMES & RESULTS

Both DS and TD control participants could anticipate the target upon hearing an informative verb, and prediction skills were positively correlated with mental age in those with DS.

CONCLUSIONS & IMPLICATIONS

This work demonstrates for the first time that children and teenagers with DS can predict linguistic information based on semantic cues from verbs, and that sentence processing is driven by predictive relationships between verbs and arguments, as in children with typical development. Clinicians can take advantage of these prediction skills, using them in therapy to support weaker areas.

Whereof one cannot speak: How language and capture of visual attention interact

Our research addresses the important question whether language influences cognition by studying crosslinguistic differences in nonlinguistic visual search tasks. We investigated whether capture of visual attention is mediated by characteristics corresponding to concepts that are differently expressed across different languages. Korean grammatically distinguishes between tight- (kkita) and loose-fit (nehta) containment whereas German collapses them into a single semantic category (in). Although linguistic processing was neither instructed nor necessary to perform the visual search task, we found that Korean speakers showed attention capture by non-instructed but target-coincident (Experiment 1) or distractor-coincident (Experiments 4 and 5) spatial fitness of the stimuli, whereas German speakers were not sensitive to it. As the tight- versus loose-fit distinction is grammaticalized only in the Korean but not the German language, our results demonstrate that language influences which visual features capture attention even in non-linguistic tasks that do not require paying attention to these features. In separate control experiments (Experiments 2 and 3), we ruled out cultural or general cognitive group differences between Korean and German speaking participants as alternative explanations. We outline the mechanisms underlying these crosslinguistic differences in nonlinguistic visual search behaviors. This is the first study showing that linguistic spatial relational concepts held in long-term memory can affect attention capture in visual search tasks.

Modulation of parafoveal word processing by cognitive load during modified visual search tasks

During visual search for simple items, the amount of information that can be processed in parafoveal vision depends on the cognitive resources that are available. However, whether this applies to the semantic processing of words remains controversial. This work was designed to manipulate simultaneously two sources of cognitive load to study their impact on the depth of parafoveal word processing during a modified visual search task. The participants had to search for target words among parafoveally presented semantic, orthographic or target-unrelated distractor words while their electroencephalogram was recorded. The task-related load was manipulated by either giving target words in advance (literal task) or giving only a semantic clue to define them (categorical task). The foveal load was manipulated by displaying either a word or hash symbols at the centre of the screen. Parafoveal orthographic and semantic distractors had an impact on the early event-related potential component P2a only in the literal task and when hash symbols were displayed at the fovea, i.e., when both the task-related and foveal loads were low. The data show that all sources of cognitive load must be considered to understand how parafoveal words are processed in visual search contexts.

Is it dangerous? The role of an emotional visual search strategy and threat-relevant training in the detection of guns and knives

Counter-terrorism strategies rely on the assumption that it is possible to increase threat detection by providing explicit verbal instructions to orient people's attention to dangerous objects and hostile behaviours in their environment. Nevertheless, whether verbal cues can be used to enhance threat detection performance under laboratory conditions is currently unclear. In Experiment 1, student participants were required to detect a picture of a dangerous or neutral object embedded within a visual search display on the basis of an emotional strategy 'is it dangerous?' or a semantic strategy 'is it an object?'. The results showed a threat superiority effect that was enhanced by the emotional visual search strategy. In Experiment 2, whilst trainee police officers displayed a greater threat superiority effect than student controls, both groups benefitted from performing the task under the emotional than semantic visual search strategy. Manipulating situational threat levels (high vs. low) in the experimental instructions had no effect on visual search performance. The current findings provide new support for the language-as-context hypothesis. They are also consistent with a dual-processing account of threat detection involving a verbally mediated route in working memory and the deployment of a visual template developed as a function of training.

Multiple representations in visual working memory simultaneously guide attention: The type of memory-matching representation matters

Whether multiple visual working memory (VWM) representations can simultaneously become active templates to guide attention is controversial. The single-item-template hypothesis argues that only one VWM representation can be active at a time, whereas the multiple-item-template hypothesis argues that multiple VWM templates can simultaneously guide attention. The present study examined the two hypotheses in three (out of four) experiments, using three different types of memory objects: Experiment 1: shapes; Experiment 2: colors; and Experiment 3: colored shapes. Participants were required to hold one (memory-1) or two objects (memory-2) in VWM while performing a tilted line search task. Zero (match-0), one (match-1), or two (match-2) memory stimuli reappeared as distractors in the search array. Guidance effects were found for each type of memory stimuli. More importantly, the guidance effect for memory-2/match-2 trials was significantly larger than that for memory-2/match-1 and memory-1/match-1 trials when holding two colors or two colored shapes in VWM, which is in line with the multiple-item-template hypothesis. However, the pattern of simultaneous guidance effect is not perfectly found for two memory shapes, which may indicate that a reliable simultaneous guidance effect from two representations in VWM can be observed only when the memory-matching stimuli is more effective in guiding attention. Experiment 4 directly compared the guidance effect induced by feature-based matches (partial matching) with the guidance effect induced object-based matches (complete matching) in memory-set-size 2. Reliable guidance effects in match-1 and match-2 trials for object-based matches but not for feature-based matches confirmed the crucial role of the type of memory-matching stimuli in guiding attention.

Words affect visual perception by activating object shape representations

Linguistic labels are known to facilitate object recognition, yet the mechanism of this facilitation is not well understood. Previous psychophysical studies have suggested that words guide visual perception by activating information about visual object shape. Here we aimed to test this hypothesis at the neural level, and to tease apart the visual and semantic contribution of words to visual object recognition. We created a set of object pictures from two semantic categories with varying shapes, and obtained subjective ratings of their shape and category similarity. We then conducted a word-picture matching experiment, while recording participants’ EEG, and tested if the shape or the category similarity between the word’s referent and target picture explained the spatiotemporal pattern of the picture-evoked responses. The results show that hearing a word activates representations of its referent’s shape, which interacts with the visual processing of a subsequent picture within 100 ms from its onset. Furthermore, non-visual categorical information, carried by the word, affects the visual processing at later stages. These findings advance our understanding of the interaction between language and visual perception and provide insights into how the meanings of words are represented in the brain.

Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions

Language-processing accounts are beginning to accommodate different visual context effects, but they remain underspecified regarding differences between cues, both during sentence comprehension and subsequent recall. We monitored participants' eye movements to mentioned characters while they listened to transitive sentences. We varied whether speaker gaze, a depicted action, neither, or both of these visual cues were available, as well as whether both cues were deictic (Experiment 1) or only speaker gaze (Experiment 2). Speaker gaze affected eye movements during comprehension similarly early to a single deictic action depiction, but significantly earlier than non-deictic action depictions; conversely, depicted actions but not speaker gaze positively affected later recall of sentence content. Thus, cue type and cue-language relations must be accommodated in characterising real-time situated language comprehension and subsequent recall of sentence content.

Watching diagnoses develop: Eye movements reveal symptom processing during diagnostic reasoning

Finding a probable explanation for observed symptoms is a highly complex task that draws on information retrieval from memory. Recent research suggests that observed symptoms are interpreted in a way that maximizes coherence for a single likely explanation. This becomes particularly clear if symptom sequences support more than one explanation. However, there are no existing process data available that allow coherence maximization to be traced in ambiguous diagnostic situations, where critical information has to be retrieved from memory. In this experiment, we applied memory indexing, an eye-tracking method that affords rich time-course information concerning memory-based cognitive processing during higher order thinking, to reveal symptom processing and the preferred interpretation of symptom sequences. Participants first learned information about causes and symptoms presented in spatial frames. Gaze allocation to emptied spatial frames during symptom processing and during the diagnostic response reflected the subjective status of hypotheses held in memory and the preferred interpretation of ambiguous symptoms. Memory indexing traced how the diagnostic decision developed and revealed instances of hypothesis change and biases in symptom processing. Memory indexing thus provided direct online evidence for coherence maximization in processing ambiguous information.

The interplay of language and visual perception in working memory

How do perception and language interact to form the representations that guide our thoughts and actions over the short-term? Here, we provide a first examination of this question by investigating the role of verbal labels in a continuous visual working memory (WM) task. Across four experiments, participants retained in memory the continuous color of a set of dots which were presented sequentially (Experiments 1-3) or simultaneously (Experiment 4). At test, they reproduced the colors of all dots using a color wheel. During stimulus presentation participants were required to either label the colors (color labeling) or to repeat "bababa" aloud (articulatory suppression), hence prompting or preventing verbal labeling, respectively. We tested four competing hypotheses of the labeling effect: (1) labeling generates a verbal representation that overshadows the visual representation; (2) labeling yields a verbal representation in addition to the visual one; (3) the labels function as a retrieval cue, adding distinctiveness to items in memory; and (4) labels activate visual categorical representations in long-term memory. Collectively, our experiments show that labeling does not overshadow the visual input; it augments it. Mixture modeling showed that labeling increased the quantity and quality of information in WM. Our findings are consistent with the hypothesis that labeling activates visual long-term categorical representations which help in reducing the noise in the internal representations of the visual stimuli in WM.

ANALISIS DE LA PERCEPCION DE EVENTOS ESTATICOS Y DINAMICOS EN PERSONAS CON ENFERMEDAD DE ALZHEIMER

Uno de los aspectos afectados en la Enfermedad de Alzheimer es el lenguaje. La naturaleza y manifestaciones de estas dificultades se relacionan con la forma como los pacientes con EA perciben y comprenden el mundo que les rodea. En la presente investigación se analizaron las fijaciones visuales de siete pacientes con EA y sus controles durante tareas de percepción de escenas estáticas y dinámicas (imagen y video respectivamente). De igual forma se analizó una muestra de lenguaje producida por los pacientes al narrar el evento dinámico. Los resultados indican diferencias significativas en cuanto a la búsqueda visual, en la cual el grupo con EA presento disminución de la velocidad. En tareas de rastreo se evidencia que las personas con EA identifican menos elementos en una imagen realizando menos fijaciones, con estrategias de exploración poco eficientes. En cuanto al evento dinámico, el rastreo visual fue similar entre los dos grupos sin embargo la expresión lingüística de lo observado está afectada en el grupo con EA revelando la relación de la percepción y el lenguaje ya que a pesar de observar los eventos dentro de una escena en movimiento, estos no son recobrados posteriormente para ser expresados lingüísticamente. Estos resultados tienen importantes implicaciones tanto en la identificación de la naturaleza de las dificultades lingüísticas en la EA como en el manejo de la misma.

Semantic and Syntactic Associations During Word Search Modulate the Relationship Between Attention and Subsequent Memory

Two experiments were conducted to investigate how linguistic information influences attention allocation in visual search and memory for words. In Experiment 1, participants searched for the synonym of a cue word among five words. The distractors included one antonym and three unrelated words. In Experiment 2, participants were asked to judge whether the five words presented on the screen comprise a valid sentence. The relationships among words were sentential, semantically related or unrelated. A memory recognition task followed. Results in both experiments showed that linguistically related words produced better memory performance. We also found that there were significant interactions between linguistic relation conditions and memorization on eye-movement measures, indicating that good memory for words relied on frequent and long fixations during search in the unrelated condition but to a much lesser extent in linguistically related conditions. We conclude that semantic and syntactic associations attenuate the link between overt attention allocation and subsequent memory performance, suggesting that linguistic relatedness can somewhat compensate for a relative lack of attention during word search.

Blindness to background: an inbuilt bias for visual objects

Sixty-eight 2- to 12-year-olds and 30 adults were shown colorful displays on a touchscreen monitor and trained to point to the location of a named color. Participants located targets near-perfectly when presented with four abutting colored patches. When presented with three colored patches on a colored background, toddlers failed to locate targets in the background. Eye tracking demonstrated that the effect was partially mediated by a tendency not to fixate the background. However, the effect was abolished when the targets were named as nouns, whilst the change to nouns had little impact on eye movement patterns. Our results imply a powerful, inbuilt tendency to attend to objects, which may slow the development of color concepts and acquisition of color words. A video abstract of this article can be viewed at: https://youtu.be/TKO1BPeAiOI. [Correction added on 27 January 2017, after first online publication: The video abstract link was added.].

Modulation of scene consistency and task demand on language-driven eye movements for audio-visual integration

Previous psycholinguistic studies have demonstrated that people tend to direct fixations toward the visual object to which spoken input refers during language comprehension. However, it is still unclear how the visual scene, especially the semantic consistency between object and background, affects the word-object mapping process during comprehension. Two visual world paradigm experiments were conducted to investigate how the scene consistency dynamically influenced the language-driven eye movements in a speech comprehension and a scene comprehension task. In each trial, participants listened to a spoken sentence while viewing a picture with two critical objects: one is the mentioned target object (e.g., tiger), which was embedded in either a consistent (e.g., field), inconsistent (e.g., sky) or blank background; the other is an unmentioned non-target object (e.g., eagle), which was always consistent with its background. The results showed that the fixation proportion of the inconsistent target was higher than the consistent target, and the task demand can affect the strength and the direction of the inconsistency effect before and after the target had been mentioned. In summary, the spoken language, scene-based knowledge and task demand were intertwined to determine eye movements during audio-visual integration for comprehension.

Am I looking at a cat or a dog? Gaze in the semantic variant of primary progressive aphasia is subject to excessive taxonomic capture

Object naming impairments or anomias are the most frequent symptom in aphasia, and can be caused by a variety of underlying neurocognitive mechanisms. Anomia in neurodegenerative or primary progressive aphasias (PPA) often appears to be based on taxonomic blurring of word meaning: words such as "dog" and "cat" are still recognized generically as referring to animals, but are no longer conceptually differentiated from each other, leading to coordinate errors in word-object matching. This blurring is the hallmark symptom of the "semantic variant" of PPA, who invariably show focal atrophy in the left anterior temporal lobe. In this study we used eye tracking to characterize information processing online (in real time) as non-aphasic controls, semantic and non-semantic PPA participants completed a word-to-object matching task. All participants (including controls) showed taxonomic capture of gaze, spending more time viewing foils that were from the same category as the target compared to unrelated foils, but capture was more extreme in the semantic PPA group. The semantic group showed heightened capture even on trials where they ultimately pointed to the correct target, demonstrating the superiority of eye movements over traditional testing methods in detecting subtle processing impairments. Heightened capture was primarily driven by a tendency to direct gaze back and forth, repeatedly, between a set of related foils on each trial, a behavior almost never shown by controls or non-semantic participants. This suggests semantic PPA participants were accumulating and weighing evidence for a probabilistic rather than definitive mapping between the noun and several candidate objects. Neurodegeneration in PPA thus appears to distort lexical concepts prior to extinguishing them altogether, causing uncertainty in recognition and word-object matching.

Speakers of different languages process the visual world differently

Language and vision are highly interactive. Here we show that people activate language when they perceive the visual world, and that this language information impacts how speakers of different languages focus their attention. For example, when searching for an item (e.g., clock) in the same visual display, English and Spanish speakers look at different objects. Whereas English speakers searching for the clock also look at a cloud, Spanish speakers searching for the clock also look at a gift, because the Spanish names for gift (regalo) and clock (reloj) overlap phonologically. These different looking patterns emerge despite an absence of direct language input, showing that linguistic information is automatically activated by visual scene processing. We conclude that the varying linguistic information available to speakers of different languages affects visual perception, leading to differences in how the visual world is processed.

Eye movements as probes of lexico-semantic processing in a patient with primary progressive aphasia

Eye movement trajectories during a verbally cued object search task were used as probes of lexico-semantic associations in an anomic patient with primary progressive aphasia. Visual search was normal on trials where the target object could be named but became lengthy and inefficient on trials where the object failed to be named. The abnormality was most profound if the noun denoting the object could not be recognized. Even trials where the name of the target object was recognized but not retrieved triggered abnormal eye movements, demonstrating that retrieval failures can have underlying associative components despite intact comprehension of the corresponding noun.

Task-dependent modulation of word processing mechanisms during modified visual search tasks

During visual search for words, the impact of the visual and semantic features of words varies as a function of the search task. This event-related potential (ERP) study focused on the way these features of words are used to detect similarities between the distractor words that are glanced at and the target word, as well as to then reject the distractor words. The participants had to search for a target word that was either given literally or defined by a semantic clue among words presented sequentially. The distractor words included words that resembled the target and words that were semantically related to the target. The P2a component was the first component to be modulated by the visual and/or semantic similarity of distractors to the target word, and these modulations varied according to the task. The same held true for the later N300 and N400 components, which confirms that, depending on the task, distinct processing pathways were sensitized through attentional modulation. Hence, the process that matches what is perceived with the target acts during the first 200 ms after word presentation, and both early detection and late rejection processes of words depend on the search task and on the representation of the target stored in memory.

Four central questions about prediction in language processing

The notion that prediction is a fundamental principle of human information processing has been en vogue over recent years. The investigation of language processing may be particularly illuminating for testing this claim. Linguists traditionally have argued prediction plays only a minor role during language understanding because of the vast possibilities available to the language user as each word is encountered. In the present review I consider four central questions of anticipatory language processing: Why (i.e. what is the function of prediction in language processing)? What (i.e. what are the cues used to predict up-coming linguistic information and what type of representations are predicted)? How (what mechanisms are involved in predictive language processing and what is the role of possible mediating factors such as working memory)? When (i.e. do individuals always predict up-coming input during language processing)? I propose that prediction occurs via a set of diverse PACS (production-, association-, combinatorial-, and simulation-based prediction) mechanisms which are minimally required for a comprehensive account of predictive language processing. Models of anticipatory language processing must be revised to take multiple mechanisms, mediating factors, and situational context into account. Finally, I conjecture that the evidence considered here is consistent with the notion that prediction is an important aspect but not a fundamental principle of language processing. This article is part of a Special Issue entitled SI: Prediction and Attention.

Visual attention during spatial language comprehension

Spatial terms such as “above”, “in front of”, and “on the left of” are all essential for describing the location of one object relative to another object in everyday communication. Apprehending such spatial relations involves relating linguistic to object representations by means of attention. This requires at least one attentional shift, and models such as the Attentional Vector Sum (AVS) predict the direction of that attention shift, from the sausage to the box for spatial utterances such as “The box is above the sausage”. To the extent that this prediction generalizes to overt gaze shifts, a listener’s visual attention should shift from the sausage to the box. However, listeners tend to rapidly look at referents in their order of mention and even anticipate them based on linguistic cues, a behavior that predicts a converse attentional shift from the box to the sausage. Four eye-tracking experiments assessed the role of overt attention in spatial language comprehension by examining to which extent visual attention is guided by words in the utterance and to which extent it also shifts “against the grain” of the unfolding sentence. The outcome suggests that comprehenders’ visual attention is predominantly guided by their interpretation of the spatial description. Visual shifts against the grain occurred only when comprehenders had some extra time, and their absence did not affect comprehension accuracy. However, the timing of this reverse gaze shift on a trial correlated with that trial’s verification time. Thus, while the timing of these gaze shifts is subtly related to the verification time, their presence is not necessary for successful verification of spatial relations.