Preferential Inspection of Recent Real-World Events Over Future Events: Evidence from Eye Tracking during Spoken Sentence Comprehension

Integration of visual context in early and late bilingual language processing: evidence from eye-tracking

Previous research on the processing of language embedded in a rich visual context has revealed the strong effect that a recently viewed action event has on language comprehension. It has been shown that listeners are more likely to view the target object of a recently performed event than look at the target object of a plausible future event during sentence utterance, regardless of the tense cue. In the current visual-world eye-tracking experiments, we tested the strength of the recently observed visual context with a group of English monolingual and two groups of English-French early and late bilingual speakers. By comparing these different groups, we examined whether bilingual speakers, as a consequence of greater cognitive flexibility when integrating visual context and language information, show early anticipatory eye-movements toward the target object. We further asked whether early and late bilinguals show differences in their processing. The findings of the three eye-tracking experiments revealed an overall preference for the recently seen event. However, as a result of the early provision of tense cue, this preference was quickly diminished in all three groups. Moreover, the bilingual groups showed an earlier decrease in reliance on the recently seen event compared to monolingual speakers and the early bilinguals showed anticipatory eye-movements toward the plausible future event target. Furthermore, a post-experimental memory test revealed that the bilingual groups recalled the future events marginally better than the recent events, whereas the reverse was found in the monolingual groups.

Influence of Actor's Congruent and Incongruent Gaze on Language Processing

In interpreting spoken sentences in event contexts, comprehenders both integrate their current interpretation of language with the recent past (e.g., events they have witnessed) and develop expectations about future event possibilities. Tense cues can disambiguate this linking but temporary ambiguity in their interpretation may lead comprehenders to also rely on further, situation-specific cues (e.g., an actor's gaze as a cue to his future actions). How comprehenders reconcile these different cues in real time is an open issue that we must address to accommodate comprehension. It has been suggested that relating a referential expression (e.g., a verb) to a referent (e.g., a recent event) is preferred over relying on other cues that refer to the future and are not yet referentially grounded (“recent-event preference”). Two visual-world eye-tracking experiments compared this recent-event preference with effects of an actor's gaze and of tense/temporal adverbs as cues to a future action event. The results revealed that people overall preferred to focus on the recent (vs. future) event target in their interpretation, suggesting that while a congruent and incongruent actor gaze can jointly with futuric linguistic cues neutralize the recent-event preference late in the sentence, the latter still plays a key role in shaping participants' initial verb-based event interpretation. Additional post-experimental memory tests provided insight into the longevity of the gaze effects.

To catch a Snitch: Brain potentials reveal variability in the functional organization of (fictional) world knowledge during reading

We harnessed the temporal sensitivity of event-related brain potentials (ERPs) alongside individual differences in Harry Potter (HP) knowledge to investigate the extent to which the availability and timing of information relevant for real-time written word processing are influenced by variation in domain knowledge. We manipulated meaningful (category, event) relationships between sentence fragments about HP stories and their sentence final words. During word-by-word reading, N400 amplitudes to (a) linguistically supported and (b) unsupported but meaningfully related, but not to (c) unsupported, unrelated sentence endings varied with HP domain knowledge. Single-trial analyses revealed that only the N400s to linguistically supported (but not to either type of unsupported) sentence-final words varied as a function of whether individuals knew (or could remember) the correct (supported) ending for each HP "fact." We conclude that the quick availability of information relevant for word understanding in sentences is a function of individuals' knowledge of both specific facts and the domain to which the facts belong. During written sentence processing, as domain knowledge increases, it is clearly evident that individuals can make use of the relevant knowledge systematically organized around themes, events, and categories in that domain, to the extent they have it.

Semantic processing of adjectives and nouns in American Sign Language: effects of reference ambiguity and word order across development

When processing spoken language sentences, listeners continuously make and revise predictions about the upcoming linguistic signal. In contrast, during comprehension of American Sign Language (ASL), signers must simultaneously attend to the unfolding linguistic signal and the surrounding scene via the visual modality. This may affect how signers activate potential lexical candidates and allocate visual attention as a sentence unfolds. To determine how signers resolve referential ambiguity during real-time comprehension of ASL adjectives and nouns, we presented deaf adults (n = 18, 19-61 years) and deaf children (n = 20, 4-8 years) with videos of ASL sentences in a visual world paradigm. Sentences had either an adjective-noun ("SEE YELLOW WHAT? FLOWER") or a noun-adjective ("SEE FLOWER WHICH? YELLOW") structure. The degree of ambiguity in the visual scene was manipulated at the adjective and noun levels (i.e., including one or more yellow items and one or more flowers in the visual array). We investigated effects of ambiguity and word order on target looking at early and late points in the sentence. Analysis revealed that adults and children made anticipatory looks to a target when it could be identified early in the sentence. Further, signers looked more to potential lexical candidates than to unrelated competitors in the early window, and more to matched than unrelated competitors in the late window. Children's gaze patterns largely aligned with those of adults with some divergence. Together, these findings suggest that signers allocate referential attention strategically based on the amount and type of ambiguity at different points in the sentence when processing adjectives and nouns in ASL.

Visual attention during spatial language comprehension

Spatial terms such as “above”, “in front of”, and “on the left of” are all essential for describing the location of one object relative to another object in everyday communication. Apprehending such spatial relations involves relating linguistic to object representations by means of attention. This requires at least one attentional shift, and models such as the Attentional Vector Sum (AVS) predict the direction of that attention shift, from the sausage to the box for spatial utterances such as “The box is above the sausage”. To the extent that this prediction generalizes to overt gaze shifts, a listener’s visual attention should shift from the sausage to the box. However, listeners tend to rapidly look at referents in their order of mention and even anticipate them based on linguistic cues, a behavior that predicts a converse attentional shift from the box to the sausage. Four eye-tracking experiments assessed the role of overt attention in spatial language comprehension by examining to which extent visual attention is guided by words in the utterance and to which extent it also shifts “against the grain” of the unfolding sentence. The outcome suggests that comprehenders’ visual attention is predominantly guided by their interpretation of the spatial description. Visual shifts against the grain occurred only when comprehenders had some extra time, and their absence did not affect comprehension accuracy. However, the timing of this reverse gaze shift on a trial correlated with that trial’s verification time. Thus, while the timing of these gaze shifts is subtly related to the verification time, their presence is not necessary for successful verification of spatial relations.

Losing track of time? Processing of time reference inflection in agrammatic and healthy speakers of German

BACKGROUND

Individuals with agrammatic aphasia (IWAs) have problems with grammatical decoding of tense inflection. However, these difficulties depend on the time frame that the tense refers to. Verb morphology with reference to the past is more difficult than with reference to the non-past, because a link needs to be made to the past event in discourse, as captured in the PAst DIscourse LInking Hypothesis (PADILIH; Bastiaanse, R., Bamyaci, E., Hsu, C., Lee, J., Yarbay Duman, T., Thompson, C. K., 2011. Time reference in agrammatic aphasia: A cross-linguistic study. J. Neurolinguist. 24, 652-673). With respect to reference to the (non-discourse-linked) future, data so far indicate that IWAs experience less difficulties as compared to past time reference (Bastiaanse, R., Bamyaci, E., Hsu, C., Lee, J., Yarbay Duman, T., Thompson, C. K., 2011. Time reference in agrammatic aphasia: A cross-linguistic study. J. Neurolinguist. 24, 652-673), supporting the assumptions of the PADILIH. Previous online studies of time reference in aphasia used methods such as reaction times analysis (e.g., Faroqi-Shah, Y., Dickey, M. W., 2009. On-line processing of tense and temporality in agrammatic aphasia. Brain Lang. 108, 97-111). So far, no such study used eye-tracking, even though this technique can bring additional insights (Burchert, F., Hanne, S., Vasishth, S., 2013. Sentence comprehension disorders in aphasia: the concept of chance performance revisited. Aphasiology 27, 112-125, doi:10.1080/02687038.2012.730603).

AIMS

This study investigated (1) whether processing of future and past time reference inflection differs between non-brain-damaged individuals (NBDs) and IWAs, and (2) underlying mechanisms of time reference comprehension failure by IWAs.

METHODS AND PROCEDURES

A visual-world experiment combining sentence-picture matching and eye-tracking was administered to 12 NBDs and 6 IWAs, all native speakers of German. Participants heard German sentences with periphrastic future ('will+V') or periphrastic past ('has+V-d') verb forms while they were presented with corresponding pictures on a computer screen.

RESULTS AND DISCUSSION

NBDs scored at ceiling and significantly higher than the IWAs. IWAs had below-ceiling performance on the future condition, and both participant groups were faster to respond to the past than to the future condition. These differences are attributed to a pre-existing preference to look at a past picture, which has to be overcome. Eye movement patterns suggest that both groups interpret future time reference similarly, while IWAs show a delay relative to NBDs in interpreting past time reference inflection. The eye tracking results support the PADILIH, because processing reference to the past in discourse syntax requires additional resources and thus, is problematic and delayed for people with aphasia.

Effects of speaker emotional facial expression and listener age on incremental sentence processing

We report two visual-world eye-tracking experiments that investigated how and with which time course emotional information from a speaker's face affects younger (N = 32, Mean age  = 23) and older (N = 32, Mean age  = 64) listeners’ visual attention and language comprehension as they processed emotional sentences in a visual context. The age manipulation tested predictions by socio-emotional selectivity theory of a positivity effect in older adults. After viewing the emotional face of a speaker (happy or sad) on a computer display, participants were presented simultaneously with two pictures depicting opposite-valence events (positive and negative; IAPS database) while they listened to a sentence referring to one of the events. Participants' eye fixations on the pictures while processing the sentence were increased when the speaker's face was (vs. wasn't) emotionally congruent with the sentence. The enhancement occurred from the early stages of referential disambiguation and was modulated by age. For the older adults it was more pronounced with positive faces, and for the younger ones with negative faces. These findings demonstrate for the first time that emotional facial expressions, similarly to previously-studied speaker cues such as eye gaze and gestures, are rapidly integrated into sentence processing. They also provide new evidence for positivity effects in older adults during situated sentence processing.