Fixations in the visual world paradigm: where, when, why?

The development of lexical processing: Real-time phonological competition and semantic activation in school age children

Prior research suggests that the development of speech perception and word recognition stabilises in early childhood. However, recent work suggests that development of these processes continues throughout adolescence. This study aimed to investigate whether these developmental changes are based solely within the lexical system or are due to domain general changes, and to extend this investigation to lexical-semantic processing. We used two Visual World Paradigm tasks: one to examine phonological and semantic processing, one to capture non-linguistic domain-general skills. We tested 43 seven- to nine-year-olds, 42 ten- to thirteen-year-olds, and 30 sixteen- to seventeen-year-olds. Older children were quicker to fixate the target word and exhibited earlier onset and offset of fixations to both semantic and phonological competitors. Visual/cognitive skills explained significant, but not all, variance in the development of these effects. Developmental changes in semantic activation were largely attributable to changes in upstream phonological processing. These results suggest that the concurrent development of linguistic processes and broader visual/cognitive skills lead to developmental changes in real-time phonological competition, while semantic activation is more stable across these ages.

The Importance of Linguistic Factors: He Likes Subject Referents

We report the results of one visual-world eye-tracking experiment and two referent selection tasks in which we investigated the effects of information structure in the form of prosody and word order manipulation on the processing of subject pronouns er and der in German. Factors such as subjecthood, focus, and topicality, as well as order of mention have been linked to an increased probability of certain referents being selected as the pronoun's antecedent and described as increasing this referent's prominence, salience, or accessibility. The goal of this study was to find out whether pronoun processing is primarily guided by linguistic factors (e.g., grammatical role) or nonlinguistic factors (e.g., first-mention), and whether pronoun interpretation can be described in terms of referents' "prominence" / "accessibility" / "salience." The results showed an overall subject preference for er, whereas der was affected by the object role and focus marking. While focus increases the attentional load and enhances memory representation for the focused referent making the focused referent more available, ultimately it did not affect the final interpretation of er, suggesting that "prominence" or the related concepts do not explain referent selection preferences. Overall, the results suggest a primacy of linguistic factors in determining pronoun resolution.

A network model of referent identification by toddlers in a visual world task

We present a neural network model of referent identification in a visual world task. Inputs are visual representations of item pairs unfolding with sequences of phonemes identifying the target item. The model is trained to output the semantic representation of the target and to suppress the distractor. The training set uses a 200-word lexicon typically known by toddlers. The phonological, visual, and semantic representations are derived from real corpora. Successful performance requires correct association between labels and visual and semantic representations, as well as correct location identification. The model reproduces experimental evidence that phonological, perceptual, and categorical relationships modulate item preferences. The model provides an account of how language can drive visual attention in the inter-modal preferential looking task.

Analysing data from the psycholinguistic visual-world paradigm: Comparison of different analysis methods

In this paper, we discuss key characteristics and typical experimental designs of the visual-world paradigm and compare different methods of analysing eye-movement data. We discuss the nature of the eye-movement data from a visual-world study and provide data analysis tutorials on ANOVA, t-tests, linear mixed-effects model, growth curve analysis, cluster-based permutation analysis, bootstrapped differences of timeseries, generalised additive modelling, and divergence point analysis to enable psycholinguists to apply each analytical method to their own data. We discuss advantages and disadvantages of each method and offer recommendations about how to select an appropriate method depending on the research question and the experimental design.

Delayed Lexical Access and Cascading Effects on Spreading Semantic Activation During Spoken Word Recognition in Children With Hearing Aids and Cochlear Implants: Evidence From Eye-Tracking

OBJECTIVE

The objective of this study was to characterize the dynamics of real-time lexical access, including lexical competition among phonologically similar words, and spreading semantic activation in school-age children with hearing aids (HAs) and children with cochlear implants (CIs). We hypothesized that developing spoken language via degraded auditory input would lead children with HAs or CIs to adapt their approach to spoken word recognition, especially by slowing down lexical access.

DESIGN

Participants were children ages 9- to 12-years old with normal hearing (NH), HAs, or CIs. Participants completed a Visual World Paradigm task in which they heard a spoken word and selected the matching picture from four options. Competitor items were either phonologically similar, semantically similar, or unrelated to the target word. As the target word unfolded, children's fixations to the target word, cohort competitor, rhyme competitor, semantically related item, and unrelated item were recorded as indices of ongoing lexical access and spreading semantic activation.

RESULTS

Children with HAs and children with CIs showed slower fixations to the target, reduced fixations to the cohort competitor, and increased fixations to the rhyme competitor, relative to children with NH. This wait-and-see profile was more pronounced in the children with CIs than the children with HAs. Children with HAs and children with CIs also showed delayed fixations to the semantically related item, although this delay was attributable to their delay in activating words in general, not to a distinct semantic source.

CONCLUSIONS

Children with HAs and children with CIs showed qualitatively similar patterns of real-time spoken word recognition. Findings suggest that developing spoken language via degraded auditory input causes long-term cognitive adaptations to how listeners recognize spoken words, regardless of the type of hearing device used. Delayed lexical access directly led to delays in spreading semantic activation in children with HAs and CIs. This delay in semantic processing may impact these children's ability to understand connected speech in everyday life.

I'm not sure that curve means what you think it means: Toward a [more] realistic understanding of the role of eye-movement generation in the Visual World Paradigm

The Visual World Paradigm (VWP) is a powerful experimental paradigm for language research. Listeners respond to speech in a "visual world" containing potential referents of the speech. Fixations to these referents provides insight into the preliminary states of language processing as decisions unfold. The VWP has become the dominant paradigm in psycholinguistics and extended to every level of language, development, and disorders. Part of its impact is the impressive data visualizations which reveal the millisecond-by-millisecond time course of processing, and advances have been made in developing new analyses that precisely characterize this time course. All theoretical and statistical approaches make the tacit assumption that the time course of fixations is closely related to the underlying activation in the system. However, given the serial nature of fixations and their long refractory period, it is unclear how closely the observed dynamics of the fixation curves are actually coupled to the underlying dynamics of activation. I investigated this assumption with a series of simulations. Each simulation starts with a set of true underlying activation functions and generates simulated fixations using a simple stochastic sampling procedure that respects the sequential nature of fixations. I then analyzed the results to determine the conditions under which the observed fixations curves match the underlying functions, the reliability of the observed data, and the implications for Type I error and power. These simulations demonstrate that even under the simplest fixation-based models, observed fixation curves are systematically biased relative to the underlying activation functions, and they are substantially noisier, with important implications for reliability and power. I then present a potential generative model that may ultimately overcome many of these issues.

Increased reliance on world knowledge during language comprehension in healthy aging: evidence from verb-argument prediction

Cognitive aging negatively impacts language comprehension performance. . However, there is evidence that older adults skillfully use linguistic context and their crystallized world knowledge to offset age-related changes that negatively impact comprehension. Two visual-world paradigm experiments examined how aging changes verb-argument prediction, a comprehension process that relies on world knowledge but has rarely been examined in the cognitive-aging literature. Older adults did not differ from younger adults in their activation of an upcoming likely verb argument, particularly when cued by a semantically-rich agent+verb combination (Experiment 1). However, older adults showed elevated activation of previously-mentioned agents (Experiment 1) and of unlikely but verb-congruent referents (Experiment 2). This is novel evidence that older adults exploit semantic context and world knowledge during comprehension to successfully activate upcoming referents. However, older adults also show elevated activation of irrelevant information, consistent with previous findings demonstrating that older adults may experience greater proactive interference and competition from task-irrelevant information.

Language is activated by visual input regardless of memory demands or capacity

In the present study, we provide compelling evidence that viewing objects automatically activates linguistic labels and that this activation is not due to task-specific memory demands. In two experiments, eye-movements of English speakers were tracked while they identified a visual target among an array of four images, including a phonological competitor (e.g., flower-flag). Experiment 1 manipulated the capacity to subvocally rehearse the target label by imposing linguistic, spatial, or no working memory load. Experiment 2 manipulated the need to encode target objects by presenting target images either before or concurrently with the search display. While the timing and magnitude of competitor activation varied across conditions, we observed consistent evidence of language activation regardless of the capacity or need to maintain object labels in memory. We propose that language activation is automatic and not contingent upon working memory capacity or demands, and conclude that objects' labels influence visual search.

The pictures who shall not be named: Empirical support for benefits of preview in the Visual World Paradigm

A common critique of the Visual World Paradigm (VWP) in psycholinguistic studies is that what is designed as a measure of language processes is meaningfully altered by the visual context of the task. This is crucial, particularly in studies of spoken word recognition, where the displayed images are usually seen as just a part of the measure and are not of fundamental interest. Many variants of the VWP allow participants to sample the visual scene before a trial begins. However, this could bias their interpretations of the later speech or even lead to abnormal processing strategies (e.g., comparing the input to only preactivated working memory representations). Prior work has focused only on whether preview duration changes fixation patterns. However, preview could affect a number of processes, such as visual search, that would not challenge the interpretation of the VWP. The present study uses a series of targeted manipulations of the preview period to ask if preview alters looking behavior during a trial, and why. Results show that evidence of incremental processing and phonological competition seen in the VWP are not dependent on preview, and are not enhanced by manipulations that directly encourage phonological prenaming. Moreover, some forms of preview can eliminate nuisance variance deriving from object recognition and visual search demands in order to produce a more sensitive measure of linguistic processing. These results deepen our understanding of how the visual scene interacts with language processing to drive fixations patterns in the VWP, and reinforce the value of the VWP as a tool for measuring real-time language processing. Stimuli, data and analysis scripts are available at https://osf.io/b7q65/.

The profile of real-time competition in spoken and written word recognition: More similar than different

Word recognition occurs across two sensory modalities: auditory (spoken words) and visual (written words). While each faces different challenges, they are often described in similar terms as a competition process by which multiple lexical candidates are activated and compete for recognition. While there is a general consensus regarding the types of words that compete during spoken word recognition, there is less consensus for written word recognition. The present study develops a novel version of the Visual World Paradigm (VWP) to examine written word recognition and uses this to assess the nature of the competitor set during word recognition in both modalities using the same experimental design. For both spoken and written words, we found evidence for activation of onset competitors (cohorts, e.g., cat, cap) and words that contain the same phonemes or letters in reverse order (anadromes, e.g., cat, tack). We found no evidence of activation for rhymes (e.g., cat, hat). The results across modalities were quite similar, with the exception that for spoken words, cohorts were more active than anadromes, whereas for written words activation was similar. These results suggest a common characterisation of lexical similarity across spoken and written words: temporal or spatial order is coarsely coded, and onsets may receive more weight in both systems. However, for spoken words, temporary ambiguity during the moment of processing gives cohorts an additional boost during real-time recognition.

A Neural Network Model of Lexical-Semantic Competition During Spoken Word Recognition

Visual world studies show that upon hearing a word in a target-absent visual context containing related and unrelated items, toddlers and adults briefly direct their gaze toward phonologically related items, before shifting toward semantically and visually related ones. We present a neural network model that processes dynamic unfolding phonological representations of words and maps them to static internal lexical, semantic, and visual representations. The model, trained on representations derived from real corpora, simulates this early phonological over semantic/visual preference. Our results support the hypothesis that incremental unfolding of a spoken word is in itself sufficient to account for the transient preference for phonological competitors over both unrelated and semantically and visually related ones. Phonological representations mapped dynamically in a bottom-up fashion to semantic-visual representations capture the early phonological preference effects reported in visual world tasks. The semantic visual preference typically observed later in such a task does not require top-down feedback from a semantic or visual system.

I see what you mean: Semantic but not lexical factors modulate image processing in bilingual adults

Bilinguals frequently juggle competing representations from their two languages when they interact with their environment (i.e., nonselective activation). As a result, both first (L1) and second language (L2) communication may be impeded when words share orthographic form but not meaning (i.e., interlingual homographs; e.g., CRANE, a machine in English, a skull in French). Similarly, bilinguals' reduced exposure to each known language makes bilingual lexical processing more vulnerable to larger frequency effects. While much is known about processes within the language system, less is known about how the bilingual language system interacts with the visual system, specifically in the context of image processing. We investigated this by testing whether commonly observed semantic (homograph interference) and lexical (frequency) effects extend to a visual word-image matching task. We tested 48 bilinguals, who were asked to determine whether an image corresponded to a written word that was presented immediately beforehand. By modulating the complexity of visual referents and the semantic (Analysis 1) or lexical (Analysis 2) complexity of word cues, we simultaneously burdened the visual and language systems. The results showed that both semantic and lexical factors modulated response accuracy and correct reaction time on the word-image matching task. Crucially, we observed an interaction between the image factor (visual complexity) with the semantic (homograph status) but not the lexical factor (word frequency). We conclude that it is possible for the language and image processing systems to interact, although the extent to which this occurs depends on the degree of linguistic processing involved.

Stroke Encoding Processes of Chinese Character During Sentence Reading

The present study was conducted to examine whether traditional and simplified Chinese readers (TCRs and SCRs) differed in stroke encoding in character processing by an eye-tracking experiment. We recruited 66 participants (32 TCRs and 34 SCRs) to read sentences comprising characters with different proportions and types of strokes removed in order to explore whether any visual complexity effect existed in their processing of simplified and traditional Chinese characters. The present study found a cross-script visual complexity effect and that SCRs were more influenced by visual complexity change in lexical access than were TCRs. In addition, the stroke-order effect appeared to be more salient for TCRs than for SCRs.

Language and perception: Introduction to the Special Issue “Speakers and Listeners in the Visual World”

Language and perception are two central cognitive systems. Until relatively recently, however, the interaction between them has been examined only partially and not from an over-arching theoretical perspective. Yet it has become clear that linguistic and perceptual interactions are essential to understanding both typical and atypical human behaviour. In this editorial, we examine the link between language and perception across three domains. First, we present a brief review of work investigating the importance of perceptual features, particularly shape bias, when learning names for novel objects—a critical skill acquired during language development. Second, we describe the Visual World Paradigm, an experimental method uniquely suited to investigate the language-perception relationship. Studies using the Visual World Paradigm demonstrate that the relationship between linguistic and perceptual information during processing is both intricate and bi-directional: linguistic cues guide interpretation of visual scenes, while perceptual information shapes interpretation of linguistic input. Finally, we turn to a discussion of co-speech gesture focusing on iconic gestures which depict aspects of the visual world (e.g., motion, shape). The relationship between language and these semantically-meaningful gestures is likewise complex and bi-directional. However, more research is needed to illuminate the exact circumstances under which iconic gestures shape language production and comprehension. In conclusion, although strong evidence exists supporting a critical relationship between linguistic and perceptual systems, the exact levels at which these two systems interact, the time-course of the interaction, and what is driving the interaction, remain largely open questions in need of future research.