Prediction is Production: The missing link between language production and comprehension

Oral reading promotes predictive processing in Chinese sentence reading: eye movement evidence

Background

Fluent sentence reading is widely acknowledged to depend on top-down contextual prediction, wherein sentential and contextual cues guide the pre-activation of linguistic representations before encountering stimuli, facilitating subsequent comprehension. The Prediction-by-Production hypothesis posits an explanation for predictive processes in language comprehension, suggesting that prediction during comprehension involves processes associated with language production. However, there is a lack of eye movement evidence supporting this hypothesis within sentence reading contexts. Thus, we manipulated reading mode and word predictability to examine the influence of language production on predictive processing.

Methods

Participants engaged in silent or oral reading of sentences containing either high or low-predictable target words. Eye movements were recorded using the Eyelink1000 eye tracker.

Results

The findings revealed a higher skipping rate and shorter fixation times for high-predictable words compared to low-predictable ones, and for silent compared to oral reading. Notably, interactive effects were observed in the time measures (FFD, SFD, GD) during first-pass reading, indicating that word predictability effects were more pronounced during oral reading than silent reading.

Discussion

The observed pattern of results suggests that the activation of the production system enhances predictive processing during the early lexical access, providing empirical support for the Prediction-by-Production hypothesis in eye movement sentence reading situations, extending the current understanding of the timing and nature of predictions in reading comprehension.

Predictive language processing: integrating comprehension and production, and what atypical populations can tell us

Predictive processing, a crucial aspect of human cognition, is also relevant for language comprehension. In everyday situations, we exploit various sources of information to anticipate and therefore facilitate processing of upcoming linguistic input. In the literature, there are a variety of models that aim at accounting for such ability. One group of models propose a strict relationship between prediction and language production mechanisms. In this review, we first introduce very briefly the concept of predictive processing during language comprehension. Secondly, we focus on models that attribute a prominent role to language production and sensorimotor processing in language prediction ("prediction-by-production" models). Contextually, we provide a summary of studies that investigated the role of speech production and auditory perception on language comprehension/prediction tasks in healthy, typical participants. Then, we provide an overview of the limited existing literature on specific atypical/clinical populations that may represent suitable testing ground for such models-i.e., populations with impaired speech production and auditory perception mechanisms. Ultimately, we suggest a more widely and in-depth testing of prediction-by-production accounts, and the involvement of atypical populations both for model testing and as targets for possible novel speech/language treatment approaches.

Dissociating the pre-activation of word meaning and form during sentence comprehension: Evidence from EEG representational similarity analysis

During language comprehension, the processing of each incoming word is facilitated in proportion to its predictability. Here, we asked whether anticipated upcoming linguistic information is actually pre-activated before new bottom-up input becomes available, and if so, whether this pre-activation is limited to the level of semantic features, or whether extends to representations of individual word-forms (orthography/phonology). We carried out Representational Similarity Analysis on EEG data while participants read highly constraining sentences. Prior to the onset of the expected target words, sentence pairs predicting semantically related words (financial "bank" - "loan") and form-related words (financial "bank" - river "bank") produced more similar neural patterns than pairs predicting unrelated words ("bank" - "lesson"). This provides direct neural evidence for item-specific semantic and form predictive pre-activation. Moreover, the semantic pre-activation effect preceded the form pre-activation effect, suggesting that top-down pre-activation is propagated from higher to lower levels of the linguistic hierarchy over time.

Prediction during language comprehension: what is next?

Prediction is often regarded as an integral aspect of incremental language comprehension, but little is known about the cognitive architectures and mechanisms that support it. We review studies showing that listeners and readers use all manner of contextual information to generate multifaceted predictions about upcoming input. The nature of these predictions may vary between individuals owing to differences in language experience, among other factors. We then turn to unresolved questions which may guide the search for the underlying mechanisms. (i) Is prediction essential to language processing or an optional strategy? (ii) Are predictions generated from within the language system or by domain-general processes? (iii) What is the relationship between prediction and memory? (iv) Does prediction in comprehension require simulation via the production system? We discuss promising directions for making progress in answering these questions and for developing a mechanistic understanding of prediction in language.

Electrophysiological evidence for a subject-first strategy in visually situated auditory sentence processing in Korean

This study investigated a subject-first strategy in prediction mechanism in visually situated sentence processing in Korean, using event-related potentials (ERPs). According to the subject-first strategy, parsers tend to generate sentences conforming to canonical sentence word order (i.e., SOV in Korean), subject-first sentence, mapping conceptually more prominent referent such as agent of the event on the subject position of the sentence. Therefore, in the predictive mechanism of language comprehension, the subject is pre-activated and anticipated for the first NP of the sentence at the initial phase of bottom-up language processing. This study tested this subject-first strategy in Korean by examining brain responses to object-initial sentences (OV) compared with subject-initial sentences (SV) under the context of clear thematic role relations set by a visual image. The results of an ERP experiment with 30 native Korean speakers identified neural effects for object-initial sentences compared with subject-initial sentences at the NP and Verb, reflecting a conflict between the pre-activated representation in the parser's mind and the encountered bottom-up input. An N400 effect was elicited at the NP, as early as at the noun, not at the following object case marker. Late frontal positivity (LFP) was also found in the sentence-final verb, proving the processing difficulty of non-canonical object-initial sentences compared with canonical subject-initial sentences. These results indicate that Korean native speakers build linguistic representation conforming to a canonical sentence in SOV language in the predictive mechanism supporting subject-first strategy but revise the predicted event structure rapidly upon newly encountering input.

Word prediction using closely and moderately related verbs in Down syndrome

People with Down syndrome (DS) have several difficulties in language learning, and one of the areas most affected is language production. Theoretical frameworks argue that prediction depends on the production system. Yet, people with DS can predict upcoming nouns using semantically related verbs. Possibly, prediction skills in people with DS are driven by their associative mechanism rather than by the prediction mechanism based on the production system. This study explores prediction mechanisms in people with DS and their relationship with production skills. Three groups were evaluated in a preferential-looking task: young adults, children with DS, and a typically developing control group paired by sex and mental age. Participants saw two images, a target and a distractor. They also heard a sentence in one of the three conditions: with a verb that was closely related to the object (e.g., "The woman read the book"), with a verb that was moderately related to the object (e.g., "My uncle waited for the bus"), or with a verb that was unrelated to the object (e.g., "My sister threw a broom"). Their productive vocabulary was then measured. In the young adult and typically developing groups, the results showed prediction in sentences with highly and moderately related verbs. Participants with DS, however, showed prediction skills only in the highly related context. There was no influence of chronological age, mental age, or production on prediction skills. These results indicate that people with DS base prediction mainly on associative mechanisms and they have difficulty in generating top-down predictions.

MSGWO-MKL-SVM: A Missing Link Prediction Method for UAV Swarm Network Based on Time Series

Missing link prediction technology (MLP) is always a hot research area in the field of complex networks, and it has been extensively utilized in UAV swarm network reconstruction recently. UAV swarm is an artificial network with strong randomness, in the face of which prediction methods based on network similarity often perform poorly. To solve those problems, this paper proposes a Multi Kernel Learning algorithm with a multi-strategy grey wolf optimizer based on time series (MSGWO-MKL-SVM). The Multiple Kernel Learning (MKL) method is adopted in this algorithm to extract the advanced features of time series, and the Support Vector Machine (SVM) algorithm is used to determine the hyperplane of threshold value in nonlinear high dimensional space. Besides that, we propose a new measurable indicator of Multiple Kernel Learning based on cluster, transforming a Multiple Kernel Learning problem into a multi-objective optimization problem. Some adaptive neighborhood strategies are used to enhance the global searching ability of grey wolf optimizer algorithm (GWO). Comparison experiments were conducted on the standard UCI datasets and the professional UAV swarm datasets. The classification accuracy of MSGWO-MKL-SVM on UCI datasets is improved by 6.2% on average, and the link prediction accuracy of MSGWO-MKL-SVM on professional UAV swarm datasets is improved by 25.9% on average.

Prediction during simultaneous interpreting: Evidence from the visual-world paradigm

We report the results of an eye-tracking study which used the Visual World Paradigm (VWP) to investigate the time-course of prediction during a simultaneous interpreting task. Twenty-four L1 French professional conference interpreters and twenty-four L1 French professional translators untrained in simultaneous interpretation listened to sentences in English and interpreted them simultaneously into French while looking at a visual scene. Sentences contained a highly predictable word (e.g., The dentist asked the man to open his mouth a little wider). The visual scene comprised four objects, one of which depicted either the target object (mouth; bouche), an English phonological competitor (mouse; souris), a French phonological competitor (cork; bouchon), or an unrelated word (bone; os). We considered 1) whether interpreters and translators predict upcoming nouns during a simultaneous interpreting task, 2) whether interpreters and translators predict the form of these nouns in English and in French and 3) whether interpreters and translators manifest different predictive behaviour. Our results suggest that both interpreters and translators predict upcoming nouns, but neither group predicts the word-form of these nouns. In addition, we did not find significant differences between patterns of prediction in interpreters and translators. Thus, evidence from the visual-world paradigm shows that prediction takes place in simultaneous interpreting, regardless of training and experience. However, we were unable to establish whether word-form was predicted.

Dropping Beans or Spilling Secrets: How Idiomatic Context Bias Affects Prediction

Idioms can have both a literal interpretation and a figurative interpretation (e.g., to "kick the bucket"). Which interpretation should be activated can be disambiguated by a preceding context (e.g., "The old man was sick. He kicked the bucket."). We investigated whether the idiomatic and literal uses of idioms have different predictive properties when the idiom has been biased toward a literal or figurative sentence interpretation. EEG was recorded as participants performed a lexical decision task on idiom-final words in biased idioms and literal (compositional) sentences. Targets in idioms were identified faster in both figuratively and literally used idioms than in compositional sentences. Time-frequency analysis of a prestimulus interval revealed relatively more alpha-beta power decreases in literally than figuratively used idiomatic sequences and compositional sentences. We argue that lexico-semantic retrieval plays a larger role in literally than figuratively biased idioms, as retrieval of the word meaning is less relevant in the latter and the word form has to be matched to a template. The results are interpreted in terms of context integration and word retrieval and have implications for models of language processing and predictive processing in general.

Verb-Mediated Prediction in Bilingual Toddlers

Prediction is an important mechanism for efficient language processing. It has been shown that as a part of sentence processing, both children and adults predict nouns based on semantically constraining verbs. Language proficiency is said to modulate prediction: the higher proficiency, the better the predictive skill. Children growing up acquiring two languages are often more proficient in one of them, and as such, investigation of the predictive ability in young bilingual children can shed light on the role of language proficiency. Furthermore, according to production-based models, the language production system drives the predictive ability. The present study investigates whether bilingual toddlers predict upcoming nouns based on verb meanings in both their languages, and whether this ability is associated with expressive vocabulary. Seventeen Norwegian-English bilingual toddlers (aged 2;5-3;3), dominant in Norwegian, participated in the study. Verb-mediated predictive ability was measured via a visual world paradigm (VWP) experiment, including sentences with semantically constraining and neutral verbs. Expressive vocabulary was measured by MacArthur-Bates CDI II. The results suggested that the toddler group predicted upcoming noun arguments in both their dominant and non-dominant languages, but were faster in their dominant language. This finding highlights the importance of language dominance for predictive processing. There was no significant relationship between predictive ability and expressive vocabulary in either language.

The effects of dual-task interference in predicting turn-ends in speech and music

Determining when a partner's spoken or musical turn will end requires well-honed predictive abilities. Evidence suggests that our motor systems are activated during perception of both speech and music, and it has been argued that motor simulation is used to predict turn-ends across domains. Here we used a dual-task interference paradigm to investigate whether motor simulation of our partner's action underlies our ability to make accurate turn-end predictions in speech and in music. Furthermore, we explored how specific this simulation is to the action being predicted. We conducted two experiments, one investigating speech turn-ends, and one investigating music turn-ends. In each, 34 proficient pianists predicted turn-endings while (1) passively listening, (2) producing an effector-specific motor activity (mouth/hand movement), or (3) producing a task- and effector-specific motor activity (mouthing words/fingering a piano melody). In the speech experiment, any movement during speech perception disrupted predictions of spoken turn-ends, whether the movement was task-specific or not. In the music experiment, only task-specific movement (i.e., fingering a piano melody) disrupted predictions of musical turn-ends. These findings support the use of motor simulation to make turn-end predictions in both speech and music but suggest that the specificity of this simulation may differ between domains.

The role of language production in making predictions during comprehension

Language comprehension depends heavily upon prediction, but how predictions are generated remains poorly understood. Several recent theories propose that these predictions are in fact generated by the language production system. Here, we directly test this claim. Participants read sentence contexts that either were or were not highly predictive of a final word, and we measured how quickly participants recognised that final word (Experiment 1), named that final word (Experiment 2), or used that word to name a picture (Experiment 3). We manipulated engagement of the production system by asking participants to read the sentence contexts either aloud or silently. Across the experiments, participants responded more quickly following highly predictive contexts. Importantly, the effect of contextual predictability was greater when participants had read the sentence contexts aloud rather than silently, a finding that was significant in Experiment 3, marginally significant in Experiment 2, and again significant in combined analyses of Experiments 1-3. These results indicate that language production (as used in reading aloud) can be used to facilitate prediction. We consider whether prediction benefits from production only in particular contexts and discuss the theoretical implications of our evidence.

The Same Ultra-Rapid Parallel Brain Dynamics Underpin the Production and Perception of Speech

The temporal dynamics by which linguistic information becomes available is one of the key properties to understand how language is organized in the brain. An unresolved debate between different brain language models is whether words, the building blocks of language, are activated in a sequential or parallel manner. In this study, we approached this issue from a novel perspective by directly comparing the time course of word component activation in speech production versus perception. In an overt object naming task and a passive listening task, we analyzed with mixed linear models at the single-trial level the event-related brain potentials elicited by the same lexico-semantic and phonological word knowledge in the two language modalities. Results revealed that both word components manifested simultaneously as early as 75 ms after stimulus onset in production and perception; differences between the language modalities only became apparent after 300 ms of processing. The data provide evidence for ultra-rapid parallel dynamics of language processing and are interpreted within a neural assembly framework where words recruit the same integrated cell assemblies across production and perception. These word assemblies ignite early on in parallel and only later on reverberate in a behavior-specific manner.

The role of crosslinguistic differences in second language anticipatory processing: An event-related potentials study

The present study uses event-related potentials to investigate how crosslinguistic (dis)similarities modulate anticipatory processing in the second language (L2). Participants read predictive stories in English that made a genitive construction consisting of a third-person singular possessive pronoun and a kinship noun (e.g., his mother) likely in an upcoming continuation. The possessive pronoun's form depended on the antecedent's natural gender, which had been previously established in the stories. The continuation included either the expected genitive construction or an unexpected one with a possessive pronoun of the opposite gender. We manipulated crosslinguistic (dis)similarity by comparing advanced English learners with either Swedish or Spanish as their L1. While Swedish has equivalent possessive pronouns that mark the antecedent's natural gender (i.e., hans/hennes "his/her"), Spanish does not. In fact, Spanish possessive pronouns mark the syntactic features (number, gender) of the possessed noun (e.g., nosotros queremos a nuestra madre "we-_MASC love our-_FEM mother-_FEM). Twenty-four native speakers of English elicited an N400 effect for prenominal possessives that were unexpected based on the possessor noun's natural gender, consistent with the possibility that they activated the pronoun's form or its semantic features (natural gender). Thirty-two Swedish-speaking learners yielded a qualitatively and quantitatively native-like N400 for unexpected prenominal possessives. In contrast, twenty-five Spanish-speaking learners showed a P600 effect for unexpected possessives, consistent with the possibility that they experienced difficulty integrating a pronoun that mismatched the expected gender. Results suggest that differences with respect to the features encoded in the activated representation result in different predictive mechanisms among adult L2 learners.

Anticipating words during spoken discourse comprehension: A large-scale, pre-registered replication study using brain potentials

Numerous studies report brain potential evidence for the anticipation of specific words during language comprehension. In the most convincing demonstrations, highly predictable nouns exert an influence on processing even before they appear to a reader or listener, as indicated by the brain's neural response to a prenominal adjective or article when it mismatches the expectations about the upcoming noun. However, recent studies suggest that some well-known demonstrations of prediction may be hard to replicate. This could signal the use of data-contingent analysis, but might also mean that readers and listeners do not always use prediction-relevant information in the way that psycholinguistic theories typically suggest. To shed light on this issue, we performed a close replication of one of the best-cited ERP studies on word anticipation (Van Berkum, Brown, Zwitserlood, Kooijman & Hagoort, 2005; Experiment 1), in which participants listened to Dutch spoken mini-stories. In the original study, the marking of grammatical gender on pre-nominal adjectives ('groot/grote') elicited an early positivity when mismatching the gender of an unheard, highly predictable noun, compared to matching gender. The current pre-registered study involved that same manipulation, but used a novel set of materials twice the size of the original set, an increased sample size (N = 187), and Bayesian mixed-effects model analyses that better accounted for known sources of variance than the original. In our study, mismatching gender elicited more negative voltage than matching gender at posterior electrodes. However, this N400-like effect was small in size and lacked support from Bayes Factors. In contrast, we successfully replicated the original's noun effects. While our results yielded some support for prediction, they do not support the Van Berkum et al. effect and highlight the risks associated with commonly employed data-contingent analyses and small sample sizes. Our results also raise the question whether Dutch listeners reliably or consistently use adjectival inflection information to inform their noun predictions.

Commonalities in alpha and beta neural desynchronizations during prediction in language comprehension and production

The present study investigates whether predictions during language comprehension are generated by engaging the language production system. Previous studies investigating either prediction or production highlighted M/EEG desynchronization (power decrease) in the alpha (8-10 Hz) and beta (13-30 Hz) frequency bands preceding the target. However, it is unclear whether this electrophysiological modulation underlies common mechanisms. We recorded EEG from participants performing both a comprehension and a production task in two separate blocks. Participants listened to high and low constraint incomplete sentences and were asked either to name a picture to complete them (production) or to simply listen to the final word (comprehension). We found that in a silent gap before the final stimulus, predictable stimuli elicited alpha and beta desynchronization in both tasks, signaling the pre-activation of linguistic information. Source estimation highlighted the involvement of left-lateralized language areas and temporo-parietal areas in the right hemisphere. Furthermore, correlations between the desynchronizations in comprehension and production showed spatiotemporal commonalities in language-relevant areas of the left hemisphere. As proposed by prediction-by-production models, our results suggest that comprehenders engage the production system while predicting upcoming words.

Electrophysiological Dynamics of Visual Speech Processing and the Role of Orofacial Effectors for Cross-Modal Predictions

The human brain generates predictions about future events. During face-to-face conversations, visemic information is used to predict upcoming auditory input. Recent studies suggest that the speech motor system plays a role in these cross-modal predictions, however, usually only audio-visual paradigms are employed. Here we tested whether speech sounds can be predicted on the basis of visemic information only, and to what extent interfering with orofacial articulatory effectors can affect these predictions. We registered EEG and employed N400 as an index of such predictions. Our results show that N400's amplitude was strongly modulated by visemic salience, coherent with cross-modal speech predictions. Additionally, N400 ceased to be evoked when syllables' visemes were presented backwards, suggesting that predictions occur only when the observed viseme matched an existing articuleme in the observer's speech motor system (i.e., the articulatory neural sequence required to produce a particular phoneme/viseme). Importantly, we found that interfering with the motor articulatory system strongly disrupted cross-modal predictions. We also observed a late P1000 that was evoked only for syllable-related visual stimuli, but whose amplitude was not modulated by interfering with the motor system. The present study provides further evidence of the importance of the speech production system for speech sounds predictions based on visemic information at the pre-lexical level. The implications of these results are discussed in the context of a hypothesized trimodal repertoire for speech, in which speech perception is conceived as a highly interactive process that involves not only your ears but also your eyes, lips and tongue.

Linguistic alignment and theory of mind impairments in schizophrenia patients' dialogic interactions

BACKGROUND

Impairments of contextual processing and theory of mind (ToM) have both been offered as accounts of the deviant language characterising formal thought disorder (FTD) in schizophrenia. This study investigated these processes in patients' dialogue. We predicted that FTD patients would show a decrement in linguistic alignment, associated with impaired ToM in dialogue.

METHODS

Speech samples were elicited via participation in an interactive computer-based task and a semi-structured interview to assess contextual processing abilities and ToM skills in dialogue, respectively, and from an interactive card-sorting task to measure syntactic alignment. Degree of alignment in dialogue and the syntactic task, and evidence of ToM in (i) dialogue and (ii) a traditional ToM task were compared across schizophrenia patients with FTD (n = 21), non-FTD patients (n = 22) and healthy controls (n = 21).

RESULTS

FTD patients showed less alignment than the other two groups in dialogue, and than healthy controls on the syntactic task. FTD patients showed poorer performance on the ToM task than the other two groups, but only compared to the healthy controls in dialogue. The FTD group's degree of alignment in dialogue was correlated with ToM performance in dialogue but not with the traditional ToM task or with syntactic alignment.

CONCLUSIONS

In dialogue, FTD patients demonstrate an impairment in employing available contextual information to facilitate their own subsequent production, which is associated with a ToM deficit. These findings indicate that a contextual processing deficit impacts on exploiting representations via the production system impoverishing the ability to make predictions about upcoming utterances in dialogue.

When scenes speak louder than words: Verbal encoding does not mediate the relationship between scene meaning and visual attention

The complexity of the visual world requires that we constrain visual attention and prioritize some regions of the scene for attention over others. The current study investigated whether verbal encoding processes influence how attention is allocated in scenes. Specifically, we asked whether the advantage of scene meaning over image salience in attentional guidance is modulated by verbal encoding, given that we often use language to process information. In two experiments, 60 subjects studied scenes (N1 = 30 and N2 = 60) for 12 s each in preparation for a scene-recognition task. Half of the time, subjects engaged in a secondary articulatory suppression task concurrent with scene viewing. Meaning and saliency maps were quantified for each of the experimental scenes. In both experiments, we found that meaning explained more of the variance in visual attention than image salience did, particularly when we controlled for the overlap between meaning and salience, with and without the suppression task. Based on these results, verbal encoding processes do not appear to modulate the relationship between scene meaning and visual attention. Our findings suggest that semantic information in the scene steers the attentional ship, consistent with cognitive guidance theory.

Listeners are better at predicting speakers similar to themselves

Although it takes several hundred milliseconds to prepare a spoken contribution, gaps between turns in conversation tend to be much shorter. To produce these short gaps, it appears that interlocutors predict the end of their partner's turn. The theory of prediction-by-simulation proposes that individuals use their own motor system to model a partner's upcoming actions by referring to prior production experience. In this study we investigate the role of motor experience for both predicting a turn-end and producing a spoken response by manipulating the similarity of heard speech to participants' own production style. We hypothesised that they would be better at predicting, and initiating responses to, speech produced in the style they speak themselves. Participants recorded a series of questions in two sessions, and several months later they listened to their own speech and that of a stylistically similar and a stylistically dissimilar participant (as assessed by independent raters). Participants predicted the end of 60 of these questions by pressing a button, and for the remaining 60 questions, by producing a spoken response. An analysis of response times showed that participants' button-press responses were faster for utterances spoken by themselves and by a stylistically similar partner, than for utterances spoken by a stylistically dissimilar partner. We conclude that simulation facilitates prediction of similar speakers.

•

Prediction-by-simulation proposes that prediction is based on one's motor experience.

•

If this is the case, predictions should be better for people more like oneself.

•

We tested if speech-style similarity affected prediction during speech listening.

•

Similarity led to better predictions of turn-ends, but not quicker turn-taking.

•

We argue simulation facilitates prediction of similar speakers.

Prediction of orthographic information during listening comprehension: A printed-word visual world study

Two visual world eye-tracking experiments examined the role of orthographic information in the visual context in pre-activation of orthographic and phonological information using Japanese. Participants heard sentences that contained a predictable target word and viewed a display showing four words in a logogram, kanji (Experiment 1) or in a phonogram, hiragana (Experiment 2). The four words included either the target word (e.g., /sakana/; fish), an orthographic competitor (e.g., /tuno/; horn), a phonological competitor (e.g., /sakura/; cherry blossom), or an unrelated word (e.g., /hon/; book), together with three distractor words. The orthographic competitor was orthographically or phonologically dissimilar to the target in hiragana. In Experiment 1, target and orthographic competitor words attracted more fixations than unrelated words before the target word was mentioned, suggesting that participants pre-activated orthographic form of the target word. In Experiment 2, target and phonological competitor words attracted more predictive fixations than unrelated words, but orthographic competitor words did not, suggesting a critical role of the visual context. This pre-activation pattern does not fit with the pattern of lexical activation in auditory word recognition, where orthography and phonology interact. However, it is compatible with the pattern of lexical activation in spoken word production, where orthographic information is not automatically activated, in line with production-based prediction accounts.