Prospective and retrospective semantic processing: prediction, time, and relationship strength in event-related potentials

Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.

Speech register influences listeners' word expectations

We utilized the N400 effect to investigate the influence of speech register on predictive language processing. Participants listened to long stretches (4 - 15 min) of naturalistic speech from different registers (dialogues, news broadcasts, and read-aloud books), totalling approximately 50,000 words, while the EEG signal was recorded. We estimated the surprisal of words in the speech materials with the aid of a statistical language model in such a manner that it reflected different predictive processing strategies; generic, register-specific, or recency-based. The N400 amplitude was best predicted with register-specific word surprisal, indicating that the statistics of the wider context (i.e., register) influences predictive language processing. Furthermore, adaptation to speech register cannot merely be explained by recency effects; instead, listeners adapt their word anticipations to the presented speech register.

Dynamic branching in a neural network model for probabilistic prediction of sequences

An important function of the brain is to predict which stimulus is likely to occur based on the perceived cues. The present research studied the branching behavior of a computational network model of populations of excitatory and inhibitory neurons, both analytically and through simulations. Results show how synaptic efficacy, retroactive inhibition and short-term synaptic depression determine the dynamics of selection between different branches predicting sequences of stimuli of different probabilities. Further results show that changes in the probability of the different predictions depend on variations of neuronal gain. Such variations allow the network to optimize the probability of its predictions to changing probabilities of the sequences without changing synaptic efficacy.

Capacity limits in sentence comprehension: Evidence from dual-task judgements and event-related potentials

There is an ongoing controversy over whether readers can access the meaning of multiple words, simultaneously. To date, different experimental methods have generated seemingly contradictory evidence in support of serial or parallel processing accounts. For example, dual-task studies suggest that readers can process a maximum of one word at a time (White, Palmer & Boynton, 2018), while ERP studies have demonstrated neural priming effects that are more consistent with parallel activation (Wen, Snell & Grainger, 2019). To help reconcile these views, I measured neural responses and behavioral accuracy in a dual-task sentence comprehension paradigm. Participants saw masked sentences and two-word phrases and had to judge whether or not they were grammatical. Grammatically correct sentences (This girl is neat) produced smaller N400 responses compared to scrambled sentences (Those girl is fled): an N400 sentence superiority effect. Critically, participants' grammaticality judgements on the same trials showed striking capacity limitations, with dual-task deficits closely matching the predictions of a serial, all-or-none processing account. Together, these findings suggest that the N400 sentence superiority effect is fully compatible with serial word recognition, and that readers are unable to process multiple sentence positions simultaneously.

DHH Students' Phoneme Repetition Awareness in Sentence Reading

To examine deaf and hard-of-hearing (DHH) students' awareness of phoneme repetition in Chinese sentence reading, two experiments were conducted in the self-paced, moving-window reading paradigm. The materials comprised sentences in which Chinese characters that sequentially followed each other shared similar spelling initials and finals in Experiments 1 and 2, respectively. In comparison to hearing participants, DHH participants were more likely to find it more time-consuming to read sentences with, as opposed to without, phoneme repetitions. However, their difficulty in phonological processing seemed to be linked to their weakness at syntactic skilfulness, thus having a negative influence on their reading performance. It is concluded that Chinese DHH college students have developed a phoneme repetition awareness which is different from how hearing college students are aware of phoneme repetitions in Chinese sentence reading. It is implicated that DHH students are able to develop their own skills of phonological information processing in sentence reading as a result of many practices.

DHH Students' Comprehension of Irony in Self-paced Reading

Irony comprehension can be a kind of challenge to those who are relatively less skillful in reading. To examine how DHH college students (DCSs) were different from hearing college students (HCSs) in the reading of ironic discourses, we conducted two experiments in the self-paced reading task. In Experiment 1, the statement was either literally congruent with the preceding context or had to be understood in an ironic way in each trial; In Experiment 2, the statement was the same but the context was not across the two levels of discourse type. The DCSs generally had a poorer performance than the hearing participants. Although able to comprehend ironies, they had a significantly lower efficiency than their hearing counterparts. The results were consistently in agreement with the prediction of the graded salience hypothesis (Giora, R. (1997). Understanding figurative and literal language: The graded salience hypothesis. Cognitive Linguistics, 7, 183-206. doi:10.1515/cogl.1997.8.3.183) and the parallel-constraint-satisfaction framework (Pexman, P. M. (2008). It's fascinating research. Current Directions in Psychological Science, 17(4)286-290. doi:10.1111/j.1467-8721.2008.00591.x), and the DCSs' performance appears to indicate an amplified version of this support. It is implied that educational environments should be created in which deaf and DHH students are encouraged to do as much reading as possible. Exercises should be designed in helping them to improve vocabulary and syntactic skills in general and to improve skills of inference-making in particular.

Are Sensory-Motor Relationships Encoded ad hoc or by Default?: An ERP Study

In this event-related potentials study we tested whether sensory-motor relations between concrete words are encoded by default or only under explicit ad hoc instructions. In Exp. 1, participants were explicitly asked to encode sensory-motor relations (e.g., “do the following objects fit in a pencil-cup?”), while other possible semantic relations remained implicit. In Exp. 2, using the same materials other group of participants were explicitly asked to encode semantic relations (e.g., “are the following objects related to a pencil-cup?”), and the possible sensory-motor relations remained implicit. The N400 component was sensitive to semantic relations (e.g., “desk” related to “pencil-cup”) both under implicit (Exp. 1) and explicit instructions (Exp. 2). By contrast, most sensory-motor relations (e.g., “pea” fitting in “pencil-cup”) were encoded ad hoc under explicit instructions (Exp. 1). Interestingly some sensory-motor relations were also encoded implicitly, but only when they corresponded to “functional” actions associated with high-related objects (e.g., “eraser” fitting in “pencil-cup”) and occurring at a late time window (500–650 ms; Exp. 2), suggesting that this type of sensory-motor relations were encoding by default.

N400 amplitude does not recover from disappearance after repetitions despite reinitiated semantic integration difficulty

It remains an open question whether the amplitude of N400 reflects combinatory postlexical semantic integration processing. To examine the issue, we repeatedly presented strictly simplified, N400-eliciting three-word structures for seven times, mixed with their plausible counterparts, followed immediately by a much more enriched and informative sentence containing two keywords of the incongruous structure, for the purpose of reinitiating semantic integration processing. Event-related potentials were recorded and compared at the first, fourth, seventh, and eighth time. It was found that multiple repetitions attenuated the N400 effect to almost nonexistent and that the follow-up semantic integration reinitiating sentence did not recover N400 amplitude. The results suggest that combinatory postlexical semantic integration does not significantly modulate N400 amplitude, and provide evidence for noncombinatory processes underlying N400 such as automatic spreading activation and expectancy/prediction.

Exploring the Organization of Semantic Memory through Unsupervised Analysis of Event-related Potentials

Modern multivariate methods have enabled the application of unsupervised techniques to analyze neurophysiological data without strict adherence to predefined experimental conditions. We demonstrate a multivariate method that leverages priming effects on the evoked potential to perform hierarchical clustering on a set of word stimuli. The current study focuses on the semantic relationships that play a key role in the organization of our mental lexicon of words and concepts. The N400 component of the event-related potential is considered a reliable neurophysiological response that is indicative of whether accessing one concept facilitates subsequent access to another (i.e., one "primes" the other). To further our understanding of the organization of the human mental lexicon, we propose to utilize the N400 component to drive a clustering algorithm that can uncover, given a set of words, which particular subsets of words show mutual priming. Such a scheme requires a reliable measurement of the amplitude of the N400 component without averaging across many trials, which was here achieved using a recently developed multivariate analysis method based on beamforming. We validated our method by demonstrating that it can reliably detect, without any prior information about the nature of the stimuli, a well-known feature of the organization of our semantic memory: the distinction between animate and inanimate concepts. These results motivate further application of our method to data-driven exploration of disputed or unknown relationships between stimuli.

Latching dynamics in neural networks with synaptic depression

Prediction is the ability of the brain to quickly activate a target concept in response to a related stimulus (prime). Experiments point to the existence of an overlap between the populations of the neurons coding for different stimuli, and other experiments show that prime-target relations arise in the process of long term memory formation. The classical modelling paradigm is that long term memories correspond to stable steady states of a Hopfield network with Hebbian connectivity. Experiments show that short term synaptic depression plays an important role in the processing of memories. This leads naturally to a computational model of priming, called latching dynamics; a stable state (prime) can become unstable and the system may converge to another transiently stable steady state (target). Hopfield network models of latching dynamics have been studied by means of numerical simulation, however the conditions for the existence of this dynamics have not been elucidated. In this work we use a combination of analytic and numerical approaches to confirm that latching dynamics can exist in the context of a symmetric Hebbian learning rule, however lacks robustness and imposes a number of biologically unrealistic restrictions on the model. In particular our work shows that the symmetry of the Hebbian rule is not an obstruction to the existence of latching dynamics, however fine tuning of the parameters of the model is needed.

Episodic Specificity in Acquiring Thematic Knowledge of Novel Words from Descriptive Episodes

The current study examined whether thematic relations of the novel words could be acquired via descriptive episodes, and if yes, whether it could be generalized to thematically related words in a different scenario. In Experiment 1, a lexical decision task was used where the novel words served as primes for target words in four conditions: (1) corresponding concepts of the novel words, (2) thematically related words in the same episodes as that in learning condition, (3) thematically related words in different episodes, or (4) unrelated words served as targets. Event related potentials elicited by the targets revealed that compared to the unrelated words, the corresponding concepts and thematically related words in the same episodes elicited smaller N400s with a frontal-central distribution, whereas the thematically related words in different episodes elicited an enhanced late positive component. Experiment 2 further showed a priming effect of the corresponding concepts on the thematically related words in the same episodes as well as in a different episode, indicating that the absence of a priming effect of the learned novel words on the thematically related words in different episode could not be attributed to inappropriate selection of thematically related words in the two conditions. These results indicate that only the corresponding concepts and the thematically related words in the learning episodes were successfully primed, whereas the thematic association between the novel words and the thematically related words in different scenarios could only be recognized in a late processing stage. Our findings suggest that thematic knowledge of novel words is organized via separate scenarios, which are represented in a clustered manner in the semantic network.

Semantic integration by pattern priming: experiment and cortical network model

Neural network models describe semantic priming effects by way of mechanisms of activation of neurons coding for words that rely strongly on synaptic efficacies between pairs of neurons. Biologically inspired Hebbian learning defines efficacy values as a function of the activity of pre- and post-synaptic neurons only. It generates only pair associations between words in the semantic network. However, the statistical analysis of large text databases points to the frequent occurrence not only of pairs of words (e.g., "the way") but also of patterns of more than two words (e.g., "by the way"). The learning of these frequent patterns of words is not reducible to associations between pairs of words but must take into account the higher level of coding of three-word patterns. The processing and learning of pattern of words challenges classical Hebbian learning algorithms used in biologically inspired models of priming. The aim of the present study was to test the effects of patterns on the semantic processing of words and to investigate how an inter-synaptic learning algorithm succeeds at reproducing the experimental data. The experiment manipulates the frequency of occurrence of patterns of three words in a multiple-paradigm protocol. Results show for the first time that target words benefit more priming when embedded in a pattern with the two primes than when only associated with each prime in pairs. A biologically inspired inter-synaptic learning algorithm is tested that potentiates synapses as a function of the activation of more than two pre- and post-synaptic neurons. Simulations show that the network can learn patterns of three words to reproduce the experimental results.

Gradients versus dichotomies: how strength of semantic context influences event-related potentials and lexical decision times

In experiments devoted to word recognition and/or language comprehension, reaction time in the lexical decision task is perhaps the most commonly used behavioral dependent measure, and the amplitude of the N400 component of the event-related potential (ERP) is the most common neural measure. Both are sensitive to multiple factors, including frequency of usage, orthographic similarity to other words, concreteness of word meaning, and preceding semantic context. All of these factors vary continuously. Published results have shown that both lexical decision times and N400 amplitudes show graded responses to graded changes of word frequency and orthographic similarity, but a puzzling discrepancy in their responsivity to the strength of a semantic context has received little attention. In three experiments, we presented pairs of words varying in the strengths of their semantic relationships, as well as unrelated pairs. In all three experiments, N400 amplitudes showed a gradient from unrelated to weakly associated to strongly associated target words, whereas lexical decision times showed a binary division rather than a gradient across strengths of relationship. This pattern of results suggests that semantic context effects in lexical decision and ERP measures arise from fundamentally different processes.

Effects of prediction and contextual support on lexical processing: prediction takes precedence

Readers may use contextual information to anticipate and pre-activate specific lexical items during reading. However, prior studies have not clearly dissociated the effects of accurate lexical prediction from other forms of contextual facilitation such as plausibility or semantic priming. In this study, we measured electrophysiological responses to predicted and unpredicted target words in passages providing varying levels of contextual support. This method was used to isolate the neural effects of prediction from other potential contextual influences on lexical processing. While both prediction and discourse context influenced ERP amplitudes within the time range of the N400, the effects of prediction occurred much more rapidly, preceding contextual facilitation by approximately 100 ms. In addition, a frontal, post-N400 positivity (PNP) was modulated by both prediction accuracy and the overall plausibility of the preceding passage. These results suggest a unique temporal primacy for prediction in facilitating lexical access. They also suggest that the frontal PNP may index the costs of revising discourse representations following an incorrect lexical prediction.

Examining the N400 semantic context effect item-by-item: relationship to corpus-based measures of word co-occurrence

With increasing availability of digital text, there has been an explosion of computational methods designed to turn patterns of word co-occurrence in large text corpora into numerical scores expressing the "semantic distance" between any two words. The success of such methods is typically evaluated by how well they predict human judgments of similarity. Here, I examine how well corpus-based methods predict amplitude of the N400 component of the event-related potential (ERP), an online measure of lexical processing in brain electrical activity. ERPs elicited by the second words of 303 word pairs were analyzed at the level of individual items. Three corpus-based measures (mutual information, distributional similarity, and latent semantic analysis) were compared to a traditional measure of free association strength. In a regression analysis, corpus-based and free association measures each explained some of the variance in N400 amplitude, suggesting that these may tap distinct aspects of word relationships. Lexical factors of concreteness of word meaning, word frequency, number of semantic associates, and orthographic similarity also explained variance in N400 amplitude at the single-item level.