Neural correlates of word representation vectors in natural language processing models: Evidence from representational similarity analysis of event-related brain potentials

DERCo: A Dataset for Human Behaviour in Reading Comprehension Using EEG

This paper introduces the DERCo (Dublin EEG-based Reading Experiment Corpus), a language resource combining electroencephalography (EEG) and next-word prediction data obtained from participants reading narrative texts. The dataset comprises behavioral data collected from 500 participants recruited through the Amazon Mechanical Turk online crowd-sourcing platform, along with EEG recordings from 22 healthy adult native English speakers. The online experiment was designed to examine the context-based word prediction by a large sample of participants, while the EEG-based experiment was developed to extend the validation of behavioral next-word predictability. Online participants were instructed to predict upcoming words and complete entire stories. Cloze probabilities were then calculated for each word so that this predictability measure could be used to support various analyses pertaining to semantic context effects in the EEG recordings. EEG-based analyses revealed significant differences between high and low predictable words, demonstrating one important type of potential analysis that necessitates close integration of these two datasets. This material is a valuable resource for researchers in neurolinguistics due to the word-level EEG recordings in context.

The primacy of taxonomic semantic organization over thematic semantic organization during picture naming

Different organizational structures have been argued to underlie semantic knowledge about concepts; taxonomic organization, based on shared features, and thematic organization based on co-occurrence in common scenes and scenarios. The goal of the current study is to examine which of the two organizational systems are more engaged in the semantic context of a picture naming task. To address this question, we examined the representational structure underlying the semantic space in different picture naming tasks by applying representational similarity analysis (RSA) to electroencephalography (EEG) datasets. In a series of experiments, EEG signals were collected while participants named pictures under different semantic contexts. Study 1 reanalyzes existing data from semantic contexts directing attention to taxonomic organization and semantic contexts that are not biased towards either taxonomic or thematic organization. In Study 2 we keep the stimuli the same and vary semantic contexts to draw attention to either taxonomic or thematic organization. The RSA approach allows us to examine the pairwise similarity in scalp-recorded amplitude patterns at each time point following the onset of the picture and relate it to theoretical taxonomic and thematic measures derived from computational models of semantics. Across all tasks, the similarity structure of scalp-recorded neural activity correlated better with taxonomic than thematic measures, in time windows associated with semantic processing. Most strikingly, we found that the scalp-recorded patterns of neural activity between taxonomically related items were more similar to each other than the scalp-recorded patterns of neural activity for thematically related or unrelated items, even in tasks that makes thematic information more salient. These results suggest that the principle semantic organization of these concepts during picture naming is taxonomic, at least in the context of picture naming.

Dissociating the pre-activation of word meaning and form during sentence comprehension: Evidence from EEG representational similarity analysis

During language comprehension, the processing of each incoming word is facilitated in proportion to its predictability. Here, we asked whether anticipated upcoming linguistic information is actually pre-activated before new bottom-up input becomes available, and if so, whether this pre-activation is limited to the level of semantic features, or whether extends to representations of individual word-forms (orthography/phonology). We carried out Representational Similarity Analysis on EEG data while participants read highly constraining sentences. Prior to the onset of the expected target words, sentence pairs predicting semantically related words (financial "bank" - "loan") and form-related words (financial "bank" - river "bank") produced more similar neural patterns than pairs predicting unrelated words ("bank" - "lesson"). This provides direct neural evidence for item-specific semantic and form predictive pre-activation. Moreover, the semantic pre-activation effect preceded the form pre-activation effect, suggesting that top-down pre-activation is propagated from higher to lower levels of the linguistic hierarchy over time.

Multivariate pattern analysis of EEG reveals nuanced impact of negation on sentence processing in the N400 and later time windows

The neurocognitive mechanism underlying negation processing remains controversial. While negation is suggested to modulate the access of word meaning, no such evidence has been observed in the event-related potential (ERP) literature on sentence processing. In the current study, we applied both univariate ERP and multivariate pattern analysis (MVPA) methods to examine the processing of sentence negation. We investigated two types of negative congruent/incongruent sentence pairs with truth-value evaluation (e.g., "A robin is a/not a bird") and without (e.g., "The woman reads a/no book"). In the N400 time window, ERPs consistently showed increased negativity for negative and incongruent conditions. MVPA, on the other hand, revealed nuanced interactions between polarity and congruency. In the later P600 time window, MVPA but not the ERPs revealed an effect of congruency, which may be functionally distinct from the N400 window. We further used cross-decoding to show that the cognitive processes underlying the N400 window for both affirmative and negative sentences are comparable, whereas in the P600 window, only for the truth sentences, negative sentences showed a distinct pattern from their affirmative counterparts. Our results thus speak for a more interactive, but nevertheless serial and biphasic, and potentially construction-specific processing account of negation. We also discuss the advantage of applying MVPA in addition to the classical univariate methods for a better understanding of the neurobiology of negation processing and language comprehension alike.

Intention affects fairness processing: Evidence from behavior and representational similarity analysis of event-related potential signals

In an ultimatum game, the responder must decide between pursuing self-interest and insisting on fairness, and these choices are affected by the intentions of the proposer. However, the time course of this social decision-making process is unclear. Representational similarity analysis (RSA) is a useful technique for linking brain activity with rich behavioral data sets. In this study, electroencephalography (EEG) was used to measure the time course of neural responses to proposed allocation schemes with different intentions. Twenty-eight participants played an ultimatum game as responders. They had to choose between accepting and rejecting the fair or unfair money allocation schemes of proposers. The schemes were offered based on the proposer's selfish intention (monetary gain), altruistic intention (donation to charity), or ambiguous intention (unknown to the responder). We used a spatiotemporal RSA and inter-subject RSA (IS-RSA) to explore the connections between event-related potentials (ERPs) after offer presentation and intention presentation with four types of behavioral data (acceptance, response time, fairness ratings, and pleasantness ratings). The spatiotemporal RSA results revealed that only response time variation was linked with the difference in ERPs at 432-592 ms after offer presentation on the posterior parietal and prefrontal regions. Meanwhile, the IS-RSA results found a significant association between inter-individual differences in response time and differences in ERP activity at 596-812 ms after the presentation of ambiguous intention, particularly in the prefrontal region. This study expands the intention-based reciprocal model to the third-party context and demonstrates that brain activity can represent response time differences in social decision-making.

Half-listening or zoned out? It's about the same: the impact of attentional state on word processing in context

Language comprehension must require some degree of attentional focus, but how do periods of inattention and/or split attention impact how language is processed? Here EEG was recorded while participants listened to full-length stories, and were periodically asked about whether they were fully attentive, were completely inattentive, or felt that they were in a split attention state. The ERP response to the words immediately preceding these attention questions was examined as a function of participant response, which allowed for the comparison of word processing in each of these attentional states. When participants were on-task, typical N400 effects of lexical frequency (smaller N400 for common compared to less common words), word position (smaller N400 for words appearing late in a sentence compared to words appearing with less preceding context), and surprisal (smaller N400 for relatively expected words compared to relatively unexpected words) were observed. When participants were in a fully inattentive state, the word-level effect of frequency was intact, but the context-dependent effects of word position and surprisal were significantly reduced. Interestingly, the pattern of results when participants were in a split attention state closely matched that of the fully inattentive state. Overall, the results demonstrate how attentional state influences sensitivity to language context during comprehension, and show that the consequences of inattention and split attention on word processing in context are quite similar, at least on the indices measured here.