1
|
Weissman B, Cohn N, Tanner D. The electrophysiology of lexical prediction of emoji and text. Neuropsychologia 2024; 198:108881. [PMID: 38579906 DOI: 10.1016/j.neuropsychologia.2024.108881] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 02/18/2024] [Accepted: 03/27/2024] [Indexed: 04/07/2024]
Abstract
As emoji often appear naturally alongside text in utterances, they provide a way to study how prediction unfolds in multimodal sentences in direct comparison to unimodal sentences. In this experiment, participants (N = 40) read sentences in which the sentence-final noun appeared in either word form or emoji form, a between-subjects manipulation. The experiment featured both high constraint sentences and low constraint sentences to examine how the lexical processing of emoji interacts with prediction processes in sentence comprehension. Two well-established ERP components linked to lexical processing and prediction - the N400 and the Late Frontal Positivity - are investigated for sentence-final words and emoji to assess whether, to what extent, and in what linguistic contexts emoji are processed like words. Results indicate that the expected effects, namely an N400 effect to an implausible lexical item compared to a plausible one and an LFP effect to an unexpected lexical item compared to an expected one, emerged for both words and emoji. This paper discusses the similarities and differences between the stimulus types and constraint conditions, contextualized within theories of linguistic prediction, ERP components, and a multimodal lexicon.
Collapse
Affiliation(s)
- Benjamin Weissman
- Department of Cognitive Science Rensselaer Polytechnic Institute 110 8th Street, Troy, NY, 12180, USA; Department of Linguistics University of Illinois at Urbana-Champaign 707 S Mathews Ave, Urbana, IL, 61801, USA.
| | - Neil Cohn
- Department of Communication and Cognition Tilburg University PO Box 90153, 5000, LE Tilburg, the Netherlands
| | - Darren Tanner
- Department of Linguistics University of Illinois at Urbana-Champaign 707 S Mathews Ave, Urbana, IL, 61801, USA; AI For Good Lab Microsoft 1 Microsoft Way, Redmond, WA, USA
| |
Collapse
|
2
|
Ding R, Ten Oever S, Martin AE. Delta-band Activity Underlies Referential Meaning Representation during Pronoun Resolution. J Cogn Neurosci 2024; 36:1472-1492. [PMID: 38652108 DOI: 10.1162/jocn_a_02163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/25/2024]
Abstract
Human language offers a variety of ways to create meaning, one of which is referring to entities, objects, or events in the world. One such meaning maker is understanding to whom or to what a pronoun in a discourse refers to. To understand a pronoun, the brain must access matching entities or concepts that have been encoded in memory from previous linguistic context. Models of language processing propose that internally stored linguistic concepts, accessed via exogenous cues such as phonological input of a word, are represented as (a)synchronous activities across a population of neurons active at specific frequency bands. Converging evidence suggests that delta band activity (1-3 Hz) is involved in temporal and representational integration during sentence processing. Moreover, recent advances in the neurobiology of memory suggest that recollection engages neural dynamics similar to those which occurred during memory encoding. Integrating from these two research lines, we here tested the hypothesis that neural dynamic patterns, especially in delta frequency range, underlying referential meaning representation, would be reinstated during pronoun resolution. By leveraging neural decoding techniques (i.e., representational similarity analysis) on a magnetoencephalogram data set acquired during a naturalistic story-listening task, we provide evidence that delta-band activity underlies referential meaning representation. Our findings suggest that, during spoken language comprehension, endogenous linguistic representations such as referential concepts may be proactively retrieved and represented via activation of their underlying dynamic neural patterns.
Collapse
Affiliation(s)
- Rong Ding
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
| | - Sanne Ten Oever
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Radboud University Donders Centre for Cognitive Neuroimaging, Nijmegen, The Netherlands
- Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | - Andrea E Martin
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Radboud University Donders Centre for Cognitive Neuroimaging, Nijmegen, The Netherlands
| |
Collapse
|
3
|
Ryskin R, Nieuwland MS. Prediction during language comprehension: what is next? Trends Cogn Sci 2023; 27:1032-1052. [PMID: 37704456 DOI: 10.1016/j.tics.2023.08.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2022] [Revised: 08/03/2023] [Accepted: 08/04/2023] [Indexed: 09/15/2023]
Abstract
Prediction is often regarded as an integral aspect of incremental language comprehension, but little is known about the cognitive architectures and mechanisms that support it. We review studies showing that listeners and readers use all manner of contextual information to generate multifaceted predictions about upcoming input. The nature of these predictions may vary between individuals owing to differences in language experience, among other factors. We then turn to unresolved questions which may guide the search for the underlying mechanisms. (i) Is prediction essential to language processing or an optional strategy? (ii) Are predictions generated from within the language system or by domain-general processes? (iii) What is the relationship between prediction and memory? (iv) Does prediction in comprehension require simulation via the production system? We discuss promising directions for making progress in answering these questions and for developing a mechanistic understanding of prediction in language.
Collapse
Affiliation(s)
- Rachel Ryskin
- Department of Cognitive and Information Sciences, University of California Merced, 5200 Lake Road, Merced, CA 95343, USA.
| | - Mante S Nieuwland
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands; Donders Institute for Brain, Cognition, and Behaviour, Nijmegen, The Netherlands
| |
Collapse
|
4
|
Maran M, Numssen O, Hartwigsen G, Zaccarella E. Online neurostimulation of Broca's area does not interfere with syntactic predictions: A combined TMS-EEG approach to basic linguistic combination. Front Psychol 2022; 13:968836. [PMID: 36619118 PMCID: PMC9815778 DOI: 10.3389/fpsyg.2022.968836] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2022] [Accepted: 09/13/2022] [Indexed: 01/11/2023] Open
Abstract
Categorical predictions have been proposed as the key mechanism supporting the fast pace of syntactic composition in language. Accordingly, grammar-based expectations are formed-e.g., the determiner "a" triggers the prediction for a noun-and facilitate the analysis of incoming syntactic information, which is then checked against a single or few other word categories. Previous functional neuroimaging studies point towards Broca's area in the left inferior frontal gyrus (IFG) as one fundamental cortical region involved in categorical prediction during incremental language processing. Causal evidence for this hypothesis is however still missing. In this study, we combined Electroencephalography (EEG) and Transcranial Magnetic Stimulation (TMS) to test whether Broca's area is functionally relevant in predictive mechanisms for language. We transiently perturbed Broca's area during the first word in a two-word construction, while simultaneously measuring the Event-Related Potential (ERP) correlates of syntactic composition. We reasoned that if Broca's area is involved in predictive mechanisms for syntax, disruptive TMS during the first word would mitigate the difference in the ERP responses for predicted and unpredicted categories in basic two-word constructions. Contrary to this hypothesis, perturbation of Broca's area at the predictive stage did not affect the ERP correlates of basic composition. The correlation strength between the electrical field induced by TMS and the ERP responses further confirmed this pattern. We discuss the present results considering an alternative account of the role of Broca's area in syntactic composition, namely the bottom-up integration of words into constituents, and of compensatory mechanisms within the language predictive network.
Collapse
Affiliation(s)
- Matteo Maran
- Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany,International Max Planck Research School on Neuroscience of Communication: Function, Structure, and Plasticity, Leipzig, Germany,*Correspondence: Matteo Maran,
| | - Ole Numssen
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Gesa Hartwigsen
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Emiliano Zaccarella
- Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| |
Collapse
|
5
|
Poulton VR, Nieuwland MS. Can You Hear What's Coming? Failure to Replicate ERP Evidence for Phonological Prediction. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2022; 3:556-574. [PMID: 37215344 PMCID: PMC10158594 DOI: 10.1162/nol_a_00078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/05/2021] [Accepted: 07/18/2022] [Indexed: 05/24/2023]
Abstract
Prediction-based theories of language comprehension assume that listeners predict both the meaning and phonological form of likely upcoming words. In alleged event-related potential (ERP) demonstrations of phonological prediction, prediction-mismatching words elicit a phonological mismatch negativity (PMN), a frontocentral negativity that precedes the centroparietal N400 component. However, classification and replicability of the PMN has proven controversial, with ongoing debate on whether the PMN is a distinct component or merely an early part of the N400. In this electroencephalography (EEG) study, we therefore attempted to replicate the PMN effect and its separability from the N400, using a participant sample size (N = 48) that was more than double that of previous studies. Participants listened to sentences containing either a predictable word or an unpredictable word with/without phonological overlap with the predictable word. Preregistered analyses revealed a widely distributed negative-going ERP in response to unpredictable words in both the early (150-250 ms) and the N400 (300-500 ms) time windows. Bayes factor analysis yielded moderate evidence against a different scalp distribution of the effects in the two time windows. Although our findings do not speak against phonological prediction during sentence comprehension, they do speak against the PMN effect specifically as a marker of phonological prediction mismatch. Instead of an PMN effect, our results demonstrate the early onset of the auditory N400 effect associated with unpredictable words. Our failure to replicate further highlights the risk associated with commonly employed data-contingent analyses (e.g., analyses involving time windows or electrodes that were selected based on visual inspection) and small sample sizes in the cognitive neuroscience of language.
Collapse
Affiliation(s)
- Victoria R. Poulton
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK
| | - Mante S. Nieuwland
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, Radboud University, The Netherlands
| |
Collapse
|
6
|
Romero-Ferreiro V, Rodríguez-Gómez P, Pozo MÁ, Moreno EM. Can you change your mind? An ERP study of cognitive flexibility and new evidence integration. Biol Psychol 2022; 172:108354. [DOI: 10.1016/j.biopsycho.2022.108354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2021] [Revised: 04/29/2022] [Accepted: 05/09/2022] [Indexed: 11/16/2022]
|
7
|
Carter GA, Nieuwland MS. Predicting Definite and Indefinite Referents During Discourse Comprehension: Evidence from Event-Related Potentials. Cogn Sci 2022; 46:e13092. [PMID: 35122304 PMCID: PMC9286847 DOI: 10.1111/cogs.13092] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2021] [Revised: 10/10/2021] [Accepted: 12/14/2021] [Indexed: 11/30/2022]
Abstract
Linguistic predictions may be generated from and evaluated against a representation of events and referents described in the discourse. Compatible with this idea, recent work shows that predictions about novel noun phrases include their definiteness. In the current follow‐up study, we ask whether people engage similar prediction‐related processes for definite and indefinite referents. This question is relevant for linguistic theories that imply a processing difference between definite and indefinite noun phrases, typically because definiteness is thought to require a uniquely identifiable referent in the discourse. We addressed this question in an event‐related potential (ERP) study (N = 48) with preregistration of data acquisition, preprocessing, and Bayesian analysis. Participants read Dutch mini‐stories with a definite or indefinite novel noun phrase (e.g., “het/een huis,” the/a house), wherein (in)definiteness of the article was either expected or unexpected and the noun was always strongly expected. Unexpected articles elicited enhanced N400s, but unexpectedly indefinite articles also elicited a positive ERP effect at frontal channels compared to expectedly indefinite articles. We tentatively link this effect to an antiuniqueness violation, which may force people to introduce a new referent over and above the already anticipated one. Interestingly, expectedly definite nouns elicited larger N400s than unexpectedly definite nouns (replicating a previous surprising finding) and indefinite nouns. Although the exact nature of these noun effects remains unknown, expectedly definite nouns may have triggered the strongest semantic activation because they alone refer to specific and concrete referents. In sum, results from both the articles and nouns clearly demonstrate that definiteness marking has a rapid effect on processing, counter to recent claims regarding definiteness processing.
Collapse
Affiliation(s)
- Georgia-Ann Carter
- Max Planck Institute for Psycholinguistics, Nijmegen.,ILCC, School of Informatics, University of Edinburgh, Edinburgh
| | - Mante S Nieuwland
- Max Planck Institute for Psycholinguistics, Nijmegen.,Donders Institute for Brain, Cognition and Behaviour, Nijmegen
| |
Collapse
|
8
|
Amos RM, Seeber KG, Pickering MJ. Prediction during simultaneous interpreting: Evidence from the visual-world paradigm. Cognition 2021; 220:104987. [PMID: 34922159 DOI: 10.1016/j.cognition.2021.104987] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2020] [Revised: 12/01/2021] [Accepted: 12/05/2021] [Indexed: 11/03/2022]
Abstract
We report the results of an eye-tracking study which used the Visual World Paradigm (VWP) to investigate the time-course of prediction during a simultaneous interpreting task. Twenty-four L1 French professional conference interpreters and twenty-four L1 French professional translators untrained in simultaneous interpretation listened to sentences in English and interpreted them simultaneously into French while looking at a visual scene. Sentences contained a highly predictable word (e.g., The dentist asked the man to open his mouth a little wider). The visual scene comprised four objects, one of which depicted either the target object (mouth; bouche), an English phonological competitor (mouse; souris), a French phonological competitor (cork; bouchon), or an unrelated word (bone; os). We considered 1) whether interpreters and translators predict upcoming nouns during a simultaneous interpreting task, 2) whether interpreters and translators predict the form of these nouns in English and in French and 3) whether interpreters and translators manifest different predictive behaviour. Our results suggest that both interpreters and translators predict upcoming nouns, but neither group predicts the word-form of these nouns. In addition, we did not find significant differences between patterns of prediction in interpreters and translators. Thus, evidence from the visual-world paradigm shows that prediction takes place in simultaneous interpreting, regardless of training and experience. However, we were unable to establish whether word-form was predicted.
Collapse
Affiliation(s)
- Rhona M Amos
- University of Geneva, Uni Mail, 40, Boulevard du Pont d'Arve, 1205 Genève, Switzerland.
| | - Kilian G Seeber
- University of Geneva, Uni Mail, 40, Boulevard du Pont d'Arve, 1205 Genève, Switzerland.
| | - Martin J Pickering
- University of Edinburgh, 7 George Square, Edinburgh, EH8 9JZ, Scotland, United Kingdom.
| |
Collapse
|
9
|
Ferreira F, Qiu Z. Predicting syntactic structure. Brain Res 2021; 1770:147632. [PMID: 34453937 DOI: 10.1016/j.brainres.2021.147632] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2021] [Revised: 08/20/2021] [Accepted: 08/21/2021] [Indexed: 11/19/2022]
Abstract
Prediction in language processing has been a topic of major interest in psycholinguistics for at least the last two decades, but most investigations focus on semantic rather than syntactic prediction. This review begins with a discussion of some influential models of parsing which assume that comprehenders have the ability to anticipate syntactic nodes, beginning with left-corner parsers and the garden-path model and ending with current information-theoretic approaches that emphasize online probabilistic prediction. We then turn to evidence for the prediction of specific syntactic forms, including coordinate clauses and noun phrases, verb arguments, and individual nouns, as well as studies that use morphosyntactic constraints to assess whether a specific semantic prediction has been made. The last section considers the implications of syntactic prediction for theories of language architecture and describes four avenues for future research.
Collapse
Affiliation(s)
| | - Zhuang Qiu
- University of California, Davis, United States.
| |
Collapse
|
10
|
Macuch Silva V, Franke M. Pragmatic Prediction in the Processing of Referring Expressions Containing Scalar Quantifiers. Front Psychol 2021; 12:662050. [PMID: 34531781 PMCID: PMC8438145 DOI: 10.3389/fpsyg.2021.662050] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2021] [Accepted: 08/05/2021] [Indexed: 11/17/2022] Open
Abstract
Previous research in cognitive science and psycholinguistics has shown that language users are able to predict upcoming linguistic input probabilistically, pre-activating material on the basis of cues emerging from different levels of linguistic abstraction, from phonology to semantics. Current evidence suggests that linguistic prediction also operates at the level of pragmatics, where processing is strongly constrained by context. To test a specific theory of contextually-constrained processing, termed pragmatic surprisal theory here, we used a self-paced reading task where participants were asked to view visual scenes and then read descriptions of those same scenes. Crucially, we manipulated whether the visual context biased readers into specific pragmatic expectations about how the description might unfold word by word. Contrary to the predictions of pragmatic surprisal theory, we found that participants took longer reading the main critical term in scenarios where they were biased by context and pragmatic constraints to expect a given word, as opposed to scenarios where there was no pragmatic expectation for any particular referent.
Collapse
Affiliation(s)
- Vinicius Macuch Silva
- Cognitive Modeling Group, Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany
| | - Michael Franke
- Cognitive Modeling Group, Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany
| |
Collapse
|
11
|
Nieuwland MS. How 'rational' is semantic prediction? A critique and re-analysis of. Cognition 2021; 215:104848. [PMID: 34274557 DOI: 10.1016/j.cognition.2021.104848] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2020] [Revised: 05/08/2021] [Accepted: 07/11/2021] [Indexed: 01/04/2023]
Abstract
In a recent article in Cognition, Delaney-Busch et al. (2019) claim evidence for 'rational', Bayesian adaptation of semantic predictions, using ERP data from Lau, Holcomb, and Kuperberg (2013). Participants read associatively related and unrelated prime-target word pairs in a first block with only 10% related trials and a second block with 50%. Related words elicited smaller N400s than unrelated words, and this difference was strongest in the second block, suggesting greater engagement in predictive processing. Using a rational adaptor model, Delaney-Busch et al. argue that the stronger N400 reduction for related words in the second block developed as a function of the number of related trials, and concluded therefore that participants predicted related words more strongly when their predictions were fulfilled more often. In this critique, I discuss two critical flaws in their analyses, namely the confounding of prediction effects with those of lexical frequency and the neglect of data from the first block. Re-analyses suggest a different picture: related words by themselves did not yield support for their conclusion, and the effect of relatedness gradually strengthened in othe two blocks in a similar way. Therefore, the N400 did not yield evidence that participants rationally adapted their semantic predictions. Within the framework proposed by Delaney-Busch et al., presumed semantic predictions may even be thought of as 'irrational'. While these results yielded no evidence for rational or probabilistic prediction, they do suggest that participants became increasingly better at predicting target words from prime words.
Collapse
Affiliation(s)
- Mante S Nieuwland
- Max-Planck-Institute for Psycholinguistics, The Netherlands; Donders Centre for Cognition, Brain and Behaviour, The Netherlands.
| |
Collapse
|
12
|
Vega-Mendoza M, Pickering MJ, Nieuwland MS. Concurrent use of animacy and event-knowledge during comprehension: Evidence from event-related potentials. Neuropsychologia 2021; 152:107724. [PMID: 33347913 DOI: 10.1016/j.neuropsychologia.2020.107724] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2019] [Revised: 10/02/2020] [Accepted: 12/08/2020] [Indexed: 10/22/2022]
Abstract
In two ERP experiments, we investigated whether readers prioritize animacy over real-world event-knowledge during sentence comprehension. We used the paradigm of Paczynski and Kuperberg (2012), who argued that animacy is prioritized based on the observations that the 'related anomaly effect' (reduced N400s for context-related anomalous words compared to unrelated words) does not occur for animacy violations, and that animacy violations but not relatedness violations elicit P600 effects. Participants read passive sentences with plausible agents (e.g., The prescription for the mental disorder was written by the psychiatrist) or implausible agents that varied in animacy and semantic relatedness (schizophrenic/guard/pill/fence). In Experiment 1 (with a plausibility judgment task), plausible sentences elicited smaller N400s relative to all types of implausible sentences. Crucially, animate words elicited smaller N400s than inanimate words, and related words elicited smaller N400s than unrelated words, but Bayesian analysis revealed substantial evidence against an interaction between animacy and relatedness. Moreover, at the P600 time-window, we observed more positive ERPs for animate than inanimate words and for related than unrelated words at anterior regions. In Experiment 2 (without judgment task), we observed an N400 effect with animacy violations, but no other effects. Taken together, the results of our experiments fail to support a prioritized role of animacy information over real-world event-knowledge, but they support an interactive, constraint-based view on incremental semantic processing.
Collapse
Affiliation(s)
- Mariana Vega-Mendoza
- Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, Scotland, UK; Department of Business Administration, Technology and Social Sciences. Engineering Psychology, Luleå University of Technology, Luleå, Sweden.
| | - Martin J Pickering
- Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, Scotland, UK
| | - Mante S Nieuwland
- Department of Psychology, School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, Scotland, UK; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands; Donders Institute for Cognition, Brain and Behaviour, Nijmegen, the Netherlands; Heinrich-Heine-University, Düsseldorf, Germany
| |
Collapse
|