Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension

The Language Network Reliably "Tracks" Naturalistic Meaningful Nonverbal Stimuli

The language network, comprised of brain regions in the left frontal and temporal cortex, responds robustly and reliably during language comprehension but shows little or no response during many nonlinguistic cognitive tasks (e.g., Fedorenko & Blank, 2020). However, one domain whose relationship with language remains debated is semantics-our conceptual knowledge of the world. Given that the language network responds strongly to meaningful linguistic stimuli, could some of this response be driven by the presence of rich conceptual representations encoded in linguistic inputs? In this study, we used a naturalistic cognition paradigm to test whether the cognitive and neural resources that are responsible for language processing are also recruited for processing semantically rich nonverbal stimuli. To do so, we measured BOLD responses to a set of ∼5-minute-long video and audio clips that consisted of meaningful event sequences but did not contain any linguistic content. We then used the intersubject correlation (ISC) approach (Hasson et al., 2004) to examine the extent to which the language network "tracks" these stimuli, that is, exhibits stimulus-related variation. Across all the regions of the language network, meaningful nonverbal stimuli elicited reliable ISCs. These ISCs were higher than the ISCs elicited by semantically impoverished nonverbal stimuli (e.g., a music clip), but substantially lower than the ISCs elicited by linguistic stimuli. Our results complement earlier findings from controlled experiments (e.g., Ivanova et al., 2021) in providing further evidence that the language network shows some sensitivity to semantic content in nonverbal stimuli.

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Current large language models (LLMs) rely on word prediction as their backbone pretraining task. Although word prediction is an important mechanism underlying language processing, human language comprehension occurs at multiple levels, involving the integration of words and sentences to achieve a full understanding of discourse. This study models language comprehension by using the next sentence prediction (NSP) task to investigate mechanisms of discourse-level comprehension. We show that NSP pretraining enhanced a model's alignment with brain data especially in the right hemisphere and in the multiple demand network, highlighting the contributions of nonclassical language regions to high-level language understanding. Our results also suggest that NSP can enable the model to better capture human comprehension performance and to better encode contextual information. Our study demonstrates that the inclusion of diverse learning objectives in a model leads to more human-like representations, and investigating the neurocognitive plausibility of pretraining tasks in LLMs can shed light on outstanding questions in language neuroscience.

The language network as a natural kind within the broader landscape of the human brain

Language behaviour is complex, but neuroscientific evidence disentangles it into distinct components supported by dedicated brain areas or networks. In this Review, we describe the 'core' language network, which includes left-hemisphere frontal and temporal areas, and show that it is strongly interconnected, independent of input and output modalities, causally important for language and language-selective. We discuss evidence that this language network plausibly stores language knowledge and supports core linguistic computations related to accessing words and constructions from memory and combining them to interpret (decode) or generate (encode) linguistic messages. We emphasize that the language network works closely with, but is distinct from, both lower-level - perceptual and motor - mechanisms and higher-level systems of knowledge and reasoning. The perceptual and motor mechanisms process linguistic signals, but, in contrast to the language network, are sensitive only to these signals' surface properties, not their meanings; the systems of knowledge and reasoning (such as the system that supports social reasoning) are sometimes engaged during language use but are not language-selective. This Review lays a foundation both for in-depth investigations of these different components of the language processing pipeline and for probing inter-component interactions.

Multilingualism at the Market: A Pre-registered Immersive Virtual Reality Study of Bilingual Language Switching

Bilinguals, by definition, are capable of expressing themselves in more than one language. But which cognitive mechanisms allow them to switch from one language to another? Previous experimental research using the cued language-switching paradigm supports theoretical models that assume that both transient, reactive and sustained, proactive inhibitory mechanisms underlie bilinguals' capacity to flexibly and efficiently control which language they use. Here we used immersive virtual reality to test the extent to which these inhibitory mechanisms may be active when unbalanced Dutch-English bilinguals i) produce full sentences rather than individual words, ii) to a life-size addressee rather than only into a microphone, iii) using a message that is relevant to that addressee rather than communicatively irrelevant, iv) in a rich visual environment rather than in front of a computer screen. We observed a reversed language dominance paired with switch costs for the L2 but not for the L1 when participants were stand owners in a virtual marketplace and informed their monolingual customers in full sentences about the price of their fruits and vegetables. These findings strongly suggest that the subtle balance between the application of reactive and proactive inhibitory mechanisms that support bilingual language control may be different in the everyday life of a bilingual compared to in the (traditional) psycholinguistic laboratory.

How does the human brain process noisy speech in real life? Insights from the second-person neuroscience perspective

Comprehending speech with the existence of background noise is of great importance for human life. In the past decades, a large number of psychological, cognitive and neuroscientific research has explored the neurocognitive mechanisms of speech-in-noise comprehension. However, as limited by the low ecological validity of the speech stimuli and the experimental paradigm, as well as the inadequate attention on the high-order linguistic and extralinguistic processes, there remains much unknown about how the brain processes noisy speech in real-life scenarios. A recently emerging approach, i.e., the second-person neuroscience approach, provides a novel conceptual framework. It measures both of the speaker's and the listener's neural activities, and estimates the speaker-listener neural coupling with regarding of the speaker's production-related neural activity as a standardized reference. The second-person approach not only promotes the use of naturalistic speech but also allows for free communication between speaker and listener as in a close-to-life context. In this review, we first briefly review the previous discoveries about how the brain processes speech in noise; then, we introduce the principles and advantages of the second-person neuroscience approach and discuss its implications to unravel the linguistic and extralinguistic processes during speech-in-noise comprehension; finally, we conclude by proposing some critical issues and calls for more research interests in the second-person approach, which would further extend the present knowledge about how people comprehend speech in noise.

The neural and cognitive basis of expository text comprehension

As science and technology rapidly progress, it becomes increasingly important to understand how individuals comprehend expository technical texts that explain these advances. This study examined differences in individual readers' technical comprehension performance and differences among texts, using functional brain imaging to measure regional brain activity while students read passages on technical topics and then took a comprehension test. Better comprehension of the technical passages was related to higher activation in regions of the left inferior frontal gyrus, left superior parietal lobe, bilateral dorsolateral prefrontal cortex, and bilateral hippocampus. These areas are associated with the construction of a mental model of the passage and with the integration of new and prior knowledge in memory. Poorer comprehension of the passages was related to greater activation of the ventromedial prefrontal cortex and the precuneus, areas involved in autobiographical and episodic memory retrieval. More comprehensible passages elicited more brain activation associated with establishing links among different types of information in the text and activation associated with establishing conceptual coherence within the text representation. These findings converge with previous behavioral research in their implications for teaching technical learners to become better comprehenders and for improving the structure of instructional texts, to facilitate scientific and technological comprehension.

A Deep Learning Approach to Analyzing Continuous-Time Cognitive Processes

The dynamics of the mind are complex. Mental processes unfold continuously in time and may be sensitive to a myriad of interacting variables, especially in naturalistic settings. But statistical models used to analyze data from cognitive experiments often assume simplistic dynamics. Recent advances in deep learning have yielded startling improvements to simulations of dynamical cognitive processes, including speech comprehension, visual perception, and goal-directed behavior. But due to poor interpretability, deep learning is generally not used for scientific analysis. Here, we bridge this gap by showing that deep learning can be used, not just to imitate, but to analyze complex processes, providing flexible function approximation while preserving interpretability. To do so, we define and implement a nonlinear regression model in which the probability distribution over the response variable is parameterized by convolving the history of predictors over time using an artificial neural network, thereby allowing the shape and continuous temporal extent of effects to be inferred directly from time series data. Our approach relaxes standard simplifying assumptions (e.g., linearity, stationarity, and homoscedasticity) that are implausible for many cognitive processes and may critically affect the interpretation of data. We demonstrate substantial improvements on behavioral and neuroimaging data from the language processing domain, and we show that our model enables discovery of novel patterns in exploratory analyses, controls for diverse confounds in confirmatory analyses, and opens up research questions in cognitive (neuro)science that are otherwise hard to study.

Word Frequency and Predictability Dissociate in Naturalistic Reading

Many studies of human language processing have shown that readers slow down at less frequent or less predictable words, but there is debate about whether frequency and predictability effects reflect separable cognitive phenomena: are cognitive operations that retrieve words from the mental lexicon based on sensory cues distinct from those that predict upcoming words based on context? Previous evidence for a frequency-predictability dissociation is mostly based on small samples (both for estimating predictability and frequency and for testing their effects on human behavior), artificial materials (e.g., isolated constructed sentences), and implausible modeling assumptions (discrete-time dynamics, linearity, additivity, constant variance, and invariance over time), which raises the question: do frequency and predictability dissociate in ordinary language comprehension, such as story reading? This study leverages recent progress in open data and computational modeling to address this question at scale. A large collection of naturalistic reading data (six datasets, >2.2 M datapoints) is analyzed using nonlinear continuous-time regression, and frequency and predictability are estimated using statistical language models trained on more data than is currently typical in psycholinguistics. Despite the use of naturalistic data, strong predictability estimates, and flexible regression models, results converge with earlier experimental studies in supporting dissociable and additive frequency and predictability effects.

Large-scale evidence for logarithmic effects of word predictability on reading time

During real-time language comprehension, our minds rapidly decode complex meanings from sequences of words. The difficulty of doing so is known to be related to words' contextual predictability, but what cognitive processes do these predictability effects reflect? In one view, predictability effects reflect facilitation due to anticipatory processing of words that are predictable from context. This view predicts a linear effect of predictability on processing demand. In another view, predictability effects reflect the costs of probabilistic inference over sentence interpretations. This view predicts either a logarithmic or a superlogarithmic effect of predictability on processing demand, depending on whether it assumes pressures toward a uniform distribution of information over time. The empirical record is currently mixed. Here, we revisit this question at scale: We analyze six reading datasets, estimate next-word probabilities with diverse statistical language models, and model reading times using recent advances in nonlinear regression. Results support a logarithmic effect of word predictability on processing difficulty, which favors probabilistic inference as a key component of human language processing.

Beyond the tried and true: How virtual reality, dialog setups, and a focus on multimodality can take bilingual language production research forward

Bilinguals possess the ability of expressing themselves in more than one language, and typically do so in contextually rich and dynamic settings. Theories and models have indeed long considered context factors to affect bilingual language production in many ways. However, most experimental studies in this domain have failed to fully incorporate linguistic, social, or physical context aspects, let alone combine them in the same study. Indeed, most experimental psycholinguistic research has taken place in isolated and constrained lab settings with carefully selected words or sentences, rather than under rich and naturalistic conditions. We argue that the most influential experimental paradigms in the psycholinguistic study of bilingual language production fall short of capturing the effects of context on language processing and control presupposed by prominent models. This paper therefore aims to enrich the methodological basis for investigating context aspects in current experimental paradigms and thereby move the field of bilingual language production research forward theoretically. After considering extensions of existing paradigms proposed to address context effects, we present three far-ranging innovative proposals, focusing on virtual reality, dialog situations, and multimodality in the context of bilingual language production.

Semantic processing of verbal narratives compared to semantic processing of visual narratives: an ERP study of school-aged children

There is a misconception that pictures are easy to comprehend, which is problematic in pedagogical practices that include pictures. For example, if a child has difficulties with verbal narration to picture sequences, it may be interpreted as specific to spoken language even though the child may have additional difficulties with comprehension of visual narratives in the form of picture sequences. The purpose of the present study was therefore to increase our understanding of semantic processing in the pictorial domain in relation to semantic processing in the verbal domain, focusing on 9-13 years-old children with typical language development. To this end, we measured electrical brain responses (event related potentials, ERPs) in 17 children to (i) pictures (panels) that were predicted versus unpredicted in sequences of panels that conveyed visual narratives and (ii) words that were predicted versus unpredicted in sentences that conveyed verbal narratives. Results demonstrated similarities as there were no significant difference in the magnitude of the N400 effect across domains. The only difference between domains was the predicted difference in distribution, that is, a more posterior N400 effect in the verbal domain than in the pictorial domain. The study contributes to an increased understanding of the complexity of processing of visual narratives and its shared features with processing of verbal narratives, which should be considered in pedagogical practices.

Motivation and genre as social action: a phenomenological perspective on academic writing

This article discusses the relationship between motivation and genre in the context of academic writing, aiming to further bridge the gap between information-processing (IP) cognitive approaches and socio-cultural or dialogical approaches to understanding cognition. The author takes one significant recent article bridging the gap, Graham's Writers Within Communities (WWC) model, as a starting point and attempts to add concepts from genre as social action and Deci and Ryan's Self-Determination Theory (SDT). The article explores how genre as social action is intimately connected with motivation and how SDT's principles of competence, autonomy, and relatedness align with the phenomenological perspective on genre and motivation. The author suggests that these theories provide a more comprehensive understanding of writing motivation, emphasizing that the perception of genre as social action is a crucial motivator for writers and that self-determination is vital to authentic self-regulation in academic writing. The article illustrates the uses of the additional theories with an interview-based case study of a dissertation writer. It ends by discussing the possible implications of this theoretical research for empirical research on student motivation from both IP cognitive and sociocultural perspectives.

Combining EEG and 3D-eye-tracking to study the prediction of upcoming speech in naturalistic virtual environments: A proof of principle

EEG and eye-tracking provide complementary information when investigating language comprehension. Evidence that speech processing may be facilitated by speech prediction comes from the observation that a listener's eye gaze moves towards a referent before it is mentioned if the remainder of the spoken sentence is predictable. However, changes to the trajectory of anticipatory fixations could result from a change in prediction or an attention shift. Conversely, N400 amplitudes and concurrent spectral power provide information about the ease of word processing the moment the word is perceived. In a proof-of-principle investigation, we combined EEG and eye-tracking to study linguistic prediction in naturalistic, virtual environments. We observed increased processing, reflected in theta band power, either during verb processing - when the verb was predictive of the noun - or during noun processing - when the verb was not predictive of the noun. Alpha power was higher in response to the predictive verb and unpredictable nouns. We replicated typical effects of noun congruence but not predictability on the N400 in response to the noun. Thus, the rich visual context that accompanied speech in virtual reality influenced language processing compared to previous reports, where the visual context may have facilitated processing of unpredictable nouns. Finally, anticipatory fixations were predictive of spectral power during noun processing and the length of time fixating the target could be predicted by spectral power at verb onset, conditional on the object having been fixated. Overall, we show that combining EEG and eye-tracking provides a promising new method to answer novel research questions about the prediction of upcoming linguistic input, for example, regarding the role of extralinguistic cues in prediction during language comprehension.

Ecological Meanings: A Consensus Paper on Individual Differences and Contextual Influences in Embodied Language

Embodied theories of cognition consider many aspects of language and other cognitive domains as the result of sensory and motor processes. In this view, the appraisal and the use of concepts are based on mechanisms of simulation grounded on prior sensorimotor experiences. Even though these theories continue receiving attention and support, increasing evidence indicates the need to consider the flexible nature of the simulation process, and to accordingly refine embodied accounts. In this consensus paper, we discuss two potential sources of variability in experimental studies on embodiment of language: individual differences and context. Specifically, we show how factors contributing to individual differences may explain inconsistent findings in embodied language phenomena. These factors include sensorimotor or cultural experiences, imagery, context-related factors, and cognitive strategies. We also analyze the different contextual modulations, from single words to sentences and narratives, as well as the top-down and bottom-up influences. Similarly, we review recent efforts to include cultural and language diversity, aging, neurodegenerative diseases, and brain disorders, as well as bilingual evidence into the embodiment framework. We address the importance of considering individual differences and context in clinical studies to drive translational research more efficiently, and we indicate recommendations on how to correctly address these issues in future research. Systematically investigating individual differences and context may contribute to understanding the dynamic nature of simulation in language processes, refining embodied theories of cognition, and ultimately filling the gap between cognition in artificial experimental settings and cognition in the wild (i.e., in everyday life).

Impaired social concept processing in persons with autistic-like traits

Situated models suggest that social concepts are grounded in interpersonal experience. However, few studies have tested this notion experimentally, and none has targeted individuals with reduced social interaction. Here, we assessed comprehension of text-level social and non-social concepts in persons with and without autistic-like traits. Participants read a social and a non-social text and answered questionnaires targeting social and non-social concepts, respectively. We compared behavioral outcomes, gauged their contribution to subject-level classification, and examined their association with validated measures of autism. Persons with autistic-like traits showed selective deficits in grasping text-level social concepts, even adjusting for intelligence, memory, and vocabulary. Also, social concept comprehension was the only variable that significantly classified between groups. Finally, social concept outcomes correlated negatively with measures of autism, including social interaction. Our results suggest that reduced interpersonal experience selectively compromises text-level social concept processing, offering empirical constraints for situated models of social semantics.

Neural signatures of second language proficiency in narrative processing

Making sense of speech in a second language relies on multiple abilities. Differences in brain activity related to proficiency in language tasks have often been attributed to processing demands. However, during naturalistic narrative comprehension, listeners at different proficiency levels may form different representations of the same speech. We hypothesized that the intersubject synchronization of these representations could be used to measure second-language proficiency. Using a searchlight-shared response model, we found highly proficient participants showed synchronization in regions similar to those of native speakers, including in the default mode network and the lateral prefrontal cortex. In contrast, participants with low proficiency showed more synchronization in auditory cortex and word-level semantic processing areas in the temporal lobe. Moderate proficiency showed the greatest neural diversity, suggesting lower consistency in the source of this partial proficiency. Based on these synchronization differences, we were able to classify the proficiency level or predict behavioral performance on an independent English test in held-out participants, suggesting the identified neural systems represented proficiency-sensitive information that was generalizable to other individuals. These findings suggest higher second-language proficiency leads to more native-like neural processing of naturalistic language, including in systems beyond the cognitive control network or the core language network.

Face2face: advancing the science of social interaction

Face-to-face interaction is core to human sociality and its evolution, and provides the environment in which most of human communication occurs. Research into the full complexities that define face-to-face interaction requires a multi-disciplinary, multi-level approach, illuminating from different perspectives how we and other species interact. This special issue showcases a wide range of approaches, bringing together detailed studies of naturalistic social-interactional behaviour with larger scale analyses for generalization, and investigations of socially contextualized cognitive and neural processes that underpin the behaviour we observe. We suggest that this integrative approach will allow us to propel forwards the science of face-to-face interaction by leading us to new paradigms and novel, more ecologically grounded and comprehensive insights into how we interact with one another and with artificial agents, how differences in psychological profiles might affect interaction, and how the capacity to socially interact develops and has evolved in the human and other species. This theme issue makes a first step into this direction, with the aim to break down disciplinary boundaries and emphasizing the value of illuminating the many facets of face-to-face interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Situation updating during discourse comprehension recruits right posterior portion of the multiple-demand network

Discourse comprehension involves the construction of a mental representation of the situation model as well as a continuous update of this representation. This mental update is cognitively demanding and likely engages the multiple-demand network. However, there is little evidence for the involvement of the multiple-demand network during situation updating. In this study, we used fMRI to test whether situation updating based on the change of spatial location activated the multiple-demand network. In a discourse comprehension task, readers read two-sentence discourses in which the second sentence either continues or introduces a shift of the spatial location information presented in the first sentence. Compared to situation continuation, situation updating reliably activated the right superior parietal lobule. This area is a part of the multiple-demand network as defined by a digit N-back localizer task and locates within the dorsal attention network as defined in the previous study by Yeo et al. in 2011. Our results provide evidence for the reliable involvement of a specific area of the multiple-demand network in situation updating during high-level discourse processing.

ROSE: A Neurocomputational Architecture for Syntax

A comprehensive model of natural language processing in the brain must accommodate four components: representations, operations, structures and encoding. It further requires a principled account of how these different components mechanistically, and causally, relate to each another. While previous models have isolated regions of interest for structure-building and lexical access, and have utilized specific neural recording measures to expose possible signatures of syntax, many gaps remain with respect to bridging distinct scales of analysis that map onto these four components. By expanding existing accounts of how neural oscillations can index various linguistic processes, this article proposes a neurocomputational architecture for syntax, termed the ROSE model (Representation, Operation, Structure, Encoding). Under ROSE, the basic data structures of syntax are atomic features, types of mental representations (R), and are coded at the single-unit and ensemble level. Elementary computations (O) that transform these units into manipulable objects accessible to subsequent structure-building levels are coded via high frequency broadband γ activity. Low frequency synchronization and cross-frequency coupling code for recursive categorial inferences (S). Distinct forms of low frequency coupling and phase-amplitude coupling (δ-θ coupling via pSTS-IFG; θ-γ coupling via IFG to conceptual hubs in lateral and ventral temporal cortex) then encode these structures onto distinct workspaces (E). Causally connecting R to O is spike-phase/LFP coupling; connecting O to S is phase-amplitude coupling; connecting S to E is a system of frontotemporal traveling oscillations; connecting E back to lower levels is low-frequency phase resetting of spike-LFP coupling. This compositional neural code has important implications for algorithmic accounts, since it makes concrete predictions for the appropriate level of study for psycholinguistic parsing models. ROSE is reliant on neurophysiologically plausible mechanisms, is supported at all four levels by a range of recent empirical research, and provides an anatomically precise and falsifiable grounding for the basic property of natural language syntax: hierarchical, recursive structure-building.

Different kinds of simulation during literary reading: Insights from a combined fMRI and eye-tracking study

Mental simulation is an important aspect of narrative reading. In a previous study, we found that gaze durations are differentially impacted by different kinds of mental simulation. Motor simulation, perceptual simulation, and mentalizing as elicited by literary short stories influenced eye movements in distinguishable ways (Mak & Willems, 2019). In the current study, we investigated the existence of a common neural locus for these different kinds of simulation. We additionally investigated whether individual differences during reading, as indexed by the eye movements, are reflected in domain-specific activations in the brain. We found a variety of brain areas activated by simulation-eliciting content, both modality-specific brain areas and a general simulation area. Individual variation in percent signal change in activated areas was related to measures of story appreciation as well as personal characteristics (i.e., transportability, perspective taking). Taken together, these findings suggest that mental simulation is supported by both domain-specific processes grounded in previous experiences, and by the neural mechanisms that underlie higher-order language processing (e.g., situation model building, event indexing, integration).

Social concepts and the cerebellum: behavioural and functional connectivity signatures in cerebellar ataxic patients

Neurocognitive research on social concepts underscores their reliance on fronto-temporo-limbic regions mediating broad socio-cognitive skills. Yet, the field has neglected another structure increasingly implicated in social cognition: the cerebellum. The present exploratory study examines this link combining a novel naturalistic text paradigm, a relevant atrophy model and functional magnetic resonance imaging. Fifteen cerebellar ataxia (CA) patients with focal cerebellar atrophy and 29 matched controls listened to a social text (highlighting interpersonal events) as well as a non-social text (focused on a single person's actions), and answered comprehension questionnaires. We compared behavioural outcomes between groups and examined their association with cerebellar connectivity. CA patients showed deficits in social text comprehension and normal scores in the non-social text. Also, social text outcomes in controls selectively correlated with connectivity between the cerebellum and key regions subserving multi-modal semantics and social cognition, including the superior and medial temporal gyri, the temporal pole and the insula. Conversely, brain-behaviour associations involving the cerebellum were abolished in the patients. Thus, cerebellar structures and connections seem involved in processing social concepts evoked by naturalistic discourse. Such findings invite new theoretical and translational developments integrating social neuroscience with embodied semantics. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.

Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience

In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective ("lateral processing pathway"). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.

Semantic Adaptation to the Interpretation of Gradable Adjectives via Active Linguistic Interaction

When communicating, people adapt their linguistic representations to those of their interlocutors. Previous studies have shown that this also occurs at the semantic level for vague and context-dependent terms such as quantifiers and uncertainty expressions. However, work to date has mostly focused on passive exposure to a given speaker's interpretation, without considering the possible role of active linguistic interaction. In this study, we focus on gradable adjectives big and small and develop a novel experimental paradigm that allows participants to ask clarification questions to figure out their interlocutor's interpretation. We find that, when in doubt, speakers do resort to this strategy, despite its inherent cognitive cost, and that doing so results in higher semantic alignment measured in terms of communicative success. While not all question-answer pairs are equally informative, we show that speakers become better questioners as the interaction progresses. Yet, the higher semantic alignment observed when speakers are able to ask questions does not increase over time. This suggests that conversational interaction's key advantage may be to boost coordination without committing to long-term semantic updates. Our findings shed new light on the mechanisms used by speakers to achieve semantic alignment and on how language is shaped by communication.

Body into Narrative: Behavioral and Neurophysiological Signatures of Action Text Processing After Ecological Motor Training

Embodied cognition research indicates that sensorimotor training can influence action concept processing. Yet, most studies employ isolated (pseudo)randomized stimuli and require repetitive single-effector responses, thus lacking ecological validity. Moreover, the neural signatures of these effects remain poorly understood. Here, we examined whether immersive bodily training can modulate behavioral and functional connectivity correlates of action-verb processing in naturalistic narratives. The study involved three phases. First, in the Pre-training phase, 32 healthy persons listened to an action text (rich in movement descriptions) and a non-action text (focused on its characters' perceptual and mental processes), completed comprehension questionnaires, and underwent resting-state electroencephalogram (EEG) recordings. Second, in the four-day Training phase, half the participants completed an exergaming intervention (eliciting full-body movements for 60 min a day) while the remaining half played static videogames (requiring no bodily engagement other than button presses). Finally, in the Post-training phase, all participants repeated the Pre-training protocol with different action and non-action texts and a new EEG session. We found that exergaming selectively reduced action-verb outcomes and fronto-posterior functional connectivity in the motor-sensitive ∼ 10-20 Hz range, both patterns being positively correlated. Conversely, static videogame playing yielded no specific effect on any linguistic category and did not modulate functional connectivity. Together, these findings suggest that action-verb processing and key neural correlates can be focally influenced by full-body motor training in a highly ecological setting. Our study illuminates the role of situated experience and sensorimotor circuits in action-concept processing, addressing calls for naturalistic insights on language embodiment.

Robust Effects of Working Memory Demand during Naturalistic Language Comprehension in Language-Selective Cortex

To understand language, we must infer structured meanings from real-time auditory or visual signals. Researchers have long focused on word-by-word structure building in working memory as a mechanism that might enable this feat. However, some have argued that language processing does not typically involve rich word-by-word structure building, and/or that apparent working memory effects are underlyingly driven by surprisal (how predictable a word is in context). Consistent with this alternative, some recent behavioral studies of naturalistic language processing that control for surprisal have not shown clear working memory effects. In this fMRI study, we investigate a range of theory-driven predictors of word-by-word working memory demand during naturalistic language comprehension in humans of both sexes under rigorous surprisal controls. In addition, we address a related debate about whether the working memory mechanisms involved in language comprehension are language specialized or domain general. To do so, in each participant, we functionally localize (1) the language-selective network and (2) the "multiple-demand" network, which supports working memory across domains. Results show robust surprisal-independent effects of memory demand in the language network and no effect of memory demand in the multiple-demand network. Our findings thus support the view that language comprehension involves computationally demanding word-by-word structure building operations in working memory, in addition to any prediction-related mechanisms. Further, these memory operations appear to be primarily conducted by the same neural resources that store linguistic knowledge, with no evidence of involvement of brain regions known to support working memory across domains.SIGNIFICANCE STATEMENT This study uses fMRI to investigate signatures of working memory (WM) demand during naturalistic story listening, using a broad range of theoretically motivated estimates of WM demand. Results support a strong effect of WM demand in the brain that is distinct from effects of word predictability. Further, these WM demands register primarily in language-selective regions, rather than in "multiple-demand" regions that have previously been associated with WM in nonlinguistic domains. Our findings support a core role for WM in incremental language processing, using WM resources that are specialized for language.

Neural correlates of multisensory enhancement in audiovisual narrative speech perception: a fMRI investigation

This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.

Multimodal Neurocognitive Markers of Naturalistic Discourse Typify Diverse Neurodegenerative Diseases

Neurodegeneration has multiscalar impacts, including behavioral, neuroanatomical, and neurofunctional disruptions. Can disease-differential alterations be captured across such dimensions using naturalistic stimuli? To address this question, we assessed comprehension of four naturalistic stories, highlighting action, nonaction, social, and nonsocial events, in Parkinson's disease (PD) and behavioral variant frontotemporal dementia (bvFTD) relative to Alzheimer's disease patients and healthy controls. Text-specific correlates were evaluated via voxel-based morphometry, spatial (fMRI), and temporal (hd-EEG) functional connectivity. PD patients presented action-text deficits related to the volume of action-observation regions, connectivity across motor-related and multimodal-semantic hubs, and frontal hd-EEG hypoconnectivity. BvFTD patients exhibited social-text deficits, associated with atrophy and spatial connectivity patterns along social-network hubs, alongside right frontotemporal hd-EEG hypoconnectivity. Alzheimer's disease patients showed impairments in all stories, widespread atrophy and spatial connectivity patterns, and heightened occipitotemporal hd-EEG connectivity. Our framework revealed disease-specific signatures across behavioral, neuroanatomical, and neurofunctional dimensions, highlighting the sensitivity and specificity of a single naturalistic task. This investigation opens a translational agenda combining ecological approaches and multimodal cognitive neuroscience for the study of neurodegeneration.

Differential Tracking of Linguistic vs. Mental State Content in Naturalistic Stimuli by Language and Theory of Mind (ToM) Brain Networks

Language and social cognition, especially the ability to reason about mental states, known as theory of mind (ToM), are deeply related in development and everyday use. However, whether these cognitive faculties rely on distinct, overlapping, or the same mechanisms remains debated. Some evidence suggests that, by adulthood, language and ToM draw on largely distinct-though plausibly interacting-cortical networks. However, the broad topography of these networks is similar, and some have emphasized the importance of social content / communicative intent in the linguistic signal for eliciting responses in the language areas. Here, we combine the power of individual-subject functional localization with the naturalistic-cognition inter-subject correlation approach to illuminate the language-ToM relationship. Using functional magnetic resonance imaging (fMRI), we recorded neural activity as participants (n = 43) listened to stories and dialogues with mental state content (+linguistic, +ToM), viewed silent animations and live action films with mental state content but no language (-linguistic, +ToM), or listened to an expository text (+linguistic, -ToM). The ToM network robustly tracked stimuli rich in mental state information regardless of whether mental states were conveyed linguistically or non-linguistically, while tracking a +linguistic / -ToM stimulus only weakly. In contrast, the language network tracked linguistic stimuli more strongly than (a) non-linguistic stimuli, and than (b) the ToM network, and showed reliable tracking even for the linguistic condition devoid of mental state content. These findings suggest that in spite of their indisputably close links, language and ToM dissociate robustly in their neural substrates-and thus plausibly cognitive mechanisms-including during the processing of rich naturalistic materials.

Shared computational principles for language processing in humans and deep language models

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Deep language models have revolutionized natural language processing. The paper discovers three computational principles shared between deep language models and the human brain, which can transform our understanding of the neural basis of language.

What is Functional Communication? A Theoretical Framework for Real-World Communication Applied to Aphasia Rehabilitation

Aphasia is an impairment of language caused by acquired brain damage such as stroke or traumatic brain injury, that affects a person’s ability to communicate effectively. The aim of rehabilitation in aphasia is to improve everyday communication, improving an individual’s ability to function in their day-to-day life. For that reason, a thorough understanding of naturalistic communication and its underlying mechanisms is imperative. The field of aphasiology currently lacks an agreed, comprehensive, theoretically founded definition of communication. Instead, multiple disparate interpretations of functional communication are used. We argue that this makes it nearly impossible to validly and reliably assess a person’s communicative performance, to target this behaviour through therapy, and to measure improvements post-therapy. In this article we propose a structured, theoretical approach to defining the concept of functional communication. We argue for a view of communication as “situated language use”, borrowed from empirical psycholinguistic studies with non-brain damaged adults. This framework defines language use as: (1) interactive, (2) multimodal, and (3) contextual. Existing research on each component of the framework from non-brain damaged adults and people with aphasia is reviewed. The consequences of adopting this approach to assessment and therapy for aphasia rehabilitation are discussed. The aim of this article is to encourage a more systematic, comprehensive approach to the study and treatment of situated language use in aphasia.

Arousal States as a Key Source of Variability in Speech Perception and Learning

The human brain exhibits the remarkable ability to categorize speech sounds into distinct, meaningful percepts, even in challenging tasks like learning non-native speech categories in adulthood and hearing speech in noisy listening conditions. In these scenarios, there is substantial variability in perception and behavior, both across individual listeners and individual trials. While there has been extensive work characterizing stimulus-related and contextual factors that contribute to variability, recent advances in neuroscience are beginning to shed light on another potential source of variability that has not been explored in speech processing. Specifically, there are task-independent, moment-to-moment variations in neural activity in broadly-distributed cortical and subcortical networks that affect how a stimulus is perceived on a trial-by-trial basis. In this review, we discuss factors that affect speech sound learning and moment-to-moment variability in perception, particularly arousal states—neurotransmitter-dependent modulations of cortical activity. We propose that a more complete model of speech perception and learning should incorporate subcortically-mediated arousal states that alter behavior in ways that are distinct from, yet complementary to, top-down cognitive modulations. Finally, we discuss a novel neuromodulation technique, transcutaneous auricular vagus nerve stimulation (taVNS), which is particularly well-suited to investigating causal relationships between arousal mechanisms and performance in a variety of perceptual tasks. Together, these approaches provide novel testable hypotheses for explaining variability in classically challenging tasks, including non-native speech sound learning.

The genetic architecture of language functional connectivity

Language is a unique trait of the human species, of which the genetic architecture remains largely unknown. Through language disorders studies, many candidate genes were identified. However, such complex and multifactorial trait is unlikely to be driven by only few genes and case-control studies, suffering from a lack of power, struggle to uncover significant variants. In parallel, neuroimaging has significantly contributed to the understanding of structural and functional aspects of language in the human brain and the recent availability of large scale cohorts like UK Biobank have made possible to study language via image-derived endophenotypes in the general population. Because of its strong relationship with task-based fMRI (tbfMRI) activations and its easiness of acquisition, resting-state functional MRI (rsfMRI) have been more popularised, making it a good surrogate of functional neuronal processes. Taking advantage of such a synergistic system by aggregating effects across spatially distributed traits, we performed a multivariate genome-wide association study (mvGWAS) between genetic variations and resting-state functional connectivity (FC) of classical brain language areas in the inferior frontal (pars opercularis, triangularis and orbitalis), temporal and inferior parietal lobes (angular and supramarginal gyri), in 32,186 participants from UK Biobank. Twenty genomic loci were found associated with language FCs, out of which three were replicated in an independent replication sample. A locus in 3p11.1, regulating EPHA3 gene expression, is found associated with FCs of the semantic component of the language network, while a locus in 15q14, regulating THBS1 gene expression is found associated with FCs of the perceptual-motor language processing, bringing novel insights into the neurobiology of language.

Aesthetic appraisals of literary style and emotional intensity in narrative engagement are neurally dissociable

Humans are deeply affected by stories, yet it is unclear how. In this study, we explored two aspects of aesthetic experiences during narrative engagement - literariness and narrative fluctuations in appraised emotional intensity. Independent ratings of literariness and emotional intensity of two literary stories were used to predict blood-oxygen-level-dependent signal changes in 52 listeners from an existing fMRI dataset. Literariness was associated with increased activation in brain areas linked to semantic integration (left angular gyrus, supramarginal gyrus, and precuneus), and decreased activation in bilateral middle temporal cortices, associated with semantic representations and word memory. Emotional intensity correlated with decreased activation in a bilateral frontoparietal network that is often associated with controlled attention. Our results confirm a neural dissociation in processing literary form and emotional content in stories and generate new questions about the function of and interaction between attention, social cognition, and semantic systems during literary engagement and aesthetic experiences.

Similarity of computations across domains does not imply shared implementation: The case of language comprehension

Understanding language requires applying cognitive operations (e.g., memory retrieval, prediction, structure building)-relevant across many cognitive domains-to specialized knowledge structures (a particular language's phonology, lexicon, and syntax). Are these computations carried out by domain-general circuits or by circuits that store domain-specific representations? Recent work has characterized the roles in language comprehension of the language-selective network and the multiple demand (MD) network, which has been implicated in executive functions and linked to fluid intelligence, making it a prime candidate for implementing computations that support information processing across domains. The language network responds robustly to diverse aspects of comprehension, but the MD network shows no sensitivity to linguistic variables. We therefore argue that the MD network does not play a core role in language comprehension, and that past claims to the contrary are likely due to methodological artifacts. Although future studies may discover some aspects of language that require the MD network, evidence to date suggests that those will not be related to core linguistic processes like lexical access or composition. The finding that the circuits that store linguistic knowledge carry out computations on those representations aligns with general arguments against the separation between memory and computation in the mind and brain.

Inhibitory Mechanisms in the Processing of Negations: A Neural Reuse Hypothesis

Negation is known to have inhibitory consequences for the information under its scope. However, how it produces such effects remains poorly understood. Recently, it has been proposed that negation processing might be implemented at the neural level by the recruitment of inhibitory and cognitive control mechanisms. On this line, this manuscript offers the hypothesis that negation reuses general-domain mechanisms that subserve inhibition in other non-linguistic cognitive functions. The first two sections describe the inhibitory effects of negation on conceptual representations and its embodied effects, as well as the theoretical foundations for the reuse hypothesis. The next section describes the neurophysiological evidence that linguistic negation interacts with response inhibition, along with the suggestion that both functions share inhibitory mechanisms. Finally, the manuscript concludes that the functional relation between negation and inhibition observed at the mechanistic level could be easily integrated with predominant cognitive models of negation processing.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment

Comprehending speech in noise is an essential cognitive skill for verbal communication. However, it remains unclear how our brain adapts to the noisy environment to achieve comprehension. The present study investigated the neural mechanisms of speech comprehension in noise using an functional near-infrared spectroscopy-based inter-brain approach. A group of speakers was invited to tell real-life stories. The recorded speech audios were added with meaningless white noise at four signal-to-noise levels and then played to listeners. Results showed that speaker-listener neural couplings of listener's left inferior frontal gyri (IFG), that is, sensorimotor system, and right middle temporal gyri (MTG), angular gyri (AG), that is, auditory system, were significantly higher in listening conditions than in the baseline. More importantly, the correlation between neural coupling of listener's left IFG and the comprehension performance gradually became more positive with increasing noise level, indicating an adaptive role of sensorimotor system in noisy speech comprehension; however, the top behavioral correlations for the coupling of listener's right MTG and AG were only obtained in mild noise conditions, indicating a different and less robust mechanism. To sum up, speaker-listener coupling analysis provides added value and new sight to understand the neural mechanism of speech-in-noise comprehension.

Situating Language in the Real-World: The Role of Multimodal Iconicity and Indexicality

In the last decade, a growing body of work has convincingly demonstrated that languages embed a certain degree of non-arbitrariness (mostly in the form of iconicity, namely the presence of imagistic links between linguistic form and meaning). Most of this previous work has been limited to assessing the degree (and role) of non-arbitrariness in the speech (for spoken languages) or manual components of signs (for sign languages). When approached in this way, non-arbitrariness is acknowledged but still considered to have little presence and purpose, showing a diachronic movement towards more arbitrary forms. However, this perspective is limited as it does not take into account the situated nature of language use in face-to-face interactions, where language comprises categorical components of speech and signs, but also multimodal cues such as prosody, gestures, eye gaze etc. We review work concerning the role of context-dependent iconic and indexical cues in language acquisition and processing to demonstrate the pervasiveness of non-arbitrary multimodal cues in language use and we discuss their function. We then move to argue that the online omnipresence of multimodal non-arbitrary cues supports children and adults in dynamically developing situational models.

Incremental Language Comprehension Difficulty Predicts Activity in the Language Network but Not the Multiple Demand Network

What role do domain-general executive functions play in human language comprehension? To address this question, we examine the relationship between behavioral measures of comprehension and neural activity in the domain-general "multiple demand" (MD) network, which has been linked to constructs like attention, working memory, inhibitory control, and selection, and implicated in diverse goal-directed behaviors. Specifically, functional magnetic resonance imaging data collected during naturalistic story listening are compared with theory-neutral measures of online comprehension difficulty and incremental processing load (reading times and eye-fixation durations). Critically, to ensure that variance in these measures is driven by features of the linguistic stimulus rather than reflecting participant- or trial-level variability, the neuroimaging and behavioral datasets were collected in nonoverlapping samples. We find no behavioral-neural link in functionally localized MD regions; instead, this link is found in the domain-specific, fronto-temporal "core language network," in both left-hemispheric areas and their right hemispheric homotopic areas. These results argue against strong involvement of domain-general executive circuits in language comprehension.

Incremental Language Comprehension Difficulty Predicts Activity in the Language Network but Not the Multiple Demand Network

What role do domain-general executive functions play in human language comprehension? To address this question, we examine the relationship between behavioral measures of comprehension and neural activity in the domain-general "multiple demand" (MD) network, which has been linked to constructs like attention, working memory, inhibitory control, and selection, and implicated in diverse goal-directed behaviors. Specifically, functional magnetic resonance imaging data collected during naturalistic story listening are compared with theory-neutral measures of online comprehension difficulty and incremental processing load (reading times and eye-fixation durations). Critically, to ensure that variance in these measures is driven by features of the linguistic stimulus rather than reflecting participant- or trial-level variability, the neuroimaging and behavioral datasets were collected in nonoverlapping samples. We find no behavioral-neural link in functionally localized MD regions; instead, this link is found in the domain-specific, fronto-temporal "core language network," in both left-hemispheric areas and their right hemispheric homotopic areas. These results argue against strong involvement of domain-general executive circuits in language comprehension.

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

Continuous-time deconvolutional regression for psycholinguistic modeling

The influence of stimuli in psycholinguistic experiments diffuses across time because the human response to language is not instantaneous. The linear models typically used to analyze psycholinguistic data are unable to account for this phenomenon due to strong temporal independence assumptions, while existing deconvolutional methods for estimating diffuse temporal structure model time discretely and therefore cannot be directly applied to natural language stimuli where events (words) have variable duration. In light of evidence that continuous-time deconvolutional regression (CDR) can address these issues (Shain & Schuler, 2018), this article motivates the use of CDR for many experimental settings, exposits some of its mathematical properties, and empirically evaluates the influence of various experimental confounds (noise, multicollinearity, and impulse response misspecification), hyperparameter settings, and response types (behavioral and fMRI). Results show that CDR (1) yields highly consistent estimates across a variety of hyperparameter configurations, (2) faithfully recovers the data-generating model on synthetic data, even under adverse training conditions, and (3) outperforms widely-used statistical approaches when applied to naturalistic reading and fMRI data. In addition, procedures for testing scientific hypotheses using CDR are defined and demonstrated, and empirically-motivated best-practices for CDR modeling are proposed. Results support the use of CDR for analyzing psycholinguistic time series, especially in a naturalistic experimental paradigm.

Influence of homophone processing during auditory language comprehension on executive control processes: A dual-task paradigm

In the present preregistered study, we evaluated the possibility of a shared cognitive mechanism during verbal and non-verbal tasks and therefore the implication of domain-general cognitive control during language comprehension. We hypothesized that a behavioral cost will be observed during a dual-task including both verbal and non-verbal difficult processing. Specifically, to test this claim, we designed a dual-task paradigm involving: an auditory language comprehension task (sentence comprehension) and a non-verbal Flanker task (including congruent and incongruent trials). We manipulated sentence ambiguity and evaluated if the ambiguity effect modified behavioral performances in the non-verbal Flanker task. Under the assumption that ambiguous sentences induce a more difficult process than unambiguous sentences, we expected non-verbal flanker task performances to be impaired only when a simultaneous difficult language processing is performed. This would be specifically reflected by a performance cost during incongruent Flanker items only during ambiguous sentence presentation. Conversely, we observed a facilitatory effect for the incongruent Flanker items during ambiguous sentence suggesting better non-verbal inhibitory performances when an ambiguous sentence was simultaneously processed. Exploratory data analysis suggests that this effect is not only related to a more difficult language processing but also to the previous (n-1) Flanker item. Indeed, results showed that incongruent n-1 Flanker items led to a facilitation of the incongruent synchronized Flanker items only when ambiguous sentences were conjointly presented. This result, even if it needs to be corroborated in future studies, suggests that the recruitment of executive control mechanisms facilitates subsequent executive control implication during difficult language processing. The present study suggests a common executive control mechanism during difficult verbal and non-verbal tasks.

Core Cognitive Mechanisms Underlying Syntactic Priming: A Comparison of Three Alternative Models

Syntactic priming (SP) is the effect by which, in a dialogue, the current speaker tends to re-use the syntactic constructs of the previous speakers. SP has been used as a window into the nature of syntactic representations within and across languages. Because of its importance, it is crucial to understand the mechanisms behind it. Currently, two competing theories exist. According to the transient activation account, SP is driven by the re-activation of declarative memory structures that encode structures. According to the error-based implicit learning account, SP is driven by prediction errors while processing sentences. By integrating both transient activation and associative learning, Reitter et al.'s hybrid model 2011 assumes that SP is achieved by both mechanisms, and predicts a priming enhancement for rare or unusual constructions. Finally, a recently proposed account, the reinforcement learning account, claims that SP driven by the successful application of procedural knowledge will be reversed when the prime sentence includes grammatical errors. These theories make different assumptions about the representation of syntactic rules (declarative vs. procedural) and the nature of the mechanism that drives priming (frequency and repetition, attention, and feedback signals, respectively). To distinguish between these theories, they were all implemented as computational models in the ACT-R cognitive architecture, and their specific predictions were examined through grid-search computer simulations. Two experiments were then carried out to empirically test the central prediction of each theory as well as the individual fits of each participant's responses to different parameterizations of each model. The first experiment produced results that were best explained by the associative account, but could also be accounted for by a modified reinforcement model with a different parsing algorithm. The second experiment, whose stimuli were designed to avoid the parsing ambiguity of the first, produced somewhat weaker effects. Its results, however, were also best predicted by the model implementing the associative account. We conclude that the data overall points to SP being due to prediction violations that direct attentional resources, in turn suggesting a declarative rather than a RL based procedural representation of syntactic rules.

How context changes the neural basis of perception and language

Cognitive processes-from basic sensory analysis to language understanding-are typically contextualized. While the importance of considering context for understanding cognition has long been recognized in psychology and philosophy, it has not yet had much impact on cognitive neuroscience research, where cognition is often studied in decontextualized paradigms. Here, we present examples of recent studies showing that context changes the neural basis of diverse cognitive processes, including perception, attention, memory, and language. Within the domains of perception and language, we review neuroimaging results showing that context interacts with stimulus processing, changes activity in classical perception and language regions, and recruits additional brain regions that contribute crucially to naturalistic perception and language. We discuss how contextualized cognitive neuroscience will allow for discovering new principles of the mind and brain.

Syllable Inference as a Mechanism for Spoken Language Understanding

A classic problem in spoken language comprehension is how listeners perceive speech as being composed of discrete words, given the variable time-course of information in continuous signals. We propose a syllable inference account of spoken word recognition and segmentation, according to which alternative hierarchical models of syllables, words, and phonemes are dynamically posited, which are expected to maximally predict incoming sensory input. Generative models are combined with current estimates of context speech rate drawn from neural oscillatory dynamics, which are sensitive to amplitude rises. Over time, models which result in local minima in error between predicted and recently experienced signals give rise to perceptions of hearing words. Three experiments using the visual world eye-tracking paradigm with a picture-selection task tested hypotheses motivated by this framework. Materials were sentences that were acoustically ambiguous in numbers of syllables, words, and phonemes they contained (cf. English plural constructions, such as "saw (a) raccoon(s) swimming," which have two loci of grammatical information). Time-compressing, or expanding, speech materials permitted determination of how temporal information at, or in the context of, each locus affected looks to, and selection of, pictures with a singular or plural referent (e.g., one or more than one raccoon). Supporting our account, listeners probabilistically interpreted identical chunks of speech as consistent with a singular or plural referent to a degree that was based on the chunk's gradient rate in relation to its context. We interpret these results as evidence that arriving temporal information, judged in relation to language model predictions generated from context speech rate evaluated on a continuous scale, informs inferences about syllables, thereby giving rise to perceptual experiences of understanding spoken language as words separated in time.

Language Dysfunction in Schizophrenia: Assessing Neural Tracking to Characterize the Underlying Disorder(s)?

Deficits in language production and comprehension are characteristic of schizophrenia. To date, it remains unclear whether these deficits arise from dysfunctional linguistic knowledge, or dysfunctional predictions derived from the linguistic context. Alternatively, the deficits could be a result of dysfunctional neural tracking of auditory information resulting in decreased auditory information fidelity and even distorted information. Here, we discuss possible ways for clinical neuroscientists to employ neural tracking methodology to independently characterize deficiencies on the auditory-sensory and abstract linguistic levels. This might lead to a mechanistic understanding of the deficits underlying language related disorder(s) in schizophrenia. We propose to combine naturalistic stimulation, measures of speech-brain synchronization, and computational modeling of abstract linguistic knowledge and predictions. These independent but likely interacting assessments may be exploited for an objective and differential diagnosis of schizophrenia, as well as a better understanding of the disorder on the functional level-illustrating the potential of neural tracking methodology as translational tool in a range of psychotic populations.

More Than Words: Extra-Sylvian Neuroanatomic Networks Support Indirect Speech Act Comprehension and Discourse in Behavioral Variant Frontotemporal Dementia

Indirect speech acts—responding “I forgot to wear my watch today” to someone who asked for the time—are ubiquitous in daily conversation, but are understudied in current neurobiological models of language. To comprehend an indirect speech act like this one, listeners must not only decode the lexical-semantic content of the utterance, but also make a pragmatic, bridging inference. This inference allows listeners to derive the speaker’s true, intended meaning—in the above dialog, for example, that the speaker cannot provide the time. In the present work, we address this major gap by asking non-aphasic patients with behavioral variant frontotemporal dementia (bvFTD, n = 21) and brain-damaged controls with amnestic mild cognitive impairment (MCI, n = 17) to judge simple question-answer dialogs of the form: “Do you want some cake for dessert?” “I’m on a very strict diet right now,” and relate the results to structural and diffusion MRI. Accuracy and reaction time results demonstrate that subjects with bvFTD, but not MCI, are selectively impaired in indirect relative to direct speech act comprehension, due in part to their social and executive limitations, and performance is related to caregivers’ judgment of communication efficacy. MRI imaging associates the observed impairment in bvFTD to cortical thinning not only in traditional language-associated regions, but also in fronto-parietal regions implicated in social and executive cerebral networks. Finally, diffusion tensor imaging analyses implicate white matter tracts in both dorsal and ventral projection streams, including superior longitudinal fasciculus, frontal aslant, and uncinate fasciculus. These results have strong implications for updated neurobiological models of language, and emphasize a core, language-mediated social disorder in patients with bvFTD.

Age-related effects on the neural processing of semantic complexity in a continuous narrative: Modulation by gestures already present in young to middle-aged adults

The processing of semantically complex speech is a demanding task which can be facilitated by speech-associated arm and hand gestures. However, the role of age concerning the perception of semantic complexity and the influence of gestures in this context remains unclear. The goal of this study was to investigate if age-related differences are already present in early adulthood during the processing of semantic complexity and gestures. To this end, we analyzed fMRI images of a sample of 38 young and middle-aged participants (age-range: 19-55). They had the task to listen and to watch a narrative. The narrative contained segments varying in the degree of semantic complexity, and they were spontaneously accompanied by gestures. The semantic complexity of the story was measured by the idea density. Consistent with previous findings in young adults, we observed increased activation for passages with lower compared to higher complexity in bilateral temporal areas and the precuneus. BOLD signal in the left frontal and left parietal regions correlated during the perception of complex passages with increasing age. This correlation was reduced for passages presented with gestures. Median-split based post-hoc comparisons confirmed that group differences between younger (19-23 years) and older adults within the early adult lifespan (24-55 years) were significantly reduced in passages with gestures. Our results suggest that older adults within early adulthood adapt to the requirements of highly complex passages activating additional regions when no gesture information is available. Gestures might play a facilitative role with increasing age, especially when speech is complex.

Common and distinctive cognitive processes between categorization and category-based induction: Evidence from event-related potentials

Categorization involves forming equivalence classes of discriminable entities, whereas category-based induction (CBI) involves employing categorical knowledge to generalize novel properties. Previous studies have suggested either common or distinctive cognitive processing between categorization and CBI. However, no study has compared cognitive processes with the same stimuli sets using event-related potentials (ERPs), which help to determine the cognitive processes with a high temporal solution. In this study, we compared the ERP responses to categorization and CBI using two separate experiments (i.e., generic and specific conclusions), with the same task materials. Results from both experiments identified distinctive cognitive processing between categorization and CBI based on a greater proportion of "definitely" responses and smaller amplitudes of sustained negativity during categorization. These observations suggest that categorization involves decreased conflict monitoring and control than CBI under single-premise conditions. Contrastingly, categorization and CBI elicited similar FN400 amplitudes in both experiments, which suggests a common cognitive process between them. These findings present the common and distinctive cognitive processes between categorization and CBI.