Grounding the neurobiology of language in first principles: The necessity of non-language-centric explanations for language comprehension

Common and distinctive cognitive processes between categorization and category-based induction: Evidence from event-related potentials

Categorization involves forming equivalence classes of discriminable entities, whereas category-based induction (CBI) involves employing categorical knowledge to generalize novel properties. Previous studies have suggested either common or distinctive cognitive processing between categorization and CBI. However, no study has compared cognitive processes with the same stimuli sets using event-related potentials (ERPs), which help to determine the cognitive processes with a high temporal solution. In this study, we compared the ERP responses to categorization and CBI using two separate experiments (i.e., generic and specific conclusions), with the same task materials. Results from both experiments identified distinctive cognitive processing between categorization and CBI based on a greater proportion of "definitely" responses and smaller amplitudes of sustained negativity during categorization. These observations suggest that categorization involves decreased conflict monitoring and control than CBI under single-premise conditions. Contrastingly, categorization and CBI elicited similar FN400 amplitudes in both experiments, which suggests a common cognitive process between them. These findings present the common and distinctive cognitive processes between categorization and CBI.

Electrifying discourse: Anodal tDCS of the primary motor cortex selectively reduces action appraisal in naturalistic narratives

Non-invasive stimulation of the primary motor cortex (M1) modulates processing of decontextualized action words and sentences (i.e., verbal units denoting bodily motion). This suggests that language comprehension hinges on brain circuits mediating the bodily experiences evoked by verbal material. Yet, despite its relevance to constrain mechanistic language models, such a finding fails to reveal whether and how relevant circuits operate in the face of full-blown, everyday texts. Using a novel naturalistic discourse paradigm, we examined whether direct modulation of M1 excitability influences the grasping of narrated actions. Following random group assignment, participants received anodal transcranial direct current stimulation over the left M1, or sham stimulation of the same area, or anodal stimulation of the left ventrolateral prefrontal cortex. Immediately afterwards, they listened to action-laden and neutral stories and answered questions on information realized by verbs (denoting action and non-action processes) and circumstances (conveying locative or temporal details). Anodal stimulation of the left M1 selectively decreased outcomes on action-relative to non-action information -a pattern that discriminated between stimulated and sham participants with 74% accuracy. This result was particular to M1 and held irrespective of the subjects' working memory and vocabulary skills, further attesting to its specificity. Our findings suggest that offline modulation of motor-network excitability might lead to transient unavailability of putative resources needed to evoke actions in naturalistic texts, opening promising avenues for the language embodiment framework.

Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models

Language is crucial for human intelligence, but what exactly is its role? We take language to be a part of a system for understanding and communicating about situations. In humans, these abilities emerge gradually from experience and depend on domain-general principles of biological neural networks: connection-based learning, distributed representation, and context-sensitive, mutual constraint satisfaction-based processing. Current artificial language processing systems rely on the same domain general principles, embodied in artificial neural networks. Indeed, recent progress in this field depends on query-based attention, which extends the ability of these systems to exploit context and has contributed to remarkable breakthroughs. Nevertheless, most current models focus exclusively on language-internal tasks, limiting their ability to perform tasks that depend on understanding situations. These systems also lack memory for the contents of prior situations outside of a fixed contextual span. We describe the organization of the brain's distributed understanding system, which includes a fast learning system that addresses the memory problem. We sketch a framework for future models of understanding drawing equally on cognitive neuroscience and artificial intelligence and exploiting query-based attention. We highlight relevant current directions and consider further developments needed to fully capture human-level language understanding in a computational system.

The phonological loop: is speech special?

It has been proposed that the maintenance of phonological information in verbal working memory (vWM) is carried by a domain-specific short-term storage center-the phonological loop-which is composed of a phonological store and an articulatory rehearsal system. Several brain regions including the left posterior inferior frontal gyrus (pIFG) and anterior supramarginal gyri (aSMG) are thought to support these processes. However, recent behavioral evidence suggests that verbal and non-verbal auditory information may be processed as part of a unique domain general short-term storage center instead of through specialized subsystems such as the phonological loop. In the current study, we used a single-pulse transcranial magnetic stimulation (TMS)-delayed priming paradigm with speech (syllables) and acoustically complex non-speech sounds (bird songs) to examine whether the pIFG and aSMG are involved in the processing of verbal information or, alternatively, in the processing of any complex auditory information. Our results demonstrate that TMS delivered to both regions had an effect on performance for speech and non-speech stimuli, but the nature of the effect was different. That is, priming was reduced for the speech sounds because TMS facilitated the detection of different but not identical stimuli, and accuracy was decreased for non-speech sounds. Since TMS interfered with both speech and non-speech sounds, these findings support the existence of an auditory short-term storage center located within the dorsal auditory stream.

The Storytelling Brain: How Neuroscience Stories Help Bridge the Gap between Research and Society

Active communication between researchers and society is necessary for the scientific community's involvement in developing science-based policies. This need is recognized by governmental and funding agencies that compel scientists to increase their public engagement and disseminate research findings in an accessible fashion. Storytelling techniques can help convey science by engaging people's imagination and emotions. Yet, many researchers are uncertain about how to approach scientific storytelling, or feel they lack the tools to undertake it. Here we explore some of the techniques intrinsic to crafting scientific narratives, as well as the reasons why scientific storytelling may be an optimal way of communicating research to nonspecialists. We also point out current communication gaps between science and society, particularly in the context of neurodiverse audiences and those that include neurological and psychiatric patients. Present shortcomings may turn into areas of synergy with the potential to link neuroscience education, research, and advocacy.

No title, no theme: The joined neural space between speakers and listeners during production and comprehension of multi-sentence discourse

Speakers and listeners usually interact in larger discourses than single words or even single sentences. The goal of the present study was to identify the neural bases reflecting how the mental representation of the situation denoted in a multi-sentence discourse (situation model) is constructed and shared between speakers and listeners. An fMRI study using a variant of the ambiguous text paradigm was designed. Speakers (n = 15) produced ambiguous texts in the scanner and listeners (n = 27) subsequently listened to these texts in different states of ambiguity: preceded by a highly informative, intermediately informative or no title at all. Conventional BOLD activation analyses in listeners, as well as inter-subject correlation analyses between the speakers' and the listeners' hemodynamic time courses were performed. Critically, only the processing of disambiguated, coherent discourse with an intelligible situation model representation involved (shared) activation in bilateral lateral parietal and medial prefrontal regions. This shared spatiotemporal pattern of brain activation between the speaker and the listener suggests that the process of memory retrieval in medial prefrontal regions and the binding of retrieved information in the lateral parietal cortex constitutes a core mechanism underlying the communication of complex conceptual representations.

The Domain-General Multiple Demand (MD) Network Does Not Support Core Aspects of Language Comprehension: A Large-Scale fMRI Investigation

Aside from the language-selective left-lateralized frontotemporal network, language comprehension sometimes recruits a domain-general bilateral frontoparietal network implicated in executive functions: the multiple demand (MD) network. However, the nature of the MD network's contributions to language comprehension remains debated. To illuminate the role of this network in language processing in humans, we conducted a large-scale fMRI investigation using data from 30 diverse word and sentence comprehension experiments (481 unique participants [female and male], 678 scanning sessions). In line with prior findings, the MD network was active during many language tasks. Moreover, similar to the language-selective network, which is robustly lateralized to the left hemisphere, these responses were stronger in the left-hemisphere MD regions. However, in contrast with the language-selective network, the MD network responded more strongly (1) to lists of unconnected words than to sentences, and (2) in paradigms with an explicit task compared with passive comprehension paradigms. Indeed, many passive comprehension tasks failed to elicit a response above the fixation baseline in the MD network, in contrast to strong responses in the language-selective network. Together, these results argue against a role for the MD network in core aspects of sentence comprehension, such as inhibiting irrelevant meanings or parses, keeping intermediate representations active in working memory, or predicting upcoming words or structures. These results align with recent evidence of relatively poor tracking of the linguistic signal by the MD regions during naturalistic comprehension, and instead suggest that the MD network's engagement during language processing reflects effort associated with extraneous task demands.SIGNIFICANCE STATEMENT Domain-general executive processes, such as working memory and cognitive control, have long been implicated in language comprehension, including in neuroimaging studies that have reported activation in domain-general multiple demand (MD) regions for linguistic manipulations. However, much prior evidence has come from paradigms where language interpretation is accompanied by extraneous tasks. Using a large fMRI dataset (30 experiments/481 participants/678 sessions), we demonstrate that MD regions are engaged during language comprehension in the presence of task demands, but not during passive reading/listening, conditions that strongly activate the frontotemporal language network. These results present a fundamental challenge to proposals whereby linguistic computations, such as inhibiting irrelevant meanings, keeping representations active in working memory, or predicting upcoming elements, draw on domain-general executive resources.

Language comprehension in the social brain: Electrophysiological brain signals of social presence effects during syntactic and semantic sentence processing

Although, evolutionarily, language emerged predominantly for social purposes, much has yet to be uncovered regarding how language processing is affected by social context. Social presence research studies the ways in which the presence of a conspecific affects processing, but has yet to be thoroughly applied to language processes. The principal aim of this study was to see how syntactic and semantic language processing might be subject to mere social presence effects by studying Event-Related brain Potentials (ERP). In a sentence correctness task, participants read sentences with a semantic or syntactic anomaly while being either alone or in the mere presence of a confederate. Compared to the alone condition, the presence condition was associated with an enhanced N400 component and a more centro-posterior LAN component (interpreted as an N400). The results seem to imply a boosting of heuristic language processing strategies, proper of lexico-semantic operations, which actually entails a shift in the strategy to process morphosyntactic violations, typically based on algorithmic or rule-based strategies. The effects cannot be related to increased arousal levels. The apparent enhancement of the activity in the precuneus while in presence of another person suggests that the effects conceivably relate to social cognitive and attentional factors. The present results suggest that understanding language comprehension would not be complete without considering the impact of social presence effects, inherent to the most natural and fundamental communicative scenarios.

Motor-system dynamics during naturalistic reading of action narratives in first and second language

Do embodied semantic systems play different roles depending on when and how well a given language was learned? Emergent evidence suggests that this is the case for isolated, decontextualized stimuli, but no study has addressed the issue considering naturalistic narratives. Seeking to bridge this gap, we assessed motor-system dynamics in 26 Spanish-English bilinguals as they engaged in free, unconstrained reading of naturalistic action texts (ATs, highlighting the characters’ movements) and neutral texts (NTs, featuring low motility) in their first and second language (L1, L2). To explore functional connectivity spread over each reading session, we recorded ongoing high-density electroencephalographic signals and subjected them to functional connectivity analysis via a spatial clustering approach. Results showed that, in L1, AT (relative to NT) reading involved increased connectivity between left and right central electrodes consistently implicated in action-related processes, as well as distinct source-level modulations in motor regions. In L2, despite null group-level effects, enhanced motor-related connectivity during AT reading correlated positively with L2 proficiency and negatively with age of L2 learning. Taken together, these findings suggest that action simulations during unconstrained narrative reading involve neural couplings between motor-sensitive mechanisms, in proportion to how consolidated a language is. More generally, such evidence addresses recent calls to test the ecological validity of motor-resonance effects while offering new insights on their relation with experiential variables.

Broca's Area Is Not a Natural Kind

Theories of human cognition prominently feature 'Broca's area', which causally contributes to a myriad of mental functions. However, Broca's area is not a monolithic, multipurpose unit - it is structurally and functionally heterogeneous. Some functions engaging (subsets of) this area share neurocognitive resources, whereas others rely on separable circuits. A decade of converging evidence has now illuminated a fundamental distinction between two subregions of Broca's area that likely play computationally distinct roles in cognition: one belongs to the domain-specific 'language network', the other to the domain-general 'multiple-demand (MD) network'. Claims about Broca's area should be (re)cast in terms of these (and other, as yet undetermined) functional components, to establish a cumulative research enterprise where empirical findings can be replicated and theoretical proposals can be meaningfully compared and falsified.

Expectancy effects in the EEG during joint and spontaneous word-by-word sentence production in German

Our aim in the present study is to measure neural correlates during spontaneous interactive sentence production. We present a novel approach using the word-by-word technique from improvisational theatre, in which two speakers jointly produce one sentence. This paradigm allows the assessment of behavioural aspects, such as turn-times, and electrophysiological responses, such as event-related-potentials (ERPs). Twenty-five participants constructed a cued but spontaneous four-word German sentence together with a confederate, taking turns for each word of the sentence. In 30% of the trials, the confederate uttered an unexpected gender-marked article. To complete the sentence in a meaningful way, the participant had to detect the violation and retrieve and utter a new fitting response. We found significant increases in response times after unexpected words and - despite allowing unscripted language production and naturally varying speech material - successfully detected significant N400 and P600 ERP effects for the unexpected word. The N400 EEG activity further significantly predicted the response time of the subsequent turn. Our results show that combining behavioural and neuroscientific measures of verbal interactions while retaining sufficient experimental control is possible, and that this combination provides promising insights into the mechanisms of spontaneous spoken dialogue.

fMRI reveals language-specific predictive coding during naturalistic sentence comprehension

Much research in cognitive neuroscience supports prediction as a canonical computation of cognition across domains. Is such predictive coding implemented by feedback from higher-order domain-general circuits, or is it locally implemented in domain-specific circuits? What information sources are used to generate these predictions? This study addresses these two questions in the context of language processing. We present fMRI evidence from a naturalistic comprehension paradigm (1) that predictive coding in the brain's response to language is domain-specific, and (2) that these predictions are sensitive both to local word co-occurrence patterns and to hierarchical structure. Using a recently developed continuous-time deconvolutional regression technique that supports data-driven hemodynamic response function discovery from continuous BOLD signal fluctuations in response to naturalistic stimuli, we found effects of prediction measures in the language network but not in the domain-general multiple-demand network, which supports executive control processes and has been previously implicated in language comprehension. Moreover, within the language network, surface-level and structural prediction effects were separable. The predictability effects in the language network were substantial, with the model capturing over 37% of explainable variance on held-out data. These findings indicate that human sentence processing mechanisms generate predictions about upcoming words using cognitive processes that are sensitive to hierarchical structure and specialized for language processing, rather than via feedback from high-level executive control mechanisms.

Quasi-compositional mapping from form to meaning: a neural network-based approach to capturing neural responses during human language comprehension

We argue that natural language can be usefully described as quasi-compositional and we suggest that deep learning-based neural language models bear long-term promise to capture how language conveys meaning. We also note that a successful account of human language processing should explain both the outcome of the comprehension process and the continuous internal processes underlying this performance. These points motivate our discussion of a neural network model of sentence comprehension, the Sentence Gestalt model, which we have used to account for the N400 component of the event-related brain potential (ERP), which tracks meaning processing as it happens in real time. The model, which shares features with recent deep learning-based language models, simulates N400 amplitude as the automatic update of a probabilistic representation of the situation or event described by the sentence, corresponding to a temporal difference learning signal at the level of meaning. We suggest that this process happens relatively automatically, and that sometimes a more-controlled attention-dependent process is necessary for successful comprehension, which may be reflected in the subsequent P600 ERP component. We relate this account to current deep learning models as well as classic linguistic theory, and use it to illustrate a domain general perspective on some specific linguistic operations postulated based on compositional analyses of natural language. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Is less readable liked better? The case of font readability in poetry appreciation

Previous research shows conflicting findings for the effect of font readability on comprehension and memory for language. It has been found that—perhaps counterintuitively–a hard to read font can be beneficial for language comprehension, especially for difficult language. Here we test how font readability influences the subjective experience of poetry reading. In three experiments we tested the influence of poem difficulty and font readability on the subjective experience of poems. We specifically predicted that font readability would have opposite effects on the subjective experience of easy versus difficult poems. Participants read poems which could be more or less difficult in terms of conceptual or structural aspects, and which were presented in a font that was either easy or more difficult to read. Participants read existing poems and subsequently rated their subjective experience (measured through four dependent variables: overall liking, perceived flow of the poem, perceived topic clarity, and perceived structure). In line with previous literature we observed a Poem Difficulty x Font Readability interaction effect for subjective measures of poetry reading. We found that participants rated easy poems as nicer when presented in an easy to read font, as compared to when presented in a hard to read font. Despite the presence of the interaction effect, we did not observe the predicted opposite effect for more difficult poems. We conclude that font readability can influence reading of easy and more difficult poems differentially, with strongest effects for easy poems.

An Integrated Neural Decoder of Linguistic and Experiential Meaning

The brain is thought to combine linguistic knowledge of words and nonlinguistic knowledge of their referents to encode sentence meaning. However, functional neuroimaging studies aiming at decoding language meaning from neural activity have mostly relied on distributional models of word semantics, which are based on patterns of word co-occurrence in text corpora. Here, we present initial evidence that modeling nonlinguistic "experiential" knowledge contributes to decoding neural representations of sentence meaning. We model attributes of peoples' sensory, motor, social, emotional, and cognitive experiences with words using behavioral ratings. We demonstrate that fMRI activation elicited in sentence reading is more accurately decoded when this experiential attribute model is integrated with a text-based model than when either model is applied in isolation (participants were 5 males and 9 females). Our decoding approach exploits a representation-similarity-based framework, which benefits from being parameter free, while performing at accuracy levels comparable with those from parameter fitting approaches, such as ridge regression. We find that the text-based model contributes particularly to the decoding of sentences containing linguistically oriented "abstract" words and reveal tentative evidence that the experiential model improves decoding of more concrete sentences. Finally, we introduce a cross-participant decoding method to estimate an upper bound on model-based decoding accuracy. We demonstrate that a substantial fraction of neural signal remains unexplained, and leverage this gap to pinpoint characteristics of weakly decoded sentences and hence identify model weaknesses to guide future model development.SIGNIFICANCE STATEMENT Language gives humans the unique ability to communicate about historical events, theoretical concepts, and fiction. Although words are learned through language and defined by their relations to other words in dictionaries, our understanding of word meaning presumably draws heavily on our nonlinguistic sensory, motor, interoceptive, and emotional experiences with words and their referents. Behavioral experiments lend support to the intuition that word meaning integrates aspects of linguistic and nonlinguistic "experiential" knowledge. However, behavioral measures do not provide a window on how meaning is represented in the brain and tend to necessitate artificial experimental paradigms. We present a model-based approach that reveals early evidence that experiential and linguistically acquired knowledge can be detected in brain activity elicited in reading natural sentences.

Vector-Space Models of Semantic Representation From a Cognitive Perspective: A Discussion of Common Misconceptions

Models that represent meaning as high-dimensional numerical vectors—such as latent semantic analysis (LSA), hyperspace analogue to language (HAL), bound encoding of the aggregate language environment (BEAGLE), topic models, global vectors (GloVe), and word2vec—have been introduced as extremely powerful machine-learning proxies for human semantic representations and have seen an explosive rise in popularity over the past 2 decades. However, despite their considerable advancements and spread in the cognitive sciences, one can observe problems associated with the adequate presentation and understanding of some of their features. Indeed, when these models are examined from a cognitive perspective, a number of unfounded arguments tend to appear in the psychological literature. In this article, we review the most common of these arguments and discuss (a) what exactly these models represent at the implementational level and their plausibility as a cognitive theory, (b) how they deal with various aspects of meaning such as polysemy or compositionality, and (c) how they relate to the debate on embodied and grounded cognition. We identify common misconceptions that arise as a result of incomplete descriptions, outdated arguments, and unclear distinctions between theory and implementation of the models. We clarify and amend these points to provide a theoretical basis for future research and discussions on vector models of semantic representation.

Multimodal Language Processing in Human Communication

The natural ecology of human language is face-to-face interaction comprising the exchange of a plethora of multimodal signals. Trying to understand the psycholinguistic processing of language in its natural niche raises new issues, first and foremost the binding of multiple, temporally offset signals under tight time constraints posed by a turn-taking system. This might be expected to overload and slow our cognitive system, but the reverse is in fact the case. We propose cognitive mechanisms that may explain this phenomenon and call for a multimodal, situated psycholinguistic framework to unravel the full complexities of human language processing.

Simple Acoustic Features Can Explain Phoneme-Based Predictions of Cortical Responses to Speech

When we listen to speech, we have to make sense of a waveform of sound pressure. Hierarchical models of speech perception assume that, to extract semantic meaning, the signal is transformed into unknown, intermediate neuronal representations. Traditionally, studies of such intermediate representations are guided by linguistically defined concepts, such as phonemes. Here, we argue that in order to arrive at an unbiased understanding of the neuronal responses to speech, we should focus instead on representations obtained directly from the stimulus. We illustrate our view with a data-driven, information theoretic analysis of a dataset of 24 young, healthy humans who listened to a 1 h narrative while their magnetoencephalogram (MEG) was recorded. We find that two recent results, the improved performance of an encoding model in which annotated linguistic and acoustic features were combined and the decoding of phoneme subgroups from phoneme-locked responses, can be explained by an encoding model that is based entirely on acoustic features. These acoustic features capitalize on acoustic edges and outperform Gabor-filtered spectrograms, which can explicitly describe the spectrotemporal characteristics of individual phonemes. By replicating our results in publicly available electroencephalography (EEG) data, we conclude that models of brain responses based on linguistic features can serve as excellent benchmarks. However, we believe that in order to further our understanding of human cortical responses to speech, we should also explore low-level and parsimonious explanations for apparent high-level phenomena.

Bi-Directional Evidence Linking Sentence Production and Comprehension: A Cross-Modality Structural Priming Study

Natural language involves both speaking and listening. Recent models claim that production and comprehension share aspects of processing and are linked within individuals (Pickering and Garrod, 2004, 2013; MacDonald, 2013; Dell and Chang, 2014). Evidence for this claim has come from studies of cross-modality structural priming, mainly examining processing in the direction of comprehension to production. The current study replicated these comprehension to production findings and developed a novel cross-modal structural priming paradigm from production to comprehension using a temporally sensitive online measure of comprehension, Event-Related Potentials. For Comprehension-to-Production priming, participants first listened to active or passive sentences and then described target pictures using either structure. In Production-to-Comprehension priming, participants first described a picture using either structure and then listened to target passive sentences while EEG was recorded. Comprehension-to-Production priming showed the expected passive sentence priming for syntactic choice, but not response time (RT) or average syllable duration. In Production-to-Comprehension priming, primed, versus unprimed, passive sentences elicited a reduced N400. These effects support the notion that production and comprehension share aspects of processing and are linked within the individual. Moreover, this paradigm can be used for the exploration priming at different linguistic levels as well as the influence of extra-linguistic factors on natural language use.

I see what you meant to say: Anticipating speech errors during online sentence processing

Everyday speech is rife with errors and disfluencies, yet processing what we hear usually feels effortless. How does the language comprehension system accomplish such an impressive feat? The current experiment tests the hypothesis that listeners draw on relevant contextual and linguistic cues to anticipate speech errors and mentally correct them, even before receiving an explicit correction from the speaker. In the current visual-world eye-tracking experiment, we monitored participants' eye movements to objects in a display while they listened to utterances containing reparandum-repair speech errors (e.g., . . . his cat, uh I mean his dog . . .). The contextual plausibility of the misspoken word and the certainty with which the speaker uttered this word were systematically manipulated. Results showed that listeners immediately exploited these cues to generate top-down expectations regarding the speaker's communicative intention. Crucially, listeners used these expectations to constrain the bottom-up speech input and mentally correct perceived speech errors, even before the speaker initiated the correction. The results provide powerful evidence regarding the joint process of correcting speech errors that involves both the speaker and the listener. (PsycINFO Database Record (c) 2019 APA, all rights reserved).