The fractionation of spoken language understanding by measuring electrical and magnetic brain signals

Episodic long-term memory formation during slow-wave sleep

We are unresponsive during slow-wave sleep but continue monitoring external events for survival. Our brain wakens us when danger is imminent. If events are non-threatening, our brain might store them for later consideration to improve decision-making. To test this hypothesis, we examined whether novel vocabulary consisting of simultaneously played pseudowords and translation words are encoded/stored during sleep, and which neural-electrical events facilitate encoding/storage. An algorithm for brain-state-dependent stimulation selectively targeted word pairs to slow-wave peaks or troughs. Retrieval tests were given 12 and 36 hr later. These tests required decisions regarding the semantic category of previously sleep-played pseudowords. The sleep-played vocabulary influenced awake decision-making 36 hr later, if targeted to troughs. The words' linguistic processing raised neural complexity. The words' semantic-associative encoding was supported by increased theta power during the ensuing peak. Fast-spindle power ramped up during a second peak likely aiding consolidation. Hence, new vocabulary played during slow-wave sleep was stored and influenced decision-making days later.

Extending the Architecture of Language From a Multimodal Perspective

Language is inherently multimodal. In spoken languages, combined spoken and visual signals (e.g., co-speech gestures) are an integral part of linguistic structure and language representation. This requires an extension of the parallel architecture, which needs to include the visual signals concomitant to speech. We present the evidence for the multimodality of language. In addition, we propose that distributional semantics might provide a format for integrating speech and co-speech gestures in a common semantic representation.

Modulating Complex Sentence Processing in Aphasia Through Attention and Semantic Networks

PURPOSE

Lexical processing impairments such as delayed and reduced activation of lexical-semantic information have been linked to syntactic processing disruptions and sentence comprehension deficits in individuals with aphasia (IWAs). Lexical-level deficits can also preclude successful lexical encoding during sentence processing and amplify the processing costs of similarity-based interference during syntactic retrieval. We investigate whether two manipulations to engage attention and pre-activate semantic features of a target (to-be-retrieved) noun will (a) boost lexical activation during initial lexical encoding and (b) facilitate syntactic dependency linking through improved resolution of interference in IWAs and neurologically unimpaired age-matched controls (AMCs).

METHOD

Eye-tracking-while-listening with a visual world paradigm was used to investigate whether semantic and attentional manipulations modulated initial lexical processing and downstream syntactic retrieval of the direct-object noun in object-relative sentences.

RESULTS

In the attention and semantic manipulations, the AMC group showed no changes in initial lexical access levels; however, gaze patterns revealed clear facilitations in dependency linking and interference resolution. In the IWA group, the attentional cue increased and maintained activation of N1 with modest facilitations in dependency linking. In the semantic condition, IWA results showed a greater degree of facilitation during dependency linking.

CONCLUSIONS

The results suggest that attention and semantic activation are parameters that may be manipulated to strengthen encoding of lexical representations to facilitate retrieval (i.e., dependency linking) and mitigate similarity-based interference. In IWAs, these manipulations may help to reduce lexical processing deficits that can preclude successful encoding.

Processing of Grammatical Agreement in the Face of Variation in Lexical Stress: A Mismatch Negativity Study

Previous electroencephalography studies have yielded evidence for automatic processing of syntax and lexical stress. However, these studies looked at both effects in isolation, limiting their generalizability to everyday language comprehension. In the current study, we investigated automatic processing of grammatical agreement in the face of variation in lexical stress. Using an oddball paradigm, we measured the Mismatch Negativity (MMN) in Dutch-speaking participants while they listened to Dutch subject-verb sequences (linguistic context) or acoustically similar sequences in which the subject was replaced by filtered noise (nonlinguistic context). The verb forms differed in the inflectional suffix, rendering the subject-verb sequences grammatically correct or incorrect, and leading to a difference in the stress pattern of the verb forms. We found that the MMNs were modulated in both the linguistic and nonlinguistic condition, suggesting that the processing load induced by variation in lexical stress can hinder early automatic processing of grammatical agreement. However, as the morphological differences between the verb forms correlated with differences in number of syllables, an interpretation in terms of the prosodic structure of the sequences cannot be ruled out. Future research is needed to determine which of these factors (i.e., lexical stress, syllabic structure) most strongly modulate early syntactic processing.

The pale spear-nosed bat: A neuromolecular and transgenic model for vocal learning

Vocal learning, the ability to produce modified vocalizations via learning from acoustic signals, is a key trait in the evolution of speech. While extensively studied in songbirds, mammalian models for vocal learning are rare. Bats present a promising study system given their gregarious natures, small size, and the ability of some species to be maintained in captive colonies. We utilize the pale spear-nosed bat (Phyllostomus discolor) and report advances in establishing this species as a tractable model for understanding vocal learning. We have taken an interdisciplinary approach, aiming to provide an integrated understanding across genomics (Part I), neurobiology (Part II), and transgenics (Part III). In Part I, we generated new, high-quality genome annotations of coding genes and noncoding microRNAs to facilitate functional and evolutionary studies. In Part II, we traced connections between auditory-related brain regions and reported neuroimaging to explore the structure of the brain and gene expression patterns to highlight brain regions. In Part III, we created the first successful transgenic bats by manipulating the expression of FoxP2, a speech-related gene. These interdisciplinary approaches are facilitating a mechanistic and evolutionary understanding of mammalian vocal learning and can also contribute to other areas of investigation that utilize P. discolor or bats as study species.

Evoking the N400 Event-related Potential (ERP) Component Using a Publicly Available Novel Set of Sentences with Semantically Incongruent or Congruent Eggplants (Endings)

During speech comprehension, the ongoing context of a sentence is used to predict sentence outcome by limiting subsequent word likelihood. Neurophysiologically, violations of context-dependent predictions result in amplitude modulations of the N400 event-related potential (ERP) component. While N400 is widely used to measure semantic processing and integration, no publicly-available auditory stimulus set is available to standardize approaches across the field. Here, we developed an auditory stimulus set of 442 sentences that utilized the semantic anomaly paradigm, provided cloze probability for all stimuli, and was developed for both children and adults. With 20 neurotypical adults, we validated that this set elicits robust N400′s, as well as two additional semantically-related ERP components: the recognition potential (~250 ms) and the late positivity component (~600 ms). This stimulus set (https://doi.org/10.5061/dryad.9ghx3ffkg) and the 20 high-density (128-channel) electrophysiological datasets (https://doi.org/10.5061/dryad.6wwpzgmx4) are made publicly available to promote data sharing and reuse. Future studies that use this stimulus set to investigate sentential semantic comprehension in both control and clinical populations may benefit from the increased comparability and reproducibility within this field of research.

Fine-grained time course of verb aspect processing

Sentence processing is known to be highly incremental. Speakers make incremental commitments as the sentence unfolds, dynamically updating their representations based on the smallest pieces of information from the incoming speech stream. Less is known about linguistic processing on the sub-word level, especially with regard to abstract grammatical information. This study employs the Visual World Paradigm to investigate the processing of grammatical aspect by Russian-speaking adults (n = 124). Aspectual information is encoded relatively early within the Russian verb which makes this an ideal testing ground to investigate the incrementality of grammatical processing on the sub-word level. Participants showed preference for pictures of ongoing events when they heard sentences involving Imperfective verbs, and for pictures of completed events when they heard sentences involving Perfective verbs. Crucially, the analysis of the participants’ eye-movements showed that they exhibited preference for the target picture already before they heard the end of the verb. Moreover, the latency of this effect depended on where the aspectual information was encoded within the verb. These results indicate that the processing and integration of grammatical aspect information can happen rapidly and incrementally on a fine-grained word-internal level. Methodologically, the study draws together a set of analytical techniques which can be fruitfully applied to the analysis of effect latencies in a wide range of studies within the Visual World eye-tracking paradigm.

Does the character-based dimension of stories impact narrative processing? An event-related potentials (ERPs) study

This event-related potentials (ERPs) study investigated online processes of integration of information relating to characters in narrative comprehension. The final sample included twenty-nine participants who read short third-person stories in which the plausibility of the characters' actions was manipulated. Stories were administered in three conditions: a character-based congruent condition including a target word that was consistent with the character's job; a character-based incongruent condition with a target word inconsistent with the character's job; a character-based neutral condition, narrating the action of a character presented by his/her proper name without information about his/her job. Results comparing the ERPs elicited by the experimental conditions revealed a greater negative amplitude of the N400 in the right temporal regions in response to the character-based incongruent compared to the character-based congruent narratives. This finding shows that implicit background character-based information affects the N400, with readers rapidly using this information to comprehend narratives.

The electrophysiology of aphasia: A scoping review

OBJECTIVE

To systematically assess the body of literature using N400 and P600 as they relate to people with aphasia. The primary aim was to reveal patterns in the literature which could be used to direct future research in the development of clinically relevant Event-Related Potentials (ERPs) for language assessment, while also identifying gaps in existing knowledge and highlight areas of further inquiry.

METHODS

A literature search was performed on studies published before May 2021. Relevant studies on aphasia and the two ERPs of interest were assessed for quality, and the relationship between aphasia and these ERPs was explored.

RESULTS

A total of 721 articles were identified, with 30 meeting inclusion criteria. Although there is significant variation in the literature, this scoping review revealed people with aphasia show reduced amplitude, delayed latency and different distribution compared to controls, and that ERPs are modulated by severity of aphasia.

CONCLUSIONS

To develop a relevant clinical tool for the management of aphasia, future research must strive to improve consistency within ERP methodology, with a greater number of diverse aphasia subtypes included in research.

SIGNIFICANCE

This scoping review reveals N400 and P600 represent promising potential biomarkers for the diagnosis and ongoing management of aphasia.

The Same Ultra-Rapid Parallel Brain Dynamics Underpin the Production and Perception of Speech

The temporal dynamics by which linguistic information becomes available is one of the key properties to understand how language is organized in the brain. An unresolved debate between different brain language models is whether words, the building blocks of language, are activated in a sequential or parallel manner. In this study, we approached this issue from a novel perspective by directly comparing the time course of word component activation in speech production versus perception. In an overt object naming task and a passive listening task, we analyzed with mixed linear models at the single-trial level the event-related brain potentials elicited by the same lexico-semantic and phonological word knowledge in the two language modalities. Results revealed that both word components manifested simultaneously as early as 75 ms after stimulus onset in production and perception; differences between the language modalities only became apparent after 300 ms of processing. The data provide evidence for ultra-rapid parallel dynamics of language processing and are interpreted within a neural assembly framework where words recruit the same integrated cell assemblies across production and perception. These word assemblies ignite early on in parallel and only later on reverberate in a behavior-specific manner.

Age-related dissociation of N400 effect and lexical priming

The use of contextual information is an important capability to facilitate language comprehension. This can be shown by studying behavioral and neurophysiological measures of accelerated word recognition when semantically or phonemically related information is provided in advance, resulting in accompanying attenuation of the respective event-related potential, i.e. the N400 effect. Against the background of age-dependent changes in a broad variety of lexical capacities, we aimed to study whether word priming is accomplished differently in elderly compared to young persons. 19 young (29.9 ± 5.6 years) and 15 older (69.0 ± 7.2 years) healthy adults participated in a primed lexical decision task that required the classification of target stimuli (words or pseudo-words) following related or unrelated prime words. We assessed reaction time, task accuracy and N400 responses. Acceleration of word recognition by semantic and phonemic priming was significant in both groups, but resulted in overall larger priming effects in the older participants. Compared with young adults, the older participants were slower and less accurate in responding to unrelated word-pairs. The expected N400 effect was smaller in older than young adults, particularly during phonemic word and pseudo-word priming, with a rather similar N400 amplitude reduction by semantic relatedness. The observed pattern of results is consistent with preserved or even enhanced lexical context sensitivity in older compared to young adults. This, however, appears to involve compensatory cognitive strategies with higher lexical processing costs during phonological processing in particular, suggested by a reduced N400 effect in the elderly.

Gricean Expectations in Online Sentence Comprehension: An ERP Study on the Processing of Scalar Inferences

There is substantial support for the general idea that a formalization of comprehenders' expectations about the likely next word in a sentence helps explaining data related to online sentence processing. While much research has focused on syntactic, semantic, and discourse expectations, the present event-related potentials (ERPs) study investigates neurolinguistic correlates of pragmatic expectations, which arise when comprehenders expect a sentence to conform to Gricean Maxims of Conversation. For predicting brain responses associated with pragmatic processing, we introduce a formal model of such Gricean pragmatic expectations, using an idealized incremental interpreter. We examine whether pragmatic expectancies derived from this model modulate the amplitude of the N400, a component that has been associated with predictive processing. As part of its parameterization, the model distinguishes genuine pragmatic interpreters, who expect maximally informative true utterances, from literal interpreters, who only expect truthfulness. We explore the model's non-trivial predictions for an experimental setup which uses picture-sentence verification with ERPs recorded at several critical positions in sentences containing the scalar implicature trigger some. We find that Gricean expectations indeed affect the N400, largely in line with the predictions of our model, but also discuss discrepancies between model predictions and observations critically.

Design and evaluation of the effectiveness of a corpus of congruent and incongruent English sentences for the study of event related potentials

OBJECTIVE

To design and evaluate the effectiveness of a stimulus material in eliciting the N400 event related potential (ERP).

DESIGN

A set of 700 semantically congruent and incongruent sentences was developed in accordance with current linguistic norms, and validated with an electroencephalography (EEG) study, in which the influence of age and gender on the N400 ERP magnitude was analysed.

STUDY SAMPLE

Forty-five normal-hearing subjects (19-57 years, 21 females) participated in the EEG study.

RESULTS

The stimulus material used in the EEG study elicited a robust N400 ERP, with a morphology consistent with the literature. Results also showed no statistically significant effect of age or gender on the N400 magnitude.

CONCLUSIONS

The material presented in this paper constitutes the largest complete stimulus set suitable for both auditory and text-based N400 experiments. This material may help facilitate the efficient implementation of future N400 ERP studies, as well as promote standardisation and consistency across studies.

Auditory neural tracking and lexical processing of speech in noise: Masker type, spatial location, and language experience

The present study investigated how single-talker and babble maskers affect auditory and lexical processing during native (L1) and non-native (L2) speech recognition. Electroencephalogram (EEG) recordings were made while L1 and L2 (Korean) English speakers listened to sentences in the presence of single-talker and babble maskers that were colocated or spatially separated from the target. The predictability of the sentences was manipulated to measure lexical-semantic processing (N400), and selective auditory processing of the target was assessed using neural tracking measures. The results demonstrate that intelligible single-talker maskers cause listeners to attend more to the semantic content of the targets (i.e., greater context-related N400 changes) than when targets are in babble, and that listeners track the acoustics of the target less accurately with single-talker maskers. L1 and L2 listeners both modulated their processing in this way, although L2 listeners had more difficulty with the materials overall (i.e., lower behavioral accuracy, less context-related N400 variation, more listening effort). The results demonstrate that auditory and lexical processing can be simultaneously assessed within a naturalistic speech listening task, and listeners can adjust lexical processing to more strongly track the meaning of a sentence in order to help ignore competing lexical content.

Changes in Speech-Related Brain Activity During Adaptation to Electro-Acoustic Hearing

Objectives: Hearing improves significantly with bimodal provision, i.e., a cochlear implant (CI) at one ear and a hearing aid (HA) at the other, but performance shows a high degree of variability resulting in substantial uncertainty about the performance that can be expected by the individual CI user. The objective of this study was to explore how auditory event-related potentials (AERPs) of bimodal listeners in response to spoken words approximate the electrophysiological response of normal hearing (NH) listeners. Study Design: Explorative prospective analysis during the first 6 months of bimodal listening using a within-subject repeated measures design. Setting: Academic tertiary care center. Participants: Twenty-seven adult participants with bilateral sensorineural hearing loss who received a HiRes 90K CI and continued use of a HA at the non-implanted ear. Age-matched NH listeners served as controls. Intervention: Cochlear implantation. Main Outcome Measures: Obligatory auditory evoked potentials N1 and P2, and the event-related N2 potential in response to monosyllabic words and their reversed sound traces before, as well as 3 and 6 months post-implantation. The task required word/non-word classification. Stimuli were presented within speech-modulated noise. Loudness of word/non-word signals was adjusted individually to achieve the same intelligibility across groups and assessments. Results: Intelligibility improved significantly with bimodal hearing, and the N1-P2 response approximated the morphology seen in NH with enhanced and earlier responses to the words compared to their reversals. For bimodal listeners, a prominent negative deflection was present between 370 and 570 ms post stimulus onset (N2), irrespective of stimulus type. This was absent for NH controls; hence, this response did not approximate the NH response during the study interval. N2 source localization evidenced extended activation of general cognitive areas in frontal and prefrontal brain areas in the CI group. Conclusions: Prolonged and spatially extended processing in bimodal CI users suggests employment of additional auditory-cognitive mechanisms during speech processing. This does not reduce within 6 months of bimodal experience and may be a correlate of the enhanced listening effort described by CI listeners.

fMRI evidence that left posterior temporal cortex contributes to N400 effects of predictability independent of congruity

Previous electrophysiological work argues that predictability and semantic incongruity rapidly impact comprehension, as indicated by modulation of the N400 component between ~300 and 500 ms. An ongoing question is whether effects of predictability in fact reflect pre-activation in long-term memory as opposed to modulating the kind of integration processes triggered by incongruity. Using fMRI, we compared the impact of predictability and incongruity in adjective-noun phrases, in regions identified with lexical and phrasal localizer scans. We found that predictability impacted activity in left posterior middle temporal gyrus (pMTG), while incongruity impacted activity in left precentral gyrus. Together with parallel data from ERP, these data are consistent with the hypothesis that left pMTG activity is a key contributor to N400 effects of predictability and that the relevant mechanism is reduced activation of stored lexical representations. We tentatively suggest that the left precentral region may play a role in reanalysis when incongruity is encountered.

MVPA Analysis of Intertrial Phase Coherence of Neuromagnetic Responses to Words Reliably Classifies Multiple Levels of Language Processing in the Brain

Neural processing of language is still among the most poorly understood functions of the human brain, whereas a need to objectively assess the neurocognitive status of the language function in a participant-friendly and noninvasive fashion arises in various situations. Here, we propose a solution for this based on a short task-free recording of MEG responses to a set of spoken linguistic contrasts. We used spoken stimuli that diverged lexically (words/pseudowords), semantically (action-related/abstract), or morphosyntactically (grammatically correct/ungrammatical). Based on beamformer source reconstruction we investigated intertrial phase coherence (ITPC) in five canonical bands (α, β, and low, medium, and high γ) using multivariate pattern analysis (MVPA). Using this approach, we could successfully classify brain responses to meaningful words from meaningless pseudowords, correct from incorrect syntax, as well as semantic differences. The best classification results indicated distributed patterns of activity dominated by core temporofrontal language circuits and complemented by other areas. They varied between the different neurolinguistic properties across frequency bands, with lexical processes classified predominantly by broad γ, semantic distinctions by α and β, and syntax by low γ feature patterns. Crucially, all types of processing commenced in a near-parallel fashion from ∼100 ms after the auditory information allowed for disambiguating the spoken input. This shows that individual neurolinguistic processes take place simultaneously and involve overlapping yet distinct neuronal networks that operate at different frequency bands. This brings further hope that brain imaging can be used to assess neurolinguistic processes objectively and noninvasively in a range of populations.

Combined predictive effects of sentential and visual constraints in early audiovisual speech processing

In language comprehension, a variety of contextual cues act in unison to render upcoming words more or less predictable. As a sentence unfolds, we use prior context (sentential constraints) to predict what the next words might be. Additionally, in a conversation, we can predict upcoming sounds through observing the mouth movements of a speaker (visual constraints). In electrophysiological studies, effects of visual constraints have typically been observed early in language processing, while effects of sentential constraints have typically been observed later. We hypothesized that the visual and the sentential constraints might feed into the same predictive process such that effects of sentential constraints might also be detectable early in language processing through modulations of the early effects of visual salience. We presented participants with audiovisual speech while recording their brain activity with magnetoencephalography. Participants saw videos of a person saying sentences where the last word was either sententially constrained or not, and began with a salient or non-salient mouth movement. We found that sentential constraints indeed exerted an early (N1) influence on language processing. Sentential modulations of the N1 visual predictability effect were visible in brain areas associated with semantic processing, and were differently expressed in the two hemispheres. In the left hemisphere, visual and sentential constraints jointly suppressed the auditory evoked field, while the right hemisphere was sensitive to visual constraints only in the absence of strong sentential constraints. These results suggest that sentential and visual constraints can jointly influence even very early stages of audiovisual speech comprehension.

Grammatical class modulates the (left) inferior frontal gyrus within 100 milliseconds when syntactic context is predictive

The current study set out to examine the spatiotemporal dynamics of predictive processing during syntactic processing. To do so, we conducted an MEG experiment in which we contrasted MRI-constrained sources elicited by nouns and verbs when they were preceded by a predictive syntactic context (i.e., possessive pronouns for nouns, and personal pronouns for verbs) versus a non-predictive syntactic context (visually matched symbols). The results showed rapid (from ~80 ms onwards) noun-verb differences in the left and (to a lesser extent) right inferior frontal gyri (IFG), but only when those nouns and verbs were preceded by the syntactically predictive context (i.e. their corresponding pronoun). Furthermore, the contrast between possessive and personal pronouns that preceded the rapid noun-verb modulations in the (L)IFG also produced differences in source activation in various regions of the prefrontal cortex (the superior frontal and orbitofrontal cortex). We suggest the data show that syntactic unification manifests very early on during processing in the LIFG. The speed of such syntactic unification operations is hypothesized to be driven by predictive top-down activations stemming from a domain-general network in the prefrontal cortex.

Oscillatory dynamics of cortical functional connections in semantic prediction

An event related potential, known as the N400, has been particularly useful in investigating language processing as it serves as a neural index for semantic prediction. There are numerous studies on the functional segregation of N400 neural sources; however, the oscillatory dynamics of functional connections among the relevant sources has remained elusive. In this study we acquired magnetoencephalography data during a classic N400 paradigm, where the semantic predictability of a fixed target noun was manipulated in simple German sentences. We conducted inter-regional functional connectivity (FC) and time-frequency analysis on known regions of the semantic network, encompassing bilateral temporal, and prefrontal cortices. Increased FC was found in less predicted (LP) nouns compared with highly predicted (HP) nouns in three connections: (a) right inferior frontal gyrus (IFG) and right middle temporal gyrus (MTG) from 0 to 300 ms mainly within the alpha band, (b) left lateral orbitofrontal (LOF) and right IFG around 400 ms within the beta band, and (c) left superior temporal gyrus (STG) and left LOF from 300 to 700 ms in the beta and low gamma bands. Furthermore, gamma spectral power (31-70 Hz) was stronger in HP nouns than in LP nouns in left anterior temporal cortices in earlier time windows (0-200 ms). Our findings support recent theories in language comprehension, suggesting fronto-temporal top-down connections are mainly mediated through beta oscillations while gamma band frequencies are involved in matching between prediction and input.

Morphological processing in the brain: The good (inflection), the bad (derivation) and the ugly (compounding)

There is considerable behavioral evidence that morphologically complex words such as 'tax-able' and 'kiss-es' are processed and represented combinatorially. In other words, they are decomposed into their constituents 'tax' and '-able' during comprehension (reading or listening), and producing them might also involve on-the-spot combination of these constituents (especially for inflections). However, despite increasing amount of neurocognitive research, the neural mechanisms underlying these processes are still not fully understood. The purpose of this critical review is to offer a comprehensive overview on the state-of-the-art of the research on the neural mechanisms of morphological processing. In order to take into account all types of complex words, we include findings on inflected, derived, and compound words presented both visually and aurally. More specifically, we cover a wide range of electro- and magnetoencephalography (EEG and MEG, respectively) as well as structural/functional magnetic resonance imaging (s/fMRI) studies that focus on morphological processing. We present the findings with respect to the temporal course and localization of morphologically complex word processing. We summarize the observed findings, their interpretations with respect to current psycholinguistic models, and discuss methodological approaches as well as their possible limitations.

Listening effort during speech perception enhances auditory and lexical processing for non-native listeners and accents

Speech communication in a non-native language (L2) can feel effortful, and the present study suggests that this effort affects both auditory and lexical processing. EEG recordings (electroencephalography) were made from native English (L1) and Korean listeners while they listened to English sentences spoken with two accents (English and Korean) in the presence of a distracting talker. Neural entrainment (i.e., phase locking between the EEG recording and the speech amplitude envelope) was measured for target and distractor talkers. L2 listeners had relatively greater entrainment for target talkers than did L1 listeners, likely because their difficulty with L2 speech recognition caused them to focus more attention on the speech signal. N400 was measured for the final word in each sentence, and L2 listeners had greater lexical processing in high-predictability sentences than did L1 listeners. L1 listeners had greater target-talker entrainment when listening to the more difficult L2 accent than their own L1 accent, and similarly had larger N400 responses for the L2 accent. It thus appears that the increased effort of L2 listeners, as well as L1 listeners understanding L2 speech, modulates their auditory and lexical processing during speech recognition. This may provide a mechanism to compensate for their perceptual challenges under adverse conditions.

Brain indices of disagreement with one's social values predict EU referendum voting behavior

Pre-electoral surveys typically attempt, and sometimes fail, to predict voting behavior on the basis of explicit measures of agreement or disagreement with a candidate or political position. Here, we assessed whether a specific brain signature of disagreement with one’s social values, the event-related potential component N400, could be predictive of voting behavior. We examined this possibility in the context of the EU referendum in the UK. In the 5 weeks preceding the referendum, we recorded the N400 while participants with different vote intentions expressed their agreement or disagreement with pro- and against-EU statements. We showed that the N400 responded to statements incongruent with one’s view regarding the EU. Crucially, this effect predicted actual voting behavior in decided as well as undecided voters. The N400 was a better predictor of voting choice than an explicit index of preference based on the behavioral responses. Our findings demonstrate that well-defined patterns of brain activity can forecast future voting behavior.

Vowel-speech versus pure-tone processing in healthy subjects

To investigate the characteristics of speech perception, we evaluated the differences in mismatch negativity (MMN) between vowel change and frequency change. Additionally, we examined the effects of gender, age, and educational length on MMN. Forty healthy adults (21 females), who were native Japanese speakers, participated in the study. A Japanese vowel-speech pair (standard/a/vs. deviant/o/) and a pure-tone pair (standard 1000 Hz vs. deviant 1050 Hz) were constructed. MMN elicited by vowel-speech sounds was larger and earlier compared with pure-tone sounds. Larger and earlier MMNs for vowel-speech sounds than for pure-tone sounds suggest different processing of linguistically relevant information at the early stage in the auditory cortex. In conclusion, the factors influencing on MMN are different between vowel-speech sounds and pure-tone sounds.

Building by Syntax: The Neural Basis of Minimal Linguistic Structures

Language comes in utterances in which words are bound together according to a simple rule-based syntactic computation (merge), which creates linguistic hierarchies of potentially infinite length-phrases and sentences. In the current functional magnetic resonance imaging study, we compared prepositional phrases and sentences-both involving merge-to word lists-not involving merge-to explore how this process is implemented in the brain. We found that merge activates the pars opercularis of the left inferior frontal gyrus (IFG; Brodmann Area [BA] 44) and a smaller region in the posterior superior temporal sulcus (pSTS). Within the IFG, sentences engaged a more anterior portion of the area (pars triangularis, BA 45)-compared with phrases-which showed activity peak in BA 44. As prepositional phrases, in contrast to sentences, do not contain verbs, activity in BA 44 may reflect structure-building syntactic processing, while the involvement of BA 45 may reflect the encoding of propositional meaning initiated by the verb. The pSTS appears to work together with the IFG during thematic role assignment not only at the sentential level, but also at the phrasal level. The present results suggest that merge, the process of binding words together into syntactic hierarchies, is primarily supported by BA 44 in the IFG.

From meaning to categorization: The hierarchical recruitment of brain circuits selective for action verbs

Sensorimotor and affective brain systems are known to be involved in language processing. However, to date it is still debated whether this involvement is a crucial step of semantic processing or, on the contrary, it is dependent on the specific context or strategy adopted to solve a task at hand. The present electroencephalographic (EEG) study is aimed at investigating which brain circuits are engaged when processing written verbs. By aligning event-related potentials (ERPs) both to the verb onset and to the motor response indexing the accomplishment of a semantic task of categorization, we were able to dissociate the relative stimulus-related and response-related cognitive components at play, respectively. EEG signal source reconstruction showed that while the recruitment of sensorimotor fronto-parietal circuits was time-locked with action verb onset, a left temporal-parietal circuit was time-locked to the task accomplishment. Crucially, by comparing the time course of both these bottom-up and top-down cognitive components, it appears that the frontal motor involvement precedes the task-related temporal-parietal activity. The present findings suggest that the recruitment of fronto-parietal sensorimotor circuits is independent of the specific strategy adopted to solve a semantic task and, given its temporal hierarchy, it may provide crucial information to brain circuits involved in the categorization task. Eventually, a discussion on how the present results may contribute to the clinical literature on patients affected by disorders specifically impairing the motor system is provided.

Don't Get Me Wrong: ERP Evidence from Cueing Communicative Intentions

How to make sure that one’s utterances are understood as intended when not facing each other? In order to convey communicative intentions, in digital communication emoticons and pragmatic cues are frequently used. Such cueing becomes even more crucial for implied interpretations (e.g., irony) that cannot be understood literally, but require extra information. Sentences, such as ‘That’s fantastic,’ may achieve either a literal or ironic meaning depending on the contextual constraints. In two experiments using event-related brain potentials (ERPs), we examined the effects of cueing communicative intentions (i.e., by means of quotation marks) on ironic and literal language comprehension. An impact of cueing on language processing was seen as early as 200 ms post-stimulus onset by the emergence of a P300 preceding a sustained positivity for cued irony relative to literal language, while for uncued irony a P200-P600 pattern was obtained. In presence of additional information for ironic intentions, pragmatic reanalysis allowing inferences on the message level may have occured immediately. Moreover, by examining the way of cueing (i.e., ambiguous vs. unambiguous cueing) this type of information for communicative intentions appeared to be only effective when the cues were unambiguous by matching pragmatic conventions. The findings suggest that cueing communicative intentions may immediately affect language comprehension, albeit depending on pragmatic conventions of the cues’ usage.

A simple technique to study embodied language processes: the grip force sensor

Research in cognitive neuroscience has shown that brain structures serving perceptual, emotional, and motor processes are also recruited during the understanding of language when it refers to emotion, perception, and action. However, the exact linguistic and extralinguistic conditions under which such language-induced activity in modality-specific cortex is triggered are not yet well understood. The purpose of this study is to introduce a simple experimental technique that allows for the online measure of language-induced activity in motor structures of the brain. This technique consists in the use of a grip force sensor that captures subtle grip force variations while participants listen to words and sentences. Since grip force reflects activity in motor brain structures, the continuous monitoring of force fluctuations provides a fine-grained estimation of motor activity across time. In other terms, this method allows for both localization of the source of language-induced activity to motor brain structures and high temporal resolution of the recorded data. To facilitate comparison of the data to be collected with this tool, we present two experiments that describe in detail the technical setup, the nature of the recorded data, and the analyses (including justification about the data filtering and artifact rejection) that we applied. We also discuss how the tool could be used in other domains of behavioral research.

Event-Related Potentials Reflect Deficits in Lexical Access: The N200 in Prematurely Born School-Aged Children

OBJECTIVES

Children born preterm have a high prevalence of neurocognitive deficits early in life. We examined whether the neural correlates of lexical access are atypical in 9-year-old children born preterm, and whether the findings of acoustic mapping correlate with language- and attention-related skills.

PATIENTS AND METHODS

The subjects were fourteen 9-year-old children born preterm and 14 full-term, typically developing controls. Two auditory event-related potential (ERP) components, the N200 and the N400, were used to assess discrimination response and word recognition. A set of behavioral tests (naming ability, auditory attention, phonological processing, pseudoword repetition, and comprehension of instructions) was performed, and the results were compared with the amplitudes, latencies, and scalp distribution of the ERP results.

RESULTS

In prematurely born children, neurophysiological deficits were associated with difficulties in auditory discrimination. The N200 amplitude correlated significantly with auditory attention and pseudoword repetition. The scalp distribution of both the N200 and the N400 was broader in children born preterm than in the controls. Low scores in the neuropsychological tasks referred to difficulties in auditory processing and memory.

CONCLUSIONS

Children born preterm have difficulties in lexical access together with memory- and attention-related processes, which may have a longstanding impact on their school outcomes and academic skills.

Semantic Processing of Sentences in Preschoolers With Specific Language Impairment: Evidence From the N400 Effect

PURPOSE

Given the complexity of sentence processing and the specific problems that children with specific language impairment (SLI) experience, we investigated the time course and characteristics of semantic processing at the sentence level in Dutch preschoolers with SLI.

METHOD

We measured N400 responses to semantically congruent and incongruent spoken sentences (e.g., "My father is eating an apple/*blanket") in a group of 37 Dutch preschoolers with SLI and in a group of 25 typically developing (TD) peers. We compared the time course and amplitude of the N400 effect between the two groups.

RESULTS

The TD group showed a strong posterior N400 effect in time windows 300-500 ms and 500-800 ms. In contrast, the SLI group demonstrated only a reliable N400 effect in the later time window, 500-800 ms, and did not show a stronger presence at posterior electrodes.

CONCLUSION

The findings suggest that the neuronal processing of semantic information at sentence level is atypical in preschoolers with SLI compared with TD children.

The Electrophysiology of Basic Phrase Building

A defining trait of linguistic competence is the ability to combine elements into increasingly complex structures to denote, and to comprehend, a potentially infinite number of meanings. Recent magnetoencephalography (MEG) work has investigated these processes by comparing the response to nouns in combinatorial (blue car) and non-combinatorial (rnsh car) contexts. In the current study we extended this paradigm using electroencephalography (EEG) to dissociate the role of semantic content from phonological well-formedness (yerl car). We used event-related potential (ERP) recordings in order to better relate the observed neurophysiological correlates of basic combinatorial operations to prior ERP work on comprehension. We found that nouns in combinatorial contexts (blue car) elicited a greater centro-parietal negativity between 180-400ms, independent of the phonological well-formedness of the context word. We discuss the potential relationship between this ‘combinatorial’ effect and classic N400 effects. We also report preliminary evidence for an early anterior negative deflection immediately preceding the critical noun in combinatorial contexts, which we tentatively interpret as an electrophysiological reflex of syntactic structure initialization.

Neural correlate of the construction of sentence meaning

The neural processes that underlie your ability to read and understand this sentence are unknown. Sentence comprehension occurs very rapidly, and can only be understood at a mechanistic level by discovering the precise sequence of underlying computational and neural events. However, we have no continuous and online neural measure of sentence processing with high spatial and temporal resolution. Here we report just such a measure: intracranial recordings from the surface of the human brain show that neural activity, indexed by γ-power, increases monotonically over the course of a sentence as people read it. This steady increase in activity is absent when people read and remember nonword-lists, despite the higher cognitive demand entailed, ruling out accounts in terms of generic attention, working memory, and cognitive load. Response increases are lower for sentence structure without meaning ("Jabberwocky" sentences) and word meaning without sentence structure (word-lists), showing that this effect is not explained by responses to syntax or word meaning alone. Instead, the full effect is found only for sentences, implicating compositional processes of sentence understanding, a striking and unique feature of human language not shared with animal communication systems. This work opens up new avenues for investigating the sequence of neural events that underlie the construction of linguistic meaning.

From Sensory Perception to Lexical-Semantic Processing: An ERP Study in Non-Verbal Children with Autism

This study examines electrocortical activity associated with visual and auditory sensory perception and lexical-semantic processing in nonverbal (NV) or minimally-verbal (MV) children with Autism Spectrum Disorder (ASD). Currently, there is no agreement on whether these children comprehend incoming linguistic information and whether their perception is comparable to that of typically developing children. Event-related potentials (ERPs) of 10 NV/MV children with ASD and 10 neurotypical children were recorded during a picture-word matching paradigm. Atypical ERP responses were evident at all levels of processing in children with ASD. Basic perceptual processing was delayed in both visual and auditory domains but overall was similar in amplitude to typically-developing children. However, significant differences between groups were found at the lexical-semantic level, suggesting more atypical higher-order processes. The results suggest that although basic perception is relatively preserved in NV/MV children with ASD, higher levels of processing, including lexical- semantic functions, are impaired. The use of passive ERP paradigms that do not require active participant response shows significant potential for assessment of non-compliant populations such as NV/MV children with ASD.

Evaluation of induced and evoked changes in EEG during selective attention to verbal stimuli

BACKGROUND

Two challenges need to be addressed before bringing non-motor mental tasks for brain-computer interface (BCI) control to persons in a minimally conscious state (MCS), who can be behaviorally unresponsive even when proven to be consciously aware: first, keeping the cognitive demands as low as possible so that they could be fulfilled by persons with MCS. Second, increasing the control of experimental protocol (i.e. type and timing of the task performance).

NEW METHOD

The goal of this study is twofold: first goal is to develop an experimental paradigm that can facilitate the performance of brain-teasers (e.g. mental subtraction and word generation) on the one hand, and can increase the control of experimental protocol on the other hand. The second goal of this study is to exploit the similar findings for mentally attending to someone else's verbal performance of brain-teaser tasks and self-performing the same tasks to setup an online BCI, and to compare it in healthy participants to the current "state-of-the-art" motor imagery (MI, sports).

RESULTS

The response accuracies for the best performing healthy participants indicate that selective attention to verbal performance of mental subtraction (SUB) is a viable alternative to the MI. Time-frequency analysis of the SUB task in one participant with MCS did not reveal any significant (p<0.05) EEG changes, whereas imagined performance of one sport of participants' choice (SPORT) revealed task-related EEG changes over neurophysiological plausible cortical areas.

COMPARISON WITH EXISTING METHODS

We found that mentally attending to someone else's verbal performance of brain-teaser tasks leads to similar results as in self-performing the same tasks.

CONCLUSIONS

In this work we demonstrated that a single auditory selective attention task (i.e. mentally attending to someone else's verbal performance of mental subtraction) can modulate both induced and evoked changes in EEG, and be used for yes/no communication in an auditory scanning paradigm.

Neural Correlates of Morphological Processing: Evidence from Chinese

Morphological decomposition is an important part of complex word processing. In Chinese, this requires a comprehensive consideration of phonological, orthographic and morphemic information. The left inferior frontal gyrus (L-IFG) has been implicated in this process in alphabetic languages. However, it is unclear whether the neural mechanisms underlying morphological processing in alphabetic languages would be the same in Chinese, a logographic language. To investigate the neural basis of morphological processing in Chinese compound words, an fMRI experiment was conducted using an explicit auditory morphological judgment task. Results showed the L-IFG to be a core area in Chinese morphological processing, consistent with research in alphabetic languages. Additionally, a broad network consisting of the L-MTG, the bilateral STG and the L-FG that taps phonological, orthographic, and semantic information was found to be involved. These results provide evidence that the L-IFG plays an important role in morphological processing even in languages that are typologically different.

Auditory perceptual objects as generative models: Setting the stage for communication by sound

Communication by sounds requires that the communication channels (i.e. speech/speakers and other sound sources) had been established. This allows to separate concurrently active sound sources, to track their identity, to assess the type of message arriving from them, and to decide whether and when to react (e.g., reply to the message). We propose that these functions rely on a common generative model of the auditory environment. This model predicts upcoming sounds on the basis of representations describing temporal/sequential regularities. Predictions help to identify the continuation of the previously discovered sound sources to detect the emergence of new sources as well as changes in the behavior of the known ones. It produces auditory event representations which provide a full sensory description of the sounds, including their relation to the auditory context and the current goals of the organism. Event representations can be consciously perceived and serve as objects in various cognitive operations.

The linguistic context effects on the processing of body-object interaction words: An ERP study on second language learners

Embodied theories of cognition argue that the processing of both concrete and abstract concepts requires the activation of sensorimotor systems. The present study examined the time course for embedding a sensorimotor context in order to elicit sensitivity to the sensorimotor consequences of understanding body-object interaction (BOI) words. In the study, Event-Related Potentials (ERPs) were recorded while subjects performed a sentence acceptability task. Target BOI words were preceded by rich or poor sensorimotor sentential contexts. The behavioural results replicated previous findings in that high BOI words received a response faster than low BOI words. In addition to this, however, there was a context effect in the sensorimotor region as well as a BOI effect in the parietal region (involved in object representation). The results indicate that the sentential sensorimotor context contributes to the subsequent BOI processing and that action-and perception-related language leads to the activation of the same brain areas, which is consistent with the embodiment theory.

Conversation electrified: ERP correlates of speech act recognition in underspecified utterances

The ability to recognize speech acts (verbal actions) in conversation is critical for everyday interaction. However, utterances are often underspecified for the speech act they perform, requiring listeners to rely on the context to recognize the action. The goal of this study was to investigate the time-course of auditory speech act recognition in action-underspecified utterances and explore how sequential context (the prior action) impacts this process. We hypothesized that speech acts are recognized early in the utterance to allow for quick transitions between turns in conversation. Event-related potentials (ERPs) were recorded while participants listened to spoken dialogues and performed an action categorization task. The dialogues contained target utterances that each of which could deliver three distinct speech acts depending on the prior turn. The targets were identical across conditions, but differed in the type of speech act performed and how it fit into the larger action sequence. The ERP results show an early effect of action type, reflected by frontal positivities as early as 200 ms after target utterance onset. This indicates that speech act recognition begins early in the turn when the utterance has only been partially processed. Providing further support for early speech act recognition, actions in highly constraining contexts did not elicit an ERP effect to the utterance-final word. We take this to show that listeners can recognize the action before the final word through predictions at the speech act level. However, additional processing based on the complete utterance is required in more complex actions, as reflected by a posterior negativity at the final word when the speech act is in a less constraining context and a new action sequence is initiated. These findings demonstrate that sentence comprehension in conversational contexts crucially involves recognition of verbal action which begins as soon as it can.

Dispositional mindfulness and semantic integration of emotional words: Evidence from event-related brain potentials

Initial research shows that mindfulness training can enhance attention and modulate the affective response. However, links between mindfulness and language processing remain virtually unexplored despite the prominent role of overt and silent negative ruminative speech in depressive and anxiety-related symptomatology. Here, we measured dispositional mindfulness and recorded participants' event-related brain potential responses to positive and negative target words preceded by words congruent or incongruent with the targets in terms of semantic relatedness and emotional valence. While the low mindfulness group showed similar N400 effect pattern for positive and negative targets, high dispositional mindfulness was associated with larger N400 effect to negative targets. This result suggests that negative meanings are less readily accessible in people with high dispositional mindfulness. Furthermore, high dispositional mindfulness was associated with reduced P600 amplitudes to emotional words, suggesting less post-analysis and attentional effort which possibly relates to a lower inclination to ruminate. Overall, these findings provide initial evidence on associations between modifications in language systems and mindfulness.

Context affects L1 but not L2 during bilingual word recognition: an MEG study

How do bilinguals manage the activation levels of the two languages and prevent interference from the irrelevant language? Using magnetoencephalography, we studied the effect of context on the activation levels of languages by manipulating the composition of word lists (the probability of the languages) presented auditorily to late Finnish-English bilinguals. We first determined the upper limit time-window for semantic access, and then focused on the preceding responses during which the actual word recognition processes were assumedly ongoing. Between 300 and 500 ms in the temporal cortices (in the N400 m response) we found an asymmetric language switching effect: the responses to L1 Finnish words were affected by the presentation context unlike the responses to L2 English words. This finding suggests that the stronger language is suppressed in an L2 context, supporting models that allow auditory word recognition to be affected by contextual factors and the language system to be subject to inhibitory influence.

EEG decoding of spoken words in bilingual listeners: from words to language invariant semantic-conceptual representations

Spoken word recognition and production require fast transformations between acoustic, phonological, and conceptual neural representations. Bilinguals perform these transformations in native and non-native languages, deriving unified semantic concepts from equivalent, but acoustically different words. Here we exploit this capacity of bilinguals to investigate input invariant semantic representations in the brain. We acquired EEG data while Dutch subjects, highly proficient in English listened to four monosyllabic and acoustically distinct animal words in both languages (e.g., “paard”–“horse”). Multivariate pattern analysis (MVPA) was applied to identify EEG response patterns that discriminate between individual words within one language (within-language discrimination) and generalize meaning across two languages (across-language generalization). Furthermore, employing two EEG feature selection approaches, we assessed the contribution of temporal and oscillatory EEG features to our classification results. MVPA revealed that within-language discrimination was possible in a broad time-window (~50–620 ms) after word onset probably reflecting acoustic-phonetic and semantic-conceptual differences between the words. Most interestingly, significant across-language generalization was possible around 550–600 ms, suggesting the activation of common semantic-conceptual representations from the Dutch and English nouns. Both types of classification, showed a strong contribution of oscillations below 12 Hz, indicating the importance of low frequency oscillations in the neural representation of individual words and concepts. This study demonstrates the feasibility of MVPA to decode individual spoken words from EEG responses and to assess the spectro-temporal dynamics of their language invariant semantic-conceptual representations. We discuss how this method and results could be relevant to track the neural mechanisms underlying conceptual encoding in comprehension and production.

Two routes to actorhood: lexicalized potency to act and identification of the actor role

The inference of causality is a crucial cognitive ability and language processing is no exception: recent research suggests that, across different languages, the human language comprehension system attempts to identify the primary causer of the state of affairs described (the "actor") quickly and unambiguously (Bornkessel-Schlesewsky and Schlesewsky, 2009). This identification can take place verb-independently based on certain prominence cues (e.g., case, word order, animacy). Here, we present two experiments demonstrating that actor potential is also encoded at the level of individual nouns (a king is a better actor than a beggar). Experiment 1 collected ratings for 180 German nouns on 12 scales defined by adjective oppositions and deemed relevant for actorhood potential. By means of structural equation modeling, an actor potential (ACT) value was calculated for each noun. Experiment 2, an event-related potential study, embedded nouns from Experiment 1 in verb-final sentences, in which they were either actors or non-actors. N400 amplitude increased with decreasing ACT values and this modulation was larger for highly frequent nouns and for actor versus non-actor nouns. We argue that potency to act is lexically encoded for individual nouns and, since it modulates the N400 even for non-actor participants, it should be viewed as a property that modulates ease of lexical access (akin, for example, to lexical frequency). We conclude that two separate dimensions of actorhood computation are crucial to language comprehension: an experience-based, lexically encoded (bottom-up) representation of actorhood potential, and a prominence-based, computational mechanism for calculating goodness-of-fit to the actor role in a particular (top-down) sentence context.