Articulation Artifacts During Overt Language Production in Event-Related Brain Potentials: Description and Correction

Neural Decoding of Spontaneous Overt and Intended Speech

PURPOSE

The aim of this study was to decode intended and overt speech from neuromagnetic signals while the participants performed spontaneous overt speech tasks without cues or prompts (stimuli).

METHOD

Magnetoencephalography (MEG), a noninvasive neuroimaging technique, was used to collect neural signals from seven healthy adult English speakers performing spontaneous, overt speech tasks. The participants randomly spoke the words yes or no at a self-paced rate without cues. Two machine learning models, namely, linear discriminant analysis (LDA) and one-dimensional convolutional neural network (1D CNN), were employed to classify the two words from the recorded MEG signals.

RESULTS

LDA and 1D CNN achieved average decoding accuracies of 79.02% and 90.40%, respectively, in decoding overt speech, significantly surpassing the chance level (50%). The accuracy for decoding intended speech was 67.19% using 1D CNN.

CONCLUSIONS

This study showcases the possibility of decoding spontaneous overt and intended speech directly from neural signals in the absence of perceptual interference. We believe that these findings make a steady step toward the future spontaneous speech-based brain-computer interface.

Decoding kinematic information from beta-band motor rhythms of speech motor cortex: a methodological/analytic approach using concurrent speech movement tracking and magnetoencephalography

Introduction

Articulography and functional neuroimaging are two major tools for studying the neurobiology of speech production. Until now, however, it has generally not been feasible to use both in the same experimental setup because of technical incompatibilities between the two methodologies.

Methods

Here we describe results from a novel articulography system dubbed Magneto-articulography for the Assessment of Speech Kinematics (MASK), which is technically compatible with magnetoencephalography (MEG) brain scanning systems. In the present paper we describe our methodological and analytic approach for extracting brain motor activities related to key kinematic and coordination event parameters derived from time-registered MASK tracking measurements. Data were collected from 10 healthy adults with tracking coils on the tongue, lips, and jaw. Analyses targeted the gestural landmarks of reiterated utterances/ipa/ and /api/, produced at normal and faster rates.

Results

The results show that (1) Speech sensorimotor cortex can be reliably located in peri-rolandic regions of the left hemisphere; (2) mu (8-12 Hz) and beta band (13-30 Hz) neuromotor oscillations are present in the speech signals and contain information structures that are independent of those present in higher-frequency bands; and (3) hypotheses concerning the information content of speech motor rhythms can be systematically evaluated with multivariate pattern analytic techniques.

Discussion

These results show that MASK provides the capability, for deriving subject-specific articulatory parameters, based on well-established and robust motor control parameters, in the same experimental setup as the MEG brain recordings and in temporal and spatial co-register with the brain data. The analytic approach described here provides new capabilities for testing hypotheses concerning the types of kinematic information that are encoded and processed within specific components of the speech neuromotor system.

Evaluating the effectiveness of artifact correction and rejection in event-related potential research

Eyeblinks and other large artifacts can create two major problems in event-related potential (ERP) research, namely confounds and increased noise. Here, we developed a method for assessing the effectiveness of artifact correction and rejection methods at minimizing these two problems. We then used this method to assess a common artifact minimization approach, in which independent component analysis (ICA) is used to correct ocular artifacts, and artifact rejection is used to reject trials with extreme values resulting from other sources (e.g., movement artifacts). This approach was applied to data from five common ERP components (P3b, N400, N170, mismatch negativity, and error-related negativity). Four common scoring methods (mean amplitude, peak amplitude, peak latency, and 50% area latency) were examined for each component. We found that eyeblinks differed systematically across experimental conditions for several of the components. We also found that artifact correction was reasonably effective at minimizing these confounds, although it did not usually eliminate them completely. In addition, we found that the rejection of trials with extreme voltage values was effective at reducing noise, with the benefits of eliminating these trials outweighing the reduced number of trials available for averaging. For researchers who are analyzing similar ERP components and participant populations, this combination of artifact correction and rejection approaches should minimize artifact-related confounds and lead to improved data quality. Researchers who are analyzing other components or participant populations can use the method developed in this study to determine which artifact minimization approaches are effective in their data.

Cleaning up the Brickyard: How Theory and Methodology Shape Experiments in Cognitive Neuroscience of Language

The capacity for language is a defining property of our species, yet despite decades of research, evidence on its neural basis is still mixed and a generalized consensus is difficult to achieve. We suggest that this is partly caused by researchers defining "language" in different ways, with focus on a wide range of phenomena, properties, and levels of investigation. Accordingly, there is very little agreement among cognitive neuroscientists of language on the operationalization of fundamental concepts to be investigated in neuroscientific experiments. Here, we review chains of derivation in the cognitive neuroscience of language, focusing on how the hypothesis under consideration is defined by a combination of theoretical and methodological assumptions. We first attempt to disentangle the complex relationship between linguistics, psychology, and neuroscience in the field. Next, we focus on how conclusions that can be drawn from any experiment are inherently constrained by auxiliary assumptions, both theoretical and methodological, on which the validity of conclusions drawn rests. These issues are discussed in the context of classical experimental manipulations as well as study designs that employ novel approaches such as naturalistic stimuli and computational modeling. We conclude by proposing that a highly interdisciplinary field such as the cognitive neuroscience of language requires researchers to form explicit statements concerning the theoretical definitions, methodological choices, and other constraining factors involved in their work.

The temporal dynamics of the Stroop effect from childhood to young and older adulthood

The processes involved in the Stroop task/effect are thought to involve conflict detection and resolution stages. Little is known about the evolution of these two components over the lifespan. It is well admitted that children and older adults tend to show longer response latencies than young adults. The present study aims at clarifying the rational of such changes from childhood to adulthood and in aging by comparing the impacted cognitive processes across age groups. More precisely, the aim was to clarify if all processes take more time to be executed, hence implying that longer latencies rely mainly on processing speed or if an additional process lengthens the resolution of the conflict in children and/or older adults. To this aim we recorded brain electrical activity using EEG in school-age children, young and older adults while they performed a classic verbal Stroop task. The signal was decomposed in microstate brain networks, and age groups and conditions were compared. Behavioral results evolved following an inverted U-shaped curve. In children, different brain states from the ones observed in adults were highlighted, both in the conflict detection and resolution time-windows. Longer latencies in the incongruent condition were mainly attributed to an overly increased duration of the microstates involved in the conflict resolution time window. In aging, the same microstate maps were reported for both young and older adult groups. The differences in performances between groups could be explained by a disproportionally long conflict detection phase, even compressing the latest stage of response articulation. These results tend to favor a specific immaturity of the brain networks involved coupled with a slowing of the processes in children, while cognitive decline could be mostly explained by a general slowing.

Morphological facilitation and semantic interference in compound production: An ERP study

This study investigates the production of nominal compounds (Experiment 1) and simple nouns (Experiment 2) in a picture-word interference (PWI) paradigm to test models of morpho-lexical representation and processing. The continuous electroencephalogram (EEG) was registered and event-related brain potentials [ERPs] were analyzed in addition to picture-naming latencies. Experiment 1 used morphologically and semantically related distractor words to tap into different pre-articulatory planning stages during compound production. Relative to unrelated distractors, naming was speeded when distractors corresponded to morphemes of the compound (sun or flower for the target sunflower), but slowed when distractors were from the same semantic category as the compound (tulip ➔ sunflower). Distractors from the same category as the compound's first constituent (moon ➔ sunflower) had no influence. The diverging effects for semantic and morphological distractors replicate results from earlier studies. ERPs revealed different effects of morphological and semantic distractors with an interesting time course: morphological effects had an earlier onset. Comparable to the naming latencies, no ERP effects were obtained for distractors from the same semantic category as the compound's first constituent. Experiment 2 investigated the effectiveness of the latter distractors, presenting them with pictures of the compounds' first constituents (e.g., moon ➔ sun). Interference was confirmed both behaviorally and in the ERPs, showing that the absence of an effect in Experiment 1 was not due to the materials used. Considering current models of speech production, the data are best explained by a cascading flow of activation throughout semantic, lexical and morpho-phonological steps of speech planning.

Native language inhibition predicts more successful second language learning: Evidence of two ERP pathways during learning

How some individuals succeed in learning a second language as adults is still an unsolved question in cognitive neuroscience. At the brain level, adults' electrophysiological responses to input in a second language may differ after completing different types of training. However, there is limited understanding of what neural pathways are activated as learning unfolds, and which patterns of activation lead to successful learning. Using brain event-related potentials, this study explored whether individual brain responses to practice difficulty during second language learning predict learning outcomes. English-speaking learners of Spanish practiced completing newly learned phrases in their second language. For some learners, all the choices presented during practice were "easy" because non-target choices were unrelated distractors. In the more "difficult" practice mode, however, learners had to avoid choosing a competing word that would be acceptable based on their native language, but not in the second language being learned. Performance during practice was similar in both groups of learners. Critically, divergence in event-related potentials indicated alternative strategies to practice, based on the level of difficulty. At the group level, learners completing the easier practice revealed increased monitoring when making responses; in the difficult condition, learners showed inhibition of their native language (i.e., an N400 for phrases congruent with the native language) to avoid interference during word selection. Individual brain responses indexing the degree of native language inhibition predicted learning rates in tests.

Microstate ERP Analyses to Pinpoint the Articulatory Onset in Speech Production

The use of electroencephalography (EEG) to study overt speech production has increased substantially in the past 15 years and the alignment of evoked potential (ERPs) on the response onset has become an extremely useful method to target “latest” stages of speech production. Yet, response-locked ERPs raise a methodological issue: on which event should the point of alignment be placed? Response-locked ERPs are usually aligned to the vocal (acoustic) onset, although it is well known that articulatory movements may start up to a hundred milliseconds prior to the acoustic onset and that this “articulatory onset to acoustic onset interval” (AAI) depends on the phoneme properties. Given the previously reported difficulties to measure the AAI, the purpose of this study was to determine if the AAI could be reliably detected with EEG-microstates. High-density EEG was recorded during delayed speech production of monosyllabic pseudowords with four different onset consonants. Whereas the acoustic response onsets varied depending on the onset consonant, the response-locked spatiotemporal EEG analysis revealed a clear asynchrony of the same sequence of microstates across onset consonants. A specific microstate, the latest observed in the ERPs locked to the vocal onset, presented longer duration for phonemes with longer acoustic response onsets. Converging evidences seemed to confirm that this microstate may be related to the articulatory onset of motor execution: its scalp topography corresponded to those previously associated with muscle activity and source localization highlighted the involvement of motor areas. Finally, the analyses on the duration of such microstate in single trials further fit with the AAI intervals for specific phonemes reported in previous studies. These results thus suggest that a particular ERP-microstate is a reliable index of articulation onset and of the AAI.

When a second language hits a native language. What ERPs (do and do not) tell us about language retrieval difficulty in bilingual language production

The accumulating evidence suggests that prior usage of a second language (L2) leads to processing costs on the subsequent production of a native language (L1). However, it is unclear what mechanism underlies this effect. It has been proposed that the L1 cost reflects inhibition of L1 representation acting during L1 production; however, previous studies exploring this issue were inconclusive. It is also unsettled whether the mechanism operates on the whole-language level or is restricted to translation equivalents in the two languages. We report a study that allowed us to address both issues behaviorally with the use of ERPs while focusing on the consequences of using L2 on the production of L1. In our experiment, native speakers of Polish (L1) and learners of English (L2) named a set of pictures in L1 following a set of pictures in either L1 or L2. Half of the pictures were repeated from the preceding block and half were new; this enabled dissociation of the effects on the level of the whole language from those specific to individual lexical items. Our results are consistent with the notion that language after-effects operate at a whole-language level. Behaviorally, we observed a clear processing cost on the whole-language level and a small facilitation on the item-specific level. The whole-language effect was accompanied by an enhanced, fronto-centrally distributed negativity in the 250-350 ms time-window which we identified as the N300 (in contrast to previous research, which probably misidentified the effect as the N2), a component that presumably reflects retrieval difficulty of relevant language representations during picture naming. As such, unlike previous studies that reported N2 for naming pictures in L1 after L2 use, we propose that the reported ERPs (N300) indicate that prior usage of L2 hampers lexical access to names in L1. Based on the literature, the after-effects could be caused by L1 inhibition and/or L2 interference, but the ERPs so far have not been informative about the causal mechanism.

The Effect of Lexical Cohort Size Is Independent of Semantic Context Effects in a Picture-Word Interference Task: A Combined ERP and sLORETA Study

Lexical cohort size is known to play an important role in the magnitude of semantic interference during picture naming in continuous and blocking naming tasks. Nevertheless, whether and how lexical cohort size influences semantic context effects in a picture-word interference (PWI) task remains unclear. To address this issue, participants were required to name pictures, which were paired with both semantically related and unrelated distractors, from both large and small lexical cohorts while electroencephalogram (EEG) signals were recorded. Behavior results showed a semantic interference effect but no interaction between semantic relatedness and lexical cohort size in naming latencies. ERPs and correlation analyses revealed that semantic interference effects occurred at the lexical level in the time windows of 200-400 and 400-600 ms, and lexical cohort size effects occurred at the conceptual level in the time window of 100-200 ms and at the lexical level in the time windows of 200-400 ms. Critically, no interaction between two variables was found, reflecting that lexical cohort size is independent of semantic interference for categorical relations in the PWI. sLORETA results found stronger brain activations for large lexical cohorts at the left superior temporal gyrus and inferior frontal gyrus in the time interval of 250-300 ms, which may relate to lexical selection and self-monitoring. Our findings provide evidence for the swinging lexical network rather than the response exclusion hypothesis in spoken production.

Do bilingual children lag behind? A study of morphological encoding using ERPs

The current study investigates how bilingual children encode and produce morphologically complex words. We employed a silent-production-plus-delayed-vocalization paradigm in which event-related brain potentials (ERPs) were recorded during silent encoding of inflected words which were subsequently cued to be overtly produced. The bilingual children's spoken responses and their ERPs were compared to previous datasets from monolingual children on the same task. We found an enhanced negativity for regular relative to irregular forms during silent production in both bilingual children's languages, replicating the ERP effect previously obtained from monolingual children. Nevertheless, the bilingual children produced more morphological errors (viz. over-regularizations) than monolingual children. We conclude that mechanisms of morphological encoding (as measured by ERPs) are parallel for bilingual and monolingual children, and that the increased over-regularization rates are due to their reduced exposure to each of the two languages (relative to monolingual children).

Early and Late Effects of Semantic Distractors on Electroencephalographic Responses During Overt Picture Naming

This study investigated the nature of the interference effect of semantically related distractors in the picture-word interference paradigm, which has been claimed to be caused by either competition between lexical representations of target and distractor or by a late response exclusion mechanism that removes the distractor from a response buffer. EEG was recorded while participants overtly named pictures accompanied by categorically related versus unrelated written distractor words. In contrast to previous studies, stimuli were presented for only 250 ms to avoid any re-processing. ERP effects of relatedness were found around 290, 470, 540, and 660 ms post stimulus onset. In addition, related distractors led to an increase in midfrontal theta power, especially from about 440 to 540 ms, as well as to decreased high beta power between 40 and 110 ms and increased high beta power between 275 and 340 ms post stimulus onset. Response-locked analyses showed no differences in ERPs, however increased low and high beta power for related distractors in various time windows, most importantly a high beta power increase between -175 and -155 ms before speech onset. These results suggest that the semantic distractor effect is a combination of various effects and that the lexical competition account and the response exclusion account each capture a part, but not all aspects of the effect.

Copycat of dynamic facial expressions: Superior volitional motor control for expressions of disgust

In social situations facial expressions are often strategically employed. Despite the many research on motor control of limb movements, little is known about the control over facial expressions. Using a response-priming task, we investigated motor control over three facial expressions, smiles, disgust and emotionally neutral jaw drops. Prime stimuli consisted of videos of a facial expression to be prepared or - as a neutral prime - an abstract symbol superimposed to a scrambled face. In valid trials an equal symbol (=) indicated to produce the primed expression. In invalid trials, an unequal symbol (‡) prompted participants to produce an alternative, unprimed expression. We examined the impact of emotion in preparing and revoking a prepared expression, and possible facilitation for dynamic facial expressions relative to symbolic primes. Participants' facial responses were scored using automated analyses of facial expressions with computer software. The underlying neurocognitive processes were tracked with event-related-potentials. Reprogramming costs, in the form of longer reaction times (RTs) in trials where participants had prepared an invalidly primed expression and had to quickly switch to the correct one, were more pronounced for smiles and jaw drops than for disgust, possibly indicating the need for being fast when showing disgust. Data from the P3 component related the behavioral effect to a more efficient updating of the correct response in brain systems responsible for motor control. Priming participants with dynamic facial expressions as examples for imitation, improved performance accuracy as compared to the symbolic abstract stimuli, but it not did affect RTs. Priming with dynamic videos also resulted in larger validity effects of the P3 component when disgust was the target response, indicating that the perceptual system might trigger automatic emotional responses, at least for negative affect.

Variation in the speech signal as a window into the cognitive architecture of language production

The pronunciation of words is highly variable. This variation provides crucial information about the cognitive architecture of the language production system. This review summarizes key empirical findings about variation phenomena, integrating corpus, acoustic, articulatory, and chronometric data from phonetic and psycholinguistic studies. It examines how these data constrain our current understanding of word production processes and highlights major challenges and open issues that should be addressed in future research.