Saliency or template? ERP evidence for long-term representation of word stress

From psychophysiology to brain imaging: forty-five years MMN history of investigating acoustic change sensitivity

Forty-five years have passed since the first publication of the mismatch negativity (MMN) event-related brain potential (ERP) component. The first 10 years of research hardly gained any particular attention of the scientific community interested in acoustic perception. Debates on the nature of sensation versus perception were going on, and the technical possibilities to record ERPs, called in general evoked potentials, were very limited. Subtle changes in pure tone frequency or intensity giving rise to the MMN component were first investigated in humans. The background of the theoretical model developed by Risto Näätänen was the orientation reaction model of E.N. Sokolov published in 1963 so that the MMN was seen first as an electrophysiological correlate of auditory change detection. This fundamental ability of the auditory system seen as crucial for survival led to the development of the first animal model of the MMN (Csépe et al. in Clin Neurophysiol 66: 571-578, 1987). Indeed, it was confirmed that the MMN was the brain correlate of subtle changes detected that might alert to potential threats in the environment and direct the behavioral orientation. The investigations performed after 2000 introduced complex models and more sophisticated methods, both in animal and human studies, so that the MMN method was on the way to become a tool on the first place and not the main goal of research. This approach was further strengthened by the increasing number of studies on different clinical populations aiming at future applications. The aim of our review is to describe and redefine what the MMN may reflect in auditory perception and to show why and how this brain correlate of changes in the auditory scene can be used as a valuable tool in cognitive neuroscience research. We refer to publications selected to underly the argument the MMN cannot be classified anymore as a sign of simple change detection and not all the indicators used to confirm how genuine the MMN elicited by variations of tones are valid for those to  speech contrasts. We provide a fresh view on the broadly used MMN models, provided by some influential publications as well as on the unwritten history of MMN research aiming to give revised picture on what the MMN may truly reflect. We show how the focus and terminology of the MMN research have changed and what kind of misunderstandings and seemingly contradictive results prevent the MMN community to accept a generally usable cognitive model.

Investigating the neurocognitive background of speech perception with a fast multi-feature MMN paradigm

The speech multi-feature MMN (Mismatch Negativity) offers a means to explore the neurocognitive background of the processing of multiple speech features in a short time, by capturing the time-locked electrophysiological activity of the brain known as event-related brain potentials (ERPs). Originating from Näätänen et al. (Clin Neurophysiol 115:140-144, 2004) pioneering work, this paradigm introduces several infrequent deviant stimuli alongside standard ones, each differing in various speech features. In this study, we aimed to refine the multi-feature MMN paradigm used previously to encompass both segmental and suprasegmental (prosodic) features of speech. In the experiment, a two-syllable long pseudoword was presented as a standard, and the deviant stimuli included alterations in consonants (deviation by place or place and mode of articulation), vowels (deviation by place or mode of articulation), and stress pattern in the first syllable of the pseudoword. Results indicated the emergence of MMN components across all segmental and prosodic contrasts, with the expected fronto-central amplitude distribution. Subsequent analyses revealed subtle differences in MMN responses to the deviants, suggesting varying sensitivity to phonetic contrasts. Furthermore, individual differences in MMN amplitudes were noted, partially attributable to participants' musical and language backgrounds. These findings underscore the utility of the multi-feature MMN paradigm for rapid and efficient investigation of the neurocognitive mechanisms underlying speech processing. Moreover, the paradigm demonstrated the potential to be used in further research to study the speech processing abilities in various populations.

Predictions about prosody facilitate lexical access: Evidence from P50/N100 and MMN components

Research into the neural foundation of perception asserts a model where top-down predictions modulate the bottom-up processing of sensory input. Despite becoming increasingly influential in cognitive neuroscience, the precise account of this predictive coding framework remains debated. In this study, we aim to contribute to this debate by investigating how predictions about prosody facilitate speech perception, and to shed light especially on lexical access influenced by simultaneous predictions in different domains, inter alia, prosodic and semantic. Using a passive auditory oddball paradigm, we examined neural responses to prosodic changes, leading to a semantic change as in Dutch nouns canon ['kaːnɔn] 'canon' vs kanon [kaː'nɔn] 'cannon', and used acoustically identical pseudowords as controls. Results from twenty-eight native speakers of Dutch (age range 18-32 years) indicated an enhanced P50/N100 complex to prosodic change in pseudowords as well as an MMN response to both words and pseudowords. The enhanced P50/N100 response to pseudowords is claimed to indicate that all relevant auditory information is still processed by the brain, whereas the reduced response to words might reflect the suppression of information that has already been encoded. The MMN response to pseudowords and words, on the other hand, is best justified by the unification of previously established prosodic representations with sensory and semantic input respectively. This pattern of results is in line with the predictive coding framework acting on multiple levels and is of crucial importance to indicate that predictions about linguistic prosodic information are utilized by the brain as early as 50 ms.

ERP evidence for Slavic and German word stress cue sensitivity in English

Word stress is demanding for non-native learners of English, partly because speakers from different backgrounds weight perceptual cues to stress like pitch, intensity, and duration differently. Slavic learners of English and particularly those with a fixed stress language background like Czech and Polish have been shown to be less sensitive to stress in their native and non-native languages. In contrast, German English learners are rarely discussed in a word stress context. A comparison of these varieties can reveal differences in the foreign language processing of speakers from two language families. We use electroencephalography (EEG) to explore group differences in word stress cue perception between Slavic and German learners of English. Slavic and German advanced English speakers were examined in passive multi-feature oddball experiments, where they were exposed to the word impact as an unstressed standard and as deviants stressed on the first or second syllable through higher pitch, intensity, or duration. The results revealed a robust Mismatch Negativity (MMN) component of the event-related potential (ERP) in both language groups in response to all conditions, demonstrating sensitivity to stress changes in a non-native language. While both groups showed higher MMN responses to stress changes to the second than the first syllable, this effect was more pronounced for German than for Slavic participants. Such group differences in non-native English word stress perception from the current and previous studies are argued to speak in favor of customizable language technologies and diversified English curricula compensating for non-native perceptual variation.

Processing of Grammatical Agreement in the Face of Variation in Lexical Stress: A Mismatch Negativity Study

Previous electroencephalography studies have yielded evidence for automatic processing of syntax and lexical stress. However, these studies looked at both effects in isolation, limiting their generalizability to everyday language comprehension. In the current study, we investigated automatic processing of grammatical agreement in the face of variation in lexical stress. Using an oddball paradigm, we measured the Mismatch Negativity (MMN) in Dutch-speaking participants while they listened to Dutch subject-verb sequences (linguistic context) or acoustically similar sequences in which the subject was replaced by filtered noise (nonlinguistic context). The verb forms differed in the inflectional suffix, rendering the subject-verb sequences grammatically correct or incorrect, and leading to a difference in the stress pattern of the verb forms. We found that the MMNs were modulated in both the linguistic and nonlinguistic condition, suggesting that the processing load induced by variation in lexical stress can hinder early automatic processing of grammatical agreement. However, as the morphological differences between the verb forms correlated with differences in number of syllables, an interpretation in terms of the prosodic structure of the sequences cannot be ruled out. Future research is needed to determine which of these factors (i.e., lexical stress, syllabic structure) most strongly modulate early syntactic processing.

English and Mandarin native speakers' cue-weighting of lexical stress: Results from MMN and LDN

Past research on how listeners weight stress cues such as pitch, duration and intensity has reported two inconsistent patternss: listeners' weighting conforms to 1) their native language experience (e.g., language rhythmicity, lexical tone), and 2) a general "iambic-trochaic law" (ITL), favouring innate sound groupings in cue perception. This study aims to tease apart the above effects by investigating the weighting of pitch, duration and intensity cues in stress-timed (Australian English) and non-stress-timed and tonal (Taiwan Mandarin) language speaking adults using a mismatch negativity (MMN) multi-feature paradigm. Results show effects that can be explained by language-specific rhythmic influence, but only partially by the ITL. Moreover, these findings revealed cross-linguistic differences indexed by both MMN and late discriminative negativity (LDN) responses at cue and syllable position levels, and thus call for more sophisticated perspectives for existing cue-weighting models.

Evidence For Selective Adaptation and Recalibration in the Perception of Lexical Stress

Individuals vary in how they produce speech. This variability affects both the segments (vowels and consonants) and the suprasegmental properties of their speech (prosody). Previous literature has demonstrated that listeners can adapt to variability in how different talkers pronounce the segments of speech. This study shows that listeners can also adapt to variability in how talkers produce lexical stress. Experiment 1 demonstrates a selective adaptation effect in lexical stress perception: repeatedly hearing Dutch trochaic words biased perception of a subsequent lexical stress continuum towards more iamb responses. Experiment 2 demonstrates a recalibration effect in lexical stress perception: when ambiguous suprasegmental cues to lexical stress were disambiguated by lexical orthographic context as signaling a trochaic word in an exposure phase, Dutch participants categorized a subsequent test continuum as more trochee-like. Moreover, the selective adaptation and recalibration effects generalized to novel words, not encountered during exposure. Together, the experiments demonstrate that listeners also flexibly adapt to variability in the suprasegmental properties of speech, thus expanding our understanding of the utility of listener adaptation in speech perception. Moreover, the combined outcomes speak for an architecture of spoken word recognition involving abstract prosodic representations at a prelexical level of analysis.

Perception of Prosodic Modulations of Linguistic and Paralinguistic Origin: Evidence From Early Auditory Event-Related Potentials

How listeners handle prosodic cues of linguistic and paralinguistic origin is a central question for spoken communication. In the present EEG study, we addressed this question by examining neural responses to variations in pitch accent (linguistic) and affective (paralinguistic) prosody in Swedish words, using a passive auditory oddball paradigm. The results indicated that changes in pitch accent and affective prosody elicited mismatch negativity (MMN) responses at around 200 ms, confirming the brain’s pre-attentive response to any prosodic modulation. The MMN amplitude was, however, statistically larger to the deviation in affective prosody in comparison to the deviation in pitch accent and affective prosody combined, which is in line with previous research indicating not only a larger MMN response to affective prosody in comparison to neutral prosody but also a smaller MMN response to multidimensional deviants than unidimensional ones. The results, further, showed a significant P3a response to the affective prosody change in comparison to the pitch accent change at around 300 ms, in accordance with previous findings showing an enhanced positive response to emotional stimuli. The present findings provide evidence for distinct neural processing of different prosodic cues, and statistically confirm the intrinsic perceptual and motivational salience of paralinguistic information in spoken communication.

Electrophysiological Signatures of Perceiving Alternated Tone in Mandarin Chinese: Mismatch Negativity to Underlying Tone Conflict

Although phonological alternation is prevalent in languages, the process of perceiving phonologically alternated sounds is poorly understood, especially at the neurolinguistic level. We examined the process of perceiving Mandarin 3rd tone sandhi (T3 + T3 → T2 + T3) with a mismatch negativity (MMN) experiment. Our design has two independent variables (whether the deviant undergoes tone sandhi; whether the standard and the deviant have matched underlying tone). These two independent variables modulated ERP responses in both the first and the second syllables. Notably, despite the apparent segmental conflict between the standard and the deviant in all conditions, MMN is only observed when neither the standard nor the deviant undergoes tone sandhi, suggesting that discovering the underlying representation of an alternated sound could interfere with the generation of MMN. A tentative model with three hypothesized underlying processing mechanisms is proposed to explain the observed latency and amplitude differences across conditions. The results are also discussed in light of the potential electrophysiological signatures involved in the process of perceiving alternated sounds.

The effect of lexical status on prosodic processing in infants learning a fixed stress language

In speech processing, in the first year of life, prosody and phoneme-relevant aspects serve different functions. Recent studies have assumed that the two aspects become integrated at around 9 months of age. The present study investigates the effect of lexical status on stress processing in a fixed stress language. We hypothesize that lexicality modulates stress processing, and where the stress cue is in conflict with the lexical status (legal deviant condition), we will observe differences in age indicating the stage of integration. We tested 69 6 and 10 month-old infants in an acoustic oddball event-related potential paradigm. A frequent word stimulus (baba) and a pseudoword (bebe) were used with legal versus illegal stress. We systematically swapped the standard and deviant roles of the different stress variants in two conditions. In the illegal deviant condition in the case of the word stimulus, the response pattern typical for the pseudoword (an MMR to the absence of the stress cue) was missing. This implies the suppression effect of lexicality. In the legal deviant condition, negative MMR (N-MMR) in the second time window indicated a facilitation effect of lexicality in both age groups. As only the 6-month-olds produced an N-MMR in the first time window, we concluded that in a fixed stress language, integration starts at 6 months but is only completed by the age of 10 months. Our results show that lexical status modulates stress processing at word level in a highly regularly stressed language in which stable, long-term language-specific stress representation exists from early infancy.

Near native-like stress pattern perception in English-French bilinguals as indexed by the mismatch negativity

We examined lexical stress processing in English-French bilinguals. Auditory mismatch negativity (MMN) responses were recorded in response to English and French pseudowords, whose primary stress occurred either on a language-consistent "usual" or language-inconsistent "unusual" syllable. In most conditions, the pseudowords elicited two consecutive MMNs, and somewhat surprisingly, these MMNs were not systematically modulated by bilingual experience. This suggests that it is possible to achieve native-like pre-attentive processing of lexical stress, even in a language that one has not learned since birth.

Disrupted or delayed? Stress discrimination among preterm as compared to full-term infants during the first year of life

BACKGROUND

Several papers pointed out that the language disorders are the most commonly reported cognitive deficits of the preterm (PT) infants. However, neither the exact interpretation (disrupted or delayed) of their language development nor the most powerful perinatal risk factors have been specified yet.

AIMS

We aimed to determine whether postnatal development of prosodic processing of PT infants is disrupted or delayed in the first year of life? We also tested the role of Birth Weight (BW) and Gestational Age (GA) regarding the PT and full-term (FT) infants' language perception.

METHOD

We registered the mismatch responses (MMR) of 34 PT (at 6 and 12 month of ages) and 33 FT infants (at 4 and 10 month of age) elicited by bisyllabic pseudo-words in two oddball conditions.

RESULT

Contrary to their FT peers, younger PT group detected stress changes of the legal stress form only. Analogously, a positive MMR (P-MMR) was found for the legal form discrimination exclusively in PT12 group. Furthermore, the lack of sensitivity to the standard vs. deviant difference was identified in the PT infants. In PT infants, BW explained 21 % of the total variance of the P-MMR.

CONCLUSION

Consequently, we argue that the stress sensitivity of the PT infants is unimpaired, but their stress processing seems to be disrupted from the 6th month on. We suggest for further studies to take BW into account in studies using MMR paradigms in PT infants.

ERP mismatch response to phonological and temporal regularities in speech

Predictions of our sensory environment facilitate perception across domains. During speech perception, formal and temporal predictions may be made for phonotactic probability and syllable stress patterns, respectively, contributing to the efficient processing of speech input. The current experiment employed a passive EEG oddball paradigm to probe the neurophysiological processes underlying temporal and formal predictions simultaneously. The component of interest, the mismatch negativity (MMN), is considered a marker for experience-dependent change detection, where its timing and amplitude are indicative of the perceptual system’s sensitivity to presented stimuli. We hypothesized that more predictable stimuli (i.e. high phonotactic probability and first syllable stress) would facilitate change detection, indexed by shorter peak latencies or greater peak amplitudes of the MMN. This hypothesis was confirmed for phonotactic probability: high phonotactic probability deviants elicited an earlier MMN than low phonotactic probability deviants. We do not observe a significant modulation of the MMN to variations in syllable stress. Our findings confirm that speech perception is shaped by formal and temporal predictability. This paradigm may be useful to investigate the contribution of implicit processing of statistical regularities during (a)typical language development.

Expectations about word stress modulate neural activity in speech-sensitive cortical areas

A recent dual-stream model of language processing proposed that the postero-dorsal stream performs predictive sequential processing of linguistic information via hierarchically organized internal models. However, it remains unexplored whether the prosodic segmentation of linguistic information involves predictive processes. Here, we addressed this question by investigating the processing of word stress, a major component of speech segmentation, using probabilistic repetition suppression (RS) modulation as a marker of predictive processing. In an event-related acoustic fMRI RS paradigm, we presented pairs of pseudowords having the same (Rep) or different (Alt) stress patterns, in blocks with varying Rep and Alt trial probabilities. We found that the BOLD signal was significantly lower for Rep than for Alt trials, indicating RS in the posterior and middle superior temporal gyrus (STG) bilaterally, and in the anterior STG in the left hemisphere. Importantly, the magnitude of RS was modulated by repetition probability in the posterior and middle STG. These results reveal the predictive processing of word stress in the STG areas and raise the possibility that words stress processing is related to the dorsal "where" auditory stream.

Electrophysiological Differentiation of the Effects of Stress and Accent on Lexical Integration in Highly Fluent Bilinguals

Individuals who acquire a second language (L2) after infancy often retain features of their native language (L1) accent. Cross-language priming studies have shown negative effects of L1 accent on L2 comprehension, but the role of specific speech features, such as lexical stress, is mostly unknown. Here, we investigate whether lexical stress and accent differently modulate semantic processing and cross-language lexical activation in Welsh–English bilinguals, given that English and Welsh differ substantially in terms of stress realisation. In an L2 cross-modal priming paradigm, we manipulated the stress pattern and accent of spoken primes, whilst participants made semantic relatedness judgments on visual word targets. Event-related brain potentials revealed a main effect of stress on target integration, such that stimuli with stress patterns compatible with either the L1 or L2 required less processing effort than stimuli with stress incompatible with both Welsh and English. An independent cross-language phonological overlap manipulation revealed an interaction between accent and L1 access. Interestingly, although it increased processing effort, incorrect stress did not significantly modulate semantic priming effects or covert access to L1 phonological representations. Our results are consistent with the concept of language-specific stress templates, and suggest that accent and lexical stress affect speech comprehension mechanisms differentially.

Does intra-uterine language experience modulate word stress processing? An ERP study

BACKGROUND

Preterm birth is associated with various risks, including delayed or atypical language development. The prenatal start of prosodic tuning may affect the processing of word stress, an important suprasegmental feature of spoken utterances.

AIM

Our study focused on the expected contribution of intra-uterine experience to word stress processing. We aimed to demonstrate the hypothesized effect of intra-uterine sound exposition on stress sensitivity.

METHOD

We recorded ERP responses of 34 preterm infants elicited by bisyllabic pseudo-words in two oddball conditions by switching the stress pattern (legal vs. illegal) and role (standard vs. deviant).

RESULTS

The mismatch responses found were synchronized to each syllable of the illegally stressed stimuli with no difference between pre- and full-term infants. However, the clear role of the preterm status was demonstrated by the exaggerated processing of the native stress information. The impact of intra-uterine exposure to prosody was confirmed by our finding that moderate-late preterm infants outperformed the very preterm ones.

CONCLUSION

Intra-uterine exposition to prosodic features appears to contribute to the emergence of stable long-term stress representation. When this tuning is missing it is considered a risk for the language acquisition process.

Intonation and talker variability in the discrimination of Spanish lexical stress contrasts by Spanish, German and French listeners

The perception of stress is highly influenced by listeners' native language. In this research, the authors examined the effect of intonation and talker variability (here: phonetic variability) in the discrimination of Spanish lexical stress contrasts by native Spanish (N = 17), German (N = 21), and French (N = 27) listeners. Participants listened to 216 trials containing three Spanish disyllabic words, where one word carried a different lexical stress to the others. The listeners' task was to identify the deviant word in each trial (Odd-One-Out task). The words in the trials were produced by either the same talker or by two different talkers, and carried the same or varying intonation patterns. The German listeners' performance was lower compared to the Spanish listeners but higher than that of the French listeners. French listeners performed above chance level with and without talker variability, and performed at chance level when intonation variability was introduced. Results are discussed in the context of the stress "deafness" hypothesis.

Statistical learning leads to persistent memory: Evidence for one-year consolidation

Statistical learning is a robust mechanism of the brain that enables the extraction of environmental patterns, which is crucial in perceptual and cognitive domains. However, the dynamical change of processes underlying long-term statistical memory formation has not been tested in an appropriately controlled design. Here we show that a memory trace acquired by statistical learning is resistant to inference as well as to forgetting after one year. Participants performed a statistical learning task and were retested one year later without further practice. The acquired statistical knowledge was resistant to interference, since after one year, participants showed similar memory performance on the previously practiced statistical structure after being tested with a new statistical structure. These results could be key to understand the stability of long-term statistical knowledge.

Lexical influence on stress processing in a fixed-stress language

In the present study, we investigate how lexicality affects the processing of suprasegmental features at the word level. In contrast to earlier studies which analyzed the role of either segmental or suprasegmental feature in language processing our aim was to investigate the effect of the lexical status on the processing of violated stress pattern defined by linguistic rules. We have conducted a passive oddball ERP experiment, presenting a frequent CVCV word with legal (familiar) and illegal (unfamiliar) stress patterns. Former results obtained with pseudo-words in a similar paradigm enabled to assess the influence of lexical information on stress processing. The presence of lexically relevant information resulted in different ERP patterns compared to those obtained with pseudo-words. We obtained two consecutive MMN responses to the illegally stressed words while violating the illegal stress pattern with a legal one the deviant stimulus elicited two consecutive MMN responses as well. In the latter condition lexicality clearly enhanced the comparison of prosodic information between standard and deviant stimuli, as these components very completely missing when presenting pseudo-words. We interpret the results that lexicality acts as a filter since in the absence of lexical familiarity unfamiliar stress patterns are discriminated better. Our results highlight that even when stress is fully predictable, it is taken into account during pre-attentive processing of linguistic input.

Cortical encoding and neurophysiological tracking of intensity and pitch cues signaling English stress patterns in native and nonnative speakers

We examined cross-language differences in neural encoding and tracking of intensity and pitch cues signaling English stress patterns. Auditory mismatch negativities (MMNs) were recorded in English and Mandarin listeners in response to contrastive English pseudowords whose primary stress occurred either on the first or second syllable (i.e., "nocTICity" vs. "NOCticity"). The contrastive syllable stress elicited two consecutive MMNs in both language groups, but English speakers demonstrated larger responses to stress patterns than Mandarin speakers. Correlations between the amplitude of ERPs and continuous changes in the running intensity and pitch of speech assessed how well each language group's brain activity tracked these salient acoustic features of lexical stress. We found that English speakers' neural responses tracked intensity changes in speech more closely than Mandarin speakers (higher brain-acoustic correlation). Findings demonstrate more robust and precise processing of English stress (intensity) patterns in early auditory cortical responses of native relative to nonnative speakers.