The phonemic restoration effect reveals pre-N400 effect of supportive sentence context in speech perception

Conventionality matters in Chinese metaphor but not simile comprehension: evidence from event-related potentials

Metaphor and simile, two prevalent forms of figurative language widely employed in daily communication, serve as significant research subjects in linguistics. The Career of Metaphor Theory in cognitive linguistics posits that as conventionality increases, the cognitive mechanisms of metaphor comprehension shift from "comparison" to "categorization." In line with this notion, prior electrophysiological investigations have revealed that novel metaphors elicit a stronger N400 brain response compared to conventional metaphors. However, the observed N400 difference between conventional and novel metaphors may merely stem from the familiarity contrast between them, as conventional metaphors are typically more familiar than novel ones. To address this dichotomy, the present study not only compared the N400 responses between conventional and novel metaphors but also between conventional and novel similes. While conventional and novel similes differ in familiarity, similar to conventional and novel metaphors, both are processed via "comparison" mechanisms. The results revealed that novel metaphors elicited larger N400 amplitudes compared to conventional metaphors, aligning with previous findings. In contrast, no significant N400 differences were observed between conventional and novel similes, suggesting that familiarity disparity is unlikely to account for N400 distinctions. Our findings imply that conventional and novel metaphors undergo distinct cognitive processing mechanisms ("comparison" versus "categorization"), thereby providing further empirical validation for the Career of Metaphor Theory.

Dissociating the pre-activation of word meaning and form during sentence comprehension: Evidence from EEG representational similarity analysis

During language comprehension, the processing of each incoming word is facilitated in proportion to its predictability. Here, we asked whether anticipated upcoming linguistic information is actually pre-activated before new bottom-up input becomes available, and if so, whether this pre-activation is limited to the level of semantic features, or whether extends to representations of individual word-forms (orthography/phonology). We carried out Representational Similarity Analysis on EEG data while participants read highly constraining sentences. Prior to the onset of the expected target words, sentence pairs predicting semantically related words (financial "bank" - "loan") and form-related words (financial "bank" - river "bank") produced more similar neural patterns than pairs predicting unrelated words ("bank" - "lesson"). This provides direct neural evidence for item-specific semantic and form predictive pre-activation. Moreover, the semantic pre-activation effect preceded the form pre-activation effect, suggesting that top-down pre-activation is propagated from higher to lower levels of the linguistic hierarchy over time.

Can You Hear What's Coming? Failure to Replicate ERP Evidence for Phonological Prediction

Prediction-based theories of language comprehension assume that listeners predict both the meaning and phonological form of likely upcoming words. In alleged event-related potential (ERP) demonstrations of phonological prediction, prediction-mismatching words elicit a phonological mismatch negativity (PMN), a frontocentral negativity that precedes the centroparietal N400 component. However, classification and replicability of the PMN has proven controversial, with ongoing debate on whether the PMN is a distinct component or merely an early part of the N400. In this electroencephalography (EEG) study, we therefore attempted to replicate the PMN effect and its separability from the N400, using a participant sample size (N = 48) that was more than double that of previous studies. Participants listened to sentences containing either a predictable word or an unpredictable word with/without phonological overlap with the predictable word. Preregistered analyses revealed a widely distributed negative-going ERP in response to unpredictable words in both the early (150-250 ms) and the N400 (300-500 ms) time windows. Bayes factor analysis yielded moderate evidence against a different scalp distribution of the effects in the two time windows. Although our findings do not speak against phonological prediction during sentence comprehension, they do speak against the PMN effect specifically as a marker of phonological prediction mismatch. Instead of an PMN effect, our results demonstrate the early onset of the auditory N400 effect associated with unpredictable words. Our failure to replicate further highlights the risk associated with commonly employed data-contingent analyses (e.g., analyses involving time windows or electrodes that were selected based on visual inspection) and small sample sizes in the cognitive neuroscience of language.

(Not so) Great Expectations: Listening to Foreign-Accented Speech Reduces the Brain's Anticipatory Processes

This study examines the effect of foreign-accented speech on the predictive ability of our brain. Listeners actively anticipate upcoming linguistic information in the speech signal so as to facilitate and reduce processing load. However, it is unclear whether or not listeners also do this when they are exposed to speech from non-native speakers. In the present study, we exposed native Dutch listeners to sentences produced by native and non-native speakers while measuring their brain activity using electroencephalography. We found that listeners’ brain activity differed depending on whether they listened to native or non-native speech. However, participants’ overall performance as measured by word recall rate was unaffected. We discussed the results in relation to previous findings as well as the automaticity of anticipation.

Context Effect on L2 Word Recognition: Visual Versus Auditory Modalities

This study tested the effect of constraining sentence context on word recognition time (RT) in the first and second language. Native (L1) and nonnative (L2) speakers of English performed self-paced reading and listening tasks to see whether a semantically-rich preceding context would lead to the activation of a probable upcoming word prior to encountering it. The pre-access prediction model (e.g., Altmann and Kamide in Cognition 73(3):247-264, 1999; McClelland and Rumelhart in Psychol Rev 88:375-407, 1981) posits that when the preceding context is semantically high-constraining, the perceptual system anticipates a probable upcoming word prior to encountering it. In contrast, the post-access model (e.g., Fodor in The modularity of mind, MIT Press, Cambridge, 1983; Forster in Q J Exp Psychol Sect A 33(4):465-495, 1981; Traxler et al. in J Psycholinguist Res 29(6):581-595, 2000; Van Petten and Luka in Int J Psychophysiol 83(2): 176-190, 2012) suggests that it is only after a word is encountered that a subsequent process integrates it into the preceding context. The integration process is easier and faster when the word is more congruent with the preceding context. In line with these two models of visual word recognition, auditory word recognition is modeled by the TRACE model (McClelland and Elman in Cogn Psychol 18(1):1-86, 1986), which suggests that a spoken word is influenced by the preceding sentence context, whereas models such as the cohort model (Marslen-Wilson in Cognition 25(1-2):71-102, 1987) support a post-access integration process. Some studies observe a facilitative effect of sentence context on L2 word recognition (e.g., Kamide et al. in J Mem Lang 49(1):133-156, 2003), while others find no effect of sentence context in L2 (e.g., Ito et al. in Lang Cogn Neurosci 32(8):954-965, 2017; Martin et al. in J Mem Lang 69(4):574-588, 2013). In the present experiments, the RTs of native English speakers (L1) and non-native English speakers (L2) were collected in both visual and auditory word recognition in semantically high-constraint sentences and semantically low-constraint sentences. A linear, mixed-effects model shows that both groups of participants are faster to recognize a word when it is preceded by a semantically high-constraining context. This result is observed in both the visual and the auditory modalities, lending some support for a mechanism facilitating the access of target words based on sentence context in both L1 and L2.

Neural Signal to Violations of Abstract Rules Using Speech-Like Stimuli

As the evidence of predictive processes playing a role in a wide variety of cognitive domains increases, the brain as a predictive machine becomes a central idea in neuroscience. In auditory processing, a considerable amount of progress has been made using variations of the Oddball design, but most of the existing work seems restricted to predictions based on physical features or conditional rules linking successive stimuli. To characterize the predictive capacity of the brain to abstract rules, we present here two experiments that use speech-like stimuli to overcome limitations and avoid common confounds. Pseudowords were presented in isolation, intermixed with infrequent deviants that contained unexpected phoneme sequences. As hypothesized, the occurrence of unexpected sequences of phonemes reliably elicited an early prediction error signal. These prediction error signals do not seemed to be modulated by attentional manipulations due to different task instructions, suggesting that the predictions are deployed even when the task at hand does not volitionally involve error detection. In contrast, the amount of syllables congruent with a standard pseudoword presented before the point of deviance exerted a strong modulation. Prediction error's amplitude doubled when two congruent syllables were presented instead of one, despite keeping local transitional probabilities constant. This suggests that auditory predictions can be built integrating information beyond the immediate past. In sum, the results presented here further contribute to the understanding of the predictive capabilities of the human auditory system when facing complex stimuli and abstract rules.

Effects of Task Difficulty on Neural Processes Underlying Semantics: An Event-Related Potentials Study

Purpose Improving the ability to listen efficiently in noisy environments is a critical goal for hearing rehabilitation. However, understanding of the impact of difficult listening conditions on language processing is limited. The current study evaluated the neural processes underlying semantics in challenging listening conditions. Method Thirty adults with normal hearing completed an auditory sentence processing task in 4-talker babble. Event-related brain potentials were elicited by the final word in high- or low-context sentences, where the final word was either highly expected or not expected, followed by a 4-alternative forced-choice response with either longer (1,000 ms), middle (700 ms), or shorter (400 ms) response time deadlines (RTDs). Results Behavioral accuracy was reduced, and reactions times were faster for shorter RTDs. N400 amplitudes, reflecting ease of lexical access, were larger when elicited by target words in low-context sentences followed by shorter compared with longer RTDs. Conclusions These results reveal that more neural resources are allocated for semantic processing/lexical access when listening difficulty increases. Differences between RTDs may reflect increased attentional allocation for shorter RTDs. These findings suggest that situational listening demands can impact the demands for cognitive resources engaged in language processing, which could significantly impact listener experiences across environments.

The role of the root in auditory word recognition of Hebrew

Evidence from visual word recognition has shown that the root morpheme plays a particularly important role in recognition of nouns in templatic languages [e.g., Velan & Frost, 2009 (Hebrew), Perea, abu Mallouh, & Carreiras, 2010 (Arabic)]. Letter transposition studies in masked priming have proved a useful tool for investigating letter flexibility in the visual domain. Due to the linear nature of the auditory signal, such manipulation is not possible for spoken words. In this study, we use a novel application of the phonemic restoration paradigm to explore the role of morphology in auditory word recognition. In two separate experiments, we show that in auditory word recognition the root plays an important role in Hebrew noun recognition, with words with masked root sounds being especially difficult to recover. This study provides additional evidence in favor of the privileged role of the root in Semitic lexical access and its function in morphological decomposition.

Speech rhythm alterations in Spanish-speaking individuals with Alzheimer's disease

Rhythm is the speech property related to the temporal organization of sounds. Considerable evidence is now available for suggesting that dementia of Alzheimer's type is associated with impairments in speech rhythm. The aim of this study is to assess the use of an automatic computerized system for measuring speech rhythm characteristics in an oral reading task performed by 45 patients with Alzheimer's disease (AD) compared with those same characteristics among 82 healthy older adults without a diagnosis of dementia, and matched by age, sex and cultural background. Ranges of rhythmic-metric and clinical measurements were applied. The results show rhythmic differences between the groups, with higher variability of syllabic intervals in AD patients. Signal processing algorithms applied to oral reading recordings prove to be capable of differentiating between AD patients and older adults without dementia with an accuracy of 87% (specificity 81.7%, sensitivity 82.2%), based on the standard deviation of the duration of syllabic intervals. Experimental results show that the syllabic variability measurements extracted from the speech signal can be used to distinguish between older adults without a diagnosis of dementia and those with AD, and may be useful as a tool for the objective study and quantification of speech deficits in AD.

What do we mean by prediction in language comprehension?

We consider several key aspects of prediction in language comprehension: its computational nature, the representational level(s) at which we predict, whether we use higher level representations to predictively pre-activate lower level representations, and whether we 'commit' in any way to our predictions, beyond pre-activation. We argue that the bulk of behavioral and neural evidence suggests that we predict probabilistically and at multiple levels and grains of representation. We also argue that we can, in principle, use higher level inferences to predictively pre-activate information at multiple lower representational levels. We also suggest that the degree and level of predictive pre-activation might be a function of the expected utility of prediction, which, in turn, may depend on comprehenders' goals and their estimates of the relative reliability of their prior knowledge and the bottom-up input. Finally, we argue that all these properties of language understanding can be naturally explained and productively explored within a multi-representational hierarchical actively generative architecture whose goal is to infer the message intended by the producer, and in which predictions play a crucial role in explaining the bottom-up input.

Sentential influences on acoustic-phonetic processing: A Granger causality analysis of multimodal imaging data

Sentential context influences the way that listeners identify phonetically ambiguous or perceptual degraded speech sounds. Unfortunately, inherent inferential limitations on the interpretation of behavioral or BOLD imaging results make it unclear whether context influences perceptual processing directly, or acts at a post-perceptual decision stage. In this paper, we use Kalman-filter enabled Granger causation analysis of MR-constrained MEG/EEG data to distinguish between these possibilities. Using a retrospective probe verification task, we found that sentential context strongly affected the interpretation of words with ambiguous initial voicing (e.g. DUSK-TUSK). This behavioral context effect coincided with increased influence by brain regions associated with lexical representation on regions associated with acoustic-phonetic processing. These results support an interactive view of sentence context effects on speech perception.

Regression-based estimation of ERP waveforms: I. The rERP framework

ERP averaging is an extraordinarily successful method, but can only be applied to a limited range of experimental designs. We introduce the regression-based rERP framework, which extends ERP averaging to handle arbitrary combinations of categorical and continuous covariates, partial confounding, nonlinear effects, and overlapping responses to distinct events, all within a single unified system. rERPs enable a richer variety of paradigms (including high-N naturalistic designs) while preserving the advantages of traditional ERPs. This article provides an accessible introduction to what rERPs are, why they are useful, how they are computed, and when we should expect them to be effective, particularly in cases of partial confounding. A companion article discusses how nonlinear effects and overlap correction can be handled within this framework, as well as practical considerations around baselining, filtering, statistical testing, and artifact rejection. Free software implementing these techniques is available.

Perceptual restoration of degraded speech is preserved with advancing age

Cognitive skills, such as processing speed, memory functioning, and the ability to divide attention, are known to diminish with aging. The present study shows that, despite these changes, older adults can successfully compensate for degradations in speech perception. Critically, the older participants of this study were not pre-selected for high performance on cognitive tasks, but only screened for normal hearing. We measured the compensation for speech degradation using phonemic restoration, where intelligibility of degraded speech is enhanced using top-down repair mechanisms. Linguistic knowledge, Gestalt principles of perception, and expectations based on situational and linguistic context are used to effectively fill in the inaudible masked speech portions. A positive compensation effect was previously observed only with young normal hearing people, but not with older hearing-impaired populations, leaving the question whether the lack of compensation was due to aging or due to age-related hearing problems. Older participants in the present study showed poorer intelligibility of degraded speech than the younger group, as expected from previous reports of aging effects. However, in conditions that induce top-down restoration, a robust compensation was observed. Speech perception by the older group was enhanced, and the enhancement effect was similar to that observed with the younger group. This effect was even stronger with slowed-down speech, which gives more time for cognitive processing. Based on previous research, the likely explanations for these observations are that older adults can overcome age-related cognitive deterioration by relying on linguistic skills and vocabulary that they have accumulated over their lifetime. Alternatively, or simultaneously, they may use different cerebral activation patterns or exert more mental effort. This positive finding on top-down restoration skills by the older individuals suggests that new cognitive training methods can teach older adults to effectively use compensatory mechanisms to cope with the complex listening environments of everyday life.

Prediction in the service of comprehension: modulated early brain responses to omitted speech segments

Speech signals are often compromised by disruptions originating from external (e.g., masking noise) or internal (e.g., inaccurate articulation) sources. Speech comprehension thus entails detecting and replacing missing information based on predictive and restorative neural mechanisms. The present study targets predictive mechanisms by investigating the influence of a speech segment's predictability on early, modality-specific electrophysiological responses to this segment's omission. Predictability was manipulated in simple physical terms in a single-word framework (Experiment 1) or in more complex semantic terms in a sentence framework (Experiment 2). In both experiments, final consonants of the German words Lachs ([laks], salmon) or Latz ([lats], bib) were occasionally omitted, resulting in the syllable La ([la], no semantic meaning), while brain responses were measured with multi-channel electroencephalography (EEG). In both experiments, the occasional presentation of the fragment La elicited a larger omission response when the final speech segment had been predictable. The omission response occurred ∼125-165 msec after the expected onset of the final segment and showed characteristics of the omission mismatch negativity (MMN), with generators in auditory cortical areas. Suggestive of a general auditory predictive mechanism at work, this main observation was robust against varying source of predictive information or attentional allocation, differing between the two experiments. Source localization further suggested the omission response enhancement by predictability to emerge from left superior temporal gyrus and left angular gyrus in both experiments, with additional experiment-specific contributions. These results are consistent with the existence of predictive coding mechanisms in the central auditory system, and suggestive of the general predictive properties of the auditory system to support spoken word recognition.

Dissociating N400 effects of prediction from association in single-word contexts

When a word is preceded by a supportive context such as a semantically associated word or a strongly constraining sentence frame, the N400 component of the ERP is reduced in amplitude. An ongoing debate is the degree to which this reduction reflects a passive spread of activation across long-term semantic memory representations as opposed to specific predictions about upcoming input. We addressed this question by embedding semantically associated prime-target pairs within an experimental context that encouraged prediction to a greater or lesser degree. The proportion of related items was used to manipulate the predictive validity of the prime for the target while holding semantic association constant. A semantic category probe detection task was used to encourage semantic processing and to preclude the need for a motor response on the trials of interest. A larger N400 reduction to associated targets was observed in the high than the low relatedness proportion condition, consistent with the hypothesis that predictions about upcoming stimuli make a substantial contribution to the N400 effect. We also observed an earlier priming effect (205-240 msec) in the high-proportion condition, which may reflect facilitation because of form-based prediction. In summary, the results suggest that predictability modulates N400 amplitude to a greater degree than the semantic content of the context.

Advanced EEG analysis using threshold-free cluster-enhancement and non-parametric statistics

Advances in EEG signal analysis and its combination with other investigative techniques make appropriate statistical analysis of large EEG datasets a crucial issue. With an increasing number of available channels and samples, as well as more exploratory experimental designs, it has become necessary to develop a statistical process with a high level of statistical integrity, signal sensitivity which nonetheless produces results which are interpretable to the common user. Threshold-free cluster-enhancement has recently been proposed as a useful analysis tool for fMRI datasets. This approach essentially takes into account both a data point's statistical intensity and neighbourhood to transform the original signal into a more intuitive understanding of 'real' differences between groups or conditions. Here we adapt this approach to optimally deal with EEG datasets and use permutation-based statistics to build an efficient statistical analysis. Furthermore we compare the results with several other non-parametric and parametric approaches currently available using realistic simulated EEG signals. The proposed method is shown to be generally more sensitive to the variety of signal types common to EEG datasets without the need for any arbitrary adjusting of parameters. Moreover, a unique p-value is produced for each channel-sample pair such that specific questions can still be asked of the dataset while providing general information regarding the large-scale experimental effects.

Neural restoration of degraded audiovisual speech

When speech is interrupted by noise, listeners often perceptually "fill-in" the degraded signal, giving an illusion of continuity and improving intelligibility. This phenomenon involves a neural process in which the auditory cortex (AC) response to onsets and offsets of acoustic interruptions is suppressed. Since meaningful visual cues behaviorally enhance this illusory filling-in, we hypothesized that during the illusion, lip movements congruent with acoustic speech should elicit a weaker AC response to interruptions relative to static (no movements) or incongruent visual speech. AC response to interruptions was measured as the power and inter-trial phase consistency of the auditory evoked theta band (4-8 Hz) activity of the electroencephalogram (EEG) and the N1 and P2 auditory evoked potentials (AEPs). A reduction in the N1 and P2 amplitudes and in theta phase-consistency reflected the perceptual illusion at the onset and/or offset of interruptions regardless of visual condition. These results suggest that the brain engages filling-in mechanisms throughout the interruption, which repairs degraded speech lasting up to ~250 ms following the onset of the degradation. Behaviorally, participants perceived speech continuity over longer interruptions for congruent compared to incongruent or static audiovisual streams. However, this specific behavioral profile was not mirrored in the neural markers of interest. We conclude that lip-reading enhances illusory perception of degraded speech not by altering the quality of the AC response, but by delaying it during degradations so that longer interruptions can be tolerated.

Mass univariate analysis of event-related brain potentials/fields I: a critical tutorial review

Event-related potentials (ERPs) and magnetic fields (ERFs) are typically analyzed via ANOVAs on mean activity in a priori windows. Advances in computing power and statistics have produced an alternative, mass univariate analyses consisting of thousands of statistical tests and powerful corrections for multiple comparisons. Such analyses are most useful when one has little a priori knowledge of effect locations or latencies, and for delineating effect boundaries. Mass univariate analyses complement and, at times, obviate traditional analyses. Here we review this approach as applied to ERP/ERF data and four methods for multiple comparison correction: strong control of the familywise error rate (FWER) via permutation tests, weak control of FWER via cluster-based permutation tests, false discovery rate control, and control of the generalized FWER. We end with recommendations for their use and introduce free MATLAB software for their implementation.

Mass univariate analysis of event-related brain potentials/fields II: Simulation studies

Mass univariate analysis is a relatively new approach for the study of ERPs/ERFs. It consists of many statistical tests and one of several powerful corrections for multiple comparisons. Multiple comparison corrections differ in their power and permissiveness. Moreover, some methods are not guaranteed to work or may be overly sensitive to uninteresting deviations from the null hypothesis. Here we report the results of simulations assessing the accuracy, permissiveness, and power of six popular multiple comparison corrections (permutation-based control of the familywise error rate [FWER], weak control of FWER via cluster-based permutation tests, permutation-based control of the generalized FWER, and three false discovery rate control procedures) using realistic ERP data. In addition, we look at the sensitivity of permutation tests to differences in population variance. These results will help researchers apply and interpret these procedures.