The influence of increased working memory load on semantic neural systems: a high-resolution event-related brain potential study

Tracking Cognitive Spare Capacity During Speech Perception With EEG/ERP: Effects of Cognitive Load and Sentence Predictability

OBJECTIVES

Listening to speech in adverse listening conditions is effortful. Objective assessment of cognitive spare capacity during listening can serve as an index of the effort needed to understand speech. Cognitive spare capacity is influenced both by signal-driven demands posed by listening conditions and top-down demands intrinsic to spoken language processing, such as memory use and semantic processing. Previous research indicates that electrophysiological responses, particularly alpha oscillatory power, may index listening effort. However, it is not known how these indices respond to memory and semantic processing demands during spoken language processing in adverse listening conditions. The aim of the present study was twofold: first, to assess the impact of memory demands on electrophysiological responses during recognition of degraded, spoken sentences, and second, to examine whether predictable sentence contexts increase or decrease cognitive spare capacity during listening.

DESIGN

Cognitive demand was varied in a memory load task in which young adult participants (n = 20) viewed either low-load (one digit) or high-load (seven digits) sequences of digits, then listened to noise-vocoded spoken sentences that were either predictable or unpredictable, and then reported the final word of the sentence and the digits. Alpha oscillations in the frequency domain and event-related potentials in the time domain of the electrophysiological data were analyzed, as was behavioral accuracy for both words and digits.

RESULTS

Measured during sentence processing, event-related desynchronization of alpha power was greater (more negative) under high load than low load and was also greater for unpredictable than predictable sentences. A complementary pattern was observed for the P300/late positive complex (LPC) to sentence-final words, such that P300/LPC amplitude was reduced under high load compared with low load and for unpredictable compared with predictable sentences. Both words and digits were identified more quickly and accurately on trials in which spoken sentences were predictable.

CONCLUSIONS

Results indicate that during a sentence-recognition task, both cognitive load and sentence predictability modulate electrophysiological indices of cognitive spare capacity, namely alpha oscillatory power and P300/LPC amplitude. Both electrophysiological and behavioral results indicate that a predictive sentence context reduces cognitive demands during listening. Findings contribute to a growing literature on objective measures of cognitive demand during listening and indicate predictable sentence context as a top-down factor that can support ease of listening.

Verbal working memory and co-speech gesture processing

Multimodal discourse requires an assembly of cognitive processes that are uniquely recruited for language comprehension in social contexts. In this study, we investigated the role of verbal working memory for the online integration of speech and iconic gestures. Participants memorized and rehearsed a series of auditorily presented digits in low (one digit) or high (four digits) memory load conditions. To observe how verbal working memory load impacts online discourse comprehension, ERPs were recorded while participants watched discourse videos containing either congruent or incongruent speech-gesture combinations during the maintenance portion of the memory task. While expected speech-gesture congruity effects were found in the low memory load condition, high memory load trials elicited enhanced frontal positivities that indicated a unique interaction between online speech-gesture integration and the availability of verbal working memory resources. This work contributes to an understanding of discourse comprehension by demonstrating that language processing in a multimodal context is subject to the relationship between cognitive resource availability and the degree of controlled processing required for task performance. We suggest that verbal working memory is less important for speech-gesture integration than it is for mediating speech processing under high task demands.

Brain Vital Signs Detect Information Processing Differences When Neuromodulation Is Used During Cognitive Skills Training

Background: Neuromodulation through translingual neurostimulation (TLNS) has been shown to initiate long-lasting processes of neuronal reorganization with a variety of outcomes (i.e., neuroplasticity). Non-invasive TLNS is increasingly accessible through the Portable Neuromodulation Stimulator (PoNS^®), a medical device that delivers electrical stimulation to the tongue to activate the trigeminal (V) and facial (VII) cranial nerves. Anecdotal reports from previous clinical studies have suggested incidental improvements in cognitive function. To objectively explore this observation, we examined TLNS-related effects on the semantic N400 brain vital sign cognitive response during cognitive skills training in healthy individuals. Methods: Thirty-seven healthy volunteers were randomized to receive simultaneous TLNS (treatment) or no TLNS (control) while undergoing cognitive skills training. Cognitive training was conducted for two 20-min sessions (morning and afternoon/evening) over 3 consecutive days. Brain vital signs were evaluated at baseline, Day 1, and Day 3. Analyses focused on cognitive processing as measured by N400 changes in amplitude and latency. Results: Over the 3-day course of cognitive training, the N400 amplitude decreased significantly in the control group due to habituation (p = 0.028). In contrast, there was no significant change in the TLNS treatment group. Conclusion: TLNS led to a sustained N400 response during cognitive skills training, as measured by the brain's vital signs framework. The study findings suggest differential learning effects due to neuromodulation, consistent with increased attention and cognitive vigilance.

Brain Vital Signs: Expanding From the Auditory to Visual Modality

The critical need for rapid objective, physiological evaluation of brain function at point-of-care has led to the emergence of brain vital signs-a framework encompassing a portable electroencephalography (EEG) and an automated, quick test protocol. This framework enables access to well-established event-related potential (ERP) markers, which are specific to sensory, attention, and cognitive functions in both healthy and patient populations. However, all our applications to-date have used auditory stimulation, which have highlighted application challenges in persons with hearing impairments (e.g., aging, seniors, dementia). Consequently, it has become important to translate brain vital signs into a visual sensory modality. Therefore, the objectives of this study were to: 1) demonstrate the feasibility of visual brain vital signs; and 2) compare and normalize results from visual and auditory brain vital signs. Data were collected from 34 healthy adults (33 ± 13 years) using a 64-channel EEG system. Visual and auditory sequences were kept as comparable as possible to elicit the N100, P300, and N400 responses. Visual brain vital signs were elicited successfully for all three responses across the group (N100: F = 29.8380, p < 0.001; P300: F = 138.8442, p < 0.0001; N400: F = 6.8476, p = 0.01). Initial auditory-visual comparisons across the three components showed attention processing (P300) was found to be the most transferrable across modalities, with no group-level differences and correlated peak amplitudes (rho = 0.7, p = 0.0001) across individuals. Auditory P300 latencies were shorter than visual (p < 0.0001) but normalization and correlation (r = 0.5, p = 0.0033) implied a potential systematic difference across modalities. Reduced auditory N400 amplitudes compared to visual (p = 0.0061) paired with normalization and correlation across individuals (r = 0.6, p = 0.0012), also revealed potential systematic modality differences between reading and listening language comprehension. This study provides an initial understanding of the relationship between the visual and auditory sequences, while importantly establishing a visual sequence within the brain vital signs framework. With both auditory and visual stimulation capabilities available, it is possible to broaden applications across the lifespan.

Multimodal characterization of the semantic N400 response within a rapid evaluation brain vital sign framework

Background

For nearly four decades, the N400 has been an important brainwave marker of semantic processing. It can be recorded non-invasively from the scalp using electrical and/or magnetic sensors, but largely within the restricted domain of research laboratories specialized to run specific N400 experiments. However, there is increasing evidence of significant clinical utility for the N400 in neurological evaluation, particularly at the individual level. To enable clinical applications, we recently reported a rapid evaluation framework known as “brain vital signs” that successfully incorporated the N400 response as one of the core components for cognitive function evaluation. The current study characterized the rapidly evoked N400 response to demonstrate that it shares consistent features with traditional N400 responses acquired in research laboratory settings—thereby enabling its translation into brain vital signs applications.

Methods

Data were collected from 17 healthy individuals using magnetoencephalography (MEG) and electroencephalography (EEG), with analysis of sensor-level effects as well as evaluation of brain sources. Individual-level N400 responses were classified using machine learning to determine the percentage of participants in whom the response was successfully detected.

Results

The N400 response was observed in both M/EEG modalities showing significant differences to incongruent versus congruent condition in the expected time range (p < 0.05). Also as expected, N400-related brain activity was observed in the temporal and inferior frontal cortical regions, with typical left-hemispheric asymmetry. Classification robustly confirmed the N400 effect at the individual level with high accuracy (89%), sensitivity (0.88) and specificity (0.90).

Conclusion

The brain vital sign N400 characteristics were highly consistent with features of the previously reported N400 responses acquired using traditional laboratory-based experiments. These results provide important evidence supporting clinical translation of the rapidly acquired N400 response as a potential tool for assessments of higher cognitive functions.

Localized Fluctuant Oscillatory Activity by Working Memory Load: A Simultaneous EEG-fMRI Study

Working memory (WM) is a resource-limited memory system for temporary storage and processing of brain information during the execution of cognitive tasks. Increased WM load will increase the amount and difficulty of memory information. Several studies have used electroencephalography (EEG) or functional magnetic resonance imaging (fMRI) to explore load-dependent cognition processing according to the time courses of electrophysiological activity or the spatial pattern of blood oxygen metabolic activity. However, the relationships between these two activities and the underlying neural mechanism are still unclear. In this study, using simultaneously collected EEG and fMRI data under an n-back verbal WM task, we modeled the spectral perturbation of EEG oscillation and fMRI activation through joint independent component analysis (JICA). Multi-channel oscillation features were also introduced into the JICA model for further analysis. The results showed that time-locked activity of theta and beta were modulated by memory load in the early stimuli evaluation stage, corresponding to the enhanced activation in the frontal and parietal lobe, which were involved in stimulus discrimination, information encoding and delay-period activity. In the late response selection stage, alpha and gamma activity changes dependent on the load correspond to enhanced activation in the areas of frontal, temporal and parietal lobes, which played important roles in attention, information extraction and memory retention. These findings suggest that the increases in memory load not only affect the intensity and time course of the EEG activities, but also lead to the enhanced activation of brain regions which plays different roles during different time periods of cognitive process of WM.

How Distance Affects Semantic Integration in Discourse: Evidence from Event-Related Potentials

Event-related potentials were used to investigate whether semantic integration in discourse is influenced by the number of intervening sentences between the endpoints of integration. Readers read discourses in which the last sentence contained a critical word that was either congruent or incongruent with the information introduced in the first sentence. Furthermore, for the short discourses, the first and last sentence were intervened by only one sentence while for the long discourses, they were intervened by three sentences. We found that the incongruent words elicited an N400 effect for both the short and long discourses. However, a P600 effect was only observed for the long discourses, but not for the short ones. These results suggest that although readers can successfully integrate upcoming words into the existing discourse representation, the effort required for this integration process is modulated by the number of intervening sentences. Thus, discourse distance as measured by the number of intervening sentences should be taken as an important factor for semantic integration in discourse.

Semantic Relations between Visual Objects Can Be Unconsciously Processed but Not Reported under Change Blindness

Change blindness—the failure to detect changes in visual scenes—has often been interpreted as a result of impoverished visual information encoding or as a failure to compare the prechange and postchange scene. In the present electroencephalography study, we investigated whether semantic features of prechange and postchange information are processed unconsciously, even when observers are unaware that a change has occurred. We presented scenes composed of natural objects in which one object changed from one presentation to the next. Object changes were either semantically related (e.g., rail car changed to rail) or unrelated (e.g., rail car changed to sausage). Observers were first asked to detect whether any change had occurred and then to judge the semantic relation of the two objects involved in the change. We found a semantic mismatch ERP effect, that is, a more negative-going ERP for semantically unrelated compared to related changes, originating from a cortical network including the left middle temporal gyrus and occipital cortex and resembling the N400 effect, albeit at longer latencies. Importantly, this semantic mismatch effect persisted even when observers were unaware of the change and the semantic relationship of prechange and postchange object. This finding implies that change blindness does not preclude the encoding of the prechange and postchange objects' identities and possibly even the comparison of their semantic content. Thus, change blindness cannot be interpreted as resulting from impoverished or volatile visual representations or as a failure to process the prechange and postchange object. Instead, change detection appears to be limited at a later, postperceptual stage.

Narrowed Expectancies under Degraded Speech: Revisiting the N400

Under adverse listening conditions, speech comprehension profits from the expectancies that listeners derive from the semantic context. However, the neurocognitive mechanisms of this semantic benefit are unclear: How are expectancies formed from context and adjusted as a sentence unfolds over time under various degrees of acoustic degradation? In an EEG study, we modified auditory signal degradation by applying noise-vocoding (severely degraded: four-band, moderately degraded: eight-band, and clear speech). Orthogonal to that, we manipulated the extent of expectancy: strong or weak semantic context (±con) and context-based typicality of the sentence-last word (high or low: ±typ). This allowed calculation of two distinct effects of expectancy on the N400 component of the evoked potential. The sentence-final N400 effect was taken as an index of the neural effort of automatic word-into-context integration; it varied in peak amplitude and latency with signal degradation and was not reliably observed in response to severely degraded speech. Under clear speech conditions in a strong context, typical and untypical sentence completions seemed to fulfill the neural prediction, as indicated by N400 reductions. In response to moderately degraded signal quality, however, the formed expectancies appeared more specific: Only typical (+con +typ), but not the less typical (+con −typ) context–word combinations led to a decrease in the N400 amplitude. The results show that adverse listening “narrows,” rather than broadens, the expectancies about the perceived speech signal: limiting the perceptual evidence forces the neural system to rely on signal-driven expectancies, rather than more abstract expectancies, while a sentence unfolds over time.

Control-display mapping in brain-computer interfaces

Event-related potential (ERP) based brain-computer interfaces (BCIs) employ differences in brain responses to attended and ignored stimuli. When using a tactile ERP-BCI for navigation, mapping is required between navigation directions on a visual display and unambiguously corresponding tactile stimuli (tactors) from a tactile control device: control-display mapping (CDM). We investigated the effect of congruent (both display and control horizontal or both vertical) and incongruent (vertical display, horizontal control) CDMs on task performance, the ERP and potential BCI performance. Ten participants attended to a target (determined via CDM), in a stream of sequentially vibrating tactors. We show that congruent CDM yields best task performance, enhanced the P300 and results in increased estimated BCI performance. This suggests a reduced availability of attentional resources when operating an ERP-BCI with incongruent CDM. Additionally, we found an enhanced N2 for incongruent CDM, which indicates a conflict between visual display and tactile control orientations.

PRACTITIONER SUMMARY

Incongruency in control-display mapping reduces task performance. In this study, brain responses, task and system performance are related to (in)congruent mapping of command options and the corresponding stimuli in a brain-computer interface (BCI). Directional congruency reduces task errors, increases available attentional resources, improves BCI performance and thus facilitates human-computer interaction.

Spatial MEG laterality maps for language: clinical applications in epilepsy

Functional imaging is increasingly being used to provide a noninvasive alternative to intracarotid sodium amobarbitol testing (i.e., the Wada test). Although magnetoencephalography (MEG) has shown significant potential in this regard, the resultant output is often reduced to a simplified estimate of laterality. Such estimates belie the richness of functional imaging data and consequently limit the potential value. We present a novel approach that utilizes MEG data to compute "complex laterality vectors" and consequently "laterality maps" for a given function. Language function was examined in healthy controls and in people with epilepsy. When compared with traditional laterality index (LI) approaches, the resultant maps provided critical information about the magnitude and spatial characteristics of lateralized function. Specifically, it was possible to more clearly define low LI scores resulting from strong bilateral activation, high LI scores resulting from weak unilateral activation, and most importantly, the spatial distribution of lateralized activation. We argue that the laterality concept is better presented with the inherent spatial sensitivity of activation maps, rather than being collapsed into a one-dimensional index.

Event-related potentials in clinical research: guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400

This paper describes recommended methods for the use of event-related brain potentials (ERPs) in clinical research and reviews applications to a variety of psychiatric and neurological disorders. Techniques are presented for eliciting, recording, and quantifying three major cognitive components with confirmed clinical utility: mismatch negativity (MMN), P300, and N400. Also highlighted are applications of each of the components as methods of investigating central nervous system pathology. The guidelines are intended to assist investigators who use ERPs in clinical research, in an effort to provide clear and concise recommendations and thereby to standardize methodology and facilitate comparability of data across laboratories.

The cognitive consequences of emotion regulation: an ERP investigation

Increasing evidence suggests that emotion regulation (ER) strategies modulate encoding of information presented during regulation; however, no studies have assessed the impact of cognitive reappraisal ER strategies on the processing of stimuli presented after the ER period. Participants in the present study regulated emotions to unpleasant pictures and then judged whether a word was negative or neutral. Electromyographic measures (corrugator supercilli) confirmed that individuals increased and decreased negative affect according to ER condition. Event-related potential analyses revealed smallest N400 amplitudes to negative and neutral words presented after decreasing unpleasant emotions and smallest P300 amplitudes to words presented after increasing unpleasant emotions whereas reaction time data failed to show ER modulations. Results are discussed in the context of the developing ER literature, as well as theories of emotional incongruity (N400) and resource allocation (P300).

Event-related brain potentials in reading disabled children during an inverse serial digit detection task

It has been reported that limitations in different components of working memory could underlie reading disabilities. In addition, reading-disabled (RD) children seem to perform worse when digit name processing is required. With the purpose to explore further these assumptions one inverse serial digit detection task was evaluated using event-related brain potentials in fifteen 8-year-old RD children and a control group (CG). CG obtained significantly more correct responses than RD, but had similar reaction times. The experimental task performance significantly correlated with the performance on reading tests. Difference event-related potentials showed a voltage component peaking at 160 ms over frontocentral leads (P160d) that reached significantly higher amplitude in RD group, and was interpreted as an index of the amount of neural resources involved in visual working memory load. The amplitude of P160d significantly correlated with reading speed, the backward digit span and with the experimental task performance. Present results point out that highly demanding working memory tasks reveal behavioral and electrophysiological differences in RD children with respect to healthy controls.

ERP assessment of functional status in the temporal lobe: Examining spatiotemporal correlates of object recognition

We have previously used functional magnetic resonance imaging to assess specific anterolateral temporal regions involved in object recognition, providing a novel technique for evaluating functional status in temporal lobe epilepsy (i.e., a site directed approach). However, a method that combines high temporal resolution with spatial mapping is needed to isolate the specific processes associated with these regions (i.e., a process specific approach). In the current study, we evaluated the cognitive processes associated with object recognition and their relationship to the temporal lobe using high-density event-related potentials (ERPs). N300 and late positive component (LPC) responses were examined using a word-picture matching task, with three factors: congruity (match/mismatch), level of abstraction (basic/superordinate), and object category (natural/artificial). The N300 and LPC were analyzed in terms of their temporal characteristics as well as their spatial characteristics through source analysis. The results showed that the N300 primarily indexed processing of congruity and level of abstraction, whereas the LPC primarily indexed processing of object category. In agreement with previous functional imaging studies, N300 and LPC source analysis results confirmed temporal lobe involvement in object recognition. Importantly, LPC object category differences were detected in the anterior temporal lobe. Individual subject analyses revealed that these anterior temporal lobe differences were reliable--with greater activity for natural objects in 84% of subjects. The findings are discussed in terms of clinical applications that use spatiotemporal ERP differences to evaluate functional status of the temporal lobes.

Wechsler Memory Scale Revised Edition: neural correlates of the visual paired associates subtest adapted for fMRI

Memory deficits in neurological and psychiatric patients are evaluated by neuropsychological tests such as the Wechsler Memory Scale Revised Edition (WMS-R). Neuropsychological data from patients with circumscribed lesions point to single elements of the underlying neural network but fail to identify its whole extent. We report the fMRI adaptation of a subtest of the WMS-R, the Visual Paired Associates. Fifteen healthy, right-handed male volunteers were studied using a 1.5T MRI scanner. The encoding of the combination between a shape and a colour, the assessment of the retrieval of this combination immediately after encoding took place, and the underlying network employed during retrieval a second time after approximately 25 min were investigated. The results show a fronto-parieto-occipital network with left frontal accentuation for encoding and a fronto-parieto-occipital network for immediate and delayed retrieval. Noteworthy is the specific role of the thalamus. During immediate retrieval, the thalamus showed significant bilateral activation; during delayed retrieval, there was no significant activation. The thalami are part of an extended hippocampal-diencephalic system which is critical for efficient encoding and normal retrieval of new episodic information. We describe the probability of thalamocortical connections during retrieval based on the Thalamus Connectivity Atlas. The cerebellum showed significant activation in all conditions; its part in higher cognitive functions such as memory was thereby confirmed.

Working memory effects on semantic processing: Priming differences in pars orbitalis

Both working memory (WM) and controlled (attention-mediated) semantic processing functions have been thought to operate as limited capacity systems, but the possible link between these processes has not been investigated. We found that increased WM load attenuated semantic priming (i.e., reduced the response time advantage for semantically primed relative to unprimed items) and changed fMRI signal intensities in brain regions usually associated with both WM (dorsolateral prefrontal cortex) and controlled semantic retrieval (inferior frontal gyrus [IFG], pars orbitalis). fMRI signal changes in dorsolateral prefrontal cortex were negatively correlated with signal changes in pars orbitalis. The findings suggest that controlled semantic processing and working memory share neural system resources.

The evoked potential as a measure of perceptual and semantic differences

This review discusses the information content of perceptual and semantic evoked potentials arising in humans as a result of instantaneous changes in nonverbal and verbal stimuli. The amplitudes of perceptual and semantic evoked potentials were found to correlate positively with subjects' assessments of the differences between these stimuli. Multidimensional scaling matrixes of evoked potential amplitudes and subjective assessments of differences on pairwise substitution of stimuli showed that the actual colors and color names occupied a four-dimensional spherical color space and were encoded by excitation vectors of identical lengths. Color differences were equal to the absolute differences between their excitation vectors, while semantic differences in color names corresponded to the absolute difference vectors represented by long-term color memory traces. These data were reviewed in the framework of a spherical model of cognitive processes.

Localizing the distributed language network responsible for the N400 measured by MEG during auditory sentence processing

We studied auditory sentence comprehension using magnetoencephalography while subjects listened to sentences whose correctness they had to judge subsequently. The localization and the time course of brain electrical activity during processing of correct and semantically incorrect sentences were estimated by computing a brain surface current density within a cortical layer for both conditions. Finally, a region of interest (ROI) analysis was conducted to determine the time course of specific locations. A magnetic N400 was present in six spatially different ROIs. Semantic anomalies caused an exclusive involvement of the ventral portion of the left inferior frontal gyrus (BA 47) and left pars triangularis (BA 45). The anterior parts of the superior (BA 22) and inferior (BA 20/21) temporal gyri bilaterally were activated by both conditions. The activation for the correct condition, however, peaked earlier in both left temporal regions (approximately 32 ms). In general, activation due to semantic violations was more pronounced, started later, and lasted longer as compared to correct sentences. The findings reveal a clear left-hemispheric dominance during language processing indicated firstly by the mere number of activated regions (four in the left vs. two in the right hemisphere) and secondly by the observed specificity of the left inferior frontal ROIs to semantic violations. The temporal advantage observed for the correct condition in the left temporal regions is supporting the notion that the established context eases the processing of the final word. Semantically incorrect words that do not fit into the context result in longer integration times.

Effects of acoustic distortion and semantic context on event-related potentials to spoken words

This study examined the neurophysiological effects of acoustic degradation on auditory semantic processing. Event-related potentials were recorded to target words presented in a sentence context. Targets were semantically congruent or incongruent with the context, which was acoustically intact or low-pass filtered. In unaltered contexts, N400 amplitude was significantly greater for incongruent than congruent words. Filtering significantly reduced this effect, even though participants were highly accurate in interpreting the degraded stimuli, as shown by an anomaly detection task. This reduction in the N400 effect appeared to be driven by decreased N400 amplitudes over posterior electrode sites for incongruent targets. These results demonstrate that acoustic degradation influences the neural response to words in context by reducing the availability of semantic information in on-line sentence comprehension.