Using single-trial EEG to predict and analyze subsequent memory

Decoding EEG for optimizing naturalistic memory

BACKGROUND

Spectral features of human electroencephalographic (EEG) recordings during learning predict subsequent recall variability.

NEW METHOD

Capitalizing on these fluctuating neural features, we develop a non-invasive closed-loop (NICL) system for real-time optimization of human learning. Participants play a virtual navigation-and-memory game; recording multi-session data across days allowed us to build participant-specific classification models of recall success. In subsequent closed-loop sessions, our platform manipulated the timing of memory encoding, selectively presenting items during periods of predicted good or poor memory function based on EEG features decoded in real time.

RESULTS

The induced memory effect (the difference between recall rates when presenting items during predicted good vs. poor learning periods) increased with the accuracy of neural decoding.

COMPARISON WITH EXISTING METHODS

This study demonstrates greater-than-chance memory decoding from EEG recordings in a naturalistic virtual navigation task with greater real-world validity than basic word-list recall paradigms. Here we modulate memory by timing stimulus presentation based on noninvasive scalp EEG recordings, whereas prior closed-loop studies for memory improvement involved intracranial recordings and direct electrical stimulation. Other noninvasive studies have investigated the use of neurofeedback or remedial study for memory improvement.

CONCLUSIONS

These findings present a proof-of-concept for using non-invasive closed-loop technology to optimize human learning and memory through principled stimulus timing, but only in those participants for whom classifiers reliably predict out-of-sample memory function.

More than the sum of its parts: investigating episodic memory as a multidimensional cognitive process

Why do we remember some events but forget others? Previous studies attempting to decode successful vs. unsuccessful brain states to investigate this question have met with limited success, potentially due, in part, to assessing episodic memory as a unidimensional process, despite evidence that multiple domains contribute to episodic encoding. Using a novel machine learning algorithm known as "transfer learning", we leveraged visual perception, sustained attention, and selective attention brain states to better predict episodic memory performance from trial-to-trial encoding electroencephalography (EEG) activity. We found that this multidimensional treatment of memory decoding improved prediction performance compared to traditional, unidimensional, methods, with each cognitive domain explaining unique variance in decoding of successful encoding-related neural activity. Importantly, this approach could be applied to cognitive domains outside of memory. Overall, this study provides critical insight into the underlying reasons why some events are remembered while others are not.

Improved data quality and statistical power of trial-level event-related potentials with Bayesian random-shift Gaussian processes

Studies of cognitive processes via electroencephalogram (EEG) recordings often analyze group-level event-related potentials (ERPs) averaged over multiple subjects and trials. This averaging procedure can obscure scientifically relevant variability across subjects and trials, but has been necessary due to the difficulties posed by inference of trial-level ERPs. We introduce the Bayesian Random Phase-Amplitude Gaussian Process (RPAGP) model, for inference of trial-level amplitude, latency, and ERP waveforms. We apply RPAGP to data from a study of ERP responses to emotionally arousing images. The model estimates of trial-specific signals are shown to greatly improve statistical power in detecting significant differences in experimental conditions compared to existing methods. Our results suggest that replacing the observed data with the de-noised RPAGP predictions can potentially improve the sensitivity and accuracy of many of the existing ERP analysis pipelines.

EEG decoders track memory dynamics

Encoding- and retrieval-related neural activity jointly determine mnemonic success. We ask whether electroencephalographic activity can reliably predict encoding and retrieval success on individual trials. Each of 98 participants performed a delayed recall task on 576 lists across 24 experimental sessions. Logistic regression classifiers trained on spectral features measured immediately preceding spoken recall of individual words successfully predict whether or not those words belonged to the target list. Classifiers trained on features measured during word encoding also reliably predict whether those words will be subsequently recalled and further predict the temporal and semantic organization of the recalled items. These findings link neural variability predictive of successful memory with item-to-context binding, a key cognitive process thought to underlie episodic memory function.

Electroencephalography-based classification of Alzheimer's disease spectrum during computer-based cognitive testing

Alzheimer's disease (AD) is a progressive disease leading to cognitive decline, and to prevent it, researchers seek to diagnose mild cognitive impairment (MCI) early. Particularly, non-amnestic MCI (naMCI) is often mistaken for normal aging as the representative symptom of AD, memory decline, is absent. Subjective cognitive decline (SCD), an intermediate step between normal aging and MCI, is crucial for prediction or early detection of MCI, which determines the presence of AD spectrum pathology. We developed a computer-based cognitive task to classify the presence or absence of AD pathology and stage within the AD spectrum, and attempted to perform multi-stage classification through electroencephalography (EEG) during resting and memory encoding state. The resting and memory-encoding states of 58 patients (20 with SCD, 10 with naMCI, 18 with aMCI, and 10 with AD) were measured and classified into four groups. We extracted features that could reflect the phase, spectral, and temporal characteristics of the resting and memory-encoding states. For the classification, we compared nine machine learning models and three deep learning models using Leave-one-subject-out strategy. Significant correlations were found between the existing neurophysiological test scores and performance of our computer-based cognitive task for all cognitive domains. In all models used, the memory-encoding states realized a higher classification performance than resting states. The best model for the 4-class classification was cKNN. The highest accuracy using resting state data was 67.24%, while it was 93.10% using memory encoding state data. This study involving participants with SCD, naMCI, aMCI, and AD focused on early Alzheimer's diagnosis. The research used EEG data during resting and memory encoding states to classify these groups, demonstrating the significance of cognitive process-related brain waves for diagnosis. The computer-based cognitive task introduced in the study offers a time-efficient alternative to traditional neuropsychological tests, showing a strong correlation with their results and serving as a valuable tool to assess cognitive impairment with reduced bias.

Unveiling the time course of visual stabilization through human electrophysiology

Object positions are coded relative to their surroundings, presumably providing visual stability during eye movements. But when does this perceived stability arise? Here we used a visual illusion, the frame-induced position shift, and measured electrophysiological activity elicited by an object whose perceived position was either shifted because of a surrounding frame or not, thus dissociating perceived and physical locations. We found that visually evoked responses were sensitive to only physical location earlier in time (∼70 ms), but both physical and illusory location information was present at a later time point (∼140 ms). Furthermore, location information could be reliably decoded across physical and illusory locations during the later time interval but not during the earlier time interval, demonstrating that neural activity patterns are shared between the two processes at a later stage. These results suggest that visual stability of objects emerges relatively late and is thus dependent on recurrent feedback from higher processing stages.

Judgments of learning reveal conscious access to stimulus memorability

Despite the massive capacity of visual long-term memory, individuals do not successfully encode all visual information they wish to remember. This variability in encoding success has been traditionally ascribed to fluctuations in individuals' cognitive states (e.g., sustained attention) and differences in memory encoding processes (e.g., depth of encoding). However, recent work has shown that a considerable amount of variability in encoding success stems from intrinsic stimulus properties that determine the ease of encoding across individuals. While researchers have identified several perceptual and semantic properties that contribute to stimulus memorability, much remains unknown, including whether individuals are aware of the memorability of stimuli they encounter. In the present study, we investigated whether individuals have conscious access to the memorability of real-world stimuli while forming self-referential judgments of learning (JOL) during explicit memory encoding (Experiments 1A-B) and when asked about the perceived memorability of a stimulus in the absence of attempted encoding (Experiments 2A-B). We found that JOLs and perceived memorability estimates (PME) were consistent across individuals and predictive of memorability, confirming that individuals can access memorability with or without stimulus encoding. At the same time, access to memorability was not comprehensive. We found that individuals unexpectedly remembered and forgot consistent sets of stimuli as well. When we compared access to memorability between JOLs and PMEs, we found that individuals had more access during JOLs. Thus, our findings demonstrate that individuals have partial access to stimulus memorability and that explicit encoding increases the amount of access that is available.

Decoding declarative memory process for predicting memory retrieval based on source localization

Many studies have focused on understanding memory processes due to their importance in daily life. Differences in timing and power spectra of brain signals during encoding task have been linked to later remembered items and were recently used to predict memory retrieval performance. However, accuracies remain low when using non-invasive methods for acquiring brain signals, mainly due to the low spatial resolution. This study investigates the prediction of successful retrieval using estimated source activity corresponding either to cortical or subcortical structures through source localization. Electroencephalogram (EEG) signals were recorded while participants performed a declarative memory task. Frequency-time analysis was performed using signals from encoding and retrieval tasks to confirm the importance of neural oscillations and their relationship with later remembered and forgotten items. Significant differences in the power spectra between later remembered and forgotten items were found before and during the presentation of the stimulus in the encoding task. Source activity estimation revealed differences in the beta band power over the medial parietal and medial prefrontal areas prior to the presentation of the stimulus, and over the cuneus and lingual areas during the presentation of the stimulus. Additionally, there were significant differences during the stimuli presentation during the retrieval task. Prediction of later remembered items was performed using surface potentials and estimated source activity. The results showed that source localization increases classification performance compared to the one using surface potentials. These findings support the importance of incorporating spatial features of neural activity to improve the prediction of memory retrieval.

Stress diminishes outcome but enhances response representations during instrumental learning

Stress may shift behavioural control from a goal-directed system that encodes action-outcome relationships to a habitual system that learns stimulus-response associations. Although this shift to habits is highly relevant for stress-related psychopathologies, limitations of existing behavioural paradigms hinder research from answering the fundamental question of whether the stress-induced bias to habits is due to reduced outcome processing or enhanced response processing at the time of stimulus presentation, or both. Here, we used EEG-based multivariate pattern analysis to decode neural outcome representations crucial for goal-directed control, as well as response representations during instrumental learning. We show that stress reduced outcome representations but enhanced response representations. Both were directly associated with a behavioural index of habitual responding. Furthermore, changes in outcome and response representations were uncorrelated, suggesting that these may reflect distinct processes. Our findings indicate that habitual behaviour under stress may be the result of both enhanced stimulus-response processing and diminished outcome processing.

Alpha modulation in younger and older adults during distracted encoding

To successfully encode information into long-term memory, we need top-down control to focus our attention on target stimuli. This attentional focus is achieved by the modulation of sensory neuronal excitability through alpha power. Failure to modulate alpha power and to inhibit distracting information has been reported in older adults during attention and working memory tasks. Given that alpha power during encoding can predict subsequent memory performance, aberrant oscillatory modulations might play a role in age-related memory deficits. However, it is unknown whether there are age-related differences in memory performance or alpha modulation when encoding targets with distraction. Here we show that both older and younger adults are able to encode targets paired with distractors and that the level of alpha power modulation during encoding predicted recognition success. Even though older adults showed signs of higher distractibility, this did not harm their episodic memory for target information. Also, we demonstrate that older adults only modulated alpha power during high distraction, both by enhancing target processing and inhibiting distractor processing. These results indicate that both younger and older adults are able to employ the same inhibitory control mechanisms successfully, but that older adults fail to call upon these when distraction is minimal. The findings of this study give us more insight into the mechanisms involved in memory encoding across the lifespan.

Sleep spindles track cortical learning patterns for memory consolidation

Memory consolidation-the transformation of labile memory traces into stable long-term representations-is facilitated by post-learning sleep. Computational and biophysical models suggest that sleep spindles may play a key mechanistic role for consolidation, igniting structural changes at cortical sites involved in prior learning. Here, we tested the resulting prediction that spindles are most pronounced over learning-related cortical areas and that the extent of this learning-spindle overlap predicts behavioral measures of memory consolidation. Using high-density scalp electroencephalography (EEG) and polysomnography (PSG) in healthy volunteers, we first identified cortical areas engaged during a temporospatial associative memory task (power decreases in the alpha/beta frequency range, 6-20 Hz). Critically, we found that participant-specific topographies (i.e., spatial distributions) of post-learning sleep spindle amplitude correlated with participant-specific learning topographies. Importantly, the extent to which spindles tracked learning patterns further predicted memory consolidation across participants. Our results provide empirical evidence for a role of post-learning sleep spindles in tracking learning networks, thereby facilitating memory consolidation.

Evaluation of classification approaches for distinguishing brain states predictive of episodic memory performance from electroencephalography: Abbreviated Title: Evaluating methods of classifying memory states from EEG

Previous studies have attempted to separate single trial neural responses for events a person is likely to remember from those they are likely to forget using machine learning classification methods. Successful single trial classification holds potential for translation into the clinical realm for real-time detection of memory and other cognitive states to provide real-time interventions (i.e., brain-computer interfaces). However, most of these studies-and classification analyses in general- do not make clear if the chosen methodology is optimally suited for the classification of memory-related brain states. To address this problem, we systematically compared different methods for every step of classification (i.e., feature extraction, feature selection, classifier selection) to investigate which methods work best for decoding episodic memory brain states-the first analysis of its kind. Using an adult lifespan sample EEG dataset collected during performance of an episodic context encoding and retrieval task, we found that no specific feature type (including Common Spatial Pattern (CSP)-based features, mean, variance, correlation, features based on AR model, entropy, phase, and phase synchronization) outperformed others consistently in distinguishing different memory classes. However, extracting all of these feature types consistently outperformed extracting only one type of feature. Additionally, the combination of filtering and sequential forward selection was the optimal method to select the effective features compared to filtering alone or performing no feature selection at all. Moreover, although all classifiers performed at a fairly similar level, LASSO was consistently the highest performing classifier compared to other commonly used options (i.e., naïve Bayes, SVM, and logistic regression) while naïve Bayes was the fastest classifier. Lastly, for multiclass classification (i.e., levels of context memory confidence and context feature perception), generalizing the binary classification using the binary decision tree performed better than the voting or one versus rest method. These methods were shown to outperform alternative approaches for three orthogonal datasets (i.e., EEG working memory, EEG motor imagery, and MEG working memory), supporting their generalizability. Our results provide an optimized methodological process for classifying single-trial neural data and provide important insight and recommendations for a cognitive neuroscientist's ability to make informed choices at all stages of the classification process for predicting memory and other cognitive states.

Tendency to ruminate and anxiety are associated with altered alpha and beta oscillatory power dynamics during memory for contextual details

Rumination occurs when an individual becomes mentally stuck and cannot redirect attention away from an unwanted thought demonstrating cognitive inflexibility. Cognitive flexibility is important for various cognitive functions, including episodic memory. Trait rumination is a partial mediator in the relationship between depression and overgeneral episodic memory, suggesting that rumination may negatively influence memory for contextual details. Oscillations in the alpha (8-12 Hz) and beta (13-30 Hz) frequency bands are crucial for various cognitive functions (e.g., attention control and episodic memory) and may help to explain the relationship between trait rumination and memory for contextual details. Our study uses EEG recorded during a source memory task to assess how alpha and beta oscillations during memory for contextual details may change as a function of trait rumination, anxiety, and depression level (n = 43). The source memory task instructs participants to remember objects and their associated contextual details. Memory for contextual details is lessened for participants higher in trait rumination paired with higher trait anxiety. Oscillations were analyzed in posterior parietal/occipital regions. During encoding, an interaction of nonclinical depression level and rumination predicts higher alpha power for items that were later not successfully remembered. During test, depression and rumination interact and predict higher alpha power for both successful and unsuccessful memory. These results suggest that trait anxiety, depression, and rumination impact accuracy and alpha oscillatory dynamics during contextual memory via changes in attention control.

Prediction of Successful Memory Encoding Based on Lateral Temporal Cortical Gamma Power

Prediction of successful memory encoding is important for learning. High-frequency activity (HFA), such as gamma frequency activity (30–150 Hz) of cortical oscillations, is induced during memory tasks and is thought to reflect underlying neuronal processes. Previous studies have demonstrated that medio-temporal electrophysiological characteristics are related to memory formation, but the effects of neocortical neural activity remain underexplored. The main aim of the present study was to evaluate the ability of gamma activity in human electrocorticography (ECoG) signals to differentiate memory processes into remembered and forgotten memories. A support vector machine (SVM) was employed, and ECoG recordings were collected from six subjects during verbal memory recognition task performance. Two-class classification using an SVM was performed to predict subsequently remembered vs. forgotten trials based on individually selected frequencies (low gamma, 30–60 Hz; high gamma, 60–150 Hz) at time points during pre- and during stimulus intervals. The SVM classifier distinguished memory performance between remembered and forgotten trials with a mean maximum accuracy of 87.5% using temporal cortical gamma activity during the 0- to 1-s interval. Our results support the functional relevance of ECoG for memory formation and suggest that lateral temporal cortical HFA may be utilized for memory prediction.

Disrupted Modulation of Alpha and Low Beta Oscillations Mediates Temporal Sequence Memory Deficits in People With Schizophrenia

BACKGROUND

People with schizophrenia (SZ) exhibit impaired episodic memory when relating objects to each other in time and space. Empirical studies and computational models suggest that low-frequency neural oscillations may be a mechanism by which the brain keeps track of temporal relationships during encoding and retrieval, with modulation of oscillatory power as sequences are learned. It is unclear whether sequence memory deficits in SZ are associated with altered neural oscillations.

METHODS

Using electroencephalography, this study examined neural oscillations in 51 healthy control subjects and 37 people with SZ during a temporal sequence learning task. Multiple 5-object picture sequences were presented across 4 study-test blocks in either fixed or random order. Participants answered semantic questions for each object (e.g., living/nonliving), and sequence memory was operationalized as faster responses for fixed versus random sequences. Differences in oscillatory power between fixed versus random sequences provided a neural index of temporal sequence memory.

RESULTS

Although both groups showed reaction time differences in late blocks (blocks 3 and 4), this evidence of sequence memory was reduced in people with SZ relative to healthy control subjects. Decreases in globally distributed prestimulus alpha (8-12 Hz) and beta 1 (13-20 Hz) power for fixed versus random sequences in late blocks were also attenuated in people with SZ relative to healthy control subjects. Moreover, changes in oscillatory power predicted individual reaction time differences and fully mediated the relationship between group and sequence memory.

CONCLUSIONS

Disrupted modulation of alpha and beta 1 electroencephalography oscillations is a candidate mechanism of temporal sequence memory deficits in people with SZ.

Predicting memory from study-related brain activity

For both basic and applied reasons, an important goal is to identify brain activity present while people study materials that enable us to predict whether they will remember those materials. We show that this is possible with the conventional event-related potential “subsequent-memory-effect” signals as well as with machine learning classifiers, but only to a small degree. This is in line with behavioral research, which supports many determinants of memory apart from the cognitive processes during study.

Single-Trial EEG Connectivity of Default Mode Network Before and During Encoding Predicts Subsequent Memory Outcome

The successful memory process produces specific activity in the brain network. As the brain activity of the prestimulus and encoding phases has a crucial effect on subsequent memory outcomes (e.g., remembered or forgotten), previous studies have tried to predict the memory performance in this period. Conventional studies have used the spectral power or event-related potential of specific regions as the classification feature. However, as multiple brain regions work collaboratively to process memory, it could be a better option to use functional connectivity within the memory-related brain network to predict subsequent memory performance. In this study, we acquired the EEG signals while performing an associative memory task that remembers scene-word pairs. For the connectivity analysis, we estimated the cross-mutual information within the default mode network with the time-frequency spectra at the prestimulus and encoding phases. Then, we predicted the success or failure of subsequent memory outcome with the connectivity features. We found that the classifier with support vector machine achieved the highest classification accuracy of 80.83% ± 12.65% (mean ± standard deviation) using the beta (13-30 Hz) connectivity at encoding phase among the multiple frequency bands and task phases. Using the prestimulus beta connectivity, the classification accuracy of 72.45% ± 12.52% is also achieved. Among the features, the connectivity related to the dorsomedial prefrontal cortex was found to contribute to successful memory encoding. The connectivity related to the posterior cingulate cortex was found to contribute to the failure of memory encoding. The present study showed for the first time the successful prediction with high accuracy of subsequent memory outcome using single-trial functional connectivity.

Prediction of Memory Retrieval Performance Using Ear-EEG Signals

Many studies have explored brain signals during the performance of a memory task to predict later remembered items. However, prediction methods are still poorly used in real life and are not practical due to the use of electroencephalography (EEG) recorded from the scalp. Ear-EEG has been recently used to measure brain signals due to its flexibility when applying it to real world environments. In this study, we attempt to predict whether a shown stimulus is going to be remembered or forgotten using ear-EEG and compared its performance with scalp-EEG. Our results showed that there was no significant difference between ear-EEG and scalp-EEG. In addition, the higher prediction accuracy was obtained using a convolutional neural network (pre-stimulus: 74.06%, on-going stimulus: 69.53%) and it was compared to other baseline methods. These results showed that it is possible to predict performance of a memory task using ear-EEG signals and it could be used for predicting memory retrieval in a practical brain-computer interface.

EEG-Based Prediction of Successful Memory Formation During Vocabulary Learning

Previous Electroencephalography (EEG) and neuroimaging studies have found differences between brain signals for subsequently remembered and forgotten items during learning of items - it has even been shown that single trial prediction of memorization success is possible with a few target items. There has been little attempt, however, in validating the findings in an application-oriented context involving longer test spans with realistic learning materials encompassing more items. Hence, the present study investigates subsequent memory prediction within the application context of foreign-vocabulary learning. We employed an off-line, EEG-based paradigm in which Korean participants without prior German language experience learned 900 German words in paired-associate form. Our results using convolutional neural networks optimized for EEG-signal analysis show that above-chance classification is possible in this context allowing us to predict during learning which of the words would be successfully remembered later.

Stress Alters the Neural Context for Building New Memories

Stressful events affect mnemonic processing, in particular for emotionally arousing events. Previous research on the mechanisms underlying stress effects on human memory focused on stress-induced changes in the neural activity elicited by a stimulus. We tested an alternative mechanism and hypothesized that stress may already alter the neural context for successful memory formation, reflected in the neural activity preceding a stimulus. Therefore, 69 participants underwent a stress or control procedure before encoding neutral and negative pictures. During encoding, we recorded high-density EEG and analyzed-based on multivariate searchlight analyses-oscillatory activity and cross-frequency coupling patterns before stimulus onset that were predictive of memory tested 24 hr later. Prestimulus theta predicted subsequent memory in controls but not in stressed participants. Instead, prestimulus gamma predicted successful memory formation after stress, specifically for emotional material. Likewise, stress altered the patterns of prestimulus theta-beta and theta-gamma phase-amplitude coupling predictive of subsequent memory, again depending on the emotionality of the presented material. Our data suggest that stress changes the neural context for building new memories, tuning this neural context specifically to the encoding of emotionally salient events. These findings point to a yet unknown mechanism through which stressful events may change (emotional) memory formation.

Auditory cortical alpha/beta desynchronization prioritizes the representation of memory items during a retention period

To-be-memorized information in working-memory could be protected against distracting influences by processes of functional inhibition or prioritization. Modulations of oscillations in the alpha to beta range in task-relevant sensory regions have been suggested to play an important role for both mechanisms. We adapted a Sternberg task variant to the auditory modality, with a strong or a weak distracting sound presented at a predictable time during the retention period. Using a time-generalized decoding approach, relatively decreased strength of memorized information was found prior to strong distractors, paralleled by decreased pre-distractor alpha/beta power in the left superior temporal gyrus (lSTG). Over the entire group, reduced beta power in lSTG was associated with relatively increased strength of memorized information. The extent of alpha power modulations within participants was negatively correlated with strength of memorized information. Overall, our results are compatible with a prioritization account, but point to nuanced differences between alpha and beta oscillations.

Oscillatory Mechanisms of Successful Memory Formation in Younger and Older Adults Are Related to Structural Integrity

We studied oscillatory mechanisms of memory formation in 48 younger and 51 older adults in an intentional associative memory task with cued recall. While older adults showed lower memory performance than young adults, we found subsequent memory effects (SME) in alpha/beta and theta frequency bands in both age groups. Using logistic mixed effects models, we investigated whether interindividual differences in structural integrity of key memory regions could account for interindividual differences in the strength of the SME. Structural integrity of inferior frontal gyrus (IFG) and hippocampus was reduced in older adults. SME in the alpha/beta band were modulated by the cortical thickness of IFG, in line with its hypothesized role for deep semantic elaboration. Importantly, this structure–function relationship did not differ by age group. However, older adults were more frequently represented among the participants with low cortical thickness and consequently weaker SME in the alpha band. Thus, our results suggest that differences in the structural integrity of the IFG contribute not only to interindividual, but also to age differences in memory formation.

Spatio-temporal analysis of error-related brain activity in active and passive brain-computer interfaces

Electroencephalography (EEG)-based brain–computer interface (BCI) systems infer brain signals recorded via EEG without using common neuromuscular pathways. User brain response to BCI error is a contributor to non-stationarity of the EEG signal and poses challenges in developing reliable active BCI control. Many passive BCI implementations, on the other hand, have the detection of error-related brain activity as their primary goal. Therefore, reliable detection of this signal is crucial in both active and passive BCIs. In this work, we propose CREST: a novel covariance-based method that uses Riemannian and Euclidean geometry and combines spatial and temporal aspects of the feedback-related brain activity in response to BCI error. We evaluate our proposed method with two datasets: an active BCI for 1-D cursor control using motor imagery and a passive BCI for 2-D cursor control. We show significant improvement across participants in both datasets compared to existing methods.

Directional coupling of slow and fast hippocampal gamma with neocortical alpha/beta oscillations in human episodic memory

Episodic memories hinge upon our ability to process a wide range of multisensory information and bind this information into a coherent, memorable representation. On a neural level, these 2 processes are thought to be supported by neocortical alpha/beta desynchronization and hippocampal theta/gamma synchronization, respectively. Intuitively, these 2 processes should couple to successfully create and retrieve episodic memories, yet this hypothesis has not been tested empirically. We address this by analyzing human intracranial electroencephalogram data recorded during 2 associative memory tasks. We find that neocortical alpha/beta (8 to 20 Hz) power decreases reliably precede and predict hippocampal "fast" gamma (60 to 80 Hz) power increases during episodic memory formation; during episodic memory retrieval, however, hippocampal "slow" gamma (40 to 50 Hz) power increases reliably precede and predict later neocortical alpha/beta power decreases. We speculate that this coupling reflects the flow of information from the neocortex to the hippocampus during memory formation, and hippocampal pattern completion inducing information reinstatement in the neocortex during memory retrieval.

Reduced memory-related ongoing oscillatory activity in healthy older adults

Age-related impairments in episodic memory have been linked to alterations in encoding-induced neural activity. In young individuals, even prestimulus activity has been shown to influence the encoding of an upcoming stimulus, with ongoing theta and beta oscillations being predictive of subsequent recognition. The present study investigated if these memory-related ongoing oscillations are also affected by aging. In an EEG experiment, healthy older and young individuals performed an encoding task with a subsequent recognition test on picture and word stimuli. The group of younger participants showed an increased oscillatory activity in the lower frequency range (ranging from 3 to 17 Hz) in the pre- and post-stimulus period compared with the older adults. Only in young participants, ongoing beta power during encoding was related to later memory in both stimulus categories, whereas in older participants, this effect was diminished. Interestingly, there was no general age-related decrease in recognition performance. These results indicate that ongoing low beta oscillations might constitute a functional indicator of cognitive aging that reveals itself even before a strong decline in behavioral performance is noticeable, and that could be a potential target for neuromodulatory interventions.

Electrophysiological and behavioral evidence for attentional up-regulation, but not down-regulation, when encoding pictures into long-term memory

Visual long-term memory allows us to store a virtually infinite amount of visual information (Brady, Konkle, Alvarez, & Oliva in Proceedings of the National Academy of Sciences of the United States of America, 105(38), 14325-14329, 2008; Standing in Quarterly Journal of Experimental Psychology, 25(2), 207-222, 1973). However, our ability to encode new visual information fluctuates from moment to moment. In Experiment 1, we tested the hypothesis that we have voluntary control over these periodic fluctuations in our ability to encode representations into visual long-term memory using a precueing paradigm combined with behavioral and electrophysiological indices of memory encoding. We found that visual memory encoding can be up-regulated, but it was much more difficult, if not impossible, to down-regulate encoding on a trial-by-trial basis. In Experiment 2, we tested the hypothesis that voluntary up-regulation of visual memory encoding for an item incurs a cost to memory encoding of other items by manipulating the cueing probability. Here, we found that, although the cueing benefit was constant for both low (20%) and high (50%) cueing probabilities, the benefit in the high cueing probability condition came with the overall impairment of memory encoding. Taken together, our findings demonstrate that top-down control of visual long-term memory encoding may be primarily to prioritize certain memories, but this prioritization has a cost and should not be overused to avoid its negative consequences.

Out and about: Subsequent memory effect captured in a natural outdoor environment with smartphone EEG

Spatiotemporal context plays an important role in episodic memory. While temporal context effects have been frequently studied in the laboratory, ecologically valid spatial context manipulations are difficult to implement in stationary conditions. We investigated whether the neural correlates of successful encoding (subsequent memory effect) can be captured in a real-world environment. An off-the-shelf Android smartphone was used for wireless mobile EEG acquisition and stimulus presentation. Participants encoded single words, each of which was presented at a different location on a university campus. Locations were approximately 10-12 m away from each other, half of them with striking features (landmarks) nearby. We predicted landmarks would improve recall performance. After a first free recall task of verbal stimuli indoors, participants performed a subsequent recall outdoors, in which words and locations were recalled. As predicted, significantly more words presented at landmark locations as well as significantly more landmark than nonlandmark locations were recalled. ERP analysis yielded a larger posterior positive deflection during encoding for hits compared to misses in the 400-800 ms interval. Likewise, time-frequency analysis revealed a significant difference during encoding for hits compared to misses in the form of stronger alpha (200-300 ms) and theta (300-400 ms) power increases. Our results confirm that a vibrant spatial context is beneficial in episodic memory processing and that the underlying neural correlates can be captured with unobtrusive smartphone EEG technology. The advent of mobile EEG technology promises to unveil the relevance of natural physical activity and natural environments on memory.

The Role of Meaning in Visual Memory: Face-Selective Brain Activity Predicts Memory for Ambiguous Face Stimuli

How people process images is known to affect memory for those images, but these effects have typically been studied using explicit task instructions to vary encoding. Here, we investigate the effects of intrinsic variation in processing on subsequent memory, testing whether recognizing an ambiguous stimulus as meaningful (as a face vs as shape blobs) predicts subsequent visual memory even when matching the perceptual features and the encoding strategy between subsequently remembered and subsequently forgotten items. We show in adult humans of either sex that single trial EEG activity can predict whether participants will subsequently remember an ambiguous Mooney face image (e.g., an image that will sometimes be seen as a face and sometimes not be seen as a face). In addition, we show that a classifier trained only to discriminate between whether participants perceive a face versus non-face can generalize to predict whether an ambiguous image is subsequently remembered. Furthermore, when we examine the N170, an event-related potential index of face processing, we find that images that elicit larger N170s are more likely to be remembered than those that elicit smaller N170s, even when the exact same image elicited larger or smaller N170s across participants. Thus, images processed as meaningful, in this case as a face, during encoding are better remembered than identical images that are not processed as a face. This provides strong evidence that understanding the meaning of a stimulus during encoding plays a critical role in visual memory.SIGNIFICANCE STATEMENT Is visual memory inherently visual or does meaning and other conceptual information necessarily play a role even in memory for detailed visual information? Here we show that it is easier to remember an image when it is processed in a meaningful way, as indexed by the amount of category-specific brain activity it elicits. In particular, we use single-trial EEG activity to predict whether an image will be subsequently remembered, and show that the main driver of this prediction ability is whether or not an image is seen as meaningful or non-meaningful. This shows that the extent to which an image is processed as meaningful can be used to predict subsequent memory even when controlling for perceptual factors and encoding strategies that typically differ across images.

Single-Trial EEG Analysis Predicts Memory Retrieval and Reveals Source-Dependent Differences

We used pattern classifiers to extract features related to recognition memory retrieval from the temporal information in single-trial electroencephalography (EEG) data during attempted memory retrieval. Two-class classification was conducted on correctly remembered trials with accurate context (or source) judgments vs. correctly rejected trials. The average accuracy for datasets recorded in a single session was 61% while the average accuracy for datasets recorded in two separate sessions was 56%. To further understand the basis of the classifier’s performance, two other pattern classifiers were trained on different pairs of behavioral conditions. The first of these was designed to use information related to remembering the item and the second to use information related to remembering the contextual information (or source) about the item. Mollison and Curran (2012) had earlier shown that subjects’ familiarity judgments contributed to improved memory of spatial contextual information but not of extrinsic associated color information. These behavioral results were similarly reflected in the event-related potential (ERP) known as the FN400 (an early frontal effect relating to familiarity) which revealed differences between correct and incorrect context memories in the spatial but not color conditions. In our analyses we show that a classifier designed to distinguish between correct and incorrect context memories, more strongly involves early activity (400–500 ms) over the frontal channels for the location distinctions, than for the extrinsic color associations. In contrast, the classifier designed to classify memory for the item (without memory for the context), had more frontal channel involvement for the color associated experiments than for the spatial experiments. Taken together these results argue that location may be bound more tightly with the item than an extrinsic color association. The multivariate classification approach also showed that trial-by-trial variation in EEG corresponding to these ERP components were predictive of subjects’ behavioral responses. Additionally, the multivariate classification approach enabled analysis of error conditions that did not have sufficient trials for standard ERP analyses. These results suggested that false alarms were primarily attributable to item memory (as opposed to memory of associated context), as commonly predicted, but with little previous corroborating EEG evidence.

The Sync/deSync Model: How a Synchronized Hippocampus and a Desynchronized Neocortex Code Memories

Neural oscillations are important for memory formation in the brain. The desynchronization of alpha (10 Hz) oscillations in the neocortex has been shown to predict successful memory encoding and retrieval. However, when engaging in learning, it has been found that the hippocampus synchronizes in theta (4 Hz) oscillations, and that learning is dependent on the phase of theta. This inconsistency as to whether synchronization is "good" for memory formation leads to confusion over which oscillations we should expect to see and where during learning paradigm experiments. This paper seeks to respond to this inconsistency by presenting a neural network model of how a well functioning learning system could exhibit both of these phenomena, i.e., desynchronization of alpha and synchronization of theta during successful memory encoding.We present a spiking neural network (the Sync/deSync model) of the neocortical and hippocampal system. The simulated hippocampus learns through an adapted spike-time dependent plasticity rule, in which weight change is modulated by the phase of an extrinsically generated theta oscillation. Additionally, a global passive weight decay is incorporated, which is also modulated by theta phase. In this way, the Sync/deSync model exhibits theta phase-dependent long-term potentiation and long-term depression. We simulated a learning paradigm experiment and compared the oscillatory dynamics of our model with those observed in single-cell and scalp-EEG studies of the medial temporal lobe. Our Sync/deSync model suggests that both the desynchronization of neocortical alpha and the synchronization of hippocampal theta are necessary for successful memory encoding and retrieval.SIGNIFICANCE STATEMENT A fundamental question is the role of rhythmic activation of neurons, i.e., how and why their firing oscillates between high and low rates. A particularly important question is how oscillatory dynamics between the neocortex and hippocampus support memory formation. We present a spiking neural-network model of such memory formation, with the central ideas that (1) in neocortex, neurons need to break out of an alpha oscillation to represent a stimulus (i.e., alpha desynchronizes), whereas (2) in hippocampus, the firing of neurons at theta facilitates formation of memories (i.e., theta synchronizes). Accordingly, successful memory formation is marked by reduced neocortical alpha and increased hippocampal theta. This pattern has been observed experimentally and gives our model its name-the Sync/deSync model.

Differential effects of ongoing EEG beta and theta power on memory formation

Recently, elevated ongoing pre-stimulus beta power (13–17 Hz) at encoding has been associated with subsequent memory formation for visual stimulus material. It is unclear whether this activity is merely specific to visual processing or whether it reflects a state facilitating general memory formation, independent of stimulus modality. To answer that question, the present study investigated the relationship between neural pre-stimulus oscillations and verbal memory formation in different sensory modalities. For that purpose, a within-subject design was employed to explore differences between successful and failed memory formation in the visual and auditory modality. Furthermore, associative memory was addressed by presenting the stimuli in combination with background images. Results revealed that similar EEG activity in the low beta frequency range (13–17 Hz) is associated with subsequent memory success, independent of stimulus modality. Elevated power prior to stimulus onset differentiated successful from failed memory formation. In contrast, differential effects between modalities were found in the theta band (3–7 Hz), with an increased oscillatory activity before the onset of later remembered visually presented words. In addition, pre-stimulus theta power dissociated between successful and failed encoding of associated context, independent of the stimulus modality of the item itself. We therefore suggest that increased ongoing low beta activity reflects a memory promoting state, which is likely to be moderated by modality-independent attentional or inhibitory processes, whereas high ongoing theta power is suggested as an indicator of the enhanced binding of incoming interlinked information.

Remembered or Forgotten?-An EEG-Based Computational Prediction Approach

Prediction of memory performance (remembered or forgotten) has various potential applications not only for knowledge learning but also for disease diagnosis. Recently, subsequent memory effects (SMEs)—the statistical differences in electroencephalography (EEG) signals before or during learning between subsequently remembered and forgotten events—have been found. This finding indicates that EEG signals convey the information relevant to memory performance. In this paper, based on SMEs we propose a computational approach to predict memory performance of an event from EEG signals. We devise a convolutional neural network for EEG, called ConvEEGNN, to predict subsequently remembered and forgotten events from EEG recorded during memory process. With the ConvEEGNN, prediction of memory performance can be achieved by integrating two main stages: feature extraction and classification. To verify the proposed approach, we employ an auditory memory task to collect EEG signals from scalp electrodes. For ConvEEGNN, the average prediction accuracy was 72.07% by using EEG data from pre-stimulus and during-stimulus periods, outperforming other approaches. It was observed that signals from pre-stimulus period and those from during-stimulus period had comparable contributions to memory performance. Furthermore, the connection weights of ConvEEGNN network can reveal prominent channels, which are consistent with the distribution of SME studied previously.

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

Oscillations and Episodic Memory: Addressing the Synchronization/Desynchronization Conundrum

Brain oscillations are one of the core mechanisms underlying episodic memory. However, while some studies highlight the role of synchronized oscillatory activity, others highlight the role of desynchronized activity. We here describe a framework to resolve this conundrum and integrate these two opposing oscillatory behaviors. Specifically, we argue that the synchronization and desynchronization reflect a division of labor between a hippocampal and a neocortical system, respectively. We describe a novel oscillatory framework that integrates synchronization and desynchronization mechanisms to explain how the two systems interact in the service of episodic memory.

Data from rodent as well as human studies suggest that theta/gamma synchronization in the hippocampus (i.e., theta phase to gamma power cross-frequency coupling) mediates the binding of different elements in episodic memory.

In vivo and in vitro animal studies suggest that theta provides selective time windows for fast-acting synaptic modifications and recent computational models have implemented these mechanisms to explain human memory formation and retrieval.

Recent data from human experiments suggest that low-frequency power decreases in the neocortex, most evident in the alpha/beta frequency range, mediate encoding and reinstatement of episodic memories.

The content of reinstated memories can be decoded from cortical low-frequency patterns.

Prediction of successful memory encoding based on single-trial rhinal and hippocampal phase information

Mediotemporal EEG characteristics are closely related to long-term memory formation. It has been reported that rhinal and hippocampal EEG measures reflecting the stability of phases across trials are better suited to distinguish subsequently remembered from forgotten trials than event-related potentials or amplitude-based measures. Theoretical models suggest that the phase of EEG oscillations reflects neural excitability and influences cellular plasticity. However, while previous studies have shown that the stability of phase values across trials is indeed a relevant predictor of subsequent memory performance, the effect of absolute single-trial phase values has been little explored. Here, we reanalyzed intracranial EEG recordings from the mediotemporal lobe of 27 epilepsy patients performing a continuous word recognition paradigm. Two-class classification using a support vector machine was performed to predict subsequently remembered vs. forgotten trials based on individually selected frequencies and time points. We demonstrate that it is possible to successfully predict single-trial memory formation in the majority of patients (23 out of 27) based on only three single-trial phase values given by a rhinal phase, a hippocampal phase, and a rhinal-hippocampal phase difference. Overall classification accuracy across all subjects was 69.2% choosing frequencies from the range between 0.5 and 50Hz and time points from the interval between -0.5s and 2s. For 19 patients, above chance prediction of subsequent memory was possible even when choosing only time points from the prestimulus interval (overall accuracy: 65.2%). Furthermore, prediction accuracies based on single-trial phase surpassed those based on single-trial power. Our results confirm the functional relevance of mediotemporal EEG phase for long-term memory operations and suggest that phase information may be utilized for memory enhancement applications based on deep brain stimulation.

The "silent" imprint of musical training

Playing a musical instrument at a professional level is a complex multimodal task requiring information integration between different brain regions supporting auditory, somatosensory, motor, and cognitive functions. These kinds of task-specific activations are known to have a profound influence on both the functional and structural architecture of the human brain. However, until now, it is widely unknown whether this specific imprint of musical practice can still be detected during rest when no musical instrument is used. Therefore, we applied high-density electroencephalography and evaluated whole-brain functional connectivity as well as small-world topologies (i.e., node degree) during resting state in a sample of 15 professional musicians and 15 nonmusicians. As expected, musicians demonstrate increased intra- and interhemispheric functional connectivity between those brain regions that are typically involved in music perception and production, such as the auditory, the sensorimotor, and prefrontal cortex as well as Broca's area. In addition, mean connectivity within this specific network was positively related to musical skill and the total number of training hours. Thus, we conclude that musical training distinctively shapes intrinsic functional network characteristics in such a manner that its signature can still be detected during a task-free condition. Hum Brain Mapp 37:536-546, 2016. © 2015 Wiley Periodicals, Inc.

Intention to encode boosts memory-related pre-stimulus EEG beta power

Pre-stimulus oscillatory brain activity can predict the degree to which an upcoming stimulus will be remembered at a later point in time. Recently, increased pre-stimulus power in ongoing theta (5-8Hz) and low beta (13-17Hz) bands during encoding has been associated with enhanced memory performance. When a cue is presented before stimulus onset, encoding-related brain activations may be regarded as a sign of preparation for the upcoming stimulus. Here, we investigated whether the intention to encode the following stimulus into long-term memory affects these preparatory pre-stimulus activations during encoding. Two groups of 18 participants took part in a subsequent memory paradigm. Electroencephalogram (EEG) was recorded while participants were presented with a series of pictures, each one preceded by a cue, which were supposed to be classified according to animacy. One group was informed about the upcoming recognition task and therefore was enabled to develop the intention to encode the stimuli (intentional encoding), whereas the other group did not receive this information (incidental encoding). Afterwards, recognition of the pictures was tested. During intentional encoding only, power in theta and low beta bands was found to be significantly increased before the onset of pictures that were later remembered compared to later forgotten ones. Group comparisons confirmed greater memory-related power increases in the low beta band for intentional than incidental encoding. These findings indicate that oscillatory states that are associated with successful encoding can be initiated voluntarily if the intention to encode the stimuli is given. We therefore suggest low beta band activation before stimulus onset to be an indicator of memory-specific preparation for an upcoming stimulus.

The Influence of Pre-stimulus EEG Activity on Reaction Time During a Verbal Sternberg Task is Related to Musical Expertise

Previous work highlighted the possibility that musical training has an influence on cognitive functioning. The suggested reason for this influence is the strong recruitment of attention, planning, and working memory functions during playing a musical instrument. The purpose of the present work was twofold, namely to evaluate the general relationship between pre-stimulus electrophysiological activity and cognition, and more specifically the influence of musical expertise on working memory functions. With this purpose in mind, we used covariance mapping analyses to evaluate whether pre-stimulus electroencephalographic activity is predictive for reaction time during a visual working memory task (Sternberg paradigm) in musicians and non-musicians. In line with our hypothesis, we replicated previous findings pointing to a general predictive value of pre-stimulus activity for working memory performance. Most importantly, we also provide first evidence for an influence of musical expertise on working memory performance that could distinctively be predicted by pre-stimulus spectral power. Our results open novel perspectives for better comprehending the vast influences of musical expertise on cognition.