Decoding oscillatory representations and mechanisms in memory

Decoding cognition in neurodevelopmental, psychiatric and neurological conditions with multivariate pattern analysis of EEG data

Multivariate pattern analysis (MVPA) of electroencephalographic (EEG) data represents a revolutionary approach to investigate how the brain encodes information. By considering complex interactions among spatio-temporal features at the individual level, MVPA overcomes the limitations of univariate techniques, which often fail to account for the significant inter- and intra-individual neural variability. This is particularly relevant when studying clinical populations, and therefore MVPA of EEG data has recently started to be employed as a tool to study cognition in brain disorders. Here, we review the insights offered by this methodology in the study of anomalous patterns of neural activity in conditions such as autism, ADHD, schizophrenia, dyslexia, neurological and neurodegenerative disorders, within different cognitive domains (perception, attention, memory, consciousness). Despite potential drawbacks that should be attentively addressed, these studies reveal a peculiar sensitivity of MVPA in unveiling dysfunctional and compensatory neurocognitive dynamics of information processing, which often remain blind to traditional univariate approaches. Such higher sensitivity in characterizing individual neurocognitive profiles can provide unique opportunities to optimise assessment and promote personalised interventions.

The relationship between frequency content and representational dynamics in the decoding of neurophysiological data

Decoding of high temporal resolution, stimulus-evoked neurophysiological data is increasingly used to test theories about how the brain processes information. However, a fundamental relationship between the frequency spectra of the neural signal and the subsequent decoding accuracy timecourse is not widely recognised. We show that, in commonly used instantaneous signal decoding paradigms, each sinusoidal component of the evoked response is translated to double its original frequency in the subsequent decoding accuracy timecourses. We therefore recommend, where researchers use instantaneous signal decoding paradigms, that more aggressive low pass filtering is applied with a cut-off at one quarter of the sampling rate, to eliminate representational alias artefacts. However, this does not negate the accompanying interpretational challenges. We show that these can be resolved by decoding paradigms that utilise both a signal's instantaneous magnitude and its local gradient information as features for decoding. On a publicly available MEG dataset, this results in decoding accuracy metrics that are higher, more stable over time, and free of the technical and interpretational challenges previously characterised. We anticipate that a broader awareness of these fundamental relationships will enable stronger interpretations of decoding results by linking them more clearly to the underlying signal characteristics that drive them.

Contributions of representational distinctiveness and stability to memory performance and age differences

Long-standing theories of cognitive aging suggest that memory decline is associated with age-related differences in the way information is neurally represented. Multivariate pattern similarity analyses enabled researchers to take a representational perspective on brain and cognition, and allowed them to study the properties of neural representations that support successful episodic memory. Two representational properties have been identified as crucial for memory performance, namely the distinctiveness and the stability of neural representations. Here, we review studies that used multivariate analysis tools for different neuroimaging techniques to clarify how these representational properties relate to memory performance across adulthood. While most evidence on age differences in neural representations involved stimulus category information , recent studies demonstrated that particularly item-level stability and specificity of activity patterns are linked to memory success and decline during aging. Overall, multivariate methods offer a versatile tool for our understanding of age differences in the neural representations underlying memory.

Decoding distributed oscillatory signals driven by memory and perception in the prefrontal cortex

Sensory perception and memory recall generate different conscious experiences. Although externally and internally driven neural activities signifying the same perceptual content overlap in the sensory cortex, their distribution in the prefrontal cortex (PFC), an area implicated in both perception and memory, remains elusive. Here, we test whether the local spatial configurations and frequencies of neural oscillations driven by perception and memory recall overlap in the macaque PFC using high-density electrocorticography and multivariate pattern analysis. We find that dynamically changing oscillatory signals distributed across the PFC in the delta-, theta-, alpha-, and beta-band ranges carry significant, but mutually different, information predicting the same feature of memory-recalled internal targets and passively perceived external objects. These findings suggest that the frequency-specific distribution of oscillatory neural signals in the PFC serves cortical signatures responsible for distinguishing between different types of cognition driven by external perception and internal memory.

Encoding contexts are incidentally reinstated during competitive retrieval and track the temporal dynamics of memory interference

The ability to remember an episode from our past is often hindered by competition from similar events. For example, if we want to remember the article a colleague recommended during the last lab meeting, we may need to resolve interference from other article recommendations from the same colleague. This study investigates if the contextual features specifying the encoding episodes are incidentally reinstated during competitive memory retrieval. Competition between memories was created through the AB/AC interference paradigm. Individual word-pairs were presented embedded in a slowly drifting real-word-like context. Multivariate pattern analysis (MVPA) of high temporal-resolution electroencephalographic (EEG) data was used to investigate context reactivation during memory retrieval. Behaviorally, we observed proactive (but not retroactive) interference; that is, performance for AC competitive retrieval was worse compared with a control DE noncompetitive retrieval, whereas AB retrieval did not suffer from competition. Neurally, proactive interference was accompanied by an early reinstatement of the competitor context and interference resolution was associated with the ensuing reinstatement of the target context. Together, these findings provide novel evidence showing that the encoding contexts of competing discrete events are incidentally reinstated during competitive retrieval and that such reinstatement tracks retrieval competition and subsequent interference resolution.

Spectral Pattern Similarity Analysis: Tutorial and Application in Developmental Cognitive Neuroscience

The human brain encodes information in neural activation patterns. While standard approaches to analyzing neural data focus on brain (de-)activation (e.g., regarding the location, timing, or magnitude of neural responses), multivariate neural pattern similarity analyses target the informational content represented by neural activity. In adults, a number of representational properties have been identified that are linked to cognitive performance, in particular the stability, distinctiveness, and specificity of neural patterns. However, although growing cognitive abilities across childhood suggest advancements in representational quality, developmental studies still rarely utilize information-based pattern similarity approaches, especially in electroencephalography (EEG) research. Here, we provide a comprehensive methodological introduction and step-by-step tutorial for pattern similarity analysis of spectral (frequency-resolved) EEG data including a publicly available pipeline and sample dataset with data from children and adults. We discuss computation of single-subject pattern similarities and their statistical comparison at the within-person to the between-group level as well as the illustration and interpretation of the results. This tutorial targets both novice and more experienced EEG researchers and aims to facilitate the usage of spectral pattern similarity analyses, making these methodologies more readily accessible for (developmental) cognitive neuroscientists.

Decoding Object-Based Auditory Attention from Source-Reconstructed MEG Alpha Oscillations

How do we attend to relevant auditory information in complex naturalistic scenes? Much research has focused on detecting which information is attended, without regarding underlying top-down control mechanisms. Studies investigating attentional control generally manipulate and cue specific features in simple stimuli. However, in naturalistic scenes it is impossible to dissociate relevant from irrelevant information based on low-level features. Instead, the brain has to parse and select auditory objects of interest. The neural underpinnings of object-based auditory attention remain not well understood. Here we recorded MEG while 15 healthy human subjects (9 female) prepared for the repetition of an auditory object presented in one of two overlapping naturalistic auditory streams. The stream containing the repetition was prospectively cued with 70% validity. Crucially, this task could not be solved by attending low-level features, but only by processing the objects fully. We trained a linear classifier on the cortical distribution of source-reconstructed oscillatory activity to distinguish which auditory stream was attended. We could successfully classify the attended stream from alpha (8-14 Hz) activity in anticipation of repetition onset. Importantly, attention could only be classified from trials in which subjects subsequently detected the repetition, but not from miss trials. Behavioral relevance was further supported by a correlation between classification accuracy and detection performance. Decodability was not sustained throughout stimulus presentation, but peaked shortly before repetition onset, suggesting that attention acted transiently according to temporal expectations. We thus demonstrate anticipatory alpha oscillations to underlie top-down control of object-based auditory attention in complex naturalistic scenes.SIGNIFICANCE STATEMENT In everyday life, we often find ourselves bombarded with auditory information, from which we need to select what is relevant to our current goals. Previous research has highlighted how we attend to specific highly controlled aspects of the auditory input. Although invaluable, it is still unclear how this relates to attentional control in naturalistic auditory scenes. Here we used the high precision of magnetoencephalography in space and time to investigate the brain mechanisms underlying top-down control of object-based attention in ecologically valid sound scenes. We show that rhythmic activity in auditory association cortex at a frequency of ∼10 Hz (alpha waves) controls attention to currently relevant segments within the auditory scene and predicts whether these segments are subsequently detected.

EEG and fMRI evidence for autobiographical memory reactivation in empathy

Empathy relies on the ability to mirror and to explicitly infer others' inner states. Theoretical accounts suggest that memories play a role in empathy, but direct evidence of reactivation of autobiographical memories (AM) in empathy is yet to be shown. We addressed this question in two experiments. In Experiment 1, electrophysiological activity (EEG) was recorded from 28 participants. Participants performed an empathy task in which targets for empathy were depicted in contexts for which participants either did or did not have an AM, followed by a task that explicitly required memory retrieval of the AM and non‐AM contexts. The retrieval task was implemented to extract the neural fingerprints of AM and non‐AM contexts, which were then used to probe data from the empathy task. An EEG pattern classifier was trained and tested across tasks and showed evidence for AM reactivation when participants were preparing their judgement in the empathy task. Participants self‐reported higher empathy for people depicted in situations they had experienced themselves as compared to situations they had not experienced. A second independent fMRI experiment replicated this behavioural finding and showed increased activation for AM compared to non‐AM in the brain networks underlying empathy: precuneus, posterior parietal cortex, superior and inferior parietal lobule, and superior frontal gyrus. Together, our study reports behavioural, electrophysiological, and fMRI evidence that robustly supports AM reactivation in empathy.

Replay bursts in humans coincide with activation of the default mode and parietal alpha networks

Our brains at rest spontaneously replay recently acquired information, but how this process is orchestrated to avoid interference with ongoing cognition is an open question. Here we investigated whether replay coincided with spontaneous patterns of whole-brain activity. We found, in two separate datasets, that replay sequences were packaged into transient bursts occurring selectively during activation of the default mode network (DMN) and parietal alpha networks. These networks are believed to support inwardly oriented attention and inhibit bottom-up sensory processing and were characterized by widespread synchronized oscillations coupled to increases in high frequency power, mechanisms thought to coordinate information flow between disparate cortical areas. Our data reveal a tight correspondence between two widely studied phenomena in neural physiology and suggest that the DMN may coordinate replay bursts in a manner that minimizes interference with ongoing cognition.

•

Replay in humans coincides with activity in specific resting brain networks

•

Clusters of heightened default mode and alpha activity are linked to replay bursts

•

These networks are characterized by highly synchronized brain-wide oscillations

•

High-frequency power bursts are uniquely linked to default mode network activation

Electroencephalography decoding of Chinese characters in primary school children and its prediction for word reading performance and development

Research on what neural mechanisms facilitate word reading development in non-alphabetic scripts is relatively rare. The present study was among the first to adopt a multivariate pattern classification analysis to decode electroencephalographic signals recorded for primary school children (N = 236) while performing a Chinese character decision task. Chinese is an ideal script for studying the relationship between neural discriminability (i.e., decodability) of the orthography and behavioral word reading skills since the mapping from orthography to phonology is relatively arbitrary in Chinese. This was also among the first empirical attempts to examine the extent to which decoding performance can predict current and subsequent word reading skills using a longitudinal design. Results showed that neural activation patterns of real characters can be distinguished from activation patterns for pseudo-characters, non-characters, and random stroke combinations in both younger and older children. Topography of the transformed classifier weights revealed two distinct cognitive sub-processes underlying single character recognition, but temporal generalization analysis suggested common neural mechanisms between the distinct cognitive sub-processes. Suggestive evidence from correlational and hierarchical regression analyses showed that decoding performance, assessed on average 2 months before the year 2 behavioral testing, predicted both year 1 word reading performance and the development of word reading fluency over the year. Results demonstrate that decoding performance, one indicator of how the neural system is functionally organized in processing characters and character-like stimuli, can serve as a useful neural marker in predicting current word reading skills and the capacity to learn to read.

Memory: Theta Rhythm Couples Periodic Reactivation during Memory Retrieval

Memory retrieval involves the reactivation of memory traces distributed throughout the brain. New research suggests that these memory reactivations have an oscillatory nature and that they are coupled to preferential stages of the hippocampal theta cycle.

Geometric classification of brain network dynamics via conic derivative discriminants

BACKGROUND

Over the past decade, pattern decoding techniques have granted neuroscientists improved anatomical specificity in mapping neural representations associated with function and cognition. Dynamical patterns are of particular interest, as evidenced by the proliferation and success of frequency domain methods that reveal structured spatiotemporal rhythmic brain activity. One drawback of such approaches, however, is the need to estimate spectral power, which limits the temporal resolution of classification.

NEW METHOD

We propose an alternative method that enables classification of dynamical patterns with high temporal fidelity. The key feature of the method is a conversion of time-series data into temporal derivatives. By doing so, dynamically-coded information may be revealed in terms of geometric patterns in the phase space of the derivative signal.

RESULTS

We derive a geometric classifier for this problem which simplifies into a straightforward calculation in terms of covariances. We demonstrate the relative advantages and disadvantages of the technique with simulated data and benchmark its performance with an EEG dataset of covert spatial attention. We reveal the timecourse of covert spatial attention and, by mapping the classifier weights anatomically, its retinotopic organization.

COMPARISON WITH EXISTING METHOD

We especially highlight the ability of the method to provide strong group-level classification performance compared to existing benchmarks, while providing information that is complementary with classical spectral-based techniques. The robustness and sensitivity of the method to noise is also examined relative to spectral-based techniques.

CONCLUSION

The proposed classification technique enables decoding of dynamic patterns with high temporal resolution, performs favorably to benchmark methods, and facilitates anatomical inference.

Neural Pattern Classification Tracks Transfer-Appropriate Processing in Episodic Memory

The transfer-appropriate processing (TAP) account holds that episodic memory depends on the overlap between encoding and retrieval processing. In the current study, we employed multivariate pattern analysis (MVPA) of electroencephalography to examine the relevance of spontaneously engaged visual processing during encoding for later retrieval. Human participants encoded word-picture associations, where the picture could be a famous face, a landmark, or an object. At test, we manipulated the retrieval demands by asking participants to retrieve either visual or verbal information about the pictures. MVPA revealed classification between picture categories during early perceptual stages of encoding (∼170 ms). Importantly, these visual category-specific neural patterns were predictive of later episodic remembering, but the direction of the relationship was contingent on the particular retrieval demand of the memory task: a benefit for the visual and a cost for the verbal. A reinstatement of the category-specific neural patterns established during encoding was observed during retrieval, and again the relationship with behavior varied with retrieval demands. Reactivation of visual representations during retrieval was associated with better memory in the visual task, but with lower performance in the verbal task. Our findings support and extend the TAP account by demonstrating that processing of particular aspects during memory formation can also have detrimental effects on later episodic remembering when other aspects of the event are called-for and shed new light on encoding and retrieval interactions in episodic memory.

Early and late components of EEG delay activity correlate differently with scene working memory performance

Sustained and elevated activity during the working memory delay period has long been considered the primary neural correlate for maintaining information over short time intervals. This idea has recently been reinterpreted in light of findings generated from multiple neural recording modalities and levels of analysis. To further investigate the sustained or transient nature of activity, the temporal-spectral evolution (TSE) of delay period activity was examined in humans with high density EEG during performance of a Sternberg working memory paradigm with a relatively long six second delay and with novel scenes as stimuli. Multiple analyses were conducted using different trial window durations and different baseline periods for TSE computation. Sensor level analyses revealed transient rather than sustained activity during delay periods. Specifically, the consistent finding among the analyses was that high amplitude activity encompassing the theta range was found early in the first three seconds of the delay period. These increases in activity early in the delay period correlated positively with subsequent ability to distinguish new from old probe scenes. Source level signal estimation implicated a right parietal region of transient early delay activity that correlated positively with working memory ability. This pattern of results adds to recent evidence that transient rather than sustained delay period activity supports visual working memory performance. The findings are discussed in relation to synchronous and desynchronous intra- and inter-regional neural transmission, and choosing an optimal baseline for expressing temporal-spectral delay activity change.

Decoding Dynamic Brain Patterns from Evoked Responses: A Tutorial on Multivariate Pattern Analysis Applied to Time Series Neuroimaging Data

Multivariate pattern analysis (MVPA) or brain decoding methods have become standard practice in analyzing fMRI data. Although decoding methods have been extensively applied in brain–computer interfaces, these methods have only recently been applied to time series neuroimaging data such as MEG and EEG to address experimental questions in cognitive neuroscience. In a tutorial style review, we describe a broad set of options to inform future time series decoding studies from a cognitive neuroscience perspective. Using example MEG data, we illustrate the effects that different options in the decoding analysis pipeline can have on experimental results where the aim is to “decode” different perceptual stimuli or cognitive states over time from dynamic brain activation patterns. We show that decisions made at both preprocessing (e.g., dimensionality reduction, subsampling, trial averaging) and decoding (e.g., classifier selection, cross-validation design) stages of the analysis can significantly affect the results. In addition to standard decoding, we describe extensions to MVPA for time-varying neuroimaging data including representational similarity analysis, temporal generalization, and the interpretation of classifier weight maps. Finally, we outline important caveats in the design and interpretation of time series decoding experiments.

Temporal structure in associative retrieval

Electrophysiological data disclose rich dynamics in patterns of neural activity evoked by sensory objects. Retrieving objects from memory reinstates components of this activity. In humans, the temporal structure of this retrieved activity remains largely unexplored, and here we address this gap using the spatiotemporal precision of magnetoencephalography (MEG). In a sensory preconditioning paradigm, 'indirect' objects were paired with 'direct' objects to form associative links, and the latter were then paired with rewards. Using multivariate analysis methods we examined the short-time evolution of neural representations of indirect objects retrieved during reward-learning about direct objects. We found two components of the evoked representation of the indirect stimulus, 200 ms apart. The strength of retrieval of one, but not the other, representational component correlated with generalization of reward learning from direct to indirect stimuli. We suggest the temporal structure within retrieved neural representations may be key to their function.

More than spikes: common oscillatory mechanisms for content specific neural representations during perception and memory

Although previous research into the mechanisms underlying sensory and episodic representations has primarily focused on changes in neural firing rate, more recent evidence suggests that neural oscillations also contribute to these representations. Here, we argue that multiplexed oscillatory power and phase contribute to neural representations at the mesoscopic scale, complementary to neuronal firing. Reviewing recent studies which used oscillatory activity to decipher content-specific neural representations, we identify oscillatory mechanisms common to both sensory and episodic memory representations and incorporate these into a model of episodic encoding and retrieval. This model advances the idea that oscillations provide a reference frame for phase-coded item representations during memory encoding and that shifts in oscillatory frequency and phase coordinate ensemble activity during memory retrieval.

Analytical methods and experimental approaches for electrophysiological studies of brain oscillations

Brain oscillations are increasingly the subject of electrophysiological studies probing their role in the functioning and dysfunction of the human brain. In recent years this research area has seen rapid and significant changes in the experimental approaches and analysis methods. This article reviews these developments and provides a structured overview of experimental approaches, spectral analysis techniques and methods to establish relationships between brain oscillations and behaviour.

Replay of very early encoding representations during recollection

Long-term memories are linked to cortical representations of perceived events, but it is unclear which types of representations can later be recollected. Using magnetoencephalography-based decoding, we examined which brain activity patterns elicited during encoding are later replayed during recollection in the human brain. The results show that the recollection of images depicting faces and scenes is associated with a replay of neural representations that are formed at very early (180 ms) stages of encoding. This replay occurs quite rapidly, ~500 ms after the onset of a cue that prompts recollection and correlates with source memory accuracy. Therefore, long-term memories are rapidly replayed during recollection and involve representations that were formed at very early stages of encoding. These findings indicate that very early representational information can be preserved in the memory engram and can be faithfully and rapidly reinstated during recollection. These novel insights into the nature of the memory engram provide constraints for mechanistic models of long-term memory function.

Multiple neural states of representation in short-term memory? It's a matter of attention

Short-term memory (STM) refers to the capacity-limited retention of information over a brief period of time, and working memory (WM) refers to the manipulation and use of that information to guide behavior. In recent years it has become apparent that STM and WM interact and overlap with other cognitive processes, including attention (the selection of a subset of information for further processing) and long-term memory (LTM—the encoding and retention of an effectively unlimited amount of information for a much longer period of time). Broadly speaking, there have been two classes of memory models: systems models, which posit distinct stores for STM and LTM (Atkinson and Shiffrin, 1968; Baddeley and Hitch, 1974); and state-based models, which posit a common store with different activation states corresponding to STM and LTM (Cowan, 1995; McElree, 1996; Oberauer, 2002). In this paper, we will focus on state-based accounts of STM. First, we will consider several theoretical models that postulate, based on considerable behavioral evidence, that information in STM can exist in multiple representational states. We will then consider how neural data from recent studies of STM can inform and constrain these theoretical models. In the process we will highlight the inferential advantage of multivariate, information-based analyses of neuroimaging data (fMRI and electroencephalography (EEG)) over conventional activation-based analysis approaches (Postle, in press). We will conclude by addressing lingering questions regarding the fractionation of STM, highlighting differences between the attention to information vs. the retention of information during brief memory delays.

Frontal midline theta oscillations during working memory maintenance and episodic encoding and retrieval

Neural oscillations in the theta band (4-8 Hz) are prominent in the human electroencephalogram (EEG), and many recent electrophysiological studies in animals and humans have implicated scalp-recorded frontal midline theta (FMT) in working memory and episodic memory encoding and retrieval processes. However, the functional significance of theta oscillations in human memory processes remains largely unknown. Here, we review studies in human and animals examining how scalp-recorded FMT relates to memory behaviors and also their possible neural generators. We also discuss models of the functional relevance of theta oscillations to memory processes and suggest promising directions for future research.

Population level inference for multivariate MEG analysis

Multivariate analysis is a very general and powerful technique for analysing Magnetoencephalography (MEG) data. An outstanding problem however is how to make inferences that are consistent over a group of subjects as to whether there are condition-specific differences in data features, and what are those features that maximise these differences. Here we propose a solution based on Canonical Variates Analysis (CVA) model scoring at the subject level and random effects Bayesian model selection at the group level. We apply this approach to beamformer reconstructed MEG data in source space. CVA estimates those multivariate patterns of activation that correlate most highly with the experimental design; the order of a CVA model is then determined by the number of significant canonical vectors. Random effects Bayesian model comparison then provides machinery for inferring the optimal order over the group of subjects. Absence of a multivariate dependence is indicated by the null model being the most likely. This approach can also be applied to CVA models with a fixed number of canonical vectors but supplied with different feature sets. We illustrate the method by identifying feature sets based on variable-dimension MEG power spectra in the primary visual cortex and fusiform gyrus that are maximally discriminative of data epochs before versus after visual stimulation.

Feature-specific information processing precedes concerted activation in human visual cortex

Current knowledge about the precise timing of visual input to the cortex relies largely on spike timings in monkeys and evoked-response latencies in humans. However, quantifying the activation onset does not unambiguously describe the timing of stimulus-feature-specific information processing. Here, we investigated the information content of the early human visual cortical activity by decoding low-level visual features from single-trial magnetoencephalographic (MEG) responses. MEG was measured from nine healthy subjects as they viewed annular sinusoidal gratings (spanning the visual field from 2 to 10° for a duration of 1 s), characterized by spatial frequency (0.33 cycles/degree or 1.33 cycles/degree) and orientation (45° or 135°); gratings were either static or rotated clockwise or anticlockwise from 0 to 180°. Time-resolved classifiers using a 20 ms moving window exceeded chance level at 51 ms (the later edge of the window) for spatial frequency, 65 ms for orientation, and 98 ms for rotation direction. Decoding accuracies of spatial frequency and orientation peaked at 70 and 90 ms, respectively, coinciding with the peaks of the onset evoked responses. Within-subject time-insensitive pattern classifiers decoded spatial frequency and orientation simultaneously (mean accuracy 64%, chance 25%) and rotation direction (mean 82%, chance 50%). Classifiers trained on data from other subjects decoded the spatial frequency (73%), but not the orientation, nor the rotation direction. Our results indicate that unaveraged brain responses contain decodable information about low-level visual features already at the time of the earliest cortical evoked responses, and that representations of spatial frequency are highly robust across individuals.