Tracking cognitive processing stages with MEG: A spatio-temporal model of associative recognition in the brain

Word Type and Frequency Effects on Lexical Decisions Are Process-dependent and Start Early

When encountering letter strings, we rapidly determine whether they are words. The speed of such lexical decisions (LDs) is affected by word frequency. Apart from influencing late, decision-related, processing stages, frequency has also been shown to affect very early stages, and even the processing of nonwords. We developed a detailed account of the different frequency effects involved in LDs by (1) dividing LDs into processing stages using a combination of hidden semi-Markov models and multivariate pattern analysis applied to EEG data and (2) using generalized additive mixed models to investigate how the effect of continuous word and nonword frequency differs between these stages. We discovered six stages shared between word types, with the fifth stage consisting of two substages for pseudowords only. In the earliest stages, visual processing was completed faster for frequent words, but took longer for word-like nonwords. Later stages involved an orthographic familiarity assessment followed by an elaborate decision process, both affected differently by frequency. We therefore conclude that frequency indeed affects all processes involved in LDs and that the magnitude and direction of these effects differ both by process and word type.

A whole-task brain model of associative recognition that accounts for human behavior and neuroimaging data

Brain models typically focus either on low-level biological detail or on qualitative behavioral effects. In contrast, we present a biologically-plausible spiking-neuron model of associative learning and recognition that accounts for both human behavior and low-level brain activity across the whole task. Based on cognitive theories and insights from machine-learning analyses of M/EEG data, the model proceeds through five processing stages: stimulus encoding, familiarity judgement, associative retrieval, decision making, and motor response. The results matched human response times and source-localized MEG data in occipital, temporal, prefrontal, and precentral brain regions; as well as a classic fMRI effect in prefrontal cortex. This required two main conceptual advances: a basal-ganglia-thalamus action-selection system that relies on brief thalamic pulses to change the functional connectivity of the cortex, and a new unsupervised learning rule that causes very strong pattern separation in the hippocampus. The resulting model shows how low-level brain activity can result in goal-directed cognitive behavior in humans.

Spatiotemporally resolved multivariate pattern analysis for M/EEG

An emerging goal in neuroscience is tracking what information is represented in brain activity over time as a participant completes some task. While electroencephalography (EEG) and magnetoencephalography (MEG) offer millisecond temporal resolution of how activity patterns emerge and evolve, standard decoding methods present significant barriers to interpretability as they obscure the underlying spatial and temporal activity patterns. We instead propose the use of a generative encoding model framework that simultaneously infers the multivariate spatial patterns of activity and the variable timing at which these patterns emerge on individual trials. An encoding model inversion maps from these parameters to the equivalent decoding model, allowing predictions to be made about unseen test data in the same way as in standard decoding methodology. These SpatioTemporally Resolved MVPA (STRM) models can be flexibly applied to a wide variety of experimental paradigms, including classification and regression tasks. We show that these models provide insightful maps of the activity driving predictive accuracy metrics; demonstrate behaviourally meaningful variation in the timing of pattern emergence on individual trials; and achieve predictive accuracies that are either equivalent or surpass those achieved by more widely used methods. This provides a new avenue for investigating the brain's representational dynamics and could ultimately support more flexible experimental designs in the future.

Thalamic bursts modulate cortical synchrony locally to switch between states of global functional connectivity in a cognitive task

Performing a cognitive task requires going through a sequence of functionally diverse stages. Although it is typically assumed that these stages are characterized by distinct states of cortical synchrony that are triggered by sub-cortical events, little reported evidence supports this hypothesis. To test this hypothesis, we first identified cognitive stages in single-trial MEG data of an associative recognition task, showing with a novel method that each stage begins with local modulations of synchrony followed by a state of directed functional connectivity. Second, we developed the first whole-brain model that can simulate cortical synchrony throughout a task. The model suggests that the observed synchrony is caused by thalamocortical bursts at the onset of each stage, targeted at cortical synapses and interacting with the structural anatomical connectivity. These findings confirm that cognitive stages are defined by distinct states of cortical synchrony and explains the network-level mechanisms necessary for reaching stage-dependent synchrony states.

Discovering the brain stages of lexical decision: Behavioral effects originate from a single neural decision process

Lexical decision (LD) - judging whether a sequence of letters constitutes a word - has been widely investigated. In a typical lexical decision task (LDT), participants are asked to respond whether a sequence of letters is an actual word or a nonword. Although behavioral differences between types of words/nonwords have been robustly detected in LDT, there is an ongoing discussion about the exact cognitive processes that underlie the word identification process in this task. To obtain data-driven evidence on the underlying processes, we recorded electroencephalographic (EEG) data and applied a novel machine-learning method, hidden semi-Markov model multivariate pattern analysis (HsMM-MVPA). In the current study, participants performed an LDT in which we varied the frequency of words (high, low frequency) and "wordlikeness" of non-words (pseudowords, random non-words). The results revealed that models with six processing stages accounted best for the data in all conditions. While most stages were shared, Stage 5 differed between conditions. Together, these results indicate that the differences in word frequency and lexicality effects are driven by a single cognitive processing stage. Based on its latency and topology, we interpret this stage as a Decision process during which participants discriminate between words and nonwords using activated lexical information.

The Discovery and Interpretation of Evidence Accumulation Stages

To improve the understanding of cognitive processing stages, we combined two prominent traditions in cognitive science: evidence accumulation models and stage discovery methods. While evidence accumulation models have been applied to a wide variety of tasks, they are limited to tasks in which decision-making effects can be attributed to a single processing stage. Here, we propose a new method that first uses machine learning to discover processing stages in EEG data and then applies evidence accumulation models to characterize the duration effects in the identified stages. To evaluate this method, we applied it to a previously published associative recognition task (Application 1) and a previously published random dot motion task with a speed-accuracy trade-off manipulation (Application 2). In both applications, the evidence accumulation models accounted better for the data when we first applied the stage-discovery method, and the resulting parameter estimates where generally in line with psychological theories. In addition, in Application 1 the results shed new light on target-foil effects in associative recognition, while in Application 2 the stage discovery method identified an additional stage in the accuracy-focused condition — challenging standard evidence accumulation accounts. We conclude that the new framework provides a powerful new tool to investigate processing stages.

High-pass filtering artifacts in multivariate classification of neural time series data

BACKGROUND

Traditionally, EEG/MEG data are high-pass filtered and baseline-corrected to remove slow drifts. Minor deleterious effects of high-pass filtering in traditional time-series analysis have been well-documented, including temporal displacements. However, its effects on time-resolved multivariate pattern classification analyses (MVPA) are largely unknown.

NEW METHOD

To prevent potential displacement effects, we extend an alternative method of removing slow drift noise - robust detrending - with a procedure in which we mask out all cortical events from each trial. We refer to this method as trial-masked robust detrending.

RESULTS

In both real and simulated EEG data of a working memory experiment, we show that both high-pass filtering and standard robust detrending create artifacts that result in the displacement of multivariate patterns into activity silent periods, particularly apparent in temporal generalization analyses, and especially in combination with baseline correction. We show that trial-masked robust detrending is free from such displacements.

COMPARISON WITH EXISTING METHOD(S)

Temporal displacement may emerge even with modest filter cut-off settings such as 0.05 Hz, and even in regular robust detrending. However, trial-masked robust detrending results in artifact-free decoding without displacements. Baseline correction may unwittingly obfuscate spurious decoding effects and displace them to the rest of the trial.

CONCLUSIONS

Decoding analyses benefit from trial-masked robust detrending, without the unwanted side effects introduced by filtering or regular robust detrending. However, for sufficiently clean data sets and sufficiently strong signals, no filtering or detrending at all may work adequately. Implications for other types of data are discussed, followed by a number of recommendations.

EEG, MEG and neuromodulatory approaches to explore cognition: Current status and future directions

Neural oscillations and their association with brain states and cognitive functions have been object of extensive investigation over the last decades. Several electroencephalography (EEG) and magnetoencephalography (MEG) analysis approaches have been explored and oscillatory properties have been identified, in parallel with the technical and computational advancement. This review provides an up-to-date account of how EEG/MEG oscillations have contributed to the understanding of cognition. Methodological challenges, recent developments and translational potential, along with future research avenues, are discussed.

The Common Time Course of Memory Processes Revealed

Magnetoencephalography (MEG) was used to compare memory processes in two experiments, one involving recognition of word pairs and the other involving recall of newly learned arithmetic facts. A combination of hidden semi-Markov models and multivariate pattern analysis was used to locate brief "bumps" in the sensor data that marked the onset of different stages of cognitive processing. These bumps identified a separation between a retrieval stage that identified relevant information in memory and a decision stage that determined what response was implied by that information. The encoding, retrieval, decision, and response stages displayed striking similarities across the two experiments in their duration and brain activation patterns. Retrieval and decision processes involve distinct brain activation patterns. We conclude that memory processes for two different tasks, associative recognition versus arithmetic retrieval, follow a common spatiotemporal neural pattern and that both tasks have distinct retrieval and decision stages.

Characterizing synchrony patterns across cognitive task stages of associative recognition memory

Numerous studies seek to understand the role of oscillatory synchronization in cognition. This problem is particularly challenging in the context of complex cognitive behavior, which consists of a sequence of processing steps with uncertain duration. In this study, we analyzed oscillatory connectivity measures in time windows that previous computational models had associated with a specific sequence of processing steps in an associative memory recognition task (visual encoding, familiarity, memory retrieval, decision making, and motor response). The timing of these processing steps was estimated on a single‐trial basis with a novel hidden semi‐Markov model multivariate pattern analysis (HSMM‐MVPA) method. We show that different processing stages are associated with specific patterns of oscillatory connectivity. Visual encoding is characterized by a dense network connecting frontal, posterior, and temporal areas as well as frontal and occipital phase locking in the 4–9 Hz theta band. Familiarity is associated with frontal phase locking in the 9–14 Hz alpha band. Decision making is associated with frontal and temporo‐central interhemispheric connections in the alpha band. During decision making, a second network in the theta band that connects left‐temporal, central, and occipital areas bears similarity to the neural signature for preparing a motor response. A similar theta band network is also present during the motor response, with additionally alpha band connectivity between right‐temporal and posterior areas. This demonstrates that the processing stages discovered with the HSMM‐MVPA method are indeed linked to distinct synchronization patterns, leading to a closer understanding of the functional role of oscillations in cognition.

Inter-subject alignment of MEG datasets in a common representational space

Pooling neural imaging data across subjects requires aligning recordings from different subjects. In magnetoencephalography (MEG) recordings, sensors across subjects are poorly correlated both because of differences in the exact location of the sensors, and structural and functional differences in the brains. It is possible to achieve alignment by assuming that the same regions of different brains correspond across subjects. However, this relies on both the assumption that brain anatomy and function are well correlated, and the strong assumptions that go into solving the under-determined inverse problem given the high-dimensional source space. In this article, we investigated an alternative method that bypasses source-localization. Instead, it analyzes the sensor recordings themselves and aligns their temporal signatures across subjects. We used a multivariate approach, multiset canonical correlation analysis (M-CCA), to transform individual subject data to a low-dimensional common representational space. We evaluated the robustness of this approach over a synthetic dataset, by examining the effect of different factors that add to the noise and individual differences in the data. On an MEG dataset, we demonstrated that M-CCA performs better than a method that assumes perfect sensor correspondence and a method that applies source localization. Last, we described how the standard M-CCA algorithm could be further improved with a regularization term that incorporates spatial sensor information. Hum Brain Mapp 38:4287-4301, 2017. © 2017 Wiley Periodicals, Inc.

From connectome to cognition: The search for mechanism in human functional brain networks

Recent developments in functional connectivity research have expanded the scope of human neuroimaging, from identifying changes in regional activation amplitudes to detailed mapping of large-scale brain networks. However, linking network processes to a clear role in cognition demands advances in the theoretical frameworks, algorithms, and experimental approaches applied. This would help evolve the field from a descriptive to an explanatory state, by targeting network interactions that can mechanistically account for cognitive effects. In the present review, we provide an explicit framework to aid this search for "network mechanisms", which anchors recent methodological advances in functional connectivity estimation to a renewed emphasis on careful experimental design. We emphasize how this framework can address specific questions in network neuroscience. These span ambiguity over the cognitive relevance of resting-state networks, how to characterize task-evoked and spontaneous network dynamics, how to identify directed or "effective" connections, and how to apply multivariate pattern analysis at the network level. In parallel, we apply the framework to highlight the mechanistic interaction of network components that remain "stable" across task domains and more "flexible" components associated with on-task reconfiguration. By emphasizing the need to structure the use of diverse analytic approaches with sound experimentation, our framework promotes an explanatory mapping between the workings of the cognitive mind and the large-scale network mechanisms of the human brain.