Using brain imaging to track problem solving in a complex state space

Cognitive processing stages in mental rotation - How can cognitive modelling inform HsMM-EEG models?

The aspiration for insight into human cognitive processing has traditionally driven research in cognitive science. With methods such as the Hidden semi-Markov Model-Electroencephalography (HsMM-EEG) method, new approaches have been developed that help to understand the temporal structure of cognition by identifying temporally discrete processing stages. However, it remains challenging to assign concrete functional contributions by specific processing stages to the overall cognitive process. In this paper, we address this challenge by linking HsMM-EEG3 with cognitive modelling, with the aim of further validating the HsMM-EEG3 method and demonstrating the potential of cognitive models to facilitate functional interpretation of processing stages. For this purpose, we applied HsMM-EEG3 to data from a mental rotation task and developed an ACT-R cognitive model that is able to closely replicate human performance in this task. Applying HsMM-EEG3 to the mental rotation experiment data revealed a strong likelihood for 6 distinct stages of cognitive processing during trials, with an additional stage for non-rotated conditions. The cognitive model predicted intra-trial mental activity patterns that project well onto the processing stages, while explaining the additional stage as a marker of non-spatial shortcut use. Thereby, this combined methodology provided substantially more information than either method by itself and suggests conclusions for cognitive processing in general.

Spatiotemporally resolved multivariate pattern analysis for M/EEG

An emerging goal in neuroscience is tracking what information is represented in brain activity over time as a participant completes some task. While electroencephalography (EEG) and magnetoencephalography (MEG) offer millisecond temporal resolution of how activity patterns emerge and evolve, standard decoding methods present significant barriers to interpretability as they obscure the underlying spatial and temporal activity patterns. We instead propose the use of a generative encoding model framework that simultaneously infers the multivariate spatial patterns of activity and the variable timing at which these patterns emerge on individual trials. An encoding model inversion maps from these parameters to the equivalent decoding model, allowing predictions to be made about unseen test data in the same way as in standard decoding methodology. These SpatioTemporally Resolved MVPA (STRM) models can be flexibly applied to a wide variety of experimental paradigms, including classification and regression tasks. We show that these models provide insightful maps of the activity driving predictive accuracy metrics; demonstrate behaviourally meaningful variation in the timing of pattern emergence on individual trials; and achieve predictive accuracies that are either equivalent or surpass those achieved by more widely used methods. This provides a new avenue for investigating the brain's representational dynamics and could ultimately support more flexible experimental designs in the future.

Levels of biological plausibility

Notions of mechanism, emergence, reduction and explanation are all tied to levels of analysis. I cover the relationship between lower and higher levels, suggest a level of mechanism approach for neuroscience in which the components of a mechanism can themselves be further decomposed and argue that scientists' goals are best realized by focusing on pragmatic concerns rather than on metaphysical claims about what is ‘real'. Inexplicably, neuroscientists are enchanted by both reduction and emergence. A fascination with reduction is misplaced given that theory is neither sufficiently developed nor formal to allow it, whereas metaphysical claims of emergence bring physicalism into question. Moreover, neuroscience's existence as a discipline is owed to higher-level concepts that prove useful in practice. Claims of biological plausibility are shown to be incoherent from a level of mechanism view and more generally are vacuous. Instead, the relevant findings to address should be specified so that model selection procedures can adjudicate between competing accounts. Model selection can help reduce theoretical confusions and direct empirical investigations. Although measures themselves, such as behaviour, blood-oxygen-level-dependent (BOLD) and single-unit recordings, are not levels of analysis, like levels, no measure is fundamental and understanding how measures relate can hasten scientific progress.

This article is part of the theme issue ‘Key relationships between non-invasive functional neuroimaging and the underlying neuronal activity'.

The Human Brain Traverses a Common Activation-Pattern State Space Across Task and Rest

Much of our lives are spent in unconstrained rest states, yet cognitive brain processes are primarily investigated using task-constrained states. It may be possible to utilize the insights gained from experimental control of task processes as reference points for investigating unconstrained rest. To facilitate comparison of rest and task functional magnetic resonance imaging data, we focused on activation amplitude patterns, commonly used for task but not rest analyses. During rest, we identified spontaneous changes in temporally extended whole-brain activation-pattern states. This revealed a hierarchical organization of rest states. The top consisted of two competing states consistent with previously identified "task-positive" and "task-negative" activation patterns. These states were composed of more specific states that repeated over time and across individuals. Contrasting with the view that rest consists of only task-negative states, task-positive states occurred 40% of the time while individuals "rested," suggesting task-focused activity may occur during rest. Together our results suggest that brain activation dynamics form a general hierarchy across task and rest, with a small number of dominant general states reflecting basic functional modes and a variety of specific states potentially reflecting a wide variety of cognitive processes.

Approximating Bayesian Inference through Model Simulation

The ultimate test of the validity of a cognitive theory is its ability to predict patterns of empirical data. Cognitive models formalize this test by making specific processing assumptions that yield mathematical predictions, and the mathematics allow the models to be fitted to data. As the field of cognitive science has grown to address increasingly complex problems, so too has the complexity of models increased. Some models have become so complex that the mathematics detailing their predictions are intractable, meaning that the model can only be simulated. Recently, new Bayesian techniques have made it possible to fit these simulation-based models to data. These techniques have even allowed simulation-based models to transition into neuroscience, where tests of cognitive theories can be biologically substantiated.

Implicit emotion regulation in adolescent girls: An exploratory investigation of Hidden Markov Modeling and its neural correlates

Numerous data demonstrate that distracting emotional stimuli cause behavioral slowing (i.e. emotional conflict) and that behavior dynamically adapts to such distractors. However, the cognitive and neural mechanisms that mediate these behavioral findings are poorly understood. Several theoretical models have been developed that attempt to explain these phenomena, but these models have not been directly tested on human behavior nor compared. A potential tool to overcome this limitation is Hidden Markov Modeling (HMM), which is a computational approach to modeling indirectly observed systems. Here, we administered an emotional Stroop task to a sample of healthy adolescent girls (N = 24) during fMRI and used HMM to implement theoretical behavioral models. We then compared the model fits and tested for neural representations of the hidden states of the most supported model. We found that a modified variant of the model posited by Mathews et al. (1998) was most concordant with observed behavior and that brain activity was related to the model-based hidden states. Particularly, while the valences of the stimuli themselves were encoded primarily in the ventral visual cortex, the model-based detection of threatening targets was associated with increased activity in the bilateral anterior insula, while task effort (i.e. adaptation) was associated with reduction in the activity of these areas. These findings suggest that emotional target detection and adaptation are accomplished partly through increases and decreases, respectively, in the perceived immediate relevance of threatening cues and also demonstrate the efficacy of using HMM to apply theoretical models to human behavior.

On the Role of Situational Stressors in the Disruption of Global Neural Network Stability during Problem Solving

When individuals are placed in stressful situations, they are likely to exhibit deficits in cognitive capacity over and above situational demands. Despite this, individuals may still persevere and ultimately succeed in these situations. Little is known, however, about neural network properties that instantiate success or failure in both neutral and stressful situations, particularly with respect to regions integral for problem-solving processes that are necessary for optimal performance on more complex tasks. In this study, we outline how hidden Markov modeling based on multivoxel pattern analysis can be used to quantify unique brain states underlying complex network interactions that yield either successful or unsuccessful problem solving in more neutral or stressful situations. We provide evidence that brain network stability and states underlying synchronous interactions in regions integral for problem-solving processes are key predictors of whether individuals succeed or fail in stressful situations. Findings also suggested that individuals utilize discriminate neural patterns in successfully solving problems in stressful or neutral situations. Findings overall highlight how hidden Markov modeling can provide myriad possibilities for quantifying and better understanding the role of global network interactions in the problem-solving process and how the said interactions predict success or failure in different contexts.

The discovery of processing stages: Extension of Sternberg's method

We introduce a method for measuring the number and durations of processing stages from the electroencephalographic signal and apply it to the study of associative recognition. Using an extension of past research that combines multivariate pattern analysis with hidden semi-Markov models, the approach identifies on a trial-by-trial basis where brief sinusoidal peaks (called bumps) are added to the ongoing electroencephalographic signal. We propose that these bumps mark the onset of critical cognitive stages in processing. The results of the analysis can be used to guide the development of detailed process models. Applied to the associative recognition task, the hidden semi-Markov models multivariate pattern analysis method indicates that the effects of associative strength and probe type are localized to a memory retrieval stage and a decision stage. This is in line with a previously developed the adaptive control of thought-rational process model, called ACT-R, of the task. As a test of the generalization of our method we also apply it to a data set on the Sternberg working memory task collected by Jacobs, Hwang, Curran, and Kahana (2006). The analysis generalizes robustly, and localizes the typical set size effect in a late comparison/decision stage. In addition to providing information about the number and durations of stages in associative recognition, our analysis sheds light on the event-related potential components implicated in the study of recognition memory. (PsycINFO Database Record

Hidden Stages of Cognition Revealed in Patterns of Brain Activation

To advance cognitive theory, researchers must be able to parse the performance of a task into its significant mental stages. In this article, we describe a new method that uses functional MRI brain activation to identify when participants are engaged in different cognitive stages on individual trials. The method combines multivoxel pattern analysis to identify cognitive stages and hidden semi-Markov models to identify their durations. This method, applied to a problem-solving task, identified four distinct stages: encoding, planning, solving, and responding. We examined whether these stages corresponded to their ascribed functions by testing whether they are affected by appropriate factors. Planning-stage duration increased as the method for solving the problem became less obvious, whereas solving-stage duration increased as the number of calculations to produce the answer increased. Responding-stage duration increased with the difficulty of the motor actions required to produce the answer.

The sequential structure of brain activation predicts skill

In an fMRI study, participants were trained to play a complex video game. They were scanned early and then again after substantial practice. While better players showed greater activation in one region (right dorsal striatum) their relative skill was better diagnosed by considering the sequential structure of whole brain activation. Using a cognitive model that played this game, we extracted a characterization of the mental states that are involved in playing a game and the statistical structure of the transitions among these states. There was a strong correspondence between this measure of sequential structure and the skill of different players. Using multi-voxel pattern analysis, it was possible to recognize, with relatively high accuracy, the cognitive states participants were in during particular scans. We used the sequential structure of these activation-recognized states to predict the skill of individual players. These findings indicate that important features about information-processing strategies can be identified from a model-based analysis of the sequential structure of brain activation.

A statistical approach for segregating cognitive task stages from multivariate fMRI BOLD time series

Multivariate pattern analysis can reveal new information from neuroimaging data to illuminate human cognition and its disturbances. Here, we develop a methodological approach, based on multivariate statistical/machine learning and time series analysis, to discern cognitive processing stages from functional magnetic resonance imaging (fMRI) blood oxygenation level dependent (BOLD) time series. We apply this method to data recorded from a group of healthy adults whilst performing a virtual reality version of the delayed win-shift radial arm maze (RAM) task. This task has been frequently used to study working memory and decision making in rodents. Using linear classifiers and multivariate test statistics in conjunction with time series bootstraps, we show that different cognitive stages of the task, as defined by the experimenter, namely, the encoding/retrieval, choice, reward and delay stages, can be statistically discriminated from the BOLD time series in brain areas relevant for decision making and working memory. Discrimination of these task stages was significantly reduced during poor behavioral performance in dorsolateral prefrontal cortex (DLPFC), but not in the primary visual cortex (V1). Experimenter-defined dissection of time series into class labels based on task structure was confirmed by an unsupervised, bottom-up approach based on Hidden Markov Models. Furthermore, we show that different groupings of recorded time points into cognitive event classes can be used to test hypotheses about the specific cognitive role of a given brain region during task execution. We found that whilst the DLPFC strongly differentiated between task stages associated with different memory loads, but not between different visual-spatial aspects, the reverse was true for V1. Our methodology illustrates how different aspects of cognitive information processing during one and the same task can be separated and attributed to specific brain regions based on information contained in multivariate patterns of voxel activity.

The algorithmic level is the bridge between computation and brain

Every scientist chooses a preferred level of analysis and this choice shapes the research program, even determining what counts as evidence. This contribution revisits Marr's (1982) three levels of analysis (implementation, algorithmic, and computational) and evaluates the prospect of making progress at each individual level. After reviewing limitations of theorizing within a level, two strategies for integration across levels are considered. One is top-down in that it attempts to build a bridge from the computational to algorithmic level. Limitations of this approach include insufficient theoretical constraint at the computation level to provide a foundation for integration, and that people are suboptimal for reasons other than capacity limitations. Instead, an inside-out approach is forwarded in which all three levels of analysis are integrated via the algorithmic level. This approach maximally leverages mutual data constraints at all levels. For example, algorithmic models can be used to interpret brain imaging data, and brain imaging data can be used to select among competing models. Examples of this approach to integration are provided. This merging of levels raises questions about the relevance of Marr's tripartite view.

Discovering the structure of mathematical problem solving

The goal of this research is to discover the stages of mathematical problem solving, the factors that influence the duration of these stages, and how these stages are related to the learning of a new mathematical competence. Using a combination of multivariate pattern analysis (MVPA) and hidden Markov models (HMM), we found that participants went through 5 major phases in solving a class of problems: A Define Phase where they identified the problem to be solved, an Encode Phase where they encoded the needed information, a Compute Phase where they performed the necessary arithmetic calculations, a Transform Phase where they performed any mathematical transformations, and a Respond Phase where they entered an answer. The Define Phase is characterized by activity in visual attention and default network regions, the Encode Phase by activity in visual regions, the Compute Phase by activity in regions active in mathematical tasks, the Transform Phase by activity in mathematical and response regions, and the Respond phase by activity in motor regions. The duration of the Compute and Transform Phases were the only ones that varied with condition. Two features distinguished the mastery trials on which participants came to understand a new problem type. First, the duration of late phases of the problem solution increased. Second, there was increased activation in the rostrolateral prefrontal cortex (RLPFC) and angular gyrus (AG), regions associated with metacognition. This indicates the importance of reflection to successful learning.

Stages of Processing in Associative Recognition: Evidence from Behavior, EEG, and Classification

In this study, we investigated the stages of information processing in associative recognition. We recorded EEG data while participants performed an associative recognition task that involved manipulations of word length, associative fan, and probe type, which were hypothesized to affect the perceptual encoding, retrieval, and decision stages of the recognition task, respectively. Analyses of the behavioral and EEG data, supplemented with classification of the EEG data using machine-learning techniques, provided evidence that generally supported the sequence of stages assumed by a computational model developed in the Adaptive Control of Thought-Rational cognitive architecture. However, the results suggested a more complex relationship between memory retrieval and decision-making than assumed by the model. Implications of the results for modeling associative recognition are discussed. The study illustrates how a classifier approach, in combination with focused manipulations, can be used to investigate the timing of processing stages.

Discovering the sequential structure of thought

Multi-voxel pattern recognition techniques combined with Hidden Markov models can be used to discover the mental states that people go through in performing a task. The combined method identifies both the mental states and how their durations vary with experimental conditions. We apply this method to a task where participants solve novel mathematical problems. We identify four states in the solution of these problems: Encoding, Planning, Solving, and Respond. The method allows us to interpret what participants are doing on individual problem-solving trials. The duration of the planning state varies on a trial-to-trial basis with novelty of the problem. The duration of solution stage similarly varies with the amount of computation needed to produce a solution once a plan is devised. The response stage similarly varies with the complexity of the answer produced. In addition, we identified a number of effects that ran counter to a prior model of the task. Thus, we were able to decompose the overall problem-solving time into estimates of its components and in way that serves to guide theory.

A Bayesian framework for simultaneously modeling neural and behavioral data

Scientists who study cognition infer underlying processes either by observing behavior (e.g., response times, percentage correct) or by observing neural activity (e.g., the BOLD response). These two types of observations have traditionally supported two separate lines of study. The first is led by cognitive modelers, who rely on behavior alone to support their computational theories. The second is led by cognitive neuroimagers, who rely on statistical models to link patterns of neural activity to experimental manipulations, often without any attempt to make a direct connection to an explicit computational theory. Here we present a flexible Bayesian framework for combining neural and cognitive models. Joining neuroimaging and computational modeling in a single hierarchical framework allows the neural data to influence the parameters of the cognitive model and allows behavioral data, even in the absence of neural data, to constrain the neural model. Critically, our Bayesian approach can reveal interactions between behavioral and neural parameters, and hence between neural activity and cognitive mechanisms. We demonstrate the utility of our approach with applications to simulated fMRI data with a recognition model and to diffusion-weighted imaging data with a response time model of perceptual choice.