A Biologically Plausible Action Selection System for Cognitive Architectures: Implications of Basal Ganglia Anatomy for Learning and Decision-Making Models

Reliance on Episodic vs. Procedural Systems in Decision-Making Depends on Individual Differences in Their Relative Neural Efficiency

Experiential decision-making can be explained as a result of either memory-based or reinforcement-based processes. Here, for the first time, we show that individual preferences between a memory-based and a reinforcement-based strategy, even when the two are functionally equivalent in terms of expected payoff, are adaptively shaped by individual differences in resting-state brain connectivity between the corresponding brain regions. Using computational cognitive models to identify which mechanism was most likely used by each participant, we found that individuals with comparatively stronger connectivity between memory regions prefer a memory-based strategy, while individuals with comparatively stronger connectivity between sensorimotor and habit-formation regions preferentially rely on a reinforcement-based strategy. These results suggest that human decision-making is adaptive and sensitive to the neural costs associated with different strategies.

Allocating Mental Effort in Cognitive Tasks: A Model of Motivation in the ACT-R Cognitive Architecture

Motivation is the driving force that influences people's behaviors and interacts with many cognitive functions. Computationally, motivation is represented as a cost-benefit analysis that weighs efforts and rewards in order to choose the optimal actions. Shenhav and colleagues proposed an elegant theory, the Expected Value of Control (EVC), which describes the relationship between cognitive efforts, costs, and rewards. In this paper, we propose a more fine-grained and detailed motivation framework that incorporates the principles of EVC into the ACT-R cognitive architecture. Specifically, motivation is represented as a specific slot in the Goal buffer with a corresponding scalar value, M, that is translated into the reward value Rt that is delivered when the goal is reached. This implementation is tested in two models. The first model is a high-level model that reproduces the EVC predictions with abstract actions. The second model is an augmented version of an existing ACT-R model of the Simon task. The motivation mechanism is shown to permit optimal effort allocation and reproduce known phenomena. Finally, the broader implications of our mechanism are discussed.

Increased Basal Ganglia Modulatory Effective Connectivity Observed in Resting-State fMRI in Individuals With Parkinson's Disease

Alterations to interactions between networked brain regions underlie cognitive impairment in many neurodegenerative diseases, providing an important physiological link between brain structure and cognitive function. Previous attempts to characterize the effects of Parkinson's disease (PD) on network functioning using resting-state functional magnetic resonance imaging (rs-fMRI), however, have yielded inconsistent and contradictory results. Potential problems with prior work arise in the specifics of how the area targeted by the diseases (the basal ganglia) interacts with other brain regions. Specifically, current computational models point to the fact that the basal ganglia contributions should be captured with modulatory (i.e., second-order) rather than direct (i.e., first-order) functional connectivity measures. Following this hypothesis, a principled but manageable large-scale brain architecture, the Common Model of Cognition, was used to identify differences in basal ganglia connectivity in PD by analyzing resting-state fMRI data from 111 participants (70 patients with PD; 41 healthy controls) using Dynamic Causal Modeling (DCM). Specifically, the functional connectivity of the basal ganglia was modeled as two second-level, modulatory connections that control projections from sensory cortices to the prefrontal cortex, and from the hippocampus and medial temporal lobe to the prefrontal cortex. We then examined group differences between patients with PD and healthy controls in estimated modulatory effective connectivity in these connections. The Modulatory variant of the Common Model of Cognition outperformed the Direct model across all subjects. It was also found that these second-level modulatory connections had higher estimates of effective connectivity in the PD group compared to the control group, and that differences in effective connectivity were observed for all direct connections between the PD and control groups.We make the case that accounting for modulatory effective connectivity better captures the effects of PD on network functioning and influences the interpretation of the directionality of the between-group results. Limitations include that the PD group was scanned on dopaminergic medication, results were derived from a reasonable but small number of individuals and the ratio of PD to healthy control participants was relatively unbalanced. Future research will examine if the observed effect holds for individuals with PD scanned off their typical dopaminergic medications.

Recovering Reliable Idiographic Biological Parameters from Noisy Behavioral Data: the Case of Basal Ganglia Indices in the Probabilistic Selection Task

Behavioral data, despite being a common index of cognitive activity, is under scrutiny for having poor reliability as a result of noise or lacking replications of reliable effects. Here, we argue that cognitive modeling can be used to enhance the test-retest reliability of the behavioral measures by recovering individual-level parameters from behavioral data. We tested this empirically with the Probabilistic Stimulus Selection (PSS) task, which is used to measure a participant’s sensitivity to positive or negative reinforcement. An analysis of 400,000 simulations from an Adaptive Control of Thought-Rational (ACT-R) model of this task showed that the poor reliability of the task is due to the instability of the end-estimates: because of the way the task works, the same participants might sometimes end up having apparently opposite scores. To recover the underlying interpretable parameters and enhance reliability, we used a Bayesian Maximum A Posteriori (MAP) procedure. We were able to obtain reliable parameters across sessions (intraclass correlation coefficient ≈ 0.5). A follow-up study on a modified version of the task also found the same pattern of results, with very poor test-retest reliability in behavior but moderate reliability in recovered parameters (intraclass correlation coefficient ≈ 0.4). Collectively, these results imply that this approach can further be used to provide superior measures in terms of reliability, and bring greater insights into individual differences.

Core Cognitive Mechanisms Underlying Syntactic Priming: A Comparison of Three Alternative Models

Syntactic priming (SP) is the effect by which, in a dialogue, the current speaker tends to re-use the syntactic constructs of the previous speakers. SP has been used as a window into the nature of syntactic representations within and across languages. Because of its importance, it is crucial to understand the mechanisms behind it. Currently, two competing theories exist. According to the transient activation account, SP is driven by the re-activation of declarative memory structures that encode structures. According to the error-based implicit learning account, SP is driven by prediction errors while processing sentences. By integrating both transient activation and associative learning, Reitter et al.'s hybrid model 2011 assumes that SP is achieved by both mechanisms, and predicts a priming enhancement for rare or unusual constructions. Finally, a recently proposed account, the reinforcement learning account, claims that SP driven by the successful application of procedural knowledge will be reversed when the prime sentence includes grammatical errors. These theories make different assumptions about the representation of syntactic rules (declarative vs. procedural) and the nature of the mechanism that drives priming (frequency and repetition, attention, and feedback signals, respectively). To distinguish between these theories, they were all implemented as computational models in the ACT-R cognitive architecture, and their specific predictions were examined through grid-search computer simulations. Two experiments were then carried out to empirically test the central prediction of each theory as well as the individual fits of each participant's responses to different parameterizations of each model. The first experiment produced results that were best explained by the associative account, but could also be accounted for by a modified reinforcement model with a different parsing algorithm. The second experiment, whose stimuli were designed to avoid the parsing ambiguity of the first, produced somewhat weaker effects. Its results, however, were also best predicted by the model implementing the associative account. We conclude that the data overall points to SP being due to prediction violations that direct attentional resources, in turn suggesting a declarative rather than a RL based procedural representation of syntactic rules.

Corticostriatal Regulation of Language Functions

The role of corticostriatal circuits in language functions is unclear. In this review, we consider evidence from language learning, syntax, and controlled language production and comprehension tasks that implicate various corticostriatal circuits. Converging evidence from neuroimaging in healthy individuals, studies in populations with subcortical dysfunction, pharmacological studies, and brain stimulation suggests a domain-general regulatory role of corticostriatal systems in language operations. The role of corticostriatal systems in language operations identified in this review is likely to reflect a broader function of the striatum in responding to uncertainty and conflict which demands selection, sequencing, and cognitive control. We argue that this role is dynamic and varies depending on the degree and form of cognitive control required, which in turn will recruit particular corticostriatal circuits and components organised in a cognitive hierarchy.

Individual Differences in Reward-Based Learning Predict Fluid Reasoning Abilities

The ability to reason and problem-solve in novel situations, as measured by the Raven's Advanced Progressive Matrices (RAPM), is highly predictive of both cognitive task performance and real-world outcomes. Here we provide evidence that RAPM performance depends on the ability to reallocate attention in response to self-generated feedback about progress. We propose that such an ability is underpinned by the basal ganglia nuclei, which are critically tied to both reward processing and cognitive control. This hypothesis was implemented in a neurocomputational model of the RAPM task, which was used to derive novel predictions at the behavioral and neural levels. These predictions were then verified in one neuroimaging and two behavioral experiments. Furthermore, an effective connectivity analysis of the neuroimaging data confirmed a role for the basal ganglia in modulating attention. Taken together, these results suggest that individual differences in a neural circuit related to reward processing underpin human fluid reasoning abilities.

The Role of Basal Ganglia Reinforcement Learning in Lexical Ambiguity Resolution

The current study aimed to elucidate the contributions of the subcortical basal ganglia to human language by adopting the view that these structures engage in a basic neurocomputation that may account for its involvement across a wide range of linguistic phenomena. Specifically, we tested the hypothesis that basal ganglia reinforcement learning (RL) mechanisms may account for variability in semantic selection processes necessary for ambiguity resolution. To test this, we used a biased homograph lexical ambiguity priming task that allowed us to measure automatic processes for resolving ambiguity toward high-frequency word meanings. Individual differences in task performance were then related to indices of basal ganglia RL, which were used to group subjects into three learning styles: (a) Choosers who learn by seeking high reward probability stimuli; (b) Avoiders, who learn by avoiding low reward probability stimuli; and (c) Balanced participants, whose learning reflects equal contributions of choose and avoid processes. The results suggest that balanced individuals had significantly lower access to subordinate, or low-frequency, homograph word meanings. Choosers and Avoiders, on the other hand, had higher access to the subordinate word meaning even after a long delay between prime and target. Experimental findings were then tested using an ACT-R computational model of RL that learns from both positive and negative feedback. Results from the computational model simulations confirm and extend the pattern of behavioral findings, providing an RL account of individual differences in lexical ambiguity resolution.

Parameter Inference for Computational Cognitive Models with Approximate Bayesian Computation

This paper addresses a common challenge with computational cognitive models: identifying parameter values that are both theoretically plausible and generate predictions that match well with empirical data. While computational models can offer deep explanations of cognition, they are computationally complex and often out of reach of traditional parameter fitting methods. Weak methodology may lead to premature rejection of valid models or to acceptance of models that might otherwise be falsified. Mathematically robust fitting methods are, therefore, essential to the progress of computational modeling in cognitive science. In this article, we investigate the capability and role of modern fitting methods—including Bayesian optimization and approximate Bayesian computation—and contrast them to some more commonly used methods: grid search and Nelder–Mead optimization. Our investigation consists of a reanalysis of the fitting of two previous computational models: an Adaptive Control of Thought—Rational model of skill acquisition and a computational rationality model of visual search. The results contrast the efficiency and informativeness of the methods. A key advantage of the Bayesian methods is the ability to estimate the uncertainty of fitted parameter values. We conclude that approximate Bayesian computation is (a) efficient, (b) informative, and (c) offers a path to reproducible results.