Reinforcement-Based Decision Making in Corticostriatal Circuits: Mutual Constraints by Neurocomputational and Diffusion Models

The subthalamic nucleus contributes causally to perceptual decision-making in monkeys

The subthalamic nucleus (STN) plays critical roles in the motor and cognitive function of the basal ganglia (BG), but the exact nature of these roles is not fully understood, especially in the context of decision-making based on uncertain evidence. Guided by theoretical predictions of specific STN contributions, we used single-unit recording and electrical microstimulation in the STN of healthy monkeys to assess its causal, computational roles in visual-saccadic decisions based on noisy evidence. The recordings identified subpopulations of STN neurons with distinct task-related activity patterns that related to different theoretically predicted functions. Microstimulation caused changes in behavioral choices and response times that reflected multiple contributions to an "accumulate-to-bound"-like decision process, including modulation of decision bounds and evidence accumulation, and to non-perceptual processes. These results provide new insights into the multiple ways that the STN can support higher brain function.

Unraveling the enigmatic role of the subthalamic nucleus

Subpopulations of neurons in the subthalamic nucleus have distinct activity patterns that relate to the three hypotheses of the Drift Diffusion Model.

Parkinson's disease impairs cortical sensori-motor decision-making cascades

The transformation from perception to action requires a set of neuronal decisions about the nature of the percept, identification and selection of response options and execution of the appropriate motor response. The unfolding of such decisions is mediated by distributed representations of the decision variables-evidence and intentions-that are represented through oscillatory activity across the cortex. Here we combine magneto-electroencephalography and linear ballistic accumulator models of decision-making to reveal the impact of Parkinson's disease during the selection and execution of action. We used a visuomotor task in which we independently manipulated uncertainty in sensory and action domains. A generative accumulator model was optimized to single-trial neurophysiological correlates of human behaviour, mapping the cortical oscillatory signatures of decision-making, and relating these to separate processes accumulating sensory evidence and selecting a motor action. We confirmed the role of widespread beta oscillatory activity in shaping the feed-forward cascade of evidence accumulation from resolution of sensory inputs to selection of appropriate responses. By contrasting the spatiotemporal dynamics of evidence accumulation in age-matched healthy controls and people with Parkinson's disease, we identified disruption of the beta-mediated cascade of evidence accumulation as the hallmark of atypical decision-making in Parkinson's disease. In frontal cortical regions, there was inefficient processing and transfer of perceptual information. Our findings emphasize the intimate connection between abnormal visuomotor function and pathological oscillatory activity in neurodegenerative disease. We propose that disruption of the oscillatory mechanisms governing fast and precise information exchanges between the sensory and motor systems contributes to behavioural changes in people with Parkinson's disease.

Reinforcement learning of adaptive control strategies

Humans can up- or downregulate the degree to which they rely on task information for goal-directed behaviour, a process often referred to as cognitive control. Adjustments in cognitive control are traditionally studied in response to experienced or expected task-rule conflict. However, recent theories suggest that people can also learn to adapt control settings through reinforcement. Across three preregistered task switching experiments (n = 415), we selectively rewarded correct performance on trials with either more (incongruent) or less (congruent) task-rule conflict. Results confirmed the hypothesis that people rewarded more on incongruent trials showed smaller task-rule congruency effects, thus optimally adapting their control settings to the reward scheme. Using drift diffusion modelling, we further show that this reinforcement of cognitive control may occur through conflict-dependent within-trial adjustments of response thresholds after conflict detection. Together, our findings suggest that, while people remain more efficient at learning stimulus-response associations through reinforcement, they can similarly learn cognitive control strategies through reinforcement.

Competing neural representations of choice shape evidence accumulation in humans

Making adaptive choices in dynamic environments requires flexible decision policies. Previously, we showed how shifts in outcome contingency change the evidence accumulation process that determines decision policies. Using in silico experiments to generate predictions, here we show how the cortico-basal ganglia-thalamic (CBGT) circuits can feasibly implement shifts in decision policies. When action contingencies change, dopaminergic plasticity redirects the balance of power, both within and between action representations, to divert the flow of evidence from one option to another. When competition between action representations is highest, the rate of evidence accumulation is the lowest. This prediction was validated in in vivo experiments on human participants, using fMRI, which showed that (1) evoked hemodynamic responses can reliably predict trial-wise choices and (2) competition between action representations, measured using a classifier model, tracked with changes in the rate of evidence accumulation. These results paint a holistic picture of how CBGT circuits manage and adapt the evidence accumulation process in mammals.

Contributions of the Basal Ganglia to Visual Perceptual Decisions

The basal ganglia (BG) make up a prominent nexus between visual and motor-related brain regions. In contrast to the BG's well-established roles in movement control and value-based decision making, their contributions to the transformation of visual input into an action remain unclear, especially in the context of perceptual decisions based on uncertain visual evidence. This article reviews recent progress in our understanding of the BG's contributions to the formation, evaluation, and adjustment of such decisions. From theoretical and experimental perspectives, the review focuses on four key stations in the BG network, namely, the striatum, pallidum, subthalamic nucleus, and midbrain dopamine neurons, which can have different roles and together support the decision process.

Measuring utility with diffusion models

The drift diffusion model (DDM) is a prominent account of how people make decisions. Many of these decisions involve comparing two alternatives based on differences of perceived stimulus magnitudes, such as economic values. Here, we propose a consistent estimator for the parameters of a DDM in such cases. This estimator allows us to derive decision thresholds, drift rates, and subjective percepts (i.e., utilities in economic choice) directly from the experimental data. This eliminates the need to measure these values separately or to assume specific functional forms for them. Our method also allows one to predict drift rates for comparisons that did not occur in the dataset. We apply the method to two datasets, one comparing probabilities of earning a fixed reward and one comparing objects of variable reward value. Our analysis indicates that both datasets conform well to the DDM. We find that utilities are linear in probability and slightly convex in reward.

Learning how to reason and deciding when to decide

Research on human reasoning has both popularized and struggled with the idea that intuitive and deliberate thoughts stem from two different systems, raising the question how people switch between them. Inspired by research on cognitive control and conflict monitoring, we argue that detecting the need for further thought relies on an intuitive, context-sensitive process that is learned in itself.

Pathways to the persistence of drug use despite its adverse consequences

The persistence of drug taking despite its adverse consequences plays a central role in the presentation, diagnosis, and impacts of addiction. Eventual recognition and appraisal of these adverse consequences is central to decisions to reduce or cease use. However, the most appropriate ways of conceptualizing persistence in the face of adverse consequences remain unclear. Here we review evidence that there are at least three pathways to persistent use despite the negative consequences of that use. A cognitive pathway for recognition of adverse consequences, a motivational pathway for valuation of these consequences, and a behavioral pathway for responding to these adverse consequences. These pathways are dynamic, not linear, with multiple possible trajectories between them, and each is sufficient to produce persistence. We describe these pathways, their characteristics, brain cellular and circuit substrates, and we highlight their relevance to different pathways to self- and treatment-guided behavior change.

Integrated diffusion models for distance effects in number memory

I evaluated three models for the representation of numbers in memory. These were integrated with the diffusion decision model to explain accuracy and response time (RT) data from a recognition memory experiment in which the stimuli were two-digit numbers. The integrated models accounted for distance/confusability effects: when a test number was numerically close to a studied number, accuracy was lower and RTs were longer than when a test number was numerically far from a studied number. For two of the models, the representations of numbers are distributed over number (with Gaussian or exponential distributions) and the overlap between the distributions of a studied number and a test number provides the evidence (drift rate) on which a decision is made. For the third, the exponential gradient model, drift rate is an exponential function of the numerical distance between studied and test numbers. The exponential gradient model fit the data slightly better than the two overlap models. Monte Carlo simulations showed that the variability in the important parameter estimates from fitting data collected over 30-40 min is smaller than the variability among individuals, allowing differences among individuals to be studied. A second experiment compared number memory and number discrimination tasks and results showed different distance effects. Number memory had an exponential-like distance-effect and number discrimination had a linear function which shows radically different representations drive the two tasks.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Relating a Spiking Neural Network Model and the Diffusion Model of Decision-Making

Many models of decision making assume accumulation of evidence to threshold as a core mechanism to predict response probabilities and response times. A spiking neural network model (Wang, 2002) instantiates these mechanisms at the level of biophysically-plausible pools of neurons with excitatory and inhibitory connections, and has numerous model parameters tuned by physiological measures. The diffusion model (Ratcliff, 1978) is a cognitive model that can be fitted to a range of behaviors and conditions. We investigated how parameters of the cognitive-level diffusion model relate to the parameters of a neural-level spiking model. In each simulated "experiment", we generated "data" from the spiking neural network by factorially combining a manipulation of choice difficulty (via the input to the spiking model) and a manipulation of one of the core parameters of the spiking model. We then fitted the diffusion model to these simulated data to observe how manipulation of each core spiking model parameter mapped on to fitted drift rate, response threshold, and non-decision time. Manipulations of parameters in the spiking model related to input sensitivity, threshold, and stimulus processing time mapped on to their conceptual analogues in the diffusion model, namely drift rate, threshold, and non-decision time. Manipulations of parameters in the spiking model with no direct analogue to the diffusion model, non-stimulus-specific background input, strength of recurrent excitation, and receptor conductances, mapped on to threshold in the diffusion model. We discuss implications of these results for interpretations of fits of the diffusion model to behavioral data.

Beyond Drift Diffusion Models: Fitting a Broad Class of Decision and Reinforcement Learning Models with HDDM

Computational modeling has become a central aspect of research in the cognitive neurosciences. As the field matures, it is increasingly important to move beyond standard models to quantitatively assess models with richer dynamics that may better reflect underlying cognitive and neural processes. For example, sequential sampling models (SSMs) are a general class of models of decision-making intended to capture processes jointly giving rise to RT distributions and choice data in n-alternative choice paradigms. A number of model variations are of theoretical interest, but empirical data analysis has historically been tied to a small subset for which likelihood functions are analytically tractable. Advances in methods designed for likelihood-free inference have recently made it computationally feasible to consider a much larger spectrum of SSMs. In addition, recent work has motivated the combination of SSMs with reinforcement learning models, which had historically been considered in separate literatures. Here, we provide a significant addition to the widely used HDDM Python toolbox and include a tutorial for how users can easily fit and assess a (user-extensible) wide variety of SSMs and how they can be combined with reinforcement learning models. The extension comes batteries included, including model visualization tools, posterior predictive checks, and ability to link trial-wise neural signals with model parameters via hierarchical Bayesian regression.

Identifying control ensembles for information processing within the cortico-basal ganglia-thalamic circuit

In situations featuring uncertainty about action-reward contingencies, mammals can flexibly adopt strategies for decision-making that are tuned in response to environmental changes. Although the cortico-basal ganglia thalamic (CBGT) network has been identified as contributing to the decision-making process, it features a complex synaptic architecture, comprised of multiple feed-forward, reciprocal, and feedback pathways, that complicate efforts to elucidate the roles of specific CBGT populations in the process by which evidence is accumulated and influences behavior. In this paper we apply a strategic sampling approach, based on Latin hypercube sampling, to explore how variations in CBGT network properties, including subpopulation firing rates and synaptic weights, map to variability of parameters in a normative drift diffusion model (DDM), representing algorithmic aspects of information processing during decision-making. Through the application of canonical correlation analysis, we find that this relationship can be characterized in terms of three low-dimensional control ensembles within the CBGT network that impact specific qualities of the emergent decision policy: responsiveness (a measure of how quickly evidence evaluation gets underway, associated with overall activity in corticothalamic and direct pathways), pliancy (a measure of the standard of evidence needed to commit to a decision, associated largely with overall activity in components of the indirect pathway of the basal ganglia), and choice (a measure of commitment toward one available option, associated with differences in direct and indirect pathways across action channels). These analyses provide mechanistic predictions about the roles of specific CBGT network elements in tuning the way that information is accumulated and translated into decision-related behavior.

Biophysical and Architectural Mechanisms of Subthalamic Theta under Response Conflict

The cortico-basal ganglia circuit is needed to suppress prepotent actions and to facilitate controlled behavior. Under conditions of response conflict, the frontal cortex and subthalamic nucleus (STN) exhibit increased spiking and theta band power, which are linked to adaptive regulation of behavioral output. The electrophysiological mechanisms underlying these neural signatures of impulse control remain poorly understood. To address this lacuna, we constructed a novel large-scale, biophysically principled model of the subthalamopallidal (STN-globus pallidus externus) network and examined the mechanisms that modulate theta power and spiking in response to cortical input. Simulations confirmed that theta power does not emerge from intrinsic network dynamics but is robustly elicited in response to cortical input as burst events representing action selection dynamics. Rhythmic burst events of multiple cortical populations, representing a state of conflict where cortical motor plans vacillate in the theta range, led to prolonged STN theta and increased spiking, consistent with empirical literature. Notably, theta band signaling required NMDA, but not AMPA, currents, which were in turn related to a triphasic STN response characterized by spiking, silence, and bursting periods. Finally, theta band resonance was also strongly modulated by architectural connectivity, with maximal theta arising when multiple cortical populations project to individual STN "conflict detector" units because of an NMDA-dependent supralinear response. Our results provide insights into the biophysical principles and architectural constraints that give rise to STN dynamics during response conflict, and how their disruption can lead to impulsivity and compulsivity.SIGNIFICANCE STATEMENT The subthalamic nucleus exhibits theta band power modulation related to cognitive control over motor actions during conditions of response conflict. However, the mechanisms of such dynamics are not understood. Here we developed a novel biophysically detailed and data-constrained large-scale model of the subthalamopallidal network, and examined the impacts of cellular and network architectural properties that give rise to theta dynamics. Our investigations implicate an important role for NMDA receptors and cortico-subthalamic nucleus topographical connectivities in theta power modulation.

The role of state uncertainty in the dynamics of dopamine

Reinforcement learning models of the basal ganglia map the phasic dopamine signal to reward prediction errors (RPEs). Conventional models assert that, when a stimulus predicts a reward with fixed delay, dopamine activity during the delay should converge to baseline through learning. However, recent studies have found that dopamine ramps up before reward in certain conditions even after learning, thus challenging the conventional models. In this work, we show that sensory feedback causes an unbiased learner to produce RPE ramps. Our model predicts that when feedback gradually decreases during a trial, dopamine activity should resemble a "bump," whose ramp-up phase should, furthermore, be greater than that of conditions where the feedback stays high. We trained mice on a virtual navigation task with varying brightness, and both predictions were empirically observed. In sum, our theoretical and experimental results reconcile the seemingly conflicting data on dopamine behaviors under the RPE hypothesis.

Cognitive Control as a Multivariate Optimization Problem

A hallmark of adaptation in humans and other animals is our ability to control how we think and behave across different settings. Research has characterized the various forms cognitive control can take-including enhancement of goal-relevant information, suppression of goal-irrelevant information, and overall inhibition of potential responses-and has identified computations and neural circuits that underpin this multitude of control types. Studies have also identified a wide range of situations that elicit adjustments in control allocation (e.g., those eliciting signals indicating an error or increased processing conflict), but the rules governing when a given situation will give rise to a given control adjustment remain poorly understood. Significant progress has recently been made on this front by casting the allocation of control as a decision-making problem. This approach has developed unifying and normative models that prescribe when and how a change in incentives and task demands will result in changes in a given form of control. Despite their successes, these models, and the experiments that have been developed to test them, have yet to face their greatest challenge: deciding how to select among the multiplicity of configurations that control can take at any given time. Here, we will lay out the complexities of the inverse problem inherent to cognitive control allocation, and their close parallels to inverse problems within motor control (e.g., choosing between redundant limb movements). We discuss existing solutions to motor control's inverse problems drawn from optimal control theory, which have proposed that effort costs act to regularize actions and transform motor planning into a well-posed problem. These same principles may help shed light on how our brains optimize over complex control configuration, while providing a new normative perspective on the origins of mental effort.

Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits

Adaptive sequential behavior is a hallmark of human cognition. In particular, humans can learn to produce precise spatiotemporal sequences given a certain context. For instance, musicians can not only reproduce learned action sequences in a context-dependent manner, they can also quickly and flexibly reapply them in any desired tempo or rhythm without overwriting previous learning. Existing neural network models fail to account for these properties. We argue that this limitation emerges from the fact that sequence information (i.e., the position of the action) and timing (i.e., the moment of response execution) are typically stored in the same neural network weights. Here, we augment a biologically plausible recurrent neural network of cortical dynamics to include a basal ganglia-thalamic module which uses reinforcement learning to dynamically modulate action. This “associative cluster-dependent chain” (ACDC) model modularly stores sequence and timing information in distinct loci of the network. This feature increases computational power and allows ACDC to display a wide range of temporal properties (e.g., multiple sequences, temporal shifting, rescaling, and compositionality), while still accounting for several behavioral and neurophysiological empirical observations. Finally, we apply this ACDC network to show how it can learn the famous “Thunderstruck” song intro and then flexibly play it in a “bossa nova” rhythm without further training.

How do humans flexibly adapt action sequences? For instance, musicians can learn a song and quickly speed up or slow down the tempo, or even play the song following a completely different rhythm (e.g., a rock song using a bossa nova rhythm). In this work, we build a biologically plausible network of cortico-basal ganglia interactions that explains how this temporal flexibility may emerge in the brain. Crucially, our model factorizes sequence order and action timing, respectively represented in cortical and basal ganglia dynamics. This factorization allows full temporal flexibility, i.e. the timing of a learned action sequence can be recomposed without interfering with the order of the sequence. As such, our model is capable of learning asynchronous action sequences, and flexibly shift, rescale, and recompose them, while accounting for biological data.

Neural Substrates of the Drift-Diffusion Model in Brain Disorders

Many studies on the drift-diffusion model (DDM) explain decision-making based on a unified analysis of both accuracy and response times. This review provides an in-depth account of the recent advances in DDM research which ground different DDM parameters on several brain areas, including the cortex and basal ganglia. Furthermore, we discuss the changes in DDM parameters due to structural and functional impairments in several clinical disorders, including Parkinson's disease, Attention Deficit Hyperactivity Disorder (ADHD), Autism Spectrum Disorders, Obsessive-Compulsive Disorder (OCD), and schizophrenia. This review thus uses DDM to provide a theoretical understanding of different brain disorders.

Dynamic decision policy reconfiguration under outcome uncertainty

In uncertain or unstable environments, sometimes the best decision is to change your mind. To shed light on this flexibility, we evaluated how the underlying decision policy adapts when the most rewarding action changes. Human participants performed a dynamic two-armed bandit task that manipulated the certainty in relative reward (conflict) and the reliability of action-outcomes (volatility). Continuous estimates of conflict and volatility contributed to shifts in exploratory states by changing both the rate of evidence accumulation (drift rate) and the amount of evidence needed to make a decision (boundary height), respectively. At the trialwise level, following a switch in the optimal choice, the drift rate plummets and the boundary height weakly spikes, leading to a slow exploratory state. We find that the drift rate drives most of this response, with an unreliable contribution of boundary height across experiments. Surprisingly, we find no evidence that pupillary responses associated with decision policy changes. We conclude that humans show a stereotypical shift in their decision policies in response to environmental changes.

The critical role of interference control in metaphor comprehension evidenced by the drift-diffusion model

We address the question of, among several executive functions, which one has a strong influence on metaphor comprehension. To this end, participants took part in a metaphor comprehension task where metaphors had varying levels of familiarity (familiar vs. novel metaphors) with different conditions of context (supporting vs. opposing contexts). We scrutinized each participant's detailed executive functions using seven neuropsychological tests. More interestingly, we modelled their responses in metaphor comprehension using the drift-diffusion model, in an attempt to provide more systematic accounts of the processes underlying metaphor comprehension. Results showed that there were significant negative correlations between response times in metaphor comprehension and scores of the Controlled Oral Word Association Test (COWAT)-Semantic, suggesting that better performances in comprehending metaphors were strongly associated with better interference control. Using the drift-diffusion model, we found that the familiarity, compared to context, had greater leverage in the decision process for metaphor comprehension. Moreover, individuals with better performance in the COWAT-Semantic test demonstrated higher drift rates. In conclusion, with more fine-grained analysis of the decisions involved in metaphor comprehension using the drift-diffusion model, we argue that interference control plays an important role in processing metaphors.

Analogous computations in working memory input, output and motor gating: Electrophysiological and computational modeling evidence

Adaptive cognitive-control involves a hierarchical cortico-striatal gating system that supports selective updating, maintenance, and retrieval of useful cognitive and motor information. Here, we developed a task that independently manipulates selective gating operations into working-memory (input gating), from working-memory (output gating), and of responses (motor gating) and tested the neural dynamics and computational principles that support them. Increases in gating demands, captured by gate switches, were expressed by distinct EEG correlates at each gating level that evolved dynamically in partially overlapping time windows. Further, categorical representations of specific maintained items and of motor responses could be decoded from EEG when the corresponding gate was switching, thereby linking gating operations to prioritization. Finally, gate switching at all levels was related to increases in the motor decision threshold as quantified by the drift diffusion model. Together these results support the notion that cognitive gating operations scaffold on top of mechanisms involved in motor gating.

Likelihood approximation networks (LANs) for fast inference of simulation models in cognitive neuroscience

In cognitive neuroscience, computational modeling can formally adjudicate between theories and affords quantitative fits to behavioral/brain data. Pragmatically, however, the space of plausible generative models considered is dramatically limited by the set of models with known likelihood functions. For many models, the lack of a closed-form likelihood typically impedes Bayesian inference methods. As a result, standard models are evaluated for convenience, even when other models might be superior. Likelihood-free methods exist but are limited by their computational cost or their restriction to particular inference scenarios. Here, we propose neural networks that learn approximate likelihoods for arbitrary generative models, allowing fast posterior sampling with only a one-off cost for model simulations that is amortized for future inference. We show that these methods can accurately recover posterior parameter distributions for a variety of neurocognitive process models. We provide code allowing users to deploy these methods for arbitrary hierarchical model instantiations without further training.

Integrity of the Prefronto-striato-thalamo-prefrontal Loop Predicts Tai Chi Chuan Training Effects on Cognitive Task-switching in Middle-aged and Older Adults

Tai Chi Chuan (TCC) exercise has been shown to improve cognitive task-switching performance in older adults, but the extent of this positive effect varies among individuals. Past research also shows that brain white matter integrity could predict behavioral gains of cognitive and motor learning. Therefore, in this randomized controlled trial (NCT02270320), we examined whether baseline integrity of three target white matter tract groups was predictive of task-switching improvement after 12-week TCC training in middle-aged and older adults. Thirty-eight eligible participants were randomly assigned to a TCC group (n = 19) and a control group (n = 19). Cognitive task-switching and physical performances were collected before and after training. Brain diffusion spectrum MR images were acquired before training and the general fractional anisotropy (GFA) of each target white matter tract group was calculated to indicate baseline white matter integrity of that group. Correlation and regression analyses between these GFAs and post-training task-switching improvement were analyzed using adjusted p-values. After 12 weeks, significant task-switching and physical performance improvements were found only in the TCC group. Moreover, higher baseline GFA of the prefronto-striato-thalamo-prefrontal loop fibers (r = −0.63, p = 0.009), but not of the prefronto-parietal/occipital (r = −0.55, p = 0.026) and callosal (r = −0.35, p = 0.189) fiber groups, was associated with greater reductions of task-switching errors after the TCC training. Multiple regression analysis revealed that baseline GFA of the prefronto-striato-thalamo-prefrontal loop fibers was the only independent white matter integrity predictor of task-switching error reductions after TCC training (β = −0.620, adjusted R² change = 0.265, p = 0.009). These findings not only highlight the important role of baseline integrity of the prefronto-striatal circuits in influencing the extent of positive cognitive task-switching effects from short-term TCC training, but also implicate that preserving good white matter integrity in the aging process may be crucial in order to gain the best cognitive effects of exercise interventions.

Human visual search follows a suboptimal Bayesian strategy revealed by a spatiotemporal computational model and experiment

There is conflicting evidence regarding whether humans can make spatially optimal eye movements during visual search. Some studies have shown that humans can optimally integrate information across fixations and determine the next fixation location, however, these models have generally ignored the control of fixation duration and memory limitation, and the model results do not agree well with the details of human eye movement metrics. Here, we measured the temporal course of the human visibility map and performed a visual search experiment. We further built a continuous-time eye movement model that considers saccadic inaccuracy, saccadic bias, and memory constraints. We show that this model agrees better with the spatial and temporal properties of human eye movements and predict that humans have a memory capacity of around eight previous fixations. The model results reveal that humans employ a suboptimal eye movement strategy to find a target, which may minimize costs while still achieving sufficiently high search performance.

Adiposity covaries with signatures of asymmetric feedback learning during adaptive decisions

Unhealthy weight gain relates, in part, to how people make decisions based on prior experience. Here we conducted post hoc analysis on an archival data set to evaluate whether individual differences in adiposity, an anthropometric construct encompassing a spectrum of body types, from lean to obese, associate with signatures of asymmetric feedback learning during value-based decision-making. In a sample of neurologically healthy adults (N = 433), ventral striatal responses to rewards, measured using fMRI, were not directly associated with adiposity, but rather moderated its relationship with feedback-driven learning in the Iowa gambling task, tested outside the scanner. Using a biologically inspired model of basal ganglia-dependent decision processes, we found this moderating effect of reward reactivity to be explained by an asymmetrical use of feedback to drive learning; that is, with more plasticity for gains than for losses, stronger reward reactivity leads to decisions that minimize exploration for maximizing long-term outcomes. Follow-up analysis confirmed that individual differences in adiposity correlated with signatures of asymmetric use of feedback cues during learning, suggesting that reward reactivity may especially relate to adiposity, and possibly obesity risk, when gains impact future decisions more than losses.

Examining aging and numerosity using an integrated diffusion model

Two experiments are presented that use tasks common in research in numerical cognition with young adults and older adults as subjects. In these tasks, one or two arrays of dots are displayed, and subjects decide whether there are more or fewer dots of one kind than another. Results show that older adults, relative to young adults, tend to rely more on the perceptual feature, area, in making numerosity judgments when area is correlated with numerosity. Also, convex hull unexpectedly shows different effects depending on the task (being either correlated with numerosity or anticorrelated). Accuracy and response time (RT) data are interpreted with the integration of the diffusion decision model with models for the representation of numerosity. One model assumes that the representation of the difference depends on the difference between the numerosities and that standard deviations (SDs) increase linearly with numerosity, and the other model assumes a log representation with constant SDs. The representational models have coefficients that are applied to differences between two numerosities to produce drift rates and SDs in drift rates in the decision process. The two tasks produce qualitatively different patterns of RTs: One model fits results from one task, but the results are mixed for the other task. The effects of age on model parameters show a modest decrease in evidence driving the decision process, an increase in the duration of processes outside the decision process (nondecision time), and an increase in the amount of evidence needed to make a decision (boundary separation). (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making

Despite recent progress in understanding multisensory decision-making, a conclusive mechanistic account of how the brain translates the relevant evidence into a decision is lacking. Specifically, it remains unclear whether perceptual improvements during rapid multisensory decisions are best explained by sensory (i.e., ‘Early’) processing benefits or post-sensory (i.e., ‘Late’) changes in decision dynamics. Here, we employ a well-established visual object categorisation task in which early sensory and post-sensory decision evidence can be dissociated using multivariate pattern analysis of the electroencephalogram (EEG). We capitalize on these distinct neural components to identify when and how complementary auditory information influences the encoding of decision-relevant visual evidence in a multisensory context. We show that it is primarily the post-sensory, rather than the early sensory, EEG component amplitudes that are being amplified during rapid audiovisual decision-making. Using a neurally informed drift diffusion model we demonstrate that a multisensory behavioral improvement in accuracy arises from an enhanced quality of the relevant decision evidence, as captured by the post-sensory EEG component, consistent with the emergence of multisensory evidence in higher-order brain areas.

A conclusive account on how the brain translates audiovisual evidence into a rapid decision is still lacking. Here, using a neurally-informed modelling approach, the authors show that sounds amplify visual evidence later in the decision process, in line with higher-order multisensory effects.

Cross-Task Contributions of Frontobasal Ganglia Circuitry in Response Inhibition and Conflict-Induced Slowing

Why are we so slow in choosing the lesser of 2 evils? We considered whether such slowing relates to uncertainty about the value of these options, which arises from the tendency to avoid them during learning, and whether such slowing relates to frontosubthalamic inhibitory control mechanisms. In total, 49 participants performed a reinforcement-learning task and a stop-signal task while fMRI was recorded. A reinforcement-learning model was used to quantify learning strategies. Individual differences in lose-lose slowing related to information uncertainty due to sampling, and independently, to less efficient response inhibition in the stop-signal task. Neuroimaging analysis revealed an analogous dissociation: subthalamic nucleus (STN) BOLD activity related to variability in stopping latencies, whereas weaker frontosubthalamic connectivity related to slowing and information sampling. Across tasks, fast inhibitors increased STN activity for successfully canceled responses in the stop task, but decreased activity for lose-lose choices. These data support the notion that fronto-STN communication implements a rapid but transient brake on response execution, and that slowing due to decision uncertainty could result from an inefficient release of this "hold your horses" mechanism.

Prosocial Preferences Condition Decision Effort and Ingroup Biased Generosity in Intergroup Decision-Making

Ingroup favoritism and discrimination against outgroups are pervasive in social interactions. To uncover the cognitive processes underlying generosity towards in- and outgroup members, we employ eye-tracking in two pre registered studies. We replicate the well-established ingroup favoritism effect and uncover that ingroup compared to outgroup decision settings are characterized by systematic differences in information search effort (i.e., increased response times and number of fixations, more inspected information) and attention distribution. Surprisingly, these results showed a stronger dependency on the in- vs. out-group setting for more individualistic compared to prosocial participants: Whereas individualistic decision makers invested relatively less effort into information search when decisions involved out-group members, prosocial decision makers’ effort differed less between in- and outgroup decisions. Therein, choice and processing findings showed differences, indicating that inferences about the decision process from choices alone can be misleading. Implications for intergroup research and the regulation of intergroup conflict are discussed.

The caudate nucleus contributes causally to decisions that balance reward and uncertain visual information

Our decisions often balance what we observe and what we desire. A prime candidate for implementing this complex balancing act is the basal ganglia pathway, but its roles have not yet been examined experimentally in detail. Here, we show that a major input station of the basal ganglia, the caudate nucleus, plays a causal role in integrating uncertain visual evidence and reward context to guide adaptive decision-making. In monkeys making saccadic decisions based on motion cues and asymmetric reward-choice associations, single caudate neurons encoded both sources of information. Electrical microstimulation at caudate sites during motion viewing affected the monkeys’ decisions. These microstimulation effects included coordinated changes in multiple computational components of the decision process that mimicked the monkeys’ similarly coordinated voluntary strategies for balancing visual and reward information. These results imply that the caudate nucleus plays causal roles in coordinating decision processes that balance external evidence and internal preferences.

Simultaneous Hierarchical Bayesian Parameter Estimation for Reinforcement Learning and Drift Diffusion Models: a Tutorial and Links to Neural Data

Cognitive models have been instrumental for generating insights into the brain processes underlying learning and decision making. In reinforcement learning it has recently been shown that not only choice proportions but also their latency distributions can be well captured when the choice function is replaced with a sequential sampling model such as the drift diffusion model. Hierarchical Bayesian parameter estimation further enhances the identifiability of distinct learning and choice parameters. One caveat is that these models can be time-consuming to build, sample from, and validate, especially when models include links between neural activations and model parameters. Here we describe a novel extension to the widely used hierarchical drift diffusion model (HDDM) toolbox, which facilitates flexible construction, estimation, and evaluation of the reinforcement learning drift diffusion model (RLDDM) using hierarchical Bayesian methods. We describe the types of experiments most applicable to the model and provide a tutorial to illustrate how to perform quantitative data analysis and model evaluation. Parameter recovery confirmed that the method can reliably estimate parameters with varying numbers of synthetic subjects and trials. We also show that the simultaneous estimation of learning and choice parameters can improve the sensitivity to detect brain–behavioral relationships, including the impact of learned values and fronto-basal ganglia activity patterns on dynamic decision parameters.

Modeling the interaction of numerosity and perceptual variables with the diffusion model

Ratcliff and McKoon (2018) proposed integrated diffusion models for numerosity judgments in which a numerosity representation provides evidence used to drive the decision process. We extend this modeling framework to examine the interaction of non-numeric perceptual variables with numerosity by assuming that drift rate and non-decision time are functions of those variables. Four experiments were conducted with two different types of stimuli: a single array of intermingled blue and yellow dots in which both numerosity and dot area vary over trials and two side-by-side arrays of dots in which numerosity, dot area, and convex hull vary over trials. The tasks were to decide whether there were more blue or yellow dots (two experiments), more dots on which side, or which dots have a larger total area. Development of models started from the principled models in Ratcliff and McKoon (2018) and became somewhat ad hoc as we attempted to capture unexpected patterns induced by the conflict between numerosity and perceptual variables. In the three tasks involving numerosity judgments, the effects of the non-numeric variables were moderated by the number of dots. Under a high conflict, judgments were dominated by perceptual variables and produced an unexpected shift in the leading edge of the reaction time (RT) distributions. Although the resulting models were able to predict most of the accuracy and RT patterns, the models were not able to completely capture this shift in the RT distributions. However, when subjects judged area, numerosity affected perceptual judgments but there was no leading edge effect. Based on the results, it appears that the integrated diffusion models provide an effective framework to study the role of numerical and perceptual variables in numerosity tasks and their context-dependency.

Incentive motivation improves numerosity discrimination: Insights from pupillometry combined with drift-diffusion modelling

Recent studies show that training the approximate number system (ANS) holds promise for improving symbolic math abilities. Extending this line of research, the present study aims to shed light on incentive motivation of numerosity discrimination and the underlying mechanisms. Thirty-two young adults performed a novel incentivized dot comparison task, that we developed, to discern the larger of two numerosities. An EZ-diffusion model was applied to decompose motivational effects on component processes of perceptual decision-making. Furthermore, phasic pupil dilation served as an indicator of the involvement of the salience network. The results of improved accuracy and a higher information accumulation rate under the reward condition suggest that incentive motivation boosts the precision of the ANS. These novel findings extend earlier evidence on reward-related enhancements of perceptual discrimination to the domain of numerosity perception. In light of the Adaptive Gain Theory, we interpret the results in terms of two processes of gain modulation driven by the locus coeruleus-norepinephrine system. Specifically, the reward-induced increase in pupil dilation may reflect incentive modulation of (i) salience attention during reward anticipation towards incentivized stimuli to upregulate stimulus processing that results in a larger drift rate; and (ii) response caution that leads to an increased decision threshold.

Uncertainty in learning, choice, and visual fixation

Uncertainty plays a critical role in reinforcement learning and decision making. However, exactly how it influences behavior remains unclear. Multiarmed-bandit tasks offer an ideal test bed, since computational tools such as approximate Kalman filters can closely characterize the interplay between trial-by-trial values, uncertainty, learning, and choice. To gain additional insight into learning and choice processes, we obtained data from subjects' overt allocation of gaze. The estimated value and estimation uncertainty of options influenced what subjects looked at before choosing; these same quantities also influenced choice, as additionally did fixation itself. A momentary measure of uncertainty in the form of absolute prediction errors determined how long participants looked at the obtained outcomes. These findings affirm the importance of uncertainty in multiple facets of behavior and help delineate its effects on decision making.

Does response modality influence conflict? Modelling vocal and manual response Stroop interference

In the Stroop task, color words are presented in colored fonts and the task of the subject is to either name the word or name the color. If the word and font color are in agreement, then the stimulus is said to be congruent (e.g., RED in red font color); however, if the word and font color are not in agreement, the stimulus is said to be incongruent (e.g., RED in blue font color). Conflict in the Stroop task is measured by both RT and accuracy. In prior research, the amount of conflict differs depending on the response modality, vocal versus manual. We applied a model for multichoice decision-making (and confidence), the RTCON2 model (Ratcliff & Starns, 2013) to the data from 4 experiments, 2 with 2-choice manual responses, 1 with 4-choice manual touch screen responses, 1 with both 4-choice vocal responses, and 4-choice manual keyboard responses. Changes in the rate of information accumulation captured conflict effects for the manual-response versions, but not for the vocal-response version. Adding an extra nondecision time parameter allowed RTCON2 to account for the data patterns in the vocal-response version. However, to fully understand conflict in the vocal-response Stroop task, a model of conflict processing in the vocal word production system must be developed that would explain the additional processing time in the nondecision time parameter. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Mid-Frontal Theta Modulates Response Inhibition and Decision Making Processes in Emotional Contexts

Inhibitory control is an integral part of executive functions. In this study, we report event-related spectral perturbation (ERSP) results from 15 healthy adults performing an emotional stop-signal task with the use of happy, disgusted, and neutral emotional faces. Our ERSP results at the group level suggest that changes in low frequency oscillatory power for emotional and neutral conditions start at as early as 200 ms after stimulus onset and 300 ms before button press for successful go trials. To quantify the dynamics of trial-by-trial theta power, we applied the hierarchical drift diffusion model to single-trial ERSP at the mid-frontal electrode site for the go condition. Hierarchical drift diffusion modeling (HDDM) assigned higher frontal low-frequency oscillatory power for evidence accumulation in emotional contexts as compared to a neutral setting. Our results provide new evidence for dynamic modulation of sensory processing of go stimuli in inhibition and extend our knowledge for processing of response inhibition in emotional contexts.

Combining error-driven models of associative learning with evidence accumulation models of decision-making

As people learn a new skill, performance changes along two fundamental dimensions: Responses become progressively faster and more accurate. In cognitive psychology, these facets of improvement have typically been addressed by separate classes of theories. Reductions in response time (RT) have usually been addressed by theories of skill acquisition, whereas increases in accuracy have been explained by associative learning theories. To date, relatively little work has examined how changes in RT relate to changes in response accuracy, and whether these changes can be accounted for quantitatively within a single theoretical framework. The current work examines joint changes in accuracy and RT in a probabilistic category learning task. We report a model-based analysis of changes in the shapes of RT distributions for different category responses at the level of individual stimuli over the course of learning. We show that changes in performance are determined solely by changes in the quality of information entering the decision process. We then develop a new model that combines an associative learning front end with a sequential sampling model of the decision process, showing that the model provides a good account of all aspects of the learning data. We conclude by discussing potential extensions of the model and future directions for theoretical development that are opened up by our findings.

Decomposing the effects of context valence and feedback information on speed and accuracy during reinforcement learning: a meta-analytical approach using diffusion decision modeling

Reinforcement learning (RL) models describe how humans and animals learn by trial-and-error to select actions that maximize rewards and minimize punishments. Traditional RL models focus exclusively on choices, thereby ignoring the interactions between choice preference and response time (RT), or how these interactions are influenced by contextual factors. However, in the field of perceptual decision-making, such interactions have proven to be important to dissociate between different underlying cognitive processes. Here, we investigated such interactions to shed new light on overlooked differences between learning to seek rewards and learning to avoid losses. We leveraged behavioral data from four RL experiments, which feature manipulations of two factors: outcome valence (gains vs. losses) and feedback information (partial vs. complete feedback). A Bayesian meta-analysis revealed that these contextual factors differently affect RTs and accuracy: While valence only affects RTs, feedback information affects both RTs and accuracy. To dissociate between the latent cognitive processes, we jointly fitted choices and RTs across all experiments with a Bayesian, hierarchical diffusion decision model (DDM). We found that the feedback manipulation affected drift rate, threshold, and non-decision time, suggesting that it was not a mere difficulty effect. Moreover, valence affected non-decision time and threshold, suggesting a motor inhibition in punishing contexts. To better understand the learning dynamics, we finally fitted a combination of RL and DDM (RLDDM). We found that while the threshold was modulated by trial-specific decision conflict, the non-decision time was modulated by the learned context valence. Overall, our results illustrate the benefits of jointly modeling RTs and choice data during RL, to reveal subtle mechanistic differences underlying decisions in different learning contexts.

Advances in techniques for imposing reciprocity in brain-behavior relations

To better understand human behavior, the emerging field of model-based cognitive neuroscience seeks to anchor psychological theory to the biological substrate from which behavior originates: the brain. Despite complex dynamics, many researchers in this field have demonstrated that fluctuations in brain activity can be related to fluctuations in components of cognitive models, which instantiate psychological theories. In this review, we discuss a number of approaches for relating brain activity to cognitive models, and expand on a framework for imposing reciprocity in the inference of mental operations from the combination of brain and behavioral data.

The Neuro-Computational Architecture of Value-Based Selection in the Human Brain

Current neural models of value-based decision-making consider choices as a 2-stage process, proceeding from the "valuation" of each option under consideration to the "selection" of the best option on the basis of their subjective values. However, little is known about the computational mechanisms at play at the selection stage and its implementation in the human brain. Here, we used drift-diffusion models combined with model-based functional magnetic resonance imaging, effective connectivity, and multivariate pattern analysis to characterize the neuro-computational architecture of value-based decisions. We found that 2 key drift-diffusion computations at the selection stage, namely integration and choice readout, engage distinct brain regions, with the dorsolateral prefrontal cortex integrating a decision value signal computed in the ventromedial prefrontal cortex, and the posterior parietal cortex reading out choice outcomes. Our findings suggest that this prefronto-parietal network acts as a hub implementing behavioral selection through a distributed drift-diffusion process.

Reward-driven changes in striatal pathway competition shape evidence evaluation in decision-making

Cortico-basal-ganglia-thalamic (CBGT) networks are critical for adaptive decision-making, yet how changes to circuit-level properties impact cognitive algorithms remains unclear. Here we explore how dopaminergic plasticity at corticostriatal synapses alters competition between striatal pathways, impacting the evidence accumulation process during decision-making. Spike-timing dependent plasticity simulations showed that dopaminergic feedback based on rewards modified the ratio of direct and indirect corticostriatal weights within opposing action channels. Using the learned weight ratios in a full spiking CBGT network model, we simulated neural dynamics and decision outcomes in a reward-driven decision task and fit them with a drift diffusion model. Fits revealed that the rate of evidence accumulation varied with inter-channel differences in direct pathway activity while boundary height varied with overall indirect pathway activity. This multi-level modeling approach demonstrates how complementary learning and decision computations can emerge from corticostriatal plasticity.

Cognitive process models such as reinforcement learning (RL) and the drift diffusion model (DDM) have helped to elucidate the basic algorithms underlying error-corrective learning and the evaluation of accumulating decision evidence leading up to a choice. While these relatively abstract models help to guide experimental and theoretical probes into associated phenomena, they remain uninformative about the actual physical mechanics by which learning and decision algorithms are carried out in a neurobiological substrate during adaptive choice behavior. Here we present an “upwards mapping” approach to bridging neural and cognitive models of value-based decision-making, showing how dopaminergic feedback alters the network-level dynamics of cortico-basal-ganglia-thalamic (CBGT) pathways during learning to bias behavioral choice towards more rewarding actions. By mapping “up” the levels of analysis, this approach yields specific predictions about aspects of neuronal activity that map to the quantities appearing in the cognitive decision-making framework.

Adapting the flow of time with dopamine

The modulation of interval timing by dopamine (DA) has been well established over decades of research. The nature of this modulation, however, has remained controversial: Although the pharmacological evidence has largely suggested that time intervals are overestimated with higher DA levels, more recent optogenetic work has shown the opposite effect. In addition, a large body of work has asserted DA's role as a "reward prediction error" (RPE), or a teaching signal that allows the basal ganglia to learn to predict future rewards in reinforcement learning tasks. Whether these two seemingly disparate accounts of DA may be related has remained an open question. By taking a reinforcement learning-based approach to interval timing, we show here that the RPE interpretation of DA naturally extends to its role as a modulator of timekeeping and furthermore that this view reconciles the seemingly conflicting observations. We derive a biologically plausible, DA-dependent plasticity rule that can modulate the rate of timekeeping in either direction and whose effect depends on the timing of the DA signal itself. This bidirectional update rule can account for the results from pharmacology and optogenetics as well as the behavioral effects of reward rate on interval timing and the temporal selectivity of striatal neurons. Hence, by adopting a single RPE interpretation of DA, our results take a step toward unifying computational theories of reinforcement learning and interval timing. NEW & NOTEWORTHY How does dopamine (DA) influence interval timing? A large body of pharmacological evidence has suggested that DA accelerates timekeeping mechanisms. However, recent optogenetic work has shown exactly the opposite effect. In this article, we relate DA's role in timekeeping to its most established role, as a critical component of reinforcement learning. This allows us to derive a neurobiologically plausible framework that reconciles a large body of DA's temporal effects, including pharmacological, behavioral, electrophysiological, and optogenetic.

Errors in Action Timing and Inhibition Facilitate Learning by Tuning Distinct Mechanisms in the Underlying Decision Process

Goal-directed behavior requires integrating action selection processes with learning systems that adapt control using environmental feedback. These functions are known to intersect at a common neural substrate with multiple known targets of plasticity (the cortico-basal ganglia-thalamic network), suggesting that feedback signals have a multifaceted impact on future decisions. Using a hybrid of accumulation-to-bound decision models and reinforcement learning, we modeled the performance of humans in a stop signal task where participants (N 75: 37 males, 38 females) learned the prior distribution of the timing of a stop signal through trial-and-error feedback. Changes in the drift rate of the action execution process were driven by errors in action timing, whereas adaptation in the boundary height served to increase caution following failed stops. These findings highlight two interactive learning mechanisms for adapting the control of goal-directed actions based on dissociable dimensions of feedback error.SIGNIFICANCE STATEMENT Many complex behavioral goals rely on the ability to regulate the timing of action execution while also maintaining enough control to cancel actions in response to "Stop" cues in the environment. Here we examined how these fundamental components of behavior become tuned to the control demands of the environment by combining principles of reinforcement learning with accumulation-to-bound models. Model fits to behavioral data in an adaptive stop signal task revealed two adaptive mechanisms: (1) timing error-related changes in the rate of the execution signal; and (2) an increase in the execution boundary after failed stops. These findings demonstrate unique effects of timing and control errors on the underlying mechanisms of control, the rate and threshold of accumulating action signals.

Impact of Context Information on Metaphor Elaboration

Abstract. In experiments by Gibbs, Kushner, and Mills (1991) , sentences were supposedly either authored by poets or by a computer. Gibbs et al. (1991) concluded from their results that the assumed source of the text influences speed of processing, with a higher speed for metaphorical sentences in the Poet condition. However, the dependent variables used (e.g., mean RTs) do not allow clear conclusions regarding processing speed. It is also possible that participants had prior biases before the presentation of the stimuli. We conducted a conceptual replication and applied the diffusion model ( Ratcliff, 1978 ) to disentangle a possible effect on processing speed from a prior bias. Our results are in accordance with the interpretation by Gibbs et al. (1991) : The context information affected processing speed, not a priori decision settings. Additionally, analyses of model fit revealed that the diffusion model provided a good account of the data of this complex verbal task.

Event-related deep brain stimulation of the subthalamic nucleus affects conflict processing

OBJECTIVE

Many lines of evidence suggest that response conflict recruits brain regions in the cortical-basal ganglia system. Within the basal ganglia, deep brain recordings from the subthalamic nucleus (STN) have shown that conflict triggers a transient increase in low-frequency oscillations (LFOs; 2-8Hz). Here, we deployed a new method of delivering short trains of event-related deep brain stimulation (DBS) to the STN to test the causal role of the STN and its associated circuits in conflict-related processing.

METHODS

In a double-blind design, we stimulated the STN in patients with Parkinson disease by locking brief trains of DBS to specific periods of the trial within a Stroop task.

RESULTS

Stimulation had a specific effect on conflict compared to nonconflict trials by relatively speeding responses on conflict trials (ie, reducing the Stroop effect, defined as the difference in reaction time between conflict and nonconflict trials) when it was delivered in the preresponse period in the preparation phase. Stimulation also increased errors when it was delivered early in the response window. This latter result corresponded to the timing of the conflict-induced increase in LFOs observed in the absence of stimulation but was not directly related to the reduction in the Stroop effect.

INTERPRETATION

These results support the theory that the time of LFO increase recorded from the STN corresponds to a conflict-processing function. They also provide one of the first demonstrations of event-related DBS of the STN in humans during a cognitive control paradigm. Ann Neurol 2018;84:515-526.

Uncertainty and Exploration

In order to discover the most rewarding actions, agents must collect information about their environment, potentially foregoing reward. The optimal solution to this "explore-exploit" dilemma is often computationally challenging, but principled algorithmic approximations exist. These approximations utilize uncertainty about action values in different ways. Some random exploration algorithms scale the level of choice stochasticity with the level of uncertainty. Other directed exploration algorithms add a "bonus" to action values with high uncertainty. Random exploration algorithms are sensitive to total uncertainty across actions, whereas directed exploration algorithms are sensitive to relative uncertainty. This paper reports a multi-armed bandit experiment in which total and relative uncertainty were orthogonally manipulated. We found that humans employ both exploration strategies, and that these strategies are independently controlled by different uncertainty computations.

Approximating Bayesian Inference through Model Simulation

The ultimate test of the validity of a cognitive theory is its ability to predict patterns of empirical data. Cognitive models formalize this test by making specific processing assumptions that yield mathematical predictions, and the mathematics allow the models to be fitted to data. As the field of cognitive science has grown to address increasingly complex problems, so too has the complexity of models increased. Some models have become so complex that the mathematics detailing their predictions are intractable, meaning that the model can only be simulated. Recently, new Bayesian techniques have made it possible to fit these simulation-based models to data. These techniques have even allowed simulation-based models to transition into neuroscience, where tests of cognitive theories can be biologically substantiated.