Mutual benefits: Combining reinforcement learning with sequential sampling models

Social reward and nonsocial reward processing across the adult lifespan: An interim multi-echo fMRI and diffusion dataset

Social relationships change across the lifespan as social networks narrow and motivational priorities shift. These changes may affect, or reflect, differences in how older adults make decisions related to processing social and non-social rewards. While we have shown initial evidence that older adults have a blunted response to some features of social reward, further work in larger samples is needed to replicate our results and probe the extent to which age-related differences translate to real world consequences, such as financial exploitation. To address this gap, we are conducting a 5-year study funded by the National Institute on Aging (NIH R01-AG067011). Over the course of the funding period (2021-2026), this study seeks to: 1) characterize neural responses to social rewards across adulthood; 2) relate those responses to risk for financial exploitation and sociodemographic factors tied to risk; and 3) examine changes in risk for financial exploitation over time in healthy and vulnerable groups of older adults. This paper describes the preliminary release of data for the larger study. Adults (N = 114; 40 male / 70 female / 4 other or non-binary; 21-80 years of age M = 42.78, SD = 17.13) were recruited from the community to undergo multi-echo fMRI while completing tasks that measure brain function during social reward and decision making. Tasks probe neural response to social reward (e.g., peer vs. monetary feedback) and social context and closeness (e.g., sharing a monetary reward with a friend compared to a stranger). Neural response to social decision making is probed via economic trust and ultimatum games. Functional data are complimented by a T1 weighted anatomical scan and multi-shell diffusion-weighted imaging (DWI) to enable tractography and assess neurite orientation dispersion and density. Overall, this dataset has extensive potential for re-use, including leveraging multimodal neuroimaging data, within subject measures of fMRI data from different tasks - data features that are rarely seen in an adult lifespan dataset. Finally, the functional data will allow for developmentally sensitive cross-sectional analyses of differences in brain response to nuanced differences in reward contexts and outcomes (e.g., monetary vs. social; sharing winnings with a friend vs. stranger; stranger vs. computer).

The Tweedledum and Tweedledee of dynamic decisions: Discriminating between diffusion decision and accumulator models

Theories of dynamic decision-making are typically built on evidence accumulation, which is modeled using racing accumulators or diffusion models that track a shifting balance of support over time. However, these two types of models are only two special cases of a more general evidence accumulation process where options correspond to directions in an accumulation space. Using this generalized evidence accumulation approach as a starting point, I identify four ways to discriminate between absolute-evidence and relative-evidence models. First, an experimenter can look at the information that decision-makers considered to identify whether there is a filtering of near-zero evidence samples, which is characteristic of a relative-evidence decision rule (e.g., diffusion decision model). Second, an experimenter can disentangle different components of drift rates by manipulating the discriminability of the two response options relative to the stimulus to delineate the balance of evidence from the total amount of evidence. Third, a modeler can use machine learning to classify a set of data according to its generative model. Finally, machine learning can also be used to directly estimate the geometric relationships between choice options. I illustrate these different approaches by applying them to data from an orientation-discrimination task, showing converging conclusions across all four methods in favor of accumulator-based representations of evidence during choice. These tools can clearly delineate absolute-evidence and relative-evidence models, and should be useful for comparing many other types of decision theories.

Revisiting the role of computational neuroimaging in the era of integrative neuroscience

Computational models have become integral to human neuroimaging research, providing both mechanistic insights and predictive tools for human cognition and behavior. However, concerns persist regarding the ecological validity of lab-based neuroimaging studies and whether their spatiotemporal resolution is not sufficient for capturing neural dynamics. This review aims to re-examine the utility of computational neuroimaging, particularly in light of the growing prominence of alternative neuroscientific methods and the growing emphasis on more naturalistic behaviors and paradigms. Specifically, we will explore how computational modeling can both enhance the analysis of high-dimensional imaging datasets and, conversely, how neuroimaging, in conjunction with other data modalities, can inform computational models through the lens of neurobiological plausibility. Collectively, this evidence suggests that neuroimaging remains critical for human neuroscience research, and when enhanced by computational models, imaging can serve an important role in bridging levels of analysis and understanding. We conclude by proposing key directions for future research, emphasizing the development of standardized paradigms and the integrative use of computational modeling across neuroimaging techniques.

Stimulus valence moderates self-learning

Self-relevance has been demonstrated to impair instrumental learning. Compared to unfamiliar symbols associated with a friend, analogous stimuli linked with the self are learned more slowly. What is not yet understood, however, is whether this effect extends beyond arbitrary stimuli to material with intrinsically meaningful properties. Take, for example, stimulus valence an established moderator of self-bias. Does the desirability of to-be-learned material influence self-learning? Here, in conjunction with computational modelling (i.e. Reinforcement Learning Drift Diffusion Model analysis), a probabilistic selection task was used to establish if and how stimulus valence (i.e. desirable/undesirable posters) impacts the acquisition of knowledge relating to object-ownership (i.e. owned-by-self vs. owned-by-friend). Several interesting results were observed. First, undesirable posters were learned more rapidly for self compared to friend, an effect that was reversed for desirable posters. Second, learning rates were accompanied by associated differences in reward sensitivity toward desirable and undesirable choice selections as a function of ownership. Third, decisional caution was greater for self-relevant (vs. friend relevant) responses. Collectively, these findings inform understanding of self-function and how valence and stimulus relevance mutually influence probabilistic learning.

Facial misfits accelerate stereotype-based associative learning

Counterstereotypes challenge the deleterious effects that gender-typed beliefs exert on people's occupational aspirations and lifestyle choices. Surprisingly, however, the critical issue of how readily unexpected person-related knowledge can be acquired remains poorly understood. Accordingly, in two experiments in which the facial appearance of targets was varied to manipulate goodness-of-stereotype-fit (i.e., high vs. low femininity/masculinity), here we used a probabilistic selection task to probe the rate at which counter-stereotypic and stereotypic individuals can be learned. Whether occupational (Expt. 1) or trait-related (Expt. 2) gender stereotypes were explored, a computational analysis yielded consistent results. Underscoring the potency of surprising information (i.e., facial misfits), knowledge acquisition was accelerated for unexpected compared to expected persons, both in counter-stereotypic and stereotypic learning contexts. These findings affirm predictive accounts of social perception and speak to the optimal characteristics of interventions designed to reduce stereotyping outside the laboratory.

Model Virtues in Computational Cognitive Neuroscience

There is an abundance of computational models in cognitive neuroscience. A framework for what is desirable in a model, what justifies the introduction of a new one, or what makes one better than another is lacking, however. In this article, we examine key qualities ("virtues") that are desirable in computational models, and how these are interrelated. To keep the scope of the article manageable, we focus on the field of cognitive control, where we identified six "model virtues": empirical accuracy, empirical scope, functional analysis, causal detail, biological plausibility, and psychological plausibility. We first illustrate their use in published work on Stroop modeling and then discuss what expert modelers in the field of cognitive control said about them in a series of qualitative interviews. We found that virtues are interrelated and that their value depends on the modeler's goals, in ways that are not typically acknowledged in the literature. We recommend that researchers make the reasons for their modeling choices more explicit in published work. Our work is meant as a first step. Although our focus here is on cognitive control, we hope that our findings will spark discussion of virtues in other fields as well.

Dysfunctional feedback processing in male methamphetamine abusers: Evidence from neurophysiological and computational approaches

Methamphetamine use disorder (MUD) as a major public health risk is associated with dysfunctional neural feedback processing. Although dysfunctional feedback processing in people who are substance dependent has been explored in several behavioral, computational, and electrocortical studies, this mechanism in MUDs requires to be well understood. Furthermore, the current understanding of latent components of their behavior such as learning speed and exploration-exploitation dilemma is still limited. In addition, the association between the latent cognitive components and the related neural mechanisms also needs to be explored. Therefore, in this study, the underlying neurocognitive mechanisms of feedback processing of such impairment, and age/gender-matched healthy controls are evaluated within a probabilistic learning task with rewards and punishments. Mathematical modeling results based on the Q-learning paradigm suggested that MUDs show less sensitivity in distinguishing optimal options. Additionally, it may be worth noting that MUDs exhibited a slight decrease in their ability to learn from negative feedback compared to healthy controls. Also through the lens of underlying neural mechanisms, MUDs showed lower theta power at the medial-frontal areas while responding to negative feedback. However, other EEG measures of reinforcement learning including feedback-related negativity, parietal-P300, and activity flow from the medial frontal to lateral prefrontal regions, remained intact in MUDs. On the other hand, the elimination of the linkage between value sensitivity and medial-frontal theta activity in MUDs was observed. The observed dysfunction could be due to the adverse effects of methamphetamine on the cortico-striatal dopamine circuit, which is reflected in the anterior cingulate cortex activity as the most likely region responsible for efficient behavior adjustment. These findings could help us to pave the way toward tailored therapeutic approaches.

Brief mindfulness-based meditation enhances the speed of learning following positive prediction errors

Recent research has demonstrated that mindfulness-based meditation facilitates basic aspects of cognition, including memory and attention. Further developing this line of inquiry, here we considered the possibility that similar effects may extend to another core psychological process-instrumental learning. To explore this matter, in combination with a probabilistic selection task, computational modelling (i.e., reinforcement drift diffusion model analysis) was adopted to establish whether and how brief mindfulness-based meditation influences learning under conditions of uncertainty (i.e., choices based on the perceived likelihood of positive and negative outcomes). Three effects were observed. Compared with performance in the control condition (i.e., no meditation), mindfulness-based meditation (1) accelerated the rate of learning following positive prediction errors; (2) elicited a preference for the exploration (vs. exploitation) of choice selections; and (3) increased response caution. Collectively, these findings elucidate the pathways through which brief meditative experiences impact learning and decision-making, with implications for interventions designed to debias aspects of social-cognitive functioning using mindfulness-based meditation.

Joint Modelling of Latent Cognitive Mechanisms Shared Across Decision-Making Domains

Decision-making behavior is often understood using the framework of evidence accumulation models (EAMs). Nowadays, EAMs are applied to various domains of decision-making with the underlying assumption that the latent cognitive constructs proposed by EAMs are consistent across these domains. In this study, we investigate both the extent to which the parameters of EAMs are related between four different decision-making domains and across different time points. To that end, we make use of the novel joint modelling approach, that explicitly includes relationships between parameters, such as covariances or underlying factors, in one combined joint model. Consequently, this joint model also accounts for measurement error and uncertainty within the estimation of these relations. We found that EAM parameters were consistent between time points on three of the four decision-making tasks. For our between-task analysis, we constructed a joint model with a factor analysis on the parameters of the different tasks. Our two-factor joint model indicated that information processing ability was related between the different decision-making domains. However, other cognitive constructs such as the degree of response caution and urgency were only comparable on some domains.

Dopamine regulates decision thresholds in human reinforcement learning in males

Dopamine fundamentally contributes to reinforcement learning, but recent accounts also suggest a contribution to specific action selection mechanisms and the regulation of response vigour. Here, we examine dopaminergic mechanisms underlying human reinforcement learning and action selection via a combined pharmacological neuroimaging approach in male human volunteers (n = 31, within-subjects; Placebo, 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist Haloperidol). We found little credible evidence for previously reported beneficial effects of L-dopa vs. Haloperidol on learning from gains and altered neural prediction error signals, which may be partly due to differences experimental design and/or drug dosages. Reinforcement learning drift diffusion models account for learning-related changes in accuracy and response times, and reveal consistent decision threshold reductions under both drugs, in line with the idea that lower dosages of D2 receptor antagonists increase striatal DA release via an autoreceptor-mediated feedback mechanism. These results are in line with the idea that dopamine regulates decision thresholds during reinforcement learning, and may help to bridge action selection and response vigor accounts of dopamine.

Computational approaches to modeling gambling behaviour: Opportunities for understanding disordered gambling

Computational modeling has become an important tool in neuroscience and psychiatry research to provide insight into the cognitive processes underlying normal and pathological behavior. There are two modeling frameworks, reinforcement learning (RL) and drift diffusion modeling (DDM), that are well-developed in cognitive science, and have begun to be applied to Gambling Disorder. RL models focus on explaining how an agent uses reward to learn about the environment and make decisions based on outcomes. The DDM is a binary choice framework that breaks down decision making into psychologically meaningful components based on choice reaction time analyses. Both approaches have begun to yield insight into aspects of cognition that are important for, but not unique to, gambling, and thus relevant to the development of Gambling Disorder. However, these approaches also oversimplify or neglect various aspects of decision making seen in real-world gambling behavior. Gambling Disorder presents an opportunity for 'bespoke' modeling approaches to consider these neglected components. In this review, we discuss studies that have used RL and DDM frameworks to investigate some of the key cognitive components in gambling and Gambling Disorder. We also include an overview of Bayesian models, a methodology that could be useful for more tailored modeling approaches. We highlight areas in which computational modeling could enable progression in the investigation of the cognitive mechanisms relevant to gambling.

The power of the unexpected: Prediction errors enhance stereotype-based learning

Stereotyping is a ubiquitous feature of social cognition, yet surprisingly little is known about how group-related beliefs influence the acquisition of person knowledge. Accordingly, in combination with computational modeling (i.e., Reinforcement Learning Drift Diffusion Model analysis), here we used a probabilistic selection task to explore the extent to which gender stereotypes impact instrumental learning. Several theoretically interesting effects were observed. First, reflecting the impact of cultural socialization on person construal, an expectancy-based preference for stereotype-consistent (vs. stereotype-inconsistent) responses was observed. Second, underscoring the potency of unexpected information, learning rates were faster for counter-stereotypic compared to stereotypic individuals, both for negative and positive prediction errors. Collectively, these findings are consistent with predictive accounts of social perception and have implications for the conditions under which stereotyping can potentially be reduced.

Evidence accumulation modelling in the wild: understanding safety-critical decisions

Evidence accumulation models (EAMs) are a class of computational cognitive model used to understand the latent cognitive processes that underlie human decisions and response times (RTs). They have seen widespread application in cognitive psychology and neuroscience. However, historically, the application of these models was limited to simple decision tasks. Recently, researchers have applied these models to gain insight into the cognitive processes that underlie observed behaviour in applied domains, such as air-traffic control (ATC), driving, forensic and medical image discrimination, and maritime surveillance. Here, we discuss how this modelling approach helps researchers understand how the cognitive system adapts to task demands and interventions, such as task automation. We also discuss future directions and argue for wider adoption of cognitive modelling in Human Factors research.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Age-related differences in ventral striatal and default mode network function during reciprocated trust

Social relationships change across the lifespan as social networks narrow and motivational priorities shift to the present. Interestingly, aging is also associated with changes in executive function, including decision-making abilities, but it remains unclear how age-related changes in both domains interact to impact financial decisions involving other people. To study this problem, we recruited 50 human participants (Nyounger = 26, ages 18-34; Nolder = 24, ages 63-80) to play an economic trust game as the investor with three partners (friend, stranger, and computer) who played the role of investee. Investors underwent functional magnetic resonance imaging (fMRI) during the trust game while investees were seated outside of the scanner. Building on our previous work with younger adults showing both enhanced striatal responses and altered default-mode network (DMN) connectivity as a function of social closeness during reciprocated trust, we predicted that these relations would exhibit age-related differences. We found that striatal responses to reciprocated trust from friends relative to strangers and computers were blunted in older adults relative to younger adults, thus supporting our primary pre-registered hypothesis regarding social closeness. We also found that older adults exhibited enhanced DMN connectivity with the temporoparietal junction (TPJ) during reciprocated trust from friends compared to computers while younger adults exhibited the opposite pattern. Taken together, these results advance our understanding of age-related differences in sensitivity to social closeness in the context of trusting others.

Sticky me: Self-relevance slows reinforcement learning

A prominent facet of social-cognitive functioning is that self-relevant information is prioritized in perception, attention, and memory. What is not yet understood, however, is whether similar effects arise during learning. In particular, compared to other people (e.g., best friend) is information about the self acquired more rapidly? To explore this matter, here we used a probabilistic selection task in combination with computational modeling (i.e., Reinforcement Learning Drift Diffusion Model analysis) to establish how self-relevance influences learning under conditions of uncertainty (i.e., choices are based on the perceived likelihood of positive and negative outcomes). Across two experiments, a consistent pattern of effects was observed. First, learning rates for both positive and negative prediction errors were slower for self-relevant compared to friend-relevant associations. Second, self-relevant (vs. friend-relevant) learning was characterized by the exploitation (vs. exploration) of choice selections. That is, in a complex (i.e., probabilistic) decision-making environment, previously rewarded self-related outcomes were selected more often than novel - but potentially riskier - alternatives. The implications of these findings for accounts of self-function are considered.

Neurocomputational models of altruistic decision-making and social motives: Advances, pitfalls, and future directions

This article discusses insights from computational models and social neuroscience into motivations, precursors, and mechanisms of altruistic decision-making and other-regard. We introduce theoretical and methodological tools for researchers who wish to adopt a multilevel, computational approach to study behaviors that promote others' welfare. Using examples from recent studies, we outline multiple mental and neural processes relevant to altruism. To this end, we integrate evidence from neuroimaging, psychology, economics, and formalized mathematical models. We introduce basic mechanisms-pertinent to a broad range of value-based decisions-and social emotions and cognitions commonly recruited when our decisions involve other people. Regarding the latter, we discuss how decomposing distinct facets of social processes can advance altruistic models and the development of novel, targeted interventions. We propose that an accelerated synthesis of computational approaches and social neuroscience represents a critical step towards a more comprehensive understanding of altruistic decision-making. We discuss the utility of this approach to study lifespan differences in social preference in late adulthood, a crucial future direction in aging global populations. Finally, we review potential pitfalls and recommendations for researchers interested in applying a computational approach to their research. This article is categorized under: Economics > Interactive Decision-Making Psychology > Emotion and Motivation Neuroscience > Cognition Economics > Individual Decision-Making.

Modeling the influence of working memory, reinforcement, and action uncertainty on reaction time and choice during instrumental learning

What determines the speed of our decisions? Various models of decision-making have focused on perceptual evidence, past experience, and task complexity as important factors determining the degree of deliberation needed for a decision. Here, we build on a sequential sampling decision-making framework to develop a new model that captures a range of reaction time (RT) effects by accounting for both working memory and instrumental learning processes. The model captures choices and RTs at various stages of learning, and in learning environments with varying complexity. Moreover, the model generalizes from tasks with deterministic reward contingencies to probabilistic ones. The model succeeds in part by incorporating prior uncertainty over actions when modeling RT. This straightforward process model provides a parsimonious account of decision dynamics during instrumental learning and makes unique predictions about internal representations of action values.

Cognitive Control of Working Memory: A Model-Based Approach

Working memory (WM)-based decision making depends on a number of cognitive control processes that control the flow of information into and out of WM and ensure that only relevant information is held active in WM's limited-capacity store. Although necessary for successful decision making, recent work has shown that these control processes impose performance costs on both the speed and accuracy of WM-based decisions. Using the reference-back task as a benchmark measure of WM control, we conducted evidence accumulation modeling to test several competing explanations for six benchmark empirical performance costs. Costs were driven by a combination of processes, running outside of the decision stage (longer non-decision time) and showing the inhibition of the prepotent response (lower drift rates) in trials requiring WM control. Individuals also set more cautious response thresholds when expecting to update WM with new information versus maintain existing information. We discuss the promise of this approach for understanding cognitive control in WM-based decision making.

Reliability assessment of temporal discounting measures in virtual reality environments

In recent years the emergence of high-performance virtual reality (VR) technology has opened up new possibilities for the examination of context effects in psychological studies. The opportunity to create ecologically valid stimulation in a highly controlled lab environment is especially relevant for studies of psychiatric disorders, where it can be problematic to confront participants with certain stimuli in real life. However, before VR can be confidently applied widely it is important to establish that commonly used behavioral tasks generate reliable data within a VR surrounding. One field of research that could benefit greatly from VR-applications are studies assessing the reactivity to addiction related cues (cue-reactivity) in participants suffering from gambling disorder. Here we tested the reliability of a commonly used temporal discounting task in a novel VR set-up designed for the concurrent assessment of behavioral and psychophysiological cue-reactivity in gambling disorder. On 2 days, thirty-four healthy non-gambling participants explored two rich and navigable VR-environments (neutral: café vs. gambling-related: casino and sports-betting facility), while their electrodermal activity was measured using remote sensors. In addition, participants completed the temporal discounting task implemented in each VR environment. On a third day, participants performed the task in a standard lab testing context. We then used comprehensive computational modeling using both standard softmax and drift diffusion model (DDM) choice rules to assess the reliability of discounting model parameters assessed in VR. Test-retest reliability estimates were good to excellent for the discount rate log(k), whereas they were poor to moderate for additional DDM parameters. Differences in model parameters between standard lab testing and VR, reflecting reactivity to the different environments, were mostly numerically small and of inconclusive directionality. Finally, while exposure to VR generally increased tonic skin conductance, this effect was not modulated by the neutral versus gambling-related VR-environment. Taken together this proof-of-concept study in non-gambling participants demonstrates that temporal discounting measures obtained in VR are reliable, suggesting that VR is a promising tool for applications in computational psychiatry, including studies on cue-reactivity in addiction.

The dynamics of explore-exploit decisions reveal a signal-to-noise mechanism for random exploration

Growing evidence suggests that behavioral variability plays a critical role in how humans manage the tradeoff between exploration and exploitation. In these decisions a little variability can help us to overcome the desire to exploit known rewards by encouraging us to randomly explore something else. Here we investigate how such 'random exploration' could be controlled using a drift-diffusion model of the explore-exploit choice. In this model, variability is controlled by either the signal-to-noise ratio with which reward is encoded (the 'drift rate'), or the amount of information required before a decision is made (the 'threshold'). By fitting this model to behavior, we find that while, statistically, both drift and threshold change when people randomly explore, numerically, the change in drift rate has by far the largest effect. This suggests that random exploration is primarily driven by changes in the signal-to-noise ratio with which reward information is represented in the brain.

A new model of decision processing in instrumental learning tasks

Learning and decision-making are interactive processes, yet cognitive modeling of error-driven learning and decision-making have largely evolved separately. Recently, evidence accumulation models (EAMs) of decision-making and reinforcement learning (RL) models of error-driven learning have been combined into joint RL-EAMs that can in principle address these interactions. However, we show that the most commonly used combination, based on the diffusion decision model (DDM) for binary choice, consistently fails to capture crucial aspects of response times observed during reinforcement learning. We propose a new RL-EAM based on an advantage racing diffusion (ARD) framework for choices among two or more options that not only addresses this problem but captures stimulus difficulty, speed-accuracy trade-off, and stimulus-response-mapping reversal effects. The RL-ARD avoids fundamental limitations imposed by the DDM on addressing effects of absolute values of choices, as well as extensions beyond binary choice, and provides a computationally tractable basis for wider applications.

Dopamine and Risky Decision-Making in Gambling Disorder

Gambling disorder is a behavioral addiction associated with impairments in value-based decision-making and cognitive control. These functions are thought to be regulated by dopamine within fronto-striatal circuits, but the role of altered dopamine neurotransmission in the etiology of gambling disorder remains controversial. Preliminary evidence suggests that increasing frontal dopamine tone might improve cognitive functioning in gambling disorder. We therefore examined whether increasing frontal dopamine tone via a single dose of the catechol-O-methyltransferase (COMT) inhibitor tolcapone would reduce risky choice in human gamblers (n = 14) in a randomized double-blind placebo-controlled crossover study. Data were analyzed using hierarchical Bayesian parameter estimation and a combined risky choice drift diffusion model (DDM). Model comparison revealed a nonlinear mapping from value differences to trial-wise drift rates, confirming recent findings. An increase in risk-taking under tolcapone versus placebo was about five times more likely, given the data, than a decrease [Bayes factor (BF) = 0.2]. Examination of drug effects on diffusion model parameters revealed that an increase in the value dependency of the drift rate under tolcapone was about thirteen times more likely than a decrease (BF = 0.073). In contrast, a reduction in the maximum drift rate under tolcapone was about seven times more likely than an increase (BF = 7.51). Results add to previous work on COMT inhibitors in behavioral addictions and to mounting evidence for the applicability of diffusion models in value-based decision-making. Future work should focus on individual genetic, clinical and cognitive factors that might account for heterogeneity in the effects of COMT inhibition.

Simultaneous Hierarchical Bayesian Parameter Estimation for Reinforcement Learning and Drift Diffusion Models: a Tutorial and Links to Neural Data

Cognitive models have been instrumental for generating insights into the brain processes underlying learning and decision making. In reinforcement learning it has recently been shown that not only choice proportions but also their latency distributions can be well captured when the choice function is replaced with a sequential sampling model such as the drift diffusion model. Hierarchical Bayesian parameter estimation further enhances the identifiability of distinct learning and choice parameters. One caveat is that these models can be time-consuming to build, sample from, and validate, especially when models include links between neural activations and model parameters. Here we describe a novel extension to the widely used hierarchical drift diffusion model (HDDM) toolbox, which facilitates flexible construction, estimation, and evaluation of the reinforcement learning drift diffusion model (RLDDM) using hierarchical Bayesian methods. We describe the types of experiments most applicable to the model and provide a tutorial to illustrate how to perform quantitative data analysis and model evaluation. Parameter recovery confirmed that the method can reliably estimate parameters with varying numbers of synthetic subjects and trials. We also show that the simultaneous estimation of learning and choice parameters can improve the sensitivity to detect brain–behavioral relationships, including the impact of learned values and fronto-basal ganglia activity patterns on dynamic decision parameters.

AUV 3D Path Planning Based on the Improved Hierarchical Deep Q Network

This study proposed the 3D path planning of an autonomous underwater vehicle (AUV) by using the hierarchical deep Q network (HDQN) combined with the prioritized experience replay. The path planning task was divided into three layers, which realized the dimensionality reduction of state space and solved the problem of dimension disaster. An artificial potential field was used to design the positive rewards of the algorithm to shorten the training time. According to the different requirements of the task, this study modified the rewards in the training process to obtain different paths. The path planning simulation and field tests were carried out. The results of the tests corroborated that the training time of the proposed method was shorter than that of the traditional method. The path obtained by simulation training was proved to be safe and effective.