A theoretical account of cognitive effects in delay discounting

Associations between PTSD and temporal discounting: The role of future thinking

BACKGROUND AND OBJECTIVES

Despite documented alterations in future thinking in posttraumatic stress disorder (PTSD), our understanding of how individuals with PTSD make future-oriented decisions is limited. We tested the hypothesis that increased discounting in association with PTSD reflects failure to spontaneously envision future rewarding situations.

METHODS

Thirty-seven trauma exposed war-zone veterans completed a standard temporal discounting task as well as a temporal discounting task accompanied by episodic future thinking cues.

RESULTS

Severity of PTSD symptoms was associated with preference for sooner, smaller rewards in the standard task. Consistent with our hypothesis, when participants engaged in future thinking, greater PTSD symptom severity was no longer associated with steeper discounting. Moreover, difficulty anticipating future events, as measured contemporaneously in a separate task (Verfaellie et al., 2024), mediated the relationship between PTSD symptom severity and degree of discounting in the standard task. Among PTSD symptom clusters, the severity of avoidance and negative alterations in cognition and mood was related to steeper discounting. Measures of depression and alcohol use were not associated with discounting.

LIMITATIONS

The sample included mostly male, predominantly White veterans who experienced primarily combat-related trauma.

CONCLUSIONS

PTSD-associated alterations in temporal discounting reflect failure to spontaneously imagine future positive events. Two common correlates of PTSD, depression and alcohol use, could not account for the observed associations between PTSD and future-oriented decisions.

Clarifying Cognitive Control Deficits in Psychosis via Drift Diffusion Modeling and Attractor Dynamics

BACKGROUND AND HYPOTHESIS

Cognitive control deficits are prominent in individuals with psychotic psychopathology. Studies providing evidence for deficits in proactive control generally examine average performance and not variation across trials for individuals-potentially obscuring detection of essential contributors to cognitive control. Here, we leverage intertrial variability through drift-diffusion models (DDMs) aiming to identify key contributors to cognitive control deficits in psychosis.

STUDY DESIGN

People with psychosis (PwP; N = 122), their first-degree biological relatives (N = 78), and controls (N = 50) each completed 120 trials of the dot pattern expectancy (DPX) cognitive control task. We fit full hierarchical DDMs to response and reaction time (RT) data for individual trials and then used classification models to compare the DDM parameters with conventional measures of proactive and reactive control.

STUDY RESULTS

PwP demonstrated slower drift rates on proactive control trials suggesting less efficient use of cue information. Both PwP and relatives showed protracted nondecision times to infrequent trial sequences suggesting slowed perceptual processing. Classification analyses indicated that DDM parameters differentiated between the groups better than conventional measures and identified drift rates during proactive control, nondecision time during reactive control, and cue bias as most important. DDM parameters were associated with real-world functioning and schizotypal traits.

CONCLUSIONS

Modeling of trial-level data revealed that slow evidence accumulation and longer preparatory periods are the strongest contributors to cognitive control deficits in psychotic psychopathology. This pattern of atypical responding during the DPX is consistent with shallow basins in attractor dynamic models that reflect difficulties in maintaining state representations, possibly mediated by excess neural excitation or poor connectivity.

Flexible coding schemes in dorsomedial prefrontal cortex underlie decision-making during delay discounting

Determining how an agent decides between a small, immediate versus a larger, delayed reward has provided insight into the psychological and neural basis of decision-making. The tendency to excessively discount the value of delayed rewards is thought to reflect deficits in brain regions critical for impulse control such as the prefrontal cortex (PFC). This study tested the hypothesis that dorsomedial PFC (dmPFC) is critically involved in flexibly managing neural representations of strategies that limit impulsive choices. Optogenetic silencing of neurons in the rat dmPFC increased impulsive choices at an 8 sec, but not 4 sec, delay. Neural recordings from dmPFC ensembles revealed that, at the 8-sec delay, the encoding landscape transitions to reflect a deliberative-like process rather than the schema-like processes observed at the 4-sec delay. These findings show that changes in the encoding landscape reflect changes in task demands and that dmPFC is uniquely involved in decisions requiring deliberation.

Investigating associations of delay discounting with brain structure, working memory, and episodic memory

INTRODUCTION

Delay discounting (DD), the preference for smaller and sooner rewards over larger and later ones, is an important behavioural phenomenon for daily functioning of increasing interest within psychopathology. The neurobiological mechanisms behind DD are not well understood and the literature on structural correlates of DD shows inconsistencies.

METHODS

Here we leveraged a large openly available dataset (n = 1196) to investigate associations with memory performance and gray and white matter correlates of DD using linked independent component analysis.

RESULTS

Greater DD was related to smaller anterior temporal gray matter volume. Associations of DD with total cortical volume, subcortical volumes, markers of white matter microscopic organization, working memory, and episodic memory scores were not significant after controlling for education and income.

CONCLUSION

Effects of size comparable to the one we identified would be unlikely to be replicated with sample sizes common in many previous studies in this domain, which may explain the incongruities in the literature. The paucity and small size of the effects detected in our data underscore the importance of using large samples together with methods that accommodate their statistical structure and appropriate control for confounders, as well as the need to devise paradigms with improved task parameter reliability in studies relating brain structure and cognitive abilities with DD.

Developmental trajectories of delay discounting from childhood to young adulthood: longitudinal associations and test-retest reliability

Delay discounting (DD) indexes an individual's preference for smaller immediate rewards over larger delayed rewards, and is considered a form of cognitive impulsivity. Cross-sectional studies have demonstrated that DD peaks in adolescence; longitudinal studies are needed to validate this putative developmental trend, and to determine whether DD assesses a temporary state, or reflects a more stable behavioral trait. In this study, 140 individuals aged 9-23 completed a delay discounting (DD) task and cognitive battery at baseline and every-two years thereafter, yielding five assessments over approximately 10 years. Models fit with the inverse effect of age best approximated the longitudinal trajectory of two DD measures, hyperbolic discounting (log[k]) and area under the indifference-point curve (AUC). Discounting of future rewards increased rapidly from childhood to adolescence and appeared to plateau in late adolescence for both models of DD. Participants with greater verbal intelligence and working memory displayed reduced DD across the duration of the study, suggesting a functional interrelationship between these domains and DD from early adolescence to adulthood. Furthermore, AUC demonstrated good to excellent reliability across assessment points that was superior to log(k), with both measures demonstrating acceptable stability once participants reached late adolescence. The developmental trajectories of DD we observed from childhood through young adulthood suggest that DD may index cognitive control more than reward sensitivity, and that despite modest developmental changes with maturation, AUC may be conceptualized as a trait variable related to cognitive control vs impulsivity.

Imagining emotional events benefits future-oriented decisions

How does imagining future events-whether positive or negative-influence our choices in the present? Prior work has shown the simulation of hypothetical future events, dubbed episodic future thinking, can alter the propensity to engage in delay discounting (the tendency to devalue future rewards) and does so in a valence-specific manner. Some research shows that positive episodic future thinking reduces delay discounting, whereas negative future thinking augments it. However, more recent research indicates that both positive and negative episodic future thinking reduce delay discounting, suggesting an effect of episodic future thinking that is independent of valence. In this study, we sought to replicate and extend these latter findings. Here, participants (N = 604; N = 572 after exclusions) completed an online study. In the baseline task, participants completed a delay discounting task. In the experimental task, they engaged in episodic future thinking before completing a second delay discounting task. Participants were randomly assigned to engage in either positive, neutral, or negative episodic future thinking. In accordance with Bulley et al., we found that episodic future thinking, regardless of valence, reduced delay discounting. Although episodic future thinking shifted decision-making in all conditions, the effect was stronger when participants engaged in positive episodic future thinking, even after accounting for personal relevance and vividness of imagined events. These findings suggest that episodic future thinking may promote future-oriented choices by contextualising the future, and this effect is further strengthened when the future is tied to positive emotion.

The rodent medial prefrontal cortex and associated circuits in orchestrating adaptive behavior under variable demands

Emerging evidence implicates rodent medial prefrontal cortex (mPFC) in tasks requiring adaptation of behavior to changing information from external and internal sources. However, the computations within mPFC and subsequent outputs that determine behavior are incompletely understood. We review the involvement of mPFC subregions, and their projections to the striatum and amygdala in two broad types of tasks in rodents: 1) appetitive and aversive Pavlovian and operant conditioning tasks that engage mPFC-striatum and mPFC-amygdala circuits, and 2) foraging-based tasks that require decision making to optimize reward. We find support for region-specific function of the mPFC, with dorsal mPFC and its projections to the dorsomedial striatum supporting action control with higher cognitive demands, and ventral mPFC engagement in translating affective signals into behavior via discrete projections to the ventral striatum and amygdala. However, we also propose that defined mPFC subdivisions operate as a functional continuum rather than segregated functional units, with crosstalk that allows distinct subregion-specific inputs (e.g., internal, affective) to influence adaptive behavior supported by other subregions.

Subjective Time in Dementia: A Critical Review

The capacity for subjective time in humans encompasses the perception of time's unfolding from moment to moment, as well as the ability to traverse larger temporal expanses of past- and future-oriented thought via mental time travel. Disruption in time perception can result in maladaptive outcomes-from the innocuous lapse in timing that leads to a burnt piece of toast, to the grievous miscalculation that produces a traffic accident-while disruption to mental time travel can impact core functions from planning appointments to making long-term decisions. Mounting evidence suggests that disturbances to both time perception and mental time travel are prominent in dementia syndromes. Given that such disruptions can have severe consequences for independent functioning in everyday life, here we aim to provide a comprehensive exposition of subjective timing dysfunction in dementia, with a view to informing the management of such disturbances. We consider the neurocognitive mechanisms underpinning changes to both time perception and mental time travel across different dementia disorders. Moreover, we explicate the functional implications of altered subjective timing by reference to two key and representative adaptive capacities: prospective memory and intertemporal decision-making. Overall, our review sheds light on the transdiagnostic implications of subjective timing disturbances in dementia and highlights the high variability in performance across clinical syndromes and functional domains.

Brain multimodal co-alterations related to delay discounting: a multimodal MRI fusion analysis in persons with and without cocaine use disorder

BACKGROUND

Delay discounting has been proposed as a behavioral marker of substance use disorders. Innovative analytic approaches that integrate information from multiple neuroimaging modalities can provide new insights into the complex effects of drug use on the brain. This study implemented a supervised multimodal fusion approach to reveal neural networks associated with delay discounting that distinguish persons with and without cocaine use disorder (CUD).

METHODS

Adults with (n = 35) and without (n = 37) CUD completed a magnetic resonance imaging (MRI) scan to acquire high-resolution anatomical, resting-state functional, and diffusion-weighted images. Pre-computed features from each data modality included whole-brain voxel-wise maps for gray matter volume, fractional anisotropy, and regional homogeneity, respectively. With delay discounting as the reference, multimodal canonical component analysis plus joint independent component analysis was used to identify co-alterations in brain structure and function.

RESULTS

The sample was 58% male and 78% African-American. As expected, participants with CUD had higher delay discounting compared to those without CUD. One joint component was identified that correlated with delay discounting across all modalities, involving regions in the thalamus, dorsal striatum, frontopolar cortex, occipital lobe, and corpus callosum. The components were negatively correlated with delay discounting, such that weaker loadings were associated with higher discounting. The component loadings were lower in persons with CUD, meaning the component was expressed less strongly.

CONCLUSIONS

Our findings reveal structural and functional co-alterations linked to delay discounting, particularly in brain regions involved in reward salience, executive control, and visual attention and connecting white matter tracts. Importantly, these multimodal networks were weaker in persons with CUD, indicating less cognitive control that may contribute to impulsive behaviors.

Advances in the computational understanding of mental illness

Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.

Variational Free Energy and Economics Optimizing With Biases and Bounded Rationality

The purpose of this paper is to offer a new framework for understanding action, optimization, and choice when applied to economic theory more generally. By drawing upon the concept known as the variational free energy principle, the paper will explore how this principle can be used to temper rational choice theory by re-formulating how agents optimize. The approach will result in agent behavior that encompasses a wide range of so-called cognitive biases, as seen in the scientific literature of behavioral economics, but instead of using these biases as further indications of market inefficiencies or market failures, the paper will likewise attempt to show the limits to which these biases can inform or critique standard economic theory. The paper therefore offers up a “middle of the road” approach, in which the neoclassical agent is not quite as “rational” as rational choice theory assumes, but at the same time, not quite as irrational as behavioral economics would often have us believe.

Envisaging Mitigation Action Can Induce Lower Discounting toward Future Environmental Gains and Promote Pro-Environmental Behavior

Low engagement with climate change may stem from the tendency to discount the distant benefits of mitigation action. Hence, a reduced tendency to discount the future should be associated with increased involvement in climate change mitigation. Prior research has demonstrated that episodic future thinking (EFT; i.e., envisioning future events that involve self-projection) can reduce discounting. In two laboratory studies, we showed that engaging in EFT about mitigation action was associated with a lower discounting tendency toward future environmental gains (Experiments 1 and 2) and a greater tendency to act pro-environmentally, as manifested by using air conditioning in an energy-saving manner (Experiment 1), choosing a meal with less environmental impact (Experiment 2), and willingness to participate in beach cleaning (Experiment 2). The present findings suggest that engagement in EFT about mitigation action may represent a promising strategy for improving personal involvement in climate change.

The rational use of causal inference to guide reinforcement learning strengthens with age

Beliefs about the controllability of positive or negative events in the environment can shape learning throughout the lifespan. Previous research has shown that adults' learning is modulated by beliefs about the causal structure of the environment such that they update their value estimates to a lesser extent when the outcomes can be attributed to hidden causes. This study examined whether external causes similarly influenced outcome attributions and learning across development. Ninety participants, ages 7 to 25 years, completed a reinforcement learning task in which they chose between two options with fixed reward probabilities. Choices were made in three distinct environments in which different hidden agents occasionally intervened to generate positive, negative, or random outcomes. Participants' beliefs about hidden-agent intervention aligned with the true probabilities of the positive, negative, or random outcome manipulation in each of the three environments. Computational modeling of the learning data revealed that while the choices made by both adults (ages 18-25) and adolescents (ages 13-17) were best fit by Bayesian reinforcement learning models that incorporate beliefs about hidden-agent intervention, those of children (ages 7-12) were best fit by a one learning rate model that updates value estimates based on choice outcomes alone. Together, these results suggest that while children demonstrate explicit awareness of the causal structure of the task environment, they do not implicitly use beliefs about the causal structure of the environment to guide reinforcement learning in the same manner as adolescents and adults.

Neural signatures underlying deliberation in human foraging decisions

Humans have a remarkable capacity to mentally project themselves far ahead in time. This ability, which entails the mental simulation of events, is thought to be fundamental to deliberative decision making, as it allows us to search through and evaluate possible choices. Many decisions that humans make are foraging decisions, in which one must decide whether an available offer is worth taking, when compared to unknown future possibilities (i.e., the background). Using a translational decision-making paradigm designed to reveal decision preferences in rats, we found that humans engaged in deliberation when making foraging decisions. A key feature of this task is that preferences (and thus, value) are revealed as a function of serial choices. Like rats, humans also took longer to respond when faced with difficult decisions near their preference boundary, which was associated with prefrontal and hippocampal activation, exemplifying cross-species parallels in deliberation. Furthermore, we found that voxels within the visual cortices encoded neural representations of the available possibilities specifically following regret-inducing experiences, in which the subject had previously rejected a good offer only to encounter a low-valued offer on the subsequent trial.

Can episodic future thinking affect food choices?

Episodic future thinking, defined as the ability to project oneself into the future, has proven useful to pre-experience the future consequences of present actions. We investigate how episodic future thinking influences the food choices of normal weight, overweight, and obese individuals. In doing so, we conduct a controlled laboratory experiment in which participants are presented with representations of weight-increased and weight-reduced modified images of themselves before performing a food choice task. This allows subjects to vividly imagine the future consequences of their actions. We also test the effect of providing health-related information on food choices to compare with the episodic future thinking effect. Our results suggest that while providing health-related information increases the number of lite snack choices of overweight and obese individuals, engaging in episodic future thinking has a positive impact on the food choices of the obese only. These findings are supported by eye-tracking data showing how visual attention and emotional arousal (measured by pupil size) impact individuals' food choices.

Rationally inattentive intertemporal choice

Discounting of future rewards is traditionally interpreted as evidence for an intrinsic preference in favor of sooner rewards. However, temporal discounting can also arise from internal uncertainty in value representations of future events, if one assumes that noisy mental simulations of the future are rationally combined with prior beliefs. Here, we further develop this idea by considering how simulation noise may be adaptively modulated by task demands, based on principles of rational inattention. We show how the optimal allocation of mental effort can give rise to the magnitude effect in intertemporal choice. In a re-analysis of two prior data sets, and in another experiment, we reveal several behavioral signatures of this theoretical account, tying choice stochasticity to the magnitude effect. We conclude that some aspects of temporal discounting may result from a cognitively plausible adaptive response to the costs of information processing.

People are disproportionately more patient when evaluating larger rewards. Here, the authors show how this magnitude effect may reflect an adaptive response to uncertainty in mental representations of future value.

Deliberating trade-offs with the future

Many fundamental choices in life are intertemporal: they involve trade-offs between sooner and later outcomes. In recent years there has been a surge of interest into how people make intertemporal decisions, given that such decisions are ubiquitous in everyday life and central in domains from substance use to climate change action. While it is clear that people make decisions according to rules, intuitions and habits, they also commonly deliberate over their options, thinking through potential outcomes and reflecting on their own preferences. In this Perspective, we bring to bear recent research into the higher-order capacities that underpin deliberation-particularly those that enable people to think about the future (prospection) and their own thinking (metacognition)-to shed light on intertemporal decision-making. We show how a greater appreciation for these mechanisms of deliberation promises to advance our understanding of intertemporal decision-making and unify a wide range of otherwise disparate choice phenomena.

Cognitive training as a component of treatment of alcohol use disorder: A review

OBJECTIVE

Cognitive training is an effective means of improving performance in a range of populations. Whether it may serve to facilitate cognitive recovery and longer-term outcomes in persons with alcohol use disorders (AUDs) is unclear. Here, we review historical and current literature and offer perspectives for model development and potential implementation.

METHOD

We considered a large literature regarding the nature of alcohol-related compromise, early efforts to clarify the nature of recovery and current models and methods underlying cognitive training paradigms. We then constructed a narrative review demonstrating evolving frameworks and empirical data informing the critical review of cognitive training methods as a means of mitigating compromise and facilitating functional outcomes.

RESULTS

Cognitive improvement with abstinence is generally noted, but training protocols may enhance performance and generalize benefit to untrained, but highly similar, tasks. Transfer of training to dissimilar tasks and functional outcomes is uncommonly reported. It is noteworthy that some work suggests that clinician ratings for participants are improved. Inconsistency in sample characteristics, training protocols, and outcome measures constrain general conclusions while suggesting opportunities for study and development.

CONCLUSIONS

Cognitive training protocols have shown benefit in a variety of populations but have been examined infrequently in persons with AUDs. This overview indicates significant opportunity for cognitive improvement and recovery and thus a strong potential role for training protocols. However, supportive data are not robustly obtained. We suggest that one step in bridging this gap is the implementation of a conceptual framework incorporating contextual, behavioral, and neurobiological variables. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Learning and forgetting using reinforced Bayesian change detection

Agents living in volatile environments must be able to detect changes in contingencies while refraining to adapt to unexpected events that are caused by noise. In Reinforcement Learning (RL) frameworks, this requires learning rates that adapt to past reliability of the model. The observation that behavioural flexibility in animals tends to decrease following prolonged training in stable environment provides experimental evidence for such adaptive learning rates. However, in classical RL models, learning rate is either fixed or scheduled and can thus not adapt dynamically to environmental changes. Here, we propose a new Bayesian learning model, using variational inference, that achieves adaptive change detection by the use of Stabilized Forgetting, updating its current belief based on a mixture of fixed, initial priors and previous posterior beliefs. The weight given to these two sources is optimized alongside the other parameters, allowing the model to adapt dynamically to changes in environmental volatility and to unexpected observations. This approach is used to implement the "critic" of an actor-critic RL model, while the actor samples the resulting value distributions to choose which action to undertake. We show that our model can emulate different adaptation strategies to contingency changes, depending on its prior assumptions of environmental stability, and that model parameters can be fit to real data with high accuracy. The model also exhibits trade-offs between flexibility and computational costs that mirror those observed in real data. Overall, the proposed method provides a general framework to study learning flexibility and decision making in RL contexts.

Developmental perspectives on risky and impulsive choice

Epidemiological data suggest that risk taking in the real world increases from childhood into adolescence and declines into adulthood. However, developmental patterns of behaviour in laboratory assays of risk taking and impulsive choice are inconsistent. In this article, we review a growing literature using behavioural economic approaches to understand developmental changes in risk taking and impulsivity. We present findings that have begun to elucidate both the cognitive and neural processes that contribute to risky and impulsive choice, as well as how age-related changes in these neurocognitive processes give rise to shifts in choice behaviour. We highlight how variability in task parameters can be used to identify specific aspects of decision contexts that may differentially influence risky and impulsive choice behaviour across development. This article is part of the theme issue 'Risk taking and impulsive behaviour: fundamental discoveries, theoretical perspectives and clinical implications'.

Habits without values

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Mix and Match: An Investigation into Whether Episodic Future Thinking Cues Need to Match Discounting Delays in Order to Be Effective

Episodic future thinking (EFT), or prospectively imagining yourself in the future, has been developed into an intervention tool to reduce delay discounting (DD), or the preference for smaller immediate over larger future rewards, and to make healthier choices that promote long-term health rather than short-term enjoyment. Most EFT interventions use EFT cues whose future events match the time delays of the DD task, which may limit the utility of EFT. The current study (N = 160, M_age = 35.25, 47.5% female) used a 2 × 2 factorial design with type of episodic thinking (matched, unmatched) and temporal perspective (EFT, episodic recent thinking (ERT)) as between-subject factors to investigate whether there were differences in DD for groups that had EFT cues matched to the time delays of the DD task in comparison to cues with unmatched temporal delays. The results showed EFT reduced DD compared to ERT controls, and no differences emerged between matched and unmatched EFT groups. Our findings suggest that either the process of generating EFT cues or the use of any positive and vivid future event, regardless of whether it is matched to the DD task, can reduce DD.

Do process simulations during episodic future thinking enhance the reduction of delay discounting for middle income participants and those living in poverty?

Two studies examined whether episodic future thinking (EFT; pre-experiencing future events) reduces discounting of future rewards (DD). No studies have investigated whether process simulations (i.e., simulating the process of executing a future event) amplify EFT's reduction of DD. Study 1 examined the effect of incorporating process simulations into EFT (N = 42, M _age = 43.27; 91% female, family income = $75,976) using a 2 × 2 factorial design with type of episodic thinking (process, nonprocess/general) and temporal perspective (EFT, episodic recent thinking) as between-subjects factors. Study 2 replicated Study 1 in a sample of adults living in poverty (N = 36; M _age = 38.44, 88% female; family income = $25,625). The results of both studies showed EFT reduced DD, but process-oriented EFT did not amplify the effect of EFT. Our findings suggest the key ingredient in EFT's effect on DD is self-projection into the future. This was also the first study to show EFT improves DD in a sample living in poverty.

Beyond simple tests of value: measuring addiction as a heterogeneous disease of computation-specific valuation processes

Addiction is considered to be a neurobiological disorder of learning and memory because addiction is capable of producing lasting changes in the brain. Recovering addicts chronically struggle with making poor decisions that ultimately lead to relapse, suggesting a view of addiction also as a neurobiological disorder of decision-making information processing. How the brain makes decisions depends on how decision-making processes access information stored as memories in the brain. Advancements in circuit-dissection tools and recent theories in neuroeconomics suggest that neurally dissociable valuation processes access distinct memories differently, and thus are uniquely susceptible as the brain changes during addiction. If addiction is to be considered a neurobiological disorder of memory, and thus decision-making, the heterogeneity with which information is both stored and processed must be taken into account in addiction studies. Addiction etiology can vary widely from person to person. We propose that addiction is not a single disease, nor simply a disorder of learning and memory, but rather a collection of symptoms of heterogeneous neurobiological diseases of distinct circuit-computation-specific decision-making processes.

Neural signatures of reinforcement learning correlate with strategy adoption during spatial navigation

Human navigation is generally believed to rely on two types of strategy adoption, route-based and map-based strategies. Both types of navigation require making spatial decisions along the traversed way although formal computational and neural links between navigational strategies and mechanisms of value-based decision making have so far been underexplored in humans. Here we employed functional magnetic resonance imaging (fMRI) while subjects located different objects in a virtual environment. We then modelled their paths using reinforcement learning (RL) algorithms, which successfully explained decision behavior and its neural correlates. Our results show that subjects used a mixture of route and map-based navigation and their paths could be well explained by the model-free and model-based RL algorithms. Furthermore, the value signals of model-free choices during route-based navigation modulated the BOLD signals in the ventro-medial prefrontal cortex (vmPFC), whereas the BOLD signals in parahippocampal and hippocampal regions pertained to model-based value signals during map-based navigation. Our findings suggest that the brain might share computational mechanisms and neural substrates for navigation and value-based decisions such that model-free choice guides route-based navigation and model-based choice directs map-based navigation. These findings open new avenues for computational modelling of wayfinding by directing attention to value-based decision, differing from common direction and distances approaches.

A Bird in the Hand Isn't Good for Long

Folk wisdom tells us that additional time to make a decision helps us to refrain from the first impulse to take the bird in the hand. However, the question why the time to decide plays an important role is still unanswered. Here we distinguish two explanations, one based on a bias in value accumulation that has to be overcome with time, the other based on cognitive control processes that need time to set in. In an intertemporal decision task, we use mouse tracking to study participants' responses to options' values and delays which were presented sequentially. We find that the information about options' delays does indeed lead to an immediate bias that is controlled afterwards, matching the prediction of control processes needed to counter initial impulses. Hence, by using a dynamic measure, we provide insight into the processes underlying short-term oriented choices in intertemporal decision making.

Dopamine and Proximity in Motivation and Cognitive Control

Cognitive control - the ability to override a salient or prepotent action to execute a more deliberate one - is required for flexible, goal-directed behavior, and yet it is subjectively costly: decision-makers avoid allocating control resources, even when doing so affords more valuable outcomes. Dopamine likely offsets effort costs just as it does for physical effort. And yet, dopamine can also promote impulsive action, undermining control. We propose a novel hypothesis that reconciles opposing effects of dopamine on cognitive control: during action selection, striatal dopamine biases benefits relative to costs, but does so preferentially for "proximal" motor and cognitive actions. Considering the nature of instrumental affordances and their dynamics during action selection facilitates a parsimonious interpretation and conserved corticostriatal mechanisms across physical and cognitive domains.

Continuous track paths reveal additive evidence integration in multistep decision making

Multistep decision making pervades daily life, but its underlying mechanisms remain obscure. We distinguish four prominent models of multistep decision making, namely serial stage, hierarchical evidence integration, hierarchical leaky competing accumulation (HLCA), and probabilistic evidence integration (PEI). To empirically disentangle these models, we design a two-step reward-based decision paradigm and implement it in a reaching task experiment. In a first step, participants choose between two potential upcoming choices, each associated with two rewards. In a second step, participants choose between the two rewards selected in the first step. Strikingly, as predicted by the HLCA and PEI models, the first-step decision dynamics were initially biased toward the choice representing the highest sum/mean before being redirected toward the choice representing the maximal reward (i.e., initial dip). Only HLCA and PEI predicted this initial dip, suggesting that first-step decision dynamics depend on additive integration of competing second-step choices. Our data suggest that potential future outcomes are progressively unraveled during multistep decision making.

The roles of valuation and reward processing in cognitive function and psychiatric disorders

In neuroeconomics, valuation refers to the process of assigning values to states and actions on the basis of the animal's current representation of the environment, while reward processing corresponds to processing the feedback received from the environment to update the values of states and actions. In this article, we review the brain circuits associated with valuation and reward processing and argue that these are fundamental processes critical to many cognitive functions. Specifically, we focus on the role of valuation and reward processing in attention, memory, decision making, and learning. Next, the extant neuroimaging literature on a number of psychiatric disorders is reviewed (i.e., addiction, pathological gambling, schizophrenia, and mood disorders), and an argument is made that associated deficits in cognitive functions can be explained in terms of abnormal valuation and reward processing. The review concludes with the impact of this framework in clinical settings and prescriptions for future research, in particular with regard to the conversions of qualitatively different valuation systems into a system of common currency.

Impulsivity in rodents with a genetic predisposition for excessive alcohol consumption is associated with a lack of a prospective strategy

Increasing evidence supports the hypothesis that impulsive decision-making is a heritable risk factor for an alcohol use disorder (AUD). Clearly identifying a link between impulsivity and AUD risk, however, is complicated by the fact that both AUDs and impulsivity are heterogeneous constructs. Understanding the link between the two requires identifying the underlying cognitive factors that lead to impulsive choices. Rodent models have established that a family history of excessive drinking can lead to the expression of a transgenerational impulsive phenotype, suggesting heritable alterations in the decision-making process. In the present study, we explored the cognitive processes underlying impulsive choice in a validated, selectively bred rodent model of excessive drinking-the alcohol-preferring ("P") rat. Impulsivity was measured via delay discounting (DD), and P rats exhibited an impulsive phenotype as compared to their outbred foundation strain-Wistar rats. Steeper discounting in P rats was associated with a lack of a prospective behavioral strategy, which was observed in Wistar rats and was directly related to DD. To further explore the underlying cognitive factors mediating these observations, a drift diffusion model of DD was constructed. These simulations supported the hypothesis that prospective memory of the delayed reward guided choice decisions, slowed discounting, and optimized the fit of the model to the experimental data. Collectively, these data suggest that a deficit in forming or maintaining a prospective behavioral plan is a critical intermediary to delaying reward, and by extension, may underlie the inability to delay reward in those with increased AUD risk.

Does goal relevant episodic future thinking amplify the effect on delay discounting?

Delay discounting (DD) is the preference for smaller immediate rewards over larger delayed rewards. Research shows episodic future thinking (EFT), or mentally simulating future experiences, reframes the choice between small immediate and larger delayed rewards, and can reduce DD. Only general EFT has been studied, whereby people reframe decisions in terms of non-goal related future events. Since future thinking is often goal-oriented and leads to greater activation of brain regions involved in prospection, goal-oriented EFT may be associated with greater reductions in DD than general goal-unrelated EFT. The present study (n=104, M_age=22.25, SD=3.42; 50% Female) used a between-subjects 2×2 factorial design with type of episodic thinking (Goal, General) and temporal perspective (Episodic future versus recent thinking; EFT vs ERT) as between factors. Results showed a significant reduction in DD for EFT groups (p<0.001, Cohen's d effect size=0.89), and goal-EFT was more effective than general-EFT on reducing DD (p=0.03, d=0.64).

The Vivid Present: Visualization Abilities Are Associated with Steep Discounting of Future Rewards

Humans and other animals discount the value of future rewards, a phenomenon known as delay discounting. Individuals vary widely in the extent to which they discount future rewards, and these tendencies have been associated with important life outcomes. Recent studies have demonstrated that imagining the future reduces subsequent discounting behavior, but no research to date has examined whether a similar principle applies at the trait level, and whether training visualization changes discounting. The current study examined if individual differences in visualization abilities are linked to individual differences in discounting and whether practicing visualization can change discounting behaviors in a lasting way. Participants (n = 48) completed the Vividness of Visual Imagery Questionnaire (VVIQ) and delay discounting task and then underwent a 4-week intervention consisting of visualization training (intervention) or relaxation training (control). Contrary to our hypotheses, participants who reported greater visualization abilities (lower scores) on the VVIQ were higher discounters. To further examine this relationship, an additional 106 participants completed the VVIQ and delay discounting task. In the total sample (n = 154), there was a significant negative correlation between VVIQ scores and discount rates, showing that individuals who are better visualizers are also higher discounters. Consistent with this relationship but again to our surprise, visualization training tended, albeit weakly, to increase discount rates, and those whose VVIQ decreased the most were those whose discount rates increased the most. These results suggest a novel association between visualization abilities and delay discounting.

Episodic Future Thinking about the Ideal Self Induces Lower Discounting, Leading to a Decreased Tendency toward Cheating

Delay discounting refers to a pervasive tendency toward preferring smaller immediate gains over larger future gains. Recent empirical research has shown that episodic future thinking (EFT; i.e., projecting oneself into the future to pre-experience forthcoming events) can reduce the tendency toward discounting. A common tenet of psychological theories of crime is that delinquency results from focusing on short-term gains while failing to consider adequately the longer-term consequences of delinquent behavior. We investigated whether an EFT intervention involving the ideal self could induce lower discounting rates and, as a consequence, reduced delinquency. The results showed that, compared with control participants, participants engaging in EFT, that is, envisaging life events that would be experienced by their ideal selves, exhibited a lower discounting rate in a monetary choice task (Experiments 1 and 2), as well as a decreased tendency to make delinquent choices in imaginary scenarios (Experiment 1) and cheat in a matrix task (Experiment 2). The discounting tendency mediated the relationship between engaging in EFT pertaining to the ideal self and the tendency toward morally questionable behavior (Experiments 1 and 2). The findings of the two experiments indicate that engagement in EFT with a focus on the ideal self is sufficient to induce lower discounting rates, by promoting consideration of distant costs and thus increasing resistance to delinquent involvement and cheating (given the temptation of the immediate benefits that may accrue from such behavior). The current research constitutes an innovative approach to delinquency prevention and the promotion of morality.

Simulating future value in intertemporal choice

The laboratory study of how humans and other animals trade-off value and time has a long and storied history, and is the subject of a vast literature. However, despite a long history of study, there is no agreed upon mechanistic explanation of how intertemporal choice preferences arise. Several theorists have recently proposed model-based reinforcement learning as a candidate framework. This framework describes a suite of algorithms by which a model of the environment, in the form of a state transition function and reward function, can be converted on-line into a decision. The state transition function allows the model-based system to make decisions based on projected future states, while the reward function assigns value to each state, together capturing the necessary components for successful intertemporal choice. Empirical work has also pointed to a possible relationship between increased prospection and reduced discounting. In the current paper, we look for direct evidence of a relationship between temporal discounting and model-based control in a large new data set (n = 168). However, testing the relationship under several different modeling formulations revealed no indication that the two quantities are related.

The relationship between intertemporal choice and following the path of least resistance across choices, preferences, and beliefs

The degree to which individuals prefer smaller sooner versus larger delayed rewards serves as a powerful predictor of their impulsivity towards a number of different kinds of rewards. Here we test the limits of its predictive ability within a variety of cognitive and social domains. Across several large samples of subjects, individuals who prefer smaller more immediate rewards (steeper discounters) are less reflective (or more impulsive) in their choices, preferences, and beliefs. First, steeper discounters used more automatic, less controlled choice strategies, giving more intuitive but incorrect responses on the Cognitive Reflection Test (replicating previous findings); employing a suboptimal probability matching heuristic for a one-shot gamble (rather than maximizing their probability of reward); and relying less on optimal planning in a two-stage reinforcement learning task. Second, steeper discounters preferred to consume information that was less complex and multi-faceted, as suggested by their self-reported Need for Cognitive Closure, their use of short-form social media (i.e., Twitter), and their preferred news sources (in particular, whether or not they preferred National Public Radio over other news sources). Third, steeper discounters had interpersonal and religious beliefs that are associated with reduced epistemic complexity: they were more likely to believe that the behavior of others could be explained by fixed rather than dynamic factors, and they believed more strongly in God and in the afterlife. Together these findings provide evidence for a link between individual differences in temporal discounting for monetary rewards and preferences for the path of least resistance (less reflective and/or more automatic modes of processing) across a variety of domains.

Serotonergic modulation of orbitofrontal activity and its relevance for decision making and impulsivity

BACKGROUND

The orbitofrontal cortex seems to play a crucial role in reward-guided learning and decision making, especially for impulsive choice procedures including delayed reward discounting. The central serotonergic system is closely involved in the regulation of impulsivity, but how the serotonergic firing rate and release, best investigated by the loudness dependence of auditory evoked potentials (LDAEP), interact with orbitofrontal activity is still unknown.

METHODS

Twenty healthy volunteers (11 males, 9 females, 31.3 ± 10.6 years old) were studied in a 3T MRI scanner (Philips, Hamburg, Germany) during a delay discounting task, after their LDAEP was recorded using a 32 electrodes EEG machine (Brain Products, Munich, Germany).

RESULTS

Significant positive correlations were only found between the LDAEP and the medial orbitofrontal part of the superior frontal gyrus (SFG/MO) [Δ immediate reward - delayed reward] for the right (r = 0.519; P = 0.019) and left side (r = 0.478; P = 0.033). This relationship was stronger for females compared with males. Orbitofrontal activity was also related to the Barratt Impulsivity Scale.

CONCLUSIONS

This study revealed that low serotonergic activity as measured by a strong LDAEP was related to a high fMRI signal intensity of SFG/MO during immediate reward behavior which is related to impulsivity. Since this relationship was only found for the infralimbic medial and not for the middle or lateral part of the orbitofrontal cortex, an exclusive projection tract of the serotonergic system to this cortical region can be assumed to regulate impulsive reward-orientated decision making. Hum Brain Mapp 38:1507-1517, 2017. © 2016 Wiley Periodicals, Inc.

Neural Systems Underlying Individual Differences in Intertemporal Decision-making

Excessively choosing immediate over larger future rewards, or delay discounting (DD), associates with multiple clinical conditions. Individual differences in DD likely depend on variations in the activation of and functional interactions between networks, representing possible endophenotypes for associated disorders, including alcohol use disorders (AUDs). Numerous fMRI studies have probed the neural bases of DD, but investigations of large-scale networks remain scant. We addressed this gap by testing whether activation within large-scale networks during Now/Later decision-making predicts individual differences in DD. To do so, we scanned 95 social drinkers (18-40 years old; 50 women) using fMRI during hypothetical choices between small monetary amounts available "today" or larger amounts available later. We identified neural networks engaged during Now/Later choice using independent component analysis and tested the relationship between component activation and degree of DD. The activity of two components during Now/Later choice correlated with individual DD rates: A temporal lobe network positively correlated with DD, whereas a frontoparietal-striatal network negatively correlated with DD. Activation differences between these networks predicted individual differences in DD, and their negative correlation during Now/Later choice suggests functional competition. A generalized psychophysiological interactions analysis confirmed a decrease in their functional connectivity during decision-making. The functional connectivity of these two networks negatively correlates with alcohol-related harm, potentially implicating these networks in AUDs. These findings provide novel insight into the neural underpinnings of individual differences in impulsive decision-making with potential implications for addiction and related disorders in which impulsivity is a defining feature.

Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex

The hippocampus and the orbitofrontal cortex (OFC) both have important roles in cognitive processes such as learning, memory and decision making. Nevertheless, research on the OFC and hippocampus has proceeded largely independently, and little consideration has been given to the importance of interactions between these structures. Here, evidence is reviewed that the hippocampus and OFC encode parallel, but interactive, cognitive 'maps' that capture complex relationships between cues, actions, outcomes and other features of the environment. A better understanding of the interactions between the OFC and hippocampus is important for understanding the neural bases of flexible, goal-directed decision making.

Using future thinking to reduce temporal discounting: Under what circumstances are the medial temporal lobes critical?

The capacity to envision the future plays an important role in many aspects of cognition, including our ability to make optimal, adaptive choices. Past work has shown that the medial temporal lobe (MTL) is necessary for decisions that draw on episodic future thinking. By contrast, little is known about the role of the MTL in decisions that draw on semantic future thinking. Accordingly, the present study investigated whether the MTL contributes to one form of decision making, namely intertemporal choice, when such decisions depend on semantic consideration of the future. In an intertemporal choice task, participants must select either a smaller amount of money that is available in the present or a larger amount of money that would be available at a future date. Amnesic individuals with MTL damage and healthy control participants performed such a task in which, prior to making a choice, they engaged in a semantic generation exercise, wherein they generated items that they would purchase with the future reward. In experiment 1, we found that, relative to a baseline condition involving standard intertemporal choice, healthy individuals were more inclined to select a larger, later reward over a smaller, present reward after engaging in semantic future thinking. By contrast, amnesic participants were paradoxically less inclined to wait for a future reward following semantic future thinking. This finding suggests that amnesics may have had difficulty "tagging" the generated item(s) as belonging to the future. Critically, experiment 2 showed that when the generated items were presented alongside the intertemporal choices, both controls and amnesic participants shifted to more patient choices. These findings suggest that the MTL is not needed for making optimal decisions that draw on semantic future thinking as long as scaffolding is provided to support accurate time tagging. Together, these findings stand to better clarify the role of the MTL in decision making.

Vicarious trial and error

When rats come to a decision point, they sometimes pause and look back and forth as if deliberating over the choice; at other times, they proceed as if they have already made their decision. In the 1930s, this pause-and-look behaviour was termed 'vicarious trial and error' (VTE), with the implication that the rat was 'thinking about the future'. The discovery in 2007 that the firing of hippocampal place cells gives rise to alternating representations of each of the potential path options in a serial manner during VTE suggested a possible neural mechanism that could underlie the representations of future outcomes. More-recent experiments examining VTE in rats suggest that there are direct parallels to human processes of deliberative decision making, working memory and mental time travel.

From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning

Theoretical models distinguish two decision-making strategies that have been formalized in reinforcement-learning theory. A model-based strategy leverages a cognitive model of potential actions and their consequences to make goal-directed choices, whereas a model-free strategy evaluates actions based solely on their reward history. Research in adults has begun to elucidate the psychological mechanisms and neural substrates underlying these learning processes and factors that influence their relative recruitment. However, the developmental trajectory of these evaluative strategies has not been well characterized. In this study, children, adolescents, and adults performed a sequential reinforcement-learning task that enabled estimation of model-based and model-free contributions to choice. Whereas a model-free strategy was apparent in choice behavior across all age groups, a model-based strategy was absent in children, became evident in adolescents, and strengthened in adults. These results suggest that recruitment of model-based valuation systems represents a critical cognitive component underlying the gradual maturation of goal-directed behavior.

Prospection and the Present Moment: The Role of Episodic Foresight in Intertemporal Choices between Immediate and Delayed Rewards

Humans are capable of imagining future rewards and the contexts in which they may be obtained. Functionally, intertemporal choices between smaller but immediate and larger but delayed rewards may be made without such episodic foresight. However, we propose that explicit simulations of this sort enable more flexible and adaptive intertemporal decision-making. Emotions triggered through the simulation of future situations can motivate people to forego immediate pleasures in the pursuit of long-term rewards. However, we stress that the most adaptive option need not always be a larger later reward. When the future is anticipated to be uncertain, for instance, it may make sense for preferences to shift toward more immediate rewards, instead. Imagining potential future scenarios and assessment of their likelihood and affective consequences allows humans to determine when it is more adaptive to delay gratification in pursuit of a larger later reward, and when the better strategy is to indulge in a present temptation. We discuss clinical studies that highlight when and how the effect of episodic foresight on intertemporal decision-making can be altered, and consider the relevance of this perspective to understanding the nature of self-control.

The Computational Complexity of Valuation and Motivational Forces in Decision-Making Processes

The concept of value is fundamental to most theories of motivation and decision making. However, value has to be measured experimentally. Different methods of measuring value produce incompatible valuation hierarchies. Taking the agent's perspective (rather than the experimenter's), we interpret the different valuation measurement methods as accessing different decision-making systems and show how these different systems depend on different information processing algorithms. This identifies the translation from these multiple decision-making systems into a single action taken by a given agent as one of the most important open questions in decision making today. We conclude by looking at how these different valuation measures accessing different decision-making systems can be used to understand and treat decision dysfunction such as in addiction.

On Weight and Waiting: Delay Discounting in Anorexia Nervosa Pretreatment and Posttreatment

BACKGROUND

Individuals with anorexia nervosa (AN) override the drive to eat, forgoing immediate rewards in favor of longer-term goals. We examined delay discounting and its neural correlates in AN before and after treatment to test a potential mechanism of illness persistence.

METHODS

Inpatients with AN (n = 59) and healthy control subjects (HC, n = 39) performed a delay discounting task at two time points. A subset (n = 30 AN, n = 22 HC) participated in functional magnetic resonance imaging scanning during the task. The task consisted of a range of monetary choices with variable delay times, yielding individual discount rates-the rate by which money loses value over time.

RESULTS

Before treatment, the AN group showed a preference for delayed over earlier rewards (i.e., less steep discount rates) compared with HC; after weight restoration, AN did not differ from HC. Underweight AN showed slower response times for earlier versus delayed choices; this reversed with treatment. Underweight AN showed abnormal neural activity in striatum and dorsal anterior cingulate; normalization of behavior was associated with increased activation in reward regions (striatum and dorsal anterior cingulate) and decision-making regions (dorsolateral prefrontal cortex and parietal cortex).

CONCLUSIONS

The undernourished state of AN may amplify the tendency to forgo immediate rewards in favor of longer-term goals. The results suggest that behavior that looks phenotypically like excessive self-control does not correspond with enhanced prefrontal recruitment. Rather, the results point to alterations in cingulostriatal circuitry that offer new insights on the potential role of abnormalities in decision-making neural systems in the perpetuation of AN.