Deliberation and Procedural Automation on a Two-Step Task for Rats

Hippocampal representations of alternative possibilities are flexibly generated to meet cognitive demands

The cognitive ability to go beyond the present to consider alternative possibilities, including potential futures and counterfactual pasts, can support adaptive decision making. Complex and changing real-world environments, however, have many possible alternatives. Whether and how the brain can select among them to represent alternatives that meet current cognitive needs remains unknown. We therefore examined neural representations of alternative spatial locations in the rat hippocampus during navigation in a complex patch foraging environment with changing reward probabilities. We found representations of multiple alternatives along paths ahead and behind the animal, including in distant alternative patches. Critically, these representations were modulated in distinct patterns across successive trials: alternative paths were represented proportionate to their evolving relative value and predicted subsequent decisions, whereas distant alternatives were prevalent during value updating. These results demonstrate that the brain modulates the generation of alternative possibilities in patterns that meet changing cognitive needs for adaptive behavior.

Pet Owners' Preferences for Quality of Life Improvements and Costs Related to Innovative Therapies in Feline Pain Associated with Osteoarthritis-A Quantitative Survey

This research aimed to explore UK cat owners' preferences for treatments for feline osteoarthritis (OA) by exploring preferences around quality of life (QoL) improvements, safety considerations, and costs associated with hypothetical innovative pain therapies. Aspects identified in an existing conceptual framework were extracted for inclusion in exploratory interviews with cat owners (n = 3) to identify key domains that contribute to the QoL of cats. QoL descriptions for cats with OA and hypothetical product attributes were developed and validated through interviews with veterinarians (n = 3). An online survey was subsequently shared with 255 pet owners in the UK. Pet owners were presented with QoL descriptions and hypothetical product attributes to gather their preferences for QoL improvements and their willingness to pay (WTP) for (unbranded) pain therapies at various price points. Pet owners were motivated to improve their cats' QoL, which translated into WTP for therapies; specifically, pet owners valued QoL improvements in mobility, pain expression, and well-being. When presented with a product profile of the hypothetical novel monoclonal antibody (mAb) and cost, 50% of cat owners were willing to pay more for a mAb that is expected to have improved efficacy and safety when compared to a hypothetical standard of care (SoC). Significantly more pet owners preferred the mAb than the SoC when price was not presented (p < 0.01), with product efficacy and safety driving pet owners' decision-making. The majority of pet owners did not agree that taking their cats to the veterinarian once a month for their treatment would be burdensome. Cat owners in the UK are motivated to improve their cats' QoL, which translates into WTP for the efficacious treatment of pain associated with osteoarthritis. Veterinarians should offer cat owners the pain treatment they feel is best suited for improving the cat's QoL and to ensure subsequent owner-pet bond is preserved.

Rodent maze studies: from following simple rules to complex map learning

More than 100 years since the first maze designed for rodent research, researchers now have the choice of a variety of mazes that come in many different shapes and sizes. Still old designs get modified and new designs are introduced to fit new research questions. Yet, which maze is the most optimal to use or which training paradigm should be applied, remains up for debate. In this review, we not only provide a historical overview of maze designs and usages in rodent learning and memory research, but also discuss the possible navigational strategies the animals can use to solve each maze. Furthermore, we summarize the different phases of learning that take place when a maze is used as the experimental task. At last, we delve into how training and maze design can affect what the rodents are actually learning in a spatial task.

Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning

Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step,reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.

Diabetes alters neuroeconomically dissociable forms of mental accounting

Those with diabetes mellitus are at high-risk of developing psychiatric disorders, yet the link between hyperglycemia and alterations in motivated behavior has not been explored in detail. We characterized value-based decision-making behavior of a streptozocin-induced diabetic mouse model on a naturalistic neuroeconomic foraging paradigm called Restaurant Row. Mice made self-paced choices while on a limited time-budget accepting or rejecting reward offers as a function of cost (delays cued by tone-pitch) and subjective value (flavors), tested daily in a closed-economy system across months. We found streptozocin-treated mice disproportionately undervalued less-preferred flavors and inverted their meal-consumption patterns shifted toward a more costly strategy that overprioritized high-value rewards. We discovered these foraging behaviors were driven by impairments in multiple decision-making systems, including the ability to deliberate when engaged in conflict and cache the value of the passage of time in the form of sunk costs. Surprisingly, diabetes-induced changes in behavior depended not only on the type of choice being made but also the salience of reward-scarcity in the environment. These findings suggest complex relationships between glycemic regulation and dissociable valuation algorithms underlying unique cognitive heuristics and sensitivity to opportunity costs can disrupt fundamentally distinct computational processes and could give rise to psychiatric vulnerabilities.

Computational approaches to modeling gambling behaviour: Opportunities for understanding disordered gambling

Computational modeling has become an important tool in neuroscience and psychiatry research to provide insight into the cognitive processes underlying normal and pathological behavior. There are two modeling frameworks, reinforcement learning (RL) and drift diffusion modeling (DDM), that are well-developed in cognitive science, and have begun to be applied to Gambling Disorder. RL models focus on explaining how an agent uses reward to learn about the environment and make decisions based on outcomes. The DDM is a binary choice framework that breaks down decision making into psychologically meaningful components based on choice reaction time analyses. Both approaches have begun to yield insight into aspects of cognition that are important for, but not unique to, gambling, and thus relevant to the development of Gambling Disorder. However, these approaches also oversimplify or neglect various aspects of decision making seen in real-world gambling behavior. Gambling Disorder presents an opportunity for 'bespoke' modeling approaches to consider these neglected components. In this review, we discuss studies that have used RL and DDM frameworks to investigate some of the key cognitive components in gambling and Gambling Disorder. We also include an overview of Bayesian models, a methodology that could be useful for more tailored modeling approaches. We highlight areas in which computational modeling could enable progression in the investigation of the cognitive mechanisms relevant to gambling.

Development of cross-species translational paradigms for psychiatric research in the Research Domain Criteria era

The past 30 years of IBNS has included research attempting to treat the cognitive and behavioral deficits observed in people with psychiatric conditions. Early work utilized drugs identified from tests thought to be cognition-relevant, however the high failure rate crossing the translational-species barrier led to focus on developing valid cross-species translational tests. The face, predictive, and neurobiological validities used to assess animal models of psychiatry can be used to validate these tests. Clinical sensitivity is another important aspect however, for if the clinical population targeted for treatment does not exhibit task deficits, then why develop treatments? This review covers some work validating cross-species translational tests and suggests future directions. Also covered is the contribution IBNS made to fostering such research and my role in IBNS, making it more available to all including fostering mentor/mentee programs plus spearheading diversity and inclusivity initiatives. All science needs support and IBNS has supported research recreating the behavioral abnormalities that define psychiatric conditions with the aim to improve the lives of people with such conditions.

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

Value representations in the rodent orbitofrontal cortex drive learning, not choice

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

Beyond Description and Deficits: How Computational Psychiatry Can Enhance an Understanding of Decision-Making in Anorexia Nervosa

PURPOSE OF REVIEW

Despite decades of research, knowledge of the mechanisms maintaining anorexia nervosa (AN) remains incomplete and clearly effective treatments elusive. Novel theoretical frameworks are needed to advance mechanistic and treatment research for this disorder. Here, we argue the utility of engaging a novel lens that differs from existing perspectives in psychiatry. Specifically, we argue the necessity of expanding beyond two historically common perspectives: (1) the descriptive perspective: the tendency to define mechanisms on the basis of surface characteristics and (2) the deficit perspective: the tendency to search for mechanisms associated with under-functioning of decision-making abilities and related circuity, rather than problems of over-functioning, in psychiatric disorders.

RECENT FINDINGS

Computational psychiatry can provide a novel framework for understanding AN because this approach emphasizes the role of computational misalignments (rather than absolute deficits or excesses) between decision-making strategies and environmental demands as the key factors promoting psychiatric illnesses. Informed by this approach, we argue that AN can be understood as a disorder of excess goal pursuit, maintained by over-engagement, rather than disengagement, of executive functioning strategies and circuits. Emerging evidence suggests that this same computational imbalance may constitute an under-investigated phenotype presenting transdiagnostically across psychiatric disorders. A variety of computational models can be used to further elucidate excess goal pursuit in AN. Most traditional psychiatric treatments do not target excess goal pursuit or associated neurocognitive mechanisms. Thus, targeting at the level of computational dysfunction may provide a new avenue for enhancing treatment for AN and related disorders.

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

Purpose of Review

Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.

Recent Findings

The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.

Summary

While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.

The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection

Behavioral control is not unitary. It comprises parallel systems, model based and model free, that respectively generate flexible and habitual behaviors. Model-based decisions use predictions of the specific consequences of actions, but how these are implemented in the brain is poorly understood. We used calcium imaging and optogenetics in a sequential decision task for mice to show that the anterior cingulate cortex (ACC) predicts the state that actions will lead to, not simply whether they are good or bad, and monitors whether outcomes match these predictions. ACC represents the complete state space of the task, with reward signals that depend strongly on the state where reward is obtained but minimally on the preceding choice. Accordingly, ACC is necessary only for updating model-based strategies, not for basic reward-driven action reinforcement. These results reveal that ACC is a critical node in model-based control, with a specific role in predicting future states given chosen actions.

Distinct roles for dopamine clearance mechanisms in regulating behavioral flexibility

Dopamine plays a crucial role in adaptive behavior, and dysfunctional dopamine is implicated in multiple psychiatric conditions characterized by inflexible or inconsistent choices. However, the precise relationship between dopamine and flexible decision making remains unclear. One reason is that, while many studies have focused on the activity of dopamine neurons, efficient dopamine signaling also relies on clearance mechanisms, notably the dopamine transporter (DAT), which predominates in striatum, and catechol-O-methyltransferase (COMT), which predominates in cortex. The exact locus, extent, and timescale of the effects of DAT and COMT are uncertain. Moreover, there is limited data on how acute disruption of either mechanism affects flexible decision making strategies mediated by cortico-striatal networks. To address these issues, we combined pharmacological modulation of DAT and COMT with electrochemistry and behavior in mice. DAT blockade, but not COMT inhibition, regulated sub-second dopamine release in the nucleus accumbens core, but surprisingly neither clearance mechanism affected evoked release in prelimbic cortex. This was not due to a lack of sensitivity, as both amphetamine and atomoxetine changed the kinetics of sub-second release. In a multi-step decision making task where mice had to respond to reversals in either reward probabilities or the choice sequence to reach the goal, DAT blockade selectively impaired, and COMT inhibition improved, performance after reward reversals, but neither manipulation affected the adaptation of choices after action-state transition reversals. Together, our data suggest that DAT and COMT shape specific aspects of behavioral flexibility by regulating different aspects of the kinetics of striatal and cortical dopamine, respectively.

Potential roles of the rodent medial prefrontal cortex in conflict resolution between multiple decision-making systems

Mammalian decision-making is mediated by the interaction of multiple, neurally and computationally separable decision systems. Having multiple systems requires a mechanism to manage conflict and converge onto the selection of singular actions. A long history of evidence has pointed to the prefrontal cortex as a central component in processing the interactions between distinct decision systems and resolving conflicts among them. In this chapter we review four theories of how that interaction might occur and identify how the medial prefrontal cortex in the rodent may be involved in each theory. We then present experimental predictions implied by the neurobiological data in the context of each theory as a starting point for future investigation of medial prefrontal cortex and decision-making.

The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals

Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.

Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning

It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. However, the role of task complexity in the arbitration between these two strategies remains largely unknown. Here, using a combination of novel task design, computational modelling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process. Participants tended to increase model-based RL control in response to increasing task complexity. However, they resorted to model-free RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex.

The brain dynamically arbitrates between two model-based and model-free reinforcement learning (RL). Here, the authors show that participants tended to increase model-based control in response to increasing task complexity, but resorted to model-free when both uncertainty and task complexity were high.

Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world.

The perirhinal cortex supports spatial intertemporal choice stability

In order to optimize outcomes in the face of uncertainty, one must recall past experiences and extrapolate to the future by assigning values to different choice outcomes. This behavior requires an interplay between memory and reward valuation, necessitating communication across many brain regions. At the anatomical nexus of this interplay is the perirhinal cortex (PRC). The PRC is densely connected to the amygdala and orbital frontal cortex, regions that have been implicated in reward-based decision making, as well as the hippocampus. Thus, the PRC could serve as a hub for integrating memory, reward, and prediction. The PRC's role in value-based decision making, however, has not been empirically examined. Therefore, we tested the role of the PRC in a spatial delay discounting task, which allows rats to choose between a 1-s delay for a small food reward and a variable delay for a large food reward, with the delay to the large reward increasing after choice of each large reward and decreasing after each small reward. The rat can therefore adjust the delay by consecutively choosing the same reward or stabilize the delay by alternating between sides. The latter has been shown to occur once the 'temporal cost' of the large reward is established and is a decision-making process termed 'exploitation'. When the PRC was bilaterally inactivated with the GABA(A) agonist muscimol, rats spent fewer trials successfully exploiting to maintain a fixed delay compared to the vehicle control condition. Moreover, PRC inactivation resulted in an increased number of vicarious trial and error (VTE) events at the choice point, where rats had to decide between the two rewards. These behavioral patterns suggest that the PRC is critical for maintaining stability in linking a choice to a reward outcome in the face of a variable cost.