Anxiety, avoidance, and sequential evaluation

Beliefs, compulsive behavior and reduced confidence in control

OCD has been conceptualized as a disorder arising from dysfunctional beliefs, such as overestimating threats or pathological doubts. Yet, how these beliefs lead to compulsions and obsessions remains unclear. Here, we develop a computational model to examine the specific beliefs that trigger and sustain compulsive behavior in a simple symptom-provoking scenario. Our results demonstrate that a single belief disturbance-a lack of confidence in the effectiveness of one's preventive (harm-avoiding) actions-can trigger and maintain compulsions and is directly linked to compulsion severity. This distrust can further explain a number of seemingly unrelated phenomena in OCD, including the role of not-just-right feelings, the link to intolerance to uncertainty, perfectionism, and overestimation of threat, and deficits in reversal and state learning. Our simulations shed new light on which underlying beliefs drive compulsive behavior and highlight the important role of perceived ability to exert control for OCD.

Thinking computationally in translational psychiatry. A commentary on Neville et al. (2024)

There is a growing focus on the computational aspects of psychiatric disorders in humans. This idea also is gaining traction in nonhuman animal studies. Commenting on a new comprehensive overview of the benefits of applying this approach in translational research by Neville et al. (Cognitive Affective & Behavioral Neuroscience 1-14, 2024), we discuss the implications for translational model validity within this framework. We argue that thinking computationally in translational psychiatry calls for a change in the way that we evaluate animal models of human psychiatric processes, with a shift in focus towards symptom-producing computations rather than the symptoms themselves. Further, in line with Neville et al.'s adoption of the reinforcement learning framework to model animal behaviour, we illustrate how this approach can be applied beyond simple decision-making paradigms to model more naturalistic behaviours.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Affect-congruent attention modulates generalized reward expectations

Positive and negative affective states are respectively associated with optimistic and pessimistic expectations regarding future reward. One mechanism that might underlie these affect-related expectation biases is attention to positive- versus negative-valence features (e.g., attending to the positive reviews of a restaurant versus its expensive price). Here we tested the effects of experimentally induced positive and negative affect on feature-based attention in 120 participants completing a compound-generalization task with eye-tracking. We found that participants' reward expectations for novel compound stimuli were modulated in an affect-congruent way: positive affect induction increased reward expectations for compounds, whereas negative affect induction decreased reward expectations. Computational modelling and eye-tracking analyses each revealed that these effects were driven by affect-congruent changes in participants' allocation of attention to high- versus low-value features of compounds. These results provide mechanistic insight into a process by which affect produces biases in generalized reward expectations.

The computational psychopathology of emotion

Mood and anxiety disorders involve recurring, maladaptive patterns of distinct emotions and moods. Here, we argue that understanding these maladaptive patterns first requires understanding how emotions and moods guide adaptive behavior. We thus review recent progress in computational accounts of emotion that aims to explain the adaptive role of distinct emotions and mood. We then highlight how this emerging approach could be used to explain maladaptive emotions in various psychopathologies. In particular, we identify three computational factors that may be responsible for excessive emotions and moods of different types: self-intensifying affective biases, misestimations of predictability, and misestimations of controllability. Finally, we outline how the psychopathological roles of these factors can be tested, and how they may be used to improve psychotherapeutic and psychopharmacological interventions.

Placing Approach-Avoidance Conflict Within the Framework of Multi-objective Reinforcement Learning

Many psychiatric disorders are marked by impaired decision-making during an approach-avoidance conflict. Current experiments elicit approach-avoidance conflicts in bandit tasks by pairing an individual's actions with consequences that are simultaneously desirable (reward) and undesirable (harm). We frame approach-avoidance conflict tasks as a multi-objective multi-armed bandit. By defining a general decision-maker as a limiting sequence of actions, we disentangle the decision process from learning. Each decision maker can then be identified as a multi-dimensional point representing its long-term average expected outcomes, while different decision making models can be associated by the geometry of their 'feasible region', the set of all possible long term performances on a fixed task. We introduce three example decision-makers based on popular reinforcement learning models and characterize their feasible regions, including whether they can be Pareto optimal. From this perspective, we find that existing tasks are unable to distinguish between the three examples of decision-makers. We show how to design new tasks whose geometric structure can be used to better distinguish between decision-makers. These findings are expected to guide the design of approach-avoidance conflict tasks and the modeling of resulting decision-making behavior.

Investigating links between creativity anxiety, creative performance, and state-level anxiety and effort during creative thinking

Identifying ways to enable people to reach their creative potential is a core goal of creativity research with implications for education and professional attainment. Recently, we identified a potential barrier to creative achievement: creativity anxiety (i.e., anxiety specific to creative thinking). Initial work found that creativity anxiety is associated with fewer real-world creative achievements. However, the more proximal impacts of creativity anxiety remain unexplored. In particular, understanding how to overcome creativity anxiety requires understanding how creativity anxiety may or may not impact creative cognitive performance, and how it may relate to state-level anxiety and effort while completing creative tasks. The present study sought to address this gap by measuring creativity anxiety alongside several measures of creative performance, while concurrently surveying state-level anxiety and effort. Results indicated that creativity anxiety was, indeed, predictive of poor creative performance, but only on some of the tasks included. We also found that creativity anxiety predicted both state anxiety and effort during creative performance. Interestingly, state anxiety and effort did not explain the associations between creativity anxiety and creative performance. Together, this work suggests that creativity anxiety can often be overcome in the performance of creative tasks, but likewise points to increased state anxiety and effort as factors that may make creative performance and achievement fragile in more demanding real-world contexts.

Interactive cognitive maps support flexible behavior under threat

In social environments, survival can depend upon inferring and adapting to other agents' goal-directed behavior. However, it remains unclear how humans achieve this, despite the fact that many decisions must account for complex, dynamic agents acting according to their own goals. Here, we use a predator-prey task (total n = 510) to demonstrate that humans exploit an interactive cognitive map of the social environment to infer other agents' preferences and simulate their future behavior, providing for flexible, generalizable responses. A model-based inverse reinforcement learning model explained participants' inferences about threatening agents' preferences, with participants using this inferred knowledge to enact generalizable, model-based behavioral responses. Using tree-search planning models, we then found that behavior was best explained by a planning algorithm that incorporated simulations of the threat's goal-directed behavior. Our results indicate that humans use a cognitive map to determine other agents' preferences, facilitating generalized predictions of their behavior and effective responses.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Disrupted state transition learning as a computational marker of compulsivity

BACKGROUND

Disorders involving compulsivity, fear, and anxiety are linked to beliefs that the world is less predictable. We lack a mechanistic explanation for how such beliefs arise. Here, we test a hypothesis that in people with compulsivity, fear, and anxiety, learning a probabilistic mapping between actions and environmental states is compromised.

METHODS

In Study 1 (n = 174), we designed a novel online task that isolated state transition learning from other facets of learning and planning. To determine whether this impairment is due to learning that is too fast or too slow, we estimated state transition learning rates by fitting computational models to two independent datasets, which tested learning in environments in which state transitions were either stable (Study 2: n = 1413) or changing (Study 3: n = 192).

RESULTS

Study 1 established that individuals with higher levels of compulsivity are more likely to demonstrate an impairment in state transition learning. Preliminary evidence here linked this impairment to a common factor comprising compulsivity and fear. Studies 2 and 3 showed that compulsivity is associated with learning that is too fast when it should be slow (i.e. when state transition are stable) and too slow when it should be fast (i.e. when state transitions change).

CONCLUSIONS

Together, these findings indicate that compulsivity is associated with a dysregulation of state transition learning, wherein the rate of learning is not well adapted to the task environment. Thus, dysregulated state transition learning might provide a key target for therapeutic intervention in compulsivity.

Breathwork Interventions for Adults with Clinically Diagnosed Anxiety Disorders: A Scoping Review

Anxiety disorders are the most common group of mental disorders, but they are often underrecognized and undertreated in primary care. Dysfunctional breathing is a hallmark of anxiety disorders; however, mainstays of treatments do not tackle breathing in patients suffering anxiety. This scoping review aims to identify the nature and extent of the available research literature on the efficacy of breathwork interventions for adults with clinically diagnosed anxiety disorders using the DSM-5 classification system. Using the PRISMA extension for scoping reviews, a search of PubMed, Embase, and Scopus was conducted using terms related to anxiety disorders and breathwork interventions. Only clinical studies using breathwork (without the combination of other interventions) and performed on adult patients diagnosed with an anxiety disorder using the DSM-5 classification system were included. From 1081 articles identified across three databases, sixteen were included for the review. A range of breathwork interventions yielded significant improvements in anxiety symptoms in patients clinically diagnosed with anxiety disorders. The results around the role of hyperventilation in treatment of anxiety were contradictory in few of the examined studies. This evidence-based review supports the clinical utility of breathwork interventions and discusses effective treatment options and protocols that are feasible and accessible to patients suffering anxiety. Current gaps in knowledge for future research directions have also been identified.

Managing fear and anxiety in development: A framework for understanding the neurodevelopment of emotion regulation capacity and tendency

How we manage emotional responses to environmental threats is central to mental health, as difficulties regulating threat-related distress can blossom into symptoms of anxiety disorders. Given that anxiety disorders emerge early in the lifespan, it is crucial we understand the multi-level processes that support effective regulation of distress. Scholars have given increased attention to behavioral and neural development of emotion regulation abilities, particularly cognitive reappraisal capacity (i.e., how strongly one can down-regulate negative affect by reinterpreting a situation to change one's emotions). However, this work has not been well integrated with research on regulatory tendency (i.e., how often one spontaneously regulates emotion in daily life). Here, we review research on the development of both emotion regulation capacity and tendency. We then propose a framework for testing hypotheses and eventually constructing a neurodevelopmental model of both dimensions of emotion regulation. Clarifying how the brain supports both effective and frequent regulation of threat-related distress across development is crucial to identifying multi-level signs of dysregulation and developing interventions that support youth mental health.

A rubric for human-like agents and NeuroAI

Researchers across cognitive, neuro- and computer sciences increasingly reference 'human-like' artificial intelligence and 'neuroAI'. However, the scope and use of the terms are often inconsistent. Contributed research ranges widely from mimicking behaviour, to testing machine learning methods as neurally plausible hypotheses at the cellular or functional levels, or solving engineering problems. However, it cannot be assumed nor expected that progress on one of these three goals will automatically translate to progress in others. Here, a simple rubric is proposed to clarify the scope of individual contributions, grounded in their commitments to human-like behaviour, neural plausibility or benchmark/engineering/computer science goals. This is clarified using examples of weak and strong neuroAI and human-like agents, and discussing the generative, corroborate and corrective ways in which the three dimensions interact with one another. The author maintains that future progress in artificial intelligence will need strong interactions across the disciplines, with iterative feedback loops and meticulous validity tests-leading to both known and yet-unknown advances that may span decades to come. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty

Anxiety has been related to decreased physical exploration, but past findings on the interaction between anxiety and exploration during decision making were inconclusive. Here we examined how latent factors of trait anxiety relate to different exploration strategies when facing volatility-induced uncertainty. Across two studies (total N = 985), we demonstrated that people used a hybrid of directed, random and undirected exploration strategies, which were respectively sensitive to relative uncertainty, total uncertainty and value difference. Trait somatic anxiety, that is, the propensity to experience physical symptoms of anxiety, was inversely correlated with directed exploration and undirected exploration, manifesting as a lesser likelihood for choosing the uncertain option and reducing choice stochasticity regardless of uncertainty. Somatic anxiety is also associated with underestimation of relative uncertainty. Together, these results reveal the selective role of trait somatic anxiety in modulating both uncertainty-driven and value-driven exploration strategies.

Emotions as computations

Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

Even in favorable circumstances, we often find it hard to remain happy with what we have. One might enjoy a newly bought car for a season, but over time it brings fewer positive feelings and one eventually begins dreaming of the next rewarding thing to pursue. Here, we present a series of computational simulations that suggest these presumable “flaws” might play an important role in promoting adaptive behavior. We explore the value of prior expectations and relative comparisons as a useful reward signal and find that across a wide range of environments, these features help an agent learn faster and adapt better to changes in the environment. Our simulations also highlight scenarios when these relative features can be harmful to decision-making and happiness. Together, our results help explain why we have the propensity to keep wanting more, even if it contributes to depression, materialism, and overconsumption.

Validity and Reliability of the Anxiety Assessment Scale: A New Three-dimensional Perspective

Objective:

Anxiety is inseparable from life due to its survival value. Up-to-date and multidimensional assessment of anxiety is necessary to develop effective interventions to cope with high anxiety levels. This study was conducted to examine the psychometrics of the Anxiety Assessment Scale (AAS).

Methods:

Data were collected between January and April 2021 from 756 students (42.9% males and 57.1% females) studying medicine at Istanbul Medeniyet University. Seven experts evaluated the items to detect content validity in the final application form. Both exploratory and confirmatory factor analyses (EFA and CFA) were used for construct validity. The Beck Anxiety Inventory was also applied for concurrent validity. Test-retest reliabilities were calculated within four weeks. IBM SPSS 25 and AMOS 24 were used for statistical analyses.

Results:

Data were suitable for factor analyses (Kaiser-Meyer-Olkin=0.800, chi-square=3018.854, df=45). The EFA showed the three-factor structure with 10 items, and 70.1% of the variance was explained. Factor loads of the items varied between 0.61 and 0.87; data-model fit was suitable (CFI=0.92, TLI=0.93, RMSEA=0.059, SRMR=0.046, chi-square/df=1.556) according to CFA. Concurrent scale validity was also confirmed by the Pearson correlation (r=0.167, p<0.01). The test-retest reliabilities (r) were all >0.5 (p<0.001). The Cronbach a coefficients were 0.845 (AAS), 0.770 (Physiological Tension=PT), 0.822 (Worrying=W), and 0.838 (Feeling Unsafe=FU).

Conclusions:

AAS is a reliable and valid measurement instrument to assess anxiety levels in three dimensions. AAS can be applied for research, psychological assessment, and other appropriate application purposes.

Mathematical modeling of risk-taking in bipolar disorder: Evidence of reduced behavioral consistency, with altered loss aversion specific to those with history of substance use disorder

Bipolar disorder (BD) is associated with excessive pleasure-seeking risk-taking behaviors that often characterize its clinical presentation. However, the mechanisms of risk-taking behavior are not well-understood in BD. Recent data suggest prior substance use disorder (SUD) in BD may represent certain trait-level vulnerabilities for risky behavior. This study examined the mechanisms of risk-taking and the role of SUD in BD via mathematical modeling of behavior on the Balloon Analogue Risk Task (BART). Three groups-18 euthymic BD with prior SUD (BD+), 15 euthymic BD without prior SUD (BD-), and 33 healthy comparisons (HC)-completed the BART. We modeled behavior using 4 competing hierarchical Bayesian models, and model comparison results favored the Exponential-Weight Mean-Variance (EWMV) model, which encompasses and delineates five cognitive components of risk-taking: prior belief, learning rate, risk preference, loss aversion, and behavioral consistency. Both BD groups, regardless of SUD history, showed lower behavioral consistency than HC. BD+ exhibited more pessimistic prior beliefs (relative to BD- and HC) and reduced loss aversion (relative to HC) during risk-taking on the BART. Traditional measures of risk-taking on the BART (adjusted pumps, total points, total pops) detected no group differences. These findings suggest that reduced behavioral consistency is a crucial feature of risky decision-making in BD and that SUD history in BD may signal additional trait vulnerabilities for risky behavior even when mood symptoms and substance use are in remission. This study also underscores the value of using mathematical modeling to understand behavior in research on complex disorders like BD.

Increased and biased deliberation in social anxiety

A goal of computational psychiatry is to ground symptoms in basic mechanisms. Theory suggests that avoidance in anxiety disorders may reflect dysregulated mental simulation, a process for evaluating candidate actions. If so, these covert processes should have observable consequences: choices reflecting increased and biased deliberation. In two online general population samples, we examined how self-report symptoms of social anxiety disorder predict choices in a socially framed reinforcement learning task, the patent race, in which the pattern of choices reflects the content of deliberation. Using a computational model to assess learning strategy, we found that self-report social anxiety was indeed associated with increased deliberative evaluation. This effect was stronger for a particular subset of feedback ('upward counterfactual') in one of the experiments, broadly matching the biased content of rumination in social anxiety disorder, and robust to controlling for other psychiatric symptoms. These results suggest a grounding of symptoms of social anxiety disorder in more basic neuro-computational mechanisms.

Computational Psychiatry Needs Time and Context

Why has computational psychiatry yet to influence routine clinical practice? One reason may be that it has neglected context and temporal dynamics in the models of certain mental health problems. We develop three heuristics for estimating whether time and context are important to a mental health problem: Is it characterized by a core neurobiological mechanism? Does it follow a straightforward natural trajectory? And is intentional mental content peripheral to the problem? For many problems the answers are no, suggesting that modeling time and context is critical. We review computational psychiatry advances toward this end, including modeling state variation, using domain-specific stimuli, and interpreting differences in context. We discuss complementary network and complex systems approaches. Novel methods and unification with adjacent fields may inspire a new generation of computational psychiatry. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Linear reinforcement learning in planning, grid fields, and cognitive control

It is thought that the brain’s judicious reuse of previous computation underlies our ability to plan flexibly, but also that inappropriate reuse gives rise to inflexibilities like habits and compulsion. Yet we lack a complete, realistic account of either. Building on control engineering, here we introduce a model for decision making in the brain that reuses a temporally abstracted map of future events to enable biologically-realistic, flexible choice at the expense of specific, quantifiable biases. It replaces the classic nonlinear, model-based optimization with a linear approximation that softly maximizes around (and is weakly biased toward) a default policy. This solution demonstrates connections between seemingly disparate phenomena across behavioral neuroscience, notably flexible replanning with biases and cognitive control. It also provides insight into how the brain can represent maps of long-distance contingencies stably and componentially, as in entorhinal response fields, and exploit them to guide choice even under changing goals.

Models of decision making have so far been unable to account for how humans’ choices can be flexible yet efficient. Here the authors present a linear reinforcement learning model which explains both flexibility, and rare limitations such as habits, as arising from efficient approximate computation

Model-based aversive learning in humans is supported by preferential task state reactivation

Harm avoidance is critical for survival, yet little is known regarding the neural mechanisms supporting avoidance in the absence of trial-and-error experience. Flexible avoidance may be supported by a mental model (i.e., model-based), a process for which neural reactivation and sequential replay have emerged as candidate mechanisms. During an aversive learning task, combined with magnetoencephalography, we show prospective and retrospective reactivation during planning and learning, respectively, coupled to evidence for sequential replay. Specifically, when individuals plan in an aversive context, we find preferential reactivation of subsequently chosen goal states. Stronger reactivation is associated with greater hippocampal theta power. At outcome receipt, unchosen goal states are reactivated regardless of outcome valence. Replay of paths leading to goal states was modulated by outcome valence, with aversive outcomes associated with stronger reverse replay than safe outcomes. Our findings are suggestive of avoidance involving simulation of unexperienced states through hippocampally mediated reactivation and replay.

Advances in the computational understanding of mental illness

Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.

Beyond the Average View of Dopamine

Dopamine (DA) responses are synonymous with the 'reward prediction error' of reinforcement learning (RL), and are thought to update neural estimates of expected value. A recent study by Dabney et al. enriches this picture, demonstrating that DA neurons track variability in rewards, providing a readout of risk in the brain.

Inferring learning rules from animal decision-making

How do animals learn? This remains an elusive question in neuroscience. Whereas reinforcement learning often focuses on the design of algorithms that enable artificial agents to efficiently learn new tasks, here we develop a modeling framework to directly infer the empirical learning rules that animals use to acquire new behaviors. Our method efficiently infers the trial-to-trial changes in an animal's policy, and decomposes those changes into a learning component and a noise component. Specifically, this allows us to: (i) compare different learning rules and objective functions that an animal may be using to update its policy; (ii) estimate distinct learning rates for different parameters of an animal's policy; (iii) identify variations in learning across cohorts of animals; and (iv) uncover trial-to-trial changes that are not captured by normative learning rules. After validating our framework on simulated choice data, we applied our model to data from rats and mice learning perceptual decision-making tasks. We found that certain learning rules were far more capable of explaining trial-to-trial changes in an animal's policy. Whereas the average contribution of the conventional REINFORCE learning rule to the policy update for mice learning the International Brain Laboratory's task was just 30%, we found that adding baseline parameters allowed the learning rule to explain 92% of the animals' policy updates under our model. Intriguingly, the best-fitting learning rates and baseline values indicate that an animal's policy update, at each trial, does not occur in the direction that maximizes expected reward. Understanding how an animal transitions from chance-level to high-accuracy performance when learning a new task not only provides neuroscientists with insight into their animals, but also provides concrete examples of biological learning algorithms to the machine learning community.