From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning

Increased Goal Tracking in Adolescent Rats Is Goal-Directed and Not Habit-Like

When a cue is paired with reward in a different location, some animals will approach the site of reward during the cue, a behavior called goal tracking, while other animals will approach and interact with the cue itself: a behavior called sign tracking. Sign tracking is thought to reflect a tendency to transfer incentive salience from the reward to the cue. Adolescence is a time of heightened sensitivity to rewards, including environmental cues that have been associated with rewards, which may account for increased impulsivity and vulnerability to drug abuse. Surprisingly, however, studies have shown that adolescents are actually less likely to interact with the cue (i.e., sign track) than adult animals. We reasoned that adolescents might show decreased sign tracking, accompanied by increased apparent goal tracking, because they tend to attribute incentive salience to a more reward-proximal "cue": the food magazine. On the other hand, adolescence is also a time of enhanced exploratory behavior, novelty-seeking, and behavioral flexibility. Therefore, adolescents might truly express more goal-directed reward-seeking and less inflexible habit-like approach to a reward-associated cue. Using a reward devaluation procedure to distinguish between these two hypotheses, we found that adolescents indeed exhibit more goal tracking, and less sign tracking, than a comparable group of adults. Moreover, adolescents' goal tracking behavior is highly sensitive to reward devaluation and therefore goal-directed and not habit-like.

Improving the Reliability of Computational Analyses: Model-Based Planning and Its Relationship With Compulsivity

BACKGROUND

Computational models show great promise in mapping latent decision-making processes onto dissociable neural substrates and clinical phenotypes. One prominent example in reinforcement learning is model-based planning, which specifically relates to transdiagnostic compulsivity. However, the reliability of computational model-derived measures such as model-based planning is unclear. Establishing reliability is necessary to ensure that such models measure stable, traitlike processes, as assumed in computational psychiatry. Although analysis approaches affect validity of reinforcement learning models and reliability of other task-based measures, their effect on reliability of reinforcement learning models of empirical data has not been systematically studied.

METHODS

We first assessed within- and across-session reliability and effects of analysis approaches (model estimation, parameterization, and data cleaning) of measures of model-based planning in patients with compulsive disorders (n = 38). The analysis approaches affecting test-retest reliability were tested in 3 large generalization samples (healthy participants: n = 541 and 111; people with a range of compulsivity: n = 1413).

RESULTS

Analysis approaches greatly influenced reliability: reliability of model-based planning measures ranged from 0 (no concordance) to above 0.9 (acceptable for clinical applications). The largest influence on reliability was whether model-estimation approaches were robust and accounted for the hierarchical structure of estimated parameters. Improvements in reliability generalized to other datasets and greatly reduced the sample size needed to find a relationship between model-based planning and compulsivity in an independent dataset.

CONCLUSIONS

These results indicate that computational psychiatry measures such as model-based planning can reliably measure latent decision-making processes, but when doing so must assess the ability of methods to estimate complex models from limited data.

Are candidate neurocognitive endophenotypes of OCD present in paediatric patients? A systematic review

To-date it has been difficult to ascertain the exact cognitive profile of childhood OCD as studies report variable results. Adult OCD research lately utilises the endophenotype approach; studying cognitive traits that are present in both patients and their unaffected first-degree relatives, and are thought to lie closer to the genotype than the full-blown disorder. By observing whether candidate endopenotypes of adult OCD are present in child patients, we can determine whether the two subtypes show cognitive overlap. We conducted a systematic review of the paediatric OCD literature focussing on proposed neurocognitive endophenotypes of OCD: cognitive flexibility, response inhibition, memory, planning, decision-making, action monitoring, and reversal learning. We found that paediatric patients present robust increases in brain error related negativity associated with abnormal action monitoring, impaired decision-making under uncertainty, planning, and visual working memory, but there is less evidence for deficits in other cognitive domains. This implies that children with OCD show some cognitive similarities with adult patients, but other dysfunctions may only manifest later in the disorder trajectory.

Sensorimotor Contingencies as a Key Drive of Development: From Babies to Robots

Much current work in robotics focuses on the development of robots capable of autonomous unsupervised learning. An essential prerequisite for such learning to be possible is that the agent should be sensitive to the link between its actions and the consequences of its actions, called sensorimotor contingencies. This sensitivity, and more particularly its role as a key drive of development, has been widely studied by developmental psychologists. However, the results of these studies may not necessarily be accessible or intelligible to roboticians. In this paper, we review the main experimental data demonstrating the role of sensitivity to sensorimotor contingencies in infants' acquisition of four fundamental motor and cognitive abilities: body knowledge, memory, generalization, and goal-directedness. We relate this data from developmental psychology to work in robotics, highlighting the links between these two domains of research. In the last part of the article we present a blueprint architecture demonstrating how exploitation of sensitivity to sensorimotor contingencies, combined with the notion of "goal," allows an agent to develop new sensorimotor skills. This architecture can be used to guide the design of specific computational models, and also to possibly envisage new empirical experiments.

Reinforcement learning across development: What insights can we draw from a decade of research?

The past decade has seen the emergence of the use of reinforcement learning models to study developmental change in value-based learning. It is unclear, however, whether these computational modeling studies, which have employed a wide variety of tasks and model variants, have reached convergent conclusions. In this review, we examine whether the tuning of model parameters that govern different aspects of learning and decision-making processes vary consistently as a function of age, and what neurocognitive developmental changes may account for differences in these parameter estimates across development. We explore whether patterns of developmental change in these estimates are better described by differences in the extent to which individuals adapt their learning processes to the statistics of different environments, or by more static learning biases that emerge across varied contexts. We focus specifically on learning rates and inverse temperature parameter estimates, and find evidence that from childhood to adulthood, individuals become better at optimally weighting recent outcomes during learning across diverse contexts and less exploratory in their value-based decision-making. We provide recommendations for how these two possibilities - and potential alternative accounts - can be tested more directly to build a cohesive body of research that yields greater insight into the development of core learning processes.

Searching for Rewards Like a Child Means Less Generalization and More Directed Exploration

How do children and adults differ in their search for rewards? We considered three different hypotheses that attribute developmental differences to (a) children's increased random sampling, (b) more directed exploration toward uncertain options, or (c) narrower generalization. Using a search task in which noisy rewards were spatially correlated on a grid, we compared the ability of 55 younger children (ages 7 and 8 years), 55 older children (ages 9-11 years), and 50 adults (ages 19-55 years) to successfully generalize about unobserved outcomes and balance the exploration-exploitation dilemma. Our results show that children explore more eagerly than adults but obtain lower rewards. We built a predictive model of search to disentangle the unique contributions of the three hypotheses of developmental differences and found robust and recoverable parameter estimates indicating that children generalize less and rely on directed exploration more than adults. We did not, however, find reliable differences in terms of random sampling.

Pediatric OCD in the era of RDoC

The NIMH Research Domain Criteria (RDoC) initiative was established with the goal of developing an alternative research classification to further research efforts in mental health. While RDoC acknowledges that constructs should be considered within a developmental framework, developmental considerations have not yet been well integrated within the existing RDoC matrix. In this paper, we consider RDoC in relation to pediatric OCD, a paradigmatic example of a neuropsychiatric disorder that often has onset in childhood but is also present across the lifespan. We discuss three RDoC subdomains with relevance to OCD as exemplars, providing for each construct a brief review of normative developmental changes, the state of construct-relevant research in pediatric OCD, and challenges and limitations related to developmental considerations within each subdomain. Finally, we conclude with a brief discussion of how RDoC may continue to evolve with regard to developmental considerations in order to further research in pediatric OCD.

Metacontrol of decision-making strategies in human aging

Humans employ different strategies when making decisions. Previous research has reported reduced reliance on model-based strategies with aging, but it remains unclear whether this is due to cognitive or motivational factors. Moreover, it is not clear how aging affects the metacontrol of decision making, that is the dynamic adaptation of decision-making strategies to varying situational demands. In this cross-sectional study, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based strategies. In contrast to previous research, model-based strategies led to higher payoffs. Moreover, we manipulated the costs and benefits of model-based strategies by varying reward magnitude and the stability of the task structure. Compared to younger adults, older adults showed reduced model-based decision making and less adaptation of decision-making strategies. Our findings suggest that aging affects the metacontrol of decision-making strategies and that reduced model-based strategies in older adults are due to limited cognitive abilities.

Credit assignment to state-independent task representations and its relationship with model-based decision making

It is widely accepted that agents learn action values based on experience in a “model-free” manner (i.e., without holding a model of the environment). Environments usually embody many features, where a subset is considered relevant for model-free outcome learning. In this study, we show that a putative model-free system assigns credit to outcome-irrelevant task representations, regardless of stimulus features. The degree of impact of these associations is strongly linked to deployment of model-based strategies. Our findings motivate a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Reinforcement Learning With Parsimonious Computation and a Forgetting Process

Decision-making is assumed to be supported by model-free and model-based systems: the model-free system is based purely on experience, while the model-based system uses a cognitive map of the environment and is more accurate. The recently developed multistep decision-making task and its computational model can dissociate the contributions of the two systems and have been used widely. This study used this task and model to understand our value-based learning process and tested alternative algorithms for the model-free and model-based learning systems. The task used in this study had a deterministic transition structure, and the degree of use of this structure in learning is estimated as the relative contribution of the model-based system to choices. We obtained data from 29 participants and fitted them with various computational models that differ in the model-free and model-based assumptions. The results of model comparison and parameter estimation showed that the participants update the value of action sequences and not each action. Additionally, the model fit was improved substantially by assuming that the learning mechanism includes a forgetting process, where the values of unselected options change to a certain default value over time. We also examined the relationships between the estimated parameters and psychopathology and other traits measured by self-reported questionnaires, and the results suggested that the difference in model assumptions can change the conclusion. In particular, inclusion of the forgetting process in the computational models had a strong impact on estimation of the weighting parameter of the model-free and model-based systems.

Annual Research Review: Developmental computational psychiatry

Most psychiatric disorders emerge during childhood and adolescence. This is also a period that coincides with the brain undergoing substantial growth and reorganisation. However, it remains unclear how a heightened vulnerability to psychiatric disorder relates to this brain maturation. Here, we propose 'developmental computational psychiatry' as a framework for linking brain maturation to cognitive development. We argue that through modelling some of the brain's fundamental cognitive computations, and relating them to brain development, we can bridge the gap between brain and cognitive development. This in turn can lead to a richer understanding of the ontogeny of psychiatric disorders. We illustrate this perspective with examples from reinforcement learning and dopamine function. Specifically, we show how computational modelling deepens an understanding of how cognitive processes, such as reward learning, effort learning, and social learning might go awry in psychiatric disorders. Finally, we sketch the promises and limitations of a developmental computational psychiatry.

Hippocampal Contributions to Model-Based Planning and Spatial Memory

Little is known about the neural mechanisms that allow humans and animals to plan actions using knowledge of task contingencies. Emerging theories hypothesize that it involves the same hippocampal mechanisms that support self-localization and memory for locations. Yet limited direct evidence supports the link between planning and the hippocampal place map. We addressed this by investigating model-based planning and place memory in healthy controls and epilepsy patients treated using unilateral anterior temporal lobectomy with hippocampal resection. Both functions were impaired in the patient group. Specifically, the planning impairment was related to right hippocampal lesion size, controlling for overall lesion size. Furthermore, although planning and boundary-driven place memory covaried in the control group, this relationship was attenuated in patients, consistent with both functions relying on the same structure in the healthy brain. These findings clarify both the neural mechanism of model-based planning and the scope of hippocampal contributions to behavior.

Developmental perspectives on risky and impulsive choice

Epidemiological data suggest that risk taking in the real world increases from childhood into adolescence and declines into adulthood. However, developmental patterns of behaviour in laboratory assays of risk taking and impulsive choice are inconsistent. In this article, we review a growing literature using behavioural economic approaches to understand developmental changes in risk taking and impulsivity. We present findings that have begun to elucidate both the cognitive and neural processes that contribute to risky and impulsive choice, as well as how age-related changes in these neurocognitive processes give rise to shifts in choice behaviour. We highlight how variability in task parameters can be used to identify specific aspects of decision contexts that may differentially influence risky and impulsive choice behaviour across development. This article is part of the theme issue 'Risk taking and impulsive behaviour: fundamental discoveries, theoretical perspectives and clinical implications'.

Neural Variability Limits Adolescent Skill Learning

Skill learning is fundamental to the acquisition of many complex behaviors that emerge during development. For example, years of practice give rise to perceptual improvements that contribute to mature speech and language skills. While fully honed learning skills might be thought to offer an advantage during the juvenile period, the ability to learn actually continues to develop through childhood and adolescence, suggesting that the neural mechanisms that support skill learning are slow to mature. To address this issue, we asked whether the rate and magnitude of perceptual learning varies as a function of age as male and female gerbils trained on an auditory task. Adolescents displayed a slower rate of perceptual learning compared with their young and mature counterparts. We recorded auditory cortical neuron activity from a subset of adolescent and adult gerbils as they underwent perceptual training. While training enhanced the sensitivity of most adult units, the sensitivity of many adolescent units remained unchanged, or even declined across training days. Therefore, the average rate of cortical improvement was significantly slower in adolescents compared with adults. Both smaller differences between sound-evoked response magnitudes and greater trial-to-trial response fluctuations contributed to the poorer sensitivity of individual adolescent neurons. Together, these findings suggest that elevated sensory neural variability limits adolescent skill learning.SIGNIFICANCE STATEMENT The ability to learn new skills emerges gradually as children age. This prolonged development, often lasting well into adolescence, suggests that children, teens, and adults may rely on distinct neural strategies to improve their sensory and motor capabilities. Here, we found that practice-based improvement on a sound detection task is slower in adolescent gerbils than in younger or older animals. Neural recordings made during training revealed that practice enhanced the sound sensitivity of adult cortical neurons, but had a weaker effect in adolescents. This latter finding was partially explained by the fact that adolescent neural responses were more variable than in adults. Our results suggest that one mechanistic basis of adult-like skill learning is a reduction in neural response variability.

Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling

A well-established notion in cognitive neuroscience proposes that multiple brain systems contribute to choice behaviour. These include: (1) a model-free system that uses values cached from the outcome history of alternative actions, and (2) a model-based system that considers action outcomes and the transition structure of the environment. The widespread use of this distinction, across a range of applications, renders it important to index their distinct influences with high reliability. Here we consider the two-stage task, widely considered as a gold standard measure for the contribution of model-based and model-free systems to human choice. We tested the internal/temporal stability of measures from this task, including those estimated via an established computational model, as well as an extended model using drift-diffusion. Drift-diffusion modeling suggested that both choice in the first stage, and RTs in the second stage, are directly affected by a model-based/free trade-off parameter. Both parameter recovery and the stability of model-based estimates were poor but improved substantially when both choice and RT were used (compared to choice only), and when more trials (than conventionally used in research practice) were included in our analysis. The findings have implications for interpretation of past and future studies based on the use of the two-stage task, as well as for characterising the contribution of model-based processes to choice behaviour.

The computational basis of following advice in adolescents

Advice taking helps one to quickly acquire knowledge and make decisions. This age-comparative study (in children [8- to 10-year-olds], adolescents [13- to 15-year-olds], and adults [18- to 22-year-olds]) investigated developmental differences in how advice, experience, and exploration influence learning. The results showed that adolescents were initially easily swayed to follow peer advice but also switched more rapidly to exploring alternatives like children. Whereas adults stayed with the advice over the task, adolescents put more weight on their own experience compared with adults. A social learning model showed that although social influence most strongly affects adolescents' initial expectations (i.e., their priors), adolescents showed higher exploration and discovered the other good option in the current task. Thus, our model resolved the apparently conflicting findings of adolescents being more and less sensitive to peer influence and provides novel insights into the dynamic interaction between social and individual learning.

Neurocognitive Development of Motivated Behavior: Dynamic Changes across Childhood and Adolescence

The ability to anticipate and respond appropriately to the challenges and opportunities present in our environments is critical for adaptive behavior. Recent methodological innovations have led to substantial advances in our understanding of the neurocircuitry supporting such motivated behavior in adulthood. However, the neural circuits and cognitive processes that enable threat- and reward-motivated behavior undergo substantive changes over the course of development, and these changes are less well understood. In this article, we highlight recent research in human and animal models demonstrating how developmental changes in prefrontal-subcortical neural circuits give rise to corresponding changes in the processing of threats and rewards from infancy to adulthood. We discuss how these developmental trajectories are altered by experiential factors, such as early-life stress, and highlight the relevance of this research for understanding the developmental onset and treatment of psychiatric disorders characterized by dysregulation of motivated behavior.

Computational Phenotyping: Using Models to Understand Individual Differences in Personality, Development, and Mental Illness

This paper reviews progress in the application of computational models to personality, developmental, and clinical neuroscience. We first describe the concept of a computational phenotype, a collection of parameters derived from computational models fit to behavioral and neural data. This approach represents individuals as points in a continuous parameter space, complementing traditional trait and symptom measures. One key advantage of this representation is that it is mechanistic: The parameters have interpretations in terms of cognitive processes, which can be translated into quantitative predictions about future behavior and brain activity. We illustrate with several examples how this approach has led to new scientific insights into individual differences, developmental trajectories, and psychopathology. We then survey some of the challenges that lay ahead.

Computational neuroscience across the lifespan: Promises and pitfalls

In recent years, the application of computational modeling in studies on age-related changes in decision making and learning has gained in popularity. One advantage of computational models is that they provide access to latent variables that cannot be directly observed from behavior. In combination with experimental manipulations, these latent variables can help to test hypotheses about age-related changes in behavioral and neurobiological measures at a level of specificity that is not achievable with descriptive analysis approaches alone. This level of specificity can in turn be beneficial to establish the identity of the corresponding behavioral and neurobiological mechanisms. In this paper, we will illustrate applications of computational methods using examples of lifespan research on risk taking, strategy selection and reinforcement learning. We will elaborate on problems that can occur when computational neuroscience methods are applied to data of different age groups. Finally, we will discuss potential targets for future applications and outline general shortcomings of computational neuroscience methods for research on human lifespan development.

Model-based learning and individual differences in depression: The moderating role of stress

Inflexible decision-making has been proposed as a transdiagnostic risk factor for mood disorders. Evidence suggests that inflexible decision-making may emerge only when individuals are experiencing increased negative affect or stress. 151 participants completed symptom measures of depression and anxiety, followed by a two-stage decision-making task that distinguishes between habitual and goal-directed choice. An experimental manipulation to induce stress was introduced halfway through the task. Individuals with higher depression levels became less model-based after the manipulation than those with lower depression levels. There was no relationship between trait anxiety and the impact of the manipulation on decision-making. Controlling for main effects of anxiety did not attenuate the association between depression and impact of stress. Anhedonia was associated with the impact of the manipulation on model-based decision-making. These results suggest that risk for depression is associated with reflexive decision-making, but these effects may only emerge under conditions of stress.

Deliberation and Procedural Automation on a Two-Step Task for Rats

Current theories suggest that decision-making arises from multiple, competing action-selection systems. Rodent studies dissociate deliberation and procedural behavior, and find a transition from procedural to deliberative behavior with experience. However, it remains unknown how this transition from deliberative to procedural control evolves within single trials, or within blocks of repeated choices. We adapted for rats a two-step task which has been used to dissociate model-based from model-free decisions in humans. We found that a mixture of model-based and model-free algorithms was more likely to explain rat choice strategies on the task than either model-based or model-free algorithms alone. This task contained two choices per trial, which provides a more complex and non-discrete per-trial choice structure. This task structure enabled us to evaluate how deliberative and procedural behavior evolved within-trial and within blocks of repeated choice sequences. We found that vicarious trial and error (VTE), a behavioral correlate of deliberation in rodents, was correlated between the two choice points on a given lap. We also found that behavioral stereotypy, a correlate of procedural automation, increased with the number of repeated choices. While VTE at the first choice point decreased [corrected] with the number of repeated choices, VTE at the second choice point did not, and only increased after unexpected transitions within the task. This suggests that deliberation at the beginning of trials may correspond to changes in choice patterns, while mid-trial deliberation may correspond to an interruption of a procedural process.

Intact striatal dopaminergic modulation of reward learning and daily-life reward-oriented behavior in first-degree relatives of individuals with psychotic disorder

BACKGROUND

Abnormalities in reward learning in psychotic disorders have been proposed to be linked to dysregulated subcortical dopaminergic (DA) neurotransmission, which in turn is a suspected mechanism for predisposition to psychosis. We therefore explored the striatal dopaminergic modulation of reward processing and its behavioral correlates in individuals at familial risk for psychosis.

METHODS

We performed a DA D2/3 receptor [18F]fallypride positron emission tomography scan during a probabilistic reinforcement learning task in 16 healthy first-degree relatives of patients with psychosis and 16 healthy volunteers, followed by a 6-day ecological momentary assessment study capturing reward-oriented behavior in the everyday life.

RESULTS

We detected significant reward-induced DA release in bilateral caudate, putamen and ventral striatum of both groups, with no group differences in its magnitude nor spatial extent. In both groups alike, greater extent of reward-induced DA release in all regions of interest was associated with better performance in the task, as well as in greater tendency to be engaged in reward-oriented behavior in the daily life.

CONCLUSIONS

These findings suggest intact striatal dopaminergic modulation of reinforcement learning and reward-oriented behavior in individuals with familial predisposition to psychosis. Furthermore, this study points towards a key link between striatal reward-related DA release and pursuit of ecologically relevant rewards.

Developmental differences in the neural dynamics of observational learning

Learning from vicarious experience is central for educational practice, but not well understood with respect to its ontogenetic development and underlying neural dynamics. In this age-comparative study we compared behavioral and electrophysiological markers of learning from vicarious and one's own experience in children (age 8-10) and young adults. Behaviorally both groups benefitted from integrating vicarious experience into their own choices however, adults learned much faster from social information than children. The electrophysiological results show learning-related changes in the P300 to experienced and observed rewards in adults, but not in children, indicating that adults were more efficient in integrating observed and experienced information during learning. In comparison to adults, children showed an enhanced FRN for observed and experienced feedback, indicating that they focus more on valence information than adults. Taken together, children compared to adults seem to be less able to rapidly assess the informational value of observed and experienced feedback during learning and consequently up-regulate their response to both, observed and experienced (particularly negative) feedback. When transferring the current findings to an applied context, educational practice should strengthen children's ability to use feedback information for learning and be particularly cautious with negative social feedback during supervised learning.

Adolescent Development of Value-Guided Goal Pursuit

Adolescents are challenged to orchestrate goal-directed actions in increasingly independent and consequential ways. In doing so, it is advantageous to use information about value to select which goals to pursue and how much effort to devote to them. Here, we examine age-related changes in how individuals use value signals to orchestrate goal-directed behavior. Drawing on emerging literature on value-guided cognitive control and reinforcement learning, we demonstrate how value and task difficulty modulate the execution of goal-directed action in complex ways across development from childhood to adulthood. We propose that the scope of value-guided goal pursuit expands with age to include increasingly challenging cognitive demands, and scaffolds on the emergence of functional integration within brain networks supporting valuation, cognition, and action.

Surviving threats: neural circuit and computational implications of a new taxonomy of defensive behaviour

Research on defensive behaviour in mammals has in recent years focused on elicited reactions; however, organisms also make active choices when responding to danger. We propose a hierarchical taxonomy of defensive behaviour on the basis of known psychological processes. Included are three categories of reactions (reflexes, fixed reactions and habits) and three categories of goal-directed actions (direct action-outcome behaviours and actions based on implicit or explicit forecasting of outcomes). We then use this taxonomy to guide a summary of findings regarding the underlying neural circuits.

The detour problem in a stochastic environment: Tolman revisited

We designed a grid world task to study human planning and re-planning behavior in an unknown stochastic environment. In our grid world, participants were asked to travel from a random starting point to a random goal position while maximizing their reward. Because they were not familiar with the environment, they needed to learn its characteristics from experience to plan optimally. Later in the task, we randomly blocked the optimal path to investigate whether and how people adjust their original plans to find a detour. To this end, we developed and compared 12 different models. These models were different on how they learned and represented the environment and how they planned to catch the goal. The majority of our participants were able to plan optimally. We also showed that people were capable of revising their plans when an unexpected event occurred. The result from the model comparison showed that the model-based reinforcement learning approach provided the best account for the data and outperformed heuristics in explaining the behavioral data in the re-planning trials.

Developmental Changes in Learning: Computational Mechanisms and Social Influences

Our ability to learn from the outcomes of our actions and to adapt our decisions accordingly changes over the course of the human lifespan. In recent years, there has been an increasing interest in using computational models to understand developmental changes in learning and decision-making. Moreover, extensions of these models are currently applied to study socio-emotional influences on learning in different age groups, a topic that is of great relevance for applications in education and health psychology. In this article, we aim to provide an introduction to basic ideas underlying computational models of reinforcement learning and focus on parameters and model variants that might be of interest to developmental scientists. We then highlight recent attempts to use reinforcement learning models to study the influence of social information on learning across development. The aim of this review is to illustrate how computational models can be applied in developmental science, what they can add to our understanding of developmental mechanisms and how they can be used to bridge the gap between psychological and neurobiological theories of development.

Model-Based Control in Dimensional Psychiatry

We use parallel interacting goal-directed and habitual strategies to make our daily decisions. The arbitration between these strategies is relevant to inflexible repetitive behaviors in psychiatric disorders. Goal-directed control, also known as model-based control, is based on an affective outcome relying on a learned internal model to prospectively make decisions. In contrast, habit control, also known as model-free control, is based on an integration of previous reinforced learning autonomous of the current outcome value and is implicit and more efficient but at the cost of greater inflexibility. The concept of model-based control can be further extended into pavlovian processes. Here we describe and compare tasks that tap into these constructs and emphasize the clinical relevance and translation of these tasks in psychiatric disorders. Together, these findings highlight a role for model-based control as a transdiagnostic impairment underlying compulsive behaviors and representing a promising therapeutic target.

Medial prefrontal cortical thinning mediates shifts in other-regarding preferences during adolescence

Adolescence is a time of significant cortical changes in the ‘social brain’, a set of brain regions involved in sophisticated social inference. However, there is limited evidence linking the structural changes in social brain to development of social behavior. The present study investigated how cortical development of the social brain relates to other-regarding behavior, in the context of fairness concerns. Participants aged between 9 to 23 years old responded to multiple rounds of ultimatum game proposals. The degree to which each participant considers fairness of intention (i.e., intention-based reciprocity) vs. outcome (i.e., egalitarianism) was quantified using economic utility models. We observed a gradual shift in other-regarding preferences from simple rule-based egalitarianism to complex intention-based reciprocity from early childhood to young adulthood. The preference shift was associated with cortical thinning of the dorsomedial prefrontal cortex and posterior temporal cortex. Meta-analytic reverse-inference analysis showed that these regions were involved in social inference. Importantly, the other-regarding preference shift was statistically mediated by cortical thinning in the dorsomedial prefrontal cortex. Together these findings suggest that development of the ability to perform sophisticated other-regarding social inference is associated with the structural changes of specific social brain regions in late adolescence.

Making better decisions in groups

We review the literature to identify common problems of decision-making in individuals and groups. We are guided by a Bayesian framework to explain the interplay between past experience and new evidence, and the problem of exploring the space of hypotheses about all the possible states that the world could be in and all the possible actions that one could take. There are strong biases, hidden from awareness, that enter into these psychological processes. While biases increase the efficiency of information processing, they often do not lead to the most appropriate action. We highlight the advantages of group decision-making in overcoming biases and searching the hypothesis space for good models of the world and good solutions to problems. Diversity of group members can facilitate these achievements, but diverse groups also face their own problems. We discuss means of managing these pitfalls and make some recommendations on how to make better group decisions.

Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems

Human behavior is sometimes determined by habit and other times by goal-directed planning. Modern reinforcement-learning theories formalize this distinction as a competition between a computationally cheap but inaccurate model-free system that gives rise to habits and a computationally expensive but accurate model-based system that implements planning. It is unclear, however, how people choose to allocate control between these systems. Here, we propose that arbitration occurs by comparing each system’s task-specific costs and benefits. To investigate this proposal, we conducted two experiments showing that people increase model-based control when it achieves greater accuracy than model-free control, and especially when the rewards of accurate performance are amplified. In contrast, they are insensitive to reward amplification when model-based and model-free control yield equivalent accuracy. This suggests that humans adaptively balance habitual and planned action through on-line cost-benefit analysis.

How Accumulated Real Life Stress Experience and Cognitive Speed Interact on Decision-Making Processes

Rationale: Advances in neurocomputational modeling suggest that valuation systems for goal-directed (deliberative) on one side, and habitual (automatic) decision-making on the other side may rely on distinct computational strategies for reinforcement learning, namely model-free vs. model-based learning. As a key theoretical difference, the model-based system strongly demands cognitive functions to plan actions prospectively based on an internal cognitive model of the environment, whereas valuation in the model-free system relies on rather simple learning rules from operant conditioning to retrospectively associate actions with their outcomes and is thus cognitively less demanding. Acute stress reactivity is known to impair model-based but not model-free choice behavior, with higher working memory capacity protecting the model-based system from acute stress. However, it is not clear which impact accumulated real life stress has on model-free and model-based decision systems and how this influence interacts with cognitive abilities. Methods: We used a sequential decision-making task distinguishing relative contributions of both learning strategies to choice behavior, the Social Readjustment Rating Scale questionnaire to assess accumulated real life stress, and the Digit Symbol Substitution Test to test cognitive speed in 95 healthy subjects. Results: Individuals reporting high stress exposure who had low cognitive speed showed reduced model-based but increased model-free behavioral control. In contrast, subjects exposed to accumulated real life stress with high cognitive speed displayed increased model-based performance but reduced model-free control. Conclusion: These findings suggest that accumulated real life stress exposure can enhance reliance on cognitive speed for model-based computations, which may ultimately protect the model-based system from the detrimental influences of accumulated real life stress. The combination of accumulated real life stress exposure and slower information processing capacities, however, might favor model-free strategies. Thus, the valence and preference of either system strongly depends on stressful experiences and individual cognitive capacities.

Cognitive components underpinning the development of model-based learning

Reinforcement learning theory distinguishes “model-free” learning, which fosters reflexive repetition of previously rewarded actions, from “model-based” learning, which recruits a mental model of the environment to flexibly select goal-directed actions. Whereas model-free learning is evident across development, recruitment of model-based learning appears to increase with age. However, the cognitive processes underlying the development of model-based learning remain poorly characterized. Here, we examined whether age-related differences in cognitive processes underlying the construction and flexible recruitment of mental models predict developmental increases in model-based choice. In a cohort of participants aged 9–25, we examined whether the abilities to infer sequential regularities in the environment (“statistical learning”), maintain information in an active state (“working memory”) and integrate distant concepts to solve problems (“fluid reasoning”) predicted age-related improvements in model-based choice. We found that age-related improvements in statistical learning performance did not mediate the relationship between age and model-based choice. Ceiling performance on our working memory assay prevented examination of its contribution to model-based learning. However, age-related improvements in fluid reasoning statistically mediated the developmental increase in the recruitment of a model-based strategy. These findings suggest that gradual development of fluid reasoning may be a critical component process underlying the emergence of model-based learning.

Reversal learning strategy in adolescence is associated with prefrontal cortex activation

Adolescence is a critical maturation period for human cognitive control and executive function. In this study, a large sample of adolescents (n = 85) performed a reversal learning task during functional magnetic resonance imaging. We analyzed behavioral data using a reinforcement learning model to provide individually fitted parameters and imaging data with regard to reward prediction errors (PE). Following a model-based approach, we formed two groups depending on whether individuals tended to update expectations predominantly for the chosen stimulus or also for the unchosen one. These groups significantly differed in their problem behavior score obtained using the child behavior checklist (CBCL) and in a measure of their developmental stage. Imaging results showed that dorsolateral striatal areas covaried with PE. Participants who relied less on learning based on task structure showed less prefrontal activation compared with participants who relied more on task structure. An exploratory analysis revealed that PE-related activity was associated with pubertal development in prefrontal areas, insula and anterior cingulate. These findings support the hypothesis that the prefrontal cortex is implicated in mediating flexible goal-directed behavioral control.

When Does Model-Based Control Pay Off?

Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to “model-free” and “model-based” strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.

When you make a choice about what groceries to get for dinner, you can rely on two different strategies. You can make your choice by relying on habit, simply buying the items you need to make a meal that is second nature to you. However, you can also plan your actions in a more deliberative way, realizing that the friend who will join you is a vegetarian, and therefore you should not make the burgers that have become a staple in your cooking. These two strategies differ in how computationally demanding and accurate they are. While the habitual strategy is less computationally demanding (costs less effort and time), the deliberative strategy is more accurate. Scientists have been able to study the distinction between these strategies using a task that allows them to measure how much people rely on habit and planning strategies. Interestingly, we have discovered that in this task, the deliberative strategy does not increase performance accuracy, and hence does not induce a trade-off between accuracy and demand. We describe why this happens, and improve the task so that it embodies an accuracy-demand trade-off, providing evidence for theories of cost-based arbitration between cognitive strategies.

The role of experience in adolescent cognitive development: Integration of executive, memory, and mesolimbic systems

Adolescence marks a time of unique neurocognitive development, in which executive functions reach adult levels of maturation. While many core facets of executive function may reach maturation in childhood, these processes continue to be refined and stabilized during adolescence. We propose that this is mediated, in part, by interactions between the hippocampus and prefrontal cortex. Specifically, we propose that development of this circuit refines adolescents' ability to extract relevant information from prior experience to support task-relevant behavior. In support of this model, we review evidence for protracted structural and functional development both within and across the hippocampus and prefrontal cortex. We describe emerging research demonstrating the refinement of adolescents' ability to integrate prior experiences to support goal-oriented behavior, which parallel hippocampal-prefrontal integration. Finally, we speculate that the development of this circuit is mediated by increases in dopaminergic neuromodulation present in adolescence, which may underlie memory processing, plasticity, and circuit integration. This model provides a novel characterization of how memory and executive systems integrate throughout adolescence to support adaptive behavior.