From Creatures of Habit to Goal-Directed Learners: Tracking the Developmental Emergence of Model-Based Reinforcement Learning

Balancing exploration and exploitation? The impact of cost and inhibitory control on information gathering in early childhood

The information-seeking behaviour of adults focuses on optimizing the gathering and utilizing information to minimize search costs. In contrast, children tend to engage in information search during decision-making with less consideration for costs. This difference in behaviour is believed to be linked to the development of executive functions. Therefore, the present study aimed to investigate the relationship between executive function and cost-related information-gathering behaviour. We assessed 56 children aged 4-6 years, involving three tasks: an information-gathering task, an inhibitory control and a working memory task. In the information-gathering task, children participated in both non-cost and cost conditions, where they were given the opportunity to freely gather information or incur a cost to acquire information. The findings revealed that children with higher inhibitory control tended to gather less information when a cost was involved. This highlights the important role of inhibitory control in shaping information-seeking behaviour in early childhood.

Disentangling negative reinforcement, working memory, and deductive reasoning deficits in elevated BMI

Neuropsychological data suggest that being overweight or obese is associated with a tendency to perseverate behavior despite negative feedback. This deficit might be observed due to other cognitive factors, such as working memory (WM) deficits or decreased ability to deduce model-based strategies when learning by trial-and-error. In the present study, a group of subjects with overweight or obesity (Ow/Ob, n = 30) was compared to normal-weight individuals (n = 42) in a modified Reinforcement Learning (RL) task. The task was designed to control WM effects on learning by manipulating cognitive load and to foster model-based learning via deductive reasoning. Computational modelling and analysis were conducted to isolate parameters related to RL mechanisms, WM use, and model-based learning (deduction parameter). Results showed that subjects with Ow/Ob had a higher number of perseverative errors and used a weaker deduction mechanism in their performance than control individuals, indicating impairments in negative reinforcement and model-based learning, whereas WM impairments were not responsible for deficits in RL. The present data suggests that obesity is associated with impairments in negative reinforcement and model-based learning.

Surprising sounds influence risky decision making

Adaptive behavior depends on appropriate responses to environmental uncertainty. Incidental sensory events might simply be distracting and increase errors, but alternatively can lead to stereotyped responses despite their irrelevance. To evaluate these possibilities, we test whether task-irrelevant sensory prediction errors influence risky decision making in humans across seven experiments (total n = 1600). Rare auditory sequences preceding option presentation systematically increase risk taking and decrease choice perseveration (i.e., increased tendency to switch away from previously chosen options). The risk-taking and perseveration effects are dissociable by manipulating auditory statistics: when rare sequences end on standard tones, including when rare sequences consist only of standard tones, participants are less likely to perseverate after rare sequences but not more likely to take risks. Computational modeling reveals that these effects cannot be explained by increased decision noise but can be explained by value-independent risky bias and perseveration parameters, decision biases previously linked to dopamine. Control experiments demonstrate that both surprise effects can be eliminated when tone sequences are presented in a balanced or fully predictable manner, and that surprise effects cannot be explained by erroneous beliefs. These findings suggest that incidental sounds may influence many of the decisions we make in daily life.

The role of intrinsic reward in adolescent word learning

Relatively little work has focused on why we are motivated to learn words. In adults, recent experiments have shown that intrinsic reward signals accompany successful word learning from context. In addition, the experience of reward facilitated long-term memory for words. In adolescence, developmental changes are seen in reward and motivation systems as well as in reading and language systems. Here, in the face of this developmental change, we ask whether adolescents experience reward from word learning, and how the reward and memory benefit seen in adults is modulated by age. We used a naturalistic reading paradigm, which involved extracting novel word meanings from sentence context without the need for explicit feedback. By exploring ratings of enjoyment during the learning phase, as well as recognition memory for words a day later, we assessed whether adolescents show the same reward and learning patterns as adults. We tested 345 children between the ages of 10-18 (N > 84 in each 2-year age-band) using this paradigm. We found evidence for our first prediction: children aged 10-18 report greater enjoyment for successful word learning. However, we did not find evidence for age-related change in this developmental period, or memory benefits. This work gives us greater insight into the process of language acquisition and sets the stage for further investigations of intrinsic reward in typical and atypical development. RESEARCH HIGHLIGHTS: We constantly learn words from context, even in the absence of explicit rewards or feedback. In adults, intrinsic reward experienced during word learning is linked to a dopaminergic circuit in the brain, which also fuels enhancements in memory for words. We find adolescents also report enhanced reward or enjoyment when they successfully learn words from sentence context. The relationship between reward and learning is maintained between the ages of 10 and 18. Unlike in adults, we did not observe ensuing memory benefits.

Personalized Cognitive Health in Psychiatry: Current State and the Promise of Computational Methods

BACKGROUND

Decades of research have firmly established that cognitive health and cognitive treatment services are a key need for people living with psychosis. However, many current clinical programs do not address this need, despite the essential role that an individual's cognitive and social cognitive capacities play in determining their real-world functioning. Preliminary practice-based research in the Early Psychosis Intervention Network early psychosis intervention network shows that it is possible to develop and implement tools that delineate an individuals' cognitive health profile and that help engage the client and the clinician in shared decision-making and treatment planning that includes cognitive treatments. These findings signify a promising shift toward personalized cognitive health.

STUDY DESIGN

Extending upon this early progress, we review the concept of interindividual variability in cognitive domains/processes in psychosis as the basis for offering personalized treatment plans. We present evidence from studies that have used traditional neuropsychological measures as well as findings from emerging computational studies that leverage trial-by-trial behavior data to illuminate the different latent strategies that individuals employ.

STUDY RESULT

We posit that these computational techniques, when combined with traditional cognitive assessments, can enrich our understanding of individual differences in treatment needs, which in turn can guide evermore personalized interventions.

CONCLUSION

As we find clinically relevant ways to decompose maladaptive behaviors into separate latent cognitive elements captured by model parameters, the ultimate goal is to develop and implement approaches that empower clients and their clinical providers to leverage individual's existing learning capacities to improve their cognitive health and well-being.

Sensitivity to the Instrumental Value of Choice Increases Across Development

Across development, people tend to demonstrate a preference for contexts in which they have the opportunity to make choices. However, it is not clear how children, adolescents, and adults learn to calibrate this preference based on the costs and benefits of agentic choice. Here, in both a primary, in-person, reinforcement-learning experiment (N = 92; age range = 10-25 years) and a preregistered online replication study (N = 150; age range = 8-25 years), we found that participants overvalued agentic choice but also calibrated their agency decisions to the reward structure of the environment, increasingly selecting agentic choice when choice had greater instrumental value. Regression analyses and computational modeling of participant choices revealed that participants' bias toward agentic choice-reflecting its intrinsic value-remained consistent across age, whereas sensitivity to the instrumental value of agentic choice increased from childhood to early adulthood.

Learning and memory processes in behavioural addiction: A systematic review

Similar to addictive substances, addictive behaviours such as gambling and gaming are associated with maladaptive modulation of key brain areas and functional networks implicated in learning and memory. Therefore, this review sought to understand how different learning and memory processes relate to behavioural addictions and to unravel their underlying neural mechanisms. Adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we systematically searched four databases - PsycINFO, PubMed, Scopus, and Web of Science using the agreed-upon search string. Findings suggest altered executive function-dependent learning processes and enhanced habit learning in behavioural addiction. Whereas the relationship between working memory and behavioural addiction is influenced by addiction type, working memory aspect, and task nature. Additionally, long-term memory is incoherent in individuals with addictive behaviours. Consistently, neurophysiological evidence indicates alterations in brain areas and networks implicated in learning and memory processes in behavioural addictions. Overall, the present review argues that, like substance use disorders, alteration in learning and memory processes may underlie the development and maintenance of behavioural addictions.

Compulsive avoidance in youths and adults with OCD: an aversive pavlovian-to-instrumental transfer study

Compulsive behaviour may often be triggered by Pavlovian cues. Assessing how Pavlovian cues drive instrumental behaviour in obsessive-compulsive disorder (OCD) is therefore crucial to understand how compulsions develop and are maintained. An aversive Pavlovian-to-Instrumental transfer (PIT) paradigm, particularly one involving avoidance/cancellation of negative outcomes, can enable such investigation and has not previously been studied in clinical-OCD. Forty-one participants diagnosed with OCD (21 adults; 20 youths) and 44 controls (21 adults; 23 youths) completed an aversive PIT task. Participants had to prevent the delivery of unpleasant noises by moving a joystick in the correct direction. They could infer these correct responses by learning appropriate response-outcome (instrumental) and stimulus-outcome (Pavlovian) associations. We then assessed whether Pavlovian cues elicited specific instrumental avoidance responses (specific PIT) and induced general instrumental avoidance (general PIT). We investigated whether task learning and confidence indices influenced PIT strength differentially between groups. There was no overall group difference in PIT performance, although youths with OCD showed weaker specific PIT than youth controls. However, urge to avoid unpleasant noises and preference for safe over unsafe stimuli influenced specific and general PIT respectively in OCD, while PIT in controls was more influenced by confidence in instrumental and Pavlovian learning. Thus, in OCD, implicit motivational factors, but not learnt knowledge, may contribute to the successful integration of aversive Pavlovian and instrumental cues. This implies that compulsive avoidance may be driven by these automatic processes. Youths with OCD show deficits in specific PIT, suggesting cue integration impairments are only apparent in adolescence. These findings may be clinically relevant as they emphasise the importance of targeting such implicit motivational processes when treating OCD.

The future of neuropsychology is digital, theory-driven, and Bayesian: a paradigmatic study of cognitive flexibility

Introduction

This study explores the transformative potential of digital, theory-driven, and Bayesian paradigms in neuropsychology by combining digital technologies, a commitment to evaluating theoretical frameworks, and Bayesian statistics. The study also examines theories of executive function and cognitive flexibility in a large sample of neurotypical individuals (N = 489).

Methods

We developed an internet-based Wisconsin Card-Sorting Task (iWCST) optimized for online assessment of perseveration errors (PE). Predictions of the percentage of PE, PE (%), in non-repetitive versus repetitive situations were derived from the established supervisory attention system (SAS) theory, non-repetitive PE (%) < repetitive PE (%), and the novel goal-directed instrumental control (GIC) theory, non-repetitive PE (%) > repetitive PE (%).

Results

Bayesian t-tests revealed the presence of a robust error suppression effect (ESE) indicating that PE are less likely in repetitive situations than in non-repetitive situations, contradicting SAS theory with posterior model probability p < 0.001 and confirming GIC theory with posterior model probability p > 0.999. We conclude that repetitive situations support cognitive set switching in the iWCST by facilitating the retrieval of goal-directed, instrumental memory that associates stimulus features, actions, and outcomes, thereby generating the ESE in neurotypical individuals. We also report exploratory data analyses, including a Bayesian network analysis of relationships between iWCST measures.

Discussion

Overall, this study serves as a paradigmatic model for combining digital technologies, theory-driven research, and Bayesian statistics in neuropsychology. It also provides insight into how this integrative, innovative approach can advance the understanding of executive function and cognitive flexibility and inform future research and clinical applications.

Evidence for a competitive relationship between executive functions and statistical learning

The ability of the brain to extract patterns from the environment and predict future events, known as statistical learning, has been proposed to interact in a competitive manner with prefrontal lobe-related networks and their characteristic cognitive or executive functions. However, it remains unclear whether these cognitive functions also possess a competitive relationship with implicit statistical learning across individuals and at the level of latent executive function components. In order to address this currently unknown aspect, we investigated, in two independent experiments (NStudy1 = 186, NStudy2 = 157), the relationship between implicit statistical learning, measured by the Alternating Serial Reaction Time task, and executive functions, measured by multiple neuropsychological tests. In both studies, a modest, but consistent negative correlation between implicit statistical learning and most executive function measures was observed. Factor analysis further revealed that a factor representing verbal fluency and complex working memory seemed to drive these negative correlations. Thus, the antagonistic relationship between implicit statistical learning and executive functions might specifically be mediated by the updating component of executive functions or/and long-term memory access.

Bayesian Reinforcement Learning With Limited Cognitive Load

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Adolescents flexibly adapt action selection based on controllability inferences

From early in life, we encounter both controllable environments, in which our actions can causally influence the reward outcomes we experience, and uncontrollable environments, in which they cannot. Environmental controllability is theoretically proposed to organize our behavior. In controllable contexts, we can learn to proactively select instrumental actions that bring about desired outcomes. In uncontrollable environments, Pavlovian learning enables hard-wired, reflexive reactions to anticipated, motivationally salient events, providing "default" behavioral responses. Previous studies characterizing the balance between Pavlovian and instrumental learning systems across development have yielded divergent findings, with some studies observing heightened expression of Pavlovian learning during adolescence and others observing a reduced influence of Pavlovian learning during this developmental stage. In this study, we aimed to investigate whether a theoretical model of controllability-dependent arbitration between learning systems might explain these seemingly divergent findings in the developmental literature, with the specific hypothesis that adolescents' action selection might be particularly sensitive to environmental controllability. To test this hypothesis, 90 participants, aged 8-27, performed a probabilistic-learning task that enables estimation of Pavlovian influence on instrumental learning, across both controllable and uncontrollable conditions. We fit participants' data with a reinforcement-learning model in which controllability inferences adaptively modulate the dominance of Pavlovian versus instrumental control. Relative to children and adults, adolescents exhibited greater flexibility in calibrating the expression of Pavlovian bias to the degree of environmental controllability. These findings suggest that sensitivity to environmental reward statistics that organize motivated behavior may be heightened during adolescence.

Goal-directed learning in adolescence: neurocognitive development and contextual influences

Adolescence is a time during which we transition to independence, explore new activities and begin pursuit of major life goals. Goal-directed learning, in which we learn to perform actions that enable us to obtain desired outcomes, is central to many of these processes. Currently, our understanding of goal-directed learning in adolescence is itself in a state of transition, with the scientific community grappling with inconsistent results. When we examine metrics of goal-directed learning through the second decade of life, we find that many studies agree there are steady gains in performance in the teenage years, but others report that adolescent goal-directed learning is already adult-like, and some find adolescents can outperform adults. To explain the current variability in results, sophisticated experimental designs are being applied to test learning in different contexts. There is also increasing recognition that individuals of different ages and in different states will draw on different neurocognitive systems to support goal-directed learning. Through adoption of more nuanced approaches, we can be better prepared to recognize and harness adolescent strengths and to decipher the purpose (or goals) of adolescence itself.

Characterizing Human Habits in the Lab

Habits pose a fundamental puzzle for those aiming to understand human behavior. They pervade our everyday lives and dominate some forms of psychopathology but are extremely hard to elicit in the lab. In this Registered Report, we developed novel experimental paradigms grounded in computational models, which suggest that habit strength should be proportional to the frequency of behavior and, in contrast to previous research, independent of value. Specifically, we manipulated how often participants performed responses in two tasks varying action repetition without, or separately from, variations in value. Moreover, we asked how this frequency-based habitization related to value-based operationalizations of habit and self-reported propensities for habitual behavior in real life. We find that choice frequency during training increases habit strength at test and that this form of habit shows little relation to value-based operationalizations of habit. Our findings empirically ground a novel perspective on the constituents of habits and suggest that habits may arise in the absence of external reinforcement. We further find no evidence for an overlap between different experimental approaches to measuring habits and no associations with self-reported real-life habits. Thus, our findings call for a rigorous reassessment of our understanding and measurement of human habitual behavior in the lab.

Greater reliance on model-free learning in adolescent anorexia nervosa: An examination of dual-system reinforcement learning

Alterations in learning and decision-making systems are thought to contribute to core features of anorexia nervosa (AN), a psychiatric disorder characterized by persistent dietary restriction and weight loss. Instrumental learning theory identifies a dual-system of habit and goal-directed decision-making, linked to model-free and model-based reinforcement learning algorithms. Difficulty arbitrating between these systems, resulting in an over-reliance on one strategy over the other, has been implicated in compulsivity and extreme goal pursuit, both of which are observed in AN. Characterizing alterations in model-free and model-based systems, and their neural correlates, in AN may clarify mechanisms contributing to symptom heterogeneity (e.g., binge/purge symptoms). This study tested whether adolescents with restricting AN (AN-R; n = 36) and binge/purge AN (AN-BP; n = 20) differentially utilized model-based and model-free learning systems compared to a healthy control group (HC; n = 28) during a Markov two-step decision-making task under conditions of reward and punishment. Associations between model-free and model-based learning and resting-state functional connectivity between neural regions of interest, including orbitofrontal cortex (OFC), nucleus accumbens (NAcc), putamen, and sensory motor cortex (SMC) were examined. AN-R showed higher utilization of model-free learning compared to HC for reward, but attenuated model-free and model-based learning for punishment. In AN-R only, higher model-based learning was associated with stronger OFC-to-left NAcc functional connectivity, regions linked to goal-directed behavior. Greater utilization of model-free learning for reward in AN-R may differentiate this group, particularly during adolescence, and facilitate dietary restriction by prioritizing habitual control in rewarding contexts.

How are overweight and obesity associated with reinforcement learning deficits? A systematic review

Reinforcement learning (RL) refers to the ability to learn stimulus-response or response-outcome associations relevant to the acquisition of behavioral repertoire and adaptation to the environment. Research data from correlational and case-control studies have shown that obesity is associated with impairments in RL. The aim of the present study was to systematically review how obesity and overweight are associated with RL performance. More specifically, the relationship between high body mass index (BMI) and task performance was explored through the analysis of specific RL processes associated with different physiological, computational, and behavioral manifestations. Our systematic analyses indicate that obesity might be associated with impairments in the use of aversive outcomes to change ongoing behavior, as revealed by results involving instrumental negative reinforcement and extinction/reversal learning, but further research needs to be conducted to confirm this association. Hypotheses regarding how obesity might be associated with altered RL were discussed.

Model-based and model-free mechanisms in methamphetamine use disorder

People with methamphetamine use disorder (MUD) struggle to shift their behaviour from methamphetamine-orientated habits to goal-oriented choices. The model-based/model-free framework is well suited to understand this difficulty by unpacking the computational mechanisms that support experienced-based (model-free) and goal-directed (model-based) choices. We aimed to examine whether 1) participants with MUD differed from controls on behavioural proxies and/or computational mechanisms of model-based/model-free choices; 2) model-based/model-free decision-making correlated with MUD symptoms; and 3) model-based/model-free deficits improved over six weeks in the group with MUD. Participants with MUD and controls with similar age, IQ and socioeconomic status completed the Two-Step Task at treatment commencement (MUD n = 30, Controls n = 31) and six weeks later (MUD n = 23, Controls n = 26). We examined behavioural proxies of model-based/model-free decisions using mixed logistic regression, and their underlying mechanisms using computational modelling. At a behavioural level, participants with MUD were more likely to switch their choices following rewarded actions, although this pattern improved at follow up. At a computational level, groups were similar in their use of model-based mechanisms, but participants with MUD were less likely to apply model-free mechanisms and less likely to repeat rewarded actions. We did not find evidence that individual differences in model-based or model-free parameters were associated with greater severity of methamphetamine dependence, nor did we find that group differences in computational parameters changed between baseline and follow-up assessment. Decision-making challenges in people with MUD are likely related to difficulties in pursuing choices previously associated with positive outcomes.

A feasibility study of goal-directed network-based real-time fMRI neurofeedback for anhedonic depression

Anhedonia is a hallmark symptom of depression that often lacks adequate interventions. The translational gap remains in clinical treatments based on neural substrates of anhedonia. Our pilot study found that depressed individuals depended less on goal-directed (GD) reward learning (RL), with reduced reward prediction error (RPE) BOLD signal. Previous studies have found that anhedonia is related to abnormal activities and/or functional connectivities of the central executive network (CEN) and salience network (SN), both of which belong to the goal-directed system. In addition, it was found that real-time functional magnetic resonance imaging (rt-fMRI) neurofeedback (NF) could improve the balance between CEN and SN in healthy individuals. Therefore, we speculate that rt-fMRI NF of the CEN and SN associated with the GD system may improve depressive and/or anhedonic symptoms. Therefore, this study (1) will examine individuals with anhedonic depression using GD-RL behavioral task, combined with functional magnetic resonance imaging and computational modeling to explore the role of CEN/SN deficits in anhedonic depression; and (2) will utilize network-based rt-fMRI NF to investigate whether it is feasible to regulate the differential signals of brain CEN/SN of GD system through rt-fMRI NF to alleviate depressive and/or anhedonic symptoms. This study highlights the need to elucidate the intervention effects of rt-fMRI NF and the underlying computational network neural mechanisms.

Model-free decision making resists improved instructions and is enhanced by stimulus-response associations

Human behaviour may be thought of as supported by two different computational-learning mechanisms, model-free and model-based respectively. In model-free strategies, stimulus-response associations are strengthened when actions are followed by a reward and weakened otherwise. In model-based learning, previous to selecting an action, the current values of the different possible actions are computed based on a detailed model of the environment. Previous research with the two-stage task suggests that participants' behaviour usually shows a mixture of both strategies. But, interestingly, a recent study by da Silva and Hare (2020) found that participants primarily deploy model-based behaviour when they are given detailed instructions about the structure of the task. In the present study, we reproduce this essential experiment. Our results confirm that improved instructions give rise to a stronger model-based component. Crucially, we also found a significant effect of reward that became stronger under conditions that favoured the development of strong stimulus-response associations. This suggests that the effect of reward, often taken as indicator of a model-free component, is related to stimulus-response learning.

The Influence of Memory on Visual Perception in Infants, Children, and Adults

Perception is not an independent, in-the-moment event. Instead, perceiving involves integrating prior expectations with current observations. How does this ability develop from infancy through adulthood? We examined how prior visual experience shapes visual perception in infants, children, and adults. Using an identical task across age groups, we exposed participants to pairs of colorful stimuli and implicitly measured their ability to discriminate relative saturation levels. Results showed that adult participants were biased by previously experienced exemplars, and exhibited weakened in-the-moment discrimination between different levels of saturation. In contrast, infants and children showed less influence of memory in their perception, and they actually outperformed adults in discriminating between current levels of saturation. Our findings suggest that as humans develop, their perception relies more on prior experience and less on current observation.

Using smartphones to optimise and scale-up the assessment of model-based planning

Model-based planning is thought to protect against over-reliance on habits. It is reduced in individuals high in compulsivity, but effect sizes are small and may depend on subtle features of the tasks used to assess it. We developed a diamond-shooting smartphone game that measures model-based planning in an at-home setting, and varied the game's structure within and across participants to assess how it affects measurement reliability and validity with respect to previously established correlates of model-based planning, with a focus on compulsivity. Increasing the number of trials used to estimate model-based planning did remarkably little to affect the association with compulsivity, because the greatest signal was in earlier trials. Associations with compulsivity were higher when transition ratios were less deterministic and depending on the reward drift utilised. These findings suggest that model-based planning can be measured at home via an app, can be estimated in relatively few trials using certain design features, and can be optimised for sensitivity to compulsive symptoms in the general population.

Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

The influence of aerobic exercise on model-based decision making in women with posttraumatic stress disorder

Individuals with PTSD often exhibit deficits in executive functioning. An unexplored aspect of neurocognitive functions associated with PTSD is the type of learning system engaged in during decision-making. A model-free (MF) system is habitual in nature and involves trial-and-error learning that is often updated based on the most recent experience (e.g., repeat action if rewarded). A model-based (MB) system is goal-directed in nature and involves the development of an abstract representation of the environment to facilitate decisions (e.g., choose sequence of actions according to current contextual state and predicted outcomes). The existing neurocognitive literature on PTSD suggests the hypothesis of greater reliance on MF vs MB learning strategies when navigating their environment. While MF systems may be more cognitively efficient, they do not afford flexibility when making prospective predictions about likely outcomes of different decision-tree branches. Emerging research suggests that an acute bout of aerobic exercise improves certain aspects of neurocognition, and thereby could promote the utilization of MB over MF systems during decision making, although prior research has not yet tested this hypothesis. Accordingly, the current study administered a lab-based two-stage Markov decision-making task capable of discriminating MF vs MB decision making, in order to determine if moderate-intensity aerobic exercise (either shortly after or 30-minutes after the exercise bout has ended) promotes greater engagement in MB behavioral strategies compared to light-intensity aerobic exercise in adult women with and without PTSD (N=61). Results revealed that control women generally displayed higher levels of MB behavior that was further increased following immediate exercise, particularly moderate-intensity exercise. By contrast, the PTSD group generally displayed lower levels of MB behavior, and exhibited greater MB behavior when completing the task following moderate-intensity aerobic exercise compared to light-intensity aerobic exercise regardless of whether there was a short or long delay between exercise and the task. Additionally, women with PTSD demonstrated less impairment in MB decision-making compared to controls following moderate-intensity aerobic exercise. These results suggest that an acute bout of moderate-intensity aerobic exercise boosts MB behavior in women with PTSD, and suggests that aerobic exercise may play an important role in enhancing cognitive outcomes for PTSD.

A neurocognitive model of early onset persistent and desistant antisocial behavior in early adulthood

It remains unclear which functional and neurobiological mechanisms are associated with persistent and desistant antisocial behavior in early adulthood. We reviewed the empirical literature and propose a neurocognitive social information processing model for early onset persistent and desistant antisocial behavior in early adulthood, focusing on how young adults evaluate, act upon, monitor, and learn about their goals and self traits. Based on the reviewed literature, we propose that persistent antisocial behavior is characterized by domain-general impairments in self-relevant and goal-related information processing, regulation, and learning, which is accompanied by altered activity in fronto-limbic brain areas. We propose that desistant antisocial development is associated with more effortful information processing, regulation and learning, that possibly balances self-relevant goals and specific situational characteristics. The proposed framework advances insights by considering individual differences such as psychopathic personality traits, and specific emotional characteristics (e.g., valence of social cues), to further illuminate functional and neural mechanisms underlying heterogenous developmental pathways. Finally, we address important open questions and offer suggestions for future research to improve scientific knowledge on general and context-specific expression and development of antisocial behavior in early adulthood.

Neurocognitive basis of model-based decision making and its metacontrol in childhood

Human behavior is supported by both goal-directed (model-based) and habitual (model-free) decision-making, each differing in its flexibility, accuracy, and computational cost. The arbitration between habitual and goal-directed systems is thought to be regulated by a process known as metacontrol. However, how these systems emerge and develop remains poorly understood. Recently, we found that while children between 5 and 11 years displayed robust signatures of model-based decision-making, which increased during this developmental period, there were substantial individual differences in the display of metacontrol. Here, we inspect the neurocognitive basis of model-based decision-making and metacontrol in childhood and focus this investigation on executive functions, fluid reasoning, and brain structure. A total of 69 participants between the ages of 6-13 completed a two-step decision-making task and an extensive behavioral test battery. A subset of 44 participants also completed a structural magnetic resonance imaging scan. We find that individual differences in metacontrol are specifically associated with performance on an inhibition task and individual differences in thickness of dorsolateral prefrontal, temporal, and superior-parietal cortices. These brain regions likely reflect the involvement of cognitive processes crucial to metacontrol, such as cognitive control and contextual processing.

Conditions under which college students cease learning

Introduction

Effective learning involves the acquisition of information toward a goal and cessation upon reaching that goal. Whereas the process of learning acquisition is well understood, comparatively little is known about how or when learning ceases under naturalistic, open-ended learning conditions in which the criterion for performance is not specified. Ideally, learning should cease once there is no progress toward the goal, although this has never been directly tested in human learners. The present set of experiments explored the conditions under which college students stopped attempting to learn a series of inductive perceptual discrimination problems.

Methods

Each problem varied by whether it was solvable and had a criterion for success. The first problem was solvable and involved a pre-determined criterion. The second problem was solvable, but with no criterion for ending the problem so that learners eventually achieved a highly accurate level of performance (overlearning). The third problem was unsolvable as the correct answer varied randomly across features. Measures included the number of trials attempted and the outcome of each problem.

Results and Discussion

Results revealed that college students rarely ceased learning in the overlearning or unsolvable problems even though there was no possibility for further progress. Learning cessation increased only by manipulating time demands for completion or reducing the opportunity for reinforcement. These results suggest that human learners show laudable, but inefficient and unproductive, attempts to master problems they should cease.

Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective

A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.

Rethinking model-based and model-free influences on mental effort and striatal prediction errors

A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Cortical Grey Matter Mediates Increases in Model-Based Control and Learning from Positive Feedback from Adolescence to Adulthood

Cognition and brain structure undergo significant maturation from adolescence into adulthood. Model-based (MB) control is known to increase across development, which is mediated by cognitive abilities. Here, we asked two questions unaddressed in previous developmental studies. First, what are the brain structural correlates of age-related increases in MB control? Second, how are age-related increases in MB control from adolescence to adulthood influenced by motivational context? A human developmental sample (n = 103; age, 12-50, male/female, 55:48) completed structural MRI and an established task to capture MB control. The task was modified with respect to outcome valence by including (1) reward and punishment blocks to manipulate the motivational context and (2) an additional choice test to assess learning from positive versus negative feedback. After replicating that an age-dependent increase in MB control is mediated by cognitive abilities, we demonstrate first-time evidence that gray matter density (GMD) in the parietal cortex mediates the increase of MB control with age. Although motivational context did not relate to age-related changes in MB control, learning from positive feedback improved with age. Meanwhile, negative feedback learning showed no age effects. We present a first report that an age-related increase in positive feedback learning was mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. Our findings indicate that brain maturation, putatively reflected in lower GMD, in distinct and partially overlapping brain regions could lead to a more efficient brain organization and might thus be a key developmental step toward age-related increases in planning and value-based choice.SIGNIFICANCE STATEMENT Changes in model-based decision-making are paralleled by extensive maturation in cognition and brain structure across development. Still, to date the neuroanatomical underpinnings of these changes remain unclear. Here, we demonstrate for the first time that parietal GMD mediates age-dependent increases in model-based control. Age-related increases in positive feedback learning were mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. A manipulation of motivational context did not have an impact on age-related changes in model-based control. These findings highlight that brain maturation in distinct and overlapping cortical regions constitutes a key developmental step toward improved value-based choices.

Distinct reinforcement learning profiles distinguish between language and attentional neurodevelopmental disorders

BACKGROUND

Theoretical models posit abnormalities in cortico-striatal pathways in two of the most common neurodevelopmental disorders (Developmental dyslexia, DD, and Attention deficit hyperactive disorder, ADHD), but it is still unclear what distinct cortico-striatal dysfunction might distinguish language disorders from others that exhibit very different symptomatology. Although impairments in tasks that depend on the cortico-striatal network, including reinforcement learning (RL), have been implicated in both disorders, there has been little attempt to dissociate between different types of RL or to compare learning processes in these two types of disorders. The present study builds upon prior research indicating the existence of two learning manifestations of RL and evaluates whether these processes can be differentiated in language and attention deficit disorders. We used a two-step RL task shown to dissociate model-based from model-free learning in human learners.

RESULTS

Our results show that, relative to neurotypicals, DD individuals showed an impairment in model-free but not in model-based learning, whereas in ADHD the ability to use both model-free and model-based learning strategies was significantly compromised.

CONCLUSIONS

Thus, learning impairments in DD may be linked to a selective deficit in the ability to form action-outcome associations based on previous history, whereas in ADHD some learning deficits may be related to an incapacity to pursue rewards based on the tasks' structure. Our results indicate how different patterns of learning deficits may underlie different disorders, and how computation-minded experimental approaches can differentiate between them.

Diminished reinforcement sensitivity in adolescence is associated with enhanced response switching and reduced coding of choice probability in the medial frontal pole

Precisely charting the maturation of core neurocognitive functions such as reinforcement learning (RL) and flexible adaptation to changing action-outcome contingencies is key for developmental neuroscience and adjacent fields like developmental psychiatry. However, research in this area is both sparse and conflicted, especially regarding potentially asymmetric development of learning for different motives (obtain wins vs avoid losses) and learning from valenced feedback (positive vs negative). In the current study, we investigated the development of RL from adolescence to adulthood, using a probabilistic reversal learning task modified to experimentally separate motivational context and feedback valence, in a sample of 95 healthy participants between 12 and 45. We show that adolescence is characterized by enhanced novelty seeking and response shifting especially after negative feedback, which leads to poorer returns when reward contingencies are stable. Computationally, this is accounted for by reduced impact of positive feedback on behavior. We also show, using fMRI, that activity of the medial frontopolar cortex reflecting choice probability is attenuated in adolescence. We argue that this can be interpreted as reflecting diminished confidence in upcoming choices. Interestingly, we find no age-related differences between learning in win and loss contexts.

Chronic escitalopram in healthy volunteers has specific effects on reinforcement sensitivity: a double-blind, placebo-controlled semi-randomised study

Several studies of the effects on cognition of selective serotonin reuptake inhibitors (SSRI), administered either acutely or sub-chronically in healthy volunteers, have found changes in learning and reinforcement outcomes. In contrast, to our knowledge, there have been no studies of chronic effects of escitalopram on cognition in healthy volunteers. This is important in view of its clinical use in major depressive disorder (MDD) and obsessive-compulsive disorder (OCD). Consequently, we aimed to investigate the chronic effect of the SSRI, escitalopram, on measures of 'cold' cognition (including inhibition, cognitive flexibility, memory) and 'hot cognition' including decision-making and particularly reinforcement learning. The study, conducted at the University of Copenhagen between May 2020 and October 2021, used a double-blind placebo-controlled design with 66 healthy volunteers, semi-randomised to receive either 20 mg of escitalopram (n = 32) or placebo (n = 34), balanced for age, sex and intelligence quotient (IQ) for at least 21 days. Questionnaires, neuropsychological tests and serum escitalopram measures were taken. We analysed group differences on the cognitive measures using linear regression models as well as innovative hierarchical Bayesian modelling of the Probabilistic Reversal Learning (PRL) task. The novel and important finding was that escitalopram reduced reinforcement sensitivity compared to placebo on both the Sequential Model-Based/Model-Free task and the PRL task. We found no other significant group differences on 'cold' or 'hot' cognition. These findings demonstrate that serotonin reuptake inhibition is involved in reinforcement learning in healthy individuals. Lower reinforcement sensitivity in response to chronic SSRI administration may reflect the 'blunting' effect often reported by patients with MDD treated with SSRIs. Trial Registration: NCT04239339 .

Computational and behavioral markers of model-based decision making in childhood

Human decision-making is underpinned by distinct systems that differ in flexibility and associated cognitive cost. A widely accepted dichotomy distinguishes between a cheap but rigid model-free system and a flexible but costly model-based system. Typically, humans use a hybrid of both types of decision-making depending on environmental demands. However, children's use of a model-based system during decision-making has not yet been shown. While prior developmental work has identified simple building blocks of model-based reasoning in young children (1-4 years old), there has been little evidence of this complex cognitive system influencing behavior before adolescence. Here, by using a modified task to make engagement in cognitively costly strategies more rewarding, we show that children aged 5-11-years (N = 85), including the youngest children, displayed multiple indicators of model-based decision making, and that the degree of its use increased throughout childhood. Unlike adults (N = 24), however, children did not display adaptive arbitration between model-free and model-based decision-making. Our results demonstrate that throughout childhood, children can engage in highly sophisticated and costly decision-making strategies. However, the flexible arbitration between decision-making strategies might be a critically late-developing component in human development.

Local and global reward learning in the lateral frontal cortex show differential development during human adolescence

Reward-guided choice is fundamental for adaptive behaviour and depends on several component processes supported by prefrontal cortex. Here, across three studies, we show that two such component processes, linking reward to specific choices and estimating the global reward state, develop during human adolescence and are linked to the lateral portions of the prefrontal cortex. These processes reflect the assignment of rewards contingently to local choices, or noncontingently, to choices that make up the global reward history. Using matched experimental tasks and analysis platforms, we show the influence of both mechanisms increase during adolescence (study 1) and that lesions to lateral frontal cortex (that included and/or disconnected both orbitofrontal and insula cortex) in human adult patients (study 2) and macaque monkeys (study 3) impair both local and global reward learning. Developmental effects were distinguishable from the influence of a decision bias on choice behaviour, known to depend on medial prefrontal cortex. Differences in local and global assignments of reward to choices across adolescence, in the context of delayed grey matter maturation of the lateral orbitofrontal and anterior insula cortex, may underlie changes in adaptive behaviour.

Neural evidence for age-related deficits in the representation of state spaces

Under high cognitive demands, older adults tend to resort to simpler, habitual, or model-free decision strategies. This age-related shift in decision behavior has been attributed to deficits in the representation of the cognitive maps, or state spaces, necessary for more complex model-based decision-making. Yet, the neural mechanisms behind this shift remain unclear. In this study, we used a modified 2-stage Markov task in combination with computational modeling and single-trial EEG analyses to establish neural markers of age-related changes in goal-directed decision-making under different demands on the representation of state spaces. Our results reveal that the shift to simpler decision strategies in older adults is due to (i) impairments in the representation of the transition structure of the task and (ii) a diminished signaling of the reward value associated with decision options. In line with the diminished state space hypothesis of human aging, our findings suggest that deficits in goal-directed, model-based behavior in older adults result from impairments in the representation of state spaces of cognitive tasks.

The Precision in Psychiatry (PIP) study: Testing an internet-based methodology for accelerating research in treatment prediction and personalisation

BACKGROUND

Evidence-based treatments for depression exist but not all patients benefit from them. Efforts to develop predictive models that can assist clinicians in allocating treatments are ongoing, but there are major issues with acquiring the volume and breadth of data needed to train these models. We examined the feasibility, tolerability, patient characteristics, and data quality of a novel protocol for internet-based treatment research in psychiatry that may help advance this field.

METHODS

A fully internet-based protocol was used to gather repeated observational data from patient cohorts receiving internet-based cognitive behavioural therapy (iCBT) (N = 600) or antidepressant medication treatment (N = 110). At baseline, participants provided > 600 data points of self-report data, spanning socio-demographics, lifestyle, physical health, clinical and other psychological variables and completed 4 cognitive tests. They were followed weekly and completed another detailed clinical and cognitive assessment at week 4. In this paper, we describe our study design, the demographic and clinical characteristics of participants, their treatment adherence, study retention and compliance, the quality of the data gathered, and qualitative feedback from patients on study design and implementation.

RESULTS

Participant retention was 92% at week 3 and 84% for the final assessment. The relatively short study duration of 4 weeks was sufficient to reveal early treatment effects; there were significant reductions in 11 transdiagnostic psychiatric symptoms assessed, with the largest improvement seen for depression. Most participants (66%) reported being distracted at some point during the study, 11% failed 1 or more attention checks and 3% consumed an intoxicating substance. Data quality was nonetheless high, with near perfect 4-week test retest reliability for self-reported height (ICC = 0.97).

CONCLUSIONS

An internet-based methodology can be used efficiently to gather large amounts of detailed patient data during iCBT and antidepressant treatment. Recruitment was rapid, retention was relatively high and data quality was good. This paper provides a template methodology for future internet-based treatment studies, showing that such an approach facilitates data collection at a scale required for machine learning and other data-intensive methods that hope to deliver algorithmic tools that can aid clinical decision-making in psychiatry.

The role of the lateral orbitofrontal cortex in creating cognitive maps

We use mental models of the world-cognitive maps-to guide behavior. The lateral orbitofrontal cortex (lOFC) is typically thought to support behavior by deploying these maps to simulate outcomes, but recent evidence suggests that it may instead support behavior by underlying map creation. We tested between these two alternatives using outcome-specific devaluation and a high-potency chemogenetic approach. Selectively inactivating lOFC principal neurons when male rats learned distinct cue-outcome associations, but before outcome devaluation, disrupted subsequent inference, confirming a role for the lOFC in creating new maps. However, lOFC inactivation surprisingly led to generalized devaluation, a result that is inconsistent with a complete mapping failure. Using a reinforcement learning framework, we show that this effect is best explained by a circumscribed deficit in credit assignment precision during map construction, suggesting that the lOFC has a selective role in defining the specificity of associations that comprise cognitive maps.

The pupillometry of the possible: an investigation of infants' representation of alternative possibilities

Contrasting possibilities has a fundamental adaptive value for prediction and learning. Developmental research, however, has yielded controversial findings. Some data suggest that preschoolers might have trouble in planning actions that take into account mutually exclusive possibilities, while other studies revealed an early understanding of alternative future outcomes based on infants' looking behaviour. To better understand the origin of such abilities, here we use pupil dilation as a potential indicator of infants' representation of possibilities. Ten- and 14-month-olds were engaged in an object-identification task by watching video animations where three different objects with identical top parts moved behind two screens. Importantly, a target object emerged from one of the screens but remained in partial occlusion, revealing only its top part, which was compatible with a varying number of possible identities. Just as adults' pupil diameter grows monotonically with the amount of information held in memory, we expected that infants' pupil size would increase with the number of alternatives sustained in memory as candidate identities for the partially occluded object. We found that pupil diameter increased with the object's potential identities in 14- but not in 10-month-olds. We discuss the implications of these results for the foundation of humans' capacities to represent alternatives. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Effects of dopamine D2/3 and opioid receptor antagonism on the trade-off between model-based and model-free behaviour in healthy volunteers

Human behaviour requires flexible arbitration between actions we do out of habit and actions that are directed towards a specific goal. Drugs that target opioid and dopamine receptors are notorious for inducing maladaptive habitual drug consumption; yet, how the opioidergic and dopaminergic neurotransmitter systems contribute to the arbitration between habitual and goal-directed behaviour is poorly understood. By combining pharmacological challenges with a well-established decision-making task and a novel computational model, we show that the administration of the dopamine D2/3 receptor antagonist amisulpride led to an increase in goal-directed or 'model-based' relative to habitual or 'model-free' behaviour, whereas the non-selective opioid receptor antagonist naltrexone had no appreciable effect. The effect of amisulpride on model-based/model-free behaviour did not scale with drug serum levels in the blood. Furthermore, participants with higher amisulpride serum levels showed higher explorative behaviour. These findings highlight the distinct functional contributions of dopamine and opioid receptors to goal-directed and habitual behaviour and support the notion that even small doses of amisulpride promote flexible application of cognitive control.

Model-based prioritization for acquiring protection

Protection often involves the capacity to prospectively plan the actions needed to mitigate harm. The computational architecture of decisions involving protection remains unclear, as well as whether these decisions differ from other beneficial prospective actions such as reward acquisition. Here we compare protection acquisition to reward acquisition and punishment avoidance to examine overlapping and distinct features across the three action types. Protection acquisition is positively valenced similar to reward. For both protection and reward, the more the actor gains, the more benefit. However, reward and protection occur in different contexts, with protection existing in aversive contexts. Punishment avoidance also occurs in aversive contexts, but differs from protection because punishment is negatively valenced and motivates avoidance. Across three independent studies (Total N = 600) we applied computational modeling to examine model-based reinforcement learning for protection, reward, and punishment in humans. Decisions motivated by acquiring protection evoked a higher degree of model-based control than acquiring reward or avoiding punishment, with no significant differences in learning rate. The context-valence asymmetry characteristic of protection increased deployment of flexible decision strategies, suggesting model-based control depends on the context in which outcomes are encountered as well as the valence of the outcome.

The effect of body image dissatisfaction on goal-directed decision making in a population marked by negative appearance beliefs and disordered eating

Eating disorders are associated with one of the highest mortality rates among all mental disorders, yet there is very little research about them within the newly emerging and promising field of computational psychiatry. As such, we focus on investigating a previously unexplored, yet core aspect of eating disorders-body image dissatisfaction. We continue a freshly opened debate about model-based learning and its trade-off against model-free learning-a proxy for goal-directed and habitual behaviour. We perform a behavioural study that utilises a two-step decision-making task and a reinforcement learning model to understand the effect of body image dissatisfaction on model-based learning in a population characterised by high scores of disordered eating and negative appearance beliefs, as recruited using Prolific. We find a significantly reduced model-based contribution in the body image dissatisfaction task condition in the population of interest as compared to a healthy control. This finding suggests general deficits in deliberate control in this population, leading to habitual, compulsive-like behaviours (body checking) dominating the experience. Importantly, the results may inform treatment approaches, which could focus on enhancing the reliance on goal-directed decision making to help cope with unwanted behaviours.

Active inference and the two-step task

Sequential decision problems distill important challenges frequently faced by humans. Through repeated interactions with an uncertain world, unknown statistics need to be learned while balancing exploration and exploitation. Reinforcement learning is a prominent method for modeling such behaviour, with a prevalent application being the two-step task. However, recent studies indicate that the standard reinforcement learning model sometimes describes features of human task behaviour inaccurately and incompletely. We investigated whether active inference, a framework proposing a trade-off to the exploration-exploitation dilemma, could better describe human behaviour. Therefore, we re-analysed four publicly available datasets of the two-step task, performed Bayesian model selection, and compared behavioural model predictions. Two datasets, which revealed more model-based inference and behaviour indicative of directed exploration, were better described by active inference, while the models scored similarly for the remaining datasets. Learning using probability distributions appears to contribute to the improved model fits. Further, approximately half of all participants showed sensitivity to information gain as formulated under active inference, although behavioural exploration effects were not fully captured. These results contribute to the empirical validation of active inference as a model of human behaviour and the study of alternative models for the influential two-step task.

A methodological perspective on learning in the developing brain

The brain undergoes profound development across childhood and adolescence, including continuous changes in brain morphology, connectivity, and functioning that are, in part, dependent on one's experiences. These neurobiological changes are accompanied by significant changes in children's and adolescents' cognitive learning. By drawing from studies in the domains of reading, reinforcement learning, and learning difficulties, we present a brief overview of methodological approaches and research designs that bridge brain- and behavioral research on learning. We argue that ultimately these methods and designs may help to unravel questions such as why learning interventions work, what learning computations change across development, and how learning difficulties are distinct between individuals.

Longitudinal Evidence of a Vicious Cycle Between Nucleus Accumbens Microstructure and Childhood Weight Gain

PURPOSE

Pediatric obesity is a growing public health concern. Previous work has observed diet to impact nucleus accumbens (NAcc) inflammation in rodents, measured by the reactive proliferation of glial cells. Recent work in humans has demonstrated a relationship between NAcc cell density-a proxy for neuroinflammation-and weight gain in youth; however, the directionality of this relationship in the developing brain and association with diet remains unknown.

METHODS

Waist circumference (WC) and NAcc cell density were collected in a large cohort of children (n > 2,000) participating in the Adolescent Brain Cognitive Development (ABCD) Study (release 3.0) at baseline (9-10 y) and at a Year 2 follow-up (11-12 y). Latent change score modeling (LCSM) was used to disentangle contributions of baseline measures to two-year changes in WC percentile and NAcc cellularity. In addition, the role of NAcc cellularity in mediating the relationship between diet and WC percentile was assessed using dietary intake data collected at Year 2.

RESULTS

LCSM indicates that baseline WC percentile influences change in NAcc cellularity and that baseline NAcc cell density influences change in WC percentile. NAcc cellularity was significantly associated with WC percentile at Year 2 and mediated the relationship between dietary fat consumption and WC percentile.

CONCLUSIONS

These results implicate a vicious cycle whereby NAcc cell density biases longitudinal changes in WC percentile and vice versa. Moreover, NAcc cell density may mediate the relationship between diet and weight gain in youth. These findings suggest that diet-induced inflammation of reward circuitry may lead to behavioral changes that further contribute to weight gain.

Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8-30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants' mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.

Developmental shifts in computations used to detect environmental controllability

Accurate assessment of environmental controllability enables individuals to adaptively adjust their behavior—exploiting rewards when desirable outcomes are contingent upon their actions and minimizing costly deliberation when their actions are inconsequential. However, it remains unclear how estimation of environmental controllability changes from childhood to adulthood. Ninety participants (ages 8–25) completed a task that covertly alternated between controllable and uncontrollable conditions, requiring them to explore different actions to discover the current degree of environmental controllability. We found that while children were able to distinguish controllable and uncontrollable conditions, accuracy of controllability assessments improved with age. Computational modeling revealed that whereas younger participants’ controllability assessments relied on evidence gleaned through random exploration, older participants more effectively recruited their task structure knowledge to make highly informative interventions. Age-related improvements in working memory mediated this qualitative shift toward increased use of an inferential strategy. Collectively, these findings reveal an age-related shift in the cognitive processes engaged to assess environmental controllability. Improved detection of environmental controllability may foster increasingly adaptive behavior over development by revealing when actions can be leveraged for one’s benefit.

The ability to determine when one’s actions are consequential organizes learning and decision making across the lifespan. However, few studies have examined how the ability to detect control over our environment changes from childhood to adulthood. Here, we leveraged a computational modeling framework to characterize the component learning processes underlying controllability assessment in children, adolescents, and adults. We observed age-related improvements in controllability assessment that stemmed from an increasing ability to represent contingencies between states and actions and to use that knowledge to make informative interventions that yield diagnostic evidence of the current degree of control. Increasing ability to accurately assess environmental controllability may confer greater recognition of opportunities to adaptively pursue rewards through goal-directed action across development.

Spontaneous discovery of novel task solutions in children

Children often perform worse than adults on tasks that require focused attention. While this is commonly regarded as a sign of incomplete cognitive development, a broader attentional focus could also endow children with the ability to find novel solutions to a given task. To test this idea, we investigated children's ability to discover and use novel aspects of the environment that allowed them to improve their decision-making strategy. Participants were given a simple choice task in which the possibility of strategy improvement was neither mentioned by instructions nor encouraged by explicit error feedback. Among 47 children (8-10 years of age) who were instructed to perform the choice task across two experiments, 27.5% showed a full strategy change. This closely matched the proportion of adults who had the same insight (28.2% of n = 39). The amount of erroneous choices, working memory capacity and inhibitory control, in contrast, indicated substantial disadvantages of children in task execution and cognitive control. A task difficulty manipulation did not affect the results. The stark contrast between age-differences in different aspects of cognitive performance might offer a unique opportunity for educators in fostering learning in children.