Reinforcement learning and bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

Adolescent-to-adult gains in cognitive flexibility are adaptively supported by reward sensitivity, exploration, and neural variability

Cognitive flexibility exhibits dynamic changes throughout development, with different forms of flexibility showing dissociable developmental trajectories. In this review, we propose that an adolescent-specific mode of flexibility in the face of changing environmental contingencies supports the emergence of adolescent-to-adult gains in cognitive shifting efficiency. We first describe how cognitive shifting abilities monotonically improve from childhood to adulthood, accompanied by increases in brain state flexibility, neural variability, and excitatory/inhibitory balance. We next summarize evidence supporting the existence of a dopamine-driven, adolescent peak in flexible behavior that results in reward seeking, undirected exploration, and environmental sampling. We propose a neurodevelopmental framework that relates these adolescent behaviors to the refinement of neural phenotypes relevant to mature cognitive flexibility, and thus highlight the importance of the adolescent period in fostering healthy neurocognitive trajectories.

Lesions to the mediodorsal thalamus, but not orbitofrontal cortex, enhance volatility beliefs linked to paranoia

Beliefs-attitudes toward some state of the environment-guide action selection and should be robust to variability but sensitive to meaningful change. Beliefs about volatility (expectation of change) are associated with paranoia in humans, but the brain regions responsible for volatility beliefs remain unknown. The orbitofrontal cortex (OFC) is central to adaptive behavior, whereas the magnocellular mediodorsal thalamus (MDmc) is essential for arbitrating between perceptions and action policies. We assessed belief updating in a three-choice probabilistic reversal learning task following excitotoxic lesions of the MDmc (n = 3) or OFC (n = 3) and compared performance with that of unoperated monkeys (n = 14). Computational analyses indicated a double dissociation: MDmc, but not OFC, lesions were associated with erratic switching behavior and heightened volatility belief (as in paranoia in humans), whereas OFC, but not MDmc, lesions were associated with increased lose-stay behavior and reward learning rates. Given the consilience across species and models, these results have implications for understanding paranoia.

Artificial neural networks for model identification and parameter estimation in computational cognitive models

Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate.

Stochastic pairwise preference convergence in Bayesian agents

Beliefs inform the behavior of forward-thinking agents in complex environments. Recently, sequential Bayesian inference has emerged as a mechanism to study belief formation among agents adapting to dynamical conditions. However, we lack critical theory to explain how preferences evolve in cases of simple agent interactions. In this paper, we derive a Gaussian, pairwise agent interaction model to study how preferences converge when driven by observation of each other's behaviors. We show that the dynamics of convergence resemble an Ornstein-Uhlenbeck process, a common model in nonequilibrium stochastic dynamics. Using standard analytical and computational techniques, we find that the hyperprior magnitudes, representing the learning time, determine the convergence value and the asymptotic entropy of the preferences across pairs of agents. We also show that the dynamical variance in preferences is characterized by a relaxation time t^{★} and compute its asymptotic upper bound. This formulation enhances the existing toolkit for modeling stochastic, interactive agents by formalizing leading theories in learning theory, and builds towards more comprehensive models of open problems in principal-agent and market theory.

Artificial neural networks for model identification and parameter estimation in computational cognitive models

Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate.

A cognitive-computational account of mood swings in adolescence

Teenagers have a reputation for being fickle, in both their choices and their moods. This variability may help adolescents as they begin to independently navigate novel environments. Recently, however, adolescent moodiness has also been linked to psychopathology. Here, we consider adolescents' mood swings from a novel computational perspective, grounded in reinforcement learning (RL). This model proposes that mood is determined by surprises about outcomes in the environment, and how much we learn from these surprises. It additionally suggests that mood biases learning and choice in a bidirectional manner. Integrating independent lines of research, we sketch a cognitive-computational account of how adolescents' mood, learning, and choice dynamics influence each other, with implications for normative and psychopathological development.

Goal-directed learning in adolescence: neurocognitive development and contextual influences

Adolescence is a time during which we transition to independence, explore new activities and begin pursuit of major life goals. Goal-directed learning, in which we learn to perform actions that enable us to obtain desired outcomes, is central to many of these processes. Currently, our understanding of goal-directed learning in adolescence is itself in a state of transition, with the scientific community grappling with inconsistent results. When we examine metrics of goal-directed learning through the second decade of life, we find that many studies agree there are steady gains in performance in the teenage years, but others report that adolescent goal-directed learning is already adult-like, and some find adolescents can outperform adults. To explain the current variability in results, sophisticated experimental designs are being applied to test learning in different contexts. There is also increasing recognition that individuals of different ages and in different states will draw on different neurocognitive systems to support goal-directed learning. Through adoption of more nuanced approaches, we can be better prepared to recognize and harness adolescent strengths and to decipher the purpose (or goals) of adolescence itself.

Environmental statistics and experience shape risk-taking across adolescence

Adolescents are often portrayed as reckless risk-takers because of their immature brains. Recent research has cast doubt on this portrayal, identifying the environment as a moderator of risk-taking. However, the key features of environments that drive risk-taking behaviors are often underspecified. We call for greater attention to the environment by drawing on research showing that its statistical structure impacts future risk-taking as people learn from outcomes they experience after taking a risk. This opinion shows that adolescents are unlikely to experience harm from many risks because environmental statistics are skewed and favor safe experiences. Environmental statistics and experience suggest entry points for policy interventions by carefully timing risk warnings and leveraging peers' potential to shape the statistics of rewarding experiences.

Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning

Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation plays a fundamental role in instrumental performance and associated mesolimbic dopamine signaling.

Understanding patch foraging strategies across development

Patch foraging is a near-ubiquitous behaviour across the animal kingdom and characterises many decision-making domains encountered by humans. We review how a disposition to explore in adolescence may reflect the evolutionary conditions under which hunter-gatherers foraged for resources. We propose that neurocomputational mechanisms responsible for reward processing, learning, and cognitive control facilitate the transition from exploratory strategies in adolescence to exploitative strategies in adulthood - where individuals capitalise on known resources. This developmental transition may be disrupted by psychopathology, as there is emerging evidence of biases in explore/exploit choices in mental health problems. Explore/exploit choices may be an informative marker for mental health across development and future research should consider this feature of decision-making as a target for clinical intervention.

A neurocognitive model of early onset persistent and desistant antisocial behavior in early adulthood

It remains unclear which functional and neurobiological mechanisms are associated with persistent and desistant antisocial behavior in early adulthood. We reviewed the empirical literature and propose a neurocognitive social information processing model for early onset persistent and desistant antisocial behavior in early adulthood, focusing on how young adults evaluate, act upon, monitor, and learn about their goals and self traits. Based on the reviewed literature, we propose that persistent antisocial behavior is characterized by domain-general impairments in self-relevant and goal-related information processing, regulation, and learning, which is accompanied by altered activity in fronto-limbic brain areas. We propose that desistant antisocial development is associated with more effortful information processing, regulation and learning, that possibly balances self-relevant goals and specific situational characteristics. The proposed framework advances insights by considering individual differences such as psychopathic personality traits, and specific emotional characteristics (e.g., valence of social cues), to further illuminate functional and neural mechanisms underlying heterogenous developmental pathways. Finally, we address important open questions and offer suggestions for future research to improve scientific knowledge on general and context-specific expression and development of antisocial behavior in early adulthood.

Expecting the unexpected: a review of learning under uncertainty across development

Many of our decisions take place under uncertainty. To successfully navigate the environment, individuals need to estimate the degree of uncertainty and adapt their behaviors accordingly by learning from experiences. However, uncertainty is a broad construct and distinct types of uncertainty may differentially influence our learning. We provide a semi-systematic review to illustrate cognitive and neurobiological processes involved in learning under two types of uncertainty: learning in environments with stochastic outcomes, and with volatile outcomes. We specifically reviewed studies (N = 26 studies) that included an adolescent population, because adolescence is a period in life characterized by heightened exploration and learning, as well as heightened uncertainty due to experiencing many new, often social, environments. Until now, reviews have not comprehensively compared learning under distinct types of uncertainties in this age range. Our main findings show that although the overall developmental patterns were mixed, most studies indicate that learning from stochastic outcomes, as indicated by increased accuracy in performance, improved with age. We also found that adolescents tended to have an advantage compared with adults and children when learning from volatile outcomes. We discuss potential mechanisms explaining these age-related differences and conclude by outlining future research directions.

Sensitivity and specificity in affective and social learning in adolescence

Adolescence is a period of heightened affective and social sensitivity. In this review we address how this increased sensitivity influences associative learning. Based on recent evidence from human and rodent studies, as well as advances in computational biology, we suggest that, compared to other age groups, adolescents show features of heightened Pavlovian learning but tend to perform worse than adults at instrumental learning. Because Pavlovian learning does not involve decision-making, whereas instrumental learning does, we propose that these developmental differences might be due to heightened sensitivity to rewards and threats in adolescence, coupled with a lower specificity of responding. We discuss the implications of these findings for adolescent mental health and education.

Confirmatory reinforcement learning changes with age during adolescence

Understanding how learning changes during human development has been one of the long-standing objectives of developmental science. Recently, advances in computational biology have demonstrated that humans display a bias when learning to navigate novel environments through rewards and punishments: they learn more from outcomes that confirm their expectations than from outcomes that disconfirm them. Here, we ask whether confirmatory learning is stable across development, or whether it might be attenuated in developmental stages in which exploration is beneficial, such as in adolescence. In a reinforcement learning (RL) task, 77 participants aged 11-32 years (four men, mean age = 16.26) attempted to maximize monetary rewards by repeatedly sampling different pairs of novel options, which varied in their reward/punishment probabilities. Mixed-effect models showed an age-related increase in accuracy as long as learning contingencies remained stable across trials, but less so when they reversed halfway through the trials. Age was also associated with a greater tendency to stay with an option that had just delivered a reward, more than to switch away from an option that had just delivered a punishment. At the computational level, a confirmation model provided increasingly better fit with age. This model showed that age differences are captured by decreases in noise or exploration, rather than in the magnitude of the confirmation bias. These findings provide new insights into how learning changes during development and could help better tailor learning environments to people of different ages. RESEARCH HIGHLIGHTS: Reinforcement learning shows age-related improvement during adolescence, but more in stable learning environments compared with volatile learning environments. People tend to stay with an option after a win more than they shift from an option after a loss, and this asymmetry increases with age during adolescence. Computationally, these changes are captured by a developing confirmatory learning style, in which people learn more from outcomes that confirm rather than disconfirm their choices. Age-related differences in confirmatory learning are explained by decreases in stochasticity, rather than changes in the magnitude of the confirmation bias.

Neurocomputational mechanisms underlying fear-biased adaptation learning in changing environments

Humans are able to adapt to the fast-changing world by estimating statistical regularities of the environment. Although fear can profoundly impact adaptive behaviors, the computational and neural mechanisms underlying this phenomenon remain elusive. Here, we conducted a behavioral experiment (n = 21) and a functional magnetic resonance imaging experiment (n = 37) with a novel cue-biased adaptation learning task, during which we simultaneously manipulated emotional valence (fearful/neutral expressions of the cue) and environmental volatility (frequent/infrequent reversals of reward probabilities). Across 2 experiments, computational modeling consistently revealed a higher learning rate for the environment with frequent versus infrequent reversals following neutral cues. In contrast, this flexible adjustment was absent in the environment with fearful cues, suggesting a suppressive role of fear in adaptation to environmental volatility. This suppressive effect was underpinned by activity of the ventral striatum, hippocampus, and dorsal anterior cingulate cortex (dACC) as well as increased functional connectivity between the dACC and temporal-parietal junction (TPJ) for fear with environmental volatility. Dynamic causal modeling identified that the driving effect was located in the TPJ and was associated with dACC activation, suggesting that the suppression of fear on adaptive behaviors occurs at the early stage of bottom-up processing. These findings provide a neuro-computational account of how fear interferes with adaptation to volatility during dynamic environments.

Adolescent neurodevelopment and psychopathology: The interplay between adversity exposure and genetic risk for accelerated brain ageing

In adulthood, stress exposure and genetic risk heighten psychological vulnerability by accelerating neurobiological senescence. To investigate whether molecular and brain network maturation processes play a similar role in adolescence, we analysed genetic, as well as longitudinal task neuroimaging (inhibitory control, incentive processing) and early life adversity (i.e., material deprivation, violence) data from the Adolescent Brain and Cognitive Development study (N = 980, age range: 9-13 years). Genetic risk was estimated separately for Major Depressive Disorder (MDD) and Alzheimer's Disease (AD), two pathologies linked to stress exposure and allegedly sharing a causal connection (MDD-to-AD). Adversity and genetic risk for MDD/AD jointly predicted functional network segregation patterns suggestive of accelerated (GABA-linked) visual/attentional, but delayed (dopamine [D2]/glutamate [GLU5R]-linked) somatomotor/association system development. A positive relationship between brain maturation and psychopathology emerged only among the less vulnerable adolescents, thereby implying that normatively maladaptive neurodevelopmental alterations could foster adjustment among the more exposed and genetically more stress susceptible youths. Transcriptomic analyses suggested that sensitivity to stress may underpin the joint neurodevelopmental effect of adversity and genetic risk for MDD/AD, in line with the proposed role of negative emotionality as a precursor to AD, likely to account for the alleged causal impact of MDD on dementia onset.

Transient food insecurity during the juvenile-adolescent period affects adult weight, cognitive flexibility, and dopamine neurobiology

A major challenge for neuroscience, public health, and evolutionary biology is to understand the effects of scarcity and uncertainty on the developing brain. Currently, a significant fraction of children and adolescents worldwide experience insecure access to food. The goal of our work was to test in mice whether the transient experience of insecure versus secure access to food during the juvenile-adolescent period produced lasting differences in learning, decision-making, and the dopamine system in adulthood. We manipulated feeding schedules in mice from postnatal day (P)21 to P40 as food insecure or ad libitum and found that when tested in adulthood (after P60), males with different developmental feeding history showed significant differences in multiple metrics of cognitive flexibility in learning and decision-making. Adult females with different developmental feeding history showed no differences in cognitive flexibility but did show significant differences in adult weight. We next applied reinforcement learning models to these behavioral data. The best fit models suggested that in males, developmental feeding history altered how mice updated their behavior after negative outcomes. This effect was sensitive to task context and reward contingencies. Consistent with these results, in males, we found that the two feeding history groups showed significant differences in the AMPAR/NMDAR ratio of excitatory synapses on nucleus-accumbens-projecting midbrain dopamine neurons and evoked dopamine release in dorsal striatal targets. Together, these data show in a rodent model that transient differences in feeding history in the juvenile-adolescent period can have significant impacts on adult weight, learning, decision-making, and dopamine neurobiology.

Design and Application of Artificial Intelligence Technology-Driven Education and Teaching System in Universities

In recent years, many colleges and universities have been experimenting and exploring the evaluation of education and teaching system and have achieved certain results. In order to understand the quality of education and teaching system in colleges and universities, to improve the school conditions, and to promote the reform of teaching management, methods and means of evaluating the quality of education and teaching system in general higher education institutions are needed. Modern university education and teaching system should realize the combination of classroom teaching and practice teaching, and education and teaching system adopts the mode of the combination of on-campus practice and off-campus practice, so the design of teaching system is the key to the quality of teaching. Aiming at the current problem that talents cultivated by colleges and universities can hardly meet social demands in terms of engineering practice ability, innovation ability, and international competitiveness, this paper proposes the evaluation and adjustment of college education and teaching system driven by algorithms based on artificial intelligence (AI). By designing the teaching system of talent cultivation, and then establishing a quantitative and controllable quality assurance system for practical teaching, a new mechanism for the design of university education system is further explored. Specifically, the framework of the instructional system is built with the aid of an actor-critic algorithm in reinforcement learning, which assists in the design of the university education system, allowing students to truly understand, master and flex their knowledge, and strengthening the correct understanding of the students' internal learning mechanisms. The practical teaching effect shows that the AI-driven instructional designs are more popular with contemporary students and have higher evaluation scores. The numerical experiment results also show the stability of the instructional design, overcoming the drawbacks of traditional manual subjectivity in the design. AI-driven college education and teaching system is conducive to cultivating students' solid technical theoretical foundation. Therefore, through the AI-driven teaching system to strengthen the training of practical ability, so as to comprehensively improve students' comprehensive quality and innovation ability.

The interpretation of computational model parameters depends on the context

Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g. tasks, models) and that they capture interpretable (i.e. unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8–30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. Still, generalization was significantly below the methodological ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g. reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models.