Reinforcement Learning during Adolescence in Rats

Social experience in adolescence shapes prefrontal cortex structure and function in adulthood

During adolescence, the prefrontal cortex (PFC) undergoes dramatic reorganization. PFC development is profoundly influenced by the social environment, disruptions to which may prime the emergence of psychopathology across the lifespan. We investigated the neurobehavioral consequences of isolation experienced in adolescence in mice, and in particular, the long-term consequences that were detectable even despite normalization of the social milieu. Isolation produced biases toward habit-like behavior at the expense of flexible goal seeking, plus anhedonic-like reward deficits. Behavioral phenomena were accompanied by neuronal dendritic spine over-abundance and hyper-excitability in the ventromedial PFC (vmPFC), which was necessary for the expression of isolation-induced habits and sufficient to trigger behavioral inflexibility in socially reared controls. Isolation activated cytoskeletal regulatory pathways otherwise suppressed during adolescence, such that repression of constituent elements prevented long-term isolation-induced neurosequelae. Altogether, our findings unveil an adolescent critical period and multi-model mechanism by which social experiences facilitate prefrontal cortical maturation.

Adolescent-to-adult gains in cognitive flexibility are adaptively supported by reward sensitivity, exploration, and neural variability

Cognitive flexibility exhibits dynamic changes throughout development, with different forms of flexibility showing dissociable developmental trajectories. In this review, we propose that an adolescent-specific mode of flexibility in the face of changing environmental contingencies supports the emergence of adolescent-to-adult gains in cognitive shifting efficiency. We first describe how cognitive shifting abilities monotonically improve from childhood to adulthood, accompanied by increases in brain state flexibility, neural variability, and excitatory/inhibitory balance. We next summarize evidence supporting the existence of a dopamine-driven, adolescent peak in flexible behavior that results in reward seeking, undirected exploration, and environmental sampling. We propose a neurodevelopmental framework that relates these adolescent behaviors to the refinement of neural phenotypes relevant to mature cognitive flexibility, and thus highlight the importance of the adolescent period in fostering healthy neurocognitive trajectories.

Regulation of reinforcement learning parameters captures long-term changes in rat behaviour

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Sex differences in oxycodone-taking behaviors are linked to disruptions in reward-guided, decision-making functions

Problematic opioid use that emerges in a subset of individuals may be due to pre-existing disruptions in the biobehavioral mechanisms that regulate drug use. The identity of these mechanisms is not known, but emerging evidence suggests that suboptimal decision-making that is observable prior to drug use may contribute to the pathology of addiction and, notably, serve as a powerful phenotype for interrogating biologically based differences in opiate-taking behaviors. The current study investigated the relationship between decision-making phenotypes and opioid-taking behaviors in male and female Long Evans rats. Adaptive decision-making processes were assessed using a probabilistic reversal-learning task and oxycodone- (or vehicle, as a control) taking behaviors assessed for 32 days using a saccharin fading procedure that promoted dynamic intake of oxycodone. Tests of motivation, extinction, and reinstatement were also performed. Computational analyses of decision-making and opioid-taking behaviors revealed that attenuated reward-guided decision-making was associated with greater self-administration of oxycodone and addiction-relevant behaviors. Moreover, pre-existing impairments in reward-guided decision-making observed in female rats was associated with greater oxycodone use and addiction-relevant behaviors when compared to males. These results provide new insights into the biobehavioral mechanisms that regulate opiate-taking behaviors and offer a novel phenotypic approach for interrogating sex differences in addiction susceptibility and opioid use disorders.

Cocaine reward and reinstatement in adolescent versus adult rodents

Adolescence is a critical juncture when initiation of drug use intersects with profound developmental changes in the brain. Adolescent drug use increases the risk to develop substance use disorders (SUDs) later in life, but the mechanisms that confer this vulnerability are not understood. SUDs are defined by cycles of use, abstinence, and relapse. Intense craving during drug-free periods is often triggered by cues and environmental contexts associated with previous use. In contrast to our understanding of stimuli that elicit craving and relapse in adults, the behavioral processes that occur during periods of abstinence and relapse in adolescents are poorly understood. The current mini-review will summarize findings from preclinical rodent studies that used cocaine conditioned place preference and operant cocaine self-administration to examine subsequent effects on reward, relapse and incubation of craving.

Associations between nesting, stereotypy, and working memory in deer mice: response to levetiracetam

BACKGROUND

Some deer mice (Peromyscus maniculatus bairdii) exhibit various phenotypes of persistent behaviors. It remains unknown if and how said phenotypes associate with early-life and adult cognitive perturbations, and whether potentially cognitive enhancing drugs might modify such associations. Here, we explored the longitudinal relationship between early-life behavioral flexibility and the expression of persistent behavior in adulthood. We also investigated how said phenotypes might associate with working memory in adulthood, and how this association might respond to chronic exposure to the putative cognitive enhancer, levetiracetam (LEV).

METHODS

76 juvenile deer mice were assessed for habit-proneness in the Barnes maze (BM) and divided into two exposure groups (n = 37-39 per group), i.e., control and LEV (75 mg/kg/day). After 56 days of uninterrupted exposure, mice were screened for nesting and stereotypical behavior, and then assessed for working memory in the T-maze.

RESULTS

Juvenile deer mice overwhelmingly utilize habit-like response strategies, regardless of LNB and HS behavior in adulthood. Further, LNB and HS are unrelated in terms of their expression, while LEV reduces the expression of LNB, but bolsters CR (but not VA). Last, an increased level of control over high stereotypical expression may facilitate improved working memory performance.

CONCLUSION

LNB, VA and CR, are divergent in terms of their neurocognitive underpinnings. Chronic LEV administration throughout the entire rearing period may be of benefit to some phenotypes, e.g., LNB, but not others (CR). We also show that an increased level of control over the expression of stereotypy may facilitate improved working memory performance.

Adolescent reinforcement-learning trajectories predict cocaine-taking behaviors in adult male and female rats

The anatomical, structural, and functional adaptations that occur in the brain during adolescence are thought to facilitate improvements in decision-making functions that are known to occur during this stage of development. The mechanisms that underlie these neural adaptations are not known, but deviations in developmental trajectories have been proposed to contribute to the emergence of mental illness, including addiction. Direct evidence supporting this hypothesis, however, has been limited. Here, we used a recently developed reversal-learning protocol to investigate the predictive relationship between adolescent decision-making trajectories and cocaine-taking behaviors in adulthood. Decision-making functions in the reversal-learning task were assessed throughout adolescence and into adulthood in male and female Long-Evans rats. Trial-by-trial choice data was fitted with a reinforcement-learning model to quantify the degree to which choice behavior of individual rats was influenced by rewarded (e.g., ∆₊ parameter) and unrewarded (e.g., ∆₀ parameter) outcomes. We report that reversal-learning performance improved during adolescence and that this was due to an increase in value updating for rewarded outcomes (e.g., ∆₊ parameter). Furthermore, the rate of change in the ∆₊ parameter predicted individual differences in the ∆₊ parameter and, notably, cocaine-taking behaviors in adulthood: Rats that had a shallower adolescent trajectory were found to have a lower ∆₊ parameter and greater cocaine self-administration in adulthood. These data indicate that adolescent development plays a critical role in drug use susceptibility. Future studies aimed at understanding the neurobiological mechanisms that underlie these age-related changes in decision-making could provide new insights into the biobehavioral mechanisms mediating addiction susceptibility.

Pruning recurrent neural networks replicates adolescent changes in working memory and reinforcement learning

SignificanceAdolescence is a period during which there are important changes in behavior and the structure of the brain. In this manuscript, we use theoretical modeling to show how improvements in working memory and reinforcement learning that occur during adolescence can be explained by the reduction in synaptic connectivity in prefrontal cortex that occurs during a similar period. We train recurrent neural networks to solve working memory and reinforcement learning tasks and show that when we prune connectivity in these networks, they perform the tasks better. The improvement in task performance, however, can come at the cost of flexibility as the pruned networks are not able to learn some new tasks as well.

Cell adhesion presence during adolescence controls the architecture of projection-defined prefrontal cortical neurons and reward-related action strategies later in life

Adolescent brain development is characterized by neuronal remodeling in the prefrontal cortex; relationships with behavior are largely undefined. Integrins are cell adhesion factors that link the extracellular matrix with intracellular actin cytoskeleton. We find that β1-integrin presence in the prelimbic prefrontal cortex (PL) during adolescence, but not adulthood, is necessary for mice to select actions based on reward likelihood and value. As such, adult mice that lacked β1-integrin during adolescence failed to modify response strategies when rewards lost value or failed to be delivered. This pattern suggests that β1-integrin-mediated neuronal development is necessary for PL function in adulthood. We next visualized adolescent PL neurons, including those receiving input from the basolateral amygdala (BLA) - thought to signal salience - and projecting to the dorsomedial striatum (DMS) - the striatal output by which the PL controls goal-seeking behavior. Firstly, we found that these projection-defined neurons had a distinct morphology relative to general layer V PL neurons. Secondly, β1-integrin loss triggered the overexpression of stubby-type dendritic spines at the expense of mature spines, including on projection-defined neurons. This phenotype was not observed when β1-integrins were silenced before or after adolescence. Altogether, our experiments localize β1-integrin-mediated cell adhesion within a developing di-synaptic circuit coordinating adaptive action.

Reinforcement learning detuned in addiction: integrative and translational approaches

Suboptimal decision-making strategies have been proposed to contribute to the pathophysiology of addiction. Decision-making, however, arises from a collection of computational components that can independently influence behavior. Disruptions in these different components can lead to decision-making deficits that appear similar behaviorally, but differ at the computational, and likely the neurobiological, level. Here, we discuss recent studies that have used computational approaches to investigate the decision-making processes underlying addiction. Studies in animal models have found that value updating following positive, but not negative, outcomes is predictive of drug use, whereas value updating following negative, but not positive, outcomes is disrupted following drug self-administration. We contextualize these findings with studies on the circuit and biological mechanisms of decision-making to develop a framework for revealing the biobehavioral mechanisms of addiction.

Persistent behavioral and neurobiological consequences of social isolation during adolescence

Meaningful social interactions are a fundamental human need, the lack of which can pose serious risks to an individual's physical and mental health. Across species, peer-oriented social behaviors are dramatically reshaped during adolescence, a developmental period characterized by dynamic changes in brain structure and function as individuals transition into adulthood. Thus, the experience of social isolation during this critical developmental stage may be especially pernicious, as it could permanently derail typical neurobiological processes that are necessary for establishing adaptive adult behaviors. The purpose of this review is to summarize investigations in which rodents were isolated during adolescence, then re-housed in typical social groups prior to testing, thus allowing the investigators to resolve the long-term consequences of social adversity experienced during adolescent sensitive periods, despite subsequent normalization of the social environment. Here, we discuss alterations in social, anxiety-like, cognitive, and decision-making behaviors in previously isolated adult rodents. We then explore corresponding neurobiological findings, focusing on the prefrontal cortex, including changes in synaptic densities and protein levels, white matter and oligodendrocyte function, and neuronal physiology. Made more urgent by the recent wave of social deprivation resulting from the COVID-19 pandemic, especially amongst school-aged adolescents, understanding the mechanisms by which even transient social adversity can negatively impact brain function across the lifespan is of paramount importance.

Linking mPFC circuit maturation to the developmental regulation of emotional memory and cognitive flexibility

The medial prefrontal cortex (mPFC) and its abundant connections with other brain regions play key roles in memory, cognition, decision making, social behaviors, and mood. Dysfunction in mPFC is implicated in psychiatric disorders in which these behaviors go awry. The prolonged maturation of mPFC likely enables complex behaviors to emerge, but also increases their vulnerability to disruption. Many foundational studies have characterized either mPFC synaptic or behavioral development without establishing connections between them. Here, we review this rich body of literature, aligning major events in mPFC development with the maturation of complex behaviors. We focus on emotional memory and cognitive flexibility, and highlight new work linking mPFC circuit disruption to alterations of these behaviors in disease models. We advance new hypotheses about the causal connections between mPFC synaptic development and behavioral maturation and propose research strategies to establish an integrated understanding of neural architecture and behavioral repertoires.

Unlocking the reinforcement-learning circuits of the orbitofrontal cortex

Neuroimaging studies have consistently identified the orbitofrontal cortex (OFC) as being affected in individuals with neuropsychiatric disorders. OFC dysfunction has been proposed to be a key mechanism by which decision-making impairments emerge in diverse clinical populations, and recent studies employing computational approaches have revealed that distinct reinforcement-learning mechanisms of decision-making differ among diagnoses. In this perspective, we propose that these computational differences may be linked to select OFC circuits and present our recent work that has used a neurocomputational approach to understand the biobehavioral mechanisms of addiction pathology in rodent models. We describe how combining translationally analogous behavioral paradigms with reinforcement-learning algorithms and sophisticated neuroscience techniques in animals can provide critical insights into OFC pathology in biobehavioral disorders. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Sex differences in pubertal associations with fronto-accumbal white matter morphometry: Implications for understanding sensitivity to reward and punishment

Researchers have reported sex-differentiated maturation of white matter (WM) during puberty. It is not clear, however, whether such distinctions contribute to documented sex differences in sensitivity to reward and punishment during adolescence. Given the role of the orbitofrontal cortex (OFC) and nucleus accumbens (NAcc) in reward and punishment-related behaviors, we tested in a cross-sectional study whether males and females (N = 156, 89 females; ages 9-14 years) differ in the association between pubertal stage and fixel-based morphometry of WM fibers connecting the OFC and NAcc (i.e., the fronto-accumbal tract). Further, we examined whether males and females differ in associations between fronto-accumbal WM measures and self-reported sensitivity to reward and punishment. Pubertal stage was positively associated with fronto-accumbal fiber density and cross-section (FDC) in males, but not in females. Consistent with previous reports, males reported higher reward sensitivity than did females, although fronto-accumbal combined FDC was not related to reward sensitivity in either sex. Meanwhile, only males showed a negative association between fronto-accumbal tract FDC and sensitivity to punishment. Follow-up analyses revealed that fiber cross-section, but not density, was related to pubertal stage and punishment sensitivity in males, as well as to reward sensitivity in all participants. Our findings suggest there are sex differences in puberty-related maturation of the fronto-accumbal tract, and this tract is related to lower punishment sensitivity in adolescent males compared to females.