Goal-oriented and habitual decisions: Neural signatures of model-based and model-free learning

Mechanisms of brain self-regulation: psychological factors, mechanistic models and neural substrates

While neurofeedback represents a promising tool for neuroscience and a brain self-regulation approach to psychological rehabilitation, the field faces several problems and challenges. Current research has shown great variability and even failure among human participants in learning to self-regulate target features of brain activity with neurofeedback. A better understanding of cognitive mechanisms, psychological factors and neural substrates underlying self-regulation might help improve neurofeedback's scientific and clinical practices. This article reviews the current understanding of the neural mechanisms of brain self-regulation by drawing on findings from human and animal studies in neurofeedback, brain-computer/machine interfaces and neuroprosthetics. In this article, we look closer at the following topics: cognitive processes and psychophysiological factors affecting self-regulation, theoretical models and neural substrates underlying self-regulation, and finally, we provide an outlook on the outstanding gaps in knowledge and technical challenges. This article is part of the theme issue 'Neurofeedback: new territories and neurocognitive mechanisms of endogenous neuromodulation'.

Global neural encoding of behavioral strategies in mice during perceptual decision-making task with two different sensory patterns

When a simple model-free strategy does not provide sufficient outcomes, an inference-based strategy estimating a hidden task structure becomes essential for optimizing choices. However, the neural circuitry involved in inference-based strategies is still unclear. We developed a tone frequency discrimination task in head-fixed mice in which the tone category of the current trial depended on the category of the previous trial. When the tone category was repeated, the mice continued using the default model-free strategy, as well as when the tone was randomly presented, to bias choices. In contrast, when the tone was alternated, the default strategy gradually shifted to a hybrid of model-free and inference-based strategies, although we did not observe distinct strategy changes. Brain-wide electrophysiological recording suggested that the neural activity of the frontal and sensory cortices, hippocampus, and striatum was correlated with the reward expectation in different task conditions, suggesting the global encoding of multiple strategies in the brain.

A sequence bottleneck for animal intelligence and language?

We discuss recent findings suggesting that non-human animals lack memory for stimulus sequences, and therefore do not represent the order of stimuli faithfully. These observations have far-reaching consequences for animal cognition, neuroscience, and studies of the evolution of language and culture. This is because, if non-human animals do not remember or process information about order faithfully, then it is unlikely that non-human animals perform mental simulations, construct mental world models, have episodic memory, or transmit culture faithfully. If this suggested sequence bottleneck proves to be a prevalent characteristic of animal memory systems, as suggested by recent work, it would require a re-examination of some influential concepts and ideas.

Revisiting the role of computational neuroimaging in the era of integrative neuroscience

Computational models have become integral to human neuroimaging research, providing both mechanistic insights and predictive tools for human cognition and behavior. However, concerns persist regarding the ecological validity of lab-based neuroimaging studies and whether their spatiotemporal resolution is not sufficient for capturing neural dynamics. This review aims to re-examine the utility of computational neuroimaging, particularly in light of the growing prominence of alternative neuroscientific methods and the growing emphasis on more naturalistic behaviors and paradigms. Specifically, we will explore how computational modeling can both enhance the analysis of high-dimensional imaging datasets and, conversely, how neuroimaging, in conjunction with other data modalities, can inform computational models through the lens of neurobiological plausibility. Collectively, this evidence suggests that neuroimaging remains critical for human neuroscience research, and when enhanced by computational models, imaging can serve an important role in bridging levels of analysis and understanding. We conclude by proposing key directions for future research, emphasizing the development of standardized paradigms and the integrative use of computational modeling across neuroimaging techniques.

The reverberation of implementation errors in a neuroimaging meta-analytic software package: A citation analysis to a technical report on GingerALE

GingerALE, a widely used neuroimaging meta-analysis software package, contained errors in earlier versions that were later corrected. The technical report "Implementation errors in the GingerALE Software: description and recommendations" by Eickhoff et al. (2017) documented these errors and their corresponding fixes. In the current study, the papers that cited the GingerALE technical report were analyzed to identify the reasons for these citations. In August 2023, a search through Web of Science Core Collection identified 158 papers that cited the GingerALE technical report. These papers were manually examined to extract the citation statements and code the citation reasons into 12 categories. The analysis revealed that the most frequent reason for citing the report was to justify the use of a specific statistical threshold, followed by a simple acknowledgement of using GingerALE, acknowledging the impact of the errors in earlier versions of GingerALE on prior studies or the lack of effect on current results, and justifying the number of experiments in a meta-analysis. A small number of reasons related to non-GingerALE software, matters not related to activation likelihood estimation (ALE), or statements not mentioned in the GingerALE technical report.

Distinct cholinergic circuits underlie discrete effects of reward on attention

Attention and reward are functions that are critical for the control of behavior, and massive multi-region neural systems have evolved to support the discrete computations associated with each. Previous research has also identified that attention and reward interact, though our understanding of the neural mechanisms that mediate this interplay is incomplete. Here, we review the basic neuroanatomy of attention, reward, and cholinergic systems. We then examine specific contexts in which attention and reward computations interact. Building on this work, we propose two discrete neural circuits whereby acetylcholine, released from cell groups located in different parts of the brain, mediates the impact of stimulus-reward associations as well as motivation on attentional control. We conclude by examining these circuits as a potential shared loci of dysfunction across diseases states associated with deficits in attention and reward.

Distinct roles of monkey OFC-subcortical pathways in adaptive behavior

Primates must adapt to changing environments by optimizing their behavior to make beneficial choices. At the core of adaptive behavior is the orbitofrontal cortex (OFC) of the brain, which updates choice value through direct experience or knowledge-based inference. Here, we identify distinct neural circuitry underlying these two separate abilities. We designed two behavioral tasks in which two male macaque monkeys updated the values of certain items, either by directly experiencing changes in stimulus-reward associations, or by inferring the value of unexperienced items based on the task's rules. Chemogenetic silencing of bilateral OFC combined with mathematical model-fitting analysis revealed that monkey OFC is involved in updating item value based on both experience and inference. In vivo imaging of chemogenetic receptors by positron emission tomography allowed us to map projections from the OFC to the rostromedial caudate nucleus (rmCD) and the medial part of the mediodorsal thalamus (MDm). Chemogenetic silencing of the OFC-rmCD pathway impaired experience-based value updating, while silencing the OFC-MDm pathway impaired inference-based value updating. Our results thus demonstrate dissociable contributions of distinct OFC projections to different behavioral strategies, and provide new insights into the neural basis of value-based adaptive decision-making in primates.

Bad habits-good goals? Meta-analysis and translation of the habit construct to alcoholism

Excessive alcohol consumption remains a global public health crisis, with millions suffering from alcohol use disorder (AUD, or simply "alcoholism"), leading to significantly reduced life expectancy. This review examines the interplay between habitual and goal-directed behaviors and the associated neurobiological changes induced by chronic alcohol exposure. Contrary to a strict habit-goal dichotomy, our meta-analysis of the published animal experiments combined with a review of human studies reveals a nuanced transition between these behavioral control systems, emphasizing the need for refined terminology to capture the probabilistic nature of decision biases in individuals with a history of chronic alcohol exposure. Furthermore, we distinguish habitual responding from compulsivity, viewing them as separate entities with diverse roles throughout the stages of the addiction cycle. By addressing species-specific differences and translational challenges in habit research, we provide insights to enhance future investigations and inform strategies for combatting AUD.

Corticostriatal activity related to performance during continuous de novo motor learning

Corticostriatal regions play a pivotal role in visuomotor learning. However, less research has been done on how fMRI activity in their subregions is related to task performance, which is provided as visual feedback during motor learning. To address this, we conducted an fMRI experiment in which participants acquired a complex de novo motor skill using continuous or binary visual feedback related to performance. We found a highly selective response related to performance in the entire striatum in both conditions and a relatively higher response in the caudate nucleus for the binary feedback condition. However, the ventromedial prefrontal cortex (vmPFC) response was significant only for the continuous feedback condition. Furthermore, we also found functional distinction of the striatal subregions in random versus goal-directed motor control. These findings underscore the substantial effects of the visual feedback indicating performance on distinct corticostriatal responses, thereby elucidating its significance in reinforcement-based motor learning.

Model-based and model-free mechanisms in methamphetamine use disorder

People with methamphetamine use disorder (MUD) struggle to shift their behaviour from methamphetamine-orientated habits to goal-oriented choices. The model-based/model-free framework is well suited to understand this difficulty by unpacking the computational mechanisms that support experienced-based (model-free) and goal-directed (model-based) choices. We aimed to examine whether 1) participants with MUD differed from controls on behavioural proxies and/or computational mechanisms of model-based/model-free choices; 2) model-based/model-free decision-making correlated with MUD symptoms; and 3) model-based/model-free deficits improved over six weeks in the group with MUD. Participants with MUD and controls with similar age, IQ and socioeconomic status completed the Two-Step Task at treatment commencement (MUD n = 30, Controls n = 31) and six weeks later (MUD n = 23, Controls n = 26). We examined behavioural proxies of model-based/model-free decisions using mixed logistic regression, and their underlying mechanisms using computational modelling. At a behavioural level, participants with MUD were more likely to switch their choices following rewarded actions, although this pattern improved at follow up. At a computational level, groups were similar in their use of model-based mechanisms, but participants with MUD were less likely to apply model-free mechanisms and less likely to repeat rewarded actions. We did not find evidence that individual differences in model-based or model-free parameters were associated with greater severity of methamphetamine dependence, nor did we find that group differences in computational parameters changed between baseline and follow-up assessment. Decision-making challenges in people with MUD are likely related to difficulties in pursuing choices previously associated with positive outcomes.

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

Self-regulation learning as active inference: dynamic causal modeling of an fMRI neurofeedback task

Introduction

Learning to self-regulate brain activity by neurofeedback has been shown to lead to changes in the brain and behavior, with beneficial clinical and non-clinical outcomes. Neurofeedback uses a brain-computer interface to guide participants to change some feature of their brain activity. However, the neural mechanism of self-regulation learning remains unclear, with only 50% of the participants succeeding in achieving it. To bridge this knowledge gap, our study delves into the neural mechanisms of self-regulation learning via neurofeedback and investigates the brain processes associated with successful brain self-regulation.

Methods

We study the neural underpinnings of self-regulation learning by employing dynamical causal modeling (DCM) in conjunction with real-time functional MRI data. The study involved a cohort of 18 participants undergoing neurofeedback training targeting the supplementary motor area. A critical focus was the comparison between top-down hierarchical connectivity models proposed by Active Inference and alternative bottom-up connectivity models like reinforcement learning.

Results

Our analysis revealed a crucial distinction in brain connectivity patterns between successful and non-successful learners. Particularly, successful learners evinced a significantly stronger top-down effective connectivity towards the target area implicated in self-regulation. This heightened top-down network engagement closely resembles the patterns observed in goal-oriented and cognitive control studies, shedding light on the intricate cognitive processes intertwined with self-regulation learning.

Discussion

The findings from our investigation underscore the significance of cognitive mechanisms in the process of self-regulation learning through neurofeedback. The observed stronger top-down effective connectivity in successful learners indicates the involvement of hierarchical cognitive control, which aligns with the tenets of Active Inference. This study contributes to a deeper understanding of the neural dynamics behind successful self-regulation learning and provides insights into the potential cognitive architecture underpinning this process.

Cognitive and neural bases of visual-context-guided decision-making

Humans adjust their behavioral strategies based on feedback, a process that may depend on intrinsic preferences and contextual factors such as visual salience. In this study, we hypothesized that decision-making based on visual salience is influenced by habitual and goal-directed processes, which can be evidenced by changes in attention and subjective valuation systems. To test this hypothesis, we conducted a series of studies to investigate the behavioral and neural mechanisms underlying visual salience-driven decision-making. We first established the baseline behavioral strategy without salience in Experiment 1 (n = 21). We then highlighted the utility or performance dimension of the chosen outcome using colors in Experiment 2 (n = 30). We demonstrated that the difference in staying frequency increased along the salient dimension, confirming a salience effect. Furthermore, the salience effect was abolished when directional information was removed in Experiment 3 (n = 28), suggesting that the salience effect is feedback-specific. To generalize our findings, we replicated the feedback-specific salience effects using eye-tracking and text emphasis. The fixation differences between the chosen and unchosen values were enhanced along the feedback-specific salient dimension in Experiment 4 (n = 48) but unchanged after removing feedback-specific information in Experiment 5 (n = 32). Moreover, the staying frequency was correlated with fixation properties, confirming that salience guides attention deployment. Lastly, our neuroimaging study (Experiment 6, n = 25) showed that the striatum subregions encoded salience-based outcome evaluation, while the vmPFC encoded salience-based behavioral adjustments. The connectivity of the vmPFC-ventral striatum accounted for individual differences in utility-driven, whereas the vmPFC-dmPFC for performance-driven behavioral adjustments. Together, our results provide a neurocognitive account of how task-irrelevant visual salience drives decision-making by involving attention and the frontal-striatal valuation systems. PUBLIC SIGNIFICANCE STATEMENT: Humans may use the current outcome to make behavior adjustments. How this occurs may depend on stable individual preferences and contextual factors, such as visual salience. Under the hypothesis that visual salience determines attention and subsequently modulates subjective valuation, we investigated the underlying behavioral and neural bases of visual-context-guided outcome evaluation and behavioral adjustments. Our findings suggest that the reward system is orchestrated by visual context and highlight the critical role of attention and the frontal-striatal neural circuit in visual-context-guided decision-making that may involve habitual and goal-directed processes.

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

A framework for application of consumer neuroscience in pro-environmental behavior change interventions

Most consumers are aware that climate change is a growing problem and admit that action is needed. However, research shows that consumers' behavior often does not conform to their value and orientations. This value-behavior gap is due to contextual factors such as price, product design, and social norms as well as individual factors such as personal and hedonic values, environmental beliefs, and the workload capacity an individual can handle. Because of this conflict of interest, consumers have a hard time identifying the true drivers of their behavior, as they are either unaware of or unwilling to acknowledge the processes at play. Therefore, consumer neuroscience methods might provide a valuable tool to uncover the implicit measurements of pro-environmental behavior (PEB). Several studies have already defined neurophysiological differences between green and non-green individuals; however, a behavior change intervention must be developed to motivate PEB among consumers. Motivating behavior with reward or punishment will most likely get users engaged in climate change action via brain structures related to the reward system, such as the amygdala, nucleus accumbens, and (pre)frontal cortex, where the reward information and subsequent affective responses are encoded. The intensity of the reward experience can be increased when the consumer is consciously considering the action to achieve it. This makes goal-directed behavior the potential aim of behavior change interventions. This article provides an extensive review of the neuroscientific evidence for consumer attitude, behavior, and decision-making processes in the light of sustainability incentives for behavior change interventions. Based on this review, we aim to unite the current theories and provide future research directions to exploit the power of affective conditioning and neuroscience methods for promoting PEB engagement.

Mother’s physical activity during pregnancy and newborn’s brain cortical development

Background

Physical activity is known to improve mental health, and is regarded as safe and desirable for uncomplicated pregnancy. In this novel study, we aim to evaluate whether there are associations between maternal physical activity during pregnancy and neonatal brain cortical development.

Methods

Forty-four mother/newborn dyads were included in this longitudinal study. Healthy pregnant women were recruited and their physical activity throughout pregnancy were documented using accelerometers worn for 3–7 days for each of the 6 time points at 4–10, ∼12, ∼18, ∼24, ∼30, and ∼36 weeks of pregnancy. Average daily total steps and daily total activity count as well as daily minutes spent in sedentary/light/moderate/vigorous activity modes were extracted from the accelerometers for each time point. At ∼2 weeks of postnatal age, their newborns underwent an MRI examination of the brain without sedation, and 3D T1-weighted brain structural images were post-processed by the iBEAT2.0 software utilizing advanced deep learning approaches. Cortical surface maps were reconstructed from the segmented brain images and parcellated to 34 regions in each brain hemisphere, and mean cortical thickness for each region was computed for partial correlation analyses with physical activity measures, with appropriate multiple comparison corrections and potential confounders controlled.

Results

At 4–10 weeks of pregnancy, mother’s daily total activity count positively correlated (FDR corrected P ≤ 0.05) with newborn’s cortical thickness in the left caudal middle frontal gyrus (rho = 0.48, P = 0.04), right medial orbital frontal gyrus (rho = 0.48, P = 0.04), and right transverse temporal gyrus (rho = 0.48, P = 0.04); mother’s daily time in moderate activity mode positively correlated with newborn’s cortical thickness in the right transverse temporal gyrus (rho = 0.53, P = 0.03). At ∼24 weeks of pregnancy, mother’s daily total activity count positively correlated (FDR corrected P ≤ 0.05) with newborn’s cortical thickness in the left (rho = 0.56, P = 0.02) and right isthmus cingulate gyrus (rho = 0.50, P = 0.05).

Conclusion

We identified significant relationships between physical activity in healthy pregnant women during the 1st and 2nd trimester and brain cortical development in newborns. Higher maternal physical activity level is associated with greater neonatal brain cortical thickness, presumably indicating better cortical development.

An fMRI meta-analysis of the role of the striatum in everyday-life vs laboratory-developed habits

The dorsolateral striatum plays a critical role in the acquisition and expression of stimulus-response habits that are learned in experimental laboratories. Here, we use meta-analytic procedures to contrast the neural circuits activated by laboratory-acquired habits with those activated by stimulus-response behaviours acquired in everyday-life. We confirmed that newly learned habits rely more on the anterior putamen with activation extending into caudate and nucleus accumbens. Motor and associative components of everyday-life habits were identified. We found that motor-dominant stimulus-response associations developed outside the laboratory primarily engaged posterior dorsal putamen, supplementary motor area (SMA) and cerebellum. Importantly, associative components were also represented in the posterior putamen. Thus, common neural representations for both naturalistic and laboratory-based habits were found in the left posterior and right anterior putamen. These findings suggest a partial common striatal substrate for habitual actions that are performed predominantly by stimulus-response associations represented in the posterior striatum. The overlapping neural substrates for laboratory and everyday-life habits supports the use of both methods for the analysis of habitual behaviour.

What Can Reinforcement Learning Models of Dopamine and Serotonin Tell Us about the Action of Antidepressants?

Although evidence suggests that antidepressants are effective at treating depression, the mechanisms behind antidepressant action remain unclear, especially at the cognitive/computational level. In recent years, reinforcement learning (RL) models have increasingly been used to characterise the roles of neurotransmitters and to probe the computations that might be altered in psychiatric disorders like depression. Hence, RL models might present an opportunity for us to better understand the computational mechanisms underlying antidepressant effects. Moreover, RL models may also help us shed light on how these computations may be implemented in the brain (e.g., in midbrain, striatal, and prefrontal regions) and how these neural mechanisms may be altered in depression and remediated by antidepressant treatments. In this paper, we evaluate the ability of RL models to help us understand the processes underlying antidepressant action. To do this, we review the preclinical literature on the roles of dopamine and serotonin in RL, draw links between these findings and clinical work investigating computations altered in depression, and appraise the evidence linking modification of RL processes to antidepressant function. Overall, while there is no shortage of promising ideas about the computational mechanisms underlying antidepressant effects, there is insufficient evidence directly implicating these mechanisms in the response of depressed patients to antidepressant treatment. Consequently, future studies should investigate these mechanisms in samples of depressed patients and assess whether modifications in RL processes mediate the clinical effect of antidepressant treatments.

A Neuroeconomics Approach to Obesity

Obesity is a heterogeneous condition that is affected by physiological, behavioral, and environmental factors. Value-based decision making is a useful framework for integrating these factors at the individual level. The disciplines of behavioral economics and reinforcement learning provide tools for identifying specific cognitive and motivational processes that may contribute to the development and maintenance of obesity. Neuroeconomics complements these disciplines by studying the neural mechanisms underlying these processes. We surveyed recent literature on individual decision characteristics that are most frequently implicated in obesity: discounting the value of future outcomes, attitudes toward uncertainty, and learning from rewards and punishments. Our survey highlighted both consistent and inconsistent behavioral findings. These findings underscore the need to examine multiple processes within individuals to identify unique behavioral profiles associated with obesity. Such individual characterization will inform future studies on the neurobiology of obesity as well as the design of effective interventions that are individually tailored.

Action and emotion perception in Parkinson's disease: A neuroimaging meta-analysis

•

The neural substrates for action and emotion perception deficits in PD are still unclear.

•

We addressed this issue via coordinate-based meta-analyses of previous fMRI data.

•

PD patients exhibit decreased response in the basal ganglia.

•

PD patients exhibit a trend toward decreased response in the parietal areas.

•

PD patients exhibit a trend toward increased activation in the posterior cerebellum.

Patients with Parkinson disease (PD) may show impairments in the social perception. Whether these deficits have been consistently reported, it remains to be clarified which brain alterations subtend them. To this aim, we conducted a neuroimaging meta-analysis to compare the brain activity during social perception in patients with PD versus healthy controls. Our results show that PD patients exhibit a significantly decreased response in the basal ganglia (putamen and pallidum) and a trend toward decreased activity in the mirror system, particularly in the left parietal cortex (inferior parietal lobule and intraparietal sulcus). This reduced activation may be tied to a disruption of cognitive resonance mechanisms and may thus constitute the basis of impaired others’ representations underlying action and emotion perception. We also found increased activation in the posterior cerebellum in PD, although only in a within-group analysis and not in comparison with healthy controls. This cerebellar activation may reflect compensatory mechanisms, an aspect that deserves further investigation. We discuss the clinical implications of our findings for the development of novel social skill training programs for PD patients.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Examining the effect of depressive symptoms on habit formation and habit-breaking

BACKGROUND AND OBJECTIVES

Dysfunction in reward processing is a hallmark feature of depression. In the context of reinforcement learning, previous research has linked depression with reliance on simple habit-driven ('model-free') learning strategies over more complex, goal-directed ('model-based') strategies. However, the relationship between depression and habit-breaking remains an under-explored research area. The current study sought to bridge this gap by investigating the effect of depressive symptoms on habit formation and habit-breaking under monetary and social feedback conditions. Additionally, we examined whether spontaneous eyeblink rate (EBR), an indirect marker for striatal dopamine levels, would modulate such effects.

METHODS

Depressive symptoms were operationalized using self-report measures. To examine differences in habit formation and habit breaking, undergraduate participants (N = 156) completed a two-stage reinforcement learning task with a devaluation procedure using either monetary or social feedback.

RESULTS

Regression results showed that in the monetary feedback condition, spontaneous EBR moderated the relationship between depressive symptoms and model-free strategies; individuals with more depressive symptomatology and high EBR (higher dopamine levels) exhibited increased reliance on model-free strategies. Depressive symptoms negatively predicted devaluation sensitivity, indicative of difficulty in habit-breaking, in both monetary and social feedback contexts.

LIMITATIONS

Social feedback relied on fixed feedback rather than real-time peer evaluations; depressive symptoms were measured using self-report rather than diagnostic criteria for Major Depressive Disorder; dopaminergic functioning was measured using EBR rather than PET imaging; potential confounds were not controlled for.

CONCLUSIONS

These findings have implications for identifying altered patterns of habit formation and deficits in habit-breaking among those experiencing depressive symptoms.

Distributed cell assemblies spanning prefrontal cortex and striatum

Highly synchronous neuronal assembly activity is deemed essential for cognitive brain function. In theory, such synchrony could coordinate multiple brain areas performing complementary processes. However, cell assemblies have been observed only in single structures, typically cortical areas, and little is known about their synchrony with downstream subcortical structures, such as the striatum. Here, we demonstrate distributed cell assemblies activated at high synchrony (∼10 ms) spanning prefrontal cortex and striatum. In addition to including neurons at different brain hierarchical levels, surprisingly, they synchronized functionally distinct limbic and associative sub-regions. These assembly activations occurred when members shifted their firing phase relative to ongoing 4 Hz and theta rhythms, in association with high gamma oscillations. This suggests that these rhythms could mediate the emergence of cross-structural assemblies. To test for the role of assemblies in behavior, we trained the rats to perform a task requiring cognitive flexibility, alternating between two different rules in a T-maze. Overall, assembly activations were correlated with task-relevant parameters, including impending choice, reward, rule, or rule order. Moreover, these behavioral correlates were more robustly expressed by assemblies than by their individual member neurons. Finally, to verify whether assemblies can be endogenously generated, we found that they were indeed spontaneously reactivated during sleep and quiet immobility. Thus, cell assemblies are a more general coding mechanism than previously envisioned, linking distributed neocortical and subcortical areas at high synchrony.

Valence bias in metacontrol of decision making in adolescents and young adults

The development of metacontrol of decision making and its susceptibility to framing effects were investigated in a sample of 201 adolescents and adults in Germany (12-25 years, 111 female, ethnicity not recorded). In a task that dissociates model-free and model-based decision making, outcome magnitude and outcome valence were manipulated. Both adolescents and adults showed metacontrol and metacontrol tended to increase across adolescence. Furthermore, model-based decision making was more pronounced for loss compared to gain frames but there was no evidence that this framing effect differed with age. Thus, the strategic adaptation of decision making continues to develop into young adulthood and for both adolescents and adults, losses increase the motivation to invest cognitive resources into an effortful decision-making strategy.

Astrocyte-neuron interaction in the dorsal striatum-pallidal circuits and alcohol-seeking behaviors

In the striatum, two main types of GABAergic medium spiny neurons (MSNs), denoted striatonigral (or direct-pathway MSNs, dMSNs) and striatopallidal neurons (indirect-pathway MSNs, iMSNs), form circuits with distinct pallidal nuclei, which sends "GO" or "NO-GO" signals through the thalamus. These striatopallidal circuits evaluate and execute reward-seeking and taking behaviors. Especially, the dorsal striatum can be further divided into the dorsomedial striatum (DMS, equivalent to caudate in primates and humans) and dorsolateral striatum (DLS, equivalent to putamen), which orchestrates goal-directed and habitual reward-seeking and taking behaviors, respectively. Using optogenetics, chemogenetics and in vivo calcium imaging technologies combined with electrophysiology and digitalized behavior phenotyping, recent studies have revealed cell-, circuit- and context-specific functions of these microcircuits in addictive behaviors. Also, region-specific astrocytes regulate the homeostatic activities of the dMSNs and iMSNs as well as the downstream circuits, which determine the net balance of cortico-striato-pallidal activities to the thalamic neurons. This review will summarize the recent progress of striatopallidal circuits focusing on astrocyte-neuron interaction and, reward- and alcohol-seeking behaviors. Our review will also discuss the translational and clinical implications of these microcircuit studies. This article is part of the special Issue on "Neurocircuitry Modulating Drug and Alcohol Abuse".

Parallel and hierarchical neural mechanisms for adaptive and predictive behavioral control

Our brain can be recognized as a network of largely hierarchically organized neural circuits that operate to control specific functions, but when acting in parallel, enable the performance of complex and simultaneous behaviors. Indeed, many of our daily actions require concurrent information processing in sensorimotor, associative, and limbic circuits that are dynamically and hierarchically modulated by sensory information and previous learning. This organization of information processing in biological organisms has served as a major inspiration for artificial intelligence and has helped to create in silico systems capable of matching or even outperforming humans in several specific tasks, including visual recognition and strategy-based games. However, the development of human-like robots that are able to move as quickly as humans and respond flexibly in various situations remains a major challenge and indicates an area where further use of parallel and hierarchical architectures may hold promise. In this article we review several important neural and behavioral mechanisms organizing hierarchical and predictive processing for the acquisition and realization of flexible behavioral control. Then, inspired by the organizational features of brain circuits, we introduce a multi-timescale parallel and hierarchical learning framework for the realization of versatile and agile movement in humanoid robots.

The tDCS effect on Prosocial Behavior: A Meta-Analytic Review

Previous studies have shown that transcranial direct current stimulation (tDCS) could potentially promote prosocial behaviors. However, results from randomized controlled trials are inconsistent. The current meta-analysis aimed to assess the effects of anodal and cathodal tDCS using single-session protocols on prosocial behaviors in healthy young adults and explore potential moderators of these effects. The results showed that compared with sham stimulation, anodal (excitatory) stimulation significantly increased (g = 0.27, 95% CI [0.11, 0.43], Z = 3.30, P = 0.001) and cathodal (inhibitory) stimulation significantly decreased prosocial behaviors (g = −0.19, 95% CI [−0.39, −0.01], Z = −1.95, P = 0.051) using a multilevel meta-analytic model. These effects were not significantly modulated by stimulation parameters (e.g. duration, intensity and site) and types of prosocial behavior. The risk of publication bias for the included effects was minimal, and no selective reporting (e.g. P-hacking) was found in the P-curve analysis. This meta-analysis showed that both anodal and cathodal tDCS have small but significant effects on prosocial behaviors. The current study provides evidence that prosocial behaviors are linked to the activity of the ‘social brain’. Future studies are encouraged to further explore whether tDCS could effectively treat social dysfunctions in psychiatry disorders.

The neural underpinnings of allocentric thinking in a novel signaling task

The ability to adopt the perspectives of others is fundamental to effective communication in social interactions. However, the neural correlates of allocentric thinking in communicative signaling remain unclear. We adapted a novel signaling task in which the signaler was given the target word and must choose a one-word signal to help the receiver guess the target. Behavioral results suggest that speakers can use allocentric thinking to choose signals that are salient from the perspective of the receiver rather than their own point of view. At the neural level, functional magnetic resonance imaging (fMRI) data reveal that the medial prefrontal cortex (mPFC), ventral striatum, and temporal-parietal junction are more activated when signalers engage in allocentric than egocentric thinking. Moreover, functional connectivity between the mPFC and ventral striatum predicted individuals' perspective-taking ability during successful communication. These findings reveal that neural representations in the mPFC-striatum network support perspective-taking in complex social decision making, providing a new perspective on how the brain arbitrates between allocentric thinking and egocentric thinking in communication and social coordination.

Why and how the brain weights contributions from a mixture of experts

It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a "Mixture of Experts" in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavior at any one moment in time by keeping track of the reliability of the predictions within each system, and by allocating control over behavior in a manner that depends on the relative reliabilities across experts. fMRI and neurostimulation studies suggest a specific contribution of the anterior prefrontal cortex in this process. Further, such a mechanism also takes into consideration the complexity of the expert, favoring simpler over more cognitively complex experts. Results from the study of different expert systems in both experiential and social learning domains hint at the possibility that this reliability-based control mechanism is domain general, exerting control over many different expert systems simultaneously in order to produce sophisticated behavior.

The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection

Behavioral control is not unitary. It comprises parallel systems, model based and model free, that respectively generate flexible and habitual behaviors. Model-based decisions use predictions of the specific consequences of actions, but how these are implemented in the brain is poorly understood. We used calcium imaging and optogenetics in a sequential decision task for mice to show that the anterior cingulate cortex (ACC) predicts the state that actions will lead to, not simply whether they are good or bad, and monitors whether outcomes match these predictions. ACC represents the complete state space of the task, with reward signals that depend strongly on the state where reward is obtained but minimally on the preceding choice. Accordingly, ACC is necessary only for updating model-based strategies, not for basic reward-driven action reinforcement. These results reveal that ACC is a critical node in model-based control, with a specific role in predicting future states given chosen actions.

A brain network that supports consensus-seeking and conflict-resolving of college couples' shopping interaction

One of the typical campus scenes is the social interaction between college couples, and the lesson couples must keep learning is to adapt to each other. This fMRI study investigated the shopping interactions of 30 college couples, one lying inside and the other outside the scanner, beholding the same item from two connected PCs, making preference ratings and subsequent buy/not-buy decisions. The behavioral results showed the clear modulation of significant others’ preferences onto one’s own decisions, and the contrast of the “shop-together vs. shop-alone”, and the “congruent (both liked or disliked the item, 68%) vs. incongruent (one liked but the other disliked, and vice versa)” together trials, both revealed bilateral temporal parietal junction (TPJ) among other reward-related regions, likely reflecting mentalizing during preference harmony. Moreover, when contrasting “own-high/other-low vs. own-low/other-high” incongruent trials, left anterior inferior parietal lobule (l-aIPL) was parametrically mapped, and the “yield (e.g., own-high/not-buy) vs. insist (e.g., own-low/not-buy)” modulation further revealed left lateral-IPL (l-lIPL), together with left TPJ forming a local social decision network that was further constrained by the mediation analysis among left TPJ–lIPL–aIPL. In sum, these results exemplify, via the two-person fMRI, the neural substrate of shopping interactions between couples.

Striatal and hippocampal contributions to flexible navigation in rats and humans

The hippocampus has been firmly established as playing a crucial role in flexible navigation. Recent evidence suggests that dorsal striatum may also play an important role in such goal-directed behaviour in both rodents and humans. Across recent studies, activity in the caudate nucleus has been linked to forward planning and adaptation to changes in the environment. In particular, several human neuroimaging studies have found the caudate nucleus tracks information traditionally associated with that by the hippocampus. In this brief review, we examine this evidence and argue the dorsal striatum encodes the transition structure of the environment during flexible, goal-directed behaviour. We highlight that future research should explore the following: (1) Investigate neural responses during spatial navigation via a biophysically plausible framework explained by reinforcement learning models and (2) Observe the interaction between cortical areas and both the dorsal striatum and hippocampus during flexible navigation.