Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal

Valence-dependent dopaminergic modulation during reversal learning in Parkinson's disease: A neurocomputational approach

Reinforcement learning, crucial for behavior in dynamic environments, is driven by rewards and punishments, modulated by dopamine (DA) changes. This study explores the dopaminergic system's influence on learning, particularly in Parkinson's disease (PD), where medication leads to impaired adaptability. Highlighting the role of tonic DA in signaling the valence of actions, this research investigates how DA affects response vigor and decision-making in PD. DA not only influences reward and punishment learning but also indicates the cognitive effort level and risk propensity in actions, which are essential for understanding and managing PD symptoms. In this work, we adapt our existing neurocomputational model of basal ganglia (BG) to simulate two reversal learning tasks proposed by Cools et al. We first optimized a Hebb rule for both probabilistic and deterministic reversal learning, conducted a sensitivity analysis (SA) on parameters related to DA effect, and compared performances between three groups: PD-ON, PD-OFF, and control subjects. In our deterministic task simulation, we explored switch error rates after unexpected task switches and found a U-shaped relationship between tonic DA levels and switch error frequency. Through SA, we classify these three groups. Then, assuming that the valence of the stimulus affects the tonic levels of DA, we were able to reproduce the results by Cools et al. As for the probabilistic task simulation, our results are in line with clinical data, showing similar trends with PD-ON, characterized by higher tonic DA levels that are correlated with increased difficulty in both acquisition and reversal tasks. Our study proposes a new hypothesis: valence, signaled by tonic DA levels, influences learning in PD, confirming the uncorrelation between phasic and tonic DA changes. This hypothesis challenges existing paradigms and opens new avenues for understanding cognitive processes in PD, particularly in reversal learning tasks.

Neuronal diversity in the caudate nucleus: A comparative study between camel and human brains

Caudate nucleus (CN) neurons in camels and humans were examined using modified Golgi impregnation methods. Neurons were classified based on soma morphology, dendritic characteristics, and spine distribution. Three primary neuron types were identified in both species: rich-spiny (Type I), sparsely-spiny (Type II), and aspiny (Type III), each comprising subtypes with specific features. Comparative analysis revealed significant differences in soma size, dendritic morphology, and spine distribution between camels and humans. The study contributes to our understanding of structural diversity in CN neurons and provides insights into evolutionary neural adaptations.

Dynamic diversity is the answer to proxy failure

We argue that a diverse and dynamic pool of agents mitigates proxy failure. Proxy modularity plays a key role in the ongoing production of diversity. We review examples from a range of scales.

Impaired executive functioning mediates the association between aging and deterministic sequence learning

Sensitivity to the fixed ordering of actions and events, or deterministic sequence learning, is an important skill throughout adulthood. Yet, it remains unclear whether age deficits in sequencing exist, and we lack a firm understanding of which factors might contribute to age-related impairments when they arise. Though debated, executive functioning, governed by the frontal lobe, may underlie age-related sequence learning deficits in older adults. The present study asked if age predicts errors in deterministic sequence learning across the older adult lifespan (ages 55-89), and whether executive functioning accounts for any age-related declines. Healthy older adults completed a comprehensive measure of frontal-based executive abilities as well as a deterministic sequence learning task that required the step-by-step acquisition of associations through trial-and-error feedback. Among those who met a performance-based criterion, increasing age was positively correlated with higher sequencing errors; however, this relationship was no longer significant after controlling for executive functioning. Moreover, frontal-based executive abilities mediated the relationship between age and sequence learning performance. These findings suggest that executive or frontal functioning may underlie age deficits in learning judgment-based, deterministic serial operations.

Early life interpersonal stress and depression: Social reward processing as a potential mediator

Experiencing stressful events early in life is lamentably very common and widespread across the globe. Despite the strong link between experiencing such stress and developing depression, the mechanisms underlying this association remain unclear. This review addresses this critical question by drawing focus to "early life interpersonal stress" (ELIS), or stressful experiences that occur within the context of a relationship where there is close, direct interaction. Recent evidence suggests that ELIS uniquely relates to depression. A growing body of work demonstrates that ELIS impacts how youth respond to social reward (e.g., positive social stimuli/ feedback). Similar social reward-related impairments are noted in youth with depression. The current review synthesizes these two disparate, yet related, bodies of literature examining the relations between a) ELIS and neurobehavioral alterations in social reward processing; and b) behavioral and neural processing of social reward in depression. A preliminary model presents neurobehavioral disruptions in social reward processing as one mediating factor underlying the connection between ELIS and depression. Key limitations and future directions are discussed.

A computational network dynamical modeling for abnormal oscillation and deep brain stimulation control of obsessive-compulsive disorder

Obsessive-compulsive disorder (OCD) is associated with multi-nodal abnormalities in brain networks, characterized by recurrent intrusive thoughts (obsessions) and repetitive behaviours or mental acts (compulsions), which might manifest as pathological low-frequency oscillations in the frontal EEG and low-frequency bursting firing patterns in the subthalamus nucleus (STN). Abnormalities in the cortical-striatal-thalamic-cortical (CSTC) loop, including dysregulation of serotonin, dopamine, and glutamate systems, are considered to contribute to certain types of OCD. Here, we extend a biophysical computational model to investigate the effect of orbitofronto-subcortical loop abnormalities on network oscillations. Particularly, the OCD lesion process is simulated by the loss of connectivity from striatal parvalbumin interneurons (PV) to medium spiny neurons (MSNs), excessive activation to the hyperdirect pathway, and high dopamine concentrations. By calculating low-frequency oscillation power in the STN, STN burst index, and average firing rates levels of the cortex and thalamus, we demonstrate that the model can explain the pathology of glutamatergic and dopamine system dysregulation, the effects of pathway imbalance, and neuropsychiatric treatment in OCD. In addition, results indicate the abnormal brain rhythms caused by the dysregulation of orbitofronto-subcortical loop may serve as a biomarker of OCD. Our studies can help to understand the cause of OCD, thereby facilitating the diagnosis of OCD and the development of new therapeutics.

Art Value Creation and Destruction

I present a theory of creative and destructive value state referring to abstract art. Value is a probabilistic state held as a mixture of its expectation and information forces that coexist in a give-and-take relationship. Expectations are driven by the disclosure of novel information about the value state of various events of desire. Each bit of accumulated information contributes to the improvement of perception up to a threshold level, beyond which begin conscious states. The desire to disclose a value state triggers a triadic system of evaluation which uses concepts, observables and approaches. While the triadic valuation mechanisms can be used to assess various commodities, the scope of this work is limited to the case of artworks, in particular abstract paintings. I assume that art value is basically mediated by the interplay between these value state mechanisms of creation and destruction. Expectations in artwork develop attraction by challenging its contemplator to evaluate (predict) its meaning. Once the relevant information, corresponding to its creative expectations, is acquired (and conditioned), emotional states of indifference, disinterest and desensitization develop.

BrainCog: A spiking neural network based, brain-inspired cognitive intelligence engine for brain-inspired AI and brain simulation

Spiking neural networks (SNNs) serve as a promising computational framework for integrating insights from the brain into artificial intelligence (AI). Existing software infrastructures based on SNNs exclusively support brain simulation or brain-inspired AI, but not both simultaneously. To decode the nature of biological intelligence and create AI, we present the brain-inspired cognitive intelligence engine (BrainCog). This SNN-based platform provides essential infrastructure support for developing brain-inspired AI and brain simulation. BrainCog integrates different biological neurons, encoding strategies, learning rules, brain areas, and hardware-software co-design as essential components. Leveraging these user-friendly components, BrainCog incorporates various cognitive functions, including perception and learning, decision-making, knowledge representation and reasoning, motor control, social cognition, and brain structure and function simulations across multiple scales. BORN is an AI engine developed by BrainCog, showcasing seamless integration of BrainCog's components and cognitive functions to build advanced AI models and applications.

Emotional subtypes in patients with depression: A cluster analysis

Major depressive disorder (MDD) is associated with deficits in emotion experience, expression and regulation. Whilst emotion regulation deficits prolong MDD, emotion expression influences symptomatic presentations, and anticipatory pleasure deficits predict recurrence risk. Profiling MDD patients from an emotion componential perspective can characterize subtypes with different clinical and functional outcomes. This study aimed to investigate emotional subtypes of MDD. A two-stage cluster analysis applied to 150 MDD patients. Clustering variables included emotion experience measured by Temporal Experience of Pleasure Scale, emotion expression measured by Toronto Alexithymia Scale, and emotion regulation measured by Emotion Regulation Questionnaire. We validated the resultant clusters by comparing their symptoms and functioning with that of 50 controls. Cluster 1 (n = 50) exhibited intact emotion experience and expression yet adopted reappraisal rather than suppression strategy, whereas Cluster 2 (n = 66) exhibited generalized emotional deficits. Cluster 3 (n = 34) exhibited emotion expression deficits and adopted both reappraisal and suppression strategies. On validation, Cluster 2 exhibited the worst, but Cluster 1 exhibited the least symptoms and social functioning impairments. Cluster 3 was intermediate among the two other subtypes. Our findings support the existence of different emotional subtypes in MDD patients, and have clinical and theoretical implications for developing future specific treatments for MDD.

On the normative advantages of dopamine and striatal opponency for learning and choice

The basal ganglia (BG) contribute to reinforcement learning (RL) and decision-making, but unlike artificial RL agents, it relies on complex circuitry and dynamic dopamine modulation of opponent striatal pathways to do so. We develop the OpAL* model to assess the normative advantages of this circuitry. In OpAL*, learning induces opponent pathways to differentially emphasize the history of positive or negative outcomes for each action. Dynamic DA modulation then amplifies the pathway most tuned for the task environment. This efficient coding mechanism avoids a vexing explore-exploit tradeoff that plagues traditional RL models in sparse reward environments. OpAL* exhibits robust advantages over alternative models, particularly in environments with sparse reward and large action spaces. These advantages depend on opponent and nonlinear Hebbian plasticity mechanisms previously thought to be pathological. Finally, OpAL* captures risky choice patterns arising from DA and environmental manipulations across species, suggesting that they result from a normative biological mechanism.

Neurocognitive Psychiatric and Neuropsychological Alterations in Parkinson's Disease: A Basic and Clinical Approach

The main histopathological hallmarks of Parkinson's disease (PD) are the degeneration of the dopaminergic neurons of the substantia nigra pars compacta and the loss of neuromelanin as a consequence of decreased dopamine synthesis. The destruction of the striatal dopaminergic pathway and blocking of striatal dopamine receptors cause motor deficits in humans and experimental animal models induced by some environmental agents. In addition, neuropsychiatric symptoms such as mood and anxiety disorders, hallucinations, psychosis, cognitive impairment, and dementia are common in PD. These alterations may precede the appearance of motor symptoms and are correlated with neurochemical and structural changes in the brain. This paper reviews the most crucial pathophysiology of neuropsychiatric alterations in PD. It is worth noting that PD patients have global task learning deficits, and cognitive functions are compromised in a way is associated with hypoactivation within the striatum, anterior cingulate cortex, and inferior frontal sulcus regions. An appropriate and extensive neuropsychological screening battery in PD must accurately assess at least five cognitive domains with some tests for each cognitive domain. This neuropsychological screening should consider the pathophysiological and clinical heterogeneity of cognitive dysfunction in PD.

Stress effects on memory retrieval of aversive and appetitive instrumental counterconditioning in men

Extinction training creates a second inhibitory memory trace and effectively reduces conditioned responding. However, acute stress inhibits the retrieval of this extinction memory trace. It is not known whether this also applies to other forms of associative learning such as instrumental counterconditioning, where previously learned associations are reversed and paired with the opposite valence. Therefore, the current preregistered study investigates whether stress decreases the retrieval of instrumental counterconditioning memories with aversive and appetitive consequences. Fifty-two healthy men were randomly assigned to either a stress or control group and took part in a two-day instrumental learning paradigm. During a first phase, participants learned that pressing specific buttons in response to the presentation of four neutral stimuli either leads to gaining or losing money. During a second phase, two stimuli reversed their contingencies (counterconditioning). One day later, participants were exposed to acute stress or a control condition prior to the same task, which no longer included feedback about gains or losses. Stressed participants showed more approach behavior towards appetitive and less avoidance behavior towards aversive stimuli as compared to non-stressed participants. Our findings indicate that stress effects on memory retrieval differ depending on the associative learning approach in men. These differences might be related to stress effects on decision making and different motivational systems involved.

The interpretation of computational model parameters depends on the context

Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g. tasks, models) and that they capture interpretable (i.e. unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8-30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. Still, generalization was significantly below the methodological ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g. reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models.

Relating a Spiking Neural Network Model and the Diffusion Model of Decision-Making

Many models of decision making assume accumulation of evidence to threshold as a core mechanism to predict response probabilities and response times. A spiking neural network model (Wang, 2002) instantiates these mechanisms at the level of biophysically-plausible pools of neurons with excitatory and inhibitory connections, and has numerous model parameters tuned by physiological measures. The diffusion model (Ratcliff, 1978) is a cognitive model that can be fitted to a range of behaviors and conditions. We investigated how parameters of the cognitive-level diffusion model relate to the parameters of a neural-level spiking model. In each simulated "experiment", we generated "data" from the spiking neural network by factorially combining a manipulation of choice difficulty (via the input to the spiking model) and a manipulation of one of the core parameters of the spiking model. We then fitted the diffusion model to these simulated data to observe how manipulation of each core spiking model parameter mapped on to fitted drift rate, response threshold, and non-decision time. Manipulations of parameters in the spiking model related to input sensitivity, threshold, and stimulus processing time mapped on to their conceptual analogues in the diffusion model, namely drift rate, threshold, and non-decision time. Manipulations of parameters in the spiking model with no direct analogue to the diffusion model, non-stimulus-specific background input, strength of recurrent excitation, and receptor conductances, mapped on to threshold in the diffusion model. We discuss implications of these results for interpretations of fits of the diffusion model to behavioral data.

Aberrant functional connectivity within the salience network is related to cognitive deficits and disorganization in psychosis

In schizophrenia and schizoaffective disorder cognitive deficits are a reliable characteristic predicting a poor functional outcome. It has been theorized that both the default mode network (DMN) and the salience network (SN) play a crucial role in cognitive processes and aberrant functional connectivity within these networks in psychotic patients has been reported. The goal of this study was to reveal potential links between aberrant functional connectivity within these networks and impaired cognitive performance in psychosis. We chose two approaches for cognitive assessment, first the MATRICS Consensus Cognitive Battery (MCCB) combined into a global score and second the disorganization factor derived from a five-factor model of the Positive and Negative Syndrome Scale (PANSS) known to be relevant for cognitive performance. DMN and SN were identified using independent component analysis on resting-state functional magnetic resonance imaging data. We found significantly decreased connectivity within the right supplementary motor area (SMA) and bilateral putamen in patients with psychosis (n = 70; 27F/43M) compared to healthy controls (n = 72; 28F/44M). Within patients, linear regression analysis revealed that aberrant SMA connectivity was associated with impaired global cognition, while dysfunctional bilateral putamen connectivity predicted disorganization. There were no significant changes in connectivity within the DMN. Results support the hypothesis that SN dysfunctional connectivity is important in the pathobiology of cognitive deficits in psychosis. For the first time we were able to show the involvement of dysfunctional SMA connectivity in this context. We interpret the decreased SN connectivity as evidence of reduced functionality in recruiting brain areas necessary for cognitive processing.

A mechanistic model of ADHD as resulting from dopamine phasic/tonic imbalance during reinforcement learning

Attention deficit hyperactivity disorder (ADHD) is the most common neurodevelopmental disorder in children. Although the involvement of dopamine in this disorder seems to be established, the nature of dopaminergic dysfunction remains controversial. The purpose of this study was to test whether the key response characteristics of ADHD could be simulated by a mechanistic model that combines a decrease in tonic dopaminergic activity with an increase in phasic responses in cortical-striatal loops during learning reinforcement. To this end, we combined a dynamic model of dopamine with a neurocomputational model of the basal ganglia with multiple action channels. We also included a dynamic model of tonic and phasic dopamine release and control, and a learning procedure driven by tonic and phasic dopamine levels. In the model, the dopamine imbalance is the result of impaired presynaptic regulation of dopamine at the terminal level. Using this model, virtual individuals from a dopamine imbalance group and a control group were trained to associate four stimuli with four actions with fully informative reinforcement feedback. In a second phase, they were tested without feedback. Subjects in the dopamine imbalance group showed poorer performance with more variable reaction times due to the presence of fast and very slow responses, difficulty in choosing between stimuli even when they were of high intensity, and greater sensitivity to noise. Learning history was also significantly more variable in the dopamine imbalance group, explaining 75% of the variability in reaction time using quadratic regression. The response profile of the virtual subjects varied as a function of the learning history variability index to produce increasingly severe impairment, beginning with an increase in response variability alone, then accumulating a decrease in performance and finally a learning deficit. Although ADHD is certainly a heterogeneous disorder, these results suggest that typical features of ADHD can be explained by a phasic/tonic imbalance in dopaminergic activity alone.

Functional alterations in large-scale resting-state networks of amyotrophic lateral sclerosis: A multi-site study across Canada and the United States

Amyotrophic lateral sclerosis (ALS) is a multisystem neurodegenerative disorder characterized by progressive degeneration of upper motor neurons and lower motor neurons, and frontotemporal regions resulting in impaired bulbar, limb, and cognitive function. Magnetic resonance imaging studies have reported cortical and subcortical brain involvement in the pathophysiology of ALS. The present study investigates the functional integrity of resting-state networks (RSNs) and their importance in ALS. Intra- and inter-network resting-state functional connectivity (Rs-FC) was examined using an independent component analysis approach in a large multi-center cohort. A total of 235 subjects (120 ALS patients; 115 healthy controls (HC) were recruited across North America through the Canadian ALS Neuroimaging Consortium (CALSNIC). Intra-network and inter-network Rs-FC was evaluated by the FSL-MELODIC and FSLNets software packages. As compared to HC, ALS patients displayed higher intra-network Rs-FC in the sensorimotor, default mode, right and left fronto-parietal, and orbitofrontal RSNs, and in previously undescribed networks including auditory, dorsal attention, basal ganglia, medial temporal, ventral streams, and cerebellum which negatively correlated with disease severity. Furthermore, ALS patients displayed higher inter-network Rs-FC between the orbitofrontal and basal ganglia RSNs which negatively correlated with cognitive impairment. In summary, in ALS there is an increase in intra- and inter-network functional connectivity of RSNs underpinning both motor and cognitive impairment. Moreover, the large multi-center CALSNIC dataset permitted the exploration of RSNs in unprecedented detail, revealing previously undescribed network involvement in ALS.

Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8-30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants' mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.

Dopaminergic Modulation of Dynamic Emotion Perception

Emotion recognition abilities are fundamental to our everyday social interaction. A large number of clinical populations show impairments in this domain, with emotion recognition atypicalities being particularly prevalent among disorders exhibiting a dopamine system disruption (e.g., Parkinson's disease). Although this suggests a role for dopamine in emotion recognition, studies employing dopamine manipulation in healthy volunteers have exhibited mixed neural findings and no behavioral modulation. Interestingly, while a dependence of dopaminergic drug effects on individual baseline dopamine function has been well established in other cognitive domains, the emotion recognition literature so far has failed to account for these possible interindividual differences. The present within-subjects study therefore tested the effects of the dopamine D2 antagonist haloperidol on emotion recognition from dynamic, whole-body stimuli while accounting for interindividual differences in baseline dopamine. A total of 33 healthy male and female adults rated emotional point-light walkers (PLWs) once after ingestion of 2.5 mg haloperidol and once after placebo. To evaluate potential mechanistic pathways of the dopaminergic modulation of emotion recognition, participants also performed motoric and counting-based indices of temporal processing. Confirming our hypotheses, effects of haloperidol on emotion recognition depended on baseline dopamine function, where individuals with low baseline dopamine showed enhanced, and those with high baseline dopamine decreased emotion recognition. Drug effects on emotion recognition were related to drug effects on movement-based and explicit timing mechanisms, indicating possible mediating effects of temporal processing. Results highlight the need for future studies to account for baseline dopamine and suggest putative mechanisms underlying the dopaminergic modulation of emotion recognition.SIGNIFICANCE STATEMENT A high prevalence of emotion recognition difficulties among clinical conditions where the dopamine system is affected suggests an involvement of dopamine in emotion recognition processes. However, previous psychopharmacological studies seeking to confirm this role in healthy volunteers thus far have failed to establish whether dopamine affects emotion recognition and lack mechanistic insights. The present study uncovered effects of dopamine on emotion recognition in healthy individuals by controlling for interindividual differences in baseline dopamine function and investigated potential mechanistic pathways via which dopamine may modulate emotion recognition. Our findings suggest that dopamine may influence emotion recognition via its effects on temporal processing, providing new directions for future research on typical and atypical emotion recognition.

Trauma Disrupts Reinforcement Learning in Rats-A Novel Animal Model of Chronic Stress Exposure

Trauma, as well as chronic stress that characterizes a modern fast-paced lifestyle, contributes to numerous psychopathologies and psychological problems. Psychiatric patients with traumas, as well as healthy individuals who experienced traumas in the past, are often characterized by diminished cognitive abilities. In our protocol, we used an animal model to explore the influence of chronic trauma on cognitive abilities and behavior in the group of 20 rats (Rattus norvegicus). The experimental group was introduced to chronic (12 consecutive days) exposure to predator odor (bobcat urine). We measured the reinforcement learning of each individual before and after the exposition via the Probabilistic Selection Task (PST) and we used Social Interaction Test (SIT) to assess the behavioral changes of each individual before and after the trauma. In the experimental group, there was a significant decrease in reinforcement learning after exposure to a single trauma (Wilcoxon Test, p = 0.034) as well as after 11 days of chronic trauma (Wilcoxon-test, p = 0.01) in comparison to pre-trauma performance. The control group, which was not exposed to predator odor but underwent the same testing protocol, did not present significant deterioration in reinforcement learning. In cross-group comparisons, there was no difference between the experimental and control group in PST before odor protocol (U Mann-Whitney two-sided, p = 0.909). After exposure to chronic trauma, the experimental group deteriorated in PST performance compared to control (U Mann-Whitney Two-sided, p = 0.0005). In SIT, the experimental group spent less time in an Interaction Zone with an unfamiliar rat after trauma protocol (Wilcoxon two-sided test, p = 0.019). Major strengths of our models are: (1) protocol allows investigating reinforcement learning before and after exposition to chronic trauma, with the same group of rats, (2) translational scope, as the PST is displayed on touchscreen, similarly to human studies, (3) protocol delivers chronic trauma that impairs reward learning, but behaviorally does not induce full-blown anhedonia, thus rats performed voluntarily throughout all the procedures.

Damage to the dorsolateral prefrontal cortex is associated with repetitive compulsive behaviors in patients with penetrating brain injury

Background

Damage to cortico-striato-thalamo-cortical (CSTC) circuits is associated with the development of repetitive behaviours in animals and humans. However, the types of repetitive behaviours that are developed after injury to these structures are poorly defined. This study examines the effect of damage to separate elements of CSTC circuits sustained by veterans of the Vietnam War on obsessions, compulsions, and tics.

Methods

We performed partial correlations (correcting for cognition, age, education, and global brain damage) between volume loss from traumatic brain injury in specific elements of CSTC circuits (lateral and medial orbitofrontal and dorsolateral prefrontal cortices, anterior cingulate cortex, thalamus, and basal ganglia) and scores on a modified version of the Yale-Brown Obsessive Compulsive Scale Symptom Checklist and the Yale Global Tic Severity Scale in 83 Vietnam war veterans with penetrating brain injuries at different sites throughout the brain.

Results

We found that volume loss in the left dorsolateral prefrontal cortex was associated with the development of compulsive behaviours (r=0.32, padj<0.05) whereas volume loss in the basal ganglia was associated with the development of tics (r=0.33, padj<0.05).

Conclusion

Our findings indicate that damage to specific CSTC elements can be associated with the development of compulsive behaviours and tics that are not necessarily accompanied by obsessions.

Bright light exposure during simulated night work improves cognitive flexibility

Night work leads to sleepiness and reduced vigilant attention during work hours, and bright light interventions may reduce such effects. It is also known that total sleep deprivation impairs cognitive flexibility as measured by reversal learning tasks. Whether night work impairs reversal learning task performance or if bright light can mitigate reversal learning deficits during night work is unclear. In this counterbalanced crossover study (ClinicaTrials.gov Identifier NCT03203538), young healthy individuals completed a reversal learning task twice during each of three consecutive simulated night shifts (23:00-07:00 h). The night shifts were performed in a laboratory under a full-spectrum (4000 K) bright light (~900 lx) and a standard light (~90 lx) condition. Reversal learning task performance was reduced towards the end of the night shifts (04:50 h), compared to the first part of the night shifts (00:20 h) in both light conditions. However, with bright light, the reversal learning task performance improved towards the end of the night shifts, compared to standard light. The study shows that bright light may mitigate performance deficits on a reversal learning task during night work and implies that bright light interventions during night work may be beneficial not only for vigilant attention but also for cognitive flexibility.

Childhood adversity and impaired reward processing: A meta-analysis

BACKGROUND

Childhood adversity (CA) is associated with increased risk of psychopathology, and reward processing (RP) may be one of the underlying mechanisms. However, evidence on impaired RP in childhood adversity is theoretically and methodologically heterogeneous.

OBJECTIVE

To provide a quantitative overview of studies on the relation between childhood adversity and RP assessed at the behavioral and subjective levels, and identify differences between studies that influence the effect size.

PARTICIPANTS AND SETTING

Twenty-seven studies (overall N = 6801) were included.

METHODS

Peer-reviewed publications describing empirical studies on the relation between CA and behavioral and self-report measures of RP in humans were identified through systematic searches in six bibliographic databases. Effect sizes (r) were pooled using random-effects models. The potential moderator role of RP dimension, type of RP assessment, type of childhood adversity assessment, and age were examined.

RESULTS

Results indicated a small, but consistent association between CA and impaired RP (r = 0.12; 95% CI: 0.07, 0.16), with medium heterogeneity (I² = 62.43). The effect size was significantly larger (i.e., medium-sized) in studies that focused on reward learning rather than reward valuation and reward responsiveness; used cognitive tasks rather than self-report assessments of RP; and relied on official records rather than subjective reports of CA. There was evidence of publication bias, but overall effect size remained significant after imputation.

CONCLUSIONS

These results suggest that multidimensional RP impairments (e.g., deficits in reward learning, biased reward valuation) are a consistent marker of CA, and may represent mechanisms underlying the increased risk of psychopathology.

Phasic Dopamine Changes and Hebbian Mechanisms during Probabilistic Reversal Learning in Striatal Circuits: A Computational Study

Cognitive flexibility is essential to modify our behavior in a non-stationary environment and is often explored by reversal learning tasks. The basal ganglia (BG) dopaminergic system, under a top-down control of the pre-frontal cortex, is known to be involved in flexible action selection through reinforcement learning. However, how adaptive dopamine changes regulate this process and learning mechanisms for training the striatal synapses remain open questions. The current study uses a neurocomputational model of the BG, based on dopamine-dependent direct (Go) and indirect (NoGo) pathways, to investigate reinforcement learning in a probabilistic environment through a task that associates different stimuli to different actions. Here, we investigated: the efficacy of several versions of the Hebb rule, based on covariance between pre- and post-synaptic neurons, as well as the required control in phasic dopamine changes crucial to achieving a proper reversal learning. Furthermore, an original mechanism for modulating the phasic dopamine changes is proposed, assuming that the expected reward probability is coded by the activity of the winner Go neuron before a reward/punishment takes place. Simulations show that this original formulation for an automatic phasic dopamine control allows the achievement of a good flexible reversal even in difficult conditions. The current outcomes may contribute to understanding the mechanisms for active control of dopamine changes during flexible behavior. In perspective, it may be applied in neuropsychiatric or neurological disorders, such as Parkinson’s or schizophrenia, in which reinforcement learning is impaired.

Reduced learning bias towards the reward context in medication-naive first-episode schizophrenia patients

BACKGROUND

Reinforcement learning has been proposed to contribute to the development of amotivation in individuals with schizophrenia (SZ). Accumulating evidence suggests dysfunctional learning in individuals with SZ in Go/NoGo learning and expected value representation. However, previous findings might have been confounded by the effects of antipsychotic exposure. Moreover, reinforcement learning also rely on the learning context. Few studies have examined the learning performance in reward and loss-avoidance context separately in medication-naïve individuals with first-episode SZ. This study aimed to explore the behaviour profile of reinforcement learning performance in medication-naïve individuals with first-episode SZ, including the contextual performance, the Go/NoGo learning and the expected value representation performance.

METHODS

Twenty-nine medication-naïve individuals with first-episode SZ and 40 healthy controls (HCs) who have no significant difference in age and gender, completed the Gain and Loss Avoidance Task, a reinforcement learning task involving stimulus pairs presented in both the reward and loss-avoidance context. We assessed the group difference in accuracy in the reward and loss-avoidance context, the Go/NoGo learning and the expected value representation. The correlations between learning performance and the negative symptom severity were examined.

RESULTS

Individuals with SZ showed significantly lower accuracy when learning under the reward than the loss-avoidance context as compared to HCs. The accuracies under the reward context (90%win- 10%win) in the Acquisition phase was significantly and negatively correlated with the Scale for the Assessment of Negative Symptoms (SANS) avolition scores in individuals with SZ. On the other hand, individuals with SZ showed spared ability of Go/NoGo learning and expected value representation.

CONCLUSIONS

Despite our small sample size and relatively modest findings, our results suggest possible reduced learning bias towards reward context among medication-naïve individuals with first-episode SZ. The reward learning performance was correlated with amotivation symptoms. This finding may facilitate our understanding of the underlying mechanism of negative symptoms. Reinforcement learning performance under the reward context may be important to better predict and prevent the development of schizophrenia patients' negative symptom, especially amotivation.

Coarse-Grained Neural Network Model of the Basal Ganglia to Simulate Reinforcement Learning Tasks

Computational models of the basal ganglia (BG) provide a mechanistic account of different phenomena observed during reinforcement learning tasks performed by healthy individuals, as well as by patients with various nervous or mental disorders. The aim of the present work was to develop a BG model that could represent a good compromise between simplicity and completeness. Based on more complex (fine-grained neural network, FGNN) models, we developed a new (coarse-grained neural network, CGNN) model by replacing layers of neurons with single nodes that represent the collective behavior of a given layer while preserving the fundamental anatomical structures of BG. We then compared the functionality of both the FGNN and CGNN models with respect to several reinforcement learning tasks that are based on BG circuitry, such as the Probabilistic Selection Task, Probabilistic Reversal Learning Task and Instructed Probabilistic Selection Task. We showed that CGNN still has a functionality that mirrors the behavior of the most often used reinforcement learning tasks in human studies. The simplification of the CGNN model reduces its flexibility but improves the readability of the signal flow in comparison to more detailed FGNN models and, thus, can help to a greater extent in the translation between clinical neuroscience and computational modeling.

Reinforcement learning detuned in addiction: integrative and translational approaches

Suboptimal decision-making strategies have been proposed to contribute to the pathophysiology of addiction. Decision-making, however, arises from a collection of computational components that can independently influence behavior. Disruptions in these different components can lead to decision-making deficits that appear similar behaviorally, but differ at the computational, and likely the neurobiological, level. Here, we discuss recent studies that have used computational approaches to investigate the decision-making processes underlying addiction. Studies in animal models have found that value updating following positive, but not negative, outcomes is predictive of drug use, whereas value updating following negative, but not positive, outcomes is disrupted following drug self-administration. We contextualize these findings with studies on the circuit and biological mechanisms of decision-making to develop a framework for revealing the biobehavioral mechanisms of addiction.

Computational Psychiatry Needs Time and Context

Why has computational psychiatry yet to influence routine clinical practice? One reason may be that it has neglected context and temporal dynamics in the models of certain mental health problems. We develop three heuristics for estimating whether time and context are important to a mental health problem: Is it characterized by a core neurobiological mechanism? Does it follow a straightforward natural trajectory? And is intentional mental content peripheral to the problem? For many problems the answers are no, suggesting that modeling time and context is critical. We review computational psychiatry advances toward this end, including modeling state variation, using domain-specific stimuli, and interpreting differences in context. We discuss complementary network and complex systems approaches. Novel methods and unification with adjacent fields may inspire a new generation of computational psychiatry.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Impact of Perceived Stress and Immune Status on Decision-Making Abilities during COVID-19 Pandemic Lockdown

The ability to make risky decisions in stressful contexts has been largely investigated in experimental settings. We examined this ability during the first months of COVID-19 pandemic, when in Italy people were exposed to a prolonged stress condition, mainly caused by a rigid lockdown. Participants among the general population completed two cognitive tasks, an Iowa Gambling Task (IGT), which measures individual risk/reward decision-making tendencies, and a Go/No-Go task (GNG), to test impulsivity, together with two questionnaires, the Perceived Stress Scale and the Depression, Anxiety and Stress Scales. The Immune Status Questionnaire was additionally administered to explore the impact of the individual health status on decision making. The effect of the questionnaires scores on task performance was examined. The results showed that higher levels of perceived stress and a more self-reported vulnerable immune status were associated, separately, with less risky/more advantageous choices in the IGT in young male participants but with more risky/less advantageous choices in older male participants. These effects were not found in female participants. Impulsivity errors in the GNG were associated with more anxiety symptoms. These findings bring attention to the necessity of taking into account decision-making processes during stressful conditions, especially in the older and more physically vulnerable male population.

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience

Reinforcement learning (RL) is a concept that has been invaluable to fields including machine learning, neuroscience, and cognitive science. However, what RL entails differs between fields, leading to difficulties when interpreting and translating findings. After laying out these differences, this paper focuses on cognitive (neuro)science to discuss how we as a field might over-interpret RL modeling results. We too often assume-implicitly-that modeling results generalize between tasks, models, and participant populations, despite negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique (neuro)cognitive processes, a concept we call interpretability, when evidence suggests that they capture different functions across studies and tasks. We conclude that future computational research needs to pay increased attention to implicit assumptions when using RL models, and suggest that a more systematic understanding of contextual factors will help address issues and improve the ability of RL to explain brain and behavior.

Transition to substance use disorders: impulsivity for reward and learning from reward

Substance dependence constitutes a profound societal burden. Although large numbers of individuals use licit or illicit substances, few transition to dependence. The specific factors influencing this transition are not well understood. Substance-dependent individuals tend to be swayed by the immediate rewards of drug taking, but are often insensitive to delayed negative consequences of their behavior. Dependence is consequently associated with impulsivity for reward and atypical learning from feedback. Behavioral impulsivity is indexed using tasks measuring spontaneous decision-making and capacity to control impulses. While evidence indicates drug taking exacerbates behavioral impulsivity for reward, animal and human studies of drug naïve populations demonstrate it might precede any drug-related problems. Research suggests dependent individuals are also more likely to learn from rewarding (relative to punishing) feedback. This may partly explain why substance-dependent individuals fail to modify their behavior in response to negative outcomes. This enhanced learning from reward may constitute a further pre-existing risk factor for substance dependence. Although impulsivity for reward and preferential learning from rewarding feedback are both underpinned by a compromised dopaminergic system, few studies have examined the relationship between these two mechanisms. The interplay of these processes may help enrich understanding of why some individuals transition to substance dependence.

Neuropsychological spectrum in early PD: Insights from controlled and automatic behavioural regulation

Initial changes in Parkinson's disease (PD) are marked by loss of automatic movements and decline of some cognitive functions. Yet, the exact profile and extent of cognitive impairments in early stages of PD as well as their mechanisms related to automatic motor dysfunction remain unclear. Our objective was to examine the neuropsychological changes in early PD and their association to automatic and controlled modes of behavioural control. Significant relationships between early PD and cognitive dysfunction in set-shifting, abstraction ability/concept formation, processing speed, visuospatial/constructional abilities and verbal-visual memory was found. We also noted that tests with a strong effortful and controlled component were similarly affected as automatic tests by early PD, particularly those testing verbal memory, processing speed and visuospatial/constructional functions. Our findings indicate that initial stages of PD sets constraints over most of the cognitive domains normally assessed and are not easily explained in terms of either automatic or controlled mechanisms, as both appear similarly altered in early PD.

Effects of cocaine and HIV on decision-making abilities

Our study aimed to understand the impact of cocaine dependence on high-risk decision-making abilities in individuals with the human immunodeficiency virus (HIV) and individuals with cocaine dependence. We recruited 99 participants (27 HIV/Cocaine, 20 HIV Only, 26 Cocaine Only, and 26 Healthy Controls). The Iowa Gambling Task (IGT) was applied to assess decision-making abilities. Independent and interactive effects of HIV status and cocaine dependence were examined using 2 × 2 factorial ANCOVA with premorbid IQ (WRAT-4: WR) as the covariate. We found cocaine dependence had a significant adverse effect on overall IGT performance (p = 0.015). We also found individuals who were HIV-positive tended to have less total money at the end of the game than individuals who were HIV-negative (p = 0.032), suggesting individuals living with HIV had less focus on long-term gains and more focus on short-term gains. Our findings highlight the significant impact of cocaine dependence on decision-making abilities and the difficulty individuals with HIV have in adequately weighing the cost and benefits of their decisions and making appropriate changes for the future.

Stochasticity, Nonlinear Value Functions, and Update Rules in Learning Aesthetic Biases

A theoretical framework for the reinforcement learning of aesthetic biases was recently proposed based on brain circuitries revealed by neuroimaging. A model grounded on that framework accounted for interesting features of human aesthetic biases. These features included individuality, cultural predispositions, stochastic dynamics of learning and aesthetic biases, and the peak-shift effect. However, despite the success in explaining these features, a potential weakness was the linearity of the value function used to predict reward. This linearity meant that the learning process employed a value function that assumed a linear relationship between reward and sensory stimuli. Linearity is common in reinforcement learning in neuroscience. However, linearity can be problematic because neural mechanisms and the dependence of reward on sensory stimuli were typically nonlinear. Here, we analyze the learning performance with models including optimal nonlinear value functions. We also compare updating the free parameters of the value functions with the delta rule, which neuroscience models use frequently, vs. updating with a new Phi rule that considers the structure of the nonlinearities. Our computer simulations showed that optimal nonlinear value functions resulted in improvements of learning errors when the reward models were nonlinear. Similarly, the new Phi rule led to improvements in these errors. These improvements were accompanied by the straightening of the trajectories of the vector of free parameters in its phase space. This straightening meant that the process became more efficient in learning the prediction of reward. Surprisingly, however, this improved efficiency had a complex relationship with the rate of learning. Finally, the stochasticity arising from the probabilistic sampling of sensory stimuli, rewards, and motivations helped the learning process narrow the range of free parameters to nearly optimal outcomes. Therefore, we suggest that value functions and update rules optimized for social and ecological constraints are ideal for learning aesthetic biases.

Reliability and Replicability of Implicit and Explicit Reinforcement Learning Paradigms in People With Psychotic Disorders

BACKGROUND

Motivational deficits in people with psychosis may be a result of impairments in reinforcement learning (RL). Therefore, behavioral paradigms that can accurately measure these impairments and their change over time are essential.

METHODS

We examined the reliability and replicability of 2 RL paradigms (1 implicit and 1 explicit, each with positive and negative reinforcement components) given at 2 time points to healthy controls (n = 75), and people with bipolar disorder (n = 62), schizoaffective disorder (n = 60), and schizophrenia (n = 68).

RESULTS

Internal consistency was acceptable (mean α = 0.78 ± 0.15), but test-retest reliability was fair to low (mean intraclass correlation = 0.33 ± 0.25) for both implicit and explicit RL. There were no clear effects of practice for these tasks. Largely, performance on these tasks shows intact implicit and impaired explicit RL in psychosis. Symptom presentation did not relate to performance in any robust way.

CONCLUSIONS

Our findings replicate previous literature showing spared implicit RL and impaired explicit reinforcement in psychosis. This suggests typical basal ganglia dopamine release, but atypical recruitment of the orbitofrontal and dorsolateral prefrontal cortices. However, we found that these tasks have only fair to low test-retest reliability and thus may not be useful for assessing change over time in clinical trials.

Unlocking the reinforcement-learning circuits of the orbitofrontal cortex

Neuroimaging studies have consistently identified the orbitofrontal cortex (OFC) as being affected in individuals with neuropsychiatric disorders. OFC dysfunction has been proposed to be a key mechanism by which decision-making impairments emerge in diverse clinical populations, and recent studies employing computational approaches have revealed that distinct reinforcement-learning mechanisms of decision-making differ among diagnoses. In this perspective, we propose that these computational differences may be linked to select OFC circuits and present our recent work that has used a neurocomputational approach to understand the biobehavioral mechanisms of addiction pathology in rodent models. We describe how combining translationally analogous behavioral paradigms with reinforcement-learning algorithms and sophisticated neuroscience techniques in animals can provide critical insights into OFC pathology in biobehavioral disorders. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Lower neural value signaling in the prefrontal cortex is related to childhood family income and depressive symptomatology during adolescence

Lower family income during childhood is related to increased rates of adolescent depression, though the underlying mechanisms are poorly understood. Evidence suggests that individuals with depression demonstrate hypoactivation in brain regions involved in reward learning and decision-making processes (e.g., portions of the prefrontal cortex). Separately, lower family income has been associated with neural alterations in similar regions. Motivated by this research, we examined associations between family income, depression, and brain activity during a reward learning and decision-making fMRI task in a sample of adolescents (full n = 94; usable n = 78; mean age = 15.2 years). We focused on brain activity for: 1) expected value (EV), the learned subjective value of an object, and 2) prediction error, the difference between EV and the actual outcome received. Regions of interest related to reward learning were examined in connection to childhood family income and parent-reported adolescent depressive symptoms. As hypothesized, lower activity in the subgenual anterior cingulate (sACC) for EV in response to approach stimuli was associated with lower childhood family income, as well as greater symptoms of depression measured one-year after the neuroimaging session. These results are consistent with the hypothesis that lower early family income leads to disruptions in reward and decision-making brain circuitry, contributing to adolescent depression.

Reward Learning Through the Lens of RDoC: a Review of Theory, Assessment, and Empirical Findings in the Eating Disorders

PURPOSE OF REVIEW

Reward-related processes may represent important transdiagnostic factors underlying eating pathology. Using the NIMH Research Domain Criteria as a guide, the current article reviews theories, behavioral and self-report assessments, and empirical findings related to reward learning in the eating disorders.

RECENT FINDINGS

Data from behavioral tasks suggest deficits in reinforcement learning, which may become more pronounced with increasing disorder severity and duration. Self-report data strongly implicate positive eating and thinness/restriction expectancies (an element of reward prediction error) in the onset and maintenance of eating pathology. Finally, self-report measures of habit strength demonstrate relationships with eating pathology and illness duration; however, behavioral task data do not support relationships between eating pathology and a propensity towards general habit development. Existing studies are limited, but provide preliminary support for the presence of abnormal reward learning in eating disorders. Continued research is needed to address identified gaps in the literature.

Effects of methylphenidate on reinforcement learning depend on working memory capacity

RATIONALE

Brain catecholamines have long been implicated in reinforcement learning, exemplified by catecholamine drug and genetic effects on probabilistic reversal learning. However, the mechanisms underlying such effects are unclear.

OBJECTIVES AND METHODS

Here we investigated effects of an acute catecholamine challenge with methylphenidate (20 mg, oral) on a novel probabilistic reversal learning paradigm in a within-subject, double-blind randomised design. The paradigm was designed to disentangle effects on punishment avoidance from effects on reward perseveration. Given the known large individual variability in methylphenidate's effects, we stratified our effects by working memory capacity and trait impulsivity, putatively modulating the effects of methylphenidate, in a large sample (n = 102) of healthy volunteers.

RESULTS

Contrary to our prediction, methylphenidate did not alter performance in the reversal phase of the task. Our key finding is that methylphenidate altered learning of choice-outcome contingencies in a manner that depended on individual variability in working memory span. Specifically, methylphenidate improved performance by adaptively reducing the effective learning rate in participants with higher working memory capacity.

CONCLUSIONS

This finding emphasises the important role of working memory in reinforcement learning, as reported in influential recent computational modelling and behavioural work, and highlights the dependence of this interplay on catecholaminergic function.

Using supervised machine learning on neuropsychological data to distinguish OCD patients with and without sensory phenomena from healthy controls

OBJECTIVES

While theoretical models link obsessive-compulsive disorder (OCD) with executive function deficits, empirical findings from the neuropsychological literature remain mixed. These inconsistencies are likely exacerbated by the challenge of high-dimensional data (i.e., many variables per subject), which is common across neuropsychological paradigms and necessitates analytical advances. More unique to OCD is the heterogeneity of symptom presentations, each of which may relate to distinct neuropsychological features. While researchers have traditionally attempted to account for this heterogeneity using a symptom-based approach, an alternative involves focusing on underlying symptom motivations. Although the most studied symptom motivation involves fear of harmful events, 60-70% of patients also experience sensory phenomena, consisting of uncomfortable sensations or perceptions that drive compulsions. Sensory phenomena have received limited attention in the neuropsychological literature, despite evidence that symptoms motivated by these experiences may relate to distinct cognitive processes.

METHODS

Here, we used a supervised machine learning approach to characterize neuropsychological processes in OCD, accounting for sensory phenomena.

RESULTS

Compared to logistic regression and other algorithms, random forest best differentiated healthy controls (n = 59; balanced accuracy = .70), patients with sensory phenomena (n = 29; balanced accuracy = .59), and patients without sensory phenomena (n = 46; balanced accuracy = .62). Decision-making best distinguished between groups based on sensory phenomena, and among the patient subsample, those without sensory phenomena uniquely displayed greater risk sensitivity compared to healthy controls (d = .07, p = .008).

CONCLUSIONS

Results suggest that different cognitive profiles may characterize patients motivated by distinct drives. The superior performance and generalizability of the newer algorithms highlights the utility of considering multiple analytic approaches when faced with complex data.

PRACTITIONER POINTS

Practitioners should be aware that sensory phenomena are common experiences among patients with OCD. OCD patients with sensory phenomena may be distinguished from those without based on neuropsychological processes.

Episodic memory governs choices: An RNN-based reinforcement learning model for decision-making task

Typical methods to study cognitive function are to record the electrical activities of animal neurons during the training of animals performing behavioral tasks. A key problem is that they fail to record all the relevant neurons in the animal brain. To alleviate this problem, we develop an RNN-based Actor-Critic framework, which is trained through reinforcement learning (RL) to solve two tasks analogous to the monkeys' decision-making tasks. The trained model is capable of reproducing some features of neural activities recorded from animal brain, or some behavior properties exhibited in animal experiments, suggesting that it can serve as a computational platform to explore other cognitive functions. Furthermore, we conduct behavioral experiments on our framework, trying to explore an open question in neuroscience: which episodic memory in the hippocampus should be selected to ultimately govern future decisions. We find that the retrieval of salient events sampled from episodic memories can effectively shorten deliberation time than common events in the decision-making process. The results indicate that salient events stored in the hippocampus could be prioritized to propagate reward information, and thus allow decision-makers to learn a strategy faster.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Incentive-driven decision-making networks in de novo and drug-treated Parkinson's disease patients with impulsive-compulsive behaviors: A systematic review of neuroimaging studies

BACKGROUND

In Parkinson's disease (PD), impulsive-compulsive behaviors (ICBs) may develop as side-effect of dopaminergic medications. Abnormal incentive-driven decision-making, which is supported by the cognitive control and motivation interaction, may represent an ICBs signature. This systematic review explored whether structural and/or functional brain differences between PD patients with vs without ICBs encompass incentive-driven decision-making networks.

METHODS

Structural and functional neuroimaging studies comparing PD patients with and without ICBs, either de novo or medicated, were included.

RESULTS

Thirty articles were identified. No consistent evidence of structural alteration both in de novo and medicated PD patients were found. Differences in connectivity within the default mode, the salience and the central executive networks predate ICBs development and remain stable once ICBs are fully developed. Medicated PD patients with ICBs show increased metabolism and cerebral blood flow in orbitofrontal and cingulate cortices, ventral striatum, amygdala, insula, temporal and supramarginal gyri. Abnormal ventral striatum connectivity with anterior cingulate cortex and limbic structures was reported in PD patients with ICBs.

DISCUSSION

Functional brain signatures of ICBs in PD encompass areas involved in cognitive control and motivational encoding networks of the incentive-driven decision-making. Functional alterations predating ICBs may be related to abnormal synaptic plasticity in these networks.

Impaired decision-making under risk in patients with functional dyspepsia

INTRODUCTION

The cognitive processing in patients with functional dyspepsia (FD) has not been well established. Decision-making is an important component of cognitive function. Most brain regions involved in decision-making are abnormal in FD patients. This study aimed to investigate the decision-making under ambiguity and risk in FD patients.

METHODS

We recruited 40 FD patients meeting Rome III criteria and 40 healthy controls (HCs) matched for age, sex, marital status, and education level. The Hamilton Anxiety Scale (HAMA) and the 17-item Hamilton Depression Scale (HAMD-17) were used to evaluate their anxiety and depression emotions. The Iowa Gambling Task (IGT) and Game of Dice Task (GDT) were used to evaluate decision-making under ambiguity and risk, respectively. Helicobacter pylori status, disease duration, dyspeptic symptom score, and the Nepean Dyspepsia Life Quality Index (NDLQI) were obtained from all patients.

RESULTS

In IGT, FD patients had a lower total net score, chose more adverse choices, and showed a slower response to change their behavior than HCs. However, there was no significant difference in the net score of the first 2 blocks between the two groups. In GDT, FD patients had a lower total net score, higher risk score, and lower use of negative feedback than HCs. In addition, FD patients showed better GDT performance than those without early satiation.

CONCLUSIONS

FD patients showed impaired decision-making under risk. The deficiency might be related to dyspeptic symptoms of FD patients.

DRD4 polymorphisms modulate reward positivity and P3a in a gambling task: Exploring a genetic basis for cultural learning

Prior work shows that people respond more plastically to environmental influences, including cultural influences, if they carry the 7 or 2-repeat (7/2R) allelic variant of the dopamine D4 receptor gene (DRD4). The 7/2R carriers are thus more likely to endorse the norms and values of their culture. So far, however, mechanisms underlying this moderation of cultural acquisition by DRD4 are unclear. To address this gap in knowledge, we tested the hypothesis that DRD4 modulates the processing of reward cues existing in the environment. About 72 young adults, preselected for their DRD4 status, performed a gambling task, while the electroencephalogram was recorded. Principal components of event-related potentials aligned to the Reward-Positivity (associated with bottom-up processing of reward prediction errors) and frontal-P3 (associated with top-down attention) were both significantly more positive following gains than following losses. As predicted, the gain-loss differences were significantly larger for 7/2R carriers than for noncarriers. Also, as predicted, the cultural backgrounds of the participants (East Asian vs. European American) did not moderate the effects of DRD4. Our findings suggest that the 7/2R variant of DRD4 enhances (a) the detection of reward prediction errors and (b) controlled attention that updates the context for the reward, thereby suggesting one possible mechanism underlying the DRD4 × Culture interactions.

A neural network model of basal ganglia's decision-making circuitry

The basal ganglia have been increasingly recognized as an important structure involved in decision making. Neurons in the basal ganglia were found to reflect the evidence accumulation process during decision making. However, it is not well understood how the direct and indirect pathways of the basal ganglia work together for decision making. Here, we create a recurrent neural network model that is composed of the direct and indirect pathways and test it with the classic random dot motion discrimination task. The direct pathway drives the outputs, which are modulated through a gating mechanism controlled by the indirect pathway. We train the network to learn the task and find that the network reproduces the accuracy and reaction time patterns of previous animal studies. Units in the model exhibit ramping activities that reflect evidence accumulation. Finally, we simulate manipulations of the direct and indirect pathways and find that the manipulations of the direct pathway mainly affect the choice while the manipulations of the indirect pathway affect the model's reaction time. These results suggest a potential circuitry mechanism of the basal ganglia's role in decision making with predictions that can be tested experimentally in the future.