Using TD learning to simulate working memory performance in a model of the prefrontal cortex and basal ganglia

Value-based cognition and drug dependency

Value-based decision-making is thought to play an important role in drug dependency. Achieving elevated levels of euphoria or ameliorating dysphoria/pain may motivate goal-directed drug consumption in both drug-naïve and long-time users. In other words, drugs become viewed as the preferred means of attaining a desired internal state. The bias towards choosing drugs may affect one's cognition. Observed biases in learning, attention and memory systems within the brain gradually focus one's cognitive functions towards drugs and related cues to the exclusion of other stimuli. In this narrative review, the effects of drug use on learning, attention and memory are discussed with a particular focus on changes across brain-wide functional networks and the subsequent impact on behaviour. These cognitive changes are then incorporated into the cycle of addiction, an established model outlining the transition from casual drug use to chronic dependency. If drug use results in the elevated salience of drugs and their cues, the studies highlighted in this review strongly suggest that this salience biases cognitive systems towards the motivated pursuit of addictive drugs. This bias is observed throughout the cycle of addiction, possibly contributing to the persistent hold that addictive drugs have over the dependent. Taken together, the excessive valuation of drugs as the preferred means of achieving a desired internal state affects more than just decision-making, but also learning, attentional and mnemonic systems. This eventually narrows the focus of one's thoughts towards the pursuit and consumption of addictive drugs.

Value, drug addiction and the brain

Over the years, various models have been proposed to explain the psychology and biology of drug addiction, built primarily around the habit and compulsion models. Recent research indicates drug addiction may be goal-directed, motivated by excessive valuation of drugs. Drug consumption may initially occur for the sake of pleasure but may transition to a means of escaping withdrawal, stress and negative emotions. In this hypothetical paper, we propose a value-based neurobiological model for drug addiction. We posit that during dependency, the value-based decision-making system in the brain is not inactive but has instead prioritized drugs as the reward of choice. In support of this model, we consider the role of valuation in choice, its influence on pleasure and punishment, and how valuation is contrasted in impulsive and compulsive behaviours. We then discuss the neurobiology of value, beginning with the dopaminergic system and its relationship with incentive salience before moving to brain-wide networks involved in valuation, control and prospection. These value-based neurobiological components are then integrated into the cycle of addiction as we consider the development of drug dependency from a valuation perspective. We conclude with a discussion of cognitive interventions utilizing value-based decision-making, highlighting not just advances in recalibrating the valuation system to focus on non-drug rewards, but also areas for improvement in refining this approach.

Mathematics anxiety and cognition: an integrated neural network model

Many students suffer from anxiety when performing numerical calculations. Mathematics anxiety is a condition that has a negative effect on educational outcomes and future employment prospects. While there are a multitude of behavioral studies on mathematics anxiety, its underlying cognitive and neural mechanism remain unclear. This article provides a systematic review of cognitive studies that investigated mathematics anxiety. As there are no prior neural network models of mathematics anxiety, this article discusses how previous neural network models of mathematical cognition could be adapted to simulate the neural and behavioral studies of mathematics anxiety. In other words, here we provide a novel integrative network theory on the links between mathematics anxiety, cognition, and brain substrates. This theoretical framework may explain the impact of mathematics anxiety on a range of cognitive and neuropsychological tests. Therefore, it could improve our understanding of the cognitive and neurological mechanisms underlying mathematics anxiety and also has important applications. Indeed, a better understanding of mathematics anxiety could inform more effective therapeutic techniques that in turn could lead to significant improvements in educational outcomes.

Effect of fine and gross motor training or motor imagery, delivered via novel or routine modes, on cognitive function

There is extensive literature linking motor activity to cognitive effects at various stages in life, promoting both development and the reduction of aging associated pathologies. It is unclear whether the benefits of this activity on the cognitive level are associated with brain functions that are necessary for their performance or recurrence of activity or type of activity itself. The aim of this study was to evaluate whether the type of motor activity (fine, gross, and motor imagery) in two modes (novel and routine) can affect cognitive functions such as attention, executive functions, and praxis in college students. A 2 × 3 factorial design with repeated measures was used without a control group and pre- and post-training evaluation. Fifty-three young people (14 men and 39 women) participated, with mean age of 18.94 years (SD = 1.61 years) and were divided into six groups. Each of the groups performed relevant training 20 minutes per day for five days depending on the group. Measures were taken pre and post-training for attention tests, attention span, working memory, visual constructive skills, procedural memory, and motor skills. The results show a "learning effect" from the exposure to the tests in measurements after training. It was also found that between groups, there is a difference in some of the variables of procedural memory (number of errors) and working memory. More extensive training could better reflect the effects of the training, and longitudinal evaluation could show the rate of change of functions. The main clinical implication could be the evaluation of training programs for recovery and motor training in cerebral plasticity having effect on the cognitive aspects.

Interrelations between cognitive dysfunction and motor symptoms of Parkinson's disease: behavioral and neural studies

Parkinson's disease (PD) is characterized by a range of motor symptoms. Besides the cardinal symptoms (tremor, bradykinesia/akinesia, and rigidity), PD patients also show other motor deficits, including gait disturbance, speech deficits, and impaired handwriting. However, along with these key motor symptoms, PD patients also experience cognitive deficits in attention, executive function, working memory, and learning. Recent evidence suggests that these motor and cognitive deficits of PD are not completely dissociable, as aspects of cognitive dysfunction can impact motor performance in PD. In this article, we provide a review of behavioral and neural studies on the associations between motor symptoms and cognitive deficits in PD, specifically akinesia/bradykinesia, tremor, gait, handwriting, precision grip, and speech production. This review paves the way for providing a framework for understanding how treatment of cognitive dysfunction, for example cognitive rehabilitation programs, may in turn influence the motor symptoms of PD.

The roles of valuation and reward processing in cognitive function and psychiatric disorders

In neuroeconomics, valuation refers to the process of assigning values to states and actions on the basis of the animal's current representation of the environment, while reward processing corresponds to processing the feedback received from the environment to update the values of states and actions. In this article, we review the brain circuits associated with valuation and reward processing and argue that these are fundamental processes critical to many cognitive functions. Specifically, we focus on the role of valuation and reward processing in attention, memory, decision making, and learning. Next, the extant neuroimaging literature on a number of psychiatric disorders is reviewed (i.e., addiction, pathological gambling, schizophrenia, and mood disorders), and an argument is made that associated deficits in cognitive functions can be explained in terms of abnormal valuation and reward processing. The review concludes with the impact of this framework in clinical settings and prescriptions for future research, in particular with regard to the conversions of qualitatively different valuation systems into a system of common currency.

Generalized Anxiety Disorder and Social Anxiety Disorder, but Not Panic Anxiety Disorder, Are Associated with Higher Sensitivity to Learning from Negative Feedback: Behavioral and Computational Investigation

Anxiety disorders, including generalized anxiety disorder (GAD), social anxiety disorder (SAD), and panic anxiety disorder (PAD), are a group of common psychiatric conditions. They are characterized by excessive worrying, uneasiness, and fear of future events, such that they affect social and occupational functioning. Anxiety disorders can alter behavior and cognition as well, yet little is known about the particular domains they affect. In this study, we tested the cognitive correlates of medication-free patients with GAD, SAD, and PAD, along with matched healthy participants using a probabilistic category-learning task that allows the dissociation between positive and negative feedback learning. We also fitted all participants' data to a Q-learning model and various actor-critic models that examine learning rate parameters from positive and negative feedback to investigate effects of valence vs. action on performance. SAD and GAD patients were more sensitive to negative feedback than either PAD patients or healthy participants. PAD, SAD, and GAD patients did not differ in positive-feedback learning compared to healthy participants. We found that Q-learning models provide the simplest fit of the data in comparison to other models. However, computational analysis revealed that groups did not differ in terms of learning rate or exploration values. These findings argue that (a) not all anxiety spectrum disorders share similar cognitive correlates, but are rather different in ways that do not link them to the hallmark of anxiety (higher sensitivity to negative feedback); and (b) perception of negative consequences is the core feature of GAD and SAD, but not PAD. Further research is needed to examine the similarities and differences between anxiety spectrum disorders in other cognitive domains and potential implementation of behavioral therapy to remediate cognitive deficits.

The effects of clinical motor variables and medication dosage on working memory in Parkinson's disease

In this study, we investigate the interrelationship between clinical variables and working memory (WM) in Parkinson's disease (PD). Specifically, the aim of the study was to investigate the relationship between disease duration, dopaminergic medication dosage, and motor disability (UPDRS score) with WM in individuals with PD. Accordingly, we recruited three groups of subjects: unmedicated PD patients, medicated PD patients, and healthy controls. All subjects were tested on three WM tasks: short-delay WM, long-delay WM, and the n-back task. Further, PD encompasses a spectrum that can be classified either into akinesia/rigidity or resting tremor as the predominant motor presentation of the disease. In addition to studying medication effects, we tested WM performance in tremor-dominant and akinesia-dominant patients. We further correlated WM performance with disease duration and medication dosage. We found no difference between medicated and unmedicated patients in the short-delay WM task, but medicated patients outperformed unmedicated patients in the long-delay WM and n-back tasks. Interestingly, we also found that akinesia-dominant patients were more impaired than tremor-dominant patients at various WM measures, which is in agreement with prior studies of the relationship between akinesia symptom and basal ganglia dysfunction. Moreover, the results show that disease duration inversely correlates with more demanding WM tasks (long-delay WM and n-back tasks), but medication dosage positively correlates with demanding WM performance. In sum, our results show that WM impairment in PD patients depend on cognitive domain (simple vs. demanding WM task), subtype of PD patients (tremor- vs. akinesia-dominant), as well as disease duration and medication dosage. Our results have implications for the interrelationship between motor and cognitive processes in PD, and for understanding the role of cognitive training in treating motor symptoms in PD.

A model of amygdala-hippocampal-prefrontal interaction in fear conditioning and extinction in animals

Empirical research has shown that the amygdala, hippocampus, and ventromedial prefrontal cortex (vmPFC) are involved in fear conditioning. However, the functional contribution of each brain area and the nature of their interactions are not clearly understood. Here, we extend existing neural network models of the functional roles of the hippocampus in classical conditioning to include interactions with the amygdala and prefrontal cortex. We apply the model to fear conditioning, in which animals learn physiological (e.g. heart rate) and behavioral (e.g. freezing) responses to stimuli that have been paired with a highly aversive event (e.g. electrical shock). The key feature of our model is that learning of these conditioned responses in the central nucleus of the amygdala is modulated by two separate processes, one from basolateral amygdala and signaling a positive prediction error, and one from the vmPFC, via the intercalated cells of the amygdala, and signaling a negative prediction error. In addition, we propose that hippocampal input to both vmPFC and basolateral amygdala is essential for contextual modulation of fear acquisition and extinction. The model is sufficient to account for a body of data from various animal fear conditioning paradigms, including acquisition, extinction, reacquisition, and context specificity effects. Consistent with studies on lesioned animals, our model shows that damage to the vmPFC impairs extinction, while damage to the hippocampus impairs extinction in a different context (e.g., a different conditioning chamber from that used in initial training in animal experiments). We also discuss model limitations and predictions, including the effects of number of training trials on fear conditioning.

Dissociating the cognitive effects of levodopa versus dopamine agonists in a neurocomputational model of learning in Parkinson's disease

BACKGROUND/AIMS

Levodopa and dopamine agonists have different effects on the motor, cognitive, and psychiatric aspects of Parkinson's disease (PD).

METHODS

Using a computational model of basal ganglia (BG) and prefrontal cortex (PFC) dopamine, we provide a theoretical synthesis of the dissociable effects of these dopaminergic medications on brain and cognition. Our model incorporates the findings that levodopa is converted by dopamine cells into dopamine, and thus activates prefrontal and striatal D(1) and D(2) dopamine receptors, whereas antiparkinsonian dopamine agonists directly stimulate D(2) receptors in the BG and PFC (although some have weak affinity to D(1) receptors).

RESULTS

In agreement with prior neuropsychological studies, our model explains how levodopa enhances, but dopamine agonists impair or have no effect on, stimulus-response learning and working memory.

CONCLUSION

Our model explains how levodopa and dopamine agonists have differential effects on motor and cognitive processes in PD.

Computational cognitive models of prefrontal-striatal-hippocampal interactions in Parkinson's disease and schizophrenia

Disruption to different components of the prefrontal cortex, basal ganglia, and hippocampal circuits leads to various psychiatric and neurological disorders including Parkinson's disease (PD) and schizophrenia. Medications used to treat these disorders (such as levodopa, dopamine agonists, antipsychotics, among others) affect the prefrontal-striatal-hippocampal circuits in a complex fashion. We have built models of prefrontal-striatal and striatal-hippocampal interactions which simulate cognitive dysfunction in PD and schizophrenia. In these models, we argue that the basal ganglia is key for stimulus-response learning, the hippocampus for stimulus-stimulus representational learning, and the prefrontal cortex for stimulus selection during learning about multidimensional stimuli. In our models, PD is associated with reduced dopamine levels in the basal ganglia and prefrontal cortex. In contrast, the cognitive deficits in schizophrenia are associated primarily with hippocampal dysfunction, while the occurrence of negative symptoms is associated with frontostriatal deficits in a subset of patients. In this paper, we review our past models and provide new simulation results for both PD and schizophrenia. We also describe an extended model that includes simulation of the different functional role of D1 and D2 dopamine receptors in the basal ganglia and prefrontal cortex, a dissociation we argue is essential for understanding the non-uniform effects of levodopa, dopamine agonists, and antipsychotics on cognition. Motivated by clinical and physiological data, we discuss model limitations and challenges to be addressed in future models of these brain disorders.

A neurocomputational model of dopamine and prefrontal-striatal interactions during multicue category learning by Parkinson patients

Most existing models of dopamine and learning in Parkinson disease (PD) focus on simulating the role of basal ganglia dopamine in reinforcement learning. Much data argue, however, for a critical role for prefrontal cortex (PFC) dopamine in stimulus selection in attentional learning. Here, we present a new computational model that simulates performance in multicue category learning, such as the "weather prediction" task. The model addresses how PD and dopamine medications affect stimulus selection processes, which mediate reinforcement learning. In this model, PFC dopamine is key for attentional learning, whereas basal ganglia dopamine, consistent with other models, is key for reinforcement and motor learning. The model assumes that competitive dynamics among PFC neurons is the neural mechanism underlying stimulus selection with limited attentional resources, whereas competitive dynamics among striatal neurons is the neural mechanism underlying action selection. According to our model, PD is associated with decreased phasic and tonic dopamine levels in both PFC and basal ganglia. We assume that dopamine medications increase dopamine levels in both the basal ganglia and PFC, which, in turn, increase tonic dopamine levels but decrease the magnitude of phasic dopamine signaling in these brain structures. Increase of tonic dopamine levels in the simulated PFC enhances attentional shifting performance. The model provides a mechanistic account for several phenomena, including (a) medicated PD patients are more impaired at multicue probabilistic category learning than unmedicated patients and (b) medicated PD patients opt out of reversal when there are alternative and redundant cue dimensions.

A neural model of hippocampal-striatal interactions in associative learning and transfer generalization in various neurological and psychiatric patients

Building on our previous neurocomputational models of basal ganglia and hippocampal region function (and their modulation by dopamine and acetylcholine, respectively), we show here how an integration of these models can inform our understanding of the interaction between the basal ganglia and hippocampal region in associative learning and transfer generalization across various patient populations. As a common test bed for exploring interactions between these brain regions and neuromodulators, we focus on the acquired equivalence task, an associative learning paradigm in which stimuli that have been associated with the same outcome acquire a functional similarity such that subsequent generalization between these stimuli increases. This task has been used to test cognitive dysfunction in various patient populations with damages to the hippocampal region and basal ganglia, including studies of patients with Parkinson's disease (PD), schizophrenia, basal forebrain amnesia, and hippocampal atrophy. Simulation results show that damage to the hippocampal region-as in patients with hippocampal atrophy (HA), hypoxia, mild Alzheimer's (AD), or schizophrenia-leads to intact associative learning but impaired transfer generalization performance. Moreover, the model demonstrates how PD and anterior communicating artery (ACoA) aneurysm-two very different brain disorders that affect different neural mechanisms-can have similar effects on acquired equivalence performance. In particular, the model shows that simulating a loss of dopamine function in the basal ganglia module (as in PD) leads to slow acquisition learning but intact transfer generalization. Similarly, the model shows that simulating the loss of acetylcholine in the hippocampal region (as in ACoA aneurysm) also results in slower acquisition learning. We argue from this that changes in associative learning of stimulus-action pathways (in the basal ganglia) or changes in the learning of stimulus representations (in the hippocampal region) can have similar functional effects.

A neurocomputational model of classical conditioning phenomena: a putative role for the hippocampal region in associative learning

Some existing models of hippocampal function simulate performance in classical conditioning tasks using the error backpropagation algorithm to guide learning (Gluck, M.A., and Myers, C.E., (1993). Hippocampal mediation of stimulus representation: a computational theory. Hippocampus, 3(4), 491-516.). This algorithm is not biologically plausible because it requires information to be passed backward through layers of nodes and assumes that the environment provides information to the brain about what correct outputs should be. Here, we show that the same information-processing function proposed for the hippocampal region in the Gluck and Myers (1993) model can also be implemented in a network without using the backpropagation algorithm. Instead, our newer instantiation of the theory uses only (a) Hebbian learning methods which match more closely with synaptic and associative learning mechanisms ascribed to the hippocampal region and (b) a more plausible representation of input stimuli. We demonstrate here that this new more biologically plausible model is able to simulate various behavioral effects, including latent inhibition, acquired equivalence, sensory preconditioning, negative patterning, and context shift effects. In addition, the newer model is able to address some new phenomena including the effect of the number of training trials on blocking and overshadowing.

The influence of Markov decision process structure on the possible strategic use of working memory and episodic memory

Researchers use a variety of behavioral tasks to analyze the effect of biological manipulations on memory function. This research will benefit from a systematic mathematical method for analyzing memory demands in behavioral tasks. In the framework of reinforcement learning theory, these tasks can be mathematically described as partially-observable Markov decision processes. While a wealth of evidence collected over the past 15 years relates the basal ganglia to the reinforcement learning framework, only recently has much attention been paid to including psychological concepts such as working memory or episodic memory in these models. This paper presents an analysis that provides a quantitative description of memory states sufficient for correct choices at specific decision points. Using information from the mathematical structure of the task descriptions, we derive measures that indicate whether working memory (for one or more cues) or episodic memory can provide strategically useful information to an agent. In particular, the analysis determines which observed states must be maintained in or retrieved from memory to perform these specific tasks. We demonstrate the analysis on three simplified tasks as well as eight more complex memory tasks drawn from the animal and human literature (two alternation tasks, two sequence disambiguation tasks, two non-matching tasks, the 2-back task, and the 1-2-AX task). The results of these analyses agree with results from quantitative simulations of the task reported in previous publications and provide simple indications of the memory demands of the tasks which can require far less computation than a full simulation of the task. This may provide a basis for a quantitative behavioral stoichiometry of memory tasks.

A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism

Parkinson's disease (PD) patients exhibit cognitive deficits, including reinforcement learning, working memory (WM) and set shifting. Computational models of the basal ganglia-frontal system posit similar mechanisms for these deficits in terms of reduced dynamic range of striatal dopamine (DA) signals in both medicated and non-medicated states. Here, we report results from the first study that tests PD patients on and off dopaminergic medications in a modified version of the AX continuous performance (AX-CPT) working memory task, consisting of three performance phases and one phase requiring WM associations to be learned via reinforcement feedback. Patients generally showed impairments relative to controls. Medicated patients showed deficits specifically when having to ignore distracting stimuli during the delay. Our models suggest that this impairment is due to medication causing excessive WM updating by enhancing striatal "Go" signals that facilitate such updating, while concurrently suppressing "NoGo" signals. In contrast, patients off medication showed deficits consistent with an overall reduction in striatal DA and associated Go updating signals. Supporting this dichotomy, patients on and off medication both showed attentional shifting deficits, but for different reasons. Deficits in non-medicated patients were consistent with an inability to update the new attentional set, whereas those in medicated patients were evident when having to ignore distractors that had previously been task relevant. Finally, in the feedback-based WM phase, medicated patients were better than unmedicated patients, suggesting a key role of striatal DA in using feedback to update information into WM. These results lend further insight into the role of basal ganglia dopamine in WM and broadly support predictions from neurocomputational models.