Neural correlates of strategic reasoning during competitive games

Autocorrelation structure at rest predicts value correlates of single neurons during reward-guided choice

Correlates of value are routinely observed in the prefrontal cortex (PFC) during reward-guided decision making. In previous work (Hunt et al., 2015), we argued that PFC correlates of chosen value are a consequence of varying rates of a dynamical evidence accumulation process. Yet within PFC, there is substantial variability in chosen value correlates across individual neurons. Here we show that this variability is explained by neurons having different temporal receptive fields of integration, indexed by examining neuronal spike rate autocorrelation structure whilst at rest. We find that neurons with protracted resting temporal receptive fields exhibit stronger chosen value correlates during choice. Within orbitofrontal cortex, these neurons also sustain coding of chosen value from choice through the delivery of reward, providing a potential neural mechanism for maintaining predictions and updating stored values during learning. These findings reveal that within PFC, variability in temporal specialisation across neurons predicts involvement in specific decision-making computations.

DOI:http://dx.doi.org/10.7554/eLife.18937.001

Three challenges for connecting model to mechanism in decision-making

Recent years have seen a growing interest in understanding the neural mechanisms that support decision-making. The advent of new tools for measuring and manipulating neurons, alongside the inclusion of multiple new animal models and sensory systems has led to the generation of many novel datasets. The potential for these new approaches to constrain decision-making models is unprecedented. Here, we argue that to fully leverage these new approaches, three challenges must be met. First, experimenters must design well-controlled behavioral experiments that make it possible to distinguish competing behavioral strategies. Second, analyses of neural responses should think beyond single neurons, taking into account tradeoffs of single-trial versus trial-averaged approaches. Finally, quantitative model comparisons should be used, but must consider common obstacles.

Taming the beast: extracting generalizable knowledge from computational models of cognition

Generalizing knowledge from experimental data requires constructing theories capable of explaining observations and extending beyond them. Computational modeling offers formal quantitative methods for generating and testing theories of cognition and neural processing. These techniques can be used to extract general principles from specific experimental measurements, but introduce dangers inherent to theory: model-based analyses are conditioned on a set of fixed assumptions that impact the interpretations of experimental data. When these conditions are not met, model-based results can be misleading or biased. Recent work in computational modeling has highlighted the implications of this problem and developed new methods for minimizing its negative impact. Here we discuss the issues that arise when data is interpreted through models and strategies for avoiding misinterpretation of data through model fitting.

Targeting Prefrontal Cortical Systems for Drug Development: Potential Therapies for Cognitive Disorders

Medications to treat cognitive disorders are increasingly needed, yet researchers have had few successes in this challenging arena. Cognitive abilities in primates arise from highly evolved N-methyl-d-aspartate (NMDA) receptor circuits in layer III of the dorsolateral prefrontal cortex. These circuits have unique modulatory needs that can differ from the layer V neurons that predominate in rodents, but they offer multiple therapeutic targets. Cognitive improvement often requires low doses that enhance the pattern of information held in working memory, whereas higher doses can produce nonspecific changes that obscure information. Identifying appropriate doses for clinical trials may be helped by assessments in monkeys and by flexible, individualized dose designs. The use of guanfacine (Intuniv) for prefrontal cortical disorders was based on research in monkeys, supporting this approach. Coupling our knowledge of higher primate circuits with the powerful methods now available in drug design will help create effective treatments for cognitive disorders.

Monkey Prefrontal Neurons Reflect Logical Operations for Cognitive Control in a Variant of the AX Continuous Performance Task (AX-CPT)

Cognitive control is the ability to modify the behavioral response to a stimulus based on internal representations of goals or rules. We sought to characterize neural mechanisms in prefrontal cortex associated with cognitive control in a context that would maximize the potential for future translational relevance to human neuropsychiatric disease. To that end, we trained monkeys to perform a dot-pattern variant of the AX continuous performance task that is used to measure cognitive control impairment in patients with schizophrenia (MacDonald, 2008;Jones et al., 2010). Here we describe how information processing for cognitive control in this task is related to neural activity patterns in prefrontal cortex of monkeys, to advance our understanding of how behavioral flexibility is implemented by prefrontal neurons in general, and to model neural signals in the healthy brain that may be disrupted to produce cognitive control deficits in schizophrenia. We found that the neural representation of stimuli in prefrontal cortex is strongly biased toward stimuli that inhibit prepotent or automatic responses. We also found that population signals encoding different stimuli were modulated to overlap in time specifically in the case that information from multiple stimuli had to be integrated to select a conditional response. Finally, population signals relating to the motor response were biased toward less frequent and therefore less automatic actions. These data relate neuronal activity patterns in prefrontal cortex to logical information processing operations required for cognitive control, and they characterize neural events that may be disrupted in schizophrenia.

SIGNIFICANCE STATEMENT

Functional imaging studies have demonstrated that cognitive control deficits in schizophrenia are associated with reduced activation of the dorsolateral prefrontal cortex (MacDonald et al., 2005). However, these data do not reveal how the disease has disrupted the function of prefrontal neurons to produce the observed deficits in cognitive control. Relating cognitive control to neurophysiological signals at a cellular level in prefrontal cortex is a necessary first step toward understanding how disruption of these signals could lead to cognitive control failure in neuropsychiatric disease. To that end, we translated a task that measures cognitive control deficits in patients with schizophrenia to monkeys and describe here how neural signals in prefrontal cortex relate to performance.

Neural Correlates and Connectivity Underlying Stress-Related Impulse Control Difficulties in Alcoholism

BACKGROUND

Stress triggers impulsive and addictive behaviors, and alcoholism has been frequently associated with increased stress sensitivity and impulse control problems. However, neural correlates underlying the link between alcoholism and impulsivity in the context of stress in patients with alcohol use disorders (AUD) have not been well studied.

METHODS

This study investigated neural correlates and connectivity patterns associated with impulse control difficulties in abstinent AUD patients. Using functional magnetic resonance imaging, brain responses of 37 AUD inpatients, and 37 demographically matched healthy controls were examined during brief individualized imagery trials of stress, alcohol cue, and neutral-relaxing conditions. Stress-related impulsivity was measured using a subscale score of impulse control problems from Difficulties in Emotion Regulation Scale.

RESULTS

Impulse control difficulties in AUD patients were significantly associated with hypo-active response to stress in the ventromedial prefrontal cortex (VmPFC), right caudate, and left lateral PFC (LPFC) compared to the neutral condition (p < 0.01, whole-brain corrected). These regions were used as seed regions to further examine the connectivity patterns with other brain regions. With the VmPFC seed, AUD patients showed reduced connectivity with the anterior cingulate cortex compared to controls, which are core regions of emotion regulation, suggesting AUD patients' decreased ability to modulate emotional response under distressed state. With the right caudate seed, patients showed increased connectivity with the right motor cortex, suggesting increased tendency toward habitually driven behaviors. With the left LPFC seed, decreased connectivity with the dorsomedial PFC (DmPFC), but increased connectivity with sensory and motor cortices were found in AUD patients compared to controls (p < 0.05, whole-brain corrected). Reduced connectivity between the left LPFC and DmPFC was further associated with increased stress-induced anxiety in AUD patients (p < 0.05, with adjusted Bonferroni correction).

CONCLUSIONS

Hypo-active response to stress and altered connectivity in key emotion regulatory regions may account for greater stress-related impulse control problems in alcoholism.

Recording extracellular neural activity in the behaving monkey using a semichronic and high-density electrode system

We describe a technique to semichronically record the cortical extracellular neural activity in the behaving monkey employing commercial high-density electrodes. After the design and construction of low cost microdrives that allow varying the depth of the recording locations after the implantation surgery, we recorded the extracellular unit activity from pools of neurons at different depths in the presupplementary motor cortex (pre-SMA) of a rhesus monkey trained in a tapping task. The collected data were processed to classify cells as putative pyramidal cells or interneurons on the basis of their waveform features. We also demonstrate that short time cross-correlogram occasionally yields unit pairs with high short latency (<5 ms), narrow bin (<3 ms) peaks, indicative of monosynaptic spike transmission from pre- to postsynaptic neurons. These methods have been verified extensively in rodents. Finally, we observed that the pattern of population activity was repetitive over distinct trials of the tapping task. These results show that the semichronic technique is a viable option for the large-scale parallel recording of local circuit activity at different depths in the cortex of the macaque monkey and other large species.

Hierarchical decision processes that operate over distinct timescales underlie choice and changes in strategy

Decision-making in a natural environment depends on a hierarchy of interacting decision processes. A high-level strategy guides ongoing choices, and the outcomes of those choices determine whether or not the strategy should change. When the right decision strategy is uncertain, as in most natural settings, feedback becomes ambiguous because negative outcomes may be due to limited information or bad strategy. Disambiguating the cause of feedback requires active inference and is key to updating the strategy. We hypothesize that the expected accuracy of a choice plays a crucial rule in this inference, and setting the strategy depends on integration of outcome and expectations across choices. We test this hypothesis with a task in which subjects report the net direction of random dot kinematograms with varying difficulty while the correct stimulus-response association undergoes invisible and unpredictable switches every few trials. We show that subjects treat negative feedback as evidence for a switch but weigh it with their expected accuracy. Subjects accumulate switch evidence (in units of log-likelihood ratio) across trials and update their response strategy when accumulated evidence reaches a bound. A computational framework based on these principles quantitatively explains all aspects of the behavior, providing a plausible neural mechanism for the implementation of hierarchical multiscale decision processes. We suggest that a similar neural computation-bounded accumulation of evidence-underlies both the choice and switches in the strategy that govern the choice, and that expected accuracy of a choice represents a key link between the levels of the decision-making hierarchy.

Neural Basis of Strategic Decision Making

Human choice behaviors during social interactions often deviate from the predictions of game theory. This might arise partly from the limitations in the cognitive abilities necessary for recursive reasoning about the behaviors of others. In addition, during iterative social interactions, choices might change dynamically as knowledge about the intentions of others and estimates for choice outcomes are incrementally updated via reinforcement learning. Some of the brain circuits utilized during social decision making might be general-purpose and contribute to isomorphic individual and social decision making. By contrast, regions in the medial prefrontal cortex (mPFC) and temporal parietal junction (TPJ) might be recruited for cognitive processes unique to social decision making.

Opponent Identity Influences Value Learning in Simple Games

Context plays a pivotal role in many decision-making scenarios, including social interactions wherein the identities and strategies of other decision makers often shape our behaviors. However, the neural mechanisms for tracking such contextual information are poorly understood. Here, we investigated how opponent identity affects human reinforcement learning during a simulated competitive game against two independent computerized opponents. We found that strategies of participants were affected preferentially by the outcomes of the previous interactions with the same opponent. In addition, reinforcement signals from the previous trial were less discriminable throughout the brain after the opponent changed, compared with when the same opponent was repeated. These opponent-selective reinforcement signals were particularly robust in right rostral anterior cingulate and right lingual regions, where opponent-selective reinforcement signals correlated with a behavioral measure of opponent-selective reinforcement learning. Therefore, when choices involve multiple contextual frames, such as different opponents in a game, decision making and its neural correlates are influenced by multithreaded histories of reinforcement. Overall, our findings are consistent with the availability of temporally overlapping, context-specific reinforcement signals.

SIGNIFICANCE STATEMENT

In real-world decision making, context plays a strong role in determining the value of an action. Similar choices take on different values depending on setting. We examined the contextual dependence of reward-based learning and reinforcement signals using a simple two-choice matching-pennies game played by humans against two independent computer opponents that were randomly interleaved. We found that human subjects' strategies were highly dependent on opponent context in this game, a fact that was reflected in select brain regions' activity (rostral anterior cingulate and lingual cortex). These results indicate that human reinforcement histories are highly dependent on contextual factors, a fact that is reflected in neural correlates of reinforcement signals.

Spatial gradient in value representation along the medial prefrontal cortex reflects individual differences in prosociality

Despite the importance of valuing another person's welfare for prosocial behavior, currently we have only a limited understanding of how these values are represented in the brain and, more importantly, how they give rise to individual variability in prosociality. In the present study, participants underwent functional magnetic resonance imaging while performing a prosocial learning task in which they could choose to benefit themselves and/or another person. Choice behavior indicated that participants valued the welfare of another person, although less so than they valued their own welfare. Neural data revealed a spatial gradient in activity within the medial prefrontal cortex (MPFC), such that ventral parts predominantly represented self-regarding values and dorsal parts predominantly represented other-regarding values. Importantly, compared with selfish individuals, prosocial individuals showed a more gradual transition from self-regarding to other-regarding value signals in the MPFC and stronger MPFC-striatum coupling when they made choices for another person rather than for themselves. The present study provides evidence of neural markers reflecting individual differences in human prosociality.

Reinforcement learning in depression: A review of computational research

Despite being considered primarily a mood disorder, major depressive disorder (MDD) is characterized by cognitive and decision making deficits. Recent research has employed computational models of reinforcement learning (RL) to address these deficits. The computational approach has the advantage in making explicit predictions about learning and behavior, specifying the process parameters of RL, differentiating between model-free and model-based RL, and the computational model-based functional magnetic resonance imaging and electroencephalography. With these merits there has been an emerging field of computational psychiatry and here we review specific studies that focused on MDD. Considerable evidence suggests that MDD is associated with impaired brain signals of reward prediction error and expected value ('wanting'), decreased reward sensitivity ('liking') and/or learning (be it model-free or model-based), etc., although the causality remains unclear. These parameters may serve as valuable intermediate phenotypes of MDD, linking general clinical symptoms to underlying molecular dysfunctions. We believe future computational research at clinical, systems, and cellular/molecular/genetic levels will propel us toward a better understanding of the disease.

Neuronal prediction of opponent's behavior during cooperative social interchange in primates

A cornerstone of successful social interchange is the ability to anticipate each other's intentions or actions. While generating these internal predictions is essential for constructive social behavior, their single neuronal basis and causal underpinnings are unknown. Here, we discover specific neurons in the primate dorsal anterior cingulate that selectively predict an opponent's yet unknown decision to invest in their common good or defect and distinct neurons that encode the monkey's own current decision based on prior outcomes. Mixed population predictions of the other was remarkably near optimal compared to behavioral decoders. Moreover, disrupting cingulate activity selectively biased mutually beneficial interactions between the monkeys but, surprisingly, had no influence on their decisions when no net-positive outcome was possible. These findings identify a group of other-predictive neurons in the primate anterior cingulate essential for enacting cooperative interactions and may pave a way toward the targeted treatment of social behavioral disorders.

Dynamic routing of task-relevant signals for decision making in dorsolateral prefrontal cortex

Neurons in the dorsolateral prefrontal cortex (DLPFC) encode a diverse array of sensory and mnemonic signals, but little is known about how this information is dynamically routed during decision making. We analyzed the neuronal activity in the DLPFC of monkeys performing a probabilistic reversal task where information about the probability and magnitude of reward was provided by the target color and numerical cues, respectively. The location of the target of a given color was randomized across trials and therefore was not relevant for subsequent choices. DLPFC neurons encoded signals related to both task-relevant and irrelevant features, but only task-relevant mnemonic signals were encoded congruently with choice signals. Furthermore, only the task-relevant signals related to previous events were more robustly encoded following rewarded outcomes. Thus, multiple types of neural signals are flexibly routed in the DLPFC so as to favor actions that maximize reward.