The orbitofrontal cartographer

Neural mechanisms of credit assignment for delayed outcomes during contingent learning

Adaptive behavior in complex environments critically relies on the ability to appropriately link specific choices or actions to their outcomes. However, the neural mechanisms that support the ability to credit only those past choices believed to have caused the observed outcomes remain unclear. Here, we leverage multivariate pattern analyses of functional magnetic resonance imaging (fMRI) data and an adaptive learning task to shed light on the underlying neural mechanisms of such specific credit assignment. We find that the lateral orbitofrontal cortex (lOFC) and hippocampus (HC) code for the causal choice identity when credit needs to be assigned for choices that are separated from outcomes by a long delay, even when this delayed transition is punctuated by interim decisions. Further, we show when interim decisions must be made, learning is additionally supported by lateral frontopolar cortex (FPl). Our results indicate that FPl holds previous causal choices in a "pending" state until a relevant outcome is observed, and the fidelity of these representations predicts the fidelity of subsequent causal choice representations in lOFC and HC during credit assignment. Together, these results highlight the importance of the timely reinstatement of specific causes in lOFC and HC in learning choice-outcome relationships when delays and choices intervene, a critical component of real-world learning and decision making.

Ventral frontostriatal circuitry mediates the computation of reinforcement from symbolic gains and losses

Reinforcement learning (RL), particularly in primates, is often driven by symbolic outcomes. However, it is usually studied with primary reinforcers. To examine the neural mechanisms underlying learning from symbolic outcomes, we trained monkeys on a task in which they learned to choose options that led to gains of tokens and avoid choosing options that led to losses of tokens. We then recorded simultaneously from the orbitofrontal cortex (OFC), ventral striatum (VS), amygdala (AMY), and the mediodorsal thalamus (MDt). We found that the OFC played a dominant role in coding token outcomes and token prediction errors. The other areas contributed complementary functions with the VS coding appetitive outcomes and the AMY coding the salience of outcomes. The MDt coded actions and relayed information about tokens between the OFC and VS. Thus, OFC leads the process of symbolic reinforcement learning in the ventral frontostriatal circuitry.

Optogenetic Inhibition of the Orbitofrontal Cortex Disrupts Inhibitory Control during Stop-Change Performance in Male Rats

Historically, the orbitofrontal cortex (OFC) has been implicated in a variety of behaviors ranging from reversal learning and inhibitory control to more complex representations of reward value and task space. While modern interpretations of the OFC's function have focused on a role in outcome evaluation, these cognitive processes often require an organism to inhibit a maladaptive response or strategy. Single-unit recordings from the OFC in rats performing a stop-change task show that the OFC responds strongly to STOP trials. To investigate the role that the OFC plays in stop-change performance, we expressed halorhodopsin (eNpHR3.0) in excitatory neurons in the OFC and tested rats on the stop-change task. Previous work suggests that the OFC differentiates between STOP trials based on trial sequence (i.e., gS trials: STOP trials preceded by a GO vs sS trials: STOP trials preceded by a STOP). We found that yellow light activation of the eNpHR3.0-expressing neurons significantly decreased accuracy only on STOP trials that followed GO trials (gS trials). Further, optogenetic inhibition of the OFC speeded reaction times on error trials. This suggests that the OFC plays a role in inhibitory control processes and that this role needs to be accounted for in modern interpretations of OFC function.

Lateral Orbitofrontal Cortex Encodes Presence of Risk and Subjective Risk Preference During Decision-Making

Adaptive decision-making requires consideration of objective risks and rewards associated with each option, as well as subjective preference for risky/safe alternatives. Inaccurate risk/reward estimations can engender excessive risk-taking, a central trait in many psychiatric disorders. The lateral orbitofrontal cortex (lOFC) has been linked to many disorders associated with excessively risky behavior and is ideally situated to mediate risky decision-making. Here, we used single-unit electrophysiology to measure neuronal activity from lOFC of freely moving rats performing in a punishment-based risky decision-making task. Subjects chose between a small, safe reward and a large reward associated with either 0% or 50% risk of concurrent punishment. lOFC activity repeatedly encoded current risk in the environment throughout the decision-making sequence, signaling risk before, during, and after a choice. In addition, lOFC encoded reward magnitude, although this information was only evident during action selection. A Random Forest classifier successfully used neural data accurately to predict the risk of punishment in any given trial, and the ability to predict choice via lOFC activity differentiated between and risk-preferring and risk-averse rats. Finally, risk preferring subjects demonstrated reduced lOFC encoding of risk and increased encoding of reward magnitude. These findings suggest lOFC may serve as a central decision-making hub in which external, environmental information converges with internal, subjective information to guide decision-making in the face of punishment risk.

The cognitive (lateral) hypothalamus

Despite the physiological complexity of the hypothalamus, its role is typically restricted to initiation or cessation of innate behaviors. For example, theories of lateral hypothalamus argue that it is a switch to turn feeding 'on' and 'off' as dictated by higher-order structures that render when feeding is appropriate. However, recent data demonstrate that the lateral hypothalamus is critical for learning about food-related cues. Furthermore, the lateral hypothalamus opposes learning about information that is neutral or distal to food. This reveals the lateral hypothalamus as a unique arbitrator of learning capable of shifting behavior toward or away from important events. This has relevance for disorders characterized by changes in this balance, including addiction and schizophrenia. Generally, this suggests that hypothalamic function is more complex than increasing or decreasing innate behaviors.

The orbitofrontal cortex: A goal-directed cognitive map framework for social and non-social behaviors

The orbitofrontal cortex (OFC) is regarded as one of the core brain areas in a variety of value-based behaviors. Over the past two decades, tremendous knowledge about the OFC function was gained from studying the behaviors of single subjects. As a result, our previous understanding of the OFC's function of encoding decision variables, such as the value and identity of choices, has evolved to the idea that the OFC encodes a more complex representation of the task space as a cognitive map. Accumulating evidence also indicates that the OFC importantly contributes to behaviors in social contexts, especially those involved in cooperative interactions. However, it remains elusive how exactly OFC neurons contribute to social functions and how non-social and social behaviors are related to one another in the computations performed by OFC neurons. In this review, we aim to provide an integrated view of the OFC function across both social and non-social behavioral contexts. We propose that seemingly complex functions of the OFC may be explained by its role in providing a goal-directed cognitive map to guide a wide array of adaptive reward-based behaviors in both social and non-social domains.

Non-Invasive Brain Stimulation for the Modulation of Aggressive Behavior-A Systematic Review of Randomized Sham-Controlled Studies

INTRO

Aggressive behavior represents a significant public health issue, with relevant social, political, and security implications. Non-invasive brain stimulation (NIBS) techniques may modulate aggressive behavior through stimulation of the prefrontal cortex.

AIMS

To review research on the effectiveness of NIBS to alter aggression, discuss the main findings and potential limitations, consider the specifics of the techniques and protocols employed, and discuss clinical implications.

METHODS

A systematic review of the literature available in the PubMed database was carried out, and 17 randomized sham-controlled studies investigating the effectiveness of NIBS techniques on aggression were included. Exclusion criteria included reviews, meta-analyses, and articles not referring to the subject of interest or not addressing cognitive and emotional modulation aims.

CONCLUSIONS

The reviewed data provide promising evidence for the beneficial effects of tDCS, conventional rTMS, and cTBS on aggression in healthy adults, forensic, and clinical samples. The specific stimulation target is a key factor for the success of stimulation on aggression modulation. rTMS and cTBS showed opposite effects on aggression compared with tDCS. However, due to the heterogeneity of stimulation protocols, experimental designs, and samples, we cannot exclude other factors that may play a confounding role.

The role of the lateral orbitofrontal cortex in creating cognitive maps

We use mental models of the world-cognitive maps-to guide behavior. The lateral orbitofrontal cortex (lOFC) is typically thought to support behavior by deploying these maps to simulate outcomes, but recent evidence suggests that it may instead support behavior by underlying map creation. We tested between these two alternatives using outcome-specific devaluation and a high-potency chemogenetic approach. Selectively inactivating lOFC principal neurons when male rats learned distinct cue-outcome associations, but before outcome devaluation, disrupted subsequent inference, confirming a role for the lOFC in creating new maps. However, lOFC inactivation surprisingly led to generalized devaluation, a result that is inconsistent with a complete mapping failure. Using a reinforcement learning framework, we show that this effect is best explained by a circumscribed deficit in credit assignment precision during map construction, suggesting that the lOFC has a selective role in defining the specificity of associations that comprise cognitive maps.

Calcium activity is a degraded estimate of spikes

Recording action potentials extracellularly during behavior has led to fundamental discoveries regarding neural function-hippocampal neurons respond to locations in space,1 motor cortex neurons encode movement direction,2 and dopamine neurons signal reward prediction errors3-observations undergirding current theories of cognition,4 movement,5 and learning.6 Recently it has become possible to measure calcium flux, an internal cellular signal related to spiking. The ability to image calcium flux in anatomically7,8 or genetically9 identified neurons can extend our knowledge of neural circuit function by allowing activity to be monitored in specific cell types or projections, or in the same neurons across many days. However, while initial studies were grounded in prior unit recording work, it has become fashionable to assume that calcium is identical to spiking, even though the spike-to-fluorescence transformation is nonlinear, noisy, and unpredictable under real-world conditions.10 It remains an open question whether calcium provides a high-fidelity representation of single-unit activity in awake, behaving subjects. Here, we have addressed this question by recording both signals in the lateral orbitofrontal cortex (OFC) of rats during olfactory discrimination learning. Activity in the OFC during olfactory learning has been well-studied in humans,11,12,13,14 nonhuman primates,15,16 and rats,17,18,19,20,21 where it has been shown to signal information about both the sensory properties of odor cues and the rewards they predict. Our single-unit results replicated prior findings, whereas the calcium signal provided only a degraded estimate of the information available in the single-unit spiking, reflecting primarily reward value.

Orbitofrontal cortex populations are differentially recruited to support actions

The ability to use information from one's prior actions is necessary for decision-making. While orbitofrontal cortex (OFC) has been hypothesized as key for inferences made using cue and value-related information, whether OFC populations contribute to the use of information from volitional actions to guide behavior is not clear. Here, we used a self-paced lever-press hold-down task in which mice infer prior lever-press durations to guide subsequent action performance. We show that the activity of genetically identified lateral OFC (lOFC) subpopulations differentially instantiate current and prior action information during ongoing action execution. Transient state-dependent lOFC circuit disruptions of specified subpopulations reduced the encoding of ongoing press durations but did not disrupt the use of prior action information to guide future action performance. In contrast, a chronic functional loss of lOFC circuit activity resulted in increased reliance on recently executed lever-press durations and impaired contingency reversal, suggesting the recruitment of compensatory mechanisms that resulted in repetitive action control. Our results identify a novel role for lOFC in the integration of action information to guide adaptive behavior.

Hippocampal signals modify orbitofrontal representations to learn new paths

We often remember the consequences of past choices to adapt to changing circumstances. Recalling past events requires the hippocampus (HPC), and using stimuli to anticipate outcome values requires the orbitofrontal cortex (OFC).1-3 Spatial reversal tasks require both structures to navigate newly rewarded paths.4,5 Both HPC place6 and OFC value cells7,8 fire in phase with theta (4-12 Hz) oscillations. Both structures are described as cognitive maps: HPC maps space9 and OFC maps task states.10 These similarities imply that OFC-HPC interactions are crucial for using memory to predict outcomes when circumstances change, but the mechanisms remain largely unknown. To investigate possible interactions, we simultaneously recorded ensembles in OFC and CA1 as rats learned spatial reversals in a plus maze. Striking interactions occurred only while rats learned their first reversal: CA1 population vectors predicted changes in OFC activity but not vice versa, OFC spikes phase locked to hippocampal theta oscillations, mixed pairs of CA1 and OFC neurons fired together within single theta cycles, and CA1 led OFC spikes by ∼30 ms. After the new contingency became familiar, CA1 ensembles stably represented distinct spatial paths, whereas OFC ensembles developed more generalized goal arm representations in different paths to identical rewards. These frontotemporal interactions, engaged selectively when new task features inform decision-making, suggest a mechanism for linking novel episodes with expected outcomes, when HPC signals trigger "cognitive remapping" by OFC.11.

The shallow cognitive map hypothesis: A hippocampal framework for thought disorder in schizophrenia

Memories are not formed in isolation. They are associated and organized into relational knowledge structures that allow coherent thought. Failure to express such coherent thought is a key hallmark of Schizophrenia. Here we explore the hypothesis that thought disorder arises from disorganized Hippocampal cognitive maps. In doing so, we combine insights from two key lines of investigation, one concerning the neural signatures of cognitive mapping, and another that seeks to understand lower-level cellular mechanisms of cognition within a dynamical systems framework. Specifically, we propose that multiple distinct pathological pathways converge on the shallowing of Hippocampal attractors, giving rise to disorganized Hippocampal cognitive maps and driving conceptual disorganization. We discuss the available evidence at the computational, behavioural, network, and cellular levels. We also outline testable predictions from this framework, including how it could unify major chemical and psychological theories of schizophrenia and how it can provide a rationale for understanding the aetiology and treatment of the disease.

The whole prefrontal cortex is premotor cortex

We propose that the entirety of the prefrontal cortex (PFC) can be seen as fundamentally premotor in nature. By this, we mean that the PFC consists of an action abstraction hierarchy whose core function is the potentiation and depotentiation of possible action plans at different levels of granularity. We argue that the apex of the hierarchy should revolve around the process of goal-selection, which we posit is inherently a form of optimization over action abstraction. Anatomical and functional evidence supports the idea that this hierarchy originates on the orbital surface of the brain and extends dorsally to motor cortex. Accordingly, our viewpoint positions the orbitofrontal cortex in a key role in the optimization of goal-selection policies, and suggests that its other proposed roles are aspects of this more general function. Our proposed perspective will reframe outstanding questions, open up new areas of inquiry and align theories of prefrontal function with evolutionary principles. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

The learning of prospective and retrospective cognitive maps within neural circuits

Brain circuits are thought to form a "cognitive map" to process and store statistical relationships in the environment. A cognitive map is commonly defined as a mental representation that describes environmental states (i.e., variables or events) and the relationship between these states. This process is commonly conceptualized as a prospective process, as it is based on the relationships between states in chronological order (e.g., does reward follow a given state?). In this perspective, we expand this concept on the basis of recent findings to postulate that in addition to a prospective map, the brain forms and uses a retrospective cognitive map (e.g., does a given state precede reward?). In doing so, we demonstrate that many neural signals and behaviors (e.g., habits) that seem inflexible and non-cognitive can result from retrospective cognitive maps. Together, we present a significant conceptual reframing of the neurobiological study of associative learning, memory, and decision making.

Cortical Contributions to Higher-Order Conditioning: A Review of Retrosplenial Cortex Function

In higher-order conditioning paradigms, such as sensory preconditioning or second-order conditioning, discrete (e.g., phasic) or contextual (e.g., static) stimuli can gain the ability to elicit learned responses despite never being directly paired with reinforcement. The purpose of this mini-review is to examine the neuroanatomical basis of high-order conditioning, by selectively reviewing research that has examined the role of the retrosplenial cortex (RSC) in sensory preconditioning and second-order conditioning. For both forms of higher-order conditioning, we first discuss the types of associations that may occur and then review findings from RSC lesion/inactivation experiments. These experiments demonstrate a role for the RSC in sensory preconditioning, suggesting that this cortical region might contribute to higher-order conditioning via the encoding of neutral stimulus-stimulus associations. In addition, we address knowledge gaps, avenues for future research, and consider the contribution of the RSC to higher-order conditioning in relation to related brain structures.