1
|
Hoy CW, de Hemptinne C, Wang SS, Harmer CJ, Apps MAJ, Husain M, Starr PA, Little S. Beta and theta oscillations track effort and previous reward in the human basal ganglia and prefrontal cortex during decision making. Proc Natl Acad Sci U S A 2024; 121:e2322869121. [PMID: 39047043 PMCID: PMC11295073 DOI: 10.1073/pnas.2322869121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Accepted: 06/18/2024] [Indexed: 07/27/2024] Open
Abstract
Choosing whether to exert effort to obtain rewards is fundamental to human motivated behavior. However, the neural dynamics underlying the evaluation of reward and effort in humans is poorly understood. Here, we report an exploratory investigation into this with chronic intracranial recordings from the prefrontal cortex (PFC) and basal ganglia (BG; subthalamic nuclei and globus pallidus) in people with Parkinson's disease performing a decision-making task with offers that varied in levels of reward and physical effort required. This revealed dissociable neural signatures of reward and effort, with BG beta (12 to 20 Hz) oscillations tracking effort on a single-trial basis and PFC theta (4 to 7 Hz) signaling previous trial reward, with no effects of net subjective value. Stimulation of PFC increased overall acceptance of offers and sensitivity to reward while decreasing the impact of effort on choices. This work uncovers oscillatory mechanisms that guide fundamental decisions to exert effort for reward across BG and PFC, supports a causal role of PFC for such choices, and seeds hypotheses for future studies.
Collapse
Affiliation(s)
- Colin W. Hoy
- Department of Neurology, University of California, San Francisco, CA94143
| | - Coralie de Hemptinne
- Norman Fixel Institute for Neurological Diseases, University of Florida, Gainesville, FL32608
- Department of Neurology, University of Florida, Gainesville, FL32608
| | - Sarah S. Wang
- Department of Neurology, University of California, San Francisco, CA94143
| | - Catherine J. Harmer
- Department of Psychiatry, University of Oxford, OxfordOX3 7JX, United Kingdom
| | - Matthew A. J. Apps
- Department of Experimental Psychology, University of Oxford, OxfordOX2 6GG, United Kingdom
- Institute for Mental Health, School of Psychology, University of Birmingham, Birmingham UKB15 2TT, United Kingdom
- Centre for Human Brain Health, School of Psychology, University of Birmingham, BirminghamB15 2TT, United Kingdom
| | - Masud Husain
- Department of Experimental Psychology, University of Oxford, OxfordOX2 6GG, United Kingdom
- Nuffield Department of Clinical Neurosciences, University of Oxford, OxfordOX3 9DU, United Kingdom
| | - Philip A. Starr
- Department of Neurological Surgery, University of California, San Francisco, CA94143, United Kingdom
| | - Simon Little
- Department of Neurology, University of California, San Francisco, CA94143
| |
Collapse
|
2
|
Drieu C, Zhu Z, Wang Z, Fuller K, Wang A, Elnozahy S, Kuchibhotla K. Rapid emergence of latent knowledge in the sensory cortex drives learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.10.597946. [PMID: 38915657 PMCID: PMC11195094 DOI: 10.1101/2024.06.10.597946] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/26/2024]
Abstract
Rapid learning confers significant advantages to animals in ecological environments. Despite the need for speed, animals appear to only slowly learn to associate rewarded actions with predictive cues1-4. This slow learning is thought to be supported by a gradual expansion of predictive cue representation in the sensory cortex2,5. However, evidence is growing that animals learn more rapidly than classical performance measures suggest6-8, challenging the prevailing model of sensory cortical plasticity. Here, we investigated the relationship between learning and sensory cortical representations. We trained mice on an auditory go/no-go task that dissociated the rapid acquisition of task contingencies (learning) from its slower expression (performance)7. Optogenetic silencing demonstrated that the auditory cortex (AC) drives both rapid learning and slower performance gains but becomes dispensable at expert. Rather than enhancement or expansion of cue representations9, two-photon calcium imaging of AC excitatory neurons throughout learning revealed two higher-order signals that were causal to learning and performance. First, a reward prediction (RP) signal emerged rapidly within tens of trials, was present after action-related errors only early in training, and faded at expert levels. Strikingly, silencing at the time of the RP signal impaired rapid learning, suggesting it serves an associative and teaching role. Second, a distinct cell ensemble encoded and controlled licking suppression that drove the slower performance improvements. These two ensembles were spatially clustered but uncoupled from underlying sensory representations, indicating a higher-order functional segregation within AC. Our results reveal that the sensory cortex manifests higher-order computations that separably drive rapid learning and slower performance improvements, reshaping our understanding of the fundamental role of the sensory cortex.
Collapse
Affiliation(s)
- Céline Drieu
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
- Johns Hopkins Kavli Neuroscience Discovery Institute, Baltimore, MD, USA
| | - Ziyi Zhu
- Department of Neuroscience, School of Medicine, Johns Hopkins University, MD, USA
| | - Ziyun Wang
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Kylie Fuller
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Aaron Wang
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Sarah Elnozahy
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
- Present address: Sainsbury Wellcome Centre, London, UK
| | - Kishore Kuchibhotla
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD, USA
- Johns Hopkins Kavli Neuroscience Discovery Institute, Baltimore, MD, USA
- Department of Neuroscience, School of Medicine, Johns Hopkins University, MD, USA
| |
Collapse
|
3
|
Cowan RL, Davis T, Kundu B, Rahimpour S, Rolston JD, Smith EH. More widespread and rigid neuronal representation of reward expectation underlies impulsive choices. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.11.588637. [PMID: 38645037 PMCID: PMC11030340 DOI: 10.1101/2024.04.11.588637] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/23/2024]
Abstract
Impulsive choices prioritize smaller, more immediate rewards over larger, delayed, or potentially uncertain rewards. Impulsive choices are a critical aspect of substance use disorders and maladaptive decision-making across the lifespan. Here, we sought to understand the neuronal underpinnings of expected reward and risk estimation on a trial-by-trial basis during impulsive choices. To do so, we acquired electrical recordings from the human brain while participants carried out a risky decision-making task designed to measure choice impulsivity. Behaviorally, we found a reward-accuracy tradeoff, whereby more impulsive choosers were more accurate at the task, opting for a more immediate reward while compromising overall task performance. We then examined how neuronal populations across frontal, temporal, and limbic brain regions parametrically encoded reinforcement learning model variables, namely reward and risk expectation and surprise, across trials. We found more widespread representations of reward value expectation and prediction error in more impulsive choosers, whereas less impulsive choosers preferentially represented risk expectation. A regional analysis of reward and risk encoding highlighted the anterior cingulate cortex for value expectation, the anterior insula for risk expectation and surprise, and distinct regional encoding between impulsivity groups. Beyond describing trial-by-trial population neuronal representations of reward and risk variables, these results suggest impaired inhibitory control and model-free learning underpinnings of impulsive choice. These findings shed light on neural processes underlying reinforced learning and decision-making in uncertain environments and how these processes may function in psychiatric disorders.
Collapse
Affiliation(s)
- Rhiannon L Cowan
- Department of Neurosurgery, University of Utah, Salt Lake City, UT 84132, USA
| | - Tyler Davis
- Department of Neurosurgery, University of Utah, Salt Lake City, UT 84132, USA
| | - Bornali Kundu
- Department of Neurosurgery, University of Missouri, Columbia, MO 65212, USA
| | - Shervin Rahimpour
- Department of Neurosurgery, University of Utah, Salt Lake City, UT 84132, USA
| | - John D Rolston
- Department of Neurosurgery, Brigham & Women's Hospital, Harvard Medical School, Boston, MA 02115, USA
| | - Elliot H Smith
- Department of Neurosurgery, University of Utah, Salt Lake City, UT 84132, USA
| |
Collapse
|
4
|
Gabriel DB, Havugimana F, Liley AE, Aguilar I, Yeasin M, Simon NW. Lateral Orbitofrontal Cortex Encodes Presence of Risk and Subjective Risk Preference During Decision-Making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.08.588332. [PMID: 38645204 PMCID: PMC11030364 DOI: 10.1101/2024.04.08.588332] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/23/2024]
Abstract
Adaptive decision-making requires consideration of objective risks and rewards associated with each option, as well as subjective preference for risky/safe alternatives. Inaccurate risk/reward estimations can engender excessive risk-taking, a central trait in many psychiatric disorders. The lateral orbitofrontal cortex (lOFC) has been linked to many disorders associated with excessively risky behavior and is ideally situated to mediate risky decision-making. Here, we used single-unit electrophysiology to measure neuronal activity from lOFC of freely moving rats performing in a punishment-based risky decision-making task. Subjects chose between a small, safe reward and a large reward associated with either 0% or 50% risk of concurrent punishment. lOFC activity repeatedly encoded current risk in the environment throughout the decision-making sequence, signaling risk before, during, and after a choice. In addition, lOFC encoded reward magnitude, although this information was only evident during action selection. A Random Forest classifier successfully used neural data accurately to predict the risk of punishment in any given trial, and the ability to predict choice via lOFC activity differentiated between and risk-preferring and risk-averse rats. Finally, risk preferring subjects demonstrated reduced lOFC encoding of risk and increased encoding of reward magnitude. These findings suggest lOFC may serve as a central decision-making hub in which external, environmental information converges with internal, subjective information to guide decision-making in the face of punishment risk.
Collapse
Affiliation(s)
- Daniel B.K. Gabriel
- Department of Psychiatry, Indiana University School of Medicine, Indianapolis, IN 46202
| | - Felix Havugimana
- Department of Computer Engineering, University of Memphis, Memphis, TN, 38152
| | - Anna E. Liley
- Institut du Cerveau/Paris Brain Institute, Paris, France, 75013
| | - Ivan Aguilar
- Department of Psychology, University of Memphis, Memphis, TN, 38152
| | - Mohammed Yeasin
- Department of Computer Engineering, University of Memphis, Memphis, TN, 38152
| | | |
Collapse
|
5
|
Jacobs DS, Bogachuk AP, Moghaddam B. Orbitofrontal and Prelimbic Cortices Serve Complementary Roles in Adapting Reward Seeking to Learned Anxiety. Biol Psychiatry 2024:S0006-3223(24)01139-9. [PMID: 38460582 DOI: 10.1016/j.biopsych.2024.02.1015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 01/26/2024] [Accepted: 02/28/2024] [Indexed: 03/11/2024]
Abstract
BACKGROUND Anxiety is a common symptom of several mental health disorders and adversely affects motivated behaviors. Anxiety can emerge from associating risk of future harm while engaged in goal-guided actions. Using a recently developed behavioral paradigm to model this aspect of anxiety, we investigated the role of 2 cortical subregions, the prelimbic medial frontal cortex (PL) and lateral orbitofrontal cortex (lOFC), which have been implicated in anxiety and outcome expectation, in flexible representation of actions associated with harm risk. METHODS A seek-take reward-guided instrumental task design was used to train animals (N = 8) to associate the seek action with a variable risk of punishment. After learning, animals underwent extinction training for this association. Fiber photometry was used to measure and compare neuronal activity in the PL and lOFC during learning and extinction. RESULTS Animals increased action suppression in response to punishment contingencies. This increase dissipated after extinction training. These behavioral changes were associated with region-specific changes in neuronal activity. PL neuronal activity preferentially adapted to the threat of punishment, whereas lOFC activity adapted to safe aspects of the task. Moreover, correlated activity between these regions was suppressed during actions associated with harm risk, suggesting that these regions may guide behavior independently under anxiety. CONCLUSIONS These findings suggest that the PL and lOFC serve distinct but complementary roles in the representation of learned anxiety. This dissociation may provide a mechanism to explain how overlapping cortical systems are implicated in reward-guided action execution during anxiety.
Collapse
Affiliation(s)
- David S Jacobs
- Department of Behavioral Neuroscience, Oregon Health and Science University, Portland, Oregon
| | - Alina P Bogachuk
- Department of Behavioral Neuroscience, Oregon Health and Science University, Portland, Oregon
| | - Bita Moghaddam
- Department of Psychiatry, Oregon Health and Science University, Portland, Oregon; Department of Behavioral Neuroscience, Oregon Health and Science University, Portland, Oregon.
| |
Collapse
|
6
|
Simpson EH, Akam T, Patriarchi T, Blanco-Pozo M, Burgeno LM, Mohebi A, Cragg SJ, Walton ME. Lights, fiber, action! A primer on in vivo fiber photometry. Neuron 2024; 112:718-739. [PMID: 38103545 PMCID: PMC10939905 DOI: 10.1016/j.neuron.2023.11.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Revised: 10/16/2023] [Accepted: 11/15/2023] [Indexed: 12/19/2023]
Abstract
Fiber photometry is a key technique for characterizing brain-behavior relationships in vivo. Initially, it was primarily used to report calcium dynamics as a proxy for neural activity via genetically encoded indicators. This generated new insights into brain functions including movement, memory, and motivation at the level of defined circuits and cell types. Recently, the opportunity for discovery with fiber photometry has exploded with the development of an extensive range of fluorescent sensors for biomolecules including neuromodulators and peptides that were previously inaccessible in vivo. This critical advance, combined with the new availability of affordable "plug-and-play" recording systems, has made monitoring molecules with high spatiotemporal precision during behavior highly accessible. However, while opening exciting new avenues for research, the rapid expansion in fiber photometry applications has occurred without coordination or consensus on best practices. Here, we provide a comprehensive guide to help end-users execute, analyze, and suitably interpret fiber photometry studies.
Collapse
Affiliation(s)
- Eleanor H Simpson
- Department of Psychiatry, Columbia University Medical Center, New York, NY, USA; New York State Psychiatric Institute, New York, NY, USA.
| | - Thomas Akam
- Department of Experimental Psychology, University of Oxford, Oxford, UK; Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK.
| | - Tommaso Patriarchi
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland; Neuroscience Center Zürich, University and ETH Zürich, Zürich, Switzerland.
| | - Marta Blanco-Pozo
- Department of Experimental Psychology, University of Oxford, Oxford, UK; Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK
| | - Lauren M Burgeno
- Department of Experimental Psychology, University of Oxford, Oxford, UK; Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford, UK
| | - Ali Mohebi
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - Stephanie J Cragg
- Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford, UK; Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Mark E Walton
- Department of Experimental Psychology, University of Oxford, Oxford, UK; Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK
| |
Collapse
|
7
|
Venditto SJC, Miller KJ, Brody CD, Daw ND. Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.28.582617. [PMID: 38464244 PMCID: PMC10925334 DOI: 10.1101/2024.02.28.582617] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step,reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.
Collapse
|
8
|
Colas JT, O’Doherty JP, Grafton ST. Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts. PLoS Comput Biol 2024; 20:e1011950. [PMID: 38552190 PMCID: PMC10980507 DOI: 10.1371/journal.pcbi.1011950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Accepted: 02/26/2024] [Indexed: 04/01/2024] Open
Abstract
Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - John P. O’Doherty
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - Scott T. Grafton
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
| |
Collapse
|
9
|
Brown CS, Devine S, Otto AR, Bischoff-Grethe A, Wierenga CE. Greater reliance on model-free learning in adolescent anorexia nervosa: An examination of dual-system reinforcement learning. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.01.31.24302097. [PMID: 38352608 PMCID: PMC10863009 DOI: 10.1101/2024.01.31.24302097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/19/2024]
Abstract
Alterations in learning and decision-making systems are thought to contribute to core features of anorexia nervosa (AN), a psychiatric disorder characterized by persistent dietary restriction and weight loss. Instrumental learning theory identifies a dual-system of habit and goal-directed decision-making, linked to model-free and model-based reinforcement learning algorithms. Difficulty arbitrating between these systems, resulting in an over-reliance on one strategy over the other, has been implicated in compulsivity and extreme goal pursuit, both of which are observed in AN. Characterizing alterations in model-free and model-based systems, and their neural correlates, in AN may clarify mechanisms contributing to symptom heterogeneity (e.g., binge/purge symptoms). This study tested whether adolescents with restricting AN (AN-R; n = 36) and binge/purge AN (AN-BP; n = 20) differentially utilized model-based and model-free learning systems compared to a healthy control group (HC; n = 28) during a Markov two-step decision-making task under conditions of reward and punishment. Associations between model-free and model-based learning and resting-state functional connectivity between neural regions of interest, including orbitofrontal cortex (OFC), nucleus accumbens (NAcc), putamen, and sensory motor cortex (SMC) were examined. AN-R showed higher utilization of model-free learning compared to HC for reward, but attenuated model-free and model-based learning for punishment. In AN-R only, higher model-based learning was associated with stronger OFC-to-left NAcc functional connectivity, regions linked to goal-directed behavior. Greater utilization of model-free learning for reward in AN-R may differentiate this group, particularly during adolescence, and facilitate dietary restriction by prioritizing habitual control in rewarding contexts.
Collapse
Affiliation(s)
- Carina S. Brown
- San Diego State University/University of California San Diego Joint Doctoral Program in Clinical Psychology
- Department of Psychiatry, University of California, San Diego
| | | | | | | | - Christina E. Wierenga
- San Diego State University/University of California San Diego Joint Doctoral Program in Clinical Psychology
- Department of Psychiatry, University of California, San Diego
| |
Collapse
|
10
|
Guidera JA, Gramling DP, Comrie AE, Joshi A, Denovellis EL, Lee KH, Zhou J, Thompson P, Hernandez J, Yorita A, Haque R, Kirst C, Frank LM. Regional specialization manifests in the reliability of neural population codes. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.01.25.576941. [PMID: 38328245 PMCID: PMC10849741 DOI: 10.1101/2024.01.25.576941] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/09/2024]
Abstract
The brain has the remarkable ability to learn and guide the performance of complex tasks. Decades of lesion studies suggest that different brain regions perform specialized functions in support of complex behaviors1-3. Yet recent large-scale studies of neural activity reveal similar patterns of activity and encoding distributed widely throughout the brain4-6. How these distributed patterns of activity and encoding are compatible with regional specialization of brain function remains unclear. Two frontal brain regions, the dorsal medial prefrontal cortex (dmPFC) and orbitofrontal cortex (OFC), are a paradigm of this conundrum. In the setting complex behaviors, the dmPFC is necessary for choosing optimal actions2,7,8, whereas the OFC is necessary for waiting for3,9 and learning from2,7,9-12 the outcomes of those actions. Yet both dmPFC and OFC encode both choice- and outcome-related quantities13-20. Here we show that while ensembles of neurons in the dmPFC and OFC of rats encode similar elements of a cognitive task with similar patterns of activity, the two regions differ in when that coding is consistent across trials ("reliable"). In line with the known critical functions of each region, dmPFC activity is more reliable when animals are making choices and less reliable preceding outcomes, whereas OFC activity shows the opposite pattern. Our findings identify the dynamic reliability of neural population codes as a mechanism whereby different brain regions may support distinct cognitive functions despite exhibiting similar patterns of activity and encoding similar quantities.
Collapse
Affiliation(s)
- Jennifer A. Guidera
- UCSF-UC Berkeley Graduate Program in Bioengineering, University of California, San Francisco; San Francisco, 94158, USA and University of California, Berkeley; Berkely, 94720, USA
- Medical Scientist Training Program, University of California, San Francisco; San Francisco, 94158, USA
| | - Daniel P. Gramling
- Departments of Physiology and Psychiatry, University of California, San Francisco; San Francisco, 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
| | - Alison E. Comrie
- Departments of Physiology and Psychiatry, University of California, San Francisco; San Francisco, 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
| | - Abhilasha Joshi
- Departments of Physiology and Psychiatry, University of California, San Francisco; San Francisco, 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
- Howard Hughes Medical Institute, University of California, San Francisco; San Francisco, 94158, USA
| | - Eric L. Denovellis
- Departments of Physiology and Psychiatry, University of California, San Francisco; San Francisco, 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
- Howard Hughes Medical Institute, University of California, San Francisco; San Francisco, 94158, USA
| | - Kyu Hyun Lee
- Departments of Physiology and Psychiatry, University of California, San Francisco; San Francisco, 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
- Howard Hughes Medical Institute, University of California, San Francisco; San Francisco, 94158, USA
| | - Jenny Zhou
- Center for Micro- and Nano-Technology, Lawrence Livermore National Laboratory; Livermore, 94158, USA
| | - Paige Thompson
- Center for Micro- and Nano-Technology, Lawrence Livermore National Laboratory; Livermore, 94158, USA
| | - Jose Hernandez
- Center for Micro- and Nano-Technology, Lawrence Livermore National Laboratory; Livermore, 94158, USA
| | - Allison Yorita
- Center for Micro- and Nano-Technology, Lawrence Livermore National Laboratory; Livermore, 94158, USA
| | - Razi Haque
- Center for Micro- and Nano-Technology, Lawrence Livermore National Laboratory; Livermore, 94158, USA
| | - Christoph Kirst
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
- Department of Anatomy, University of California, San Francisco; San Francisco, 94158, USA
| | - Loren M. Frank
- UCSF-UC Berkeley Graduate Program in Bioengineering, University of California, San Francisco; San Francisco, 94158, USA and University of California, Berkeley; Berkely, 94720, USA
- Departments of Physiology and Psychiatry, University of California, San Francisco; San Francisco, 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco; San Francisco, 94158, USA
- Howard Hughes Medical Institute, University of California, San Francisco; San Francisco, 94158, USA
| |
Collapse
|
11
|
Mahmoodi A, Harbison C, Bongioanni A, Emberton A, Roumazeilles L, Sallet J, Khalighinejad N, Rushworth MFS. A frontopolar-temporal circuit determines the impact of social information in macaque decision making. Neuron 2024; 112:84-92.e6. [PMID: 37863039 PMCID: PMC10914637 DOI: 10.1016/j.neuron.2023.09.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2023] [Revised: 07/06/2023] [Accepted: 09/26/2023] [Indexed: 10/22/2023]
Abstract
When choosing, primates are guided not only by personal experience of objects but also by social information such as others' attitudes toward the objects. Crucially, both sources of information-personal and socially derived-vary in reliability. To choose optimally, one must sometimes override choice guidance by personal experience and follow social cues instead, and sometimes one must do the opposite. The dorsomedial frontopolar cortex (dmFPC) tracks reliability of social information and determines whether it will be attended to guide behavior. To do this, dmFPC activity enters specific patterns of interaction with a region in the mid-superior temporal sulcus (mSTS). Reversible disruption of dmFPC activity with transcranial ultrasound stimulation (TUS) led macaques to fail to be guided by social information when it was reliable but to be more likely to use it when it was unreliable. By contrast, mSTS disruption uniformly downregulated the impact of social information on behavior.
Collapse
Affiliation(s)
- Ali Mahmoodi
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK.
| | - Caroline Harbison
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Alessandro Bongioanni
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK; Cognitive Neuroimaging Unit, CEA, INSERM, Université Paris-Saclay, NeuroSpin Center, 91191 Gif/Yvette, France
| | - Andrew Emberton
- Department of Biomedical Services, University of Oxford, Oxford, UK
| | - Lea Roumazeilles
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Jerome Sallet
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK; Univ Lyon, Université Lyon 1, Inserm, Stem Cell and Brain Research Institute, U1208 Bron, France
| | - Nima Khalighinejad
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Matthew F S Rushworth
- Wellcome Centre for Integrative Neuroimaging, Department of Experimental Psychology, University of Oxford, Oxford, UK
| |
Collapse
|
12
|
Hoy CW, de Hemptinne C, Wang SS, Harmer CJ, Apps MAJ, Husain M, Starr PA, Little S. Beta and theta oscillations track effort and previous reward in human basal ganglia and prefrontal cortex during decision making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.12.05.570285. [PMID: 38106063 PMCID: PMC10723308 DOI: 10.1101/2023.12.05.570285] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/19/2023]
Abstract
Choosing whether to exert effort to obtain rewards is fundamental to human motivated behavior. However, the neural dynamics underlying the evaluation of reward and effort in humans is poorly understood. Here, we investigate this with chronic intracranial recordings from prefrontal cortex (PFC) and basal ganglia (BG; subthalamic nuclei and globus pallidus) in people with Parkinson's disease performing a decision-making task with offers that varied in levels of reward and physical effort required. This revealed dissociable neural signatures of reward and effort, with BG beta (12-20 Hz) oscillations tracking subjective effort on a single trial basis and PFC theta (4-7 Hz) signaling previous trial reward. Stimulation of PFC increased overall acceptance of offers in addition to increasing the impact of reward on choices. This work uncovers oscillatory mechanisms that guide fundamental decisions to exert effort for reward across BG and PFC, as well as supporting a causal role of PFC for such choices.
Collapse
Affiliation(s)
- Colin W. Hoy
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - Coralie de Hemptinne
- Norman Fixel Institute for Neurological Diseases, University of Florida, Gainesville, FL, USA
- Department of Neurology, University of Florida, Gainesville, FL, USA
| | - Sarah S. Wang
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | | | - Mathew A. J. Apps
- Department of Experimental Psychology, University of Oxford, Oxford, UK
- Institute for Mental Health, School of Psychology, University of Birmingham, Birmingham, UK
| | - Masud Husain
- Department of Experimental Psychology, University of Oxford, Oxford, UK
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Philip A. Starr
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Simon Little
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| |
Collapse
|
13
|
Oyama K, Majima K, Nagai Y, Hori Y, Hirabayashi T, Eldridge MAG, Mimura K, Miyakawa N, Fujimoto A, Hori Y, Iwaoki H, Inoue KI, Saunders RC, Takada M, Yahata N, Higuchi M, Richmond BJ, Minamimoto T. Distinct roles of monkey OFC-subcortical pathways in adaptive behavior. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.17.567492. [PMID: 38076986 PMCID: PMC10705585 DOI: 10.1101/2023.11.17.567492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/22/2023]
Abstract
To be the most successful, primates must adapt to changing environments and optimize their behavior by making the most beneficial choices. At the core of adaptive behavior is the orbitofrontal cortex (OFC) of the brain, which updates choice value through direct experience or knowledge-based inference. Here, we identify distinct neural circuitry underlying these two separate abilities. We designed two behavioral tasks in which macaque monkeys updated the values of certain items, either by directly experiencing changes in stimulus-reward associations, or by inferring the value of unexperienced items based on the task's rules. Chemogenetic silencing of bilateral OFC combined with mathematical model-fitting analysis revealed that monkey OFC is involved in updating item value based on both experience and inference. In vivo imaging of chemogenetic receptors by positron emission tomography allowed us to map projections from the OFC to the rostromedial caudate nucleus (rmCD) and the medial part of the mediodorsal thalamus (MDm). Chemogenetic silencing of the OFC-rmCD pathway impaired experience-based value updating, while silencing the OFC-MDm pathway impaired inference-based value updating. Our results thus demonstrate a dissociable contribution of distinct OFC projections to different behavioral strategies, and provide new insights into the neural basis of value-based adaptive decision-making in primates.
Collapse
Affiliation(s)
- Kei Oyama
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
- PRESTO, Japan Science and Technology Agency, Kawaguchi, Japan
| | - Kei Majima
- Institute for Quantum Life Science, National Institutes for Quantum Science and Technology, Chiba, Japan
- PRESTO, Japan Science and Technology Agency, Kawaguchi, Japan
| | - Yuji Nagai
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Yukiko Hori
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Toshiyuki Hirabayashi
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, USA
| | - Koki Mimura
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
- Research Center for Medical and Health Data Science, The Institute of Statistical Mathematics, Tachikawa, Japan
| | - Naohisa Miyakawa
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Atsushi Fujimoto
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Yuki Hori
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Haruhiko Iwaoki
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Ken-Ichi Inoue
- Systems Neuroscience Section, Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama, Japan
| | - Richard C Saunders
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, USA
| | - Masahiko Takada
- Systems Neuroscience Section, Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama, Japan
| | - Noriaki Yahata
- Institute for Quantum Life Science, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Makoto Higuchi
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, USA
| | - Takafumi Minamimoto
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| |
Collapse
|
14
|
Kim SJ, Affan RO, Frostig H, Scott BB, Alexander AS. Advances in cellular resolution microscopy for brain imaging in rats. NEUROPHOTONICS 2023; 10:044304. [PMID: 38076724 PMCID: PMC10704261 DOI: 10.1117/1.nph.10.4.044304] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 09/23/2023] [Accepted: 11/07/2023] [Indexed: 02/12/2024]
Abstract
Rats are used in neuroscience research because of their physiological similarities with humans and accessibility as model organisms, trainability, and behavioral repertoire. In particular, rats perform a wide range of sophisticated social, cognitive, motor, and learning behaviors within the contexts of both naturalistic and laboratory environments. Further progress in neuroscience can be facilitated by using advanced imaging methods to measure the complex neural and physiological processes during behavior in rats. However, compared with the mouse, the rat nervous system offers a set of challenges, such as larger brain size, decreased neuron density, and difficulty with head restraint. Here, we review recent advances in in vivo imaging techniques in rats with a special focus on open-source solutions for calcium imaging. Finally, we provide suggestions for both users and developers of in vivo imaging systems for rats.
Collapse
Affiliation(s)
- Su Jin Kim
- Johns Hopkins University, Department of Psychological and Brain Sciences, Baltimore, Maryland, United States
| | - Rifqi O. Affan
- Boston University, Center for Systems Neuroscience, Department of Psychological and Brain Sciences, Boston, Massachusetts, United States
- Boston University, Graduate Program in Neuroscience, Boston, Massachusetts, United States
| | - Hadas Frostig
- Boston University, Center for Systems Neuroscience, Department of Psychological and Brain Sciences, Boston, Massachusetts, United States
| | - Benjamin B. Scott
- Boston University, Center for Systems Neuroscience, Department of Psychological and Brain Sciences, Boston, Massachusetts, United States
- Boston University, Neurophotonics Center and Photonics Center, Boston, Massachusetts, United States
| | - Andrew S. Alexander
- University of California Santa Barbara, Department of Psychological and Brain Sciences, Santa Barbara, California, United States
| |
Collapse
|
15
|
Mehrotra D, Dubé L. Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus. Front Neurosci 2023; 17:1200842. [PMID: 37732307 PMCID: PMC10508350 DOI: 10.3389/fnins.2023.1200842] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 08/25/2023] [Indexed: 09/22/2023] Open
Abstract
For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.
Collapse
Affiliation(s)
- Dhruv Mehrotra
- Integrated Program in Neuroscience, McGill University, Montréal, QC, Canada
- Montréal Neurological Institute, McGill University, Montréal, QC, Canada
| | - Laurette Dubé
- Desautels Faculty of Management, McGill University, Montréal, QC, Canada
- McGill Center for the Convergence of Health and Economics, McGill University, Montréal, QC, Canada
| |
Collapse
|
16
|
Costa KM, Scholz R, Lloyd K, Moreno-Castilla P, Gardner MPH, Dayan P, Schoenbaum G. The role of the lateral orbitofrontal cortex in creating cognitive maps. Nat Neurosci 2023; 26:107-115. [PMID: 36550290 PMCID: PMC9839657 DOI: 10.1038/s41593-022-01216-0] [Citation(s) in RCA: 18] [Impact Index Per Article: 18.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2022] [Accepted: 10/26/2022] [Indexed: 12/24/2022]
Abstract
We use mental models of the world-cognitive maps-to guide behavior. The lateral orbitofrontal cortex (lOFC) is typically thought to support behavior by deploying these maps to simulate outcomes, but recent evidence suggests that it may instead support behavior by underlying map creation. We tested between these two alternatives using outcome-specific devaluation and a high-potency chemogenetic approach. Selectively inactivating lOFC principal neurons when male rats learned distinct cue-outcome associations, but before outcome devaluation, disrupted subsequent inference, confirming a role for the lOFC in creating new maps. However, lOFC inactivation surprisingly led to generalized devaluation, a result that is inconsistent with a complete mapping failure. Using a reinforcement learning framework, we show that this effect is best explained by a circumscribed deficit in credit assignment precision during map construction, suggesting that the lOFC has a selective role in defining the specificity of associations that comprise cognitive maps.
Collapse
Affiliation(s)
- Kauê Machado Costa
- National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, MD, USA.
| | - Robert Scholz
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
- Max Planck School of Cognition, Leipzig, Germany
| | - Kevin Lloyd
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Perla Moreno-Castilla
- National Institute on Aging Intramural Research Program, National Institutes of Health, Baltimore, MD, USA
| | | | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
- University of Tübingen, Tübingen, Germany
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, MD, USA.
| |
Collapse
|