1
|
Zid M, Laurie VJ, Levine-Champagne A, Shourkeshti A, Harrell D, Herman AB, Ebitz RB. Humans forage for reward in reinforcement learning tasks. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.07.08.602539. [PMID: 39026817 PMCID: PMC11257465 DOI: 10.1101/2024.07.08.602539] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/20/2024]
Abstract
How do we make good decisions in uncertain environments? In psychology and neuroscience, the classic answer is that we calculate the value of each option and then compare the values to choose the most rewarding, modulo some exploratory noise. An ethologist, conversely, would argue that we commit to one option until its value drops below a threshold, at which point we start exploring other options. In order to determine which view better describes human decision-making, we developed a novel, foraging-inspired sequential decision-making model and used it to ask whether humans compare to threshold ("Forage") or compare alternatives ("Reinforcement-Learn" [RL]). We found that the foraging model was a better fit for participant behavior, better predicted the participants' tendency to repeat choices, and predicted the existence of held-out participants with a pattern of choice that was almost impossible under RL. Together, these results suggest that humans use foraging computations, rather than RL, even in classic reinforcement learning tasks.
Collapse
Affiliation(s)
- Meriam Zid
- Department of Neuroscience, University of Montreal, Montreal, QC , H3T 1J4, Canada
| | - Veldon-James Laurie
- Department of Neuroscience, University of Montreal, Montreal, QC , H3T 1J4, Canada
| | | | - Akram Shourkeshti
- Department of Neuroscience, University of Montreal, Montreal, QC , H3T 1J4, Canada
| | - Dameon Harrell
- Department of Psychiatry, University of Minnesota, Minneapolis, MN, 55455, USA
| | - Alexander B. Herman
- Department of Psychiatry, University of Minnesota, Minneapolis, MN, 55455, USA
| | - R. Becket Ebitz
- Department of Neuroscience, University of Montreal, Montreal, QC , H3T 1J4, Canada
| |
Collapse
|
2
|
Pribut HJ, Kang N, Roesch MR. Prior cocaine self-administration does not impair the ability to delay gratification in rats during diminishing returns. Behav Pharmacol 2024; 35:147-155. [PMID: 38651979 DOI: 10.1097/fbp.0000000000000771] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/25/2024]
Abstract
Previous exposure to drugs of abuse produces impairments in studies of reversal learning, delay discounting and response inhibition tasks. While these studies contribute to the understanding of normal decision-making and how it is impaired by drugs of abuse, they do not fully capture how decision-making impacts the ability to delay gratification for greater long-term benefit. To address this issue, we used a diminishing returns task to study decision-making in rats that had previously self-administered cocaine. This task was designed to test the ability of the rat to choose to delay gratification in the short-term to obtain more reward over the course of the entire behavioral session. Rats were presented with two choices. One choice had a fixed amount of time delay needed to obtain reward [i.e. fixed delay (FD)], while the other choice had a progressive delay (PD) that started at 0 s and progressively increased by 1 s each time the PD option was selected. During the 'reset' variation of the task, rats could choose the FD option to reset the time delay associated with the PD option. Consistent with previous results, we found that prior cocaine exposure reduced rats' overall preference for the PD option in post-task reversal testing during 'no-reset' sessions, suggesting that cocaine exposure made rats more sensitive to the increasing delay of the PD option. Surprisingly, however, we found that rats that had self-administered cocaine 1-month prior, adapted behavior during 'reset' sessions by delaying gratification to obtain more reward in the long run similar to control rats.
Collapse
Affiliation(s)
- H J Pribut
- Department of Psychology
- Program in Neuroscience and Cognitive Science, University of Maryland, College Park, Maryland, USA
| | | | - Matthew R Roesch
- Department of Psychology
- Program in Neuroscience and Cognitive Science, University of Maryland, College Park, Maryland, USA
| |
Collapse
|
3
|
Grissom NM, Glewwe N, Chen C, Giglio E. Sex mechanisms as nonbinary influences on cognitive diversity. Horm Behav 2024; 162:105544. [PMID: 38643533 DOI: 10.1016/j.yhbeh.2024.105544] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 04/09/2024] [Accepted: 04/10/2024] [Indexed: 04/23/2024]
Abstract
Essentially all neuropsychiatric diagnoses show some degree of sex and/or gender differences in their etiology, diagnosis, or prognosis. As a result, the roles of sex-related variables in behavior and cognition are of strong interest to many, with several lines of research showing effects on executive functions and value-based decision making in particular. These findings are often framed within a sex binary, with behavior of females described as less optimal than male "defaults"-- a framing that pits males and females against each other and deemphasizes the enormous overlap in fundamental neural mechanisms across sexes. Here, we propose an alternative framework in which sex-related factors encompass just one subset of many sources of valuable diversity in cognition. First, we review literature establishing multidimensional, nonbinary impacts of factors related to sex chromosomes and endocrine mechanisms on cognition, focusing on value- based decision-making tasks. Next, we present two suggestions for nonbinary interpretations and analyses of sex-related data that can be implemented by behavioral neuroscientists without devoting laboratory resources to delving into mechanisms underlying sex differences. We recommend (1) shifting interpretations of behavior away from performance metrics and towards strategy assessments to avoid the fallacy that the performance of one sex is worse than another; and (2) asking how much variance sex explains in measures and whether any differences are mosaic rather than binary, to avoid assuming that sex differences in separate measures are inextricably correlated. Nonbinary frameworks in research on cognition will allow neuroscience to represent the full spectrum of brains and behaviors.
Collapse
Affiliation(s)
- Nicola M Grissom
- Department of Psychology, University of Minnesota, United States of America.
| | - Nic Glewwe
- Department of Psychology, University of Minnesota, United States of America
| | - Cathy Chen
- Department of Psychiatry and Behavioral Sciences, University of Minnesota, United States of America
| | - Erin Giglio
- Department of Psychology, University of Minnesota, United States of America
| |
Collapse
|
4
|
Hong I, Wolfe JM. Research on re-searching: interrupted foraging is not disrupted foraging. Cogn Res Princ Implic 2024; 9:30. [PMID: 38748189 PMCID: PMC11096138 DOI: 10.1186/s41235-024-00556-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2024] [Accepted: 04/26/2024] [Indexed: 05/18/2024] Open
Abstract
In classic visual search, observers typically search for the presence of a target in a scene or display. In foraging tasks, there may be multiple targets in the same display (or "patch"). Observers typically search for and collect these target items in one patch until they decide to leave that patch and move to the next one. This is a highly rule-governed behavior. The current study investigated whether these rules are disrupted when the foraging is interrupted in various manners. In Experiment 1, the foraging was briefly interrupted and then resumed in the same patch. In Experiments 2 and 3, the foraging in each patch either ended voluntarily or compulsorily after a fixed amount of time. In these cases, foraging resumed in a patch only after all patches were visited. Overall, the rules of foraging remained largely intact, though Experiment 2 shows that foraging rules can be overridden by the demand characteristics of the task. The results show that participants tended to perform approximately consistently despite interruptions. The results suggest that foraging behavior in a relatively simple foraging environment is resilient and not easily disrupted by interruption.
Collapse
Affiliation(s)
- Injae Hong
- Visual Attention Lab, Brigham and Women's Hospital, Boston, MA, 02135, USA
- Harvard Medical School, Boston, USA
- Yonsei University, Seoul, South Korea
| | - Jeremy M Wolfe
- Visual Attention Lab, Brigham and Women's Hospital, Boston, MA, 02135, USA.
- Harvard Medical School, Boston, USA.
| |
Collapse
|
5
|
Alejandro RJ, Holroyd CB. Hierarchical control over foraging behavior by anterior cingulate cortex. Neurosci Biobehav Rev 2024; 160:105623. [PMID: 38490499 DOI: 10.1016/j.neubiorev.2024.105623] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 02/14/2024] [Accepted: 03/13/2024] [Indexed: 03/17/2024]
Abstract
Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).
Collapse
Affiliation(s)
| | - Clay B Holroyd
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| |
Collapse
|
6
|
González VV, Zhang Y, Ashikyan SA, Rickard A, Yassine I, Romero-Sosa JL, Blaisdell AP, Izquierdo A. A special role for anterior cingulate cortex, but not orbitofrontal cortex or basolateral amygdala, in choices involving information. Cereb Cortex 2024; 34:bhae135. [PMID: 38610085 PMCID: PMC11014886 DOI: 10.1093/cercor/bhae135] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 02/09/2024] [Accepted: 03/13/2024] [Indexed: 04/14/2024] Open
Abstract
Subjects are often willing to pay a cost for information. In a procedure that promotes paradoxical choices, animals choose between a richer option followed by a cue that is rewarded 50% of the time (No Info) vs. a leaner option followed by one of two cues that signal certain outcomes: one always rewarded (100%) and the other never rewarded, 0% (Info). Since decisions involve comparing the subjective value of options after integrating all their features, preference for information may rely on cortico-amygdalar circuitry. To test this, male and female rats were prepared with bilateral inhibitory Designer Receptors Exclusively Activated by Designer Drugs (DREADDs) in the anterior cingulate cortex, orbitofrontal cortex, basolateral amygdala, or null virus (control). We inhibited these regions after stable preference was acquired. We found that inhibition of the anterior cingulate cortex destabilized choice preference in female rats without affecting latency to choose or response rate to cues. A logistic regression fit revealed that previous choice predicted current choice in all conditions, however previously rewarded Info trials strongly predicted preference in all conditions except in female rats following anterior cingulate cortex inhibition. The results reveal a causal, sex-dependent role for the anterior cingulate cortex in decisions involving information.
Collapse
Affiliation(s)
- Valeria V González
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
| | - Yifan Zhang
- Department of Computer Science, University of Southern California, Salvatori Computer Science Center, 941 Bloom Walk, Los Angeles, CA 90089, United States
| | - Sonya A Ashikyan
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
| | - Anne Rickard
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
| | - Ibrahim Yassine
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
| | - Juan Luis Romero-Sosa
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
| | - Aaron P Blaisdell
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
- The Brain Research Institute, University of California-Los Angeles, 695 Charles E Young Dr S, Los Angeles, CA 90095, United States
- Integrative Center for Learning and Memory, University of California-Los Angeles, 695 Charles E Young Dr S, Los Angeles, CA 90095, United States
| | - Alicia Izquierdo
- Department of Psychology, University of California-Los Angeles, 502 Portola Plaza, Los Angeles, CA 90095, United States
- The Brain Research Institute, University of California-Los Angeles, 695 Charles E Young Dr S, Los Angeles, CA 90095, United States
- Integrative Center for Learning and Memory, University of California-Los Angeles, 695 Charles E Young Dr S, Los Angeles, CA 90095, United States
- Integrative Center for Addictions, University of California-Los Angeles, 695 Charles E Young Dr S, Los Angeles, CA 90095, United States
| |
Collapse
|
7
|
Mou X, Ji D. Observational activation of anterior cingulate cortical neurons coordinates hippocampal replay in social learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.31.587484. [PMID: 38617364 PMCID: PMC11014478 DOI: 10.1101/2024.03.31.587484] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/16/2024]
Abstract
Social learning enables a subject to make decisions by observing the actions of another. How neural circuits acquire relevant information during observation to guide subsequent behavior is unknown. Utilizing an observational spatial working memory task, we show that neurons in the rat anterior cingulate cortex (ACC) associated with spatial trajectories during self-running in a maze are activated when observing another rat running the same maze. The observation-induced ACC activities are reduced in error trials and are correlated with activities of hippocampal place cells representing the same trajectories. The ACC activities during observation also predict subsequent hippocampal place cell activities during sharp-wave ripples and spatial contents of hippocampal replay prior to self-running. The results support that ACC neurons involved in decisions during self-running are reactivated during observation and coordinate hippocampal replay to guide subsequent spatial navigation.
Collapse
Affiliation(s)
- Xiang Mou
- Department of Neuroscience, Baylor College of Medicine; One Baylor Plaza, Houston, TX 77030, USA
| | - Daoyun Ji
- Department of Neuroscience, Baylor College of Medicine; One Baylor Plaza, Houston, TX 77030, USA
- Department of Molecular and Cellular Biology, Baylor College of Medicine; One Baylor Plaza, Houston, TX 77030, USA
- Lead Contact
| |
Collapse
|
8
|
Webb J, Steffan P, Hayden BY, Lee D, Kemere C, McGinley M. Foraging Under Uncertainty Follows the Marginal Value Theorem with Bayesian Updating of Environment Representations. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.30.587253. [PMID: 38585964 PMCID: PMC10996644 DOI: 10.1101/2024.03.30.587253] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Foraging theory has been a remarkably successful approach to understanding the behavior of animals in many contexts. In patch-based foraging contexts, the marginal value theorem (MVT) shows that the optimal strategy is to leave a patch when the marginal rate of return declines to the average for the environment. However, the MVT is only valid in deterministic environments whose statistics are known to the forager; naturalistic environments seldom meet these strict requirements. As a result, the strategies used by foragers in naturalistic environments must be empirically investigated. We developed a novel behavioral task and a corresponding computational framework for studying patch-leaving decisions in head-fixed and freely moving mice. We varied between-patch travel time, as well as within-patch reward depletion rate, both deterministically and stochastically. We found that mice adopt patch residence times in a manner consistent with the MVT and not explainable by simple ethologically motivated heuristic strategies. Critically, behavior was best accounted for by a modified form of the MVT wherein environment representations were updated based on local variations in reward timing, captured by a Bayesian estimator and dynamic prior. Thus, we show that mice can strategically attend to, learn from, and exploit task structure on multiple timescales simultaneously, thereby efficiently foraging in volatile environments. The results provide a foundation for applying the systems neuroscience toolkit in freely moving and head-fixed mice to understand the neural basis of foraging under uncertainty.
Collapse
Affiliation(s)
- James Webb
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA
- Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, TX, USA
| | - Paul Steffan
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA
| | - Benjamin Y. Hayden
- Department of Neurosurgery, Baylor College of Medicine, Houston, TX, USA
| | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, The Solomon H Snyder Department of Neuroscience, Department of Psychological and Brain Sciences, Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, MD, USA
| | - Caleb Kemere
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA
| | - Matthew McGinley
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, USA
- Jan and Dan Duncan Neurological Research Institute, Texas Children’s Hospital, Houston, TX, USA
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA
| |
Collapse
|
9
|
Sazhin D, Dachs A, Smith DV. Meta-Analysis Reveals That Explore-Exploit Decisions are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex and the Anterior Cingulate Cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.10.21.563317. [PMID: 37961286 PMCID: PMC10634720 DOI: 10.1101/2023.10.21.563317] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2023]
Abstract
Explore-exploit research has challenges in generalizability due to a limited theoretical basis of exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions use an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N=23 studies) where we found activation in the dorsal lateral prefrontal cortex and anterior cingulate cortex during exploration versus exploitation, providing some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the dorsal anterior cingulate cortex, dorsal medial prefrontal cortex, and anterior insula, suggesting that these brain regions do not engage in opponent processing. Further, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation in exploration and exploitation decisions is generally more similar than it is different suggests there remain significant challenges toward characterizing explore-exploit decision making. Nonetheless, dlPFC and ACC activation differentiate explore and exploit decisions and identifying these responses can help in targeted interventions aimed at manipulating these decisions.
Collapse
|
10
|
González VV, Ashikyan SA, Zhang Y, Rickard A, Yassine I, Romero-Sosa JL, Blaisdell AP, Izquierdo A. A special role for anterior cingulate cortex, but not orbitofrontal cortex or basolateral amygdala, in choices involving information. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.03.551514. [PMID: 37577596 PMCID: PMC10418268 DOI: 10.1101/2023.08.03.551514] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Subjects often are willing to pay a cost for information. In a procedure that promotes paradoxical choices, animals choose between a richer option followed by a cue that is rewarded 50% of the time (No-info) vs a leaner option followed by one of two cues that signal certain outcomes: one always rewarded (100%), and the other never rewarded, 0% (Info). Since decisions involve comparing the subjective value of options after integrating all their features, preference for information may rely on cortico-amygdalar circuitry. To test this, male and female rats were prepared with bilateral inhibitory DREADDs in the anterior cingulate cortex (ACC), orbitofrontal cortex (OFC), basolateral amygdala (BLA), or null virus (control). We inhibited these regions after stable preference was acquired. We found that inhibition of ACC destabilized choice preference in female rats without affecting latency to choose or response rate to cues. A logistic regression fit revealed that the previous choice strongly predicted preference in control animals, but not in female rats following ACC inhibition. The results reveal a causal, sex-dependent role for ACC in decisions involving information.
Collapse
|
11
|
Bustamante LA, Oshinowo T, Lee JR, Tong E, Burton AR, Shenhav A, Cohen JD, Daw ND. Effort Foraging Task reveals positive correlation between individual differences in the cost of cognitive and physical effort in humans. Proc Natl Acad Sci U S A 2023; 120:e2221510120. [PMID: 38064507 PMCID: PMC10723129 DOI: 10.1073/pnas.2221510120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2023] [Accepted: 10/26/2023] [Indexed: 12/17/2023] Open
Abstract
Effort-based decisions, in which people weigh potential future rewards against effort costs required to achieve those rewards involve both cognitive and physical effort, though the mechanistic relationship between them is not yet understood. Here, we use an individual differences approach to isolate and measure the computational processes underlying effort-based decisions and test the association between cognitive and physical domains. Patch foraging is an ecologically valid reward rate maximization problem with well-developed theoretical tools. We developed the Effort Foraging Task, which embedded cognitive or physical effort into patch foraging, to quantify the cost of both cognitive and physical effort indirectly, by their effects on foraging choices. Participants chose between harvesting a depleting patch, or traveling to a new patch that was costly in time and effort. Participants' exit thresholds (reflecting the reward they expected to receive by harvesting when they chose to travel to a new patch) were sensitive to cognitive and physical effort demands, allowing us to quantify the perceived effort cost in monetary terms. The indirect sequential choice style revealed effort-seeking behavior in a minority of participants (preferring high over low effort) that has apparently been missed by many previous approaches. Individual differences in cognitive and physical effort costs were positively correlated, suggesting that these are perceived and processed in common. We used canonical correlation analysis to probe the relationship of task measures to self-reported affect and motivation, and found correlations of cognitive effort with anxiety, cognitive function, behavioral activation, and self-efficacy, but no similar correlations with physical effort.
Collapse
Affiliation(s)
- Laura A. Bustamante
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
- Department of Psychological and Brain Sciences, Washington University in Saint Louis, Saint Louis, MO63130
| | - Temitope Oshinowo
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - Jeremy R. Lee
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - Elizabeth Tong
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - Allison R. Burton
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - Amitai Shenhav
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
- Carney Institute for Brain Science, Brown University, Providence, RI02906
| | - Jonathan D. Cohen
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| | - Nathaniel D. Daw
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ08544
| |
Collapse
|
12
|
Lloyd A, Viding E, McKay R, Furl N. Understanding patch foraging strategies across development. Trends Cogn Sci 2023; 27:1085-1098. [PMID: 37500422 DOI: 10.1016/j.tics.2023.07.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 07/05/2023] [Accepted: 07/06/2023] [Indexed: 07/29/2023]
Abstract
Patch foraging is a near-ubiquitous behaviour across the animal kingdom and characterises many decision-making domains encountered by humans. We review how a disposition to explore in adolescence may reflect the evolutionary conditions under which hunter-gatherers foraged for resources. We propose that neurocomputational mechanisms responsible for reward processing, learning, and cognitive control facilitate the transition from exploratory strategies in adolescence to exploitative strategies in adulthood - where individuals capitalise on known resources. This developmental transition may be disrupted by psychopathology, as there is emerging evidence of biases in explore/exploit choices in mental health problems. Explore/exploit choices may be an informative marker for mental health across development and future research should consider this feature of decision-making as a target for clinical intervention.
Collapse
Affiliation(s)
- Alex Lloyd
- Clinical, Educational, and Health Psychology, Psychology and Language Sciences, University College London, 26 Bedford Way, London, WC1H 0AP, UK.
| | - Essi Viding
- Clinical, Educational, and Health Psychology, Psychology and Language Sciences, University College London, 26 Bedford Way, London, WC1H 0AP, UK
| | - Ryan McKay
- Department of Psychology, Royal Holloway, University of London, Egham Hill, Egham, TW20 0EX, UK
| | - Nicholas Furl
- Department of Psychology, Royal Holloway, University of London, Egham Hill, Egham, TW20 0EX, UK
| |
Collapse
|
13
|
Garcia M, Gupta S, Wikenheiser AM. Sex differences in patch-leaving foraging decisions in rats. OXFORD OPEN NEUROSCIENCE 2023; 2:kvad011. [PMID: 38596244 PMCID: PMC11003400 DOI: 10.1093/oons/kvad011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 10/11/2023] [Accepted: 10/12/2023] [Indexed: 04/11/2024]
Abstract
The ubiquity, importance, and sophistication of foraging behavior makes it an ideal platform for studying naturalistic decision making in animals. We developed a spatial patch-foraging task for rats, in which subjects chose how long to remain in one foraging patch as the rate of food earnings steadily decreased. The cost of seeking out a new location was varied across sessions. The behavioral task was designed to mimic the structure of natural foraging problems, where distinct spatial locations are associated with different reward statistics, and decisions require navigation and movement through space. Male and female Long-Evans rats generally followed the predictions of theoretical models of foraging, albeit with a consistent tendency to persist with patches for too long compared to behavioral strategies that maximize food intake rate. The tendency to choose overly-long patch residence times was stronger in male rats. We also observed sex differences in locomotion as rats performed the task, but these differences in movement only partially accounted for the differences in patch residence durations observed between male and female rats. Together, these results suggest a nuanced relationship between movement, sex, and foraging decisions.
Collapse
Affiliation(s)
- Marissa Garcia
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Sukriti Gupta
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Andrew M Wikenheiser
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA
- Brain Research Institute, University of California, Los Angeles, Los Angeles, CA 90095, USA
| |
Collapse
|
14
|
Garcia M, Gupta S, Wikenheiser AM. Sex differences in patch-leaving foraging decisions in rats. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.02.19.529135. [PMID: 36824852 PMCID: PMC9949151 DOI: 10.1101/2023.02.19.529135] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/21/2023]
Abstract
The ubiquity, importance, and sophistication of foraging behavior makes it an ideal platform for studying naturalistic decision making in animals. We developed a spatial patch-foraging task for rats, in which subjects chose how long to remain in one foraging patch as the rate of food earnings steadily decreased. The cost of seeking out a new location was varied across sessions. The behavioral task was designed to mimic the structure of natural foraging problems, where distinct spatial locations are associated with different reward statistics, and decisions require navigation and movement through space. Male and female Long-Evans rats generally followed the predictions of theoretical models of foraging, albeit with a consistent tendency to persist with patches for too long compared to behavioral strategies that maximize food intake rate. The tendency to choose overly-long patch residence times was stronger in male rats. We also observed sex differences in locomotion as rats performed the task, but these differences in movement only partially accounted for the differences in patch residence durations observed between male and female rats. Together, these results suggest a nuanced relationship between movement, sex, and foraging decisions.
Collapse
Affiliation(s)
- Marissa Garcia
- Department of Psychology, University of California, Los Angeles, Los Angeles, California 90095
- Current address: Neurosciences Graduate Program, University of California, San Diego, San Diego, CA 92093
| | - Sukriti Gupta
- Department of Psychology, University of California, Los Angeles, Los Angeles, California 90095
| | - Andrew M. Wikenheiser
- Department of Psychology, University of California, Los Angeles, Los Angeles, California 90095
- Brain Research Institute, University of California, Los Angeles, Los Angeles, California 90095
| |
Collapse
|
15
|
Bukwich M, Campbell MG, Zoltowski D, Kingsbury L, Tomov MS, Stern J, Kim HR, Drugowitsch J, Linderman SW, Uchida N. Competitive integration of time and reward explains value-sensitive foraging decisions and frontal cortex ramping dynamics. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.05.556267. [PMID: 37732217 PMCID: PMC10508756 DOI: 10.1101/2023.09.05.556267] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/22/2023]
Abstract
The ability to make advantageous decisions is critical for animals to ensure their survival. Patch foraging is a natural decision-making process in which animals decide when to leave a patch of depleting resources to search for a new one. To study the algorithmic and neural basis of patch foraging behavior in a controlled laboratory setting, we developed a virtual foraging task for head-fixed mice. Mouse behavior could be explained by ramp-to-threshold models integrating time and rewards antagonistically. Accurate behavioral modeling required inclusion of a slowly varying "patience" variable, which modulated sensitivity to time. To investigate the neural basis of this decision-making process, we performed dense electrophysiological recordings with Neuropixels probes broadly throughout frontal cortex and underlying subcortical areas. We found that decision variables from the reward integrator model were represented in neural activity, most robustly in frontal cortical areas. Regression modeling followed by unsupervised clustering identified a subset of neurons with ramping activity. These neurons' firing rates ramped up gradually in single trials over long time scales (up to tens of seconds), were inhibited by rewards, and were better described as being generated by a continuous ramp rather than a discrete stepping process. Together, these results identify reward integration via a continuous ramping process in frontal cortex as a likely candidate for the mechanism by which the mammalian brain solves patch foraging problems.
Collapse
Affiliation(s)
- Michael Bukwich
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
- Current address: Sainsbury Wellcome Centre, University College London, London, W1T 4JG, UK
| | - Malcolm G Campbell
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| | - David Zoltowski
- Department of Statistics, Stanford University, Stanford, CA, 94305
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305
| | - Lyle Kingsbury
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| | - Momchil S Tomov
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
- Current address: Motional AD LLC, Boston, MA 02210
| | - Joshua Stern
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| | - HyungGoo R Kim
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon 16419, Republic of Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
| | - Jan Drugowitsch
- Department of Neurobiology, Harvard Medical School, Boston, MA, 02115
| | - Scott W Linderman
- Department of Statistics, Stanford University, Stanford, CA, 94305
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| |
Collapse
|
16
|
Fontanesi L, Shenhav A, Gluth S. Disentangling choice value and choice conflict in sequential decisions under risk. PLoS Comput Biol 2022; 18:e1010478. [PMID: 36206310 PMCID: PMC9581387 DOI: 10.1371/journal.pcbi.1010478] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 10/19/2022] [Accepted: 08/09/2022] [Indexed: 11/07/2022] Open
Abstract
Recent years have witnessed a surge of interest in understanding the neural and cognitive dynamics that drive sequential decision making in general and foraging behavior in particular. Due to the intrinsic properties of most sequential decision-making paradigms, however, previous research in this area has suffered from the difficulty to disentangle properties of the decision related to (a) the value of switching to a new patch versus, which increases monotonically, and (b) the conflict experienced between choosing to stay or leave, which first increases but then decreases after reaching the point of indifference between staying and switching. Here, we show how the same problems arise in studies of sequential decision-making under risk, and how they can be overcome, taking as a specific example recent research on the ‘pig’ dice game. In each round of the ‘pig’ dice game, people roll a die and accumulate rewards until they either decide to proceed to the next round or lose all rewards. By combining simulation-based dissections of the task structure with two experiments, we show how an extension of the standard paradigm, together with cognitive modeling of decision-making processes, allows to disentangle properties related to either switch value or choice conflict. Our study elucidates the cognitive mechanisms of sequential decision making and underscores the importance of avoiding potential pitfalls of paradigms that are commonly used in this research area. A large body of work has investigated how people make sequential decisions under risk. For instance, how people decide whether to continue gambling for potentially greater rewards or to cash in to avoid losing everything. Here, we identify a critical confound in this line of research, between (a) the value of switching and (b) the amount of conflict between choosing to stay or switch. Using a previously proposed paradigm (i.e., the pig dice game) as an example, we replicated behavior from a recent study and showed that switch value is highly correlated with choice conflict. By simulating behavior across hypothetical contexts, we then identified and tested novel variants of this task that allow to deconfound switch value and conflict. However, only by means of sequential sampling modeling we could conclude that it is conflict rather than switch value that drives response times in this task. Sequential sampling modeling also shows how the switch value influences other cognitive components in this task.
Collapse
Affiliation(s)
- Laura Fontanesi
- Department of Psychology, University of Basel, Basel, Switzerland
- * E-mail:
| | - Amitai Shenhav
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, Rhode Island, United States of America
| | - Sebastian Gluth
- Department of Psychology, University of Hamburg, Hamburg, Germany
| |
Collapse
|