Neuroscience of foraging

Hierarchical control over foraging behavior by anterior cingulate cortex

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

Estimating foraging behavior in rodents using a modified paradigm measuring threat imminence dynamics

Animals need to respond to threats to avoid danger and approach rewards. In nature, these responses did not evolve alone but are always accompanied by motivational conflict. A semi-naturalistic threat imminence continuum model models the approach-avoidance conflict and is able to integrate multiple behaviors into a single paradigm. However, its comprehensive application is hampered by the lack of a detailed protocol and data about some fundamental factors including sex, age, and motivational level. Here, we modified a previously established paradigm measuring threat imminence continuum dynamics, involving modifications of training and testing protocols, and utilization of commercial materials combined with open science codes, making it easier to replicate. We demonstrate that foraging behavior is modulated by age, hunger level, and sex. This paradigm can be used to study foraging behaviors in animals in a more naturalistic manner with relevance to human approach-avoid conflicts and associated psychopathologies.

Reward expectations direct learning and drive operant matching in Drosophila

Foraging animals must use decision-making strategies that dynamically adapt to the changing availability of rewards in the environment. A wide diversity of animals do this by distributing their choices in proportion to the rewards received from each option, Herrnstein's operant matching law. Theoretical work suggests an elegant mechanistic explanation for this ubiquitous behavior, as operant matching follows automatically from simple synaptic plasticity rules acting within behaviorally relevant neural circuits. However, no past work has mapped operant matching onto plasticity mechanisms in the brain, leaving the biological relevance of the theory unclear. Here, we discovered operant matching in Drosophila and showed that it requires synaptic plasticity that acts in the mushroom body and incorporates the expectation of reward. We began by developing a dynamic foraging paradigm to measure choices from individual flies as they learn to associate odor cues with probabilistic rewards. We then built a model of the fly mushroom body to explain each fly's sequential choice behavior using a family of biologically realistic synaptic plasticity rules. As predicted by past theoretical work, we found that synaptic plasticity rules could explain fly matching behavior by incorporating stimulus expectations, reward expectations, or both. However, by optogenetically bypassing the representation of reward expectation, we abolished matching behavior and showed that the plasticity rule must specifically incorporate reward expectations. Altogether, these results reveal the first synapse-level mechanisms of operant matching and provide compelling evidence for the role of reward expectation signals in the fly brain.

Arc regulates a second-guessing cognitive bias during naturalistic foraging through effects on discrete behavior modules

Foraging in animals relies on innate decision-making heuristics that can result in suboptimal cognitive biases in some contexts. The mechanisms underlying these biases are not well understood, but likely involve strong genetic effects. To explore this, we studied fasted mice using a naturalistic foraging paradigm and discovered an innate cognitive bias called "second-guessing." This involves repeatedly investigating an empty former food patch instead of consuming available food, which hinders the mice from maximizing feeding benefits. The synaptic plasticity gene Arc is revealed to play a role in this bias, as Arc-deficient mice did not exhibit second-guessing and consumed more food. In addition, unsupervised machine learning decompositions of foraging identified specific behavior sequences, or "modules", that are affected by Arc. These findings highlight the genetic basis of cognitive biases in decision making, show links between behavior modules and cognitive bias, and provide insight into the ethological roles of Arc in naturalistic foraging.

Social Determinants of Inter-Individual Variability and Vulnerability: The Role of Dopamine

Individuals differ in their traits and preferences, which shape their interactions, their prospects for survival and their susceptibility to diseases. These correlations are well documented, yet the neurophysiological mechanisms underlying the emergence of distinct personalities and their relation to vulnerability to diseases are poorly understood. Social ties, in particular, are thought to be major modulators of personality traits and psychiatric vulnerability, yet the majority of neuroscience studies are performed on rodents in socially impoverished conditions. Rodent micro-society paradigms are therefore key experimental paradigms to understand how social life generates diversity by shaping individual traits. Dopamine circuitry is implicated at the interface between social life experiences, the expression of essential traits, and the emergence of pathologies, thus proving a possible mechanism to link these three concepts at a neuromodulatory level. Evaluating inter-individual variability in automated social testing environments shows great promise for improving our understanding of the link between social life, personality, and precision psychiatry – as well as elucidating the underlying neurophysiological mechanisms.

A multi-pronged investigation of option generation using depression, PET and modafinil

Option generation is a critical process in decision making, but previous studies have largely focused on choices between options given by a researcher. Consequently, how we self-generate options for behaviour remain poorly understood. Here, we investigated option generation in major depressive disorder and how dopamine might modulate this process, as well as the effects of modafinil (a putative cognitive enhancer) on option generation in healthy individuals. We first compared differences in self-generated options between healthy non-depressed adults [n = 44, age = 26.3 years (SD 5.9)] and patients with major depressive disorder [n = 54, age = 24.8 years (SD 7.4)]. In the second study, a subset of depressed individuals [n = 22, age = 25.6 years (SD 7.8)] underwent PET scans with 11C-raclopride to examine the relationships between dopamine D2/D3 receptor availability and individual differences in option generation. Finally, a randomized, double-blind, placebo-controlled, three-way crossover study of modafinil (100 mg and 200 mg), was conducted in an independent sample of healthy people [n = 19, age = 23.2 years (SD 4.8)] to compare option generation under different doses of this drug. The first study revealed that patients with major depressive disorder produced significantly fewer options [t(96) = 2.68, P = 0.009, Cohen's d = 0.54], albeit with greater uniqueness [t(96) = -2.54, P = 0.01, Cohen's d = 0.52], on the option generation task compared to healthy controls. In the second study, we found that 11C-raclopride binding potential in the putamen was negatively correlated with fluency (r = -0.69, P = 0.001) but positively associated with uniqueness (r = 0.59, P = 0.007). Hence, depressed individuals with higher densities of unoccupied putamen D2/D3 receptors in the putamen generated fewer but more unique options, whereas patients with lower D2/D3 receptor availability were likely to produce a larger number of similar options. Finally, healthy participants were less unique [F(2,36) = 3.32, P = 0.048, partial η2 = 0.16] and diverse [F(2,36) = 4.31, P = 0.021, partial η2 = 0.19] after taking 200 mg versus 100 mg and 0 mg of modafinil, while fluency increased linearly with dosage at a trend level [F(1,18) = 4.11, P = 0.058, partial η2 = 0.19]. Our results show, for the first time, that option generation is affected in clinical depression and that dopaminergic activity in the putamen of patients with major depressive disorder may play a key role in the self-generation of options. Modafinil was also found to influence option generation in healthy people by reducing the creativity of options produced.

Synthetic Spatial Foraging With Active Inference in a Geocaching Task

Humans are highly proficient in learning about the environments in which they operate. They form flexible spatial representations of their surroundings that can be leveraged with ease during spatial foraging and navigation. To capture these abilities, we present a deep Active Inference model of goal-directed behavior, and the accompanying belief updating. Active Inference rests upon optimizing Bayesian beliefs to maximize model evidence or marginal likelihood. Bayesian beliefs are probability distributions over the causes of observable outcomes. These causes include an agent's actions, which enables one to treat planning as inference. We use simulations of a geocaching task to elucidate the belief updating-that underwrites spatial foraging-and the associated behavioral and neurophysiological responses. In a geocaching task, the aim is to find hidden objects in the environment using spatial coordinates. Here, synthetic agents learn about the environment via inference and learning (e.g., learning about the likelihoods of outcomes given latent states) to reach a target location, and then forage locally to discover the hidden object that offers clues for the next location.

Uncertainty drives deviations in normative foraging decision strategies

Nearly all animals forage to acquire energy for survival through efficient search and resource harvesting. Patch exploitation is a canonical foraging behaviour, but there is a need for more tractable and understandable mathematical models describing how foragers deal with uncertainty. To provide such a treatment, we develop a normative theory of patch foraging decisions, proposing mechanisms by which foraging behaviours emerge in the face of uncertainty. Our model foragers statistically and sequentially infer patch resource yields using Bayesian updating based on their resource encounter history. A decision to leave a patch is triggered when the certainty of the patch type or the estimated yield of the patch falls below a threshold. The time scale over which uncertainty in resource availability persists strongly impacts behavioural variables like patch residence times and decision rules determining patch departures. When patch depletion is slow, as in habitat selection, departures are characterized by a reduction of uncertainty, suggesting that the forager resides in a low-yielding patch. Uncertainty leads patch-exploiting foragers to overharvest (underharvest) patches with initially low (high) resource yields in comparison with predictions of the marginal value theorem. These results extend optimal foraging theory and motivate a variety of behavioural experiments investigating patch foraging behaviour.

Conflict Test Battery for Studying the Act of Facing Threats in Pursuit of Rewards

Survival depends on the ability of animals to avoid threats and approach rewards. Traditionally, these two opposing motivational systems have been studied separately. In nature, however, they regularly compete for the control of behavior. When threat- and reward-eliciting stimuli (learned or unlearned) occur simultaneously, a motivational conflict emerges that challenges individuals to weigh available options and execute a single behavioral response (avoid or approach). Most previous animal models using approach/avoidance conflicts have often focused on the ability to avoid threats by forgoing or delaying the opportunity to obtain rewards. In contrast, behavioral tasks designed to capitalize on the ability to actively choose to execute approach behaviors despite threats are scarce. Thus, we developed a behavioral test battery composed of three conflict tasks to directly study rats confronting threats to obtain rewards guided by innate and conditioned cues. One conflict task involves crossing a potentially electrified grid to obtain food on the opposite end of a straight alley, the second task is based on the step-down threat avoidance paradigm, and the third one is a modified version of the open field test. We used diazepam to pharmacologically validate conflict behaviors in our tasks. We found that, regardless of whether competing stimuli were conditioned or innate, a low diazepam dose decreased risk assessment and facilitated taking action to obtain rewards in the face of threats during conflict, without affecting choice behavior when there was no conflict involved. Using this pharmacologically validated test battery of ethologically designed innate/learned conflict tasks could help understand the fundamental brain mechanisms underlying the ability to confront threats to achieve goals.

Systems Neuroscience of Natural Behaviors in Rodents

Animals evolved in complex environments, producing a wide range of behaviors, including navigation, foraging, prey capture, and conspecific interactions, which vary over timescales ranging from milliseconds to days. Historically, these behaviors have been the focus of study for ecology and ethology, while systems neuroscience has largely focused on short timescale behaviors that can be repeated thousands of times and occur in highly artificial environments. Thanks to recent advances in machine learning, miniaturization, and computation, it is newly possible to study freely moving animals in more natural conditions while applying systems techniques: performing temporally specific perturbations, modeling behavioral strategies, and recording from large numbers of neurons while animals are freely moving. The authors of this review are a group of scientists with deep appreciation for the common aims of systems neuroscience, ecology, and ethology. We believe it is an extremely exciting time to be a neuroscientist, as we have an opportunity to grow as a field, to embrace interdisciplinary, open, collaborative research to provide new insights and allow researchers to link knowledge across disciplines, species, and scales. Here we discuss the origins of ethology, ecology, and systems neuroscience in the context of our own work and highlight how combining approaches across these fields has provided fresh insights into our research. We hope this review facilitates some of these interactions and alliances and helps us all do even better science, together.

Trait and State-Dependent Risk Attitude of Monkeys Measured in a Single-Option Response Task

Humans and animals show diverse preferences for risks (“trait-like” risk attitude) and shift their preference depending on the state or current needs (“state-dependent” risk attitude). For a better understanding of the neural mechanisms underlying risk-sensitive decisions, useful animal models have been required. Here we examined the risk attitude of three male monkeys in a single-option response task, in which an instrumental lever-release was required to obtain a chance of reward. In each trial, reward condition, either deterministic (100% of 1, 2, 3, and 4 drops of juice) or probabilistic (25, 50, 75, and 100% of 4-drop juice) was randomly selected and assigned by a unique visual cue, allowing the monkeys to evaluate the forthcoming reward. The subjective value of the reward was inferred from their performance. Model-based analysis incorporating known economic models revealed non-linear probability distortion in monkeys; unlike previous studies, they showed a simple convex or concave probability distortion curve. The direction of risk preference was consistent between early and late phases of the testing period, suggesting that our observation reflected the trait-like risk attitude of monkeys, at least under the current experimental setting. Regardless of the baseline risk preference, all monkeys showed an enhancement of risk preference in a session according to the satiation level (i.e., state-dependent risk attitude). Our results suggest that, without choice or cognitive demand, monkeys show naturalistic risk attitude – diverse and flexible like humans. Our novel approach may provide a useful animal model of risk-sensitive decisions, facilitating the investigation of the neural mechanisms of decision-making under risk.

Foraging as an evidence accumulation process

The patch-leaving problem is a canonical foraging task, in which a forager must decide to leave a current resource in search for another. Theoretical work has derived optimal strategies for when to leave a patch, and experiments have tested for conditions where animals do or do not follow an optimal strategy. Nevertheless, models of patch-leaving decisions do not consider the imperfect and noisy sampling process through which an animal gathers information, and how this process is constrained by neurobiological mechanisms. In this theoretical study, we formulate an evidence accumulation model of patch-leaving decisions where the animal averages over noisy measurements to estimate the state of the current patch and the overall environment. We solve the model for conditions where foraging decisions are optimal and equivalent to the marginal value theorem, and perform simulations to analyze deviations from optimal when these conditions are not met. By adjusting the drift rate and decision threshold, the model can represent different “strategies”, for example an incremental, decremental, or counting strategy. These strategies yield identical decisions in the limiting case but differ in how patch residence times adapt when the foraging environment is uncertain. To describe sub-optimal decisions, we introduce an energy-dependent marginal utility function that predicts longer than optimal patch residence times when food is plentiful. Our model provides a quantitative connection between ecological models of foraging behavior and evidence accumulation models of decision making. Moreover, it provides a theoretical framework for potential experiments which seek to identify neural circuits underlying patch-leaving decisions.

Foraging is a ubiquitous animal behavior, performed by organisms as different as worms, birds, rats, and humans. Although the behavior has been extensively studied, it is not known how the brain processes information obtained during foraging activity to make subsequent foraging decisions. We form an evidence accumulation model of foraging decisions that describes the process through which an animal gathers information and uses it to make foraging decisions. By building on studies of the neural decision mechanisms within systems neuroscience, this model connects the foraging decision process with ecological models of patch-leaving decisions, such as the marginal value theorem. The model suggests the existence of different foraging strategies, which optimize for different environmental conditions and their potential implementation by neural decision making circuits. The model also shows how state-dependence, such as satiation level, can affect evidence accumulation to lead to sub-optimal foraging decisions. Our model provides a framework for future experimental studies which seek to elucidate how neural decision making mechanisms have been shaped by evolutionary forces in an animal’s surrounding environment.

Revisiting foraging approaches in neuroscience

Many complex real-world decisions, such as deciding which house to buy or whether to switch jobs, involve trying to maximize reward across a sequence of choices. Optimal Foraging Theory is well suited to study these kinds of choices because it provides formal models for reward-maximization in sequential situations. In this article, we review recent insights from foraging neuroscience, behavioral ecology, and computational modelling. We find that a commonly used approach in foraging neuroscience, in which choice items are encountered at random, does not reflect the way animals direct their foraging efforts in certain real-world settings, nor does it reflect efficient reward-maximizing behavior. Based on this, we propose that task designs allowing subjects to encounter choice items strategically will further improve the ecological validity of foraging approaches used in neuroscience, as well as give rise to new behavioral and neural predictions that deepen our understanding of sequential, value-based choice.

Dopamine Modulates Option Generation for Behavior

Animals make innumerable decisions every day, each of which involves evaluating potential options for action. But how are options generated? Although much is now known about decision making when a fixed set of potential options is provided, surprisingly little progress has been made on self-generated options. Some researchers have proposed that such abilities might be modulated by dopamine. Here, we used a new measure of option generation that is quantitative, objective, and culture fair to investigate how humans generate different behavioral options. Participants were asked to draw as many different paths (options) as they could between two points within a fixed time. Healthy individuals (n = 96) exhibited a trade-off between uniqueness (how individually different their options were) and fluency (number of options), generating either many similar or few unique options. To assess influence of dopamine, we first examined patients with Parkinson's disease (n = 35) ON and OFF their dopaminergic medication and compared them to elderly healthy controls (n = 34). Then we conducted a double-blind, placebo-controlled crossover study of the D2 agonist cabergoline in healthy older people (n = 29). Across both studies, dopamine increased fluency but diminished overall uniqueness of options generated, due to the effect of fluency trading off with uniqueness. Crucially, however, when this trade-off was corrected for, dopamine was found to increase uniqueness for any given fluency. Three carefully designed control studies showed that performance on our option-generation task was not related to executing movements, planning actions, or selecting between generated options. These findings show that dopamine plays an important role in modulating option generation.

Decision-making under risk of gain in young children and mangabey monkeys

In contrast to human adults, risk proneness in the gain domain is usually observed in both young children and non-human primates. It is currently unclear what mechanism might be underlying such economic preferences. We investigated decision-making under risk of gain in toddlers and monkeys. The choices of 2.5-year-old children and red-capped mangabeys (Cercocebus torquatus torquatus) were examined in a gambling task for food reward in which participants have to choose between two options, a secure option and a risky option. In contrast to monkeys, toddlers showed a strong preference for the risky option over the safe option. In order to test the hypothesis that risky choices in participants reflect inhibitory control difficulties, toddlers and mangabeys were presented in Experiment 2 with a situation analogous to that used in Experiment 1 except for the fact that the opaque cover under which was placed the secure option was replaced by a transparent cover. In this second experiment, toddlers continued to show a preference for the risky option over the safe option. In contrast, mangabeys showed a preference for the safe option over the risky option in Trial 1 but they shifted their economic preferences in Trial 2. We argue that decision-making strategies under risk of gain in both toddlers and mangabeys (a) do not reflect poor behavioral control and (b) are not reducible to perception-action couplings.

Dorsal anterior cingulate and ventromedial prefrontal cortex have inverse roles in both foraging and economic choice

Recent research has highlighted a distinction between sequential foraging choices and traditional economic choices between simultaneously presented options. This was partly motivated by observations in Kolling, Behrens, Mars, and Rushworth, Science, 336(6077), 95-98 (2012) (hereafter, KBMR) that these choice types are subserved by different circuits, with dorsal anterior cingulate (dACC) preferentially involved in foraging and ventromedial prefrontal cortex (vmPFC) preferentially involved in economic choice. To support this account, KBMR used fMRI to scan human subjects making either a foraging choice (between exploiting a current offer or swapping for potentially better rewards) or an economic choice (between two reward-probability pairs). This study found that dACC better tracked values pertaining to foraging, whereas vmPFC better tracked values pertaining to economic choice. We recently showed that dACC's role in these foraging choices is better described by the difficulty of choosing than by foraging value, when correcting for choice biases and testing a sufficiently broad set of foraging values (Shenhav, Straccia, Cohen, & Botvinick Nature Neuroscience, 17(9), 1249-1254, 2014). Here, we extend these findings in 3 ways. First, we replicate our original finding with a larger sample and a task modified to address remaining methodological gaps between our previous experiments and that of KBMR. Second, we show that dACC activity is best accounted for by choice difficulty alone (rather than in combination with foraging value) during both foraging and economic choices. Third, we show that patterns of vmPFC activity, inverted relative to dACC, also suggest a common function across both choice types. Overall, we conclude that both regions are similarly engaged by foraging-like and economic choice.