The effects of the previous outcome on probabilistic choice in rats

A critical review of risk-sensitive foraging

Foraging is risk sensitive if choices depend on the variability of returns from the options as well as their mean return. Risk-sensitive foraging is important in behavioural ecology, psychology and neurophysiology. It has been explained both in terms of mechanisms and in terms of evolutionary advantage. We provide a critical review, evaluating both mechanistic and evolutionary accounts. Some derivations of risk sensitivity from mechanistic models based on psychophysics are not convincing because they depend on an inappropriate use of Jensen's inequality. Attempts have been made to link risk sensitivity to the ecology of a species, but again these are not convincing. The field of risk-sensitive foraging has provided a focus for theoretical and empirical work and has yielded important insights, but we lack a simple and empirically defendable general account of it in either mechanistic or evolutionary terms. However, empirical analysis of choice sequences under theoretically motivated experimental designs and environmental settings appears a promising avenue for mapping the scope and relative merits of existing theories. Simply put, the devil is in the sequence.

Comparing utility functions between risky and riskless choice in rhesus monkeys

Decisions can be risky or riskless, depending on the outcomes of the choice. Expected utility theory describes risky choices as a utility maximization process: we choose the option with the highest subjective value (utility), which we compute considering both the option's value and its associated risk. According to the random utility maximization framework, riskless choices could also be based on a utility measure. Neuronal mechanisms of utility-based choice may thus be common to both risky and riskless choices. This assumption would require the existence of a utility function that accounts for both risky and riskless decisions. Here, we investigated whether the choice behavior of two macaque monkeys in risky and riskless decisions could be described by a common underlying utility function. We found that the utility functions elicited in the two choice scenarios were different from each other, even after taking into account the contribution of subjective probability weighting. Our results suggest that distinct utility representations exist for risky and riskless choices, which could reflect distinct neuronal representations of the utility quantities, or distinct brain mechanisms for risky and riskless choices. The different utility functions should be taken into account in neuronal investigations of utility-based choice.

Behavioural Isomorphism, Cognitive Economy and Recursive Thought in Non-Transitive Game Strategy

Game spaces in which an organism must repeatedly compete with an opponent for mutually exclusive outcomes are critical methodologies for understanding decision-making under pressure. In the non-transitive game rock, paper, scissors (RPS), the only technique that guarantees the lack of exploitation is to perform randomly in accordance with mixed-strategy. However, such behavior is thought to be outside bounded rationality and so decision-making can become deterministic, predictable, and ultimately exploitable. This review identifies similarities across economics, neuroscience, nonlinear dynamics, human, and animal cognition literatures, and provides a taxonomy of RPS strategy. RPS strategies are discussed in terms of (a) whether the relevant computations require sensitivity to item frequency, the cyclic relationships between responses, or the outcome of the previous trial, and (b) whether the strategy is framed around the self or other. The negative implication of this taxonomy is that despite the differences in cognitive economy and recursive thought, many of the identified strategies are behaviorally isomorphic. This makes it difficult to infer strategy from behavior. The positive implication is that this isomorphism can be used as a novel design feature in furthering our understanding of the attribution, agency, and acquisition of strategy in RPS and other game spaces.

Probability Distortion Depends on Choice Sequence in Rhesus Monkeys

Humans and other primates share many decision biases, among them our subjective distortion of objective probabilities. When making choices between uncertain rewards we typically treat probabilities nonlinearly: overvaluing low probabilities of reward and undervaluing high ones. A growing body of evidence, however, points to a more flexible pattern of distortion than the classical inverse-S one, highlighting the effect of experimental conditions in shifting the weight assigned to probabilities, such as task feedback, learning, and attention. Here we investigated the role of sequence structure (the order in which gambles are presented in a choice task) in shaping the probability distortion patterns of rhesus macaques: we presented 2 male monkeys with binary choice sequences of MIXED or REPEATED gambles against safe rewards. Parametric modeling revealed that choices in each sequence type were guided by significantly different patterns of probability distortion: whereas we elicited the classical inverse-S-shaped probability distortion in pseudorandomly MIXED trial sequences of gamble-safe choices, we found the opposite pattern consisting of S-shaped distortion, with REPEATED sequences. We extended these results to binary choices between two gambles, without a safe option, and confirmed the unique influence of the sequence structure in which the animals make choices. Finally, we showed that the value of gambles experienced in the past had a significant impact on the subjective value of future ones, shaping probability distortion on a trial-by-trial basis. Together, our results suggest that differences in choice sequence are sufficient to reverse the direction of probability distortion.

SIGNIFICANCE STATEMENT Our lives are peppered with uncertain, probabilistic choices. Recent studies showed how such probabilities are subjectively distorted. In the present study, we show that probability distortions in macaque monkeys differ significantly between sequences in which single gambles are repeated (S-shaped distortion), as opposed to being pseudorandomly intermixed with other gambles (inverse-S-shaped distortion). Our findings challenge the idea of fixed probability distortions resulting from inflexible computations, and points to a more instantaneous evaluation of probabilistic information. Past trial outcomes appeared to drive the “gap” between probability distortions in different conditions. Our data suggest that, as in most adaptive systems, probability values are slowly but constantly updated from prior experience, driving measures of probability distortion to either side of the S/inverse-S debate.

Resurrecting the individual in behavioral analysis: Using mixed effects models to address nonsystematic discounting data

Delay and probability discounting functions typically take a monotonic form, but some individuals produce functions that are nonsystematic. Johnson and Bickel (2008) developed an algorithm for classifying nonsystematic functions on the basis of two different criteria. Type 1 functions were identified as nonsystematic due to random choices and Type 2 functions were identified as nonsystematic due to relatively shallow slopes, suggesting poor sensitivity to choice parameters. Since their original publication, the algorithm has become widely used in the human discounting literature for removal of participants, with studies often removing approximately 20% of the original sample (Smith & Lawyer, 2017). Because subject removal may not always be feasible due to loss of power or other factors, the present report applied a mixed effects regression modeling technique (Wileyto, Audrain-Mcgovern, Epstein, & Lerman, 2004; Young, 2017) to account for individual differences in DD and PD functions. Assessment of the model estimates for Type 1 and 2 nonsystematic functions indicated that both types of functions deviated systematically from the rest of the sample in that nonsystematic participants were more likely to show shallower slopes and increased biases for larger amounts. The results indicate that removing these participants would fundamentally alter the properties of the final sample in undesirable ways. Because mixed effects models account for between-participant variation with random effects, we advocate for the use of these models for future analyses of a wide range of functions within the behavioral analysis field, with the benefit of avoiding the negative consequences associated with subject removal.

Reinforcement learning models of risky choice and the promotion of risk-taking by losses disguised as wins in rats

Risky decisions are inherently characterized by the potential to receive gains or incur losses, and these outcomes have distinct effects on subsequent decision-making. One important factor is that individuals engage in loss-chasing, in which the reception of a loss is followed by relatively increased risk-taking. Unfortunately, the mechanisms of loss-chasing are poorly understood, despite the potential importance for understanding pathological choice behavior. The goal of the present experiment was to illuminate the mechanisms governing individual differences in loss-chasing and risky-choice behaviors. Rats chose between a low-uncertainty outcome that always delivered a variable amount of reward and a high-uncertainty outcome that probabilistically delivered reward. Loss-processing and loss-chasing were assessed in the context of losses disguised as wins (LDWs), which are loss outcomes that are presented along with gain-related stimuli. LDWs have been suggested to interfere with adaptive decision-making in humans and thus potentially increase loss-making. Here, the rats presented with LDWs were riskier, in that they made more choices for the high-uncertainty outcome. A series of nonlinear models were fit to individual rats' data to elucidate the possible psychological mechanisms that best account for individual differences in high-uncertainty choices and loss-chasing behaviors. The models suggested that the rats presented with LDWs were more prone to showing a stay bias following high-uncertainty outcomes compared to rats not presented with LDWs. These results collectively suggest that LDWs acquire conditioned reinforcement properties that encourage continued risk-taking and increase loss-chasing following previous high-risk decisions. (PsycINFO Database Record

Modeling risky decision-making in nonhuman animals: shared core features

Understanding the neural mechanisms of risky decision-making is critical to developing appropriate treatments for psychiatric disorders, problem gambling, and addiction to drugs of abuse. Probing neurobiological mechanisms requires the use of nonhuman animal models (particularly rhesus macaques, rats, and mice). However, there is considerable variation across species in risk preferences. Nevertheless, there are shared core features of risky decision-making present across species. As demonstrated with a wide variety of behavioral paradigms, modulators of risk preference observed in humans are readily replicated in model species. Thus, risky decision-making represents an important implementation of reward-guided decision-making that is feasibly modeled across species.

The effects of ethanol on diverse components of choice in the rat: reward discrimination, preference and relative valuation

Alcohol consumption impairs judgment and choice. How alcohol alters these crucial processes is primarily unknown. Choice can be fractionated into different components including reward discrimination, preference and relative valuation that can function together or in isolation depending upon diverse factors including choice context. We examined the diverse components and contextual effects by analyzing the effects of alcohol drinking on choice behavior in a task with a reduced level of temporal and spatial constraints. Rats were trained to drink 10% ethanol during 6 weeks of behavior testing using a combined sucrose-fade and two-bottle free-choice procedure. Two different sucrose pellet outcomes (e.g., constant vs. variable) were presented each week to examine the impact of voluntary drinking on reward-based decision-making. Behavioral contexts of single option, free choice and extinction were examined for each outcome set. Comparisons were made between alcohol and control groups and within the alcohol group over time to inspect choice profiles. Between-group results showed alcohol drinking animals expressed altered place preference and modified sucrose reward approach latencies. The within-group profile showed that alcohol drinking animals can express adequate reward discrimination, preference and incentive contrast during free choice. All of these components were significantly reduced during the context of extinction. Control animals were also impacted by extinction but not as severely. The findings point to a need for a greater focus on the context and the diverse components of choice when examining external and internal factors influencing decision-making during alcohol or other substance of abuse exposure.

Fractionating choice: A study on reward discrimination, preference, and relative valuation in the rat (Rattus norvegicus)

Choice behavior combines discrimination between distinctive outcomes, preference for specific outcomes and relative valuation of comparable outcomes. Previous work has focused on 1 component (i.e., preference) disregarding other influential processes that might provide a more complete understanding. Animal models of choice have been explored primarily utilizing extensive training, limited freedom for multiple decisions and sparse behavioral measures constrained to a single phase of motivated action. The present study used a paradigm that combines different elements of previous methods with the goal to distinguish among components of choice and explore how well components match predictions based on risk-sensitive foraging strategies. In order to analyze discrimination and relative valuation, it was necessary to have an option that shifted and an option that remained constant. Shifting outcomes among weeks included a change in single-option outcome (0 to 1 to 2 pellets) or a change in mixed-option outcome (0 or 5 to 0 or 3 to 0 or 1 pellets). Constant outcomes among weeks were also mixed-option (0 or 3 pellets) or single-option (1 pellet). Shifting single-option outcomes among weeks led to better discrimination, more robust preference and significant incentive contrast effects for the alternative outcome. Shifting multioptions altered choice components and led to dissociations among discrimination, preference, and reduced contrast effects. During extinction, all components were impacted with the greatest deficits during the shifting mixed-option outcome sessions. Results suggest choice behavior can be optimized for 1 component but suboptimal for others depending upon the complexity of alterations in outcome value between options. (PsycINFO Database Record

Budget Constraints Affect Male Rats' Choices between Differently Priced Commodities

Demand theory can be applied to analyse how a human or animal consumer changes her selection of commodities within a certain budget in response to changes in price of those commodities. This change in consumption assessed over a range of prices is defined as demand elasticity. Previously, income-compensated and income-uncompensated price changes have been investigated using human and animal consumers, as demand theory predicts different elasticities for both conditions. However, in these studies, demand elasticity was only evaluated over the entirety of choices made from a budget. As compensating budgets changes the number of attainable commodities relative to uncompensated conditions, and thus the number of choices, it remained unclear whether budget compensation has a trivial effect on demand elasticity by simply sampling from a different total number of choices or has a direct effect on consumers’ sequential choice structure. If the budget context independently changes choices between commodities over and above price effects, this should become apparent when demand elasticity is assessed over choice sets of any reasonable size that are matched in choice opportunities between budget conditions. To gain more detailed insight in the sequential choice dynamics underlying differences in demand elasticity between budget conditions, we trained N=8 rat consumers to spend a daily budget by making a number of nosepokes to obtain two liquid commodities under different price regimes, in sessions with and without budget compensation. We confirmed that demand elasticity for both commodities differed between compensated and uncompensated budget conditions, also when the number of choices considered was matched, and showed that these elasticity differences emerge early in the sessions. These differences in demand elasticity were driven by a higher choice rate and an increased reselection bias for the preferred commodity in compensated compared to uncompensated budget conditions, suggesting a budget context effect on relative valuation.

Relative gains, losses, and reference points in probabilistic choice in rats

Theoretical reference points have been proposed to differentiate probabilistic gains from probabilistic losses in humans, but such a phenomenon in non-human animals has yet to be thoroughly elucidated. Three experiments evaluated the effect of reward magnitude on probabilistic choice in rats, seeking to determine reference point use by examining the effect of previous outcome magnitude(s) on subsequent choice behavior. Rats were trained to choose between an outcome that always delivered reward (low-uncertainty choice) and one that probabilistically delivered reward (high-uncertainty). The probability of high-uncertainty outcome receipt and the magnitudes of low-uncertainty and high-uncertainty outcomes were manipulated within and between experiments. Both the low- and high-uncertainty outcomes involved variable reward magnitudes, so that either a smaller or larger magnitude was probabilistically delivered, as well as reward omission following high-uncertainty choices. In Experiments 1 and 2, the between groups factor was the magnitude of the high-uncertainty-smaller (H-S) and high-uncertainty-larger (H-L) outcome, respectively. The H-S magnitude manipulation differentiated the groups, while the H-L magnitude manipulation did not. Experiment 3 showed that manipulating the probability of differential losses as well as the expected value of the low-uncertainty choice produced systematic effects on choice behavior. The results suggest that the reference point for probabilistic gains and losses was the expected value of the low-uncertainty choice. Current theories of probabilistic choice behavior have difficulty accounting for the present results, so an integrated theoretical framework is proposed. Overall, the present results have implications for understanding individual differences and corresponding underlying mechanisms of probabilistic choice behavior.

Mechanisms of Individual Differences in Impulsive and Risky Choice in Rats

Individual differences in impulsive and risky choice are key risk factors for a variety of maladaptive behaviors such as drug abuse, gambling, and obesity. In our rat model, ordered individual differences are stable across choice parameters, months of testing, and span a broad spectrum, suggesting that rats, like humans, exhibit trait-level impulsive and risky choice behaviors. In addition, impulsive and risky choices are highly correlated, suggesting a degree of correlation between these two traits. An examination of the underlying cognitive mechanisms has suggested an important role for timing processes in impulsive choice. In addition, in an examination of genetic factors in impulsive choice, the Lewis rat strain emerged as a possible animal model for studying disordered impulsive choice, with this strain demonstrating deficient delay processing. Early rearing environment also affected impulsive behaviors, with rearing in an enriched environment promoting adaptable and more self-controlled choices. The combined results with impulsive choice suggest an important role for timing and reward sensitivity in moderating impulsive behaviors. Relative reward valuation also affects risky choice, with manipulation of objective reward value (relative to an alternative reference point) resulting in loss chasing behaviors that predicted overall risky choice behaviors. The combined results are discussed in relation to domain-specific versus domain-general subjective reward valuation processes and the potential neural substrates of impulsive and risky choice.

Individual differences in impulsive and risky choice: effects of environmental rearing conditions

The present experiment investigated early-rearing environment modulation of individual differences in impulsive and risky choice. Rats were reared in an isolated condition (IC; n=12), in which they lived alone without novel stimuli, or an enriched condition (EC; n=11), in which they lived among conspecifics with novel stimuli. The impulsive choice task involved choices between smaller-sooner (SS) versus larger-later (LL) rewards. The risky choice task involved choices between certain-smaller (C-S) versus uncertain-larger (U-L) rewards. Following choice testing, incentive motivation to work for food was measured using a progressive ratio task and correlated with choice behavior. HPLC analyses were conducted to determine how monoamine concentrations within the prefrontal cortex (PFC) and nucleus accumbens (NAC) related to behavior in different tasks. IC rats were more impulsive than EC rats, but they did not differ in risky choice behavior. However, choice behavior across tasks was significantly correlated (i.e., the more impulsive rats were also riskier). There were no group differences in monoamine levels, but noradrenergic and serotonergic concentrations were significantly correlated with impulsive and risky choice. Furthermore, serotonin and norepinephrine concentrations in the NAC significantly correlated with incentive motivation and the timing of the reward delays within the choice tasks. These results suggest a role for domain general processes in impulsive and risky choice and indicate the importance of the NAC and/or PFC in timing, reward processing, and choice behavior.

Effects of SKF-83566 and haloperidol on performance on progressive ratio schedules maintained by sucrose and corn oil reinforcement: quantitative analysis using a new model derived from the Mathematical Principles of Reinforcement (MPR)

RATIONALE

Mathematical models can assist the interpretation of the effects of interventions on schedule-controlled behaviour and help to differentiate between processes that may be confounded in traditional performance measures such as response rate and the breakpoint in progressive ratio (PR) schedules.

OBJECTIVE

The effects of a D1-like dopamine receptor antagonist, 8-bromo-2,3,4,5-tetrahydro-3-methyl-5-phenyl-1H-3-benzazepin-7-ol hydrobromide (SKF-83566), and a D2-like receptor antagonist, haloperidol, on rats' performance on PR schedules maintained by sucrose and corn oil reinforcers were assessed using a new model derived from Killeen's (Behav Brain Sci 17:105-172, 1994) Mathematical Principles of Reinforcement.

METHOD

Separate groups of rats were trained under a PR schedule using sucrose or corn oil reinforcers. SKF-83566 (0.015 and 0.03 mg kg(-1)) and haloperidol (0.05 and 0.1 mg kg(-1)) were administered intraperitoneally (five administrations of each treatment). Running and overall response rates in successive ratios were analysed using the new model, and estimates of the model's parameters were compared between treatments.

RESULTS

Haloperidol reduced a (the parameter expressing incentive value) in the case of both reinforcers, but did not affect the parameters related to response time and post-reinforcement pausing. SKF-83566 reduced a and k (the parameter expressing sensitivity of post-reinforcement pausing to the prior inter-reinforcement interval) in the case of sucrose, but did not affect any of the parameters in the case of corn oil.

CONCLUSIONS

The results are consistent with the hypothesis that blockade of both D1-like and D2-like receptors reduces the incentive value of sucrose, whereas the incentive value of corn oil is more sensitive to blockade of D2-like than D1-like receptors.