Neural regions associated with gain-loss frequency and average reward in older and younger adults

Research on the biological basis of reinforcement-learning has focused on how brain regions track expected value based on average reward. However, recent work suggests that humans are more attuned to reward frequency. Furthermore, older adults are less likely to use expected values to guide choice than younger adults. This raises the question of whether brain regions assumed to be sensitive to average reward, like the medial and lateral PFC, also track reward frequency, and whether there are age-based differences. Older (60-81 years) and younger (18-30 years) adults performed the Soochow Gambling task, which separates reward frequency from average reward, while undergoing fMRI. Overall, participants preferred options that provided negative net payoffs, but frequent gains. Older adults improved less over time, were more reactive to recent negative outcomes, and showed greater frequency-related activation in several regions, including DLPFC. We also found broader recruitment of prefrontal and parietal regions associated with frequency value and reward prediction errors in older adults, which may indicate compensation. The results suggest greater reliance on average reward for younger adults than older adults.

Learning reward frequency over reward probability: A tale of two learning rules

Learning about the expected value of choice alternatives associated with reward is critical for adaptive behavior. Although human choice preferences are affected by the presentation frequency of reward-related alternatives, this may not be captured by some dominant models of value learning, such as the delta rule. In this study, we examined whether reward learning is driven more by learning the probability of reward provided by each option, or how frequently each option has been rewarded, and assess how well models based on average reward (e.g. the delta model) and models based on cumulative reward (e.g. the decay model) can account for choice preferences. In a binary-outcome choice task, participants selected between pairs of options that had reward probabilities of 0.65 (A) versus 0.35 (B) or 0.75 (C) versus 0.25 (D). Crucially, during training there were twice the number of AB trials as CD trials, such that option A was associated with higher cumulative reward, while option C gave higher average reward. Participants then decided between novel combinations of options (e.g., AC). Most participants preferred option A over C, a result predicted by the Decay model, but not the Delta model. We also compared the Delta and Decay models to both more simplified as well as more complex models that assumed additional mechanisms, such as representation of uncertainty. Overall, models that assume learning about cumulative reward provided the best account of the data.

Acute stress improves long-term reward maximization in decision-making under uncertainty

Acute stress influences reward-seeking tendencies and risky decision-making. However, it is unclear how acute stress influences decision-making in situations in which individuals must learn to either maximize long-term or immediate rewards from experience. Consequently, this study sought to investigate whether acute stress enhances salience of small, immediate or large, delayed rewards on decision-making under uncertainty. The Socially Evaluated Cold Pressor Task (SECPT) was used to induce acute stress. Participants in Experiment 1 (N = 50) were exposed to either the SECPT or a warm-water control condition and then completed a decision-making task in which participants needed to learn to forego immediate rewards in favor of larger delayed rewards. The results demonstrated that acute stress enhanced decisions that maximized long-term, large rewards over immediate, small rewards. Experiment 2 (N = 50) included an assessment of salivary cortisol. Results replicated the behavioral findings in Experiment 1 and demonstrated that the acute stress manipulation increased salivary cortisol, thus providing a potential physiological mechanism for these results. This work suggests that moderate acute stress can improve decision-making under uncertainty that depends on learning to maximize long-term rewards from experience.

Gender differences in preference for reward frequency versus reward magnitude in decision-making under uncertainty

Extensive research has focused on gender differences in intertemporal choices made from description in which participants must choose from multiple options that are specified without ambiguity. However, there has been limited work examining gender differences in intertemporal choices made from experience in which the possible payoffs among choice alternatives are not initially known and can only be gained from experience. Other work suggests that females attend more to reward frequency, whereas males attend more to reward magnitude. However, the tasks used in this research have been complex and did not examine intertemporal decision-making. To specifically test whether females are more sensitive to reward frequency and males are more sensitive to reward magnitude on intertemporal decisions made from experience, we designed a simple choice task in which participants pressed a response button at a time of their own choosing on each of many trials. Faster responses led to smaller, but more frequent rewards, whereas slower responses led to larger, but less frequently given rewards. As predicted, females tended to respond quicker for more certain, smaller rewards than males, supporting our prediction that women attend more to reward frequency whereas men attend more to reward magnitude.

A Case of Divergent Predictions Made by Delta and Decay Rule Learning Models

The Delta and Decay rules are two learning rules used to update expected values in reinforcement learning (RL) models. The delta rule learns average rewards, whereas the decay rule learns cumulative rewards for each option. Participants learned to select between pairs of options that had reward probabilities of .65 (option A) versus .35 (option B) or .75 (option C) versus .25 (option D) on separate trials in a binary-outcome choice task. Crucially, during training there were twice as AB trials as CD trials, therefore participants experienced more cumulative reward from option A even though option C had a higher average reward rate (.75 versus .65). Participants then decided between novel combinations of options (e.g, A versus C). The Decay model predicted more A choices, but the Delta model predicted more C choices, because those respective options had higher cumulative versus average reward values. Results were more in line with the Decay model's predictions. This suggests that people may retrieve memories of cumulative reward to compute expected value instead of learning average rewards for each option.