A unifying Bayesian account of contextual effects in value-based choice

Range adaptive value representations in schizophrenia and major depression

Anhedonia and amotivation are core symptoms of schizophrenia (SCZ) and major depressive disorder (MDD). Reward processing involves constructing and contrasting the representations for expected value (EV) and outcome value (OV) of a given stimulus, a phenomenon termed range adaptation. Impaired range adaptation can lead to anhedonia and amotivation. This study aimed to examine range adaptation in SCZ patients and MDD patients. Fifty SCZ, 46 MDD patients and 56 controls completed the Effort-based Pleasure Experience Task to measure EV and OV adaptation. SCZ and MDD patients showed altered range adaptation, albeit in different patterns. SCZ patients exhibited over-adaptation to OV and reduced adaptation to EV. By contrast, MDD patients exhibited diminished OV adaptation but intact EV adaptation. Both OV and EV adaptation were correlated with anhedonia and amotivation in SCZ and MDD. Taken together, our findings suggest that range adaptation is altered in both SCZ and MDD patients. Associations of OV and EV adaptation with anhedonia and amotivation were consistently found in SCZ and MDD patients. Impaired range adaptation in SCZ and MDD patients may be putative neural mechanisms and potential intervention targets for anhedonia and amotivation.

Theories of Context Effects in Multialternative, Multiattribute Choice

Over the past several decades, researchers in psychology, neuroscience, marketing, and economics have been keen to understand context effects in multialternative, multiattribute decision making. These effects occur when choices among existing alternatives are altered by the addition of a new alternative to the choice set. The effects violate classic decision theories and have led to the development of computational and mathematical models that explain how underlying cognitive and neural mechanisms give rise to the effects. This article reviews dynamic models of these effects, comparing mechanisms across models. Most models of context effects incorporate an attention mechanism, which suggests that attention plays an important role in multialternative, multiattribute decision making. I conclude by discussing recent empirical studies of attention and context effects and hypothesize that changes in attention could be responsible for recently observed reversals in context effects.

Human value learning and representation reflect rational adaptation to task demands

Humans and other animals routinely make choices between goods of different values. Choices are often made within identifiable contexts, such that an efficient learner may represent values relative to their local context. However, if goods occur across multiple contexts, a relative value code can lead to irrational choices. In this case, an absolute context-independent value is preferable to a relative code. Here we test the hypothesis that value representation is not fixed but rationally adapted to context expectations. In two experiments, we manipulated participants' expectations about whether item values learned within local contexts would need to be subsequently compared across contexts. Despite identical learning experiences, the group whose expectations included choices across local contexts went on to learn more absolute-like representation than the group whose expectations covered only fixed local contexts. Human value representation is thus neither relative nor absolute but efficiently and rationally tuned to task demands.

Implicit Counterfactual Effect in Partial Feedback Reinforcement Learning: Behavioral and Modeling Approach

Context remarkably affects learning behavior by adjusting option values according to the distribution of available options. Displaying counterfactual outcomes, the outcomes of the unchosen option alongside the chosen one (i.e., providing complete feedback), would increase the contextual effect by inducing participants to compare the two outcomes during learning. However, when the context only consists of the juxtaposition of several options and there is no such explicit counterfactual factor (i.e., only partial feedback is provided), it is not clear whether and how the contextual effect emerges. In this research, we employ Partial and Complete feedback paradigms in which options are associated with different reward distributions. Our modeling analysis shows that the model that uses the outcome of the chosen option for updating the values of both chosen and unchosen options in opposing directions can better account for the behavioral data. This is also in line with the diffusive effect of dopamine on the striatum. Furthermore, our data show that the contextual effect is not limited to probabilistic rewards, but also extends to magnitude rewards. These results suggest that by extending the counterfactual concept to include the effect of the chosen outcome on the unchosen option, we can better explain why there is a contextual effect in situations in which there is no extra information about the unchosen outcome.

Establishing the laws of preferential choice behavior

Mathematical and computational decision models are powerful tools for studying choice behavior, and hundreds of distinct decision models have been proposed over the long interdisciplinary history of decision making research. The existence of so many models has led to theoretical fragmentation and redundancy, obscuring key insights into choice behavior, and preventing consensus about the essential properties of preferential choice. We provide a synthesis of formal models of risky, multiattribute, and intertemporal choice, three important domains in decision making. We identify recurring insights discovered by scholars of different generations and different disciplines across these three domains, and use these insights to classify over 150 existing models as involving various combinations of eight key mathematical and computational properties. These properties capture the main avenues of theoretical development in decision making research and can be used to understand the similarities and differences between decision models, aiding both theoretical analyses and empirical tests.

Setting the space for deliberation in decision-making

Decision-making models in the behavioral, cognitive, and neural sciences typically consist of forced-choice paradigms with two alternatives. While theoretically it is feasible to translate any decision situation to a sequence of binary choices, real-life decision-making is typically more complex and nonlinear, involving choices among multiple items, graded judgments, and deferments of decision-making. Here, we discuss how the complexity of real-life decision-making can be addressed using conventional decision-making models by focusing on the interactive dynamics between criteria settings and the collection of evidence. Decision-makers can engage in multi-stage, parallel decision-making by exploiting the space for deliberation, with non-binary readings of evidence available at any point in time. The interactive dynamics principally adhere to the speed-accuracy tradeoff, such that increasing the space for deliberation enables extended data collection. The setting of space for deliberation reflects a form of meta-decision-making that can, and should be, studied empirically as a value-based exercise that weighs the prior propensities, the economics of information seeking, and the potential outcomes. Importantly, the control of the space for deliberation raises a question of agency. Decision-makers may actively and explicitly set their own decision parameters, but these parameters may also be set by environmental pressures. Thus, decision-makers may be influenced-or nudged in a particular direction-by how decision problems are framed, with a sense of urgency or a binary definition of choice options. We argue that a proper understanding of these mechanisms has important practical implications toward the optimal usage of space for deliberation.

Tracking the Influence of Predictive Cues on the Evaluation of Food Images: Volatility Enables Nudging

In previous research on the evaluation of food images, we found that appetitive food images were rated higher following a positive prediction than following a negative prediction, and vice versa for aversive food images. The findings suggested an active confirmation bias. Here, we examine whether this influence from prediction depends on the evaluative polarization of the food images. Specifically, we divided the set of food images into “strong” and “mild” images by how polarized (i.e., extreme) their average ratings were across all conditions. With respect to the influence from prediction, we raise two alternative hypotheses. According to a predictive dissonance hypothesis, the larger the discrepancy between prediction and outcome, the stronger the active inference toward accommodating the outcome with the prediction; thus, the confirmation bias should obtain particularly with strong images. Conversely, according to a nudging-in-volatility hypothesis, the active confirmation bias operates only on images within a dynamic range, where the values of images are volatile, and not on the evaluation of images that are too obviously appetitive or aversive; accordingly, the effects from prediction should occur predominately with mild images. Across the data from two experiments, we found that the evaluation of mild images tended to exhibit the confirmation bias, with ratings that followed the direction given by the prediction. For strong images, there was no confirmation bias. Our findings corroborate the nudging-in-volatility hypothesis, suggesting that predictive cues may be able to tip the balance of evaluation particularly for food images that do not have a strongly polarized value.

Quantifying the immediate computational effects of preceding outcomes on subsequent risky choices

Forty years ago, prospect theory introduced the notion that risky options are evaluated relative to their recent context, causing a significant shift in the study of risky monetary decision-making in psychology, economics, and neuroscience. Despite the central role of past experiences, it remains unclear whether, how, and how much past experiences quantitatively influence risky monetary choices moment-to-moment in a nominally learning-free setting. We analyzed a large dataset of risky monetary choices with trial-by-trial feedback to quantify how past experiences, or recent events, influence risky choice behavior and the underlying processes. We found larger recent outcomes both negatively influence subsequent risk-taking and positively influence the weight put on potential losses. Using a hierarchical Bayesian framework to fit a modified version of prospect theory, we demonstrated that the same risks will be evaluated differently given different past experiences. The computations underlying risky decision-making are fundamentally dynamic, even if the environment is not.

Better than expected: the influence of option expectations during decision-making

Our choices often arise from a consideration of options presented in a sequence (e.g. the products in a supermarket row). However, whether the precise sequential order of option presentation affects decision-making remains poorly understood. A recent model of choice proposes that, in a set of options presented sequentially, those that are better than expected will be perceived as more valuable, even when options are objectively equivalent within the set. Inspired by this proposal, we devised a novel decision-making task where we manipulated the order of option presentation together with expectations about option value. Even when we compared trials that were exactly equivalent except for option order, we observed a striking preference for options that were better than expected. Our findings show that expectations about options affect which option will be favoured within a sequence, an influence which is manifested as a preference for better-than-expected options. The findings have potential practical implications, as for example they may help policymakers in devising nudge strategies that rely on ad hoc option orders.

Reference effects on decision-making elicited by previous rewards

Substantial evidence has highlighted reference effects occurring during decision-making, whereby subjective value is not calculated in absolute terms but relative to the distribution of rewards characterizing a context. Among these, within-choice effects are exerted by options simultaneously available during choice. These should be distinguished from between-choice effects, which depend on the distribution of options presented in the past. Influential theories on between-choice effects include Decision-by-Sampling, Expectation-as-Reference and Divisive Normalization. Surprisingly, previous literature has focused on each theory individually disregarding the others. Thus, similarities and differences among theories remain to be systematically examined. Here we fill this gap by offering an overview of the state-of-the-art of research about between-choice reference effects. Our comparison of alternative theories shows that, at present, none of them is able to account for the full range of empirical data. To address this, we propose a model inspired by previous perspectives and based on a logistic framework, hence called logistic model of subjective value. Predictions of the model are analysed in detail about reference effects and risky decision-making. We conclude that our proposal offers a compelling framework for interpreting the multifaceted manifestations of between-choice reference effects.

Bayesian computational markers of relapse in methamphetamine dependence

Methamphetamine use disorder is associated with a high likelihood of relapse. Identifying robust predictors of relapse that have explanatory power is critical to develop secondary prevention based on a mechanistic understanding of relapse. Computational approaches have the potential to identify such predictive markers of psychiatric illness, with the advantage of providing a finer mechanistic explanation of the cognitive processes underlying psychiatric vulnerability.

In this study, sixty-two recently sober methamphetamine-dependent individuals were recruited from a 28-day inpatient treatment program, and completed a Stop Signal Task (SST) while undergoing functional magnetic resonance imaging (fMRI). These individuals were prospectively followed for 1 year and assessed for relapse to methamphetamine use. Thirty-three percent of followed participants reported relapse.

We found that neural activity associated with two types of Bayesian prediction error, i.e. the difference between actual and expected need to stop on a given trial, significantly differentiated those individuals who remained abstinent and those who relapsed. Specifically, relapsed individuals exhibited smaller neural activations to such Bayesian prediction errors relative to those individuals who remained abstinent in the left temporoparietal junction (Cohen's d = 0.91), the left inferior frontal gyrus (Cohen's d = 0.57), and left anterior insula (Cohen's d = 0.63). In contrast, abstinent and relapsed participants did not differ in neural activation to non-model based task contrasts or on various self-report clinical measures.

In conclusion, Bayesian cognitive models may help identify predictive biomarkers of relapse, while providing a computational explanation of belief processing and updating deficits in individuals with methamphetamine use disorder.

•

Methamphetamine-dependent individuals (MDI) face a high rate of relapse after treatment.

•

Can a Bayesian learning modeling and fMRI be used to identify markers of relapse?

•

MDI who relapsed within 1 year have smaller activation to Bayesian model-based prediction errors.

•

Such neural pattern was observed in left temporo-parietal junction, IFG, and anterior insula.

•

MDI more likely to relapse show weaker tracking of uncertainty and updating of their belief model.

Context-Sensitive Computational Mechanisms of Decision Making

Real-world information is primarily sensory in nature, and understandably people attach value to the sensory information to prepare for appropriate behavioral responses. This review presents research from value-based, perceptual, and social decision-making domains, so far studied using isolated paradigms and their corresponding computational models. For example, in perceptual decision making, the sensory evidence accumulation rather than value computation becomes central to choice behavior. Furthermore, we identify cross-linkages between the perceptual and value-based domains to help us better understand generic processes pertaining to individual decision making. The purpose of this review is 2-fold. First, we identify the need for integrated study of different domains of decision making. Second, given that both our perception and valuation are influenced by the surrounding context, we suggest the integration of different types of information in decision making could be done by studying contextual influences in decision making. Future research needs to attempt toward a system-level understanding of various subprocesses involved in decision making.

Gain control explains the effect of distraction in human perceptual, cognitive, and economic decision making

When making decisions, humans are often distracted by irrelevant information. Distraction has a different impact on perceptual, cognitive, and value-guided choices, giving rise to well-described behavioral phenomena such as the tilt illusion, conflict adaptation, or economic decoy effects. However, a single, unified model that can account for all these phenomena has yet to emerge. Here, we offer one such account, based on adaptive gain control, and additionally show that it successfully predicts a range of counterintuitive new behavioral phenomena on variants of a classic cognitive paradigm, the Eriksen flanker task. We also report that blood oxygen level-dependent signals in a dorsal network prominently including the anterior cingulate cortex index a gain-modulated decision variable predicted by the model. This work unifies the study of distraction across perceptual, cognitive, and economic domains.

Risk preference and choice stochasticity during decisions for other people

In several contexts, such as finance and politics, people make choices that are relevant for others but irrelevant for oneself. Focusing on decision-making under risk, we compared monetary choices made for one's own interest with choices made on behalf of an anonymous individual. Consistent with the previous literature, other-interest choices were characterized by an increased gambling propensity. We also investigated choice stochasticity, which captures how much decisions vary in similar conditions. An aspect related to choice stochasticity is how much decisions are tuned to the option values, and we found that this was higher during self-interest than during other-interest choices. This effect was observed only in individuals who reported a motivation to distribute rewards unequally, suggesting that it may (at least partially) depend on a motivation to make accurate decisions for others. Our results indicate that, during decision-making under risk, choices for other people are characterized by a decreased tuning to the values of the options, in addition to enhanced risk seeking.

Local Choices: Rationality and the Contextuality of Decision-Making

Rational explanation is ubiquitous in psychology and social sciences, ranging from rational analysis, expectancy-value theories, ideal observer models, mental logic to probabilistic frameworks, rational choice theory, and informal "folk psychological" explanation. However, rational explanation appears to be challenged by apparently systematic irrationality observed in psychological experiments, especially in the field of judgement and decision-making (JDM). Here, it is proposed that the experimental results require not that rational explanation should be rejected, but that rational explanation is local, i.e., within a context. Thus, rational models need to be supplemented with a theory of contextual shifts. We review evidence in JDM that patterns of choices are often consistent within contexts, but unstable between contexts. We also demonstrate that for a limited, though reasonably broad, class of decision-making domains, recent theoretical models can be viewed as providing theories of contextual shifts. It is argued that one particular significant source of global inconsistency arises from a cognitive inability to represent absolute magnitudes, whether for perceptual variables, utilities, payoffs, or probabilities. This overall argument provides a fresh perspective on the scope and limits of human rationality.