Building Bridges between Perceptual and Economic Decision-Making: Neural and Computational Mechanisms

The online metacognitive control of decisions

Difficult decisions typically involve mental effort, which scales with the deployment of cognitive (e.g., mnesic, attentional) resources engaged in processing decision-relevant information. But how does the brain regulate mental effort? A possibility is that the brain optimizes a resource allocation problem, whereby the amount of invested resources balances its expected cost (i.e. effort) and benefit. Our working assumption is that subjective decision confidence serves as the benefit term of the resource allocation problem, hence the "metacognitive" nature of decision control. Here, we present a computational model for the online metacognitive control of decisions or oMCD. Formally, oMCD is a Markov Decision Process that optimally solves the ensuing resource allocation problem under agnostic assumptions about the inner workings of the underlying decision system. We demonstrate how this makes oMCD a quasi-optimal control policy for a broad class of decision processes, including -but not limited to- progressive attribute integration. We disclose oMCD's main properties (in terms of choice, confidence and response time), and show that they reproduce most established empirical results in the field of value-based decision making. Finally, we discuss the possible connections between oMCD and most prominent neurocognitive theories about decision control and mental effort regulation.

The Future of Decisions From Experience: Connecting Real-World Decision Problems to Cognitive Processes

In many important real-world decision domains, such as finance, the environment, and health, behavior is strongly influenced by experience. Renewed interest in studying this influence led to important advancements in the understanding of these decisions from experience (DfE) in the last 20 years. Building on this literature, we suggest ways the standard experimental design should be extended to better approach important real-world DfE. These extensions include, for example, introducing more complex choice situations, delaying feedback, and including social interactions. When acting upon experiences in these richer and more complicated environments, extensive cognitive processes go into making a decision. Therefore, we argue for integrating cognitive processes more explicitly into experimental research in DfE. These cognitive processes include attention to and perception of numeric and nonnumeric experiences, the influence of episodic and semantic memory, and the mental models involved in learning processes. Understanding these basic cognitive processes can advance the modeling, understanding and prediction of DfE in the laboratory and in the real world. We highlight the potential of experimental research in DfE for theory integration across the behavioral, decision, and cognitive sciences. Furthermore, this research could lead to new methodology that better informs decision-making and policy interventions.

Corrective feedback guides human perceptual decision-making by informing about the world state rather than rewarding its choice

Corrective feedback received on perceptual decisions is crucial for adjusting decision-making strategies to improve future choices. However, its complex interaction with other decision components, such as previous stimuli and choices, challenges a principled account of how it shapes subsequent decisions. One popular approach, based on animal behavior and extended to human perceptual decision-making, employs "reinforcement learning," a principle proven successful in reward-based decision-making. The core idea behind this approach is that decision-makers, although engaged in a perceptual task, treat corrective feedback as rewards from which they learn choice values. Here, we explore an alternative idea, which is that humans consider corrective feedback on perceptual decisions as evidence of the actual state of the world rather than as rewards for their choices. By implementing these "feedback-as-reward" and "feedback-as-evidence" hypotheses on a shared learning platform, we show that the latter outperforms the former in explaining how corrective feedback adjusts the decision-making strategy along with past stimuli and choices. Our work suggests that humans learn about what has happened in their environment rather than the values of their own choices through corrective feedback during perceptual decision-making.

Contributions of the Basal Ganglia to Visual Perceptual Decisions

The basal ganglia (BG) make up a prominent nexus between visual and motor-related brain regions. In contrast to the BG's well-established roles in movement control and value-based decision making, their contributions to the transformation of visual input into an action remain unclear, especially in the context of perceptual decisions based on uncertain visual evidence. This article reviews recent progress in our understanding of the BG's contributions to the formation, evaluation, and adjustment of such decisions. From theoretical and experimental perspectives, the review focuses on four key stations in the BG network, namely, the striatum, pallidum, subthalamic nucleus, and midbrain dopamine neurons, which can have different roles and together support the decision process.

Midfrontal theta as an index of conflict strength in approach-approach vs avoidance-avoidance conflicts

The seminal theory of motivational conflicts distinguishes between approach-approach (AP-AP) conflicts, in which a decision is made between desirable alternatives, and avoidance-avoidance (AV-AV) conflicts, in which a decision is made between undesirable alternatives. The behavioral differences between AP-AP and AV-AV conflicts are well documented: abundant research showed that AV-AV conflicts are more difficult to resolve than AP-AP ones. However, there is little to no research looking into the neural underpinnings of the differences between the two conflict types. Here, we show that midfrontal theta, an established neural marker of conflict, distinguished between the two conflict types such that midfrontal theta power was higher in AV-AV conflicts than in AP-AP conflicts. We further demonstrate that higher midfrontal theta power was associated with shorter decision times on a single-trial basis, indicating that midfrontal theta played a role in promoting successful controlled behavior. Taken together, our results show that AP-AP and AV-AV conflicts are distinguishable on the neural level. The implications of these results go beyond motivational conflicts, as they establish midfrontal theta as a measure of the continuous degree of conflict in subjective decisions.

Hierarchical inference as a source of human biases

The finding that human decision-making is systematically biased continues to have an immense impact on both research and policymaking. Prevailing views ascribe biases to limited computational resources, which require humans to resort to less costly resource-rational heuristics. Here, we propose that many biases in fact arise due to a computationally costly way of coping with uncertainty-namely, hierarchical inference-which by nature incorporates information that can seem irrelevant. We show how, in uncertain situations, Bayesian inference may avail of the environment's hierarchical structure to reduce uncertainty at the cost of introducing bias. We illustrate how this account can explain a range of familiar biases, focusing in detail on the halo effect and on the neglect of base rates. In each case, we show how a hierarchical-inference account takes the characterization of a bias beyond phenomenological description by revealing the computations and assumptions it might reflect. Furthermore, we highlight new predictions entailed by our account concerning factors that could mitigate or exacerbate bias, some of which have already garnered empirical support. We conclude that a hierarchical inference account may inform scientists and policy makers with a richer understanding of the adaptive and maladaptive aspects of human decision-making.

Stable individual differences in unfamiliar face identification: Evidence from simultaneous and sequential matching tasks

Matching identity in images of unfamiliar faces is difficult: Images of the same person can look different and images of different people can look similar. Recent studies have capitalized on individual differences in the ability to distinguish match (same ID) vs. mismatch (different IDs) face pairs to inform models of face recognition. We addressed two significant gaps in the literature by examining the stability of individual differences in both sensitivity to identity and response bias. In Study 1, 210 participants completed a battery of four tasks in each of two sessions separated by one week. Tasks varied in protocol (same/different, lineup, sorting) and stimulus characteristics (low vs. high within-person variability in appearance). In Study 2, 148 participants completed a battery of three tasks in a single session. Stimuli were presented simultaneously on some trials and sequentially on others, introducing short-term memory demands. Principal components analysis revealed two components that were stable across time and tasks: sensitivity to identity and bias. Analyses of response times suggest that individual differences in bias reflect decision-making processes. We discuss the implications of our findings in applied settings and for models of face recognition.

A comparison of distributed machine learning methods for the support of “many labs” collaborations in computational modeling of decision making

Deep learning models are powerful tools for representing the complex learning processes and decision-making strategies used by humans. Such neural network models make fewer assumptions about the underlying mechanisms thus providing experimental flexibility in terms of applicability. However, this comes at the cost of involving a larger number of parameters requiring significantly more data for effective learning. This presents practical challenges given that most cognitive experiments involve relatively small numbers of subjects. Laboratory collaborations are a natural way to increase overall dataset size. However, data sharing barriers between laboratories as necessitated by data protection regulations encourage the search for alternative methods to enable collaborative data science. Distributed learning, especially federated learning (FL), which supports the preservation of data privacy, is a promising method for addressing this issue. To verify the reliability and feasibility of applying FL to train neural networks models used in the characterization of decision making, we conducted experiments on a real-world, many-labs data pool including experiment data-sets from ten independent studies. The performance of single models trained on single laboratory data-sets was poor. This unsurprising finding supports the need for laboratory collaboration to train more reliable models. To that end we evaluated four collaborative approaches. The first approach represents conventional centralized learning (CL-based) and is the optimal approach but requires complete sharing of data which we wish to avoid. The results however establish a benchmark for the other three approaches, federated learning (FL-based), incremental learning (IL-based), and cyclic incremental learning (CIL-based). We evaluate these approaches in terms of prediction accuracy and capacity to characterize human decision-making strategies. The FL-based model achieves performance most comparable to that of the CL-based model. This indicates that FL has value in scaling data science methods to data collected in computational modeling contexts when data sharing is not convenient, practical or permissible.

Efficient coding of cognitive variables underlies dopamine response and choice behavior

Reward expectations based on internal knowledge of the external environment are a core component of adaptive behavior. However, internal knowledge may be inaccurate or incomplete due to errors in sensory measurements. Some features of the environment may also be encoded inaccurately to minimize representational costs associated with their processing. In this study, we investigated how reward expectations are affected by features of internal representations by studying behavior and dopaminergic activity while mice make time-based decisions. We show that several possible representations allow a reinforcement learning agent to model animals' overall performance during the task. However, only a small subset of highly compressed representations simultaneously reproduced the co-variability in animals' choice behavior and dopaminergic activity. Strikingly, these representations predict an unusual distribution of response times that closely match animals' behavior. These results inform how constraints of representational efficiency may be expressed in encoding representations of dynamic cognitive variables used for reward-based computations.

Strategy-dependent effects of working-memory limitations on human perceptual decision-making

Deliberative decisions based on an accumulation of evidence over time depend on working memory, and working memory has limitations, but how these limitations affect deliberative decision-making is not understood. We used human psychophysics to assess the impact of working-memory limitations on the fidelity of a continuous decision variable. Participants decided the average location of multiple visual targets. This computed, continuous decision variable degraded with time and capacity in a manner that depended critically on the strategy used to form the decision variable. This dependence reflected whether the decision variable was computed either: (1) immediately upon observing the evidence, and thus stored as a single value in memory; or (2) at the time of the report, and thus stored as multiple values in memory. These results provide important constraints on how the brain computes and maintains temporally dynamic decision variables.

Working memory, the brain’s ability to temporarily store and recall information, is a critical part of decision making – but it has its limits. The brain can only store so much information, for so long. Since decisions are not often acted on immediately, information held in working memory ‘degrades’ over time. However, it is unknown whether or not this degradation of information over time affects the accuracy of later decisions.

The tactics that people use, knowingly or otherwise, to store information in working memory also remain unclear. Do people store pieces of information such as numbers, objects and particular details? Or do they tend to compute that information, make some preliminary judgement and recall their verdict later? Does the strategy chosen impact people’s decision-making?

To investigate, Schapiro et al. devised a series of experiments to test whether the limitations of working memory, and how people store information, affect the accuracy of decisions they make. First, participants were shown an array of colored discs on a screen. Then, either immediately after seeing the disks or a few seconds later, the participants were asked to recall the position of one of the disks they had seen, or the average position of all the disks. This measured how much information degraded for a decision based on multiple items, and how much for a decision based on a single item. From this, the method of information storage used to make a decision could be inferred.

Schapiro et al. found that the accuracy of people’s responses worsened over time, whether they remembered the position of each individual disk, or computed their average location before responding. The greater the delay between seeing the disks and reporting their location, the less accurate people’s responses tended to be. Similarly, the more disks a participant saw, the less accurate their response became. This suggests that however people store information, if working memory reaches capacity, decision-making suffers and that, over time, stored information decays.

Schapiro et al. also noticed that participants remembered location information in different ways depending on the task and how many disks they were shown at once. This suggests people adopt different strategies to retain information momentarily.

In summary, these findings help to explain how people process and store information to make decisions and how the limitations of working memory impact their decision-making ability. A better understanding of how people use working memory to make decisions may also shed light on situations or brain conditions where decision-making is impaired.

Humans represent the precision and utility of information acquired across fixations

Our environment contains an abundance of objects which humans interact with daily, gathering visual information using sequences of eye-movements to choose which object is best-suited for a particular task. This process is not trivial, and requires a complex strategy where task affordance defines the search strategy, and the estimated precision of the visual information gathered from each object may be used to track perceptual confidence for object selection. This study addresses the fundamental problem of how such visual information is metacognitively represented and used for subsequent behaviour, and reveals a complex interplay between task affordance, visual information gathering, and metacogntive decision making. People fixate higher-utility objects, and most importantly retain metaknowledge about how much information they have gathered about these objects, which is used to guide perceptual report choices. These findings suggest that such metacognitive knowledge is important in situations where decisions are based on information acquired in a temporal sequence.

Extracting Summary Statistics of Rapid Numerical Sequences

We examine the ability of observers to extract summary statistics (such as the mean and the relative-variance) from rapid numerical sequences of two digit numbers presented at a rate of 4/s. In four experiments (total N = 100), we find that the participants show a remarkable ability to extract such summary statistics and that their precision in the estimation of the sequence-mean improves with the sequence-length (subject to individual differences). Using model selection for individual participants we find that, when only the sequence-average is estimated, most participants rely on a holistic process of frequency based estimation with a minority who rely on a (rule-based and capacity limited) mid-range strategy. When both the sequence-average and the relative variance are estimated, about half of the participants rely on these two strategies. Importantly, the holistic strategy appears more efficient in terms of its precision. We discuss implications for the domains of two pathways numerical processing and decision-making.

Normative Principles for Decision-Making in Natural Environments

The decisions we make are shaped by a lifetime of learning. Past experience guides the way that we encode information in neural systems for perception and valuation, and determines the information we retrieve when making decisions. Distinct literatures have discussed how lifelong learning and local context shape decisions made about sensory signals, propositional information, or economic prospects. Here, we build bridges between these literatures, arguing for common principles of adaptive rationality in perception, cognition, and economic choice. We discuss how a single common framework, based on normative principles of efficient coding and Bayesian inference, can help us understand a myriad of human decision biases, including sensory illusions, adaptive aftereffects, choice history biases, central tendency effects, anchoring effects, contrast effects, framing effects, congruency effects, reference-dependent valuation, nonlinear utility functions, and discretization heuristics. We describe a simple computational framework for explaining these phenomena. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Consumers’ ability to identify a surplus when returns to attributes are nonlinear

Previous research in multiple judgment domains has found that nonlinear functions are typically processed less accurately than linear ones. This empirical regularity has potential implications for consumer choice, given that nonlinear functions (e.g., diminishing returns) are commonplace. In two experimental studies we measured precision and bias in consumers’ ability to identify surpluses when returns to product attributes were nonlinear. We hypothesized that nonlinear functions would reduce precision and induce bias toward linearization of nonlinear relationships. Neither hypothesis was supported for monotonic nonlinearities. However, precision was greatly reduced for products with nonmonotonic attributes. Moreover, assessments of surplus were systematically and strongly biased, regardless of the shape of returns and despite feedback and incentives. The findings imply that consumers use a flexible but coarse mechanism to compare attributes against prices, with implications for the prevalence of costly mistakes.

Event-related brain potentials reflect predictive coding of anticipated economic change

Research has demonstrated the importance of economic forecasts for financial decisions at the aggregate economic level. However, little is known about the psychological and neurophysiological mechanisms that economic forecasts activate at the level of individual decision-making. In the present study, we used event-related brain potentials (ERPs) to test the hypothesis that economic forecasts influence individuals’ internal model of the economy and their subsequent decision behavior. Using a simple economic decision-making game, the Balloon Analogue of Risk Task (BART) and predictive messages about possible economic changes in the game before each block, we test the idea that brain potentials time-locked to decision outcomes can vary as a function of exposure to economic forecasts. Behavioural results indicate that economic forecasts influenced the amount of risk that participants were willing to take. Analyses of brain potentials indicated parametric increases of the N1, P2, P3a, and P3b amplitudes as a function of the level of risk in subsequent inflation steps in the BART. Mismatches between economic forecasts and decision outcomes in the BART (i.e., reward prediction errors) were reflected in the amplitude of the P2, P3a, and P3b, suggesting increased attentional processing of unexpected outcomes. These electrophysiological results corroborate the idea that economic messages may indeed influence people’s beliefs about the economy and bias their subsequent financial decision-making. Our findings present a first important step in the development of a low-level neurophysiological model that may help to explain the self-fulfilling prophecy effect of economic news in the larger economy.

Does Attention Increase the Value of Choice Alternatives?

A growing recognition of the role of attention in decision-making has been driven by both the technology of eye tracking and the development of models that explicitly incorporate attention. One result of this convergence is the arresting claim that attention, by itself, can increase the perceived value of a decision alternative. In this review, we cover the origins of that claim, its empirical foundation, and the reasoning that supports it. The conclusion is that, to date, there is not sufficient evidence to support the claim. Alternative explanations for the extant evidentiary base are discussed, as is the balance between the bottom-up influence of empirical evidence and the top-down commitment to a conceptual framework.

Attraction to similar options: The Gestalt law of proximity is related to the attraction effect

Previous studies have suggested that there are common mechanisms between perceptual and value-based processes. For instance, both perceptual and value-based choices are highly influenced by the context in which the choices are made. However, the mechanisms which allow context to influence our choice process as well as the extent of the similarity between the perceptual and preferential processes are still unclear. In this study, we examine a within-subject relation between the attraction effect, which is a well-known effect of context on preferential choice, and the Gestalt law of proximity. Then, we aim to use this link to better understand the mechanisms underlying the attraction effect. We conducted one study followed by an additional pre-registered replication study, where subjects performed a Gestalt-psychophysical task and a decoy task. Comparing the behavioral sensitivity of each subject in both tasks, we found that the more susceptible a subject is to the proximity law, the more she displayed the attraction effect. These results demonstrate a within-subject relation between a perceptual phenomenon (proximity law) and a value-based bias (attraction effect) which further strengthens the notion of common rules between perceptual and value-based processing. Moreover, this suggests that the mechanism underlying the attraction effect is related to grouping by proximity with attention as a mediator.

The caudate nucleus contributes causally to decisions that balance reward and uncertain visual information

Our decisions often balance what we observe and what we desire. A prime candidate for implementing this complex balancing act is the basal ganglia pathway, but its roles have not yet been examined experimentally in detail. Here, we show that a major input station of the basal ganglia, the caudate nucleus, plays a causal role in integrating uncertain visual evidence and reward context to guide adaptive decision-making. In monkeys making saccadic decisions based on motion cues and asymmetric reward-choice associations, single caudate neurons encoded both sources of information. Electrical microstimulation at caudate sites during motion viewing affected the monkeys’ decisions. These microstimulation effects included coordinated changes in multiple computational components of the decision process that mimicked the monkeys’ similarly coordinated voluntary strategies for balancing visual and reward information. These results imply that the caudate nucleus plays causal roles in coordinating decision processes that balance external evidence and internal preferences.

Dissociable mechanisms govern when and how strongly reward attributes affect decisions

Theories and computational models of decision-making usually focus on how strongly different attributes are weighted in choice, for example, as a function of their importance or salience to the decision-maker. However, when different attributes affect the decision process is a question that has received far less attention. Here, we investigated whether the timing of attribute consideration has a unique influence on decision-making by using a time-varying drift diffusion model and data from four separate experiments. Experimental manipulations of attention and neural activity demonstrated that we can dissociate the processes that determine the relative weighting strength and timing of attribute consideration. Thus, the processes determining either the weighting strengths or the timing of attributes in decision-making can independently adapt to changes in the environment or goals. Quantifying these separate influences of timing and weighting on choice improves our understanding and predictions of individual differences in decision behaviour.

Can the VTA Come Out to Play? Only When the mPFC's Predictions Go Astray!

Confidence in perceptual decisions scales neural responses to violations in reward expectation. In this issue of Neuron, Lak et al. (2020) show that the medial prefrontal cortex in mice computes a confidence-dependent expectation signal that influences how dopamine neurons convey reward prediction errors to guide learning.

Comparison of Markov versus quantum dynamical models of human decision making

What kind of dynamic decision process do humans use to make decisions? In this article, two different types of processes are reviewed and compared: Markov and quantum. Markov processes are based on the idea that at any given point in time a decision maker has a definite and specific level of support for available choice alternatives, and the dynamic decision process is represented by a single trajectory that traces out a path across time. When a response is requested, a person's decision or judgment is generated from the current location along the trajectory. By contrast, quantum processes are founded on the idea that a person's state can be represented by a superposition over different degrees of support for available choice options, and that the dynamics of this state form a wave moving across levels of support over time. When a response is requested, a decision or judgment is constructed out of the superposition by "actualizing" a specific degree or range of degrees of support to create a definite state. The purpose of this article is to introduce these two contrasting theories, review empirical studies comparing the two theories, and identify conditions that determine when each theory is more accurate and useful than the other. This article is categorized under: Economics > Individual Decision-Making Psychology > Reasoning and Decision Making Psychology > Theory and Methods.

Dopaminergic and Prefrontal Basis of Learning from Sensory Confidence and Reward Value

Deciding between stimuli requires combining their learned value with one's sensory confidence. We trained mice in a visual task that probes this combination. Mouse choices reflected not only present confidence and past rewards but also past confidence. Their behavior conformed to a model that combines signal detection with reinforcement learning. In the model, the predicted value of the chosen option is the product of sensory confidence and learned value. We found precise correlates of this variable in the pre-outcome activity of midbrain dopamine neurons and of medial prefrontal cortical neurons. However, only the latter played a causal role: inactivating medial prefrontal cortex before outcome strengthened learning from the outcome. Dopamine neurons played a causal role only after outcome, when they encoded reward prediction errors graded by confidence, influencing subsequent choices. These results reveal neural signals that combine reward value with sensory confidence and guide subsequent learning.

The Art of Influencing Consumer Choices: A Reflection on Recent Advances in Decision Neuroscience

In recent years, our knowledge concerning the neurobiology of choice has increased tremendously. Research in the field of decision-making has identified important brain mechanisms by which a representation of the subjective value of an option is built based on previous experience, retrieved and compared to that of other available options in order to make a choice. One body of research, in particular, has focused on simple value-based choices (e.g., choices between two types of fruits) to study situations very similar to our daily life decisions as consumers. The use of neuroimaging techniques has deepened and refined our knowledge of decision processes. Additionally, computational approaches have helped identifying and describing the mechanisms underlying newly found components of the decisional process. They provide mechanistic explanations for diverse biases that can drive decision makers away from their own preferences or from rational choices. It is now clear that both attentional and affective factors can exert robust effects on an individual's decisions. Because these factors can be manipulated externally, academic research and theories are of great interest to the marketing industry. This approach is becoming increasingly effective in manipulating consumer behavior and has the potential to become even more effective in the future. Another line of research has revealed differences in the decision-making neural circuitry that underlie sub-optimal choice behavior, rendering some individuals particularly vulnerable to marketing strategies. As neuroscientists, we wonder whether relevant institutions should direct their efforts toward raising citizens' awareness, demanding more transparency on marketing applications and regulate the most pervasive communication techniques in marketing, in view of their current use and of recent research progress.

Brain Mechanisms of Concept Learning

Concept learning, the ability to extract commonalities and highlight distinctions across a set of related experiences to build organized knowledge, is a critical aspect of cognition. Previous reviews have focused on concept learning research as a means for dissociating multiple brain systems. The current review surveys recent work that uses novel analytical approaches, including the combination of computational modeling with neural measures, focused on testing theories of specific computations and representations that contribute to concept learning. We discuss in detail the roles of the hippocampus, ventromedial prefrontal, lateral prefrontal, and lateral parietal cortices, and how their engagement is modulated by the coherence of experiences and the current learning goals. We conclude that the interaction of multiple brain systems relating to learning, memory, attention, perception, and reward support a flexible concept-learning mechanism that adapts to a range of category structures and incorporates motivational states, making concept learning a fruitful research domain for understanding the neural dynamics underlying complex behaviors.

A reinforcement learning diffusion decision model for value-based decisions

Psychological models of value-based decision-making describe how subjective values are formed and mapped to single choices. Recently, additional efforts have been made to describe the temporal dynamics of these processes by adopting sequential sampling models from the perceptual decision-making tradition, such as the diffusion decision model (DDM). These models, when applied to value-based decision-making, allow mapping of subjective values not only to choices but also to response times. However, very few attempts have been made to adapt these models to situations in which decisions are followed by rewards, thereby producing learning effects. In this study, we propose a new combined reinforcement learning diffusion decision model (RLDDM) and test it on a learning task in which pairs of options differ with respect to both value difference and overall value. We found that participants became more accurate and faster with learning, responded faster and more accurately when options had more dissimilar values, and decided faster when confronted with more attractive (i.e., overall more valuable) pairs of options. We demonstrate that the suggested RLDDM can accommodate these effects and does so better than previously proposed models. To gain a better understanding of the model dynamics, we also compare it to standard DDMs and reinforcement learning models. Our work is a step forward towards bridging the gap between two traditions of decision-making research.

A theoretical analysis of the reward rate optimality of collapsing decision criteria

A standard assumption of most sequential sampling models is that decision-makers rely on a decision criterion that remains constant throughout the decision process. However, several authors have recently suggested that, in order to maximize reward rates in dynamic environments, decision-makers need to rely on a decision criterion that changes over the course of the decision process. We used dynamic programming and simulation methods to quantify the reward rates obtained by constant and dynamic decision criteria in different environments. We further investigated what influence a decision-maker's uncertainty about the stochastic structure of the environment has on reward rates. Our results show that in most dynamic environments, both types of decision criteria yield similar reward rates, across different levels of uncertainty. This suggests that a static decision criterion might provide a robust default setting.

Vicarious Rewards Modulate the Drift Rate of Evidence Accumulation From the Drift Diffusion Model

Taking other people's interests into account is a fundamental ability allowing humans to maintain relationships. Yet, the mechanisms by which monetary incentives for close others influence perceptual decision-making processes remain elusive. Here, we compared perceptual decisions motivated by payoffs for oneself or a close relative. According to drift diffusion models (DDMs), perceptual decisions are made when sensory evidence accumulated over time - with a given drift rate - reaches one of the decision boundaries. We used these computational models to identify whether the drift rate of evidence accumulation or the decision boundary is affected by these two sources of motivation. Reaction times and sensitivity were modulated by three factors: the Difficulty (motion coherence of the moving dots), the Payoff associated with, and the Beneficiary of the decision. Reaction times (RTs) were faster for easy compared to difficult trials and faster for high payoffs as compared to low payoffs. More interestingly, RTs were also faster for self than for other-affecting decisions. Finally, using DDM, we found that these faster RTs were linked to a higher drift rate of the decision variable. This study offers a mechanistic understanding of how incentives for others and motion coherence influence decision-making processes.

Decomposing the effects of context valence and feedback information on speed and accuracy during reinforcement learning: a meta-analytical approach using diffusion decision modeling

Reinforcement learning (RL) models describe how humans and animals learn by trial-and-error to select actions that maximize rewards and minimize punishments. Traditional RL models focus exclusively on choices, thereby ignoring the interactions between choice preference and response time (RT), or how these interactions are influenced by contextual factors. However, in the field of perceptual decision-making, such interactions have proven to be important to dissociate between different underlying cognitive processes. Here, we investigated such interactions to shed new light on overlooked differences between learning to seek rewards and learning to avoid losses. We leveraged behavioral data from four RL experiments, which feature manipulations of two factors: outcome valence (gains vs. losses) and feedback information (partial vs. complete feedback). A Bayesian meta-analysis revealed that these contextual factors differently affect RTs and accuracy: While valence only affects RTs, feedback information affects both RTs and accuracy. To dissociate between the latent cognitive processes, we jointly fitted choices and RTs across all experiments with a Bayesian, hierarchical diffusion decision model (DDM). We found that the feedback manipulation affected drift rate, threshold, and non-decision time, suggesting that it was not a mere difficulty effect. Moreover, valence affected non-decision time and threshold, suggesting a motor inhibition in punishing contexts. To better understand the learning dynamics, we finally fitted a combination of RL and DDM (RLDDM). We found that while the threshold was modulated by trial-specific decision conflict, the non-decision time was modulated by the learned context valence. Overall, our results illustrate the benefits of jointly modeling RTs and choice data during RL, to reveal subtle mechanistic differences underlying decisions in different learning contexts.

The Neuro-Computational Architecture of Value-Based Selection in the Human Brain

Current neural models of value-based decision-making consider choices as a 2-stage process, proceeding from the "valuation" of each option under consideration to the "selection" of the best option on the basis of their subjective values. However, little is known about the computational mechanisms at play at the selection stage and its implementation in the human brain. Here, we used drift-diffusion models combined with model-based functional magnetic resonance imaging, effective connectivity, and multivariate pattern analysis to characterize the neuro-computational architecture of value-based decisions. We found that 2 key drift-diffusion computations at the selection stage, namely integration and choice readout, engage distinct brain regions, with the dorsolateral prefrontal cortex integrating a decision value signal computed in the ventromedial prefrontal cortex, and the posterior parietal cortex reading out choice outcomes. Our findings suggest that this prefronto-parietal network acts as a hub implementing behavioral selection through a distributed drift-diffusion process.

Proactive Information Sampling in Value-Based Decision-Making: Deciding When and Where to Saccade

Evidence accumulation has been the core component in recent development of perceptual and value-based decision-making theories. Most studies have focused on the evaluation of evidence between alternative options. What remains largely unknown is the process that prepares evidence: how may the decision-maker sample different sources of information sequentially, if they can only sample one source at a time? Here we propose a theoretical framework in prescribing how different sources of information should be sampled to facilitate the decision process: beliefs for different noisy sources are updated in a Bayesian manner and participants can proactively allocate resource for sampling (i.e., saccades) among different sources to maximize the information gain in such process. We show that our framework can account for human participants' actual choice and saccade behavior in a two-alternative value-based decision-making task. Moreover, our framework makes novel predictions about the empirical eye movement patterns.

Atomoxetine modulates the relationship between perceptual abilities and response bias

Elucidation of how neuromodulators influence motivated behaviors is a major challenge of neuroscience research. It has been proposed that the locus-cœruleus-norepinephrine system promotes behavioral flexibility and provides resources required to face challenges in a wide range of cognitive processes. Both theoretical models and computational models suggest that the locus-cœruleus-norepinephrine system tunes neural gain in brain circuits to optimize behavior. However, to the best of our knowledge, empirical proof demonstrating the role of norepinephrine in performance optimization is scarce. Here, we modulated norepinephrine transmission in monkeys performing a Go/No-Go discrimination task using atomoxetine, a norepinephrine-reuptake inhibitor. We tested the optimization hypothesis by assessing perceptual sensitivity, response bias, and their functional relationship within the framework of the signal detection theory. We also manipulated the contingencies of the task (level of stimulus discriminability, target stimulus frequency, and decision outcome values) to modulate the relationship between sensitivity and response bias. We found that atomoxetine increased the subject's perceptual sensitivity to discriminate target stimuli regardless of the task contingency. Atomoxetine also improved the functional relationship between sensitivity and response bias, leading to a closer fit with the optimal strategy in different contexts. In addition, atomoxetine tended to reduce reaction time variability. Taken together, these findings support a role of norepinephrine transmission in optimizing response strategy.

Efficient coding of subjective value

Preference-based decisions are essential for survival, for instance when deciding what we should (not) eat. Despite their importance, preference-based decisions are surprisingly variable and can appear irrational in ways that have defied mechanistic explanations. Here we propose that subjective valuation results from an inference process that accounts for the structure of values in the environment and that maximizes information in value representations in line with demands imposed by limited coding resources. A model of this inference process explains the variability in both subjective value reports and preference-based choices, and predicts a new preference illusion that we validate with empirical data. Interestingly, the same model explains the level of confidence associated with these reports. Our results imply that preference-based decisions reflect information-maximizing transmission and statistically optimal decoding of subjective values by a limited-capacity system. These findings provide a unified account of how humans perceive and valuate the environment to optimally guide behavior.

Context-Sensitive Computational Mechanisms of Decision Making

Real-world information is primarily sensory in nature, and understandably people attach value to the sensory information to prepare for appropriate behavioral responses. This review presents research from value-based, perceptual, and social decision-making domains, so far studied using isolated paradigms and their corresponding computational models. For example, in perceptual decision making, the sensory evidence accumulation rather than value computation becomes central to choice behavior. Furthermore, we identify cross-linkages between the perceptual and value-based domains to help us better understand generic processes pertaining to individual decision making. The purpose of this review is 2-fold. First, we identify the need for integrated study of different domains of decision making. Second, given that both our perception and valuation are influenced by the surrounding context, we suggest the integration of different types of information in decision making could be done by studying contextual influences in decision making. Future research needs to attempt toward a system-level understanding of various subprocesses involved in decision making.

A Biased Bayesian Inference for Decision-Making and Cognitive Control

Although classical decision-making studies have assumed that subjects behave in a Bayes-optimal way, the sub-optimality that causes biases in decision-making is currently under debate. Here, we propose a synthesis based on exponentially-biased Bayesian inference, including various decision-making and probability judgments with different bias levels. We arrange three major parameter estimation methods in a two-dimensional bias parameter space (prior and likelihood), of the biased Bayesian inference. Then, we discuss a neural implementation of the biased Bayesian inference on the basis of changes in weights in neural connections, which we regarded as a combination of leaky/unstable neural integrator and probabilistic population coding. Finally, we discuss mechanisms of cognitive control which may regulate the bias levels.

Uncertainty and Exploration

In order to discover the most rewarding actions, agents must collect information about their environment, potentially foregoing reward. The optimal solution to this "explore-exploit" dilemma is often computationally challenging, but principled algorithmic approximations exist. These approximations utilize uncertainty about action values in different ways. Some random exploration algorithms scale the level of choice stochasticity with the level of uncertainty. Other directed exploration algorithms add a "bonus" to action values with high uncertainty. Random exploration algorithms are sensitive to total uncertainty across actions, whereas directed exploration algorithms are sensitive to relative uncertainty. This paper reports a multi-armed bandit experiment in which total and relative uncertainty were orthogonally manipulated. We found that humans employ both exploration strategies, and that these strategies are independently controlled by different uncertainty computations.

Mechanisms of deliberation during preferential choice: Perspectives from computational modeling and individual differences

Computational models of decision making typically assume as people deliberate between options they mentally simulate outcomes from each one and integrate valuations of these outcomes to form a preference. In two studies, we investigated this deliberation process using a task where participants make a series of decisions between a certain and an uncertain option, which were shown as dynamic visual samples that represented possible payoffs. We developed and validated a method of reverse correlational analysis for the task that measures how this time-varying signal was used to make a choice. The first study used this method to examine how information processing during deliberation differed from a perceptual analog of the task. We found participants were less sensitive to each sample of information during preferential choice. In a second study, we investigated how these different measures of deliberation were related to impulsivity and drug and alcohol use. We found that while properties of the deliberation process were not related to impulsivity, some aspects of the process may be related to substance use. In particular, alcohol abuse was related to diminished sensitivity to the payoff information and drug use was related to how the initial starting point of evidence accumulation. We synthesized our results with a rank-dependent sequential sampling model which suggests that participants allocated more attentional weight to larger potential payoffs during preferential choice.

Dissociable neural mechanisms track evidence accumulation for selection of attention versus action

Decision-making is typically studied as a sequential process from the selection of what to attend (e.g., between possible tasks, stimuli, or stimulus attributes) to which actions to take based on the attended information. However, people often process information across these various levels in parallel. Here we scan participants while they simultaneously weigh how much to attend to two dynamic stimulus attributes and what response to give. Regions of the prefrontal cortex track information about the stimulus attributes in dissociable ways, related to either the predicted reward (ventromedial prefrontal cortex) or the degree to which that attribute is being attended (dorsal anterior cingulate cortex, dACC). Within the dACC, adjacent regions track correlates of uncertainty at different levels of the decision, regarding what to attend versus how to respond. These findings bridge research on perceptual and value-based decision-making, demonstrating that people dynamically integrate information in parallel across different levels of decision-making.

Rationality, perception, and the all-seeing eye

Seeing-perception and vision-is implicitly the fundamental building block of the literature on rationality and cognition. Herbert Simon and Daniel Kahneman's arguments against the omniscience of economic agents-and the concept of bounded rationality-depend critically on a particular view of the nature of perception and vision. We propose that this framework of rationality merely replaces economic omniscience with perceptual omniscience. We show how the cognitive and social sciences feature a pervasive but problematic meta-assumption that is characterized by an "all-seeing eye." We raise concerns about this assumption and discuss different ways in which the all-seeing eye manifests itself in existing research on (bounded) rationality. We first consider the centrality of vision and perception in Simon's pioneering work. We then point to Kahneman's work-particularly his article "Maps of Bounded Rationality"-to illustrate the pervasiveness of an all-seeing view of perception, as manifested in the extensive use of visual examples and illusions. Similar assumptions about perception can be found across a large literature in the cognitive sciences. The central problem is the present emphasis on inverse optics-the objective nature of objects and environments, e.g., size, contrast, and color. This framework ignores the nature of the organism and perceiver. We argue instead that reality is constructed and expressed, and we discuss the species-specificity of perception, as well as perception as a user interface. We draw on vision science as well as the arts to develop an alternative understanding of rationality in the cognitive and social sciences. We conclude with a discussion of the implications of our arguments for the rationality and decision-making literature in cognitive psychology and behavioral economics, along with suggesting some ways forward.

Deconstructing the human algorithms for exploration

The dilemma between information gathering (exploration) and reward seeking (exploitation) is a fundamental problem for reinforcement learning agents. How humans resolve this dilemma is still an open question, because experiments have provided equivocal evidence about the underlying algorithms used by humans. We show that two families of algorithms can be distinguished in terms of how uncertainty affects exploration. Algorithms based on uncertainty bonuses predict a change in response bias as a function of uncertainty, whereas algorithms based on sampling predict a change in response slope. Two experiments provide evidence for both bias and slope changes, and computational modeling confirms that a hybrid model is the best quantitative account of the data.

Sensory uncertainty impacts avoidance during spatial decisions

When making risky spatial decisions, humans incorporate estimates of sensorimotor variability and costs on outcomes to bias their spatial selections away from regions that incur feedback penalties. Since selection variability depends on the reliability of sensory signals, increasing the spatial variance of targets during visually guided actions should increase the degree of this avoidance. Healthy adult participants (N = 20) used a computer mouse to indicate their selection of the mean of a target, represented as a 2D Gaussian distribution of dots presented on a computer display. Reward feedback on each trial corresponded to the estimation error of the selection. Either increasing or decreasing the spatial variance of the dots modulated the spatial uncertainty of the target. A non-target distractor cue was presented as an adjacent distribution of dots. On a subset of trials, feedback scores were penalized with increased proximity to the distractor mean. As expected, increasing the spatial variance of the target distribution increased selection variability. More importantly, on trials where proximity to the distractor cue incurred a penalty, increasing variance of the target increased selection bias away from the distractor cue and prolonged reaction times. These results confirm predictions that increased sensory uncertainty increases avoidance during risky spatial decisions.

Value-based decision making under uncertainty in hoarding and obsessive- compulsive disorders

Difficulties in decision making are a core impairment in a range of disease states. For instance, both obsessive- compulsive disorder (OCD) and hoarding disorder (HD) are associated with indecisiveness, inefficient planning, and enhanced uncertainty intolerance, even in contexts unrelated to their core symptomology. We examined decision-making patterns in 19 individuals with OCD, 19 individuals with HD, 19 individuals with comorbid OCD and HD, and 57 individuals from the general population, using a well-validated choice task grounded in behavioral economic theory. Our results suggest that difficulties in decision making in individuals with OCD (with or without comorbid HD) are linked to reduced fidelity of value-based decision making (i.e. increase in inconsistent choices). In contrast, we find that performance of individuals with HD on our laboratory task is largely intact. Overall, these results support our hypothesis that decision-making impairments in OCD and HD, which can appear quite similar clinically, have importantly different underpinnings. Systematic investigation of different aspects of decision making, under varying conditions, may shed new light on commonalities between and distinctions among clinical syndromes.

A Perceptual-Like Population-Coding Mechanism of Approximate Numerical Averaging

Humans possess a remarkable ability to rapidly form coarse estimations of numerical averages. This ability is important for making decisions that are based on streams of numerical or value-based information, as well as for preference formation. Nonetheless, the mechanism underlying rapid approximate numerical averaging remains unknown, and several competing mechanism may account for it. Here, we tested the hypothesis that approximate numerical averaging relies on perceptual-like processes, instantiated by population coding. Participants were presented with rapid sequences of numerical values (four items per second) and were asked to convey the sequence average. We manipulated the sequences' length, variance, and mean magnitude and found that similar to perceptual averaging, the precision of the estimations improves with the length and deteriorates with (higher) variance or (higher) magnitude. To account for the results, we developed a biologically plausible population-coding model and showed that it is mathematically equivalent to a population vector. Using both quantitative and qualitative model comparison methods, we compared the population-coding model to several competing models, such as a step-by-step running average (based on leaky integration) and a midrange model. We found that the data support the population-coding model. We conclude that humans' ability to rapidly form estimations of numerical averages has many properties of the perceptual (intuitive) system rather than the arithmetic, linguistic-based (analytic) system and that population coding is likely to be its underlying mechanism.

Neural Signature of Value-Based Sensorimotor Prioritization in Humans

In situations in which impending sensory events demand fast action choices, we must be ready to prioritize higher-value courses of action to avoid missed opportunities. When such a situation first presents itself, stimulus-action contingencies and their relative value must be encoded to establish a value-biased state of preparation for an impending sensorimotor decision. Here, we sought to identify neurophysiological signatures of such processes in the human brain (both female and male). We devised a task requiring fast action choices based on the discrimination of a simple visual cue in which the differently valued sensory alternatives were presented 750-800 ms before as peripheral "targets" that specified the stimulus-action mapping for the upcoming decision. In response to the targets, we identified a discrete, transient, spatially selective signal in the event-related potential (ERP), which scaled with relative value and strongly predicted the degree of behavioral bias in the upcoming decision both across and within subjects. This signal is not compatible with any hitherto known ERP signature of spatial selection and also bears novel distinctions with respect to characterizations of value-sensitive, spatially selective activity found in sensorimotor areas of nonhuman primates. Specifically, a series of follow-up experiments revealed that the signal was reliably invoked regardless of response laterality, response modality, sensory feature, and reward valence. It was absent, however, when the response deadline was relaxed and the strategic need for biasing removed. Therefore, more than passively representing value or salience, the signal appears to play a versatile and active role in adaptive sensorimotor prioritization.SIGNIFICANCE STATEMENT In many situations such as fast-moving sports, we must be ready to act fast in response to sensory events and, in our preparation, prioritize courses of action that lead to greater rewards. Although behavioral effects of value biases in sensorimotor decision making have been widely studied, little is known about the neural processes that set these biases in place beforehand. Here, we report the discovery of a transient, spatially selective neural signal in humans that encodes the relative value of competing decision alternatives and strongly predicts behavioral value biases in decisions made ∼500 ms later. Follow-up manipulations of value differential, reward valence, response modality, sensory features, and time constraints establish that the signal reflects an active, feature- and effector-general preparatory mechanism for value-based prioritization.

Value-based decision making via sequential sampling with hierarchical competition and attentional modulation

In principle, formal dynamical models of decision making hold the potential to represent fundamental computations underpinning value-based (i.e., preferential) decisions in addition to perceptual decisions. Sequential-sampling models such as the race model and the drift-diffusion model that are grounded in simplicity, analytical tractability, and optimality remain popular, but some of their more recent counterparts have instead been designed with an aim for more feasibility as architectures to be implemented by actual neural systems. Connectionist models are proposed herein at an intermediate level of analysis that bridges mental phenomena and underlying neurophysiological mechanisms. Several such models drawing elements from the established race, drift-diffusion, feedforward-inhibition, divisive-normalization, and competing-accumulator models were tested with respect to fitting empirical data from human participants making choices between foods on the basis of hedonic value rather than a traditional perceptual attribute. Even when considering performance at emulating behavior alone, more neurally plausible models were set apart from more normative race or drift-diffusion models both quantitatively and qualitatively despite remaining parsimonious. To best capture the paradigm, a novel six-parameter computational model was formulated with features including hierarchical levels of competition via mutual inhibition as well as a static approximation of attentional modulation, which promotes "winner-take-all" processing. Moreover, a meta-analysis encompassing several related experiments validated the robustness of model-predicted trends in humans' value-based choices and concomitant reaction times. These findings have yet further implications for analysis of neurophysiological data in accordance with computational modeling, which is also discussed in this new light.

A unifying Bayesian account of contextual effects in value-based choice

Empirical evidence suggests the incentive value of an option is affected by other options available during choice and by options presented in the past. These contextual effects are hard to reconcile with classical theories and have inspired accounts where contextual influences play a crucial role. However, each account only addresses one or the other of the empirical findings and a unifying perspective has been elusive. Here, we offer a unifying theory of context effects on incentive value attribution and choice based on normative Bayesian principles. This formulation assumes that incentive value corresponds to a precision-weighted prediction error, where predictions are based upon expectations about reward. We show that this scheme explains a wide range of contextual effects, such as those elicited by other options available during choice (or within-choice context effects). These include both conditions in which choice requires an integration of multiple attributes and conditions where a multi-attribute integration is not necessary. Moreover, the same scheme explains context effects elicited by options presented in the past or between-choice context effects. Our formulation encompasses a wide range of contextual influences (comprising both within- and between-choice effects) by calling on Bayesian principles, without invoking ad-hoc assumptions. This helps clarify the contextual nature of incentive value and choice behaviour and may offer insights into psychopathologies characterized by dysfunctional decision-making, such as addiction and pathological gambling.

Learning Contextual Reward Expectations for Value Adaptation

Substantial evidence indicates that subjective value is adapted to the statistics of reward expected within a given temporal context. However, how these contextual expectations are learned is poorly understood. To examine such learning, we exploited a recent observation that participants performing a gambling task adjust their preferences as a function of context. We show that, in the absence of contextual cues providing reward information, an average reward expectation was learned from recent past experience. Learning dependent on contextual cues emerged when two contexts alternated at a fast rate, whereas both cue-independent and cue-dependent forms of learning were apparent when two contexts alternated at a slower rate. Motivated by these behavioral findings, we reanalyzed a previous fMRI data set to probe the neural substrates of learning contextual reward expectations. We observed a form of reward prediction error related to average reward such that, at option presentation, activity in ventral tegmental area/substantia nigra and ventral striatum correlated positively and negatively, respectively, with the actual and predicted value of options. Moreover, an inverse correlation between activity in ventral tegmental area/substantia nigra (but not striatum) and predicted option value was greater in participants showing enhanced choice adaptation to context. The findings help understanding the mechanisms underlying learning of contextual reward expectation.

The Attentional Drift Diffusion Model of Simple Perceptual Decision-Making

Perceptual decisions requiring the comparison of spatially distributed stimuli that are fixated sequentially might be influenced by fluctuations in visual attention. We used two psychophysical tasks with human subjects to investigate the extent to which visual attention influences simple perceptual choices, and to test the extent to which the attentional Drift Diffusion Model (aDDM) provides a good computational description of how attention affects the underlying decision processes. We find evidence for sizable attentional choice biases and that the aDDM provides a reasonable quantitative description of the relationship between fluctuations in visual attention, choices and reaction times. We also find that exogenous manipulations of attention induce choice biases consistent with the predictions of the model.

Dysfunctions in striatal microstructure can enhance perceptual decision making through deficits in predictive coding

An important brain function is to predict upcoming events on the basis of extracted regularities of previous inputs. These predictive coding processes can disturb performance in concurrent perceptual decision-making and are known to depend on fronto-striatal circuits. However, it is unknown whether, and if so, to what extent striatal microstructural properties modulate these processes. We addressed this question in a human disease model of striosomal dysfunction, i.e. X-linked dystonia-parkinsonism (XDP), using high-density EEG recordings and source localization. The results show faster and more accurate perceptual decision-making performance during distraction in XDP patients compared to healthy controls. The electrophysiological data show that sensory memory and predictive coding processes reflected by the mismatch negativity related to lateral prefrontal brain regions were weakened in XDP patients and thus induced less cognitive conflict than in controls as reflected by the N2 event-related potential (ERP). Consequently, attentional shifting (P3a ERP) and reorientation processes (RON ERP) were less pronounced in the XDP group. Taken together, these results suggests that striosomal dysfunction is related to predictive coding deficits leading to a better performance in concomitant perceptual decision-making, probably because predictive coding does not interfere with perceptual decision-making processes. These effects may reflect striatal imbalances between the striosomes and the matrix compartment.

Step by step: Harvesting the dynamics of delay discounting decisions

People show a tendency to devalue rewards when they are delayed in time. This so-called delay discounting often happens to an extent that seems irrational from an economical perspective. Research studying outcomes of delay discounting decisions has successfully derived descriptive models for such choice preferences. However, this outcome-based approach faces limitations in integrating the influence of contextual factors on the decision. Recently, this outcome-centred perspective on delay discounting has been complemented by a focus on the process dynamics leading to delay discounting decisions. Here, we embrace and add to this approach: We show how an attractor model can extend discounting descriptive discounting curves into the temporal dimension. From the model, we derive three predictions and study the predictions in a delay discounting experiment based on mouse tracking. We find differences in discounting depending on the order of option presentation and more direct movements to options presented first. Together with the analysis of specific temporal patterns of information integration, these results show that considering the continuous process dynamics of delay discounting decisions and harvesting them with continuous behavioural measures allow the integration of contextual factors into existing models of delay discounting, not as noise but as a signal on their own.