Optimal utility and probability functions for agents with finite computational precision

Decrease in decision noise from adolescence into adulthood mediates an increase in more sophisticated choice behaviors and performance gain

Learning and decision-making undergo substantial developmental changes, with adolescence being a particular vulnerable window of opportunity. In adolescents, developmental changes in specific choice behaviors have been observed (e.g., goal-directed behavior, motivational influences over choice). Elevated levels of decision noise, i.e., choosing suboptimal options, were reported consistently in adolescents. However, it remains unknown whether these observations, the development of specific and more sophisticated choice processes and higher decision noise, are independent or related. It is conceivable, but has not yet been investigated, that the development of specific choice processes might be impacted by age-dependent changes in decision noise. To answer this, we examined 93 participants (12 to 42 years) who completed 3 reinforcement learning (RL) tasks: a motivational Go/NoGo task assessing motivational influences over choices, a reversal learning task capturing adaptive decision-making in response to environmental changes, and a sequential choice task measuring goal-directed behavior. This allowed testing of (1) cross-task generalization of computational parameters focusing on decision noise; and (2) assessment of mediation effects of noise on specific choice behaviors. Firstly, we found only noise levels to be strongly correlated across RL tasks. Second, and critically, noise levels mediated age-dependent increases in more sophisticated choice behaviors and performance gain. Our findings provide novel insights into the computational processes underlying developmental changes in decision-making: namely a vital role of seemingly unspecific changes in noise in the specific development of more complex choice components. Studying the neurocomputational mechanisms of how varying levels of noise impact distinct aspects of learning and decision processes may also be key to better understand the developmental onset of psychiatric diseases.

Modelling dataset bias in machine-learned theories of economic decision-making

Normative and descriptive models have long vied to explain and predict human risky choices, such as those between goods or gambles. A recent study reported the discovery of a new, more accurate model of human decision-making by training neural networks on a new online large-scale dataset, choices13k. Here we systematically analyse the relationships between several models and datasets using machine-learning methods and find evidence for dataset bias. Because participants' choices in stochastically dominated gambles were consistently skewed towards equipreference in the choices13k dataset, we hypothesized that this reflected increased decision noise. Indeed, a probabilistic generative model adding structured decision noise to a neural network trained on data from a laboratory study transferred best, that is, outperformed all models apart from those trained on choices13k. We conclude that a careful combination of theory and data analysis is still required to understand the complex interactions of machine-learning models and data of human risky choices.

Rationality, preferences, and emotions with biological constraints: it all starts from our senses

Is the role of our sensory systems to represent the physical world as accurately as possible? If so, are our preferences and emotions, often deemed irrational, decoupled from these 'ground-truth' sensory experiences? We show why the answer to both questions is 'no'. Brain function is metabolically costly, and the brain loses some fraction of the information that it encodes and transmits. Therefore, if brains maximize objective functions that increase the fitness of their species, they should adapt to the objective-maximizing rules of the environment at the earliest stages of sensory processing. Consequently, observed 'irrationalities', preferences, and emotions stem from the necessity for our early sensory systems to adapt and process information while considering the metabolic costs and internal states of the organism.

Grid-like entorhinal representation of an abstract value space during prospective decision making

How valuable a choice option is often changes over time, making the prediction of value changes an important challenge for decision making. Prior studies identified a cognitive map in the hippocampal-entorhinal system that encodes relationships between states and enables prediction of future states, but does not inherently convey value during prospective decision making. In this fMRI study, participants predicted changing values of choice options in a sequence, forming a trajectory through an abstract two-dimensional value space. During this task, the entorhinal cortex exhibited a grid-like representation with an orientation aligned to the axis through the value space most informative for choices. A network of brain regions, including ventromedial prefrontal cortex, tracked the prospective value difference between options. These findings suggest that the entorhinal grid system supports the prediction of future values by representing a cognitive map, which might be used to generate lower-dimensional value signals to guide prospective decision making.

Sensory perception relies on fitness-maximizing codes

Sensory information encoded by humans and other organisms is generally presumed to be as accurate as their biological limitations allow. However, perhaps counterintuitively, accurate sensory representations may not necessarily maximize the organism's chances of survival. To test this hypothesis, we developed a unified normative framework for fitness-maximizing encoding by combining theoretical insights from neuroscience, computer science, and economics. Behavioural experiments in humans revealed that sensory encoding strategies are flexibly adapted to promote fitness maximization, a result confirmed by deep neural networks with information capacity constraints trained to solve the same task as humans. Moreover, human functional MRI data revealed that novel behavioural goals that rely on object perception induce efficient stimulus representations in early sensory structures. These results suggest that fitness-maximizing rules imposed by the environment are applied at early stages of sensory processing in humans and machines.

Adaptive tuning of human learning and choice variability to unexpected uncertainty

Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.

Strategically managing learning during perceptual decision making

Making optimal decisions in the face of noise requires balancing short-term speed and accuracy. But a theory of optimality should account for the fact that short-term speed can influence long-term accuracy through learning. Here, we demonstrate that long-term learning is an important dynamical dimension of the speed-accuracy trade-off. We study learning trajectories in rats and formally characterize these dynamics in a theory expressed as both a recurrent neural network and an analytical extension of the drift-diffusion model that learns over time. The model reveals that choosing suboptimal response times to learn faster sacrifices immediate reward, but can lead to greater total reward. We empirically verify predictions of the theory, including a relationship between stimulus exposure and learning speed, and a modulation of reaction time by future learning prospects. We find that rats' strategies approximately maximize total reward over the full learning epoch, suggesting cognitive control over the learning process.

Human value learning and representation reflect rational adaptation to task demands

Humans and other animals routinely make choices between goods of different values. Choices are often made within identifiable contexts, such that an efficient learner may represent values relative to their local context. However, if goods occur across multiple contexts, a relative value code can lead to irrational choices. In this case, an absolute context-independent value is preferable to a relative code. Here we test the hypothesis that value representation is not fixed but rationally adapted to context expectations. In two experiments, we manipulated participants' expectations about whether item values learned within local contexts would need to be subsequently compared across contexts. Despite identical learning experiences, the group whose expectations included choices across local contexts went on to learn more absolute-like representation than the group whose expectations covered only fixed local contexts. Human value representation is thus neither relative nor absolute but efficiently and rationally tuned to task demands.

Efficiently irrational: deciphering the riddle of human choice

For the past half-century, cognitive and social scientists have struggled with the irrationalities of human choice behavior; people consistently make choices that are logically inconsistent. Is human choice behavior evolutionarily adaptive or is it an inefficient patchwork of competing mechanisms? In this review, I present an interdisciplinary synthesis arguing for a novel interpretation: choice is efficiently irrational. Connecting findings across disciplines suggests that observed choice behavior reflects a precise optimization of the trade-off between the costs of increasing the precision of the choice mechanism and the declining benefits that come as precision increases. Under these constraints, a rationally imprecise strategy emerges that works toward optimal efficiency rather than toward optimal rationality. This approach rationalizes many of the puzzling inconsistencies of human choice behavior, explaining why these inconsistencies arise as an optimizing solution in biological choosers.

Control over sampling boosts numerical evidence processing in human decisions from experience

When acquiring information about choice alternatives, decision makers may have varying levels of control over which and how much information they sample before making a choice. How does control over information acquisition affect the quality of sample-based decisions? Here, combining variants of a numerical sampling task with neural recordings, we show that control over when to stop sampling can enhance (i) behavioral choice accuracy, (ii) the build-up of parietal decision signals, and (iii) the encoding of numerical sample information in multivariate electroencephalogram patterns. None of these effects were observed when participants could only control which alternatives to sample, but not when to stop sampling. Furthermore, levels of control had no effect on early sensory signals or on the extent to which sample information leaked from memory. The results indicate that freedom to stop sampling can amplify decisional evidence processing from the outset of information acquisition and lead to more accurate choices.

Rational inattention in mice

Behavior exhibited by humans and other organisms is generally inconsistent and biased and, thus, is often labeled irrational. However, the origins of this seemingly suboptimal behavior remain elusive. We developed a behavioral task and normative framework to reveal how organisms should allocate their limited processing resources such that sensory precision and its related metabolic investment are balanced to guarantee maximal utility. We found that mice act as rational inattentive agents by adaptively allocating their sensory resources in a way that maximizes reward consumption in previously unexperienced stimulus-reward association environments. Unexpectedly, perception of commonly occurring stimuli was relatively imprecise; however, this apparent statistical fallacy implies "awareness" and efficient adaptation to their neurocognitive limitations. Arousal systems carry reward distribution information of sensory signals, and distributional reinforcement learning mechanisms regulate sensory precision via top-down normalization. These findings reveal how organisms efficiently perceive and adapt to previously unexperienced environmental contexts within the constraints imposed by neurobiology.

Asymmetric reinforcement learning facilitates human inference of transitive relations

Humans and other animals are capable of inferring never-experienced relations (for example, A > C) from other relational observations (for example, A > B and B > C). The processes behind such transitive inference are subject to intense research. Here we demonstrate a new aspect of relational learning, building on previous evidence that transitive inference can be accomplished through simple reinforcement learning mechanisms. We show in simulations that inference of novel relations benefits from an asymmetric learning policy, where observers update only their belief about the winner (or loser) in a pair. Across four experiments (n = 145), we find substantial empirical support for such asymmetries in inferential learning. The learning policy favoured by our simulations and experiments gives rise to a compression of values that is routinely observed in psychophysics and behavioural economics. In other words, a seemingly biased learning strategy that yields well-known cognitive distortions can be beneficial for transitive inferential judgements.

Ciranka, Linde-Domingo et al. show that inference of transitive orderings from pairwise relations benefits from a seemingly biased learning strategy, where observers update their belief about one of the pair members but not the other.

Normative Principles for Decision-Making in Natural Environments

The decisions we make are shaped by a lifetime of learning. Past experience guides the way that we encode information in neural systems for perception and valuation, and determines the information we retrieve when making decisions. Distinct literatures have discussed how lifelong learning and local context shape decisions made about sensory signals, propositional information, or economic prospects. Here, we build bridges between these literatures, arguing for common principles of adaptive rationality in perception, cognition, and economic choice. We discuss how a single common framework, based on normative principles of efficient coding and Bayesian inference, can help us understand a myriad of human decision biases, including sensory illusions, adaptive aftereffects, choice history biases, central tendency effects, anchoring effects, contrast effects, framing effects, congruency effects, reference-dependent valuation, nonlinear utility functions, and discretization heuristics. We describe a simple computational framework for explaining these phenomena. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.