On the value of information and other rewards

Distinguishing between intrinsic and instrumental sources of the value of choice

Considerable evidence suggests that people value the freedom to choose. However, it is unclear whether this preference for choice stems purely from choice's intrinsic value, or whether people prefer to choose because it tends to provide instrumental information about desirable outcomes. To address this question, participants completed a novel choice task in which they could freely choose to exert choice or not, manipulating the level of instrumental contingency between participants' choices and eventual outcomes, which we operationalized using the information-theoretic concept of mutual information. Across two experiments (N = 100 each), we demonstrate a marked preference for choice, but importantly found that participants' preference for free choice is weakened when actions are decoupled from outcomes. Taken together, our results demonstrate that a significant factor in people's preference for choice is an assumption about the instrumental value of choice, suggesting against a purely intrinsic value of choice.

Art Value Creation and Destruction

I present a theory of creative and destructive value state referring to abstract art. Value is a probabilistic state held as a mixture of its expectation and information forces that coexist in a give-and-take relationship. Expectations are driven by the disclosure of novel information about the value state of various events of desire. Each bit of accumulated information contributes to the improvement of perception up to a threshold level, beyond which begin conscious states. The desire to disclose a value state triggers a triadic system of evaluation which uses concepts, observables and approaches. While the triadic valuation mechanisms can be used to assess various commodities, the scope of this work is limited to the case of artworks, in particular abstract paintings. I assume that art value is basically mediated by the interplay between these value state mechanisms of creation and destruction. Expectations in artwork develop attraction by challenging its contemplator to evaluate (predict) its meaning. Once the relevant information, corresponding to its creative expectations, is acquired (and conditioned), emotional states of indifference, disinterest and desensitization develop.

Revealing human sensitivity to a latent temporal structure of changes

Precisely timed behavior and accurate time perception plays a critical role in our everyday lives, as our wellbeing and even survival can depend on well-timed decisions. Although the temporal structure of the world around us is essential for human decision making, we know surprisingly little about how representation of temporal structure of our everyday environment impacts decision making. How does the representation of temporal structure affect our ability to generate well-timed decisions? Here we address this question by using a well-established dynamic probabilistic learning task. Using computational modeling, we found that human subjects' beliefs about temporal structure are reflected in their choices to either exploit their current knowledge or to explore novel options. The model-based analysis illustrates a large within-group and within-subject heterogeneity. To explain these results, we propose a normative model for how temporal structure is used in decision making, based on the semi-Markov formalism in the active inference framework. We discuss potential key applications of the presented approach to the fields of cognitive phenotyping and computational psychiatry.

Computational and Motivational Mechanisms of Human Social Decision Making Involving Close Others

Every day, human beings make decisions with social consequences. These social consequences matter most when they impact those closest to us. Recent research has shown that humans exhibit reliable preferences when deciding between conflicting outcomes involving close others - for example, prioritizing the interests of one's family member over one's friend. However, virtually nothing is known about the mechanisms that drive these preferences. We conducted a pre-registered study in a large (maximum N=375) sample to quantify the computational and motivational mechanisms of human social decision-making preferences involving close others. By pairing assessment techniques from behavioral economics and psychological science with computational modeling and random coefficient regression, we show that value-based cognitive computations (e.g., risk and loss aversion) drive social decision-making preferences involving financial outcomes, whereas socioemotional motivations (e.g., relationship quality) underlie preferences involving social outcomes. These results imply mechanistic heterogeneity, underscoring a need for greater attention to contextual specificity in social decision-making.

The ventral striatum dissociates information expectation, reward anticipation, and reward receipt

Do dopaminergic reward structures represent the expected utility of information similarly to a reward? Optimal experimental design models from Bayesian decision theory and statistics have proposed a theoretical framework for quantifying the expected value of information that might result from a query. In particular, this formulation quantifies the value of information before the answer to that query is known, in situations where payoffs are unknown and the goal is purely epistemic: That is, to increase knowledge about the state of the world. Whether and how such a theoretical quantity is represented in the brain is unknown. Here we use an event-related functional MRI (fMRI) task design to disentangle information expectation, information revelation and categorization outcome anticipation, and response-contingent reward processing in a visual probabilistic categorization task. We identify a neural signature corresponding to the expectation of information, involving the left lateral ventral striatum. Moreover, we show a temporal dissociation in the activation of different reward-related regions, including the nucleus accumbens, medial prefrontal cortex, and orbitofrontal cortex, during information expectation versus reward-related processing.

The description-experience gap in risky choice in nonhuman primates

Risk attitudes in humans depend on the format used to present the gamble: we are more risk-averse for common gambles in the gains domain whose properties are described to us verbally than for those whose properties we learned about solely through experience. This difference, which constitutes part of the description-experience gap, is important, because it highlights the role of knowledge acquisition in decision-making. The reasons for the gap remain obscure, but could depend upon uniquely human cognitive abilities, such as those associated with language. Thus, the gap may or may not extend to nonhuman animals. For this study, rhesus monkeys performed a novel task in which the properties of some gambles were explicitly cued (described), whereas others were learned through previous choices (experienced). Our monkeys displayed a description-experience gap. Overall, monkeys were more risk-seeking for experienced than for described gambles. This difference was observed for a range of gamble probabilities (from 20% to 80% likelihood of payoff), indicating that it is not limited to low probability events. These results suggest that the description-experience gap does not depend on uniquely human cognitive abilities, such as those associated with language, and support the idea that epistemic influences on risk attitudes are evolutionarily ancient.

A Neurophysiological examination of quality of learning in a feedback-based learning task

The efficiency with which one processes external feedback contributes to the speed and quality of one's learning. Previous findings that the feedback related negativity (FRN) event related potential (ERP) is modulated by learning outcomes suggested that this ERP reflects the extent to which feedback is used by the learner to improve performance. To further test this suggestion, we measured whether the FRN and the fronto-central positivity (FCP) that follows it are modulated by learning slopes, and as a function of individual differences in learning outcomes. Participants were tasked with learning names (non-words) of 42 novel objects in a two-choice feedback-based visual learning task. The items were divided into three sets of 14 items, each presented in five learning blocks and a sixth test block. Individual learning slopes based on performance on the task, as well as FRN and FCP slopes based on positive and negative feedback related activation in each block were created for 53 participants. Our data pointed to an interaction between slopes of the FRN elicited by negative feedback and learning slopes, such that a sharper decrease in the amplitude of the FRN to negative feedback was associated with sharper learning slopes. We further examined the predictive power of the FRN and FCP elicited in the training blocks on the learning outcomes as measured by performance on the test blocks. We found that small FRN to negative feedback, large FRN to positive feedback, and large FCP to negative feedback in the first training block predicted better learning outcomes. These results add to the growing evidence that the processes giving rise to the FRN and FCP are sensitive to individual differences in the extent to which feedback is used for learning.

Intrinsic Valuation of Information in Decision Making under Uncertainty

In a dynamic world, an accurate model of the environment is vital for survival, and agents ought regularly to seek out new information with which to update their world models. This aspect of behaviour is not captured well by classical theories of decision making, and the cognitive mechanisms of information seeking are poorly understood. In particular, it is not known whether information is valued only for its instrumental use, or whether humans also assign it a non-instrumental intrinsic value. To address this question, the present study assessed preference for non-instrumental information among 80 healthy participants in two experiments. Participants performed a novel information preference task in which they could choose to pay a monetary cost to receive advance information about the outcome of a monetary lottery. Importantly, acquiring information did not alter lottery outcome probabilities. We found that participants were willing to incur considerable monetary costs to acquire payoff-irrelevant information about the lottery outcome. This behaviour was well explained by a computational cognitive model in which information preference resulted from aversion to temporally prolonged uncertainty. These results strongly suggest that humans assign an intrinsic value to information in a manner inconsistent with normative accounts of decision making under uncertainty. This intrinsic value may be associated with adaptive behaviour in real-world environments by producing a bias towards exploratory and information-seeking behaviour.

Acquiring information about the external world is vital for planning and decision making. However, recent research has shown that some animals choose to acquire information at considerable cost even when the information is of no practical benefit, a counter-intuitive behavior associated with suboptimal outcomes. In this study, we demonstrate that humans also engage in this suboptimal behavior by forfeiting future monetary reward in exchange for early but unusable information about future outcomes. Our results suggest that participants attach a value to information beyond its purely instrumental value. This preference for information may help account for apparent anomalies in human choice behavior such as compulsive checking behaviors in obsessive-compulsive disorder, and excessive and wasteful use of uninformative laboratory testing in hospitals.

Modeling strategic use of human computer interfaces with novel hidden Markov models

Immersive software tools are virtual environments designed to give their users an augmented view of real-world data and ways of manipulating that data. As virtual environments, every action users make while interacting with these tools can be carefully logged, as can the state of the software and the information it presents to the user, giving these actions context. This data provides a high-resolution lens through which dynamic cognitive and behavioral processes can be viewed. In this report, we describe new methods for the analysis and interpretation of such data, utilizing a novel implementation of the Beta Process Hidden Markov Model (BP-HMM) for analysis of software activity logs. We further report the results of a preliminary study designed to establish the validity of our modeling approach. A group of 20 participants were asked to play a simple computer game, instrumented to log every interaction with the interface. Participants had no previous experience with the game's functionality or rules, so the activity logs collected during their naïve interactions capture patterns of exploratory behavior and skill acquisition as they attempted to learn the rules of the game. Pre- and post-task questionnaires probed for self-reported styles of problem solving, as well as task engagement, difficulty, and workload. We jointly modeled the activity log sequences collected from all participants using the BP-HMM approach, identifying a global library of activity patterns representative of the collective behavior of all the participants. Analyses show systematic relationships between both pre- and post-task questionnaires, self-reported approaches to analytic problem solving, and metrics extracted from the BP-HMM decomposition. Overall, we find that this novel approach to decomposing unstructured behavioral data within software environments provides a sensible means for understanding how users learn to integrate software functionality for strategic task pursuit.

Explicit knowledge enhances motor vigor and performance: motivation versus practice in sequence tasks

Motor skill learning involves a practice-induced improvement in the speed and/or accuracy of a discrete movement. It is often thought that paradigms involving repetitive practice of discrete movements performed in a fixed sequence result in a further enhancement of skill beyond practice of the individual movements in a random order. Sequence-specific performance improvements could, however, arise without practice as a result of knowledge of the sequence order; knowledge could operate by either enabling advanced motor planning of the known sequence elements or by increasing overall motivation. Here, we examined how knowledge and practice contribute to performance of a sequence of movements. We found that explicit knowledge provided through instruction produced practice-independent improvements in reaction time and execution quality. These performance improvements occurred even for random elements within a partially known sequence, indicative of a general motivational effect rather than a sequence-specific effect of advanced planning. This motivational effect suggests that knowledge influences performance in a manner analogous to reward. Additionally, practice led to similar improvements in execution quality for both known and random sequences. The lack of interaction between knowledge and practice suggests that any skill acquisition occurring during discrete sequence tasks arises solely from practice of the individual movement elements, independent of their order. We conclude that performance improvements in discrete sequence tasks arise from the combination of knowledge-based motivation and sequence-independent practice; investigating this interplay between cognition and movement may facilitate a greater understanding of the acquisition of skilled behavior.

Modelling preference and diet selection patterns by grazing ruminants: a development in a mechanistic model of a grazing dairy cow, MINDY

The work presented here represents additions to the mechanistic and dynamic model of a grazing dairy cow (MINDY). The additions include a module representing preference and selection, based on two theories, namely, post-ingestive feedback and discomfort. The model was evaluated by assessing its ability to simulate patterns of preference and selection in response to a variety of feeding management. The improvements detailed here enable a realistic simulation of patterns of food selection by grazing ruminants, based on a range of feeding situations from different studies with cattle and sheep. These simulations indicate that the concepts encoded in MINDY capture several of the underlying biological mechanisms that drive preferences and selective behaviour. Thus, simulations using MINDY allow prediction of daily and diurnal patterns of selection based on preference, derived from some post-ingestive feedbacks and total discomfort. Estimates of herbage intake and parallel measurements of ingestive behaviour, rumen function and metabolism in grazing ruminants pose experimental and technical difficulties, and matching these processes to animal preference and selective behaviour is a greater challenge. As a consequence, advances in knowledge of foraging behaviour and dietary choice are slow and costly. On completion of more thorough testing, MINDY can be used as a tool for exploratory mechanistic research, to design and organise experimental programs to address a range of factors that control intake and its ecology, helping advance knowledge faster and at a low cost.

A universal role of the ventral striatum in reward-based learning: evidence from human studies

Reinforcement learning enables organisms to adjust their behavior in order to maximize rewards. Electrophysiological recordings of dopaminergic midbrain neurons have shown that they code the difference between actual and predicted rewards, i.e., the reward prediction error, in many species. This error signal is conveyed to both the striatum and cortical areas and is thought to play a central role in learning to optimize behavior. However, in human daily life rewards are diverse and often only indirect feedback is available. Here we explore the range of rewards that are processed by the dopaminergic system in human participants, and examine whether it is also involved in learning in the absence of explicit rewards. While results from electrophysiological recordings in humans are sparse, evidence linking dopaminergic activity to the metabolic signal recorded from the midbrain and striatum with functional magnetic resonance imaging (fMRI) is available. Results from fMRI studies suggest that the human ventral striatum (VS) receives valuation information for a diverse set of rewarding stimuli. These range from simple primary reinforcers such as juice rewards over abstract social rewards to internally generated signals on perceived correctness, suggesting that the VS is involved in learning from trial-and-error irrespective of the specific nature of provided rewards. In addition, we summarize evidence that the VS can also be implicated when learning from observing others, and in tasks that go beyond simple stimulus-action-outcome learning, indicating that the reward system is also recruited in more complex learning tasks.

The Good, the Bad, or the Useful? The Examination of the Relationship between the Feedback-related Negativity (FRN) and Long-term Learning Outcomes

According to the reinforcement learning account of the error-related negativity (ERN), the ERN is a manifestation of a signal generated in ACC as a consequence of a phasic decrease in the activity of the mesencephalic dopamine system occurring when the monitoring system evaluates events as worse than expected. This signal is also hypothesized to be used to modify behavior to ascertain that future events will have better outcomes. It is therefore expected that this signal be correlated with learning outcomes. We report a study designed to examine the extent to which the ERN is related to learning outcomes within a paired-associates learning task. The feedback-related negativity (FRN) elicited by stimuli that indicated to the participants whether their response was correct or not was examined both according the degree to which the associates were learned in the session and according to whether participants recalled the associations on the next day. The results of the spatio-temporal PCA indicate that, whereas the process giving rise to the negative feedback elicited a FRN whose amplitude was not correlated with long-term learning outcomes, positive feedback was associated with a FRN-like activity, which was correlated with the learning outcomes. Another ERP component that follows the FRN temporally and shares its spatial distribution was found associated with long-term learning outcomes. Our findings shed light on the functional significance of the feedback-related ERP components and are discussed within the framework of the reinforcement learning ERN hypothesis.

The mixed instrumental controller: using value of information to combine habitual choice and mental simulation

Instrumental behavior depends on both goal-directed and habitual mechanisms of choice. Normative views cast these mechanisms in terms of model-free and model-based methods of reinforcement learning, respectively. An influential proposal hypothesizes that model-free and model-based mechanisms coexist and compete in the brain according to their relative uncertainty. In this paper we propose a novel view in which a single Mixed Instrumental Controller produces both goal-directed and habitual behavior by flexibly balancing and combining model-based and model-free computations. The Mixed Instrumental Controller performs a cost-benefits analysis to decide whether to chose an action immediately based on the available "cached" value of actions (linked to model-free mechanisms) or to improve value estimation by mentally simulating the expected outcome values (linked to model-based mechanisms). Since mental simulation entails cognitive effort and increases the reward delay, it is activated only when the associated "Value of Information" exceeds its costs. The model proposes a method to compute the Value of Information, based on the uncertainty of action values and on the distance of alternative cached action values. Overall, the model by default chooses on the basis of lighter model-free estimates, and integrates them with costly model-based predictions only when useful. Mental simulation uses a sampling method to produce reward expectancies, which are used to update the cached value of one or more actions; in turn, this updated value is used for the choice. The key predictions of the model are tested in different settings of a double T-maze scenario. Results are discussed in relation with neurobiological evidence on the hippocampus - ventral striatum circuit in rodents, which has been linked to goal-directed spatial navigation.

Putting reward in art: A tentative prediction error account of visual art

The predictive coding model is increasingly and fruitfully used to explain a wide range of findings in perception. Here we discuss the potential of this model in explaining the mechanisms underlying aesthetic experiences. Traditionally art appreciation has been associated with concepts such as harmony, perceptual fluency, and the so-called good Gestalt. We observe that more often than not great artworks blatantly violate these characteristics. Using the concept of prediction error from the predictive coding approach, we attempt to resolve this contradiction. We argue that artists often destroy predictions that they have first carefully built up in their viewers, and thus highlight the importance of negative affect in aesthetic experience. However, the viewer often succeeds in recovering the predictable pattern, sometimes on a different level. The ensuing rewarding effect is derived from this transition from a state of uncertainty to a state of increased predictability. We illustrate our account with several example paintings and with a discussion of art movements and individual differences in preference. On a more fundamental level, our theorizing leads us to consider the affective implications of prediction confirmation and violation. We compare our proposal to other influential theories on aesthetics and explore its advantages and limitations.