Alterations in choice behavior by manipulations of world model

A Markovian dynamics for Caenorhabditis elegans behavior across scales

How do we capture the breadth of behavior in animal movement, from rapid body twitches to aging? Using high-resolution videos of the nematode worm Caenorhabditis elegans, we show that a single dynamics connects posture-scale fluctuations with trajectory diffusion and longer-lived behavioral states. We take short posture sequences as an instantaneous behavioral measure, fixing the sequence length for maximal prediction. Within the space of posture sequences, we construct a fine-scale, maximum entropy partition so that transitions among microstates define a high-fidelity Markov model, which we also use as a means of principled coarse-graining. We translate these dynamics into movement using resistive force theory, capturing the statistical properties of foraging trajectories. Predictive across scales, we leverage the longest-lived eigenvectors of the inferred Markov chain to perform a top-down subdivision of the worm's foraging behavior, revealing both "runs-and-pirouettes" as well as previously uncharacterized finer-scale behaviors. We use our model to investigate the relevance of these fine-scale behaviors for foraging success, recovering a trade-off between local and global search strategies.

Plinko: Eliciting beliefs to build better models of statistical learning and mental model updating

Prior beliefs are central to Bayesian accounts of cognition, but many of these accounts do not directly measure priors. More specifically, initial states of belief heavily influence how new information is assumed to be utilized when updating a particular model. Despite this, prior and posterior beliefs are either inferred from sequential participant actions or elicited through impoverished means. We had participants to play a version of the game 'Plinko', to first elicit individual participant priors in a theoretically agnostic manner. Subsequent learning and updating of participant beliefs was then directly measured. We show that participants hold various priors that cluster around prototypical probability distributions that in turn influence learning. In follow-up studies, we show that participant priors are stable over time and that the ability to update beliefs is influenced by a simple environmental manipulation (i.e., a short break). These data reveal the importance of directly measuring participant beliefs rather than assuming or inferring them as has been widely done in the literature to date. The Plinko game provides a flexible and fecund means for examining statistical learning and mental model updating.

Decision heuristics in contexts integrating action selection and execution

Heuristics can inform human decision making in complex environments through a reduction of computational requirements (accuracy-resource trade-off) and a robustness to overparameterisation (less-is-more). However, tasks capturing the efficiency of heuristics typically ignore action proficiency in determining rewards. The requisite movement parameterisation in sensorimotor control questions whether heuristics preserve efficiency when actions are nontrivial. We developed a novel action selection-execution task requiring joint optimisation of action selection and spatio-temporal skillful execution. State-appropriate choices could be determined by a simple spatial heuristic, or by more complex planning. Computational models of action selection parsimoniously distinguished human participants who adopted the heuristic from those using a more complex planning strategy. Broader comparative analyses then revealed that participants using the heuristic showed combined decisional (selection) and skill (execution) advantages, consistent with a less-is-more framework. In addition, the skill advantage of the heuristic group was predominantly in the core spatial features that also shaped their decision policy, evidence that the dimensions of information guiding action selection might be yoked to salient features in skill learning.

Active inference and the two-step task

Sequential decision problems distill important challenges frequently faced by humans. Through repeated interactions with an uncertain world, unknown statistics need to be learned while balancing exploration and exploitation. Reinforcement learning is a prominent method for modeling such behaviour, with a prevalent application being the two-step task. However, recent studies indicate that the standard reinforcement learning model sometimes describes features of human task behaviour inaccurately and incompletely. We investigated whether active inference, a framework proposing a trade-off to the exploration-exploitation dilemma, could better describe human behaviour. Therefore, we re-analysed four publicly available datasets of the two-step task, performed Bayesian model selection, and compared behavioural model predictions. Two datasets, which revealed more model-based inference and behaviour indicative of directed exploration, were better described by active inference, while the models scored similarly for the remaining datasets. Learning using probability distributions appears to contribute to the improved model fits. Further, approximately half of all participants showed sensitivity to information gain as formulated under active inference, although behavioural exploration effects were not fully captured. These results contribute to the empirical validation of active inference as a model of human behaviour and the study of alternative models for the influential two-step task.

Motor Plans under Uncertainty Reflect a Trade-Off between Maximizing Reward and Success

When faced with multiple potential movement options, individuals either reach directly to one of the options, or initiate a reach intermediate between the options. It remains unclear why people generate these two types of behaviors. Using the go-before-you-know task (commonly used to study behavior under choice uncertainty) in humans, we examined two key questions. First, do these two types of responses actually reflect distinct movement strategies? If so, the relative desirability (i.e., weighing the success likelihood vs the attainable reward) of the two target options would not need to be computed identically for direct and intermediate reaches. We showed that indeed, when reward and success likelihood differed between the two options, reach direction was preferentially biased toward different directions for direct versus intermediate reaches. Importantly, this suggests that the computation of subjective values depends on the choice of movement strategy. Second, what drives individual differences in how people respond under uncertainty? We found that risk/reward-seeking individuals tended to generate more intermediate reaches and were more responsive to changes in reward, suggesting these movements may reflect a strategy to maximize reward versus success. Together, these findings suggest that when faced with choice uncertainty, individuals adopt movement strategies consistent with their risk/reward attitude, preferentially biasing behavior toward exogenous rewards or endogenous success and consequently modulating the relative desirability of the available options.

Distinct Neural Signatures of Outcome Monitoring After Selection and Execution Errors

Losing a point in tennis could result from poor shot selection or faulty stroke execution. To explore how the brain responds to these different types of errors, we examined feedback-locked EEG activity while participants completed a modified version of a standard three-armed bandit probabilistic reward task. Our task framed unrewarded outcomes as the result of either errors of selection or errors of execution. We examined whether amplitude of a medial frontal negativity (the feedback-related negativity [FRN]) was sensitive to the different forms of error attribution. Consistent with previous reports, selection errors elicited a large FRN relative to rewards, and amplitude of this signal correlated with behavioral adjustment after these errors. A different pattern was observed in response to execution errors. These outcomes produced a larger FRN, a frontocentral attenuation in activity preceding this component, and a subsequent enhanced error positivity in parietal sites. Notably, the only correlations with behavioral adjustment were with the early frontocentral attenuation and amplitude of the parietal signal; FRN differences between execution errors and rewarded trials did not correlate with subsequent changes in behavior. Our findings highlight distinct neural correlates of selection and execution error processing, providing insight into how the brain responds to the different classes of error that determine future action.

Neural surprise in somatosensory Bayesian learning

Tracking statistical regularities of the environment is important for shaping human behavior and perception. Evidence suggests that the brain learns environmental dependencies using Bayesian principles. However, much remains unknown about the employed algorithms, for somesthesis in particular. Here, we describe the cortical dynamics of the somatosensory learning system to investigate both the form of the generative model as well as its neural surprise signatures. Specifically, we recorded EEG data from 40 participants subjected to a somatosensory roving-stimulus paradigm and performed single-trial modeling across peri-stimulus time in both sensor and source space. Our Bayesian model selection procedure indicates that evoked potentials are best described by a non-hierarchical learning model that tracks transitions between observations using leaky integration. From around 70ms post-stimulus onset, secondary somatosensory cortices are found to represent confidence-corrected surprise as a measure of model inadequacy. Indications of Bayesian surprise encoding, reflecting model updating, are found in primary somatosensory cortex from around 140ms. This dissociation is compatible with the idea that early surprise signals may control subsequent model update rates. In sum, our findings support the hypothesis that early somatosensory processing reflects Bayesian perceptual learning and contribute to an understanding of its underlying mechanisms.

Our environment features statistical regularities, such as a drop of rain predicting imminent rainfall. Despite the importance for behavior and survival, much remains unknown about how these dependencies are learned, particularly for somatosensation. As surprise signalling about novel observations indicates a mismatch between one’s beliefs and the world, it has been hypothesized that surprise computation plays an important role in perceptual learning. By analyzing EEG data from human participants receiving sequences of tactile stimulation, we compare different formulations of surprise and investigate the employed underlying learning model. Our results indicate that the brain estimates transitions between observations. Furthermore, we identified different signatures of surprise computation and thereby provide a dissociation of the neural correlates of belief inadequacy and belief updating. Specifically, early surprise responses from around 70ms were found to signal the need for changes to the model, with encoding of its subsequent updating occurring from around 140ms. These results provide insights into how somatosensory surprise signals may contribute to the learning of environmental statistics.

Maximizing as satisficing: On pattern matching and probability maximizing in groups and individuals

Distinguishing meaningful structure from unpredictable randomness is a key challenge in many domains of life. We examined whether collaborating three-person groups (n = 81) outperform individuals (n = 81) in facing this challenge with a two-part repeated choice task, where outcomes were either serially independent (probabilistic part) or fixed in a particular sequence (pattern part). Groups performed as well as the best individuals in the probabilistic part but groups' accuracy did not credibly exceed that of the average individual in the pattern part. Qualitative coding of group discussion data revealed that failures to identify existing patterns were related to groups accepting probability maximizing as a "good enough" strategy rather than expending effort to search for patterns. These results suggest that probability maximizing can arise via two routes: recognizing that probabilistic processes cannot be outdone (maximizing as optimizing) or settling for an imperfect but easily implementable strategy (maximizing as satisficing).

Human confidence judgments reflect reliability-based hierarchical integration of contextual information

Our immediate observations must be supplemented with contextual information to resolve ambiguities. However, the context is often ambiguous too, and thus it should be inferred itself to guide behavior. Here, we introduce a novel hierarchical task (airplane task) in which participants should infer a higher-level, contextual variable to inform probabilistic inference about a hidden dependent variable at a lower level. By controlling the reliability of past sensory evidence through varying the sample size of the observations, we find that humans estimate the reliability of the context and combine it with current sensory uncertainty to inform their confidence reports. Behavior closely follows inference by probabilistic message passing between latent variables across hierarchical state representations. Commonly reported inferential fallacies, such as sample size insensitivity, are not present, and neither did participants appear to rely on simple heuristics. Our results reveal uncertainty-sensitive integration of information at different hierarchical levels and temporal scales.

Because our immediate observations are often ambiguous, we must use the context (prior beliefs) to guide inference, but the context may also be uncertain. Here, the authors show that humans can accurately estimate the reliability of the context and combine it with sensory uncertainty to form their decisions and estimate confidence.

Probability Matching on a Simple Simulated Foraging Task: The Effects of Reward Persistence and Accumulation on Choice Behavior

Over a series of decisions between two or more probabilistically rewarded options, humans have a tendency to diversify their choices, even when this will lead to diminished overall reward. In the extreme case of probability matching, this tendency is expressed through allocation of choices in proportion to their likelihood of reward. Research suggests that this behaviour is an instinctive response, driven by heuristics, and that it may be overruled through the application of sufficient deliberation and self-control. However, if this is the case, then how and why did this response become established? The present study explores the hypothesis that diversification of choices, and potentially probability matching, represents an overextension of a historically normative foraging strategy. This is done through examining choice behaviour on a simple simulated foraging task, designed to model the natural process of accumulation of unharvested resources over time. Behaviour was then directly compared with that observed on a standard fixed probability task (cf. Ellerby & Tunney, 2017). Results indicated a convergence of choice patterns on the simulated foraging task, between participants who acted intuitively and those who took a more strategic approach. These findings are also compared with those of another similarly motivated study (Schulze, van Ravenzwaaij, & Newell, 2017).

Learning What to Learn Across the Life Span: From Objects to Real-World Skills

One of the most difficult and important problems that all learners face across the life span is learning what to learn. Understanding what to learn is difficult when both relevant and irrelevant information compete for attention. In these situations, the learner can rely on cues in the environment, as well as prior knowledge. However, these sources of information sometimes conflict, and the learner has to prioritize some sources over others. Determining what to learn is important because learning relevant information helps the learner achieve goals, whereas learning irrelevant information can waste time and energy. A new theoretical approach posits that adaptation is relevant for all age groups because the environment is dynamic, suggesting that learning what to learn is a problem relevant across the life span instead of only during infancy and childhood. In this article, I review new research demonstrating the importance and ways of learning what to learn across the life span, from objects to real-world skills, before highlighting some unresolved issues for future research.

Neural Signatures of Prediction Errors in a Decision-Making Task Are Modulated by Action Execution Failures

Decisions must be implemented through actions, and actions are prone to error. As such, when an expected outcome is not obtained, an individual should be sensitive to not only whether the choice itself was suboptimal but also whether the action required to indicate that choice was executed successfully. The intelligent assignment of credit to action execution versus action selection has clear ecological utility for the learner. To explore this, we used a modified version of a classic reinforcement learning task in which feedback indicated whether negative prediction errors were, or were not, associated with execution errors. Using fMRI, we asked if prediction error computations in the human striatum, a key substrate in reinforcement learning and decision making, are modulated when a failure in action execution results in the negative outcome. Participants were more tolerant of non-rewarded outcomes when these resulted from execution errors versus when execution was successful, but reward was withheld. Consistent with this behavior, a model-driven analysis of neural activity revealed an attenuation of the signal associated with negative reward prediction errors in the striatum following execution failures. These results converge with other lines of evidence suggesting that prediction errors in the mesostriatal dopamine system integrate high-level information during the evaluation of instantaneous reward outcomes.

No Evidence That Abstract Structure Learning Disrupts Novel-Event Learning in 8- to 11-Month-Olds

Although infants acquire specific information (e.g., motion of a specific toy) and abstract information (e.g., likelihood of events repeating), it is unclear whether extraction of abstract information interferes with specific learning. In the present study, 8- to 11-month-old infants were shown four audio-visual movies, either with a mixed or uniform presentation structure. Learning of abstract information was operationally defined as the looking time to changes in presentation structure of the movies (mixed vs. uniform blocks), and learning of specific information was defined as the looking time to changes in content in the four movies (object properties and identities). We found evidence of both specific and abstract learning, but did not find evidence that extraction of the presentation structure (i.e., abstract learning) impacts specific learning of the events. We discuss the implications of the costs and benefits of the interaction between abstract and specific learning for infants.

Understanding psychiatric disorder by capturing ecologically relevant features of learning and decision-making

Recent research in cognitive neuroscience has begun to uncover the processes underlying increasingly complex voluntary behaviours, including learning and decision-making. Partly this success has been possible by progressing from simple experimental tasks to paradigms that incorporate more ecological features. More specifically, the premise is that to understand cognitions and brain functions relevant for real life, we need to introduce some of the ecological challenges that we have evolved to solve. This often entails an increase in task complexity, which can be managed by using computational models to help parse complex behaviours into specific component mechanisms. Here we propose that using computational models with tasks that capture ecologically relevant learning and decision-making processes may provide a critical advantage for capturing the mechanisms underlying symptoms of disorders in psychiatry. As a result, it may help develop mechanistic approaches towards diagnosis and treatment. We begin this review by mapping out the basic concepts and models of learning and decision-making. We then move on to consider specific challenges that emerge in realistic environments and describe how they can be captured by tasks. These include changes of context, uncertainty, reflexive/emotional biases, cost-benefit decision-making, and balancing exploration and exploitation. Where appropriate we highlight future or current links to psychiatry. We particularly draw examples from research on clinical depression, a disorder that greatly compromises motivated behaviours in real-life, but where simpler paradigms have yielded mixed results. Finally, we highlight several paradigms that could be used to help provide new insights into the mechanisms of psychiatric disorders.

The neuroanatomical basis of the Gambler's fallacy: A univariate and multivariate morphometric study

Human decision-making can be irrational, as in the case of the gambler's fallacy (GF). Converging evidence from behavioral and functional neuroimaging studies has suggested that a hyperactive cognitive system and a hypo-active affective system contribute to the false world model that generates the GF. However, the neuroanatomical basis underlying the GF remains unclear. The current study addressed this issue by collecting high-resolution magnetic resonance anatomical images from a large sample of 350 healthy Chinese adults. Univariate voxel-based morphometry (VBM) analysis suggested that the gray matter volume (GMV) in the anterior cingulate cortex (ACC) and medial temporal lobe (MTL) (two regions of the cognitive system) showed negative correlations with the degree of GF, while the GMV in the striatum and orbitofrontal cortex (OFC; two regions of the affective system) showed positive correlations. Further multivariate VBM analysis showed that the GMV in these regions could potentially predict the degree of GF. Moreover, a mediation analysis suggested that the GMV in MTL, ACC, and OFC mediated the relationships between the cognitive abilities or affective decision-making performance and the GF. Results of our study help us to understand the potential neural bases of the cognitive system's constructive role and the affective system's destructive role in decision making.

Credit Assignment in a Motor Decision Making Task Is Influenced by Agency and Not Sensory Prediction Errors

Failures to obtain reward can occur from errors in action selection or action execution. Recently, we observed marked differences in choice behavior when the failure to obtain a reward was attributed to errors in action execution compared with errors in action selection (McDougle et al., 2016). Specifically, participants appeared to solve this credit assignment problem by discounting outcomes in which the absence of reward was attributed to errors in action execution. Building on recent evidence indicating relatively direct communication between the cerebellum and basal ganglia, we hypothesized that cerebellar-dependent sensory prediction errors (SPEs), a signal indicating execution failure, could attenuate value updating within a basal ganglia-dependent reinforcement learning system. Here we compared the SPE hypothesis to an alternative, "top-down" hypothesis in which changes in choice behavior reflect participants' sense of agency. In two experiments with male and female human participants, we manipulated the strength of SPEs, along with the participants' sense of agency in the second experiment. The results showed that, whereas the strength of SPE had no effect on choice behavior, participants were much more likely to discount the absence of rewards under conditions in which they believed the reward outcome depended on their ability to produce accurate movements. These results provide strong evidence that SPEs do not directly influence reinforcement learning. Instead, a participant's sense of agency appears to play a significant role in modulating choice behavior when unexpected outcomes can arise from errors in action execution.SIGNIFICANCE STATEMENT When learning from the outcome of actions, the brain faces a credit assignment problem: Failures of reward can be attributed to poor choice selection or poor action execution. Here, we test a specific hypothesis that execution errors are implicitly signaled by cerebellar-based sensory prediction errors. We evaluate this hypothesis and compare it with a more "top-down" hypothesis in which the modulation of choice behavior from execution errors reflects participants' sense of agency. We find that sensory prediction errors have no significant effect on reinforcement learning. Instead, instructions influencing participants' belief of causal outcomes appear to be the main factor influencing their choice behavior.

Continuous track paths reveal additive evidence integration in multistep decision making

Multistep decision making pervades daily life, but its underlying mechanisms remain obscure. We distinguish four prominent models of multistep decision making, namely serial stage, hierarchical evidence integration, hierarchical leaky competing accumulation (HLCA), and probabilistic evidence integration (PEI). To empirically disentangle these models, we design a two-step reward-based decision paradigm and implement it in a reaching task experiment. In a first step, participants choose between two potential upcoming choices, each associated with two rewards. In a second step, participants choose between the two rewards selected in the first step. Strikingly, as predicted by the HLCA and PEI models, the first-step decision dynamics were initially biased toward the choice representing the highest sum/mean before being redirected toward the choice representing the maximal reward (i.e., initial dip). Only HLCA and PEI predicted this initial dip, suggesting that first-step decision dynamics depend on additive integration of competing second-step choices. Our data suggest that potential future outcomes are progressively unraveled during multistep decision making.

Prior Knowledge of Object Associations Shapes Attentional Templates and Information Acquisition

Studies on attentional selection typically use unpredictable and meaningless stimuli, such as simple shapes and oriented lines. The assumption is that using these stimuli minimizes effects due to learning or prior knowledge, such that the task performance indexes a "pure" measure of the underlying cognitive ability. However, prior knowledge of the test stimuli and related stimuli acquired before or during the task impacts performance in meaningful ways. This mini review focuses on prior knowledge of object associations, because it is an important, yet often ignored, aspect of attentional selection. We first briefly review recent studies demonstrating that how objects are selected during visual search depends on the participant's prior experience with other objects associated with the target. These effects appear with both task-relevant and task-irrelevant knowledge. We then review how existing object associations may influence subsequent learning of new information, which is both a driver and a consequence of selection processes. These insights highlight the importance of one aspect of prior knowledge for attentional selection and information acquisition. We briefly discuss how this work with young adults may inform other age groups throughout the lifespan, as learners gradually increase their prior knowledge. Importantly, these insights have implications for developing more accurate measurements of cognitive abilities.

Probability Learning: Changes in Behavior Across Time and Development

Individuals track probabilities, such as associations between events in their environments, but less is known about the degree to which experience-within a learning session and over development-influences people's use of incoming probabilistic information to guide behavior in real time. In two experiments, children (4-11 years) and adults searched for rewards hidden in locations with predetermined probabilities. In Experiment 1, children (n = 42) and adults (n = 32) changed strategies to maximize reward receipt over time. However, adults demonstrated greater strategy change efficiency. Making the predetermined probabilities more difficult to learn (Experiment 2) delayed effective strategy change for children (n = 39) and adults (n = 33). Taken together, these data characterize how children and adults alike react flexibly and change behavior according to incoming information.

Choice mechanisms for past, temporally extended outcomes

Accurate retrospection is critical in many decision scenarios ranging from investment banking to hedonic psychology. A notoriously difficult case is to integrate previously perceived values over the duration of an experience. Failure in retrospective evaluation leads to suboptimal outcome when previous experiences are under consideration for revisit. A biologically plausible mechanism underlying evaluation of temporally extended outcomes is leaky integration of evidence. The leaky integrator favours positive temporal contrasts, in turn leading to undue emphasis on recency. To investigate choice mechanisms underlying suboptimal outcome based on retrospective evaluation, we used computational and behavioural techniques to model choice between perceived extended outcomes with different temporal profiles. Second-price auctions served to establish the perceived values of virtual coins offered sequentially to humans in a rapid monetary gambling task. Results show that lesser-valued options involving successive growth were systematically preferred to better options with declining temporal profiles. The disadvantageous inclination towards persistent growth was mitigated in some individuals in whom a longer time constant of the leaky integrator resulted in fewer violations of dominance. These results demonstrate how focusing on immediate gains is less beneficial than considering longer perspectives.

Do People Take Stimulus Correlations into Account in Visual Search?

In laboratory visual search experiments, distractors are often statistically independent of each other. However, stimuli in more naturalistic settings are often correlated and rarely independent. Here, we examine whether human observers take stimulus correlations into account in orientation target detection. We find that they do, although probably not optimally. In particular, it seems that low distractor correlations are overestimated. Our results might contribute to bridging the gap between artificial and natural visual search tasks.

Flexible gating of contextual influences in natural vision

Identical sensory inputs can be perceived as strikingly different when embedded in distinct contexts. Neural responses to simple stimuli are also modulated by context, but the contribution of this modulation to the processing of natural sensory input is unclear. We measured surround suppression, a quintessential contextual influence, in macaque primary visual cortex with natural images. We found suppression strength varied substantially for different images. This variability was not well explained by existing descriptions of surround suppression, but it was predicted by Bayesian inference about statistical dependencies in images. In this framework, surround suppression was flexible: it was recruited when the image was inferred to contain redundancies, and substantially reduced in strength otherwise. Our results thus reveal a surprising gating of a basic, widespread cortical computation, by inference about the statistics of natural input.

Updating representations of temporal intervals

Effectively engaging with the world depends on accurate representations of the regularities that make up that world-what we call mental models. The success of any mental model depends on the ability to adapt to changes-to 'update' the model. In prior work, we have shown that damage to the right hemisphere of the brain impairs the ability to update mental models across a range of tasks. Given the disparate nature of the tasks we have employed in this prior work (i.e. statistical learning, language acquisition, position priming, perceptual ambiguity, strategic game play), we propose that a cognitive module important for updating mental representations should be generic, in the sense that it is invoked across multiple cognitive and perceptual domains. To date, the majority of our tasks have been visual in nature. Given the ubiquity and import of temporal information in sensory experience, we examined the ability to build and update mental models of time. We had healthy individuals complete a temporal prediction task in which intervals were initially drawn from one temporal range before an unannounced switch to a different range of intervals. Separate groups had the second range of intervals switch to one that contained either longer or shorter intervals than the first range. Both groups showed significant positive correlations between perceptual and prediction accuracy. While each group updated mental models of temporal intervals, those exposed to shorter intervals did so more efficiently. Our results support the notion of generic capacity to update regularities in the environment-in this instance based on temporal information. The task developed here is well suited to investigations in neurological patients and in neuroimaging settings.

Of matchers and maximizers: How competition shapes choice under risk and uncertainty

In a world of limited resources, scarcity and rivalry are central challenges for decision makers-animals foraging for food, corporations seeking maximal profits, and athletes training to win, all strive against others competing for the same goals. In this article, we establish the role of competitive pressures for the facilitation of optimal decision making in simple sequential binary choice tasks. In two experiments, competition was introduced with a computerized opponent whose choice behavior reinforced one of two strategies: If the opponent probabilistically imitated participant choices, probability matching was optimal; if the opponent was indifferent, probability maximizing was optimal. We observed accurate asymptotic strategy use in both conditions irrespective of the provision of outcome probabilities, suggesting that participants were sensitive to the differences in opponent behavior. An analysis of reinforcement learning models established that computational conceptualizations of opponent behavior are critical to account for the observed divergence in strategy adoption. Our results provide a novel appraisal of probability matching and show how this individually 'irrational' choice phenomenon can be socially adaptive under competition.

Decision from Models: Generalizing Probability Information to Novel Tasks

We investigate a new type of decision under risk where-to succeed-participants must generalize their experience in one set of tasks to a novel set of tasks. We asked participants to trade distance for reward in a virtual minefield where each successive step incurred the same fixed probability of failure (referred to as hazard). With constant hazard, the probability of success (the survival function) decreases exponentially with path length. On each trial, participants chose between a shorter path with smaller reward and a longer (more dangerous) path with larger reward. They received feedback in 160 training trials: encountering a mine along their chosen path resulted in zero reward and successful completion of the path led to the reward associated with the path chosen. They then completed 600 no-feedback test trials with novel combinations of path length and rewards. To maximize expected gain, participants had to learn the correct exponential model in training and generalize it to the test conditions. We compared how participants discounted reward with increasing path length to the predictions of nine choice models including the correct exponential model. The choices of a majority of the participants were best accounted for by a model of the correct exponential form although with marked overestimation of the hazard rate. The decision-from-models paradigm differs from experience-based decision paradigms such as decision-from-sampling in the importance assigned to generalizing experience-based information to novel tasks. The task itself is representative of everyday tasks involving repeated decisions in stochastically invariant environments.

Examining the influence of working memory on updating mental models

The ability to accurately build and update mental representations of our environment depends on our ability to integrate information over a variety of time scales and detect changes in the regularity of events. As such, the cognitive mechanisms that support model building and updating are likely to interact with those involved in working memory (WM). To examine this, we performed three experiments that manipulated WM demands concurrently with the need to attend to regularities in other stimulus properties (i.e., location and shape). That is, participants completed a prediction task while simultaneously performing an n-back WM task with either no load or a moderate load. The distribution of target locations (Experiment 1) or shapes (Experiments 2 and 3) included some level of probabilistic regularity, which, unbeknown to participants, changed abruptly within each block. Moderate WM load hampered the ability to benefit from target regularities and to adapt to changes in those regularities (i.e., the prediction task). This was most pronounced when both prediction and WM requirements shared the same target feature. Our results show that representational updating depends on free WM resources in a domain-specific fashion.

Probabilistic cognition in two indigenous Mayan groups

Is there a sense of chance shared by all individuals, regardless of their schooling or culture? To test whether the ability to make correct probabilistic evaluations depends on educational and cultural guidance, we investigated probabilistic cognition in preliterate and prenumerate Kaqchikel and K'iche', two indigenous Mayan groups, living in remote areas of Guatemala. Although the tested individuals had no formal education, they performed correctly in tasks in which they had to consider prior and posterior information, proportions and combinations of possibilities. Their performance was indistinguishable from that of Mayan school children and Western controls. Our results provide evidence for the universal nature of probabilistic cognition.

Learning bundles of stimuli renders stimulus order as a cue, not a confound

The order in which stimuli are presented in an experiment has long been recognized to influence behavior. Previous accounts have often attributed the effect of stimulus order to the mechanisms with which people process information. We propose that stimulus order influences cognition because it is an important cue for learning the underlying structure of a task environment. In particular, stimulus order can be used to infer a "stimulus bundle"--a sequence of consecutive stimuli that share the same underlying latent cluster. We describe a clustering model that successfully explains the perception of streak shooting in basketball games, along with two other cognitive phenomena, as the outcome of finding the statistically optimal bundle representation. We argue that the perspective of viewing stimulus order as a cue may hold the key to explaining behaviors that seemingly deviate from normative theories of cognition and that in task domains where the assumption of stimulus bundles is intuitively appropriate, it can improve the explanatory power of existing models.

Characterizing the impact of category uncertainty on human auditory categorization behavior

Categorization is an important cognitive process. However, the correct categorization of a stimulus is often challenging because categories can have overlapping boundaries. Whereas perceptual categorization has been extensively studied in vision, the analogous phenomenon in audition has yet to be systematically explored. Here, we test whether and how human subjects learn to use category distributions and prior probabilities, as well as whether subjects employ an optimal decision strategy when making auditory-category decisions. We asked subjects to classify the frequency of a tone burst into one of two overlapping, uniform categories according to the perceived tone frequency. We systematically varied the prior probability of presenting a tone burst with a frequency originating from one versus the other category. Most subjects learned these changes in prior probabilities early in testing and used this information to influence categorization. We also measured each subject's frequency-discrimination thresholds (i.e., their sensory uncertainty levels). We tested each subject's average behavior against variations of a Bayesian model that either led to optimal or sub-optimal decision behavior (i.e. probability matching). In both predicting and fitting each subject's average behavior, we found that probability matching provided a better account of human decision behavior. The model fits confirmed that subjects were able to learn category prior probabilities and approximate forms of the category distributions. Finally, we systematically explored the potential ways that additional noise sources could influence categorization behavior. We found that an optimal decision strategy can produce probability-matching behavior if it utilized non-stationary category distributions and prior probabilities formed over a short stimulus history. Our work extends previous findings into the auditory domain and reformulates the issue of categorization in a manner that can help to interpret the results of previous research within a generative framework.

Categorization is an important cognitive process that allows us to simplify, extract meaning from, and respond to objects in the sensory environment. However, categorization is complicated because an object can belong to multiple categories. Thus, to inform our categorical judgments, we must make use of prior information. Given the importance of categorization, we hypothesized that humans utilize optimal strategies for making categorical judgments that allow us to minimize categorization errors. We found, though, that whereas subjects used prior information (i.e., category prior probability), they were sub-optimal in their categorization behavior. This seems to be common in other perceptual and cognitive tasks as well. We then explored the bases for this sub-optimal behavior and found that it can be consistent with an optimal strategy if we assume that subjects have trial-by-trial noise in components of the judgment process. This work extends previous similar findings into the field of auditory categorization and provides a means to reinterpret previous results.

On the origins of suboptimality in human probabilistic inference

Humans have been shown to combine noisy sensory information with previous experience (priors), in qualitative and sometimes quantitative agreement with the statistically-optimal predictions of Bayesian integration. However, when the prior distribution becomes more complex than a simple Gaussian, such as skewed or bimodal, training takes much longer and performance appears suboptimal. It is unclear whether such suboptimality arises from an imprecise internal representation of the complex prior, or from additional constraints in performing probabilistic computations on complex distributions, even when accurately represented. Here we probe the sources of suboptimality in probabilistic inference using a novel estimation task in which subjects are exposed to an explicitly provided distribution, thereby removing the need to remember the prior. Subjects had to estimate the location of a target given a noisy cue and a visual representation of the prior probability density over locations, which changed on each trial. Different classes of priors were examined (Gaussian, unimodal, bimodal). Subjects' performance was in qualitative agreement with the predictions of Bayesian Decision Theory although generally suboptimal. The degree of suboptimality was modulated by statistical features of the priors but was largely independent of the class of the prior and level of noise in the cue, suggesting that suboptimality in dealing with complex statistical features, such as bimodality, may be due to a problem of acquiring the priors rather than computing with them. We performed a factorial model comparison across a large set of Bayesian observer models to identify additional sources of noise and suboptimality. Our analysis rejects several models of stochastic behavior, including probability matching and sample-averaging strategies. Instead we show that subjects' response variability was mainly driven by a combination of a noisy estimation of the parameters of the priors, and by variability in the decision process, which we represent as a noisy or stochastic posterior.

Sequential decisions: a computational comparison of observational and reinforcement accounts

Right brain damaged patients show impairments in sequential decision making tasks for which healthy people do not show any difficulty. We hypothesized that this difficulty could be due to the failure of right brain damage patients to develop well-matched models of the world. Our motivation is the idea that to navigate uncertainty, humans use models of the world to direct the decisions they make when interacting with their environment. The better the model is, the better their decisions are. To explore the model building and updating process in humans and the basis for impairment after brain injury, we used a computational model of non-stationary sequence learning. RELPH (Reinforcement and Entropy Learned Pruned Hypothesis space) was able to qualitatively and quantitatively reproduce the results of left and right brain damaged patient groups and healthy controls playing a sequential version of Rock, Paper, Scissors. Our results suggests that, in general, humans employ a sub-optimal reinforcement based learning method rather than an objectively better statistical learning approach, and that differences between right brain damaged and healthy control groups can be explained by different exploration policies, rather than qualitatively different learning mechanisms.

The effects of prior learned strategies on updating an opponent's strategy in the rock, paper, scissors game

To explore how model building adapts to changing environments, we had participants play "rock-paper-scissors" against a computer that played a frequency bias or a player-dependent bias and then switched. Participants demonstrated their use of prior experience in how quickly they recognized and exploited changes in the computer's play strategy; in general, the more similar the strategies, the more efficient the updating. These findings inform our understanding of previously reported updating impairments in right-brain damaged patients.

Multiplexing signals in reinforcement learning with internal models and dopamine

A fundamental challenge for computational and cognitive neuroscience is to understand how reward-based learning and decision-making are made and how accrued knowledge and internal models of the environment are incorporated. Remarkable progress has been made in the field, guided by the midbrain dopamine reward prediction error hypothesis and the underlying reinforcement learning framework, which does not involve internal models ('model-free'). Recent studies, however, have begun not only to address more complex decision-making processes that are integrated with model-free decision-making, but also to include internal models about environmental reward structures and the minds of other agents, including model-based reinforcement learning and using generalized prediction errors. Even dopamine, a classic model-free signal, may work as multiplexed signals using model-based information and contribute to representational learning of reward structure.

Cognitive control over learning: creating, clustering, and generalizing task-set structure

Learning and executive functions such as task-switching share common neural substrates, notably prefrontal cortex and basal ganglia. Understanding how they interact requires studying how cognitive control facilitates learning but also how learning provides the (potentially hidden) structure, such as abstract rules or task-sets, needed for cognitive control. We investigate this question from 3 complementary angles. First, we develop a new context-task-set (C-TS) model, inspired by nonparametric Bayesian methods, specifying how the learner might infer hidden structure (hierarchical rules) and decide to reuse or create new structure in novel situations. Second, we develop a neurobiologically explicit network model to assess mechanisms of such structured learning in hierarchical frontal cortex and basal ganglia circuits. We systematically explore the link between these modeling levels across task demands. We find that the network provides an approximate implementation of high-level C-TS computations, with specific neural mechanisms modulating distinct C-TS parameters. Third, this synergism yields predictions about the nature of human optimal and suboptimal choices and response times during learning and task-switching. In particular, the models suggest that participants spontaneously build task-set structure into a learning problem when not cued to do so, which predicts positive and negative transfer in subsequent generalization tests. We provide experimental evidence for these predictions and show that C-TS provides a good quantitative fit to human sequences of choices. These findings implicate a strong tendency to interactively engage cognitive control and learning, resulting in structured abstract representations that afford generalization opportunities and, thus, potentially long-term rather than short-term optimality.

Learning what from where: Effects of spatial regularity on nonspatial sequence learning and updating

The current study examined the influence of redundant stimulus features on our ability to build and update representations of our environment. We hypothesized that our ability to process redundant spatial features would speed our ability to adapt to changing nonspatial regularities. Using a computerized version of the children's game "rock-paper-scissors", undergraduates were instructed to win as often as possible against a computer opponent. The computer's plays were repeating sequences of five choices that were presented either with spatial regularity (i.e., "rock" would always appear on the left, "paper" in the middle, and "scissors" on the right) or without spatial regularity (i.e., the items were equally likely to appear in any of the three locations). Once participants learned a sequence, the computer switched to a different sequence without participants being informed that a switch had occurred. Redundant spatial regularity improved a participant's ability both to learn sequences of plays and to update their plays to reflect new computer sequences. Our results suggest that our perceptual system is sensitive to redundant spatial stimulus features and that this information can improve learning and updating.

Integrating probabilistic models of perception and interactive neural networks: a historical and tutorial review

This article seeks to establish a rapprochement between explicitly Bayesian models of contextual effects in perception and neural network models of such effects, particularly the connectionist interactive activation (IA) model of perception. The article is in part an historical review and in part a tutorial, reviewing the probabilistic Bayesian approach to understanding perception and how it may be shaped by context, and also reviewing ideas about how such probabilistic computations may be carried out in neural networks, focusing on the role of context in interactive neural networks, in which both bottom-up and top-down signals affect the interpretation of sensory inputs. It is pointed out that connectionist units that use the logistic or softmax activation functions can exactly compute Bayesian posterior probabilities when the bias terms and connection weights affecting such units are set to the logarithms of appropriate probabilistic quantities. Bayesian concepts such the prior, likelihood, (joint and marginal) posterior, probability matching and maximizing, and calculating vs. sampling from the posterior are all reviewed and linked to neural network computations. Probabilistic and neural network models are explicitly linked to the concept of a probabilistic generative model that describes the relationship between the underlying target of perception (e.g., the word intended by a speaker or other source of sensory stimuli) and the sensory input that reaches the perceiver for use in inferring the underlying target. It is shown how a new version of the IA model called the multinomial interactive activation (MIA) model can sample correctly from the joint posterior of a proposed generative model for perception of letters in words, indicating that interactive processing is fully consistent with principled probabilistic computation. Ways in which these computations might be realized in real neural systems are also considered.

Gamification, Serious Games, Ludic Simulation, and other Contentious Categories

This paper provides a conceptual framework for gamification, ludic simulations, and serious games. Central to this framework is the spectrum of design that differentiates work and play. Work and play help define software in purpose as games, productivity software, and entertainment. These categories are informed through cognitive feature analysis of narrative and game play structure. Both can be analyzed to determine the degree of work or play in an activity, as well as issues that influence sustained engagement, which is essential for avoiding game abandonment. To demonstrate the framework for the design and analysis of gamification, ludic simulations, and serious games, several case studies are presented with feature analysis to substantiate the categories.

The mixed instrumental controller: using value of information to combine habitual choice and mental simulation

Instrumental behavior depends on both goal-directed and habitual mechanisms of choice. Normative views cast these mechanisms in terms of model-free and model-based methods of reinforcement learning, respectively. An influential proposal hypothesizes that model-free and model-based mechanisms coexist and compete in the brain according to their relative uncertainty. In this paper we propose a novel view in which a single Mixed Instrumental Controller produces both goal-directed and habitual behavior by flexibly balancing and combining model-based and model-free computations. The Mixed Instrumental Controller performs a cost-benefits analysis to decide whether to chose an action immediately based on the available "cached" value of actions (linked to model-free mechanisms) or to improve value estimation by mentally simulating the expected outcome values (linked to model-based mechanisms). Since mental simulation entails cognitive effort and increases the reward delay, it is activated only when the associated "Value of Information" exceeds its costs. The model proposes a method to compute the Value of Information, based on the uncertainty of action values and on the distance of alternative cached action values. Overall, the model by default chooses on the basis of lighter model-free estimates, and integrates them with costly model-based predictions only when useful. Mental simulation uses a sampling method to produce reward expectancies, which are used to update the cached value of one or more actions; in turn, this updated value is used for the choice. The key predictions of the model are tested in different settings of a double T-maze scenario. Results are discussed in relation with neurobiological evidence on the hippocampus - ventral striatum circuit in rodents, which has been linked to goal-directed spatial navigation.

The brain uses adaptive internal models of scene statistics for sensorimotor estimation and planning

Because of uncertainty and noise, the brain should use accurate internal models of the statistics of objects in scenes to interpret sensory signals. Moreover, the brain should adapt its internal models to the statistics within local stimulus contexts. Consider the problem of hitting a baseball. The impoverished nature of the visual information available makes it imperative that batters use knowledge of the temporal statistics and history of previous pitches to accurately estimate pitch speed. Using a laboratory analog of hitting a baseball, we tested the hypothesis that the brain uses adaptive internal models of the statistics of object speeds to plan hand movements to intercept moving objects. We fit Bayesian observer models to subjects' performance to estimate the statistical environments in which subjects' performance would be ideal and compared the estimated statistics with the true statistics of stimuli in an experiment. A first experiment showed that subjects accurately estimated and used the variance of object speeds in a stimulus set to time hitting behavior but also showed serial biases that are suboptimal for stimuli that were uncorrelated over time. A second experiment showed that the strength of the serial biases depended on the temporal correlations within a stimulus set, even when the biases were estimated from uncorrelated stimulus pairs subsampled from the larger set. Taken together, the results show that subjects adapted their internal models of the variance and covariance of object speeds within a stimulus set to plan interceptive movements but retained a bias to positive correlations.

Learning to represent reward structure: a key to adapting to complex environments

Predicting outcomes is a critical ability of humans and animals. The dopamine reward prediction error hypothesis, the driving force behind the recent progress in neural "value-based" decision making, states that dopamine activity encodes the signals for learning in order to predict a reward, that is, the difference between the actual and predicted reward, called the reward prediction error. However, this hypothesis and its underlying assumptions limit the prediction and its error as reactively triggered by momentary environmental events. Reviewing the assumptions and some of the latest findings, we suggest that the internal state representation is learned to reflect the environmental reward structure, and we propose a new hypothesis - the dopamine reward structural learning hypothesis - in which dopamine activity encodes multiplex signals for learning in order to represent reward structure in the internal state, leading to better reward prediction.

A common rule for decision making in animal collectives across species

A diversity of decision-making systems has been observed in animal collectives. In some species, choices depend on the differences of the numbers of animals that have chosen each of the available options, whereas in other species on the relative differences (a behavior known as Weber's law), or follow more complex rules. We here show that this diversity of decision systems corresponds to a single rule of decision making in collectives. We first obtained a decision rule based on Bayesian estimation that uses the information provided by the behaviors of the other individuals to improve the estimation of the structure of the world. We then tested this rule in decision experiments using zebrafish (Danio rerio), and in existing rich datasets of argentine ants (Linepithema humile) and sticklebacks (Gasterosteus aculeatus), showing that a unified model across species can quantitatively explain the diversity of decision systems. Further, these results show that the different counting systems used by animals, including humans, can emerge from the common principle of using social information to make good decisions.

The gambler's fallacy is associated with weak affective decision making but strong cognitive ability

Humans demonstrate an inherent bias towards making maladaptive decisions, as shown by a phenomenon known as the gambler's fallacy (GF). The GF has been traditionally considered as a heuristic bias supported by the fast and automatic intuition system, which can be overcome by the reasoning system. The present study examined an intriguing hypothesis, based on emerging evidence from neuroscience research, that the GF might be attributed to a weak affective but strong cognitive decision making mechanism. With data from a large sample of college students, we found that individuals' use of the GF strategy was positively correlated with their general intelligence and executive function, such as working memory and conflict resolution, but negatively correlated with their affective decision making capacities, as measured by the Iowa Gambling Task. Our result provides a novel insight into the mechanisms underlying the GF, which highlights the significant role of affective mechanisms in adaptive decision-making.