Sources of suboptimality in a minimalistic explore-exploit task

Claustrum neurons projecting to the anterior cingulate restrict engagement during sleep and behavior

The claustrum has been linked to attention and sleep. We hypothesized that this reflects a shared function, determining responsiveness to stimuli, which spans the axis of engagement. To test this hypothesis, we recorded claustrum population dynamics from male mice during both sleep and an attentional task ('ENGAGE'). Heightened activity in claustrum neurons projecting to the anterior cingulate cortex (ACCp) corresponded to reduced sensory responsiveness during sleep. Similarly, in the ENGAGE task, heightened ACCp activity correlated with disengagement and behavioral lapses, while low ACCp activity correlated with hyper-engagement and impulsive errors. Chemogenetic elevation of ACCp activity reduced both awakenings during sleep and impulsive errors in the ENGAGE task. Furthermore, mice employing an exploration strategy in the task showed a stronger correlation between ACCp activity and performance compared to mice employing an exploitation strategy which reduced task complexity. Our results implicate ACCp claustrum neurons in restricting engagement during sleep and goal-directed behavior.

Tracking the contribution of inductive bias to individualised internal models

Internal models capture the regularities of the environment and are central to understanding how humans adapt to environmental statistics. In general, the correct internal model is unknown to observers, instead they rely on an approximate model that is continually adapted throughout learning. However, experimenters assume an ideal observer model, which captures stimulus structure but ignores the diverging hypotheses that humans form during learning. We combine non-parametric Bayesian methods and probabilistic programming to infer rich and dynamic individualised internal models from response times. We demonstrate that the approach is capable of characterizing the discrepancy between the internal model maintained by individuals and the ideal observer model and to track the evolution of the contribution of the ideal observer model to the internal model throughout training. In particular, in an implicit visuomotor sequence learning task the identified discrepancy revealed an inductive bias that was consistent across individuals but varied in strength and persistence.

Instead of mapping stimuli directly to response, humans and other complex organisms are thought to maintain internal models of the environment. These internal models represent parts of the environment that are most relevant for deciding how to act in a given situation and therefore are key to explaining human behaviour. In behavioural experiments it is often assumed that the internal model in the subject’s brain matches the true model that governs the experiment. However this assumption can be violated due to a variety of reasons, such as insufficient training. Furthermore, the deviation of the internal model from the true model is not uniform across individuals, and therefore it summarizes the subjective beliefs of humans. In this paper, we provide a method to reverse engineer the internal model for individual subjects by analysing trial by trial behavioural measurements such as reaction times. We then track and analyse these reverse engineered models over the course of the experiment to see how participants trade off between an early inductive bias towards Markovian dynamics and the model that reflects the evidence that humans accumulate during learning about the actual statistics of the stimuli.

Simple Threshold Rules Solve Explore/Exploit Trade-offs in a Resource Accumulation Search Task

How, and how well, do people switch between exploration and exploitation to search for and accumulate resources? We study the decision processes underlying such exploration/exploitation trade-offs using a novel card selection task that captures the common situation of searching among multiple resources (e.g., jobs) that can be exploited without depleting. With experience, participants learn to switch appropriately between exploration and exploitation and approach optimal performance. We model participants' behavior on this task with random, threshold, and sampling strategies, and find that a linear decreasing threshold rule best fits participants' results. Further evidence that participants use decreasing threshold-based strategies comes from reaction time differences between exploration and exploitation; however, participants themselves report non-decreasing thresholds. Decreasing threshold strategies that "front-load" exploration and switch quickly to exploitation are particularly effective in resource accumulation tasks, in contrast to optimal stopping problems like the Secretary Problem requiring longer exploration.

Cognitive models of optimal sequential search with recall

Many everyday decisions require sequential search, according to which available choice options are observed one at a time, with each observation involving some cost to the decision maker. In these tasks, decision makers need to trade-off the chances of finding better options with the cost of search. Optimal strategies in such tasks involve threshold decision rules, which terminate the search as soon as an option exceeding a reward value is found. Threshold rules can be seen as special cases of well-known algorithmic decision processes, such as the satisficing heuristic. Prior work has found that decision makers do use threshold rules, however the stopping thresholds observed in data are typically smaller than the (expected value maximizing) optimal threshold. We put forward an array of cognitive models and use parametric model fits on participant-level search data to examine why decision makers adopt seemingly suboptimal thresholds. We find that people's behavior is consistent with optimal search if we allow participants to display risk aversion, psychological effort cost, and decision error. Thus, decision makers appear to be able to search in a resource-rational manner that maximizes stochastic risk averse utility. Our findings shed light on the psychological factors that guide sequential decision making, and show how threshold models can be used to describe both computational and algorithmic aspects of search behavior.

A linear threshold model for optimal stopping behavior

In many real-life decisions, options are distributed in space and time, making it necessary to search sequentially through them, often without a chance to return to a rejected option. The optimal strategy in these tasks is to choose the first option that is above a threshold that depends on the current position in the sequence. The implicit decision-making strategies by humans vary but largely diverge from this optimal strategy. The reasons for this divergence remain unknown. We present a model of human stopping decisions in sequential decision-making tasks based on a linear threshold heuristic. The first two studies demonstrate that the linear threshold model accounts better for sequential decision making than existing models. Moreover, we show that the model accurately predicts participants' search behavior in different environments. In the third study, we confirm that the model generalizes to a real-world problem, thus providing an important step toward understanding human sequential decision making.