Choice perseveration in value-based decision making: The impact of inter-trial interval and mood

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

From single decisions to sequential choice patterns: Extending the dynamics of value-based decision-making

Every day, we make many value-based decisions where we weigh the value of options with other properties, e.g. their time of delivery. In the laboratory, such value-based decision-making is usually studied on a trial by trial basis and each decision is assumed to represent an isolated choice process. Real-life decisions however are usually embedded in a rich context of previous choices at different time scales. A fundamental question is therefore how the dynamics of value-based decision processes unfold on a time scale across several decisions. Indeed, findings from perceptual decision making suggest that sequential decisions patterns might also be present for vale-based decision making. Here, we use a neural-inspired attractor model as an instance of dynamic models from perceptual decision making, as such models incorporate inherent activation dynamics across decisions. We use the model to predict sequential patterns, namely oscillatory switching, perseveration and dependence of perseveration on the delay between decisions. Furthermore, we predict RT effects for specific sequences of trials. We validate the predictions in two new studies and a reanalysis of existing data from a novel decision game in which participants have to perform delay discounting decisions. Applying the validated reasoning to a well-established choice questionnaire, we illustrate and discuss that taking sequential choice patterns into account may be necessary to accurately analyse and model value-based decision processes, especially when considering differences between individuals.

Is value-based choice repetition susceptible to medial frontal transcranial direct current stimulation (tDCS)? A preregistered study

In value-based decision making, people have to weigh different options based on their subjective value. This process, however, also is influenced by choice biases, such as choice repetition: in a series of choices, people are more likely to repeat their decision than to switch to a different choice. Previously, it was shown that transcranial direct current stimulation (tDCS) can affect such choice biases. We applied tDCS over the medial prefrontal cortex to investigate whether tDCS can alter choice repetition in value-based decision making. In a preregistered study, we applied anodal, cathodal, and sham tDCS stimulation to 52 participants. While we found robust choice repetition effects, we did not find support for an effect of tDCS stimulation. We discuss these findings within the larger scope of the tDCS literature and highlight the potential roles of interindividual variability and current density strength.

From high- to one-dimensional dynamics of decision making: testing simplifications in attractor models

Computational models introduce simplifications that need to be understood and validated. For attractor models of decision making, the main simplification is the high-level representation of different sub-processes of the complex decision system in one dynamic description of the overall process dynamics. This simplification implies that the overall process dynamics of the decision system are independent from specific values handled in different sub-processes. Here, we test the validity of this simplification empirically by investigating choice perseveration in a nonverbal, value-based decision task. Specifically, we tested whether choice perseveration occurred irrespectively of the attribute dimension as suggested by a simulation of the computational model. We find evidence supporting the validity of the simplification. We conclude that the simplification might capture mechanistic aspects of decision-making processes, and that the summation of the overall process dynamics of decision systems into one single variable is a valid approach in computational modeling. Supplement materials such as empirical data, analysis scripts, and the computational model are publicly available at the Open Science Framework (osf.io/7fb5q).