Bounded Rational Decision-Making from Elementary Computations That Reduce Uncertainty

Bayesian Reinforcement Learning With Limited Cognitive Load

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Thermodynamic fluctuation theorems govern human sensorimotor learning

The application of thermodynamic reasoning in the study of learning systems has a long tradition. Recently, new tools relating perfect thermodynamic adaptation to the adaptation process have been developed. These results, known as fluctuation theorems, have been tested experimentally in several physical scenarios and, moreover, they have been shown to be valid under broad mathematical conditions. Hence, although not experimentally challenged yet, they are presumed to apply to learning systems as well. Here we address this challenge by testing the applicability of fluctuation theorems in learning systems, more specifically, in human sensorimotor learning. In particular, we relate adaptive movement trajectories in a changing visuomotor rotation task to fully adapted steady-state behavior of individual participants. We find that human adaptive behavior in our task is generally consistent with fluctuation theorem predictions and discuss the merits and limitations of the approach.

Humans account for cognitive costs when finding shortcuts: An information-theoretic analysis of navigation

When faced with navigating back somewhere we have been before we might either retrace our steps or seek a shorter path. Both choices have costs. Here, we ask whether it is possible to characterize formally the choice of navigational plans as a bounded rational process that trades off the quality of the plan (e.g., its length) and the cognitive cost required to find and implement it. We analyze the navigation strategies of two groups of people that are firstly trained to follow a "default policy" taking a route in a virtual maze and then asked to navigate to various known goal destinations, either in the way they want ("Go To Goal") or by taking novel shortcuts ("Take Shortcut"). We address these wayfinding problems using InfoRL: an information-theoretic approach that formalizes the cognitive cost of devising a navigational plan, as the informational cost to deviate from a well-learned route (the "default policy"). In InfoRL, optimality refers to finding the best trade-off between route length and the amount of control information required to find it. We report five main findings. First, the navigational strategies automatically identified by InfoRL correspond closely to different routes (optimal or suboptimal) in the virtual reality map, which were annotated by hand in previous research. Second, people deliberate more in places where the value of investing cognitive resources (i.e., relevant goal information) is greater. Third, compared to the group of people who receive the "Go To Goal" instruction, those who receive the "Take Shortcut" instruction find shorter but less optimal solutions, reflecting the intrinsic difficulty of finding optimal shortcuts. Fourth, those who receive the "Go To Goal" instruction modulate flexibly their cognitive resources, depending on the benefits of finding the shortcut. Finally, we found a surprising amount of variability in the choice of navigational strategies and resource investment across participants. Taken together, these results illustrate the benefits of using InfoRL to address navigational planning problems from a bounded rational perspective.

Jarzyski's Equality and Crooks' Fluctuation Theorem for General Markov Chains with Application to Decision-Making Systems

We define common thermodynamic concepts purely within the framework of general Markov chains and derive Jarzynski's equality and Crooks' fluctuation theorem in this setup. In particular, we regard the discrete-time case, which leads to an asymmetry in the definition of work that appears in the usual formulation of Crooks' fluctuation theorem. We show how this asymmetry can be avoided with an additional condition regarding the energy protocol. The general formulation in terms of Markov chains allows transferring the results to other application areas outside of physics. Here, we discuss how this framework can be applied in the context of decision-making. This involves the definition of the relevant quantities, the assumptions that need to be made for the different fluctuation theorems to hold, as well as the consideration of discrete trajectories instead of the continuous trajectories, which are relevant in physics.

Bounded rational decision-making models suggest capacity-limited concurrent motor planning in human posterior parietal and frontal cortex

While traditional theories of sensorimotor processing have often assumed a serial decision-making pipeline, more recent approaches have suggested that multiple actions may be planned concurrently and vie for execution. Evidence for the latter almost exclusively stems from electrophysiological studies in posterior parietal and premotor cortex of monkeys. Here we study concurrent prospective motor planning in humans by recording functional magnetic resonance imaging (fMRI) during a delayed response task engaging movement sequences towards multiple potential targets. We find that also in human posterior parietal and premotor cortex delay activity modulates both with sequence complexity and the number of potential targets. We tested the hypothesis that this modulation is best explained by concurrent prospective planning as opposed to the mere maintenance of potential targets in memory. We devise a bounded rationality model with information constraints that optimally assigns information resources for planning and memory for this task and determine predicted information profiles according to the two hypotheses. When regressing delay activity on these model predictions, we find that the concurrent prospective planning strategy provides a significantly better explanation of the fMRI-signal modulations. Moreover, we find that concurrent prospective planning is more costly and thus limited for most subjects, as expressed by the best fitting information capacities. We conclude that bounded rational decision-making models allow relating both behavior and neural representations to utilitarian task descriptions based on bounded optimal information-processing assumptions.

When the future is uncertain, it can be beneficial to concurrently plan several action possibilities in advance. Electrophysiological research found evidence in monkeys that brain regions in posterior parietal and promotor cortex are indeed capable of planning several actions in parallel. We now used fMRI to study brain activity in these brain regions in humans. For our analyses we applied bounded rationality models that optimally assign information resources to fMRI activity in a complex motor planning task. We find that theoretical information costs of concurrent prospective planning explained fMRI activity profiles significantly better than assuming alternative memory-based strategies. Moreover, exploiting the model allowed us to quantify the individual capacity limit for concurrent planning and to relate these individual limits to both subjects’ behavior and to their neural representations of planning.

From Bayes-optimal to heuristic decision-making in a two-alternative forced choice task with an information-theoretic bounded rationality model

Bayes optimal and heuristic decision-making schemes are often considered fundamentally opposed to each other as a framework for studying human choice behavior, although recently it has been proposed that bounded rationality may provide a natural bridge between the two when varying information-processing resources. Here, we investigate a two-alternative forced choice task with varying time constraints, where subjects have to assign multi-component symbolic patterns to one of two stimulus classes. As expected, we find that subjects' response behavior becomes more imprecise with more time pressure. However, we also see that their response behavior changes qualitatively. By regressing subjects' decision weights, we find that decisions allowing for plenty of decision time rely on weighing multiple stimulus features, whereas decisions under high time pressure are made mostly based on a single feature. While the first response pattern is in line with a Bayes-optimal decision strategy, the latter could be considered as an instantiation of heuristic decision-making with cue discounting. When fitting a bounded rational decision model with multiple feature channels and varying information-processing capacity to subjects' responses, we find that the model is able to capture subjects' behavioral change. The model successfully reflects the simplicity of heuristics as well as the efficiency of optimal decision making, thus acting as a bridge between the two approaches.

Representing preorders with injective monotones

We introduce a new class of real-valued monotones in preordered spaces, injective monotones. We show that the class of preorders for which they exist lies in between the class of preorders with strict monotones and preorders with countable multi-utilities, improving upon the known classification of preordered spaces through real-valued monotones. We extend several well-known results for strict monotones (Richter-Peleg functions) to injective monotones, we provide a construction of injective monotones from countable multi-utilities, and relate injective monotones to classic results concerning Debreu denseness and order separability. Along the way, we connect our results to Shannon entropy and the uncertainty preorder, obtaining new insights into how they are related. In particular, we show how injective monotones can be used to generalize some appealing properties of Jaynes' maximum entropy principle, which is considered a basis for statistical inference and serves as a justification for many regularization techniques that appear throughout machine learning and decision theory.

The two kinds of free energy and the Bayesian revolution

The concept of free energy has its origins in 19th century thermodynamics, but has recently found its way into the behavioral and neural sciences, where it has been promoted for its wide applicability and has even been suggested as a fundamental principle of understanding intelligent behavior and brain function. We argue that there are essentially two different notions of free energy in current models of intelligent agency, that can both be considered as applications of Bayesian inference to the problem of action selection: one that appears when trading off accuracy and uncertainty based on a general maximum entropy principle, and one that formulates action selection in terms of minimizing an error measure that quantifies deviations of beliefs and policies from given reference models. The first approach provides a normative rule for action selection in the face of model uncertainty or when information processing capabilities are limited. The second approach directly aims to formulate the action selection problem as an inference problem in the context of Bayesian brain theories, also known as Active Inference in the literature. We elucidate the main ideas and discuss critical technical and conceptual issues revolving around these two notions of free energy that both claim to apply at all levels of decision-making, from the high-level deliberation of reasoning down to the low-level information processing of perception.

Specialization in Hierarchical Learning Systems

Joining multiple decision-makers together is a powerful way to obtain more sophisticated decision-making systems, but requires to address the questions of division of labor and specialization. We investigate in how far information constraints in hierarchies of experts not only provide a principled method for regularization but also to enforce specialization. In particular, we devise an information-theoretically motivated on-line learning rule that allows partitioning of the problem space into multiple sub-problems that can be solved by the individual experts. We demonstrate two different ways to apply our method: (i) partitioning problems based on individual data samples and (ii) based on sets of data samples representing tasks. Approach (i) equips the system with the ability to solve complex decision-making problems by finding an optimal combination of local expert decision-makers. Approach (ii) leads to decision-makers specialized in solving families of tasks, which equips the system with the ability to solve meta-learning problems. We show the broad applicability of our approach on a range of problems including classification, regression, density estimation, and reinforcement learning problems, both in the standard machine learning setup and in a meta-learning setting.

Analytic Hierarchy Process (AHP)-Based Aggregation Mechanism for Resilience Measurement: NATO Aggregated Resilience Decision Support Model

Resilience is a complex system that represents dynamic behaviours through its complicated structure with various nodes, interrelations, and information flows. Like other international organizations NATO has also been dealing with the measurement of this complex phenomenon in order to have a comprehensive understanding of the civil environment and its impact on military operations. With this ultimate purpose, NATO had developed and executed a prototype model with the system dynamics modelling and simulation paradigm. NATO has created an aggregated resilience model as an upgrade of the prototype one, as discussed within this study. The structure of the model, aggregation mechanism and shock parametrization methodologies used in the development of the model comprise the scope of this study. Analytic Hierarchy Process (AHP), which is a multi-criteria decision-making technique is the methodology that is used for the development of the aggregation mechanism. The main idea of selecting the AHP methodology is its power and usefulness in mitigating bias in the decision-making process, its capability to increase the number of what-if scenarios to be created, and its contribution to the quality of causal explanations with the granularity it provides. The parametrized strategic shock input page, AHP-based weighted resilience and risk parameters input pages, one more country insertion to the model, and the decision support system page enhance the capacity of the prototype model. As part of the model, the decision support system page stands out as the strategic level cockpit where the colour codes give a clear idea at first about the overall situational picture and country-wise resilience and risk status. At the validation workshop, users not only validated the model but also discussed further development opportunities, such as adding more strategic shocks into the model and introduction of new parameters that will be determined by a big data analysis on relevant open source databases. The developed model has the potential to inspire high-level decision-makers dealing with resilience management in other international organizations, such as the United Nations.

Computer Assisted Wargame for Military Capability-Based Planning

Capability-based planning as an approach to defense planning is an almost infinitely complex engineered system with countless nodes and layers of interdependency, influenced by state and non-state diplomatic activities, information, military and economic actions creating secondary and third order effects. The main output of capability-based planning is the set of capability requirements needed to achieve the expected end-state. One revitalized qualitative technique that allows us to gain insights into unstructured and fuzzy problems in the military is wargaming-in its simplest form this involves manual wargaming. At the same time, there has been a push to bring computer assistance to such wargaming, especially to support umpire adjudication and move more generally towards full automation of human elements in wargames. However, computer assistance in wargaming should not be pushed, regardless of cost, towards quantitative techniques. The objective complexity of a problem often does not allow us to replicate the operational environment with the required fidelity to get credible experimental results. This paper discusses a discovery experiment aiming to verify the concept of applying a qualitative expert system within computer assisted wargaming for developing capability requirements in order to reduce umpire bias and risk associated with their decisions. The innovation here lies in applying system dynamics modelling and simulation paradigms when designing the theoretical model of capability development, which forms the core of the expert system. This new approach enables qualitative comparisons between different sets of proposed capability requirements. Moreover, the expert system allows us to reveal the effects of budget cuts on proposed capability requirement solutions, which the umpire was previously unable to articulate when comparing individual solutions by relying solely on his own knowledge. Players in the wargame validated the proposed concept and suggested how the study might be developed going forward: namely, by enabling users to define their own capabilities and not being limited by a predefined set of capabilities.

Optimization of the Casualties' Treatment Process: Blended Military Experiment

At the battalion level, NATO ROLE1 medical treatment command focuses on the provision of primary health care being the very first physician and higher medical equipment intervention for casualty treatments. ROLE1 has paramount importance in casualty reductions, representing a complex system in current operations. This study deals with an experiment on the optimization of ROLE1 according to the key parameters of the numbers of physicians, the number of ambulances and the distance between ROLE1 and the current battlefield. The very first step in this study is to design and implement a model of current battlefield casualties. The model uses friction data generated from an already executed computer assisted exercise (CAX) while employing a constructive simulation to produce offense and defense scenarios on the flow of casualties. The next step in the study is to design and implement a model representing the transportation to ROLE1, its structure and behavior. The deterministic model of ROLE1, employing a system dynamics simulation paradigm, uses the previously generated casualty flows as the inputs representing human decision-making processes through the recorder CAX events. A factorial experimental design for the ROLE1 model revealed the recommended variants of the ROLE1 structure for both offensive and defensive operations. The overall recommendation is for the internal structure of ROLE1 to have three ambulances and three physicians for any kind of current operation and any distance between ROLE1 and the current battlefield within the limit of 20 min. This study provides novelty in the methodology of casualty estimations involving human decision-making factors as well as the optimization of medical treatment processes through experimentation with the process model.

Analyzing Abstraction and Hierarchical Decision-Making in Absolute Identification by Information-Theoretic Bounded Rationality

In the face of limited computational resources, bounded rational decision theory predicts that information-processing should be concentrated on actions that make a significant contribution in terms of the utility achieved. Accordingly, information-processing can be simplified by choosing stereotypic actions that lead to satisfactory performance over a range of different inputs rather than choosing a specific action for each input. Such a set of similar inputs with similar action responses would then correspond to an abstraction that can be harnessed with possibly negligible loss in utility, but with potentially considerable savings in information-processing effort. Here we test this prediction in an identification task, where human subjects were asked to estimate the roundness of ellipses varying from a straight line to a perfect circle. Crucially, when reporting their estimates, subjects could choose between three different levels of precision corresponding to three levels of abstraction in a decision-making hierarchy. To induce changes in level selection, we manipulated the information-processing resources available at the perceptual and action stages by varying the difficulty of identifying the stimulus and by enforcing different response times in the action stage. In line with theoretical predictions, we find that subjects adapt their abstraction level depending on the available resources. We compare subjects' behavior to the maximum efficiency predicated by the bounded rational decision-making model and investigate possible sources of inefficiency.