An information-theoretic perspective on the costs of cognition

Bayesian Reinforcement Learning With Limited Cognitive Load

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

What Is Mental Effort: A Clinical Perspective

Although mental effort is a frequently used term, it is poorly defined and understood. Consequently, its usage is frequently loose and potentially misleading. In neuroscience research, the term is used to mean both the cognitive work that is done to meet task demands and the subjective experience of performing that work. We argue that conflating these two meanings hampers progress in understanding cognitive impairments in neuropsychiatric conditions because cognitive work and the subjective experience of it have distinct underlying mechanisms. We suggest that the most coherent and clinically useful perspective on mental effort is that it is a subjective experience. This makes a clear distinction between cognitive impairments that arise from changes in the cognitive apparatus, as in dementia and brain injury, and those that arise from subjective difficulties in carrying out the cognitive work, as in attention-deficit/hyperactivity disorder, depression, and other motivational disorders. We review recent advances in neuroscience research that suggests that the experience of effort has emerged to control task switches so as to minimize costs relative to benefits. We consider how these advances can contribute to our understanding of the experience of increased effort perception in clinical populations. This more specific framing of mental effort will offer a deeper understanding of the mechanisms of cognitive impairments in differing clinical groups and will ultimately facilitate better therapeutic interventions.

An Information-Theoretic Account of Availability Effects in Language Production

I present a computational-level model of language production in terms of a combination of information theory and control theory in which words are chosen incrementally in order to maximize communicative value subject to an information-theoretic capacity constraint. The theory generally predicts a tradeoff between ease of production and communicative accuracy. I apply the theory to two cases of apparent availability effects in language production, in which words are selected on the basis of their accessibility to a speaker who has not yet perfectly planned the rest of the utterance. Using corpus data on English relative clause complementizer dropping and experimental data on Mandarin noun classifier choice, I show that the theory reproduces the observed phenomena, providing an alternative account to Uniform Information Density and a promising general model of language production which is tightly linked to emerging theories in computational neuroscience.

Alpha oscillations encode Bayesian belief updating underlying attentional allocation in dynamic environments

In a dynamic environment, expectations of the future constantly change based on updated evidence and affect the dynamic allocation of attention. To further investigate the neural mechanisms underlying attentional expectancies, we employed a modified Central Cue Posner Paradigm in which the probability of cues being valid (that is, accurately indicated the upcoming target location) was manipulated. Attentional deployment to the cued location (α), which was governed by precision of predictions on previous trials, was estimated using a hierarchical Bayesian model and was included as a regressor in the analyses of electrophysiological (EEG) data. Our results revealed that before the target appeared, alpha oscillations (8∼13 Hz) for high-predictability cues (88 % valid) were significantly predicted by precision-dependent attention (α). This relationship was not observed under low-predictability conditions (69 % and 50 % valid cues). After the target appeared, precision-dependent attention (α) correlated with alpha band oscillations only in the valid cue condition and not in the invalid condition. Further analysis under conditions of significant attentional modulation by precision suggested a separate effect of cue orientation. These results provide new insights on how trial-by-trial Bayesian belief updating relates to alpha band encoding of environmentally-sensitive allocation of visual spatial attention.

An information-theoretic approach to the typology of spatial demonstratives

We explore systems of spatial deictic words (such as 'here' and 'there') from the perspective of communicative efficiency using typological data from over 200 languages Nintemann et al. (2020). We argue from an information-theoretic perspective that spatial deictic systems balance informativity and complexity in the sense of the Information Bottleneck (Zaslavsky et al., (2018). We find that under an appropriate choice of cost function and need probability over meanings, among all the 21,146 theoretically possible spatial deictic systems, those adopted by real languages lie near an efficient frontier of informativity and complexity. Moreover, we find that the conditions that the need probability and the cost function need to satisfy for this result are consistent with the cognitive science literature on spatial cognition, especially regarding the source-goal asymmetry. We further show that the typological data are better explained by introducing a notion of consistency into the Information Bottleneck framework, which is jointly optimized along with informativity and complexity.

Neural correlates of valence-dependent belief and value updating during uncertainty reduction: An fNIRS study

Selective use of new information is crucial for adaptive decision-making. Combining a gamble bidding task with assessing cortical responses using functional near-infrared spectroscopy (fNIRS), we investigated potential effects of information valence on behavioral and neural processes of belief and value updating during uncertainty reduction in young adults. By modeling changes in the participants' expressed subjective values using a Bayesian model, we dissociated processes of (i) updating beliefs about statistical properties of the gamble, (ii) updating values of a gamble based on new information about its winning probabilities, as well as (iii) expectancy violation. The results showed that participants used new information to update their beliefs and values about the gambles in a quasi-optimal manner, as reflected in the selective updating only in situations with reducible uncertainty. Furthermore, their updating was valence-dependent: information indicating an increase in winning probability was underweighted, whereas information about a decrease in winning probability was updated in good agreement with predictions of the Bayesian decision theory. Results of model-based and moderation analyses showed that this valence-dependent asymmetry was associated with a distinct contribution of expectancy violation, besides belief updating, to value updating after experiencing new positive information regarding winning probabilities. In line with the behavioral results, we replicated previous findings showing involvements of frontoparietal brain regions in the different components of updating. Furthermore, this study provided novel results suggesting a valence-dependent recruitment of brain regions. Individuals with stronger oxyhemoglobin responses during value updating was more in line with predictions of the Bayesian model while integrating new information that indicates an increase in winning probability. Taken together, this study provides first results showing expectancy violation as a contributing factor to sub-optimal valence-dependent updating during uncertainty reduction and suggests limitations of normative Bayesian decision theory.

Information-theoretic principles in incremental language production

I apply a recently emerging perspective on the complexity of action selection, the rate-distortion theory of control, to provide a computational-level model of errors and difficulties in human language production, which is grounded in information theory and control theory. Language production is cast as the sequential selection of actions to achieve a communicative goal subject to a capacity constraint on cognitive control. In a series of calculations, simulations, corpus analyses, and comparisons to experimental data, I show that the model directly predicts some of the major known qualitative and quantitative phenomena in language production, including semantic interference and predictability effects in word choice; accessibility-based ("easy-first") production preferences in word order alternations; and the existence and distribution of disfluencies including filled pauses, corrections, and false starts. I connect the rate-distortion view to existing models of human language production, to probabilistic models of semantics and pragmatics, and to proposals for controlled language generation in the machine learning and reinforcement learning literature.

Information-Theoretic Neural Decoding Reproduces Several Laws of Human Behavior

Human response times conform to several regularities including the Hick-Hyman law, the power law of practice, speed-accuracy trade-offs, and the Stroop effect. Each of these has been thoroughly modeled in isolation, but no account describes these phenomena as predictions of a unified framework. We provide such a framework and show that the phenomena arise as decoding times in a simple neural rate code with an entropy stopping threshold. Whereas traditional information-theoretic encoding systems exploit task statistics to optimize encoding strategies, we move this optimization to the decoder, treating it as a Bayesian ideal observer that can track transmission statistics as prior information during decoding. Our approach allays prominent concerns that applying information-theoretic perspectives to modeling brain and behavior requires complex encoding schemes that are incommensurate with neural encoding.

Is it cognitive effort you measure? Comparing three task paradigms to the Need for Cognition scale

Measuring individual differences in cognitive effort can be elusive as effort is a function of motivation and ability. We report six studies (N = 663) investigating the relationship of Need for Cognition and working memory capacity with three cognitive effort measures: demand avoidance in the Demand Selection Task, effort discounting measured as the indifference point in the Cognitive Effort Discounting paradigm, and rational reasoning score with items from the heuristic and bias literature. We measured perceived mental effort with the NASA task load index. The three tasks were not correlated with each other (all r's < .1, all p's > .1). Need for Cognition was positively associated with effort discounting (r = .168, p < .001) and rational reasoning (r = .176, p < .001), but not demand avoidance (r = .085, p = .186). Working memory capacity was related to effort discounting (r = .185, p = .004). Higher perceived effort was related to poorer rational reasoning. Our data indicate that two of the tasks are related to Need for Cognition but are also influenced by a participant's working memory capacity. We discuss whether any of the tasks measure cognitive effort.

Evidence or Confidence: What Is Really Monitored during a Decision?

Assessing our confidence in the choices we make is important to making adaptive decisions, and it is thus no surprise that we excel in this ability. However, standard models of decision-making, such as the drift-diffusion model (DDM), treat confidence assessment as a post hoc or parallel process that does not directly influence the choice, which depends only on accumulated evidence. Here, we pursue the alternative hypothesis that what is monitored during a decision is an evolving sense of confidence (that the to-be-selected option is the best) rather than raw evidence. Monitoring confidence has the appealing consequence that the decision threshold corresponds to a desired level of confidence for the choice, and that confidence improvements can be traded off against the resources required to secure them. We show that most previous findings on perceptual and value-based decisions traditionally interpreted from an evidence-accumulation perspective can be explained more parsimoniously from our novel confidence-driven perspective. Furthermore, we show that our novel confidence-driven DDM (cDDM) naturally generalizes to decisions involving any number of alternative options - which is notoriously not the case with traditional DDM or related models. Finally, we discuss future empirical evidence that could be useful in adjudicating between these alternatives.

Cognitive effort and active inference

This paper aims to integrate some key constructs in the cognitive neuroscience of cognitive control and executive function by formalising the notion of cognitive (or mental) effort in terms of active inference. To do so, we call upon a task used in neuropsychology to assess impulse inhibition-a Stroop task. In this task, participants must suppress the impulse to read a colour word and instead report the colour of the text of the word. The Stroop task is characteristically effortful, and we unpack a theory of mental effort in which, to perform this task accurately, participants must overcome prior beliefs about how they would normally act. However, our interest here is not in overt action, but in covert (mental) action. Mental actions change our beliefs but have no (direct) effect on the outside world-much like deploying covert attention. This account of effort as mental action lets us generate multimodal (choice, reaction time, and electrophysiological) data of the sort we might expect from a human participant engaging in this task. We analyse how parameters determining cognitive effort influence simulated responses and demonstrate that-when provided only with performance data-these parameters can be recovered, provided they are within a certain range.

Skilled motor control of an inverted pendulum implies low entropy of states but high entropy of actions

The mastery of skills, such as balancing an inverted pendulum, implies a very accurate control of movements to achieve the task goals. Traditional accounts of skilled action control that focus on either routinization or perceptual control make opposite predictions about the ways we achieve mastery. The notion of routinization emphasizes the decrease of the variance of our actions, whereas the notion of perceptual control emphasizes the decrease of the variance of the states we visit, but not of the actions we execute. Here, we studied how participants managed control tasks of varying levels of difficulty, which consisted of controlling inverted pendulums of different lengths. We used information-theoretic measures to compare the predictions of alternative accounts that focus on routinization and perceptual control, respectively. Our results indicate that the successful performance of the control task strongly correlates with the decrease of state variability and the increase of action variability. As postulated by perceptual control theory, the mastery of skilled pendulum control consists in achieving stable control of goals by flexible means.

Fatigue and Human Performance: An Updated Framework

Fatigue has been defined differently in the literature depending on the field of research. The inconsistent use of the term fatigue complicated scientific communication, thereby limiting progress towards a more in-depth understanding of the phenomenon. Therefore, Enoka and Duchateau (Med Sci Sports Exerc 48:2228-38, 2016, [3]) proposed a fatigue framework that distinguishes between trait fatigue (i.e., fatigue experienced by an individual over a longer period of time) and motor or cognitive task-induced state fatigue (i.e., self-reported disabling symptom derived from the two interdependent attributes performance fatigability and perceived fatigability). Thereby, performance fatigability describes a decrease in an objective performance measure, while perceived fatigability refers to the sensations that regulate the integrity of the performer. Although this framework served as a good starting point to unravel the psychophysiology of fatigue, several important aspects were not included and the interdependence of the mechanisms driving performance fatigability and perceived fatigability were not comprehensively discussed. Therefore, the present narrative review aimed to (1) update the fatigue framework suggested by Enoka and Duchateau (Med Sci Sports Exerc 48:2228-38, 2016, [3]) pertaining the taxonomy (i.e., cognitive performance fatigue and perceived cognitive fatigue were added) and important determinants that were not considered previously (e.g., effort perception, affective valence, self-regulation), (2) discuss the mechanisms underlying performance fatigue and perceived fatigue in response to motor and cognitive tasks as well as their interdependence, and (3) provide recommendations for future research on these interactions. We propose to define motor or cognitive task-induced state fatigue as a psychophysiological condition characterized by a decrease in motor or cognitive performance (i.e., motor or cognitive performance fatigue, respectively) and/or an increased perception of fatigue (i.e., perceived motor or cognitive fatigue). These dimensions are interdependent, hinge on different determinants, and depend on body homeostasis (e.g., wakefulness, core temperature) as well as several modulating factors (e.g., age, sex, diseases, characteristics of the motor or cognitive task). Consequently, there is no single factor primarily determining performance fatigue and perceived fatigue in response to motor or cognitive tasks. Instead, the relative weight of each determinant and their interaction are modulated by several factors.

Humans account for cognitive costs when finding shortcuts: An information-theoretic analysis of navigation

When faced with navigating back somewhere we have been before we might either retrace our steps or seek a shorter path. Both choices have costs. Here, we ask whether it is possible to characterize formally the choice of navigational plans as a bounded rational process that trades off the quality of the plan (e.g., its length) and the cognitive cost required to find and implement it. We analyze the navigation strategies of two groups of people that are firstly trained to follow a "default policy" taking a route in a virtual maze and then asked to navigate to various known goal destinations, either in the way they want ("Go To Goal") or by taking novel shortcuts ("Take Shortcut"). We address these wayfinding problems using InfoRL: an information-theoretic approach that formalizes the cognitive cost of devising a navigational plan, as the informational cost to deviate from a well-learned route (the "default policy"). In InfoRL, optimality refers to finding the best trade-off between route length and the amount of control information required to find it. We report five main findings. First, the navigational strategies automatically identified by InfoRL correspond closely to different routes (optimal or suboptimal) in the virtual reality map, which were annotated by hand in previous research. Second, people deliberate more in places where the value of investing cognitive resources (i.e., relevant goal information) is greater. Third, compared to the group of people who receive the "Go To Goal" instruction, those who receive the "Take Shortcut" instruction find shorter but less optimal solutions, reflecting the intrinsic difficulty of finding optimal shortcuts. Fourth, those who receive the "Go To Goal" instruction modulate flexibly their cognitive resources, depending on the benefits of finding the shortcut. Finally, we found a surprising amount of variability in the choice of navigational strategies and resource investment across participants. Taken together, these results illustrate the benefits of using InfoRL to address navigational planning problems from a bounded rational perspective.

A space of goals: the cognitive geometry of informationally bounded agents

Traditionally, Euclidean geometry is treated by scientists as a priori and objective. However, when we take the position of an agent, the problem of selecting a best route should also factor in the abilities of the agent, its embodiment and particularly its cognitive effort. In this paper, we consider geometry in terms of travel between states within a world by incorporating information processing costs with the appropriate spatial distances. This induces a geometry that increasingly differs from the original geometry of the given world as information costs become increasingly important. We visualize this 'cognitive geometry' by projecting it onto two- and three-dimensional spaces showing distinct distortions reflecting the emergence of epistemic and information-saving strategies as well as pivot states. The analogies between traditional cost-based geometries and those induced by additional informational costs invite a generalization of the notion of geodesics as cheapest routes towards the notion of infodesics. In this perspective, the concept of infodesics is inspired by the property of geodesics that, travelling from a given start location to a given goal location along a geodesic, not only the goal, but all points along the way are visited at optimal cost from the start.

Digital computing through randomness and order in neural networks

We propose that coding and decoding in the brain are achieved through digital computation using three principles: relative ordinal coding of inputs, random connections between neurons, and belief voting. Due to randomization and despite the coarseness of the relative codes, we show that these principles are sufficient for coding and decoding sequences with error-free reconstruction. In particular, the number of neurons needed grows linearly with the size of the input repertoire growing exponentially. We illustrate our model by reconstructing sequences with repertoires on the order of a billion items. From this, we derive the Shannon equations for the capacity limit to learn and transfer information in the neural population, which is then generalized to any type of neural network. Following the maximum entropy principle of efficient coding, we show that random connections serve to decorrelate redundant information in incoming signals, creating more compact codes for neurons and therefore, conveying a larger amount of information. Henceforth, despite the unreliability of the relative codes, few neurons become necessary to discriminate the original signal without error. Finally, we discuss the significance of this digital computation model regarding neurobiological findings in the brain and more generally with artificial intelligence algorithms, with a view toward a neural information theory and the design of digital neural networks.

Anticipatory attention is a stable state induced by transient control mechanisms

Anticipatory attention is a neurocognitive state in which attention control regions bias neural activity in sensory cortical areas to facilitate the selective processing of incoming targets. Previous electroencephalographic (EEG) studies have identified event-related potential (ERP) signatures of anticipatory attention, and implicated alpha band (8-12 Hz) EEG oscillatory activity in the selective control of neural excitability in visual cortex. However, the degree to which ERP and alpha band measures reflect related or distinct underlying neural processes remains to be further understood. To investigate this question, we analyzed EEG data from 20 human participants performing a cued object-based attention task. We used support vector machine (SVM) decoding analysis to compare the attentional time courses of ERP signals and alpha band power. We found that ERP signals encoding attentional instructions are dynamic and precede stable attention-related changes in alpha power, suggesting that ERP and alpha power reflect distinct neural processes. We proposed that the ERP patterns reflect transient attentional orienting signals originating in higher order control areas, whereas the patterns of synchronized oscillatory neural activity in the alpha band reflect a sustained attentional state. These findings support the hypothesis that anticipatory attention involves transient top-down control signals that establish more stable neural states in visual cortex, enabling selective sensory processing.

Resourceful Event-Predictive Inference: The Nature of Cognitive Effort

Pursuing a precise, focused train of thought requires cognitive effort. Even more effort is necessary when more alternatives need to be considered or when the imagined situation becomes more complex. Cognitive resources available to us limit the cognitive effort we can spend. In line with previous work, an information-theoretic, Bayesian brain approach to cognitive effort is pursued: to solve tasks in our environment, our brain needs to invest information, that is, negative entropy, to impose structure, or focus, away from a uniform structure or other task-incompatible, latent structures. To get a more complete formalization of cognitive effort, a resourceful event-predictive inference model (REPI) is introduced, which offers computational and algorithmic explanations about the latent structure of our generative models, the active inference dynamics that unfold within, and the cognitive effort required to steer the dynamics-to, for example, purposefully process sensory signals, decide on responses, and invoke their execution. REPI suggests that we invest cognitive resources to infer preparatory priors, activate responses, and anticipate action consequences. Due to our limited resources, though, the inference dynamics are prone to task-irrelevant distractions. For example, the task-irrelevant side of the imperative stimulus causes the Simon effect and, due to similar reasons, we fail to optimally switch between tasks. An actual model implementation simulates such task interactions and offers first estimates of the involved cognitive effort. The approach may be further studied and promises to offer deeper explanations about why we get quickly exhausted from multitasking, how we are influenced by irrelevant stimulus modalities, why we exhibit magnitude interference, and, during social interactions, why we often fail to take the perspective of others into account.

Cognitive Control as a Multivariate Optimization Problem

A hallmark of adaptation in humans and other animals is our ability to control how we think and behave across different settings. Research has characterized the various forms cognitive control can take-including enhancement of goal-relevant information, suppression of goal-irrelevant information, and overall inhibition of potential responses-and has identified computations and neural circuits that underpin this multitude of control types. Studies have also identified a wide range of situations that elicit adjustments in control allocation (e.g., those eliciting signals indicating an error or increased processing conflict), but the rules governing when a given situation will give rise to a given control adjustment remain poorly understood. Significant progress has recently been made on this front by casting the allocation of control as a decision-making problem. This approach has developed unifying and normative models that prescribe when and how a change in incentives and task demands will result in changes in a given form of control. Despite their successes, these models, and the experiments that have been developed to test them, have yet to face their greatest challenge: deciding how to select among the multiplicity of configurations that control can take at any given time. Here, we will lay out the complexities of the inverse problem inherent to cognitive control allocation, and their close parallels to inverse problems within motor control (e.g., choosing between redundant limb movements). We discuss existing solutions to motor control's inverse problems drawn from optimal control theory, which have proposed that effort costs act to regularize actions and transform motor planning into a well-posed problem. These same principles may help shed light on how our brains optimize over complex control configuration, while providing a new normative perspective on the origins of mental effort.

Effect of the subjective intensity of fatigue and interoception on perceptual regulation and performance during sustained physical activity

Background

The subjective experience of fatigue impairs an individual’s ability to sustain physical endurance performance. However, precise understanding of the specific role perceived fatigue plays in the central regulation of performance remains unclear. Here, we examined whether the subjective intensity of a perceived state of fatigue, pre-induced through prior upper body activity, differentially impacted performance and altered perceived effort and affect experienced during a sustained, isometric contraction in lower body. We also explored whether (cardiac) interoception predicted the intensity of experienced perceptual and affective responses and moderated the relationships between constructs during physical activity.

Methods

Using a repeated-measures study design, thirty male participants completed three experimental conditions, with the intensity of a pre-induced state of fatigue manipulated to evoke moderate (MOD), severe (SEV) and minimal (control; CON) intensity of perceptions prior to performance of the sustained contraction.

Results

Performance of the sustained contraction was significantly impaired under a perceived state of fatigue, with reductions of 10% and 14% observed in the MOD and SEV conditions, respectively. Performance impairment was accompanied by greater perceived effort and more negative affective valence reported during the contraction. However, effects were limited to comparisons to CON, with no difference evident between the two experimental trials (i.e. MOD vs. SEV). Individuals’ awareness of their accuracy in judging resting heartbeats was shown to predict the subjective intensity of fatigue experienced during the endurance task. However, interoception did not moderate the relationships evident between fatigue and both perceived effort and affective valence.

Conclusions

A perceived state of fatigue limits endurance performance, influencing both how effortful activity is perceived to be and the affective experience of activity. Though awareness of interoceptive representations of bodily states may be important to the subjective experience of fatigue, interoception does not modulate the relationships between perceived fatigue and other perceptual (i.e. effort) and affective constructs.

Quantifying brain state transition cost via Schrödinger Bridge

Quantifying brain state transition cost is a fundamental problem in systems neuroscience. Previous studies utilized network control theory to measure the cost by considering a neural system as a deterministic dynamical system. However, this approach does not capture the stochasticity of neural systems, which is important for accurately quantifying brain state transition cost. Here, we propose a novel framework based on optimal control in stochastic systems. In our framework, we quantify the transition cost as the Kullback-Leibler divergence from an uncontrolled transition path to the optimally controlled path, which is known as Schrödinger Bridge. To test its utility, we applied this framework to functional magnetic resonance imaging data from the Human Connectome Project and computed the brain state transition cost in cognitive tasks. We demonstrate correspondence between brain state transition cost and the difficulty of tasks. The results suggest that our framework provides a general theoretical tool for investigating cognitive functions from the viewpoint of transition cost.

Advances in modeling learning and decision-making in neuroscience

An organism's survival depends on its ability to learn about its environment and to make adaptive decisions in the service of achieving the best possible outcomes in that environment. To study the neural circuits that support these functions, researchers have increasingly relied on models that formalize the computations required to carry them out. Here, we review the recent history of computational modeling of learning and decision-making, and how these models have been used to advance understanding of prefrontal cortex function. We discuss how such models have advanced from their origins in basic algorithms of updating and action selection to increasingly account for complexities in the cognitive processes required for learning and decision-making, and the representations over which they operate. We further discuss how a deeper understanding of the real-world complexities in these computations has shed light on the fundamental constraints on optimal behavior, and on the complex interactions between corticostriatal pathways to determine such behavior. The continuing and rapid development of these models holds great promise for understanding the mechanisms by which animals adapt to their environments, and what leads to maladaptive forms of learning and decision-making within clinical populations.

Emotions as Abstract Evaluation Criteria in Biological and Artificial Intelligences

Biological as well as advanced artificial intelligences (AIs) need to decide which goals to pursue. We review nature's solution to the time allocation problem, which is based on a continuously readjusted categorical weighting mechanism we experience introspectively as emotions. One observes phylogenetically that the available number of emotional states increases hand in hand with the cognitive capabilities of animals and that raising levels of intelligence entail ever larger sets of behavioral options. Our ability to experience a multitude of potentially conflicting feelings is in this view not a leftover of a more primitive heritage, but a generic mechanism for attributing values to behavioral options that can not be specified at birth. In this view, emotions are essential for understanding the mind. For concreteness, we propose and discuss a framework which mimics emotions on a functional level. Based on time allocation via emotional stationarity (TAES), emotions are implemented as abstract criteria, such as satisfaction, challenge and boredom, which serve to evaluate activities that have been carried out. The resulting timeline of experienced emotions is compared with the “character” of the agent, which is defined in terms of a preferred distribution of emotional states. The long-term goal of the agent, to align experience with character, is achieved by optimizing the frequency for selecting individual tasks. Upon optimization, the statistics of emotion experience becomes stationary.

Toward the unity of pathological and exertional fatigue: A predictive processing model

Fatigue is a common experience in both health and disease. Yet, pathological (i.e., prolonged or chronic) and transient (i.e., exertional) fatigue symptoms are traditionally considered distinct, compounding a separation between interested research fields within the study of fatigue. Within the clinical neurosciences, nascent frameworks position pathological fatigue as a product of inference derived through hierarchical predictive processing. The metacognitive theory of dyshomeostasis (Stephan et al., 2016) states that pathological fatigue emerges from the metacognitive mechanism in which the detection of persistent mismatches between prior interoceptive predictions and ascending sensory evidence (i.e., prediction error) signals low evidence for internal generative models, which undermine an agent’s feeling of mastery over the body and is thus experienced phenomenologically as fatigue. Although acute, transient subjective symptoms of exertional fatigue have also been associated with increasing interoceptive prediction error, the dynamic computations that underlie its development have not been clearly defined. Here, drawing on the metacognitive theory of dyshomeostasis, we extend this account to offer an explicit description of the development of fatigue during extended periods of (physical) exertion. Accordingly, it is proposed that a loss of certainty or confidence in control predictions in response to persistent detection of prediction error features as a common foundation for the conscious experience of both pathological and nonpathological fatigue.

Conscious awareness of motor fluidity improves performance and decreases cognitive effort in sequence learning

Motor skill learning is improved when participants are instructed to judge after each trial whether their performed movements have reached maximal fluidity. Consequently, the conscious awareness of this maximal fluidity can be classified as a genuine learning factor for motor sequences. However, it is unknown whether this effect of conscious awareness on motor learning could be mediated by the increased cognitive effort that may accompany such judgment making. The main aim of this study was to test this hypothesis in comparing two groups with, and without, the conscious awareness of the maximal fluidity. To assess the possible involvement of cognitive effort, we have recorded the pupillary dilation to the task, which is well-known to increase in proportion to cognitive effort. Results confirmed that conscious awareness indeed improved motor sequence learning of the trained sequence specifically. Pupil dilation was smaller during trained than during novel sequence performance, indicating that sequence learning decreased the cognitive cost of sequence execution. However, we found that in the group that had to judge on their maximal fluidity, pupil dilation during sequence production was smaller than in the control group, indicating that the motor improvement induced by the fluidity judgment does not involve additional cognitive effort. We discuss these results in the context of motor learning and cognitive effort theories.

The road towards understanding embodied decisions

Most current decision-making research focuses on classical economic scenarios, where choice offers are prespecified and where action dynamics play no role in the decision. However, our brains evolved to deal with different choice situations: "embodied decisions". As examples of embodied decisions, consider a lion that has to decide which gazelle to chase in the savannah or a person who has to select the next stone to jump on when crossing a river. Embodied decision settings raise novel questions, such as how people select from time-varying choice options and how they track the most relevant choice attributes; but they have long remained challenging to study empirically. Here, we summarize recent progress in the study of embodied decisions in sports analytics and experimental psychology. Furthermore, we introduce a formal methodology to identify the relevant dimensions of embodied choices (present and future affordances) and to map them into the attributes of classical economic decisions (probabilities and utilities), hence aligning them. Studying embodied decisions will greatly expand our understanding of what decision-making is.

Using the Relative Entropy of Linguistic Complexity to Assess L2 Language Proficiency Development

This study applies relative entropy in naturalistic large-scale corpus to calculate the difference among L2 (second language) learners at different levels. We chose lemma, token, POS-trigram, conjunction to represent lexicon and grammar to detect the patterns of language proficiency development among different L2 groups using relative entropy. The results show that information distribution discrimination regarding lexical and grammatical differences continues to increase from L2 learners at a lower level to those at a higher level. This result is consistent with the assumption that in the course of second language acquisition, L2 learners develop towards a more complex and diverse use of language. Meanwhile, this study uses the statistics method of time series to process the data on L2 differences yielded by traditional frequency-based methods processing the same L2 corpus to compare with the results of relative entropy. However, the results from the traditional methods rarely show regularity. As compared to the algorithms in traditional approaches, relative entropy performs much better in detecting L2 proficiency development. In this sense, we have developed an effective and practical algorithm for stably detecting and predicting the developments in L2 learners' language proficiency.

An Information-Theoretic Account of Semantic Interference in Word Production

I present a computational-level model of semantic interference effects in online word production within a rate–distortion framework. I consider a bounded-rational agent trying to produce words. The agent's action policy is determined by maximizing accuracy in production subject to computational constraints. These computational constraints are formalized using mutual information. I show that semantic similarity-based interference among words falls out naturally from this setup, and I present a series of simulations showing that the model captures some of the key empirical patterns observed in Stroop and Picture–Word Interference paradigms, including comparisons to human data from previous experiments.

A mosaic of cost-benefit control over cortico-striatal circuitry

Dopamine contributes to cognitive control through well-established effects in both the striatum and cortex. Although earlier work suggests that dopamine affects cognitive control capacity, more recent work suggests that striatal dopamine may also impact on cognitive motivation. We consider the emerging perspective that striatal dopamine boosts control by making people more sensitive to the benefits versus the costs of cognitive effort, and we discuss how this sensitivity shapes competition between controlled and prepotent actions. We propose that dopamine signaling in distinct cortico-striatal subregions mediates different types of cost-benefit tradeoffs, and also discuss mechanisms for the local control of dopamine release, enabling selectivity among cortico-striatal circuits. In so doing, we show how this cost-benefit mosaic can reconcile seemingly conflicting findings about the impact of dopamine signaling on cognitive control.

Moral decisions in the age of COVID-19: Your choices really matter

The moral decisions we make during this period, such as deciding whether to comply with quarantine rules, have unprecedented societal effects. We simulate the "escape from Milan" that occurred on March 7th-8th 2020, when many travelers moved from a high-risk zone (Milan) to southern regions of Italy (Campania and Lazio) immediately after an imminent lockdown was announced. Our simulations show that fewer than 50 active cases might have caused the sudden spread of the virus observed afterwards in these regions. The surprising influence of the actions of few individuals on societal dynamics challenges our cognitive expectations - as in normal conditions, collective dynamics are rather robust to the decisions of few "cheaters". This situation therefore requires novel educational strategies that increase our awareness and understanding of the unprecedented effects of our individual moral decisions.

Information Rate in Humans during Visuomotor Tracking

Previous investigations concluded that the human brain’s information processing rate remains fundamentally constant, irrespective of task demands. However, their conclusion rested in analyses of simple discrete-choice tasks. The present contribution recasts the question of human information rate within the context of visuomotor tasks, which provides a more ecologically relevant arena, albeit a more complex one. We argue that, while predictable aspects of inputs can be encoded virtually free of charge, real-time information transfer should be identified with the processing of surprises. We formalise this intuition by deriving from first principles a decomposition of the total information shared by inputs and outputs into a feedforward, predictive component and a feedback, error-correcting component. We find that the information measured by the feedback component, a proxy for the brain’s information processing rate, scales with the difficulty of the task at hand, in agreement with cost-benefit models of cognitive effort.

Electroencephalographic correlates of temporal Bayesian belief updating and surprise

The brain predicts the timing of forthcoming events to optimize responses to them. Temporal predictions have been formalized in terms of the hazard function, which integrates prior beliefs on the likely timing of stimulus occurrence with information conveyed by the passage of time. However, how the human brain updates prior temporal beliefs is still elusive. Here we investigated electroencephalographic (EEG) signatures associated with Bayes-optimal updating of temporal beliefs. Given that updating usually occurs in response to surprising events, we sought to disentangle EEG correlates of updating from those associated with surprise. Twenty-six participants performed a temporal foreperiod task, which comprised a subset of surprising events not eliciting updating. EEG data were analyzed through a regression-based massive approach in the electrode and source space. Distinct late positive, centro-parietally distributed, event-related potentials (ERPs) were associated with surprise and belief updating in the electrode space. While surprise modulated the commonly observed P3b, updating was associated with a later and more sustained P3b-like waveform deflection. Results from source analyses revealed that neural encoding of surprise comprises neural activity in the cingulo-opercular network (CON) and parietal regions. These data provide evidence that temporal predictions are computed in a Bayesian manner, and that this is reflected in P3 modulations, akin to other cognitive domains. Overall, our study revealed that analyzing P3 modulations provides an important window into the Bayesian brain. Data and scripts are shared on OSF: https://osf.io/ckqa5/.

Parsing Neurodynamic Information Streams to Estimate the Frequency, Magnitude and Duration of Team Uncertainty

Neurodynamic organizations are information-based abstractions, expressed in bits, of the structure of long duration EEG amplitude levels. Neurodynamic information (NI, the variable of neurodynamic organization) is thought to continually accumulate as EEG amplitudes cycle through periods of persistent activation and deactivation in response to the activities and uncertainties of teamwork. Here we show that (1) Neurodynamic information levels were a better predictor of uncertainty and novice and expert behaviors than were the EEG power levels from which NI was derived. (2) Spatial and temporal parsing of team NI from experienced submarine navigation and healthcare teams showed that it was composed of discrete peaks with durations up to 20–60 s, and identified the involvement of activated delta waves when precise motor control was needed. (3) The relationship between NI and EEG power was complex varying by brain regions, EEG frequencies, and global vs. local brain interactions. The presence of an organizational system of information that parallels the amplitude of EEG rhythms is important as it provides a greatly reduced data dimension while retaining the essential system features, i.e., linkages to higher scale behaviors that span temporal and spatial scales of teamwork. In this way the combinatorial explosion of EEG rhythmic variables at micro levels become compressed into an intermediate system of information and organization which links to macro-scale team and team member behaviors. These studies provide an avenue for understanding how complex organizations arise from the dynamics of underlying micro-scale variables. The study also has practical implications for how micro-scale variables might be better represented, both conceptually and in terms of parsimony, for training machines to recognize human behaviors that span scales of teams.

Human complex exploration strategies are enriched by noradrenaline-modulated heuristics

An exploration-exploitation trade-off, the arbitration between sampling a lesser-known against a known rich option, is thought to be solved using computationally demanding exploration algorithms. Given known limitations in human cognitive resources, we hypothesised the presence of additional cheaper strategies. We examined for such heuristics in choice behaviour where we show this involves a value-free random exploration, that ignores all prior knowledge, and a novelty exploration that targets novel options alone. In a double-blind, placebo-controlled drug study, assessing contributions of dopamine (400 mg amisulpride) and noradrenaline (40 mg propranolol), we show that value-free random exploration is attenuated under the influence of propranolol, but not under amisulpride. Our findings demonstrate that humans deploy distinct computationally cheap exploration strategies and that value-free random exploration is under noradrenergic control.

Activation of the cognitive control network associated with information uncertainty

The cognitive control network (CCN) that comprises regions of the frontoparietal network, the cingulo-opercular network, and other sub-cortical regions as core structures is commonly activated by events with an increase in information uncertainty. However, it is not clear whether this CCN activation is associated with both information entropy that represents the information conveyed by the context formed by a sequence of events and the surprise that quantifies the information conveyed by a specific type of event in the context. We manipulated entropy and surprise in this functional magnetic resonance imaging study by varying the probability of occurrence of two types of events in both the visual and auditory modalities and measured brain response as a function of entropy and surprise. We found that activation in regions of the CCN increased as a function of entropy and surprise in both the visual and auditory tasks. The frontoparietal network and additional structures in the CCN mediated the relationship between these information measures and behavioral response. These results suggest that the CCN is a high-level modality-general neural entity for the control of the processing of information conveyed by both context and event.

Fast fronto-parietal cortical dynamics of conflict detection and context updating in a flanker task

Recent research has found that the traditional target P3 consists of a family of P3-like positivities that can be functionally and topographically dissociated from one another. The current study examined target N2 and P3-like subcomponents indexing conflict detection and context updating at low- and high-order levels in the neural hierarchy during cognitive control. Electroencephalographic signals were recorded from 45 young adults while they completed a hybrid go/nogo flanker task, and Residue Iteration Decomposition (RIDE) was applied to functionally dissociate these peaks. Analyses showed a stimulus-locked frontal N2 revealing early detection and fast perceptual categorization of nogo, congruent and incongruent trials, resulting in frontal P3-like activity elicited by nogo trials in the latency-variable RIDE cluster, and by incongruent trials in the response-locked cluster. The congruent trials did not elicit frontal P3-like activity. These findings suggest that behavioral incongruency effects are related to intermediate and later stages of motor response re-programming.

Beginning with biology: “Aspects of cognition” exist in the service of the brain's overall function as a resource-regulator

Lieder and Griffiths rightly urge that computational cognitive models be constrained by resource usage, but they should go further. The brain's primary function is to regulate resource usage. As a consequence, resource usage should not simply select among algorithmic models of “aspects of cognition.” Rather, “aspects of cognition” should be understood as existing in the service of resource management.

Fearful faces modulate cognitive control under varying levels of uncertainty: An event-related potential study

Cognitive control can reduce uncertainty, but few studies have investigated temporal dynamics of the flexible allocation of resources under varying levels of uncertainty. We used a revised majority function task with emotional faces and event-related potentials to investigate this process. The task incorporated different ratios of face orientation to quantify uncertainty. Participants performed slower in high uncertainty than in other levels. Under low uncertainty, participants showed greater amplitudes of frontal N200 and late frontal wave to neutral faces than fearful faces. Parietal P300 amplitudes decreased from low uncertainty to high uncertainty, and fearful faces elicited greater P300 amplitudes than neutral faces under all levels of uncertainty. These results suggest that emotion and uncertainty interacted in the frontal cortex during both early and late stages, while no interaction existed in the parietal cortex during the late stage. The interference of fearful faces is lessened by increasing cognitive control under high uncertainty in the frontal cortex, suggesting that humans possess the ability to flexibly allocate mental resources in the temporal domain. Our findings provide evidence to support the fronto-parietal network hypothesis of cognitive control in a novel perspective of uncertainty.

Driven by Pain, Not Gain: Computational Approaches to Aversion-Related Decision Making in Psychiatry

Although it is well known that "losses loom larger than gains," computational approaches to aversion-related decision making (ARDM) for psychiatric disorders is an underdeveloped area. Computational models of ARDM have been implemented primarily as state-dependent reinforcement learning models with bias parameters to quantify Pavlovian associations, and differential learning rates to quantify instrumental updating have been shown to depend on context, involve complex cost calculations, and include the consideration of counterfactual outcomes. Little is known about how individual differences influence these models relevant to anxiety-related conditions or addiction-related dysfunction. It is argued that model parameters reflecting 1) Pavlovian biases in the context of reinforcement learning or 2) hyperprecise prior beliefs in the context of active inference play an important role in the emergence of dysfunctional avoidance behaviors. The neural implementation of ARDM includes brain areas that are important for valuation (ventromedial prefrontal cortex) and positive reinforcement-related prediction errors (ventral striatum), but also aversive processing (insular cortex and cerebellum). Computational models of ARDM will help to establish a quantitative explanatory account of the development of anxiety disorders and addiction, but such models also face several challenges, including limited evidence for stability of individual differences, relatively low reliability of tasks, and disorder heterogeneity. Thus, it will be necessary to develop robust, reliable, and model-based experimental probes; recruit larger sample sizes; and use single case experimental designs for better pragmatic and explanatory biological models of psychiatric disorders.

The sense of should: A biologically-based framework for modeling social pressure

What is social pressure, and how could it be adaptive to conform to others' expectations? Existing accounts highlight the importance of reputation and social sanctions. Yet, conformist behavior is multiply determined: sometimes, a person desires social regard, but at other times she feels obligated to behave a certain way, regardless of any reputational benefit-i.e. she feels a sense of should. We develop a formal model of this sense of should, beginning from a minimal set of biological premises: that the brain is predictive, that prediction error has a metabolic cost, and that metabolic costs are prospectively avoided. It follows that unpredictable environments impose metabolic costs, and in social environments these costs can be reduced by conforming to others' expectations. We elaborate on a sense of should's benefits and subjective experience, its likely developmental trajectory, and its relation to embodied mental inference. From this individualistic metabolic strategy, the emergent dynamics unify social phenomenon ranging from status quo biases, to communication and motivated cognition. We offer new solutions to long-studied problems (e.g. altruistic behavior), and show how compliance with arbitrary social practices is compelled without explicit sanctions. Social pressure may provide a foundation in individuals on which societies can be built.

Semantic associations between arithmetic and space: Evidence from temporal order judgements

Spatial biases associated with subtraction or addition problem solving are generally considered as reflecting leftward or rightward attention shifts along a mental numerical continuum, but an alternative hypothesis not implying spatial attention proposes that the operator (plus or minus sign) may favour a response to one side of space (left or right) because of semantic associations. We tested these two accounts in a series of temporal order judgement experiments that consisted in the auditory presentation of addition or subtraction problems followed 200 ms (Experiments 1-2) or 800 ms (Experiment 3) later by the display of two lateralized targets in close temporal succession. To dissociate the side where the operation first brought their attention from the side they had to respond to, we asked participants to report which of the left or right target appeared first or last on screen. Under the attention-orienting account, addition should elicit more rightward responses than subtraction when participants have to focus on the first target, but more leftward responses when they have to focus on the last target, because the latter is opposite to the side where the operation first brought their attention. Under the semantic account, addition should elicit more rightward responses than subtraction, no matter the focus is on the first or last target, because participants should systematically favour the side conceptually linked to the operator. The results of the three experiments converge to indicate that, in lateralized target detection tasks, the spatial biases induced by arithmetic operations stem from semantic associations.

Eye pupil signals information gain

In conditions of constant illumination, the eye pupil diameter indexes the modulation of arousal state and responds to a large breadth of cognitive processes, including mental effort, attention, surprise, decision processes, decision biases, value beliefs, uncertainty, volatility, exploitation/exploration trade-off, or learning rate. Here, I propose an information theoretic framework that has the potential to explain the ensemble of these findings as reflecting pupillary response to information processing. In short, updates of the brain’s internal model, quantified formally as the Kullback–Leibler (KL) divergence between prior and posterior beliefs, would be the common denominator to all these instances of pupillary dilation to cognition. I show that stimulus presentation leads to pupillary response that is proportional to the amount of information the stimulus carries about itself and to the quantity of information it provides about other task variables. In the context of decision making, pupil dilation in relation to uncertainty is explained by the wandering of the evidence accumulation process, leading to large summed KL divergences. Finally, pupillary response to mental effort and variations in tonic pupil size are also formalized in terms of information theory. On the basis of this framework, I compare pupillary data from past studies to simple information-theoretic simulations of task designs and show good correspondance with data across studies. The present framework has the potential to unify the large set of results reported on pupillary dilation to cognition and to provide a theory to guide future research.

Bayesian modeling of temporal expectations in the human brain

The brain predicts the timing of forthcoming events to optimize processes in response to them. Temporal predictions are driven by both our prior expectations on the likely timing of stimulus occurrence and the information conveyed by the passage of time. Specifically, such predictions can be described in terms of the hazard function, that is, the conditional probability that an event will occur, given it has not yet occurred. Events violating expectations cause surprise and often induce updating of prior expectations. While it is well-known that the brain is able to track the temporal hazard of event occurrence, the question of how prior temporal expectations are updated is still unsettled. Here we combined a Bayesian computational approach with brain imaging to map updating of temporal expectations in the human brain. Moreover, since updating is usually highly correlated with surprise, participants performed a task that allowed partially differentiating between the two processes. Results showed that updating and surprise differently modulated activity in areas belonging to two critical networks for cognitive control, the fronto-parietal (FPN) and the cingulo-opercular network (CON). Overall, these data provide a first computational characterization of the neural correlates associated with updating and surprise related to temporal expectation.