How to set the switches on this thing

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or "cache" values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Testing hypotheses about the harm that capitalism causes to the mind and brain: a theoretical framework for neuroscience research

In this paper, we will attempt to outline the key ideas of a theoretical framework for neuroscience research that reflects critically on the neoliberal capitalist context. We argue that neuroscience can and should illuminate the effects of neoliberal capitalism on the brains and minds of the population living under such socioeconomic systems. Firstly, we review the available empirical research indicating that the socio-economic environment is harmful to minds and brains. We, then, describe the effects of the capitalist context on neuroscience itself by presenting how it has been influenced historically. In order to set out a theoretical framework that can generate neuroscientific hypotheses with regards to the effects of the capitalist context on brains and minds, we suggest a categorization of the effects, namely deprivation, isolation and intersectional effects. We also argue in favor of a neurodiversity perspective [as opposed to the dominant model of conceptualizing neural (mal-)functioning] and for a perspective that takes into account brain plasticity and potential for change and adaptation. Lastly, we discuss the specific needs for future research as well as a frame for post-capitalist research.

Conscious cognitive effort in cognitive control

Cognitive effort is thought to be familiar in everyday life, ubiquitous across multiple variations of task and circumstance, and integral to cost/benefit computations that are themselves central to the proper functioning of cognitive control. In particular, cognitive effort is thought to be closely related to the assessment of cognitive control's costs. I argue here that the construct of cognitive effort, as it is deployed in cognitive psychology and neuroscience, is problematically unclear. The result is that talk of cognitive effort may paper over significant disagreement regarding the nature of cognitive effort, and its key functions for cognitive control. I highlight key points of disagreement, and several open questions regarding what causes cognitive effort, what cognitive effort represents, cognitive effort's relationship to action, and cognitive effort's relationship to consciousness. I also suggest that pluralism about cognitive effort-that cognitive effort may manifest as a range of intentional or non-intentional actions the function of which is to promote greater success at paradigmatic cognitive control tasks-may be a fruitful and irenic way to conceive of cognitive effort. Finally, I suggest that recent trends in work on cognitive control suggests that we might fruitfully conceive of cognitive effort as one key node in a complex network of mental value, and that studying this complex network may illuminate the nature of cognitive control, and the role of consciousness in cognitive control's proper functioning. This article is categorized under: Philosophy > Consciousness Philosophy > Psychological Capacities Neuroscience > Cognition.

From perception to behavior: The neural circuits underlying prey hunting in larval zebrafish

A key challenge for neural systems is to extract relevant information from the environment and make appropriate behavioral responses. The larval zebrafish offers an exciting opportunity for studying these sensing processes and sensory-motor transformations. Prey hunting is an instinctual behavior of zebrafish that requires the brain to extract and combine different attributes of the sensory input and form appropriate motor outputs. Due to its small size and transparency the larval zebrafish brain allows optical recording of whole-brain activity to reveal the neural mechanisms involved in prey hunting and capture. In this review we discuss how the larval zebrafish brain processes visual information to identify and locate prey, the neural circuits governing the generation of motor commands in response to prey, how hunting behavior can be modulated by internal states and experience, and some outstanding questions for the field.

Phantom controllers: Misspecified models create the false appearance of adaptive control during value-based choice

Decision scientists have grown increasingly interested in how people adaptively control their decision making. Researchers have demonstrated that parameters governing the accumulation of evidence towards a choice, such as the decision threshold, are shaped by information available prior to or in parallel with one's evaluation of an option set (e.g., recent outcomes or choice conflict). A recent account has taken a bold leap forward in this approach, suggesting that adjustments in decision parameters can be motivated by the value of the options under consideration. This motivated control account predicts that when faced with difficult choices (similarly valued options) under time pressure, people will adaptively lower their decision threshold to ensure that they make a choice in time. This account was supported by drift diffusion modeling of a deadlined choice task, demonstrating that decision thresholds decrease for difficult relative to easy choices. Here, we reanalyze the data from this experiment, and show that evidence for this novel account does not hold up to further scrutiny. Using a more systematic and comprehensive modeling approach, we show that this previously observed threshold adjustment disappears (or even reverses) under a more complete model of the data. Importantly, we further show how this and other apparent evidence for motivated control arises as an artifact of model (mis)specification, where one model's putatively controlled decision process (e.g., value-driven threshold adjustments) can mimic another model's stimulus-driven decision processes (e.g., accumulator competition or collapsing bounds). Collectively, this work reveals crucial insights and constraints in the pursuit of understanding how control guides decision-making, and when it doesn't.

Adaptive control of synaptic plasticity integrates micro- and macroscopic network function

Synaptic plasticity configures interactions between neurons and is therefore likely to be a primary driver of behavioral learning and development. How this microscopic-macroscopic interaction occurs is poorly understood, as researchers frequently examine models within particular ranges of abstraction and scale. Computational neuroscience and machine learning models offer theoretically powerful analyses of plasticity in neural networks, but results are often siloed and only coarsely linked to biology. In this review, we examine connections between these areas, asking how network computations change as a function of diverse features of plasticity and vice versa. We review how plasticity can be controlled at synapses by calcium dynamics and neuromodulatory signals, the manifestation of these changes in networks, and their impacts in specialized circuits. We conclude that metaplasticity-defined broadly as the adaptive control of plasticity-forges connections across scales by governing what groups of synapses can and can't learn about, when, and to what ends. The metaplasticity we discuss acts by co-opting Hebbian mechanisms, shifting network properties, and routing activity within and across brain systems. Asking how these operations can go awry should also be useful for understanding pathology, which we address in the context of autism, schizophrenia and Parkinson's disease.

A lineage explanation of human normative guidance: the coadaptive model of instrumental rationality and shared intentionality

This paper aims to contribute to the existing literature on normative cognition by providing a lineage explanation of human social norm psychology. This approach builds upon theories of goal-directed behavioral control in the reinforcement learning and control literature, arguing that this form of control defines an important class of intentional normative mental states that are instrumental in nature. I defend the view that great ape capacities for instrumental reasoning and our capacity (or family of capacities) for shared intentionality coadapted to each other and argue that the evolution of this capacity has allowed the representation of social norms and the emergence of our capacity for normative guidance.

Vigilance, arousal, and acetylcholine: Optimal control of attention in a simple detection task

Paying attention to particular aspects of the world or being more vigilant in general can be interpreted as forms of ‘internal’ action. Such arousal-related choices come with the benefit of increasing the quality and situational appropriateness of information acquisition and processing, but incur potentially expensive energetic and opportunity costs. One implementational route for these choices is widespread ascending neuromodulation, including by acetylcholine (ACh). The key computational question that elective attention poses for sensory processing is when it is worthwhile paying these costs, and this includes consideration of whether sufficient information has yet been collected to justify the higher signal-to-noise ratio afforded by greater attention and, particularly if a change in attentional state is more expensive than its maintenance, when states of heightened attention ought to persist. We offer a partially observable Markov decision-process treatment of optional attention in a detection task, and use it to provide a qualitative model of the results of studies using modern techniques to measure and manipulate ACh in rodents performing a similar task.

Paying attention to a stimulus is costly, both in terms of energy and the lost opportunity to pay attention to something else. It is also beneficial, providing more information about its target. Thus, whether and when we pay more or less attention may best be considered as a choice of internal action that responds to this trade-off. Furthermore, measurements and manipulation of the neuromodulator acetylcholine have suggested that it is one of the instruments of attention, providing us with a window onto this choice. Here, we build an abstract model of a task in which an animal must look out for a brief visual stimulus that may or may not occur on each trial. We show that optimal attentional choices in the model depend on many factors, including how likely a signal is to occur across time, the balance between the improvement in information possible by paying greater attention and its increased cost, and whether there are also costs associated with switching between different attentional states. We also show that our model can qualitatively match results from experiments involving acetylcholine.

How Imagination and Memory Shape the Moral Mind

Interdisciplinary research has proposed a multifaceted view of human cognition and morality, establishing that inputs from multiple cognitive and affective processes guide moral decisions. However, extant work on moral cognition has largely overlooked the contributions of episodic representation. The ability to remember or imagine a specific moment in time plays a broadly influential role in cognition and behavior. Yet, existing research has only begun exploring the influence of episodic representation on moral cognition. Here, we evaluate the theoretical connections between episodic representation and moral cognition, review emerging empirical work revealing how episodic representation affects moral decision-making, and conclude by highlighting gaps in the literature and open questions. We argue that a comprehensive model of moral cognition will require including the episodic memory system, further delineating its direct influence on moral thought, and better understanding its interactions with other mental processes to fundamentally shape our sense of right and wrong.

Multistability, perceptual value, and internal foraging

Substantial experimental, theoretical, and computational insights into sensory processing have been derived from the phenomena of perceptual multistability-when two or more percepts alternate or switch in response to a single sensory input. Here, we review a range of findings suggesting that alternations can be seen as internal choices by the brain responding to values. We discuss how elements of external, experimenter-controlled values and internal, uncertainty- and aesthetics-dependent values influence multistability. We then consider the implications for the involvement in switching of regions, such as the anterior cingulate cortex, which are more conventionally tied to value-dependent operations such as cognitive control and foraging.

Freezing revisited: coordinated autonomic and central optimization of threat coping

Animals have sophisticated mechanisms for coping with danger. Freezing is a unique state that, upon threat detection, allows evidence to be gathered, response possibilities to be previsioned and preparations to be made for worst-case fight or flight. We propose that - rather than reflecting a passive fear state - the particular somatic and cognitive characteristics of freezing help to conceal overt responses, while optimizing sensory processing and action preparation. Critical for these functions are the neurotransmitters noradrenaline and acetylcholine, which modulate neural information processing and also control the sympathetic and parasympathetic branches of the autonomic nervous system. However, the interactions between autonomic systems and the brain during freezing, and the way in which they jointly coordinate responses, remain incompletely explored. We review the joint actions of these systems and offer a novel computational framework to describe their temporally harmonized integration. This reconceptualization of freezing has implications for its role in decision-making under threat and for psychopathology.

Neurofeedback through the lens of reinforcement learning

Despite decades of experimental and clinical practice, the neuropsychological mechanisms underlying neurofeedback (NF) training remain obscure. NF is a unique form of reinforcement learning (RL) task, during which participants are provided with rewarding feedback regarding desired changes in neural patterns. However, key RL considerations - including choices during practice, prediction errors, credit-assignment problems, or the exploration-exploitation tradeoff - have infrequently been considered in the context of NF. We offer an RL-based framework for NF, describing different internal states, actions, and rewards in common NF protocols, thus fashioning new proposals for characterizing, predicting, and hastening the course of learning. In this way we hope to advance current understanding of neural regulation via NF, and ultimately to promote its effectiveness, personalization, and clinical utility.

Filling the gaps: Cognitive control as a critical lens for understanding mechanisms of value-based decision-making

While often seeming to investigate rather different problems, research into value-based decision making and cognitive control have historically offered parallel insights into how people select thoughts and actions. While the former studies how people weigh costs and benefits to make a decision, the latter studies how they adjust information processing to achieve their goals. Recent work has highlighted ways in which decision-making research can inform our understanding of cognitive control. Here, we provide the complementary perspective: how cognitive control research has informed understanding of decision-making. We highlight three particular areas of research where this critical interchange has occurred: (1) how different types of goals shape the evaluation of choice options, (2) how people use control to adjust the ways they make their decisions, and (3) how people monitor decisions to inform adjustments to control at multiple levels and timescales. We show how adopting this alternate viewpoint offers new insight into the determinants of both decisions and control; provides alternative interpretations for common neuroeconomic findings; and generates fruitful directions for future research.

When will's wont wants wanting

We use neural reinforcement learning concepts including Pavlovian versus instrumental control, liking versus wanting, model-based versus model-free control, online versus offline learning and planning, and internal versus external actions and control to reflect on putative conflicts between short-term temptations and long-term goals.

Abnormal negative feedback processing in individuals with autistic traits in the Iowa gambling task: Evidence from behavior and event-related potentials

Value-based decision making plays an important role in social interaction. Previous studies have reported that individuals with autism spectrum disorder (ASD) exhibit deficits in terms of decision making. However, it is still unknown clearly whether individuals with high autistic traits within nonclinical populations employ abnormal neural substrates in value-based decision-making. To explore this issue, we investigated value-based decision making and its neural substrates in individuals with high and low autistic traits within a typically developing population who completed the revised Iowa gambling task (IGT) based on measurements of event-related potentials (ERPs). The IGT net scores were significantly lower in the group with high autistic traits than the group with low autistic traits in the fifth and sixth blocks. The ERP results showed that the feedback-related negativity (FRN) amplitude in individuals with high autistic traits allowed slight discrimination between positive and negative feedback in the low-risk option. The event-related spectral perturbations (ERSPs) and inter-trial coherence (ITC) of the theta-band frequency were also lower in the group with high autistic traits than the group with low autistic traits in the loss low-risk option. The results obtained in this study indicate that individuals with high autistic traits exhibit an unusual negative feedback process and relevant neural substrate. The FRN amplitude and theta-band oscillation may comprise a neural index of abnormal decision-making processes in individuals with high autistic traits. This study of a small sample may be considered an important step toward a more comprehensive understanding of the autism "spectrum" within a nonclinical population based on cognitive neuroscience.

Combined model-free and model-sensitive reinforcement learning in non-human primates

Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates.

We routinely solve planning problems in which present decisions have consequences in the future. These pose complex computational and statistical problems and are addressed by multiple systems in the brain which use different solutions to these problems, and which may compete and cooperate. We trained two rhesus monkeys on a paradigmatic planning task that transparently reveals canonical aspects of different strategies. We performed a detailed behavioral analysis using methods of reinforcement learning on choice and reaction time to reveal conjoint influences and structural interactions of different sources of information. We show the strengths and limitations of these analyses, at the same time as we provide a novel perspective on how different learning systems interact for choice in non-human primates.

Animal affect and decision-making

The scientific study of animal affect (emotion) is an area of growing interest. Whilst research on mechanism and causation has predominated, the study of function is less advanced. This is not due to a lack of hypotheses; in both humans and animals, affective states are frequently proposed to play a pivotal role in coordinating adaptive responses and decisions. However, exactly how they might do this (what processes might implement this function) is often left rather vague. Here we propose a framework for integrating animal affect and decision-making that is couched in modern decision theory and employs an operational definition that aligns with dimensional concepts of core affect and renders animal affect empirically tractable. We develop a model of how core affect, including short-term (emotion-like) and longer-term (mood-like) states, influence decision-making via processes that we label affective options, affective predictions, and affective outcomes and which correspond to similar concepts in schema of the links between human emotion and decision-making. Our framework is generalisable across species and generates questions for future research.

A Computational Model of Attention Control in Multi-Attribute, Context-Dependent Decision Making

Real-life decisions often require a comparison of multi-attribute options with various benefits and costs, and the evaluation of each option depends partly on the others in the choice set (i.e., the choice context). Although reinforcement learning models have successfully described choice behavior, how to account for multi-attribute information when making a context-dependent decision remains unclear. Here we develop a computational model of attention control that includes context effects on multi-attribute decisions, linking a context-dependent choice model with a reinforcement learning model. The overall model suggests that the distinctiveness of attributes guides an individual's preferences among multi-attribute options via an attention-control mechanism that determines whether choices are selectively biased toward the most distinctive attribute (selective attention) or proportionally distributed based on the relative distinctiveness of attributes (divided attention). To test the model, we conducted a behavioral experiment in rhesus monkeys, in which they made simple multi-attribute decisions over three conditions that manipulated the degree of distinctiveness between alternatives: (1) four foods of different size and calorie; (2) four pieces of the same food in different colors; and (3) four identical pieces of food. The model simulation of the choice behavior captured the preference bias (i.e., overall preference structure) and the choice persistence (repeated choices) in the empirical data, providing evidence for the respective influences of attention and memory on preference bias and choice persistence. Our study provides insights into computations underlying multi-attribute decisions, linking attentional control to decision-making processes.

Reduced engagement of the anterior cingulate cortex in the dishonest decision-making of incarcerated psychopaths

A large body of research indicates that psychopathic individuals lie chronically and show little remorse or anxiety. Yet, little is known about the neurobiological substrates of dishonesty in psychopathy. In a sample of incarcerated individuals (n = 67), we tested the hypothesis that psychopathic individuals show reduced activity in the anterior cingulate cortex (ACC) when confronted with an opportunity for dishonest gain, reflecting dishonest behavior that is relatively unhindered by response conflict. During functional magnetic resonance imaging, incarcerated offenders with different levels of psychopathy performed an incentivized prediction task wherein they were given real and repeated opportunities for dishonest gain. We found that while incarcerated offenders showed a high rate of cheating, levels of psychopathic traits did not influence the frequency of dishonesty. Higher psychopathy scores predicted decreased activity in the ACC during dishonest decision-making. Further analysis revealed that the ACC was functionally connected to the dorsolateral prefrontal cortex, and that ACC activity mediated the relationship between psychopathic traits and reduced reaction times for dishonest behavior. These findings suggest that psychopathic individuals behave dishonestly with relatively low levels of response conflict and that the ACC may play a critical role in this pattern of behavior.

Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood

Pavlovian influences are important in guiding decision-making across health and psychopathology. There is an increasing interest in using concise computational tasks to parametrise such influences in large populations, and especially to track their evolution during development and changes in mental health. However, the developmental course of Pavlovian influences is uncertain, a problem compounded by the unclear psychometric properties of the relevant measurements. We assessed Pavlovian influences in a longitudinal sample using a well characterised and widely used Go-NoGo task. We hypothesized that the strength of Pavlovian influences and other 'psychomarkers' guiding decision-making would behave like traits. As reliance on Pavlovian influence is not as profitable as precise instrumental decision-making in this Go-NoGo task, we expected this influence to decrease with higher IQ and age. Additionally, we hypothesized it would correlate with expressions of psychopathology. We found that Pavlovian effects had weak temporal stability, while model-fit was more stable. In terms of external validity, Pavlovian effects decreased with increasing IQ and experience within the task, in line with normative expectations. However, Pavlovian effects were poorly correlated with age or psychopathology. Thus, although this computational construct did correlate with important aspects of development, it does not meet conventional requirements for tracking individual development. We suggest measures that might improve psychometric properties of task-derived Pavlovian measures for future studies.

Rational metareasoning and the plasticity of cognitive control

The human brain has the impressive capacity to adapt how it processes information to high-level goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we develop and evaluate a model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert. We derive this model from a general theory according to which the function of cognitive control is to select and configure neural pathways so as to make optimal use of finite time and limited computational resources. The central idea of our Learned Value of Control model is that people use reinforcement learning to predict the value of candidate control signals of different types and intensities based on stimulus features. This model correctly predicts the learning and transfer effects underlying the adaptive control-demanding behavior observed in an experiment on visual attention and four experiments on interference control in Stroop and Flanker paradigms. Moreover, our model explained these findings significantly better than an associative learning model and a Win-Stay Lose-Shift model. Our findings elucidate how learning and experience might shape people’s ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure.

The human brain has the impressive ability to adapt how it processes information to high level goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we derive a computational model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert from a formal theory of the function of cognitive control. Across five experiments, we find that our model correctly predicts that people learn to adaptively regulate their attention and decision-making and how these learning effects transfer to novel situations. Our findings elucidate how learning and experience might shape people’s ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure.

Heuristic and optimal policy computations in the human brain during sequential decision-making

Optimal decisions across extended time horizons require value calculations over multiple probabilistic future states. Humans may circumvent such complex computations by resorting to easy-to-compute heuristics that approximate optimal solutions. To probe the potential interplay between heuristic and optimal computations, we develop a novel sequential decision-making task, framed as virtual foraging in which participants have to avoid virtual starvation. Rewards depend only on final outcomes over five-trial blocks, necessitating planning over five sequential decisions and probabilistic outcomes. Here, we report model comparisons demonstrating that participants primarily rely on the best available heuristic but also use the normatively optimal policy. FMRI signals in medial prefrontal cortex (MPFC) relate to heuristic and optimal policies and associated choice uncertainties. Crucially, reaction times and dorsal MPFC activity scale with discrepancies between heuristic and optimal policies. Thus, sequential decision-making in humans may emerge from integration between heuristic and optimal policies, implemented by controllers in MPFC.

Alhough humans often make a series of related decisions, it is unknown whether this is done by relying on optimal or heuristic strategies. Here, the authors show that humans rely on both the best heuristic and the optimal policy, and that these strategies are controlled by parts of the medial prefrontal cortex.

Nested positive feedback loops in the maintenance of major depression: An integration and extension of previous models

Several theories of Major Depressive Disorder (MDD) have previously been proposed, focusing largely on either a psychological (i.e., cognitive/affective), biological, or neural/computational level of description. These theories appeal to somewhat distinct bodies of work that have each highlighted separate factors as being of considerable potential importance to the maintenance of MDD. Such factors include a range of cognitive/attentional information-processing biases, a range of structural and functional brain abnormalities, and also dysregulation within the autonomic, endocrine, and immune systems. However, to date there have been limited efforts to integrate these complimentary perspectives into a single multi-level framework. Here we review previous work in each of these MDD research domains and illustrate how they can be synthesized into a more comprehensive model of how a depressive episode is maintained. In particular, we emphasize how plausible (but insufficiently studied) interactions between the various MDD-related factors listed above can lead to a series of nested positive feedback loops, which are each capable of maintaining an individual in a depressive episode. We also describe how these different feedback loops could be active to different degrees in different individual cases, potentially accounting for heterogeneity in both depressive symptoms and treatment response. We conclude by discussing how this integrative model might extend understanding of current treatment mechanisms, and also potentially guide the search for markers to inform treatment selection in individual cases.

A Network Neuroscience of Human Learning: Potential to Inform Quantitative Theories of Brain and Behavior

Humans adapt their behavior to their external environment in a process often facilitated by learning. Efforts to describe learning empirically can be complemented by quantitative theories that map changes in neurophysiology to changes in behavior. In this review we highlight recent advances in network science that offer a sets of tools and a general perspective that may be particularly useful in understanding types of learning that are supported by distributed neural circuits. We describe recent applications of these tools to neuroimaging data that provide unique insights into adaptive neural processes, the attainment of knowledge, and the acquisition of new skills, forming a network neuroscience of human learning. While promising, the tools have yet to be linked to the well-formulated models of behavior that are commonly utilized in cognitive psychology. We argue that continued progress will require the explicit marriage of network approaches to neuroimaging data and quantitative models of behavior.

Algorithms for survival: a comparative perspective on emotions

The nature and neural implementation of emotions is the subject of vigorous debate. Here, we use Bayesian decision theory to address key complexities in this field and conceptualize emotions in terms of their relationship to survival-relevant behavioural choices. Decision theory indicates which behaviours are optimal in a given situation; however, the calculations required are radically intractable. We therefore conjecture that the brain uses a range of pre-programmed algorithms that provide approximate solutions. These solutions seem to produce specific behavioural manifestations of emotions and can also be associated with core affective dimensions. We identify principles according to which these algorithms are implemented in the brain and illustrate our approach by considering decision making in the face of proximal threat.

The hierarchical basis of neurovisceral integration

The neurovisceral integration (NVI) model was originally proposed to account for observed relationships between peripheral physiology, cognitive performance, and emotional/physical health. This model has also garnered a considerable amount of empirical support, largely from studies examining cardiac vagal control. However, recent advances in functional neuroanatomy, and in computational neuroscience, have yet to be incorporated into the NVI model. Here we present an updated/expanded version of the NVI model that incorporates these advances. Based on a review of studies of structural/functional anatomy, we first describe an eight-level hierarchy of nervous system structures, and the contribution that each level plausibly makes to vagal control. Second, we review recent work on a class of computational models of brain function known as "predictive coding" models. We illustrate how the computational dynamics of these models, when implemented within our proposed vagal control hierarchy, can increase understanding of the relationship between vagal control and both cognitive performance and emotional/physical health. We conclude by discussing novel implications of this updated NVI model for future research.

The Neurocognitive Cost of Enhancing Cognition with Methylphenidate: Improved Distractor Resistance but Impaired Updating

A balance has to be struck between supporting distractor-resistant representations in working memory and allowing those representations to be updated. Catecholamine, particularly dopamine, transmission has been proposed to modulate the balance between the stability and flexibility of working memory representations. However, it is unclear whether drugs that increase catecholamine transmission, such as methylphenidate, optimize this balance in a task-dependent manner or bias the system toward stability at the expense of flexibility (or vice versa). Here we demonstrate, using pharmacological fMRI, that methylphenidate improves the ability to resist distraction (cognitive stability) but impairs the ability to flexibly update items currently held in working memory (cognitive flexibility). These behavioral effects were accompanied by task-general effects in the striatum and opposite and task-specific effects on neural signal in the pFC. This suggests that methylphenidate exerts its cognitive enhancing and impairing effects through acting on the pFC, an effect likely associated with methylphenidate's action on the striatum. These findings highlight that methylphenidate acts as a double-edged sword, improving one cognitive function at the expense of another, while also elucidating the neurocognitive mechanisms underlying these paradoxical effects.

Adaptive integration of habits into depth-limited planning defines a habitual-goal-directed spectrum

Behavioral and neural evidence reveal a prospective goal-directed decision process that relies on mental simulation of the environment, and a retrospective habitual process that caches returns previously garnered from available choices. Artificial systems combine the two by simulating the environment up to some depth and then exploiting habitual values as proxies for consequences that may arise in the further future. Using a three-step task, we provide evidence that human subjects use such a normative plan-until-habit strategy, implying a spectrum of approaches that interpolates between habitual and goal-directed responding. We found that increasing time pressure led to shallower goal-directed planning, suggesting that a speed-accuracy tradeoff controls the depth of planning with deeper search leading to more accurate evaluation, at the cost of slower decision-making. We conclude that subjects integrate habit-based cached values directly into goal-directed evaluations in a normative manner.

Moral Learning: Conceptual foundations and normative relevance

What is distinctive about a bringing a learning perspective to moral psychology? Part of the answer lies in the remarkable transformations that have taken place in learning theory over the past two decades, which have revealed how powerful experience-based learning can be in the acquisition of abstract causal and evaluative representations, including generative models capable of attuning perception, cognition, affect, and action to the physical and social environment. When conjoined with developments in neuroscience, these advances in learning theory permit a rethinking of fundamental questions about the acquisition of moral understanding and its role in the guidance of behavior. For example, recent research indicates that spatial learning and navigation involve the formation of non-perspectival as well as ego-centric models of the physical environment, and that spatial representations are combined with learned information about risk and reward to guide choice and potentiate further learning. Research on infants provides evidence that they form non-perspectival expected-value representations of agents and actions as well, which help them to navigate the human environment. Such representations can be formed by highly-general mental processes such as causal and empathic simulation, and thus afford a foundation for spontaneous moral learning and action that requires no innate moral faculty and can exhibit substantial autonomy with respect to community norms. If moral learning is indeed integral with the acquisition and updating of casual and evaluative models, this affords a new way of understanding well-known but seemingly puzzling patterns in intuitive moral judgment-including the notorious "trolley problems."

Dopamine Does Double Duty in Motivating Cognitive Effort

Cognitive control is subjectively costly, suggesting that engagement is modulated in relationship to incentive state. Dopamine appears to play key roles. In particular, dopamine may mediate cognitive effort by two broad classes of functions: (1) modulating the functional parameters of working memory circuits subserving effortful cognition, and (2) mediating value-learning and decision-making about effortful cognitive action. Here, we tie together these two lines of research, proposing how dopamine serves "double duty", translating incentive information into cognitive motivation.

Will big data yield new mathematics? An evolving synergy with neuroscience

New mathematics has often been inspired by new insights into the natural world. Here we describe some ongoing and possible future interactions among the massive data sets being collected in neuroscience, methods for their analysis and mathematical models of the underlying, still largely uncharted neural substrates that generate these data. We start by recalling events that occurred in turbulence modelling when substantial space-time velocity field measurements and numerical simulations allowed a new perspective on the governing equations of fluid mechanics. While no analogous global mathematical model of neural processes exists, we argue that big data may enable validation or at least rejection of models at cellular to brain area scales and may illuminate connections among models. We give examples of such models and survey some relatively new experimental technologies, including optogenetics and functional imaging, that can report neural activity in live animals performing complex tasks. The search for analytical techniques for these data is already yielding new mathematics, and we believe their multi-scale nature may help relate well-established models, such as the Hodgkin-Huxley equations for single neurons, to more abstract models of neural circuits, brain areas and larger networks within the brain. In brief, we envisage a closer liaison, if not a marriage, between neuroscience and mathematics.

Safety out of control: dopamine and defence

We enjoy a sophisticated understanding of how animals learn to predict appetitive outcomes and direct their behaviour accordingly. This encompasses well-defined learning algorithms and details of how these might be implemented in the brain. Dopamine has played an important part in this unfolding story, appearing to embody a learning signal for predicting rewards and stamping in useful actions, while also being a modulator of behavioural vigour. By contrast, although choosing correct actions and executing them vigorously in the face of adversity is at least as important, our understanding of learning and behaviour in aversive settings is less well developed. We examine aversive processing through the medium of the role of dopamine and targets such as D2 receptors in the striatum. We consider critical factors such as the degree of control that an animal believes it exerts over key aspects of its environment, the distinction between 'better' and 'good' actual or predicted future states, and the potential requirement for a particular form of opponent to dopamine to ensure proper calibration of state values.

Charting the landscape of priority problems in psychiatry, part 2: pathogenesis and aetiology

This is the second of two companion papers proposing priority problems for research on mental disorders. Whereas the first paper focuses on questions of nosology and diagnosis, this Personal View concerns pathogenesis and aetiology of psychiatric diseases. We hope that this (non-exhaustive and subjective) list of problems, nominated by scientists and clinicians from different fields and institutions, provides guidance and perspectives for choosing future directions in psychiatric science.

Habitual control of goal selection in humans

Humans choose actions based on both habit and planning. Habitual control is computationally frugal but adapts slowly to novel circumstances, whereas planning is computationally expensive but can adapt swiftly. Current research emphasizes the competition between habits and plans for behavioral control, yet many complex tasks instead favor their integration. We consider a hierarchical architecture that exploits the computational efficiency of habitual control to select goals while preserving the flexibility of planning to achieve those goals. We formalize this mechanism in a reinforcement learning setting, illustrate its costs and benefits, and experimentally demonstrate its spontaneous application in a sequential decision-making task.

Deciding How To Decide: Self-Control and Meta-Decision Making

Many different situations related to self control involve competition between two routes to decisions: default and frugal versus more resource-intensive. Examples include habits versus deliberative decisions, fatigue versus cognitive effort, and Pavlovian versus instrumental decision making. We propose that these situations are linked by a strikingly similar core dilemma, pitting the opportunity costs of monopolizing shared resources such as executive functions for some time, against the possibility of obtaining a better outcome. We offer a unifying normative perspective on this underlying rational meta-optimization, review how this may tie together recent advances in many separate areas, and connect several independent models. Finally, we suggest that the crucial mechanisms and meta-decision variables may be shared across domains.

The algorithmic anatomy of model-based evaluation

Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.

Cognitive effort: A neuroeconomic approach

Cognitive effort has been implicated in numerous theories regarding normal and aberrant behavior and the physiological response to engagement with demanding tasks. Yet, despite broad interest, no unifying, operational definition of cognitive effort itself has been proposed. Here, we argue that the most intuitive and epistemologically valuable treatment is in terms of effort-based decision-making, and advocate a neuroeconomics-focused research strategy. We first outline psychological and neuroscientific theories of cognitive effort. Then we describe the benefits of a neuroeconomic research strategy, highlighting how it affords greater inferential traction than do traditional markers of cognitive effort, including self-reports and physiologic markers of autonomic arousal. Finally, we sketch a future series of studies that can leverage the full potential of the neuroeconomic approach toward understanding the cognitive and neural mechanisms that give rise to phenomenal, subjective cognitive effort.

Depression: a decision-theoretic analysis

The manifold symptoms of depression are common and often transient features of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioral implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.

Model-based and model-free Pavlovian reward learning: revaluation, revision, and revelation

Evidence supports at least two methods for learning about reward and punishment and making predictions for guiding actions. One method, called model-free, progressively acquires cached estimates of the long-run values of circumstances and actions from retrospective experience. The other method, called model-based, uses representations of the environment, expectations, and prospective calculations to make cognitive predictions of future value. Extensive attention has been paid to both methods in computational analyses of instrumental learning. By contrast, although a full computational analysis has been lacking, Pavlovian learning and prediction has typically been presumed to be solely model-free. Here, we revise that presumption and review compelling evidence from Pavlovian revaluation experiments showing that Pavlovian predictions can involve their own form of model-based evaluation. In model-based Pavlovian evaluation, prevailing states of the body and brain influence value computations, and thereby produce powerful incentive motivations that can sometimes be quite new. We consider the consequences of this revised Pavlovian view for the computational landscape of prediction, response, and choice. We also revisit differences between Pavlovian and instrumental learning in the control of incentive motivation.

Motivation and cognitive control: from behavior to neural mechanism

Research on cognitive control and executive function has long recognized the relevance of motivational factors. Recently, however, the topic has come increasingly to center stage, with a surge of new studies examining the interface of motivation and cognitive control. In the present article we survey research situated at this interface, considering work from cognitive and social psychology and behavioral economics, but with a particular focus on neuroscience research. We organize existing findings into three core areas, considering them in the light of currently vying theoretical perspectives. Based on the accumulated evidence, we advocate for a view of control function that treats it as a domain of reward-based decision making. More broadly, we argue that neuroscientific evidence plays a critical role in understanding the mechanisms by which motivation and cognitive control interact. Opportunities for further cross-fertilization between behavioral and neuroscientific research are highlighted.