Cognitive control over learning: creating, clustering, and generalizing task-set structure

The challenge of learning adaptive mental behavior

Many psychotherapies aim to help people replace maladaptive mental behaviors (such as those leading to unproductive worry) with more adaptive ones (such as those leading to active problem solving). Yet, little is known empirically about how challenging it is to learn adaptive mental behaviors. Mental behaviors entail taking mental operations and thus may be more challenging to perform than motor actions; this challenge may enhance or impair learning. In particular, challenge when learning is often desirable because it improves retention. Yet, it is also plausible that the necessity of carrying out mental operations interferes with learning the expected values of mental actions by impeding credit assignment: the process of updating an action's value after reinforcement. Then, it may be more challenging not only to perform-but also to learn the consequences of-mental (vs. motor) behaviors. We designed a task to assess learning to take adaptive mental versus motor actions via matched probabilistic feedback. In two experiments (N = 300), most participants found it more difficult to learn to select optimal mental (vs. motor) actions, as evident in worse accuracy not only in a learning but also test (retention) phase. Computational modeling traced this impairment to an indicator of worse credit assignment (impaired construction and maintenance of expected values) when learning mental actions, accounting for worse accuracy in the learning and retention phases. The results suggest that people have particular difficulty learning adaptive mental behavior and pave the way for novel interventions to scaffold credit assignment and promote adaptive thinking. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Task structure tailors the geometry of neural representations in human lateral prefrontal cortex

How do human brains represent tasks of varying structure? The lateral prefrontal cortex (lPFC) flexibly represents task information. However, principles that shape lPFC representational geometry remain unsettled. We use fMRI and pattern analyses to reveal the structure of lPFC representational geometries as humans perform two distinct categorization tasks- one with flat, conjunctive categories and another with hierarchical, context-dependent categories. We show that lPFC encodes task-relevant information with task-tailored geometries of intermediate dimensionality. These geometries preferentially enhance the separability of task-relevant variables while encoding a subset in abstract form. Specifically, in the flat task, a global axis encodes response-relevant categories abstractly, while category-specific local geometries are high-dimensional. In the hierarchy task, a global axis abstractly encodes the higher-level context, while low-dimensional, context-specific local geometries compress irrelevant information and abstractly encode the relevant information. Comparing these task geometries exposes generalizable principles by which lPFC tailors representations to different tasks.

Thalamocortical architectures for flexible cognition and efficient learning

The brain exhibits a remarkable ability to learn and execute context-appropriate behaviors. How it achieves such flexibility, without sacrificing learning efficiency, is an important open question. Neuroscience, psychology, and engineering suggest that reusing and repurposing computations are part of the answer. Here, we review evidence that thalamocortical architectures may have evolved to facilitate these objectives of flexibility and efficiency by coordinating distributed computations. Recent work suggests that distributed prefrontal cortical networks compute with flexible codes, and that the mediodorsal thalamus provides regularization to promote efficient reuse. Thalamocortical interactions resemble hierarchical Bayesian computations, and their network implementation can be related to existing gating, synchronization, and hub theories of thalamic function. By reviewing recent findings and providing a novel synthesis, we highlight key research horizons integrating computation, cognition, and systems neuroscience.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

When stimulus variability accelerates the learning of task knowledge in adults and school-aged children

Experience with instances that vary in their surface features helps individuals to form abstract task knowledge, leading to transfer of that knowledge to novel contexts. The current study sought to examine the role of this variability effect in how adults and school-aged children learn to engage cognitive control. We focused on the engagement of cognitive control in advance (proactive control) and in response to conflicts (reactive control) in a cued task-switching paradigm, and conducted four preregistered online experiments with adults (Experiment 1A: N = 100, Experiment 1B: N = 105) and 9- to 10-year-olds (Experiment 2A: N = 98, Experiment 2B: N = 97). It was shown that prior task experience of engaging reactive control makes both adults and 9- to 10-year-olds respond more slowly in a subsequent similar-structured condition with different stimuli in which proactive control could have been engaged. 9- to 10-year-olds (Experiment 2B) exhibited more negative transfer of a reactive control mode when uninformative cue and pre-target stimuli, which do not convey task-relevant information, were changed in each block, compared with when they were fixed. Furthermore, adults showed suggestive evidence of the variability effect both when cue and target stimuli were varied (Experiment 1A) and when uninformative cue and pre-target stimuli were varied (Experiment 1B). The collective findings of these experiments provide important insights into the contribution of stimulus variability to the engagement of cognitive control.

Bayesian Reinforcement Learning With Limited Cognitive Load

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Human learning of hierarchical graphs

Humans are exposed to sequences of events in the environment, and the interevent transition probabilities in these sequences can be modeled as a graph or network. Many real-world networks are organized hierarchically and while much is known about how humans learn basic transition graph topology, whether and to what degree humans can learn hierarchical structures in such graphs remains unknown. We probe the mental estimates of transition probabilities via the surprisal effect phenomenon: humans react more slowly to less expected transitions. Using mean-field predictions and numerical simulations, we show that surprisal effects are stronger for finer-level than coarser-level hierarchical transitions, and that surprisal effects at coarser levels are difficult to detect for limited learning times or in small samples. Using a serial response experiment with human participants (n=100), we replicate our predictions by detecting a surprisal effect at the finer level of the hierarchy but not at the coarser level of the hierarchy. We then evaluate the presence of a trade-off in learning, whereby humans who learned the finer level of the hierarchy better also tended to learn the coarser level worse, and vice versa. This study elucidates the processes by which humans learn sequential events in hierarchical contexts. More broadly, our work charts a road map for future investigation of the neural underpinnings and behavioral manifestations of graph learning.

Specificity and sensitivity of the fixed-point test for binary mixture distributions

When two cognitive processes contribute to a behavioral output-each process producing a specific distribution of the behavioral variable of interest-and when the mixture proportion of these two processes varies as a function of an experimental condition, a common density point should be present in the observed distributions of the data across said conditions. In principle, one can statistically test for the presence (or absence) of a fixed point in experimental data to provide evidence in favor of (or against) the presence of a mixture of processes, whose proportions are affected by an experimental manipulation. In this paper, we provide an empirical diagnostic of this test to detect a mixture of processes. We do so using resampling of real experimental data under different scenarios, which mimic variations in the experimental design suspected to affect the sensitivity and specificity of the fixed-point test (i.e., mixture proportion, time on task, and sample size). Resampling such scenarios with real data allows us to preserve important features of data which are typically observed in real experiments while maintaining tight control over the properties of the resampled scenarios. This is of particular relevance considering such stringent assumptions underlying the fixed-point test. With this paper, we ultimately aim at validating the fixed-point property of binary mixture data and at providing some performance metrics to researchers aiming at testing the fixed-point property on their experimental data.

Hierarchical consciousness: the Nested Observer Windows model

Foremost in our experience is the intuition that we possess a unified conscious experience. However, many observations run counter to this intuition: we experience paralyzing indecision when faced with two appealing behavioral choices, we simultaneously hold contradictory beliefs, and the content of our thought is often characterized by an internal debate. Here, we propose the Nested Observer Windows (NOW) Model, a framework for hierarchical consciousness wherein information processed across many spatiotemporal scales of the brain feeds into subjective experience. The model likens the mind to a hierarchy of nested mosaic tiles-where an image is composed of mosaic tiles, and each of these tiles is itself an image composed of mosaic tiles. Unitary consciousness exists at the apex of this nested hierarchy where perceptual constructs become fully integrated and complex behaviors are initiated via abstract commands. We define an observer window as a spatially and temporally constrained system within which information is integrated, e.g. in functional brain regions and neurons. Three principles from the signal analysis of electrical activity describe the nested hierarchy and generate testable predictions. First, nested observer windows disseminate information across spatiotemporal scales with cross-frequency coupling. Second, observer windows are characterized by a high degree of internal synchrony (with zero phase lag). Third, observer windows at the same spatiotemporal level share information with each other through coherence (with non-zero phase lag). The theoretical framework of the NOW Model accounts for a wide range of subjective experiences and a novel approach for integrating prominent theories of consciousness.

Learning environment-specific learning rates

People often have to switch back and forth between different environments that come with different problems and volatilities. While volatile environments require fast learning (i.e., high learning rates), stable environments call for lower learning rates. Previous studies have shown that people adapt their learning rates, but it remains unclear whether they can also learn about environment-specific learning rates, and instantaneously retrieve them when revisiting environments. Here, using optimality simulations and hierarchical Bayesian analyses across three experiments, we show that people can learn to use different learning rates when switching back and forth between two different environments. We even observe a signature of these environment-specific learning rates when the volatility of both environments is suddenly the same. We conclude that humans can flexibly adapt and learn to associate different learning rates to different environments, offering important insights for developing theories of meta-learning and context-specific control.

The successor representation subserves hierarchical abstraction for goal-directed behavior

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

Cultivating creativity: predictive brains and the enlightened room problem

How can one conciliate the claim that humans are uncertainty minimizing systems that seek to navigate predictable and familiar environments with the claim that humans can be creative? We call this the Enlightened Room Problem (ERP). The solution, we suggest, lies not (or not only) in the error-minimizing brain but in the environment itself. Creativity emerges from various degrees of interplay between predictive brains and changing environments: ones that repeatedly move the goalposts for our own error-minimizing machinery. By (co)constructing these challenging worlds, we effectively alter and expand the space within which our own prediction engines operate, and that function as 'exploration bubbles' that enable information seeking, uncertainty minimizing minds to penetrate deeper and deeper into artistic, scientific and engineering space. In what follows, we offer a proof of principle for this kind of environmentally led cognitive expansion. This article is part of the theme issue 'Art, aesthetics and predictive processing: theoretical and empirical perspectives'.

Transfer of strategic task components across unique tasks that share some common structures

Flexible, adaptive behaviour depends on the application of prior learning to novel contexts (transfer). Transfer can take many forms, but the focus of the present study was on "task schemas"-learning strategies that guide the earliest stages of engaging in a novel task. The central aim was to examine the architecture of task schemas and determine whether strategic task components can expedite learning novel tasks that share some structural components with the training tasks. Groups of participants across two experiments were exposed to different training regimes centred around multiple unique tasks that shared some/all/none of the structural task components (the kinds of stimuli, classifications, and/or responses) but none of the surface features (the specific stimuli, classifications, and/or responses) with the test task (a dot-pattern classification task). Initial test performance was improved (to a degree) in all groups relative to a control group whose training did not include any of the structural components relevant to the test task. The strongest evidence of transfer was found in the motoric, perceptual + categorization, and full schema training groups. This observation indicates that training with some (or all) strategic task components expedited learning of a novel task that shared those components. That is, task schemas were found to be componential and were able to expedite learning a novel task where similar (learning) strategies could be applied to specific elements of the test task.

Evidence for positive and negative transfer of abstract task knowledge in adults and school-aged children

Engaging cognitive control is essential to flexibly adapt to constantly changing environments. However, relatively little is known about how prior task experience impacts on the engagement of cognitive control in novel task environments. We aimed to clarify how individuals learn and transfer the engagement of cognitive control with a focus on the hierarchical and temporal aspects of task knowledge. Highlighting two distinct cognitive control processes, the engagement of cognitive control in advance (proactive control) and in response to conflicts (reactive control), we conducted six preregistered online experiments with both adults (Experiment 1, 3, and 5: N = 71, N = 108, and N = 70) and 9- to 10-year-olds (Experiment 2, 4, 6: N = 69, N = 108, and N = 70). Using two different experimental paradigms, we demonstrated that prior task experience of engaging reactive control makes adults and 9-to 10-year-olds respond in a reactive way in a subsequent similar-structured condition with different stimuli in which proactive control could have been engaged. This indicates that individuals do learn knowledge about the temporal structure of task goal activation and, on occasion, negatively transfer this knowledge. Furthermore, individuals exhibited these negative transfer effects in a similar-structured condition with different task goals and stimuli, indicating that they learn hierarchically-structured task knowledge. The collective findings suggest a new way of understanding how hierarchical and temporal task knowledge influences the engagement of cognitive control and highlight potential mechanisms underlying the near transfer effects observed in cognitive control training.

Policy abstraction as a predictor of cognitive effort avoidance

Consistent evidence has established that people avoid cognitively effortful tasks. However, the features that make a task cognitively effortful are still not well understood. Multiple hypotheses have been proposed regarding which task demands underlie cognitive effort costs, such as time-on-task, error likelihood, and the general engagement of cognitive control. In this study, we test the novel hypothesis that tasks requiring behavior according to higher degrees of policy abstraction are experienced as more effortful. Accordingly, policy abstraction, operationalized as the levels of contextual contingency required by task rules, drives task avoidance over and above the effects of task performance, such as time-on-task or error likelihood. To test this hypothesis, we combined two previously established cognitive control tasks that parametrically manipulated policy abstraction with the demand selection task procedure. The design of these tasks allowed us to test whether people avoided tasks with higher order policy abstraction while controlling for the contribution of factors such as time-on-task and expected error rate (ER). Consistent with our hypothesis, we observed that policy abstraction was the strongest predictor of cognitive effort choices, followed by ER. This was evident across both studies and in a within-subject cross-study analysis. These results establish at least one task feature independent of performance, which is predictive of task avoidance behavior. We interpret these results within an opportunity cost framework for understanding aversive experiences of cognitive effort while performing a task. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning

Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation plays a fundamental role in instrumental performance and associated mesolimbic dopamine signaling.

A Multi-task Platform for Profiling Cognitive and Motivational Constructs in Humans and Nonhuman Primates

Background

Understanding the neurobiological substrates of psychiatric disorders requires comprehensive evaluations of cognitive and motivational functions in preclinical research settings. The translational validity of such evaluations will be supported by (1) tasks with high construct validity that are engaging and easy to teach to human and nonhuman participants, (2) software that enables efficient switching between multiple tasks in single sessions, (3) software that supports tasks across a broad range of physical experimental setups, and (4) by platform architectures that are easily extendable and customizable to encourage future optimization and development.

New Method

We describe the Multi-task Universal Suite for Experiments ( M-USE ), a software platform designed to meet these requirements. It leverages the Unity video game engine and C# programming language to (1) support immersive and engaging tasks for humans and nonhuman primates, (2) allow experimenters or participants to switch between multiple tasks within-session, (3) generate builds that function across computers, tablets, and websites, and (4) is freely available online with documentation and tutorials for users and developers. M-USE includes a task library with seven pre-existing tasks assessing cognitive and motivational constructs of perception, attention, working memory, cognitive flexibility, motivational and affective self-control, relational long-term memory, and visuo-spatial problem solving.

Results

M-USE was used to test NHPs on up to six tasks per session, all available as part of the Task Library, and to extract performance metrics for all major cognitive and motivational constructs spanning the Research Domain Criteria (RDoC) of the National Institutes of Mental Health.

Comparison with Existing Methods

Other experiment design and control systems exist, but do not provide the full range of features available in M-USE, including a pre-existing task library for cross-species assessments; the ability to switch seamlessly between tasks in individual sessions; cross-platform build capabilities; license-free availability; and its leveraging of video-engine capabilities used to gamify tasks.

Conclusions

The new multi-task platform facilitates cross-species translational research for understanding the neurobiological substrates of higher cognitive and motivational functions.

Human-like systematic generalization through a meta-learning neural network

The power of human language and thought arises from systematic compositionality-the algebraic ability to understand and produce novel combinations from known components. Fodor and Pylyshyn1 famously argued that artificial neural networks lack this capacity and are therefore not viable models of the mind. Neural networks have advanced considerably in the years since, yet the systematicity challenge persists. Here we successfully address Fodor and Pylyshyn's challenge by providing evidence that neural networks can achieve human-like systematicity when optimized for their compositional skills. To do so, we introduce the meta-learning for compositionality (MLC) approach for guiding training through a dynamic stream of compositional tasks. To compare humans and machines, we conducted human behavioural experiments using an instruction learning paradigm. After considering seven different models, we found that, in contrast to perfectly systematic but rigid probabilistic symbolic models, and perfectly flexible but unsystematic neural networks, only MLC achieves both the systematicity and flexibility needed for human-like generalization. MLC also advances the compositional skills of machine learning systems in several systematic generalization benchmarks. Our results show how a standard neural network architecture, optimized for its compositional skills, can mimic human systematic generalization in a head-to-head comparison.

Information gathering explains decision dynamics during human and monkey reward foraging

Foraging in humans and other animals requires a delicate balance between exploitation of current resources and exploration for new ones. The tendency to overharvest-lingering too long in depleting patches-is a routine behavioral deviation from predictions of optimal foraging theories. To characterize the computational mechanisms driving these deviations, we modeled foraging behavior using a virtual patch-leaving task with human participants and validated our findings in an analogous foraging task in two monkeys. Both humans and monkeys overharvested and stayed longer in patches with longer travel times compared to shorter ones. Critically, patch residence times in both species declined over the course of sessions, enhancing reward rates in humans. These decisions were best explained by a logistic transformation that integrated both current rewards and information about declining rewards. This parsimonious model demystifies both the occurrence and dynamics of overharvesting, highlighting the role of information gathering in foraging. Our findings provide insight into computational mechanisms shaped by ubiquitous foraging dilemmas, underscoring how behavioral modeling can reveal underlying motivations of seemingly irrational decisions.

The effect of abstract inter-chunk relationships on serial-order control

Hierarchical control is often thought to dissect a complex task space into isolated subspaces in order to eliminate interference. Yet, there is also evidence from serial-order control tasks that our cognitive system can make use of abstract relationships between different parts (chunks) of a sequence. Past evidence in this regard was limited to situations with ordered stimuli (e.g., numbers or positions) that may have aided the detection of relationships and allowed gradual learning and hypothesis testing. Therefore, we used a modified task-span paradigm (with no ordered elements between tasks) in which participants performed memorized sequences of tasks that were encoded in terms of separate chunks of three tasks each. To allow examination of learning effects, each sequence was "cycled" through repeatedly. Importantly, we compared sequences whose chunks were governed by a common, abstract grammar with sequences whose chunks were governed by different grammars. Experiment 1 examined the effect of relationships between shared-element chunks (e.g., ABB-BAA vs. ABB-BAB), Experiment 2 and 3 between different-element chunks (e.g., ABA-CDC vs. ABA-CCD), and Experiment 4 examined second-order relationships (e.g., ABA-ABB--CDC-CDD vs. ABA-ABB--CDC-CCD). Robust evidence in favor of beneficial effects of abstract inter-chunk relationships was obtained across all four experiments. Importantly, these effects were at least as strong in initial cycles of performing a given sequence as during later cycles, suggesting that the cognitive system operates with an "expectation of abstract relationships," rather than benefitting from them through gradual learning. We discuss the implications of these results for models of hierarchical control.

Human Learning of Hierarchical Graphs

Humans are constantly exposed to sequences of events in the environment. Those sequences frequently evince statistical regularities, such as the probabilities with which one event transitions to another. Collectively, inter-event transition probabilities can be modeled as a graph or network. Many real-world networks are organized hierarchically and understanding how these networks are learned by humans is an ongoing aim of current investigations. While much is known about how humans learn basic transition graph topology, whether and to what degree humans can learn hierarchical structures in such graphs remains unknown. Here, we investigate how humans learn hierarchical graphs of the Sierpiński family using computer simulations and behavioral laboratory experiments. We probe the mental estimates of transition probabilities via the surprisal effect: a phenomenon in which humans react more slowly to less expected transitions, such as those between communities or modules in the network. Using mean-field predictions and numerical simulations, we show that surprisal effects are stronger for finer-level than coarser-level hierarchical transitions. Notably, surprisal effects at coarser levels of the hierarchy are difficult to detect for limited learning times or in small samples. Using a serial response experiment with human participants (n=100), we replicate our predictions by detecting a surprisal effect at the finer-level of the hierarchy but not at the coarser-level of the hierarchy. To further explain our findings, we evaluate the presence of a trade-off in learning, whereby humans who learned the finer-level of the hierarchy better tended to learn the coarser-level worse, and vice versa. Taken together, our computational and experimental studies elucidate the processes by which humans learn sequential events in hierarchical contexts. More broadly, our work charts a road map for future investigation of the neural underpinnings and behavioral manifestations of graph learning.

Self-regulation learning as active inference: dynamic causal modeling of an fMRI neurofeedback task

Introduction

Learning to self-regulate brain activity by neurofeedback has been shown to lead to changes in the brain and behavior, with beneficial clinical and non-clinical outcomes. Neurofeedback uses a brain-computer interface to guide participants to change some feature of their brain activity. However, the neural mechanism of self-regulation learning remains unclear, with only 50% of the participants succeeding in achieving it. To bridge this knowledge gap, our study delves into the neural mechanisms of self-regulation learning via neurofeedback and investigates the brain processes associated with successful brain self-regulation.

Methods

We study the neural underpinnings of self-regulation learning by employing dynamical causal modeling (DCM) in conjunction with real-time functional MRI data. The study involved a cohort of 18 participants undergoing neurofeedback training targeting the supplementary motor area. A critical focus was the comparison between top-down hierarchical connectivity models proposed by Active Inference and alternative bottom-up connectivity models like reinforcement learning.

Results

Our analysis revealed a crucial distinction in brain connectivity patterns between successful and non-successful learners. Particularly, successful learners evinced a significantly stronger top-down effective connectivity towards the target area implicated in self-regulation. This heightened top-down network engagement closely resembles the patterns observed in goal-oriented and cognitive control studies, shedding light on the intricate cognitive processes intertwined with self-regulation learning.

Discussion

The findings from our investigation underscore the significance of cognitive mechanisms in the process of self-regulation learning through neurofeedback. The observed stronger top-down effective connectivity in successful learners indicates the involvement of hierarchical cognitive control, which aligns with the tenets of Active Inference. This study contributes to a deeper understanding of the neural dynamics behind successful self-regulation learning and provides insights into the potential cognitive architecture underpinning this process.

Prefrontal cortex state representations shape human credit assignment

People learn adaptively from feedback, but the rate of such learning differs drastically across individuals and contexts. Here, we examine whether this variability reflects differences in what is learned. Leveraging a neurocomputational approach that merges fMRI and an iterative reward learning task, we link the specificity of credit assignment-how well people are able to appropriately attribute outcomes to their causes-to the precision of neural codes in the prefrontal cortex (PFC). Participants credit task-relevant cues more precisely in social compared vto nonsocial contexts, a process that is mediated by high-fidelity (i.e., distinct and consistent) state representations in the PFC. Specifically, the medial PFC and orbitofrontal cortex work in concert to match the neural codes from feedback to those at choice, and the strength of these common neural codes predicts credit assignment precision. Together this work provides a window into how neural representations drive adaptive learning.

Neural representation dynamics reveal computational principles of cognitive task learning

During cognitive task learning, neural representations must be rapidly constructed for novel task performance, then optimized for robust practiced task performance. How the geometry of neural representations changes to enable this transition from novel to practiced performance remains unknown. We hypothesized that practice involves a shift from compositional representations (task-general activity patterns that can be flexibly reused across tasks) to conjunctive representations (task-specific activity patterns specialized for the current task). Functional MRI during learning of multiple complex tasks substantiated this dynamic shift from compositional to conjunctive representations, which was associated with reduced cross-task interference (via pattern separation) and behavioral improvement. Further, we found that conjunctions originated in subcortex (hippocampus and cerebellum) and slowly spread to cortex, extending multiple memory systems theories to encompass task representation learning. The formation of conjunctive representations hence serves as a computational signature of learning, reflecting cortical-subcortical dynamics that optimize task representations in the human brain.

Humans decompose tasks by trading off utility and computational cost

Human behavior emerges from planning over elaborate decompositions of tasks into goals, subgoals, and low-level actions. How are these decompositions created and used? Here, we propose and evaluate a normative framework for task decomposition based on the simple idea that people decompose tasks to reduce the overall cost of planning while maintaining task performance. Analyzing 11,117 distinct graph-structured planning tasks, we find that our framework justifies several existing heuristics for task decomposition and makes predictions that can be distinguished from two alternative normative accounts. We report a behavioral study of task decomposition (N = 806) that uses 30 randomly sampled graphs, a larger and more diverse set than that of any previous behavioral study on this topic. We find that human responses are more consistent with our framework for task decomposition than alternative normative accounts and are most consistent with a heuristic-betweenness centrality-that is justified by our approach. Taken together, our results suggest the computational cost of planning is a key principle guiding the intelligent structuring of goal-directed behavior.

Executive Function in Deaf Native Signing Children

The aim of this study is twofold: To examine if deafness is invariably associated with deficits in executive function (EF) and to investigate the relationship between sign language proficiency and EF in deaf children of deaf parents with early exposure to a sign language. It is also the first study of EF in children acquiring Polish Sign Language. Even though the mothers of the deaf children (N = 20) had lower levels of education compared with the mothers of a hearing control group, the children performed similarly to their hearing peers (N = 20) on a variety of EF task-based assessments. Only in the Go/No-go task were weaker inhibition skills observed in younger deaf children (6-9 years) compared with hearing peers, and this difference was not seen in older children (10-12 years). Hence, deafness does not necessarily impair EF; however, attentional and inhibition abilities may be acquired via a different route in deaf children. Sign language receptive skills predicted EF in deaf children. In conclusion, we highlight the importance of deaf parenting building the scaffolding for EF in deaf children.

Dissociation and integration of outcome and state uncertainty signals in cognitive control

Signals related to uncertainty are frequently observed in regions of the cognitive control network, including anterior cingulate/medial prefrontal cortex (ACC/mPFC), dorsolateral prefrontal cortex (dlPFC), and anterior insular cortex. Uncertainty generally refers to conditions in which decision variables may assume multiple possible values and can arise at multiple points in the perception-action cycle, including sensory input, inferred states of the environment, and the consequences of actions. These sources of uncertainty are frequently correlated: noisy input can lead to unreliable estimates of the state of the environment, with consequential influences on action selection. Given this correlation amongst various sources of uncertainty, dissociating the neural structures underlying their estimation presents an ongoing issue: a region associated with uncertainty related to outcomes may estimate outcome uncertainty itself, or it may reflect a cascade effect of state uncertainty on outcome estimates. In this study, we derive signals of state and outcome uncertainty from mathematical models of risk and observe regions in the cognitive control network whose activity is best explained by signals related to state uncertainty (anterior insula), outcome uncertainty (dlPFC), as well as regions that appear to integrate the two (ACC/mPFC).

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Sensitive caregiving and reward responsivity: A novel mechanism linking parenting and executive functions development in early childhood

Sensitive caregiving is an essential aspect of positive parenting that influences executive functions development, but the mechanisms underlying this association are less clear. Using data from the Family Life Project, a large prospective longitudinal sample of 1292 families residing in rural, predominately low-income communities, the current study examined whether sensitive caregiving impacts executive functions development by shaping behavioral reward processing systems in early postnatal life. Results indicated that higher levels of sensitive caregiving during infancy were associated with heightened reward responsivity at age 4, which in turn predicted superior executive functions ability at age 5. Notably, children's reward responsivity partially mediated the relationship between sensitive caregiving in infancy and executive functions ability at school entry. These findings add to prior work on early experience and children's executive functions and highlight caregiver scaffolding of developing reward processing systems as a potential foundational mechanism for supporting adaptive behavior and self-regulation across the lifespan.

Learning when effort matters: neural dynamics underlying updating and adaptation to changes in performance efficacy

To determine how much cognitive control to invest in a task, people need to consider whether exerting control matters for obtaining rewards. In particular, they need to account for the efficacy of their performance-the degree to which rewards are determined by performance or by independent factors. Yet it remains unclear how people learn about their performance efficacy in an environment. Here we combined computational modeling with measures of task performance and EEG, to provide a mechanistic account of how people (i) learn and update efficacy expectations in a changing environment and (ii) proactively adjust control allocation based on current efficacy expectations. Across 2 studies, subjects performed an incentivized cognitive control task while their performance efficacy (the likelihood that rewards are performance-contingent or random) varied over time. We show that people update their efficacy beliefs based on prediction errors-leveraging similar neural and computational substrates as those that underpin reward learning-and adjust how much control they allocate according to these beliefs. Using computational modeling, we show that these control adjustments reflect changes in information processing, rather than the speed-accuracy tradeoff. These findings demonstrate the neurocomputational mechanism through which people learn how worthwhile their cognitive control is.

Knowledge generalization and the costs of multitasking

Humans are able to rapidly perform novel tasks, but show pervasive performance costs when attempting to do two things at once. Traditionally, empirical and theoretical investigations into the sources of such multitasking interference have largely focused on multitasking in isolation to other cognitive functions, characterizing the conditions that give rise to performance decrements. Here we instead ask whether multitasking costs are linked to the system's capacity for knowledge generalization, as is required to perform novel tasks. We show how interrogation of the neurophysiological circuitry underlying these two facets of cognition yields further insights for both. Specifically, we demonstrate how a system that rapidly generalizes knowledge may induce multitasking costs owing to sharing of task contingencies between contexts in neural representations encoded in frontoparietal and striatal brain regions. We discuss neurophysiological insights suggesting that prolonged learning segregates such representations by refining the brain's model of task-relevant contingencies, thereby reducing information sharing between contexts and improving multitasking performance while reducing flexibility and generalization. These proposed neural mechanisms explain why the brain shows rapid task understanding, multitasking limitations and practice effects. In short, multitasking limits are the price we pay for behavioural flexibility.

Visuospatial information foraging describes search behavior in learning latent environmental features

In the real world, making sequences of decisions to achieve goals often depends upon the ability to learn aspects of the environment that are not directly perceptible. Learning these so-called latent features requires seeking information about them. Prior efforts to study latent feature learning often used single decisions, used few features, and failed to distinguish between reward-seeking and information-seeking. To overcome this, we designed a task in which humans and monkeys made a series of choices to search for shapes hidden on a grid. On our task, the effects of reward and information outcomes from uncovering parts of shapes could be disentangled. Members of both species adeptly learned the shapes and preferred to select tiles expected to be informative earlier in trials than previously rewarding ones, searching a part of the grid until their outcomes dropped below the average information outcome-a pattern consistent with foraging behavior. In addition, how quickly humans learned the shapes was predicted by how well their choice sequences matched the foraging pattern, revealing an unexpected connection between foraging and learning. This adaptive search for information may underlie the ability in humans and monkeys to learn latent features to support goal-directed behavior in the long run.

Contextual inference in learning and memory

Context is widely regarded as a major determinant of learning and memory across numerous domains, including classical and instrumental conditioning, episodic memory, economic decision-making, and motor learning. However, studies across these domains remain disconnected due to the lack of a unifying framework formalizing the concept of context and its role in learning. Here, we develop a unified vernacular allowing direct comparisons between different domains of contextual learning. This leads to a Bayesian model positing that context is unobserved and needs to be inferred. Contextual inference then controls the creation, expression, and updating of memories. This theoretical approach reveals two distinct components that underlie adaptation, proper and apparent learning, respectively referring to the creation and updating of memories versus time-varying adjustments in their expression. We review a number of extensions of the basic Bayesian model that allow it to account for increasingly complex forms of contextual learning.

Adaptive control of synaptic plasticity integrates micro- and macroscopic network function

Synaptic plasticity configures interactions between neurons and is therefore likely to be a primary driver of behavioral learning and development. How this microscopic-macroscopic interaction occurs is poorly understood, as researchers frequently examine models within particular ranges of abstraction and scale. Computational neuroscience and machine learning models offer theoretically powerful analyses of plasticity in neural networks, but results are often siloed and only coarsely linked to biology. In this review, we examine connections between these areas, asking how network computations change as a function of diverse features of plasticity and vice versa. We review how plasticity can be controlled at synapses by calcium dynamics and neuromodulatory signals, the manifestation of these changes in networks, and their impacts in specialized circuits. We conclude that metaplasticity-defined broadly as the adaptive control of plasticity-forges connections across scales by governing what groups of synapses can and can't learn about, when, and to what ends. The metaplasticity we discuss acts by co-opting Hebbian mechanisms, shifting network properties, and routing activity within and across brain systems. Asking how these operations can go awry should also be useful for understanding pathology, which we address in the context of autism, schizophrenia and Parkinson's disease.

Humans adaptively resolve the explore-exploit dilemma under cognitive constraints: Evidence from a multi-armed bandit task

When navigating uncertain worlds, humans must balance exploring new options versus exploiting known rewards. Longer horizons and spatially structured option values encourage humans to explore, but the impact of real-world cognitive constraints such as environment size and memory demands on explore-exploit decisions is unclear. In the present study, humans chose between options varying in uncertainty during a multi-armed bandit task with varying environment size and memory demands. Regression and cognitive computational models of choice behavior showed that with a lower cognitive load, humans are more exploratory than a simulated value-maximizing learner, but under cognitive constraints, they adaptively scale down exploration to maintain exploitation. Thus, while humans are curious, cognitive constraints force people to decrease their strategic exploration in a resource-rational-like manner to focus on harvesting known rewards.

Tracking human skill learning with a hierarchical Bayesian sequence model

Humans can implicitly learn complex perceptuo-motor skills over the course of large numbers of trials. This likely depends on our becoming better able to take advantage of ever richer and temporally deeper predictive relationships in the environment. Here, we offer a novel characterization of this process, fitting a non-parametric, hierarchical Bayesian sequence model to the reaction times of human participants' responses over ten sessions, each comprising thousands of trials, in a serial reaction time task involving higher-order dependencies. The model, adapted from the domain of language, forgetfully updates trial-by-trial, and seamlessly combines predictive information from shorter and longer windows onto past events, weighing the windows proportionally to their predictive power. As the model implies a posterior over window depths, we were able to determine how, and how many, previous sequence elements influenced individual participants' internal predictions, and how this changed with practice. Already in the first session, the model showed that participants had begun to rely on two previous elements (i.e., trigrams), thereby successfully adapting to the most prominent higher-order structure in the task. The extent to which local statistical fluctuations in trigram frequency influenced participants' responses waned over subsequent sessions, as participants forgot the trigrams less and evidenced skilled performance. By the eighth session, a subset of participants shifted their prior further to consider a context deeper than two previous elements. Finally, participants showed resistance to interference and slow forgetting of the old sequence when it was changed in the final sessions. Model parameters for individual participants covaried appropriately with independent measures of working memory and error characteristics. In sum, the model offers the first principled account of the adaptive complexity and nuanced dynamics of humans' internal sequence representations during long-term implicit skill learning.

Choice Type Impacts Human Reinforcement Learning

In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values that change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (pressing a key with the index finger), to more general choices that can be executed in a number of ways (selecting dinner at the restaurant). However, does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: Participants were slower and less accurate in learning correct choices of a general format compared with learning more concrete motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: The values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response was not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1 by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL. In both experiments, we ruled out the explanation that the difference in performance between two condition types was driven by difficulty/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.

Humans combine value learning and hypothesis testing strategically in multi-dimensional probabilistic reward learning

Realistic and complex decision tasks often allow for many possible solutions. How do we find the correct one? Introspection suggests a process of trying out solutions one after the other until success. However, such methodical serial testing may be too slow, especially in environments with noisy feedback. Alternatively, the underlying learning process may involve implicit reinforcement learning that learns about many possibilities in parallel. Here we designed a multi-dimensional probabilistic active-learning task tailored to study how people learn to solve such complex problems. Participants configured three-dimensional stimuli by selecting features for each dimension and received probabilistic reward feedback. We manipulated task complexity by changing how many feature dimensions were relevant to maximizing reward, as well as whether this information was provided to the participants. To investigate how participants learn the task, we examined models of serial hypothesis testing, feature-based reinforcement learning, and combinations of the two strategies. Model comparison revealed evidence for hypothesis testing that relies on reinforcement-learning when selecting what hypothesis to test. The extent to which participants engaged in hypothesis testing depended on the instructed task complexity: people tended to serially test hypotheses when instructed that there were fewer relevant dimensions, and relied more on gradual and parallel learning of feature values when the task was more complex. This demonstrates a strategic use of task information to balance the costs and benefits of the two methods of learning.

Updating Contextual Sensory Expectations for Adaptive Behavior

The brain has the extraordinary capacity to construct predictive models of the environment by internalizing statistical regularities in the sensory inputs. The resulting sensory expectations shape how we perceive and react to the world; at the neural level, this relates to decreased neural responses to expected than unexpected stimuli ("expectation suppression"). Crucially, expectations may need revision as context changes. However, existing research has often neglected this issue. Further, it is unclear whether contextual revisions apply selectively to expectations relevant to the task at hand, hence serving adaptive behavior. The present fMRI study examined how contextual visual expectations spread throughout the cortical hierarchy as we update our beliefs. We created a volatile environment: two alternating contexts contained different sequences of object images, thereby producing context-dependent expectations that needed revision when the context changed. Human participants of both sexes attended a training session before scanning to learn the contextual sequences. The fMRI experiment then tested for the emergence of contextual expectation suppression in two separate tasks, respectively, with task-relevant and task-irrelevant expectations. Effects of contextual expectation emerged progressively across the cortical hierarchy as participants attuned themselves to the context: expectation suppression appeared first in the insula, inferior frontal gyrus, and posterior parietal cortex, followed by the ventral visual stream, up to early visual cortex. This applied selectively to task-relevant expectations. Together, the present results suggest that an insular and frontoparietal executive control network may guide the flexible deployment of contextual sensory expectations for adaptive behavior in our complex and dynamic world.SIGNIFICANCE STATEMENT The world is structured by statistical regularities, which we use to predict the future. This is often accompanied by suppressed neural responses to expected compared with unexpected events ("expectation suppression"). Crucially, the world is also highly volatile and context-dependent: expected events may become unexpected when the context changes, thus raising the crucial need for belief updating. However, this issue has generally been neglected. By setting up a volatile environment, we show that expectation suppression emerges first in executive control regions, followed by relevant sensory areas, only when observers use their expectations to optimize behavior. This provides surprising yet clear evidence on how the brain controls the updating of sensory expectations for adaptive behavior in our ever-changing world.

Structure learning enhances concept formation in synthetic Active Inference agents

Humans display astonishing skill in learning about the environment in which they operate. They assimilate a rich set of affordances and interrelations among different elements in particular contexts, and form flexible abstractions (i.e., concepts) that can be generalised and leveraged with ease. To capture these abilities, we present a deep hierarchical Active Inference model of goal-directed behaviour, and the accompanying belief update schemes implied by maximising model evidence. Using simulations, we elucidate the potential mechanisms that underlie and influence concept learning in a spatial foraging task. We show that the representations formed–as a result of foraging–reflect environmental structure in a way that is enhanced and nuanced by Bayesian model reduction, a special case of structure learning that typifies learning in the absence of new evidence. Synthetic agents learn associations and form concepts about environmental context and configuration as a result of inferential, parametric learning, and structure learning processes–three processes that can produce a diversity of beliefs and belief structures. Furthermore, the ensuing representations reflect symmetries for environments with identical configurations.

Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning

A hallmark of human intelligence, but challenging for reinforcement learning (RL) agents, is the ability to compositionally generalise, that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, when navigating in a city, one needs to know the location of the destination and how to operate a vehicle to get there, whether it be pedalling a bike or operating a car. In RL, these correspond to the reward function and transition function, respectively. To compositionally generalize, these two components need to be transferable independently of each other: multiple modes of transport can reach the same goal, and any given mode can be used to reach multiple destinations. Yet there are also instances where it can be helpful to learn and transfer entire structures, jointly representing goals and transitions, particularly whenever these recur in natural tasks (e.g., given a suggestion to get ice cream, one might prefer to bike, even in new towns). Prior theoretical work has explored how, in model-based RL, agents can learn and generalize task components (transition and reward functions). But a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical RL agent that learns and transfers individual task components as well as entire structures (particular compositions of components) by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that it can also improve generalisation and transfer in more complex, hierarchical tasks with goal/subgoal structures. Finally, we end with a discussion of our work including how this clustering algorithm could conceivably be implemented by cortico-striatal gating circuits in the brain.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Contextual effects in sensorimotor adaptation adhere to associative learning rules

Traditional associative learning tasks focus on the formation of associations between salient events and arbitrary stimuli that predict those events. This is exemplified in cerebellar-dependent delay eyeblink conditioning, where arbitrary cues such as a tone or light act as conditioned stimuli (CSs) that predict aversive sensations at the cornea (unconditioned stimulus [US]). Here, we ask if a similar framework could be applied to another type of cerebellar-dependent sensorimotor learning - sensorimotor adaptation. Models of sensorimotor adaptation posit that the introduction of an environmental perturbation results in an error signal that is used to update an internal model of a sensorimotor map for motor planning. Here, we take a step toward an integrative account of these two forms of cerebellar-dependent learning, examining the relevance of core concepts from associative learning for sensorimotor adaptation. Using a visuomotor adaptation reaching task, we paired movement-related feedback (US) with neutral auditory or visual contextual cues that served as CSs. Trial-by-trial changes in feedforward movement kinematics exhibited three key signatures of associative learning: differential conditioning, sensitivity to the CS-US interval, and compound conditioning. Moreover, after compound conditioning, a robust negative correlation was observed between responses to the two elemental CSs of the compound (i.e. overshadowing), consistent with the additivity principle posited by theories of associative learning. The existence of associative learning effects in sensorimotor adaptation provides a proof-of-concept for linking cerebellar-dependent learning paradigms within a common theoretical framework.

Cognitive [Computational] Neuroscience Test Reliability and Clinical Applications for Serious Mental Illness (CNTRaCS) Consortium: Progress and Future Directions

The development of treatments for impaired cognition in schizophrenia has been characterized as the most important challenge facing psychiatry at the beginning of the twenty-first century. The Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) project was designed to build on the potential benefits of using tasks and tools from cognitive neuroscience to better understanding and treat cognitive impairments in psychosis. These benefits include: (1) the use of fine-grained tasks that measure discrete cognitive processes; (2) the ability to design tasks that distinguish between specific cognitive domain deficits and poor performance due to generalized deficits resulting from sedation, low motivation, poor test taking skills, etc.; and (3) the ability to link cognitive deficits to specific neural systems, using animal models, neuropsychology, and functional imaging. CNTRICS convened a series of meetings to identify paradigms from cognitive neuroscience that maximize these benefits and identified the steps need for translation into use in clinical populations. The Cognitive Neuroscience Test Reliability and Clinical Applications for Schizophrenia (CNTRaCS) Consortium was developed to help carry out these steps. CNTRaCS consists of investigators at five different sites across the country with diverse expertise relevant to a wide range of the cognitive systems identified as critical as part of CNTRICs. This work reports on the progress and current directions in the evaluation and optimization carried out by CNTRaCS of the tasks identified as part of the original CNTRICs process, as well as subsequent extensions into the Positive Valence systems domain of Research Domain Criteria (RDoC). We also describe the current focus of CNTRaCS, which involves taking a computational psychiatry approach to measuring cognitive and motivational function across the spectrum of psychosis. Specifically, the current iteration of CNTRaCS is using computational modeling to isolate parameters reflecting potentially more specific cognitive and visual processes that may provide greater interpretability in understanding shared and distinct impairments across psychiatric disorders.

Dynamics of Hierarchical Task Representations

Task representations are critical for cognitive control and adaptive behavior. The hierarchical organization of task representations allows humans to maintain goals, integrate information across varying contexts, and select potential responses. In this study we characterized the structure and interactive dynamics of task representations that facilitate cognitive control. Human participants (both males and females) performed a hierarchical task that required them to select a response rule while considering the contingencies from different contextual inputs. By applying time- and frequency-resolved representational similarity analysis to human electroencephalography data, we characterized properties of task representations that are otherwise difficult to observe. We found that participants formed multiple representations of task-relevant contexts and features from the presented stimuli, beyond simple stimulus-response mappings. These disparate representations were hierarchically structured, with higher-order contextual representations dominantly influencing subordinate representations of task features and response rules. Furthermore, this cascade of top-down interactions facilitated faster responses. Our results describe key properties of task representations that support hierarchical cognitive control.SIGNIFICANCE STATEMENT Humans can adjust their actions in response to contingencies imposed by the environment. Although it has long been hypothesized that this ability depends on mental representations of tasks, the neural dynamics of task representations have been difficult to characterize. Our study used electroencephalography data from human participants to demonstrate the neural organization and interactive dynamics of task representations. Our results revealed a top-down, hierarchically organized representational structure that encoded multiple contexts and features from the environment. To support cognitive control, higher-level contextual representations influenced subordinate representations of task-relevant features and potential responses, facilitating response selection in a context-dependent manner. Our results provide direct evidence of organizational properties of task representations, which are cornerstones of cognitive control theories.

Controllability boosts neural and cognitive signatures of changes-of-mind in uncertain environments

In uncertain environments, seeking information about alternative choice options is essential for adaptive learning and decision-making. However, information seeking is usually confounded with changes-of-mind about the reliability of the preferred option. Here, we exploited the fact that information seeking requires control over which option to sample to isolate its behavioral and neurophysiological signatures. We found that changes-of-mind occurring with control require more evidence against the current option, are associated with reduced confidence, but are nevertheless more likely to be confirmed on the next decision. Multimodal neurophysiological recordings showed that these changes-of-mind are preceded by stronger activation of the dorsal attention network in magnetoencephalography, and followed by increased pupil-linked arousal during the presentation of decision outcomes. Together, these findings indicate that information seeking increases the saliency of evidence perceived as the direct consequence of one's own actions.

Geometry of neural computation unifies working memory and planning

Real-world tasks require coordination of working memory, decision-making, and planning, yet these cognitive functions have disproportionately been studied as independent modular processes in the brain. Here, we propose that contingency representations, defined as mappings for how future behaviors depend on upcoming events, can unify working memory and planning computations. We designed a task capable of disambiguating distinct types of representations. In task-optimized recurrent neural networks, we investigated possible circuit mechanisms for contingency representations and found that these representations can explain neurophysiological observations from the prefrontal cortex during working memory tasks. Our experiments revealed that human behavior is consistent with contingency representations and not with traditional sensory models of working memory. Finally, we generated falsifiable predictions for neural data to identify contingency representations in neural data and to dissociate different models of working memory. Our findings characterize a neural representational strategy that can unify working memory, planning, and context-dependent decision-making.

The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning

We advance a novel computational theory of the hippocampal formation as a hierarchical generative model that organizes sequential experiences, such as rodent trajectories during spatial navigation, into coherent spatiotemporal contexts. We propose that the hippocampal generative model is endowed with inductive biases to identify individual items of experience (first hierarchical layer), organize them into sequences (second layer) and cluster them into maps (third layer). This theory entails a novel characterization of hippocampal reactivations as generative replay: the offline resampling of fictive sequences from the generative model, which supports the continual learning of multiple sequential experiences. We show that the model learns and efficiently retains multiple spatial navigation trajectories, by organizing them into spatial maps. Furthermore, the model reproduces flexible and prospective aspects of hippocampal dynamics that are challenging to explain within existing frameworks. This theory reconciles multiple roles of the hippocampal formation in map-based navigation, episodic memory and imagination.

Credit assignment in hierarchical option transfer

Humans have the exceptional ability to efficiently structure past knowledge during learning to enable fast generalization. Xia and Collins (2021) evaluated this ability in a hierarchically structured, sequential decision-making task, where participants could build "options" (strategy "chunks") at multiple levels of temporal and state abstraction. A quantitative model, the Option Model, captured the transfer effects observed in human participants, suggesting that humans create and compose hierarchical options and use them to explore novel contexts. However, it is not well understood how learning in a new context is attributed to new and old options (i.e., the credit assignment problem). In a new context with new contingencies, where participants can recompose some aspects of previously learned options, do they reliably create new options or overwrite existing ones? Does the credit assignment depend on how similar the new option is to an old one? In our experiment, two groups of participants (n=124 and n=104) learned hierarchically structured options, experienced different amounts of negative transfer in a new option context, and were subsequently tested on the previously learned options. Behavioral analysis showed that old options were successfully reused without interference, and new options were appropriately created and credited. This credit assignment did not depend on how similar the new option was to the old option, showing great flexibility and precision in human hierarchical learning. These behavioral results were captured by the Option Model, providing further evidence for option learning and transfer in humans.

Statistical Learning in Vision

Vision and learning have long been considered to be two areas of research linked only distantly. However, recent developments in vision research have changed the conceptual definition of vision from a signal-evaluating process to a goal-oriented interpreting process, and this shift binds learning, together with the resulting internal representations, intimately to vision. In this review, we consider various types of learning (perceptual, statistical, and rule/abstract) associated with vision in the past decades and argue that they represent differently specialized versions of the fundamental learning process, which must be captured in its entirety when applied to complex visual processes. We show why the generalized version of statistical learning can provide the appropriate setup for such a unified treatment of learning in vision, what computational framework best accommodates this kind of statistical learning, and what plausible neural scheme could feasibly implement this framework. Finally, we list the challenges that the field of statistical learning faces in fulfilling the promise of being the right vehicle for advancing our understanding of vision in its entirety. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.