Planning in the brain

Building machines that learn and think with people

What do we want from machine intelligence? We envision machines that are not just tools for thought but partners in thought: reasonable, insightful, knowledgeable, reliable and trustworthy systems that think with us. Current artificial intelligence systems satisfy some of these criteria, some of the time. In this Perspective, we show how the science of collaborative cognition can be put to work to engineer systems that really can be called 'thought partners', systems built to meet our expectations and complement our limitations. We lay out several modes of collaborative thought in which humans and artificial intelligence thought partners can engage, and we propose desiderata for human-compatible thought partnerships. Drawing on motifs from computational cognitive science, we motivate an alternative scaling path for the design of thought partners and ecosystems around their use through a Bayesian lens, whereby the partners we construct actively build and reason over models of the human and world.

The role of training variability for model-based and model-free learning of an arbitrary visuomotor mapping

A fundamental feature of the human brain is its capacity to learn novel motor skills. This capacity requires the formation of vastly different visuomotor mappings. Using a grid navigation task, we investigated whether training variability would enhance the flexible use of a visuomotor mapping (key-to-direction rule), leading to better generalization performance. Experiments 1 and 2 show that participants trained to move between multiple start-target pairs exhibited greater generalization to both distal and proximal targets compared to participants trained to move between a single pair. This finding suggests that limited variability can impair decisions even in simple tasks without planning. In addition, during the training phase, participants exposed to higher variability were more inclined to choose options that, counterintuitively, moved the cursor away from the target while minimizing its actual distance under the constrained mapping, suggesting a greater engagement in model-based computations. In Experiments 3 and 4, we showed that the limited generalization performance in participants trained with a single pair can be enhanced by a short period of variability introduced early in learning or by incorporating stochasticity into the visuomotor mapping. Our computational modeling analyses revealed that a hybrid model between model-free and model-based computations with different mixing weights for the training and generalization phases, best described participants' data. Importantly, the differences in the model-based weights between our experimental groups, paralleled the behavioral findings during training and generalization. Taken together, our results suggest that training variability enables the flexible use of the visuomotor mapping, potentially by preventing the consolidation of habits due to the continuous demand to change responses.

Learning to Move and Plan like the Knight: Sequential Decision Making with a Novel Motor Mapping

Many skills that humans acquire throughout their lives, such as playing video games or sports, require substantial motor learning and multi-step planning. While both processes are typically studied separately, they are likely to interact during the acquisition of complex motor skills. In this work, we studied this interaction by assessing human performance in a sequential decision-making task that requires the learning of a non-trivial motor mapping. Participants were tasked to move a cursor from start to target locations in a grid world, using a standard keyboard. Notably, the specific keys were arbitrarily mapped to a movement rule resembling the Knight chess piece. In Experiment 1, we showed the learning of this mapping in the absence of planning, led to significant improvements in the task when presented with sequential decisions at a later stage. Computational modeling analysis revealed that such improvements resulted from an increased learning rate about the state transitions of the motor mapping, which also resulted in more flexible planning from trial to trial (less perseveration or habitual responses). In Experiment 2, we showed that incorporating mapping learning into the planning process, allows us to capture (1) differential task improvements for distinct planning horizons and (2) overall lower performance for longer horizons. Additionally, model analysis suggested that participants may limit their search to three steps ahead. We hypothesize that this limitation in planning horizon arises from capacity constraints in working memory, and may be the reason complex skills are often broken down into individual subroutines or components during learning.

A recurrent network model of planning explains hippocampal replay and human behavior

When faced with a novel situation, people often spend substantial periods of time contemplating possible futures. For such planning to be rational, the benefits to behavior must compensate for the time spent thinking. Here, we capture these features of behavior by developing a neural network model where planning itself is controlled by the prefrontal cortex. This model consists of a meta-reinforcement learning agent augmented with the ability to plan by sampling imagined action sequences from its own policy, which we call 'rollouts'. In a spatial navigation task, the agent learns to plan when it is beneficial, which provides a normative explanation for empirical variability in human thinking times. Additionally, the patterns of policy rollouts used by the artificial agent closely resemble patterns of rodent hippocampal replays. Our work provides a theory of how the brain could implement planning through prefrontal-hippocampal interactions, where hippocampal replays are triggered by-and adaptively affect-prefrontal dynamics.

Comparative Analysis of Human-Chimpanzee Divergence in Brain Connectivity and its Genetic Correlates

Chimpanzees (Pan troglodytes) are humans' closest living relatives, making them the most directly relevant comparison point for understanding human brain evolution. Zeroing in on the differences in brain connectivity between humans and chimpanzees can provide key insights into the specific evolutionary changes that might have occured along the human lineage. However, conducting comparisons of brain connectivity between humans and chimpanzees remains challenging, as cross-species brain atlases established within the same framework are currently lacking. Without the availability of cross-species brain atlases, the region-wise connectivity patterns between humans and chimpanzees cannot be directly compared. To address this gap, we built the first Chimpanzee Brainnetome Atlas (ChimpBNA) by following a well-established connectivity-based parcellation framework. Leveraging this new resource, we found substantial divergence in connectivity patterns across most association cortices, notably in the lateral temporal and dorsolateral prefrontal cortex between the two species. Intriguingly, these patterns significantly deviate from the patterns of cortical expansion observed in humans compared to chimpanzees. Additionally, we identified regions displaying connectional asymmetries that differed between species, likely resulting from evolutionary divergence. Genes associated with these divergent connectivities were found to be enriched in cell types crucial for cortical projection circuits and synapse formation. These genes exhibited more pronounced differences in expression patterns in regions with higher connectivity divergence, suggesting a potential foundation for brain connectivity evolution. Therefore, our study not only provides a fine-scale brain atlas of chimpanzees but also highlights the connectivity divergence between humans and chimpanzees in a more rigorous and comparative manner and suggests potential genetic correlates for the observed divergence in brain connectivity patterns between the two species. This can help us better understand the origins and development of uniquely human cognitive capabilities.

Neurocomputational model of compulsivity: deviating from an uncertain goal-directed system

Despite a theory that an imbalance in goal-directed versus habitual systems serve as building blocks of compulsions, research has yet to delineate how this occurs during arbitration between the two systems in obsessive-compulsive disorder. Inspired by a brain model in which the inferior frontal cortex selectively gates the putamen to guide goal-directed or habitual actions, this study aimed to examine whether disruptions in the arbitration process via the fronto-striatal circuit would underlie imbalanced decision-making and compulsions in patients. Thirty patients with obsessive-compulsive disorder [mean (standard deviation) age = 26.93 (6.23) years, 12 females (40%)] and 30 healthy controls [mean (standard deviation) age = 24.97 (4.72) years, 17 females (57%)] underwent functional MRI scans while performing the two-step Markov decision task, which was designed to dissociate goal-directed behaviour from habitual behaviour. We employed a neurocomputational model to account for an uncertainty-based arbitration process, in which a prefrontal arbitrator (i.e. inferior frontal gyrus) allocates behavioural control to a more reliable strategy by selectively gating the putamen. We analysed group differences in the neural estimates of uncertainty of each strategy. We also compared the psychophysiological interaction effects of system preference (goal-directed versus habitual) on fronto-striatal coupling between groups. We examined the correlation between compulsivity score and the neural activity and connectivity involved in the arbitration process. The computational model captured the subjects' preferences between the strategies. Compared with healthy controls, patients had a stronger preference for the habitual system (t = -2.88, P = 0.006), which was attributed to a more uncertain goal-directed system (t = 2.72, P = 0.009). Before the allocation of controls, patients exhibited hypoactivity in the inferior frontal gyrus compared with healthy controls when this region tracked the inverse of uncertainty (i.e. reliability) of goal-directed behaviour (P = 0.001, family-wise error rate corrected). When reorienting behaviours to reach specific goals, patients exhibited weaker right ipsilateral ventrolateral prefronto-putamen coupling than healthy controls (P = 0.001, family-wise error rate corrected). This hypoconnectivity was correlated with more severe compulsivity (r = -0.57, P = 0.002). Our findings suggest that the attenuated top-down control of the putamen by the prefrontal arbitrator underlies compulsivity in obsessive-compulsive disorder. Enhancing fronto-striatal connectivity may be a potential neurotherapeutic approach for compulsivity and adaptive decision-making.

Reactivation strength during cued recall is modulated by graph distance within cognitive maps

Declarative memory retrieval is thought to involve reinstatement of neuronal activity patterns elicited and encoded during a prior learning episode. Furthermore, it is suggested that two mechanisms operate during reinstatement, dependent on task demands: individual memory items can be reactivated simultaneously as a clustered occurrence or, alternatively, replayed sequentially as temporally separate instances. In the current study, participants learned associations between images that were embedded in a directed graph network and retained this information over a brief 8 min consolidation period. During a subsequent cued recall session, participants retrieved the learned information while undergoing magnetoencephalographic recording. Using a trained stimulus decoder, we found evidence for clustered reactivation of learned material. Reactivation strength of individual items during clustered reactivation decreased as a function of increasing graph distance, an ordering present solely for successful retrieval but not for retrieval failure. In line with previous research, we found evidence that sequential replay was dependent on retrieval performance and was most evident in low performers. The results provide evidence for distinct performance-dependent retrieval mechanisms, with graded clustered reactivation emerging as a plausible mechanism to search within abstract cognitive maps.

Synergizing habits and goals with variational Bayes

Behaving efficiently and flexibly is crucial for biological and artificial embodied agents. Behavior is generally classified into two types: habitual (fast but inflexible), and goal-directed (flexible but slow). While these two types of behaviors are typically considered to be managed by two distinct systems in the brain, recent studies have revealed a more sophisticated interplay between them. We introduce a theoretical framework using variational Bayesian theory, incorporating a Bayesian intention variable. Habitual behavior depends on the prior distribution of intention, computed from sensory context without goal-specification. In contrast, goal-directed behavior relies on the goal-conditioned posterior distribution of intention, inferred through variational free energy minimization. Assuming that an agent behaves using a synergized intention, our simulations in vision-based sensorimotor tasks explain the key properties of their interaction as observed in experiments. Our work suggests a fresh perspective on the neural mechanisms of habits and goals, shedding light on future research in decision making.

From task structures to world models: what do LLMs know?

In what sense does a large language model (LLM) have knowledge? We answer by granting LLMs 'instrumental knowledge': knowledge gained by using next-word generation as an instrument. We then ask how instrumental knowledge is related to the ordinary, 'worldly knowledge' exhibited by humans, and explore this question in terms of the degree to which instrumental knowledge can be said to incorporate the structured world models of cognitive science. We discuss ways LLMs could recover degrees of worldly knowledge and suggest that such recovery will be governed by an implicit, resource-rational tradeoff between world models and tasks. Our answer to this question extends beyond the capabilities of a particular AI system and challenges assumptions about the nature of knowledge and intelligence.

The generative neural microdynamics of cognitive processing

The entorhinal cortex and hippocampus form a recurrent network that informs many cognitive processes, including memory, planning, navigation, and imagination. Neural recordings from these regions reveal spatially organized population codes corresponding to external environments and abstract spaces. Aligning the former cognitive functionalities with the latter neural phenomena is a central challenge in understanding the entorhinal-hippocampal circuit (EHC). Disparate experiments demonstrate a surprising level of complexity and apparent disorder in the intricate spatiotemporal dynamics of sequential non-local hippocampal reactivations, which occur particularly, though not exclusively, during immobile pauses and rest. We review these phenomena with a particular focus on their apparent lack of physical simulative realism. These observations are then integrated within a theoretical framework and proposed neural circuit mechanisms that normatively characterize this neural complexity by conceiving different regimes of hippocampal microdynamics as neuromarkers of diverse cognitive computations.

Local prediction-learning in high-dimensional spaces enables neural networks to plan

Planning and problem solving are cornerstones of higher brain function. But we do not know how the brain does that. We show that learning of a suitable cognitive map of the problem space suffices. Furthermore, this can be reduced to learning to predict the next observation through local synaptic plasticity. Importantly, the resulting cognitive map encodes relations between actions and observations, and its emergent high-dimensional geometry provides a sense of direction for reaching distant goals. This quasi-Euclidean sense of direction provides a simple heuristic for online planning that works almost as well as the best offline planning algorithms from AI. If the problem space is a physical space, this method automatically extracts structural regularities from the sequence of observations that it receives so that it can generalize to unseen parts. This speeds up learning of navigation in 2D mazes and the locomotion with complex actuator systems, such as legged bodies. The cognitive map learner that we propose does not require a teacher, similar to self-attention networks (Transformers). But in contrast to Transformers, it does not require backpropagation of errors or very large datasets for learning. Hence it provides a blue-print for future energy-efficient neuromorphic hardware that acquires advanced cognitive capabilities through autonomous on-chip learning.

A robot-rodent interaction arena with adjustable spatial complexity for ethologically relevant behavioral studies

Outside of the laboratory, animals behave in spaces where they can transition between open areas and coverage as they interact with others. Replicating these conditions in the laboratory can be difficult to control and record. This has led to a dominance of relatively simple, static behavioral paradigms that reduce the ethological relevance of behaviors and may alter the engagement of cognitive processes such as planning and decision-making. Therefore, we developed a method for controllable, repeatable interactions with others in a reconfigurable space. Mice navigate a large honeycomb lattice of adjustable obstacles as they interact with an autonomous robot coupled to their actions. We illustrate the system using the robot as a pseudo-predator, delivering airpuffs to the mice. The combination of obstacles and a mobile threat elicits a diverse set of behaviors, such as increased path diversity, peeking, and baiting, providing a method to explore ethologically relevant behaviors in the laboratory.

Delayed escape behavior requires claustral activity

Animals are known to exhibit innate and learned forms of defensive behaviors, but it is unclear whether animals can escape through methods other than these forms. In this study, we develop the delayed escape task, in which male rats temporarily hold the information required for future escape, and we demonstrate that this task, in which the subject extrapolates from past experience without direct experience of its behavioral outcome, does not fall into either of the two forms of behavior. During the holding period, a subset of neurons in the rostral-to-striatum claustrum (rsCla), only when pooled together, sustain enhanced population activity without ongoing sensory stimuli. Transient inhibition of rsCla neurons during the initial part of the holding period produces prolonged inhibition of the enhanced activity. The transient inhibition also attenuates the delayed escape behavior. Our data suggest that the rsCla activity bridges escape-inducing stimuli to the delayed onset of escape.

Putting it Together, Together

People are not as fast or as strong as many other creatures that evolved around us. What gives us an evolutionary advantage is working together to achieve common aims. Coordinating joint action begins at a tender age with such cooperative activities as alternating babbling and clapping games. Adult joint activities are far more complex and use multiple means of coordination. Joint action has attracted qualitative analyses by sociolinguists, cognitive scientists, and philosophers as well as empirical analyses and theories by cognitive scientists. Here, we analyze how joint action is spontaneously coordinated from start to finish in a novel complex real-life joint activity, assembling a piece of furniture, a task that captures the essentials of joint action, collaborators, things in the world, and communicative devices. Pairs of strangers assembled a TV cart from a stack of parts and a photo of the completed cart. Coordination prior to each assembly action was coded as explicit, using speech or gesture, or implicit, actions that both advanced the task and communicated the next step. Initial planning relied on explicit communication about structure, but not action nor division of labor, which were improvised. That served to establish a joint representation of the goal that informed actions and monitored progress. As assembly progressed, coordination was increasingly implicit, through actions alone. Joint action is a dynamic interplay of explicit and implicit signaling with respect to things in the world to coordinate ongoing progress, guided by a shared representation of the goal.

Proactive and reactive construction of memory-based preferences

We are often faced with decisions we have never encountered before, requiring us to infer possible outcomes before making a choice. Computational theories suggest that one way to make these types of decisions is by accessing and linking related experiences stored in memory. Past work has shown that such memory-based preference construction can occur at a number of different timepoints relative to the moment a decision is made. Some studies have found that memories are integrated at the time a decision is faced (reactively) while others found that memory integration happens earlier, when memories are encoded (proactively). Here we offer a resolution to this inconsistency. We demonstrate behavioral and neural evidence for both strategies and for how they tradeoff rationally depending on the associative structure of memory. Using fMRI to decode patterns of brain responses unique to categories of images in memory, we found that proactive memory access is more common and allows more efficient inference. However, participants also use reactive access when choice options are linked to more numerous memory associations. Together, these results indicate that the brain judiciously conducts proactive inference by accessing memories ahead of time in conditions when this strategy is most favorable.

The technology, opportunities, and challenges of Synthetic Biological Intelligence

Integrating neural cultures developed through synthetic biology methods with digital computing has enabled the early development of Synthetic Biological Intelligence (SBI). Recently, key studies have emphasized the advantages of biological neural systems in some information processing tasks. However, neither the technology behind this early development, nor the potential ethical opportunities or challenges, have been explored in detail yet. Here, we review the key aspects that facilitate the development of SBI and explore potential applications. Considering these foreseeable use cases, various ethical implications are proposed. Ultimately, this work aims to provide a robust framework to structure ethical considerations to ensure that SBI technology can be both researched and applied responsibly.

Rational Simplification and Rigidity in Human Planning

Planning underpins the impressive flexibility of goal-directed behavior. However, even when planning, people can display surprising rigidity in how they think about problems (e.g., "functional fixedness") that lead them astray. How can our capacity for behavioral flexibility be reconciled with our susceptibility to conceptual inflexibility? We propose that these tendencies reflect avoidance of two cognitive costs: the cost of representing task details and the cost of switching between representations. To test this hypothesis, we developed a novel paradigm that affords participants opportunities to choose different families of simplified representations to plan. In two preregistered, online studies (Ns = 377 and 294 adults), we found that participants' optimal behavior, suboptimal behavior, and reaction time were explained by a computational model that formalized people's avoidance of representational complexity and switching. These results demonstrate how the selection of simplified, rigid representations leads to the otherwise puzzling combination of flexibility and inflexibility observed in problem solving.

Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

Hope for brain health: impacting the life course and society

Hope is a cognitive process by which an individual can identify their personal goals and develop actionable steps to achieve results. It has the potential to positively impact people's lives by building resilience, and can be meaningfully experienced at both the individual and group level. Despite this significance, there are sizable gaps in our understanding of the neurobiology of hope. In this perspective paper, the authors discuss why further research is needed on hope and its potency to be harnessed in society as a "tool" to promote brain health across healthy and patient populations. Avenues for future research in hope and the brain are proposed. The authors conclude by identifying strategies for the possible applications of hope in brain health promotion within the areas of technology, arts, media, and education.

Expertise increases planning depth in human gameplay

A hallmark of human intelligence is the ability to plan multiple steps into the future1,2. Despite decades of research3-5, it is still debated whether skilled decision-makers plan more steps ahead than novices6-8. Traditionally, the study of expertise in planning has used board games such as chess, but the complexity of these games poses a barrier to quantitative estimates of planning depth. Conversely, common planning tasks in cognitive science often have a lower complexity9,10 and impose a ceiling for the depth to which any player can plan. Here we investigate expertise in a complex board game that offers ample opportunity for skilled players to plan deeply. We use model fitting methods to show that human behaviour can be captured using a computational cognitive model based on heuristic search. To validate this model, we predict human choices, response times and eye movements. We also perform a Turing test and a reconstruction experiment. Using the model, we find robust evidence for increased planning depth with expertise in both laboratory and large-scale mobile data. Experts memorize and reconstruct board features more accurately. Using complex tasks combined with precise behavioural modelling might expand our understanding of human planning and help to bridge the gap with progress in artificial intelligence.

Computational models of episodic-like memory in food-caching birds

Birds of the crow family adapt food-caching strategies to anticipated needs at the time of cache recovery and rely on memory of the what, where and when of previous caching events to recover their hidden food. It is unclear if this behavior can be explained by simple associative learning or if it relies on higher cognitive processes like mental time-travel. We present a computational model and propose a neural implementation of food-caching behavior. The model has hunger variables for motivational control, reward-modulated update of retrieval and caching policies and an associative neural network for remembering caching events with a memory consolidation mechanism for flexible decoding of the age of a memory. Our methodology of formalizing experimental protocols is transferable to other domains and facilitates model evaluation and experiment design. Here, we show that memory-augmented, associative reinforcement learning without mental time-travel is sufficient to explain the results of 28 behavioral experiments with food-caching birds.

Computational cross-species views of the hippocampal formation

The discovery of place cells and head direction cells in the hippocampal formation of freely foraging rodents has led to an emphasis of its role in encoding allocentric spatial relationships. In contrast, studies in head-fixed primates have additionally found representations of spatial views. We review recent experiments in freely moving monkeys that expand upon these findings and show that postural variables such as eye/head movements strongly influence neural activity in the hippocampal formation, suggesting that the function of the hippocampus depends on where the animal looks. We interpret these results in the light of recent studies in humans performing challenging navigation tasks which suggest that depending on the context, eye/head movements serve one of two roles-gathering information about the structure of the environment (active sensing) or externalizing the contents of internal beliefs/deliberation (embodied cognition). These findings prompt future experimental investigations into the information carried by signals flowing between the hippocampal formation and the brain regions controlling postural variables, and constitute a basis for updating computational theories of the hippocampal system to accommodate the influence of eye/head movements.

Spatiotemporal Precision of Neuroimaging in Psychiatry

Aberrant patterns of cognition, perception, and behavior seen in psychiatric disorders are thought to be driven by a complex interplay of neural processes that evolve at a rapid temporal scale. Understanding these dynamic processes in vivo in humans has been hampered by a trade-off between spatial and temporal resolutions inherent to current neuroimaging technology. A recent trend in psychiatric research has been the use of high temporal resolution imaging, particularly magnetoencephalography, often in conjunction with sophisticated machine learning decoding techniques. Developments here promise novel insights into the spatiotemporal dynamics of cognitive phenomena, including domains relevant to psychiatric illnesses such as reward and avoidance learning, memory, and planning. This review considers recent advances afforded by exploiting this increased spatiotemporal precision, with specific reference to applications that seek to drive a mechanistic understanding of psychopathology and the realization of preclinical translation.

Neural learning rules for generating flexible predictions and computing the successor representation

The predictive nature of the hippocampus is thought to be useful for memory-guided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.

Diminished prospective mental representations of reward mediate reward learning strategies among youth with internalizing symptoms

BACKGROUND

Adolescent internalizing symptoms and trauma exposure have been linked with altered reward learning processes and decreased ventral striatal responses to rewarding cues. Recent computational work on decision-making highlights an important role for prospective representations of the imagined outcomes of different choices. This study tested whether internalizing symptoms and trauma exposure among youth impact the generation of prospective reward representations during decision-making and potentially mediate altered behavioral strategies during reward learning.

METHODS

Sixty-one adolescent females with varying exposure to interpersonal violence exposure (n = 31 with histories of physical or sexual assault) and severity of internalizing symptoms completed a social reward learning task during fMRI. Multivariate pattern analyses (MVPA) were used to decode neural reward representations at the time of choice.

RESULTS

MVPA demonstrated that rewarding outcomes could accurately be decoded within several large-scale distributed networks (e.g. frontoparietal and striatum networks), that these reward representations were reactivated prospectively at the time of choice in proportion to the expected probability of receiving reward, and that youth with behavioral strategies that favored exploiting high reward options demonstrated greater prospective generation of reward representations. Youth internalizing symptoms, but not trauma exposure characteristics, were negatively associated with both the behavioral strategy of exploiting high reward options as well as the prospective generation of reward representations in the striatum.

CONCLUSIONS

These data suggest diminished prospective mental simulation of reward as a mechanism of altered reward learning strategies among youth with internalizing symptoms.

Replay and compositional computation

Replay in the brain has been viewed as rehearsal or, more recently, as sampling from a transition model. Here, we propose a new hypothesis: that replay is able to implement a form of compositional computation where entities are assembled into relationally bound structures to derive qualitatively new knowledge. This idea builds on recent advances in neuroscience, which indicate that the hippocampus flexibly binds objects to generalizable roles and that replay strings these role-bound objects into compound statements. We suggest experiments to test our hypothesis, and we end by noting the implications for AI systems which lack the human ability to radically generalize past experience to solve new problems.

Spatial uncertainty and environmental geometry in navigation

Variations in the geometry of the environment, such as the shape and size of an enclosure, have profound effects on navigational behavior and its neural underpinning. Here, we show that these effects arise as a consequence of a single, unifying principle: to navigate efficiently, the brain must maintain and update the uncertainty about one's location. We developed an image-computable Bayesian ideal observer model of navigation, continually combining noisy visual and self-motion inputs, and a neural encoding model optimized to represent the location uncertainty computed by the ideal observer. Through mathematical analysis and numerical simulations, we show that the ideal observer accounts for a diverse range of sometimes paradoxical distortions of human homing behavior in anisotropic and deformed environments, including 'boundary tethering', and its neural encoding accounts for distortions of rodent grid cell responses under identical environmental manipulations. Our results demonstrate that spatial uncertainty plays a key role in navigation.

The influence of action on perception spans different effectors

Perception and action are fundamental processes that characterize our life and our possibility to modify the world around us. Several pieces of evidence have shown an intimate and reciprocal interaction between perception and action, leading us to believe that these processes rely on a common set of representations. The present review focuses on one particular aspect of this interaction: the influence of action on perception from a motor effector perspective during two phases, action planning and the phase following execution of the action. The movements performed by eyes, hands, and legs have a different impact on object and space perception; studies that use different approaches and paradigms have formed an interesting general picture that demonstrates the existence of an action effect on perception, before as well as after its execution. Although the mechanisms of this effect are still being debated, different studies have demonstrated that most of the time this effect pragmatically shapes and primes perception of relevant features of the object or environment which calls for action; at other times it improves our perception through motor experience and learning. Finally, a future perspective is provided, in which we suggest that these mechanisms can be exploited to increase trust in artificial intelligence systems that are able to interact with humans.

Goals, usefulness and abstraction in value-based choice

Colombian drug lord Pablo Escobar, while on the run, purportedly burned two million dollars in banknotes to keep his daughter warm. A stark reminder that, in life, circumstances and goals can quickly change, forcing us to reassess and modify our values on-the-fly. Studies in decision-making and neuroeconomics have often implicitly equated value to reward, emphasising the hedonic and automatic aspect of the value computation, while overlooking its functional (concept-like) nature. Here we outline the computational and biological principles that enable the brain to compute the usefulness of an option or action by creating abstractions that flexibly adapt to changing goals. We present different algorithmic architectures, comparing ideas from artificial intelligence (AI) and cognitive neuroscience with psychological theories and, when possible, drawing parallels.

In vitro neurons learn and exhibit sentience when embodied in a simulated game-world

Integrating neurons into digital systems may enable performance infeasible with silicon alone. Here, we develop DishBrain, a system that harnesses the inherent adaptive computation of neurons in a structured environment. In vitro neural networks from human or rodent origins are integrated with in silico computing via a high-density multielectrode array. Through electrophysiological stimulation and recording, cultures are embedded in a simulated game-world, mimicking the arcade game "Pong." Applying implications from the theory of active inference via the free energy principle, we find apparent learning within five minutes of real-time gameplay not observed in control conditions. Further experiments demonstrate the importance of closed-loop structured feedback in eliciting learning over time. Cultures display the ability to self-organize activity in a goal-directed manner in response to sparse sensory information about the consequences of their actions, which we term synthetic biological intelligence. Future applications may provide further insights into the cellular correlates of intelligence.

Sampling motion trajectories during hippocampal theta sequences

Efficient planning in complex environments requires that uncertainty associated with current inferences and possible consequences of forthcoming actions is represented. Representation of uncertainty has been established in sensory systems during simple perceptual decision making tasks but it remains unclear if complex cognitive computations such as planning and navigation are also supported by probabilistic neural representations. Here, we capitalized on gradually changing uncertainty along planned motion trajectories during hippocampal theta sequences to capture signatures of uncertainty representation in population responses. In contrast with prominent theories, we found no evidence of encoding parameters of probability distributions in the momentary population activity recorded in an open-field navigation task in rats. Instead, uncertainty was encoded sequentially by sampling motion trajectories randomly and efficiently in subsequent theta cycles from the distribution of potential trajectories. Our analysis is the first to demonstrate that the hippocampus is well equipped to contribute to optimal planning by representing uncertainty.

Dynamic Causal Modelling of Hierarchical Planning

Hierarchical planning (HP) is a strategy that optimizes the planning by storing the steps towards the goal (lower-level planning) into subgoals (higher-level planning). In the framework of model-based reinforcement learning, HP requires the computation through the transition value between higher-level hierarchies. Previous study identified the dmPFC, PMC and SPL were involved in the computation process of HP respectively. However, it is still unclear about how these regions interaction with each other to support the computation in HP, which could deepen our understanding about the implementation of plan algorithm in hierarchical environment. To address this question, we conducted an fMRI experiment using a virtual subway navigation task. We identified the activity of the dmPFC, premotor cortex (PMC) and superior parietal lobe (SPL) with general linear model (GLM) in HP. Then, Dynamic Causal Modelling (DCM) was performed to quantify the influence of the higher- and lower-planning on the connectivity between the brain areas identified by the GLM. The strongest modulation effect of the higher-level planning was found on the dmPFC→right PMC connection. Furthermore, using Parametric Empirical Bayes (PEB), we found the modulation of higher-level planning on the dmPFC→right PMC and right PMC→SPL connections could explain the individual difference of the response time. We conclude that the dmPFC-related connectivity takes the response to the higher-level planning, while the PMC acts as the bridge between the higher-level planning to behavior outcome.

Eye movements reveal spatiotemporal dynamics of visually-informed planning in navigation

Goal-oriented navigation is widely understood to depend upon internal maps. Although this may be the case in many settings, humans tend to rely on vision in complex, unfamiliar environments. To study the nature of gaze during visually-guided navigation, we tasked humans to navigate to transiently visible goals in virtual mazes of varying levels of difficulty, observing that they took near-optimal trajectories in all arenas. By analyzing participants' eye movements, we gained insights into how they performed visually-informed planning. The spatial distribution of gaze revealed that environmental complexity mediated a striking trade-off in the extent to which attention was directed towards two complimentary aspects of the world model: the reward location and task-relevant transitions. The temporal evolution of gaze revealed rapid, sequential prospection of the future path, evocative of neural replay. These findings suggest that the spatiotemporal characteristics of gaze during navigation are significantly shaped by the unique cognitive computations underlying real-world, sequential decision making.