Computational Neural Mechanisms of Goal-Directed Planning and Problem Solving

Debates on the dorsomedial prefrontal/dorsal anterior cingulate cortex: insights for future research

The dorsomedial prefrontal cortex/dorsal anterior cingulate cortex (dmPFC/dACC) is a brain area subject to many theories and debates over its function(s). Even its precise anatomical borders are subject to much controversy. In the past decades, the dmPFC/dACC has been associated with more than 15 different cognitive processes, which sometimes appear quite unrelated (e.g. body perception, cognitive conflict). As a result, understanding what the dmPFC/dACC does has become a real challenge for many neuroscientists. Several theories of this brain area's function(s) have been developed, leading to successive and competitive publications bearing different models, which sometimes contradict each other. During the last two decades, the lively scientific exchanges around the dmPFC/dACC have promoted fruitful research in cognitive neuroscience. In this review, we provide an overview of the anatomy of the dmPFC/dACC, summarize the state of the art of functions that have been associated with this brain area and present the main theories aiming at explaining the dmPFC/dACC function(s). We explore the commonalities and the arguments between the different theories. Finally, we explain what can be learned from these debates for future investigations of the dmPFC/dACC and other brain regions' functions.

Having multiple selves helps learning agents explore and adapt in complex changing worlds

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of subagents, each dedicated to a separate need, powerfully enhanced the agent's capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multiobjective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in nonstationary environments, and c) scaled gracefully in their ability to maintain homeostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of "multiple selves."

Foundations of human spatial problem solving

Despite great strides in both machine learning and neuroscience, we do not know how the human brain solves problems in the general sense. We approach this question by drawing on the framework of engineering control theory. We demonstrate a computational neural model with only localist learning laws that is able to find solutions to arbitrary problems. The model and humans perform a multi-step task with arbitrary and changing starting and desired ending states. Using a combination of computational neural modeling, human fMRI, and representational similarity analysis, we show here that the roles of a number of brain regions can be reinterpreted as interacting mechanisms of a control theoretic system. The results suggest a new set of functional perspectives on the orbitofrontal cortex, hippocampus, basal ganglia, anterior temporal lobe, lateral prefrontal cortex, and visual cortex, as well as a new path toward artificial general intelligence.

The whole prefrontal cortex is premotor cortex

We propose that the entirety of the prefrontal cortex (PFC) can be seen as fundamentally premotor in nature. By this, we mean that the PFC consists of an action abstraction hierarchy whose core function is the potentiation and depotentiation of possible action plans at different levels of granularity. We argue that the apex of the hierarchy should revolve around the process of goal-selection, which we posit is inherently a form of optimization over action abstraction. Anatomical and functional evidence supports the idea that this hierarchy originates on the orbital surface of the brain and extends dorsally to motor cortex. Accordingly, our viewpoint positions the orbitofrontal cortex in a key role in the optimization of goal-selection policies, and suggests that its other proposed roles are aspects of this more general function. Our proposed perspective will reframe outstanding questions, open up new areas of inquiry and align theories of prefrontal function with evolutionary principles. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

The Best Laid Plans: Computational Principles of Anterior Cingulate Cortex

Despite continual debate for the past 30 years about the function of anterior cingulate cortex (ACC), its key contribution to neurocognition remains unknown. However, recent computational modeling work has provided insight into this question. Here we review computational models that illustrate three core principles of ACC function, related to hierarchy, world models, and cost. We also discuss four constraints on the neural implementation of these principles, related to modularity, binding, encoding, and learning and regulation. These observations suggest a role for ACC in hierarchical model-based hierarchical reinforcement learning (HMB-HRL), which instantiates a mechanism motivating the execution of high-level plans.