Sequential replay of nonspatial task states in the human hippocampus

Repetition dynamically and rapidly increases cortical, but not hippocampal, offline reactivation

No sooner is an experience over than its neural representation begins to be transformed through memory reactivation during offline periods. The lion's share of prior research has focused on understanding offline reactivation within the hippocampus. However, it is hypothesized that consolidation processes involve offline reactivation in cortical regions as well as coordinated reactivation in the hippocampus and cortex. Using fMRI, we presented novel and repeated paired associates to participants during encoding and measured offline memory reactivation for those events during an immediate post-encoding rest period. post-encoding reactivation frequency of repeated and once-presented events did not differ in the hippocampus. However, offline reactivation in widespread cortical regions and hippocampal-cortical coordinated reactivation were significantly enhanced for repeated events. These results provide evidence that repetition might facilitate the distribution of memory representations across cortical networks, a hallmark of systems-level consolidation. Interestingly, we found that offline reactivation frequency in both hippocampus and cortex explained variance in behavioral success on an immediate associative recognition test for the once-presented information, potentially indicating a role of offline reactivation in maintaining these novel, weaker, memories. Together, our findings highlight that endogenous offline reactivation can be robustly and significantly modulated by study repetition.

Replay shapes abstract cognitive maps for efficient social navigation

To make adaptive social decisions, people must anticipate how information flows through their social network. While this requires knowledge of how people are connected, networks are too large to have first-hand experience with every possible route between individuals. How, then, are people able to accurately track information flow through social networks? Here we find that people immediately cache abstract knowledge about social network structure as they learn who is friends with whom, which enables the identification of efficient routes between remotely connected individuals. These cognitive maps of social networks, which are built while learning, are then reshaped through overnight rest. During these extended periods of rest, a replay-like mechanism helps to make these maps increasingly abstract, which privileges improvements in social navigation accuracy for the longest communication paths that span distinct communities within the network. Together, these findings provide mechanistic insight into the sophisticated mental representations humans use for social navigation.

Replay-triggered brain-wide activation in humans

The consolidation of discrete experiences into a coherent narrative shapes the cognitive map, providing structured mental representations of our experiences. In this process, past memories are reactivated and replayed in sequence, fostering hippocampal-cortical dialogue. However, brain-wide engagement coinciding with sequential reactivation (or replay) of memories remains largely unexplored. In this study, employing simultaneous EEG-fMRI, we capture both the spatial and temporal dynamics of memory replay. We find that during mental simulation, past memories are replayed in fast sequences as detected via EEG. These transient replay events are associated with heightened fMRI activity in the hippocampus and medial prefrontal cortex. Replay occurrence strengthens functional connectivity between the hippocampus and the default mode network, a set of brain regions key to representing the cognitive map. On the other hand, when subjects are at rest following learning, memory reactivation of task-related items is stronger than that of pre-learning rest, and is also associated with heightened hippocampal activation and augmented hippocampal connectivity to the entorhinal cortex. Together, our findings highlight a distributed, brain-wide engagement associated with transient memory reactivation and its sequential replay.

Neural mechanisms of credit assignment for delayed outcomes during contingent learning

Adaptive behavior in complex environments critically relies on the ability to appropriately link specific choices or actions to their outcomes. However, the neural mechanisms that support the ability to credit only those past choices believed to have caused the observed outcomes remain unclear. Here, we leverage multivariate pattern analyses of functional magnetic resonance imaging (fMRI) data and an adaptive learning task to shed light on the underlying neural mechanisms of such specific credit assignment. We find that the lateral orbitofrontal cortex (lOFC) and hippocampus (HC) code for the causal choice identity when credit needs to be assigned for choices that are separated from outcomes by a long delay, even when this delayed transition is punctuated by interim decisions. Further, we show when interim decisions must be made, learning is additionally supported by lateral frontopolar cortex (FPl). Our results indicate that FPl holds previous causal choices in a "pending" state until a relevant outcome is observed, and the fidelity of these representations predicts the fidelity of subsequent causal choice representations in lOFC and HC during credit assignment. Together, these results highlight the importance of the timely reinstatement of specific causes in lOFC and HC in learning choice-outcome relationships when delays and choices intervene, a critical component of real-world learning and decision making.

Helpless infants are learning a foundation model

Humans have a protracted postnatal helplessness period, typically attributed to human-specific maternal constraints causing an early birth when the brain is highly immature. By aligning neurodevelopmental events across species, however, it has been found that humans are not born with especially immature brains compared with animal species with a shorter helpless period. Consistent with this, the rapidly growing field of infant neuroimaging has found that brain connectivity and functional activation at birth share many similarities with the mature brain. Inspired by machine learning, where deep neural networks also benefit from a 'helpless period' of pre-training, we propose that human infants are learning a foundation model: a set of fundamental representations that underpin later cognition with high performance and rapid generalisation.

State-dependent connectivity in auditory-reward networks predicts peak pleasure experiences to music

Music can evoke pleasurable and rewarding experiences. Past studies that examined task-related brain activity revealed individual differences in musical reward sensitivity traits and linked them to interactions between the auditory and reward systems. However, state-dependent fluctuations in spontaneous neural activity in relation to music-driven rewarding experiences have not been studied. Here, we used functional MRI to examine whether the coupling of auditory-reward networks during a silent period immediately before music listening can predict the degree of musical rewarding experience of human participants (N = 49). We used machine learning models and showed that the functional connectivity between auditory and reward networks, but not others, could robustly predict subjective, physiological, and neurobiological aspects of the strong musical reward of chills. Specifically, the right auditory cortex-striatum/orbitofrontal connections predicted the reported duration of chills and the activation level of nucleus accumbens and insula, whereas the auditory-amygdala connection was associated with psychophysiological arousal. Furthermore, the predictive model derived from the first sample of individuals was generalized in an independent dataset using different music samples. The generalization was successful only for state-like, pre-listening functional connectivity but not for stable, intrinsic functional connectivity. The current study reveals the critical role of sensory-reward connectivity in pre-task brain state in modulating subsequent rewarding experience.

Shared structure facilitates working memory of multiple sequences

Daily experiences often involve the processing of multiple sequences, yet storing them challenges the limited capacity of working memory (WM). To achieve efficient memory storage, relational structures shared by sequences would be leveraged to reorganize and compress information. Here, participants memorized a sequence of items with different colors and spatial locations and later reproduced the full color and location sequences one after another. Crucially, we manipulated the consistency between location and color sequence trajectories. First, sequences with consistent trajectories demonstrate improved memory performance and a trajectory correlation between reproduced color and location sequences. Second, sequences with consistent trajectories show neural reactivation of common trajectories, and display spontaneous replay of color sequences when recalling locations. Finally, neural reactivation correlates with WM behavior. Our findings suggest that a shared common structure is leveraged for the storage of multiple sequences through compressed encoding and neural replay, together facilitating efficient information organization in WM.

The Prediction of Brain Activity from Connectivity: Advances and Applications

The human brain is composed of multiple, discrete, functionally specialized regions that are interconnected to form large-scale distributed networks. Using advanced brain-imaging methods and machine-learning analytical approaches, recent studies have demonstrated that regional brain activity during the performance of various cognitive tasks can be accurately predicted from patterns of task-independent brain connectivity. In this review article, we first present evidence for the predictability of brain activity from structural connectivity (i.e., white matter connections) and functional connectivity (i.e., temporally synchronized task-free activations). We then discuss the implications of such predictions to clinical populations, such as patients diagnosed with psychiatric disorders or neurologic diseases, and to the study of brain-behavior associations. We conclude that connectivity may serve as an infrastructure that dictates brain activity, and we pinpoint several open questions and directions for future research.

Reactivation strength during cued recall is modulated by graph distance within cognitive maps

Declarative memory retrieval is thought to involve reinstatement of neuronal activity patterns elicited and encoded during a prior learning episode. Furthermore, it is suggested that two mechanisms operate during reinstatement, dependent on task demands: individual memory items can be reactivated simultaneously as a clustered occurrence or, alternatively, replayed sequentially as temporally separate instances. In the current study, participants learned associations between images that were embedded in a directed graph network and retained this information over a brief 8 min consolidation period. During a subsequent cued recall session, participants retrieved the learned information while undergoing magnetoencephalographic recording. Using a trained stimulus decoder, we found evidence for clustered reactivation of learned material. Reactivation strength of individual items during clustered reactivation decreased as a function of increasing graph distance, an ordering present solely for successful retrieval but not for retrieval failure. In line with previous research, we found evidence that sequential replay was dependent on retrieval performance and was most evident in low performers. The results provide evidence for distinct performance-dependent retrieval mechanisms, with graded clustered reactivation emerging as a plausible mechanism to search within abstract cognitive maps.

Association between resting-state connectivity patterns in the defensive system network and treatment response in spider phobia-a replication approach

Although highly effective on average, exposure-based treatments do not work equally well for all patients with anxiety disorders. The identification of pre-treatment response-predicting patient characteristics may enable patient stratification. Preliminary research highlights the relevance of inhibitory fronto-limbic networks as such. We aimed to identify pre-treatment neural signatures differing between exposure treatment responders and non-responders in spider phobia and to validate results through rigorous replication. Data of a bi-centric intervention study comprised clinical phenotyping and pre-treatment resting-state functional connectivity (rsFC) data of n = 79 patients with spider phobia (discovery sample) and n = 69 patients (replication sample). RsFC data analyses were accomplished using the Matlab-based CONN-toolbox with harmonized analyses protocols at both sites. Treatment response was defined by a reduction of >30% symptom severity from pre- to post-treatment (Spider Phobia Questionnaire Score, primary outcome). Secondary outcome was defined by a reduction of >50% in a Behavioral Avoidance Test (BAT). Mean within-session fear reduction functioned as a process measure for exposure. Compared to non-responders and pre-treatment, results in the discovery sample seemed to indicate that responders exhibited stronger negative connectivity between frontal and limbic structures and were characterized by heightened connectivity between the amygdala and ventral visual pathway regions. Patients exhibiting high within-session fear reduction showed stronger excitatory connectivity within the prefrontal cortex than patients with low within-session fear reduction. Whereas these results could be replicated by another team using the same data (cross-team replication), cross-site replication of the discovery sample findings in the independent replication sample was unsuccessful. Results seem to support negative fronto-limbic connectivity as promising ingredient to enhance response rates in specific phobia but lack sufficient replication. Further research is needed to obtain a valid basis for clinical decision-making and the development of individually tailored treatment options. Notably, future studies should regularly include replication approaches in their protocols.

Neural Trajectories of Conceptually Related Events

In a series of conceptually related episodes, meaning arises from the link between these events rather than from each event individually. How does the brain keep track of conceptually related sequences of events (i.e., conceptual trajectories)? In a particular kind of conceptual trajectory-a social relationship-meaning arises from a specific sequence of interactions. To test whether such abstract sequences are neurally tracked, we had participants complete a naturalistic narrative-based social interaction game, during functional magnetic resonance imaging. We modeled the simulated relationships as trajectories through an abstract affiliation and power space. In two independent samples, we found evidence of individual social relationships being tracked with unique sequences of hippocampal states. The neural states corresponded to the accumulated trial-to-trial affiliation and power relations between the participant and each character, such that each relationship's history was captured by its own neural trajectory. Each relationship had its own sequence of states, and all relationships were embedded within the same manifold. As such, we show that the hippocampus represents social relationships with ordered sequences of low-dimensional neural patterns. The number of distinct clusters of states on this manifold is also related to social function, as measured by the size of real-world social networks. These results suggest that our evolving relationships with others are represented in trajectory-like neural patterns.

Grid-like entorhinal representation of an abstract value space during prospective decision making

How valuable a choice option is often changes over time, making the prediction of value changes an important challenge for decision making. Prior studies identified a cognitive map in the hippocampal-entorhinal system that encodes relationships between states and enables prediction of future states, but does not inherently convey value during prospective decision making. In this fMRI study, participants predicted changing values of choice options in a sequence, forming a trajectory through an abstract two-dimensional value space. During this task, the entorhinal cortex exhibited a grid-like representation with an orientation aligned to the axis through the value space most informative for choices. A network of brain regions, including ventromedial prefrontal cortex, tracked the prospective value difference between options. These findings suggest that the entorhinal grid system supports the prediction of future values by representing a cognitive map, which might be used to generate lower-dimensional value signals to guide prospective decision making.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Grid codes underlie multiple cognitive maps in the human brain

Grid cells fire at multiple positions that organize the vertices of equilateral triangles tiling a 2D space and are well studied in rodents. The last decade witnessed rapid progress in two other research lines on grid codes-empirical studies on distributed human grid-like representations in physical and multiple non-physical spaces, and cognitive computational models addressing the function of grid cells based on principles of efficient and predictive coding. Here, we review the progress in these fields and integrate these lines into a systematic organization. We also discuss the coordinate mechanisms of grid codes in the human entorhinal cortex and medial prefrontal cortex and their role in neurological and psychiatric diseases.

Prefrontal Cortex Codes Representations of Target Identity and Feature Uncertainty

Many objects in the real world have features that vary over time, creating uncertainty in how they will look in the future. This uncertainty makes statistical knowledge about the likelihood of features critical to attention demanding processes such as visual search. However, little is known about how the uncertainty of visual features is integrated into predictions about search targets in the brain. In the current study, we test the idea that regions prefrontal cortex code statistical knowledge about search targets before the onset of search. Across 20 human participants (13 female; 7 male), we observe target identity in the multivariate pattern and uncertainty in the overall activation of dorsolateral prefrontal cortex (DLPFC) and inferior frontal junction (IFJ) in advance of the search display. This indicates that the target identity (mean) and uncertainty (variance) of the target distribution are coded independently within the same regions. Furthermore, once the search display appears the univariate IFJ signal scaled with the distance of the actual target from the expected mean, but more so when expected variability was low. These results inform neural theories of attention by showing how the prefrontal cortex represents both the identity and expected variability of features in service of top-down attentional control.SIGNIFICANCE STATEMENT Theories of attention and working memory posit that when we engage in complex cognitive tasks our performance is determined by how precisely we remember task-relevant information. However, in the real world the properties of objects change over time, creating uncertainty about many aspects of the task. There is currently a gap in our understanding of how neural systems represent this uncertainty and combine it with target identity information in anticipation of attention demanding cognitive tasks. In this study, we show that the prefrontal cortex represents identity and uncertainty as unique codes before task onset. These results advance theories of attention by showing that the prefrontal cortex codes both target identity and uncertainty to implement top-down attentional control.

A spontaneous neural replay account for involuntary autobiographical memories and déjà vu experiences

Barzykowski and Moulin argue both involuntary autobiographical memories and déjà vu experiences rely on the same involuntary memory retrieval processes but their underlying neurological basis remains unclear. We propose spontaneous neural replay in the default mode network (DMN) and hippocampus as the basis for involuntary autobiographical memories, whereas for déjà vu experiences such transient activation is limited to the DMN.

Diminished State Space Theory of Human Aging

Many new technologies, such as smartphones, computers, or public-access systems (like ticket-vending machines), are a challenge for older adults. One feature that these technologies have in common is that they involve underlying, partially observable, structures (state spaces) that determine the actions that are necessary to reach a certain goal (e.g., to move from one menu to another, to change a function, or to activate a new service). In this work we provide a theoretical, neurocomputational account to explain these behavioral difficulties in older adults. Based on recent findings from age-comparative computational- and cognitive-neuroscience studies, we propose that age-related impairments in complex goal-directed behavior result from an underlying deficit in the representation of state spaces of cognitive tasks. Furthermore, we suggest that these age-related deficits in adaptive decision-making are due to impoverished neural representations in the orbitofrontal cortex and hippocampus.

Functional neuroimaging as a catalyst for integrated neuroscience

Functional magnetic resonance imaging (fMRI) enables non-invasive access to the awake, behaving human brain. By tracking whole-brain signals across a diverse range of cognitive and behavioural states or mapping differences associated with specific traits or clinical conditions, fMRI has advanced our understanding of brain function and its links to both normal and atypical behaviour. Despite this headway, progress in human cognitive neuroscience that uses fMRI has been relatively isolated from rapid advances in other subdomains of neuroscience, which themselves are also somewhat siloed from one another. In this Perspective, we argue that fMRI is well-placed to integrate the diverse subfields of systems, cognitive, computational and clinical neuroscience. We first summarize the strengths and weaknesses of fMRI as an imaging tool, then highlight examples of studies that have successfully used fMRI in each subdomain of neuroscience. We then provide a roadmap for the future advances that will be needed to realize this integrative vision. In this way, we hope to demonstrate how fMRI can help usher in a new era of interdisciplinary coherence in neuroscience.

Hippocampal and medial prefrontal ensemble spiking represents episodes and rules in similar task spaces

Episodic memory requires the hippocampus and prefrontal cortex to guide decisions by representing events in spatial, temporal, and personal contexts. Both brain regions have been described by cognitive theories that represent events in context as locations in maps or memory spaces. We query whether ensemble spiking in these regions described spatial structures as rats performed memory tasks. From each ensemble, we construct a state-space with each point defined by the coordinated spiking of single and pairs of units in 125-ms bins and investigate how state-space locations discriminate task features. Trajectories through state-spaces correspond with behavioral episodes framed by spatial, temporal, and internal contexts. Both hippocampal and prefrontal ensembles distinguish maze locations, task intervals, and goals by distances between state-space locations, consistent with cognitive mapping and relational memory space theories of episodic memory. Prefrontal modulation of hippocampal activity may guide choices by directing memory representations toward appropriate state-space goal locations.

Generative replay underlies compositional inference in the hippocampal-prefrontal circuit

Human reasoning depends on reusing pieces of information by putting them together in new ways. However, very little is known about how compositional computation is implemented in the brain. Here, we ask participants to solve a series of problems that each require constructing a whole from a set of elements. With fMRI, we find that representations of novel constructed objects in the frontal cortex and hippocampus are relational and compositional. With MEG, we find that replay assembles elements into compounds, with each replay sequence constituting a hypothesis about a possible configuration of elements. The content of sequences evolves as participants solve each puzzle, progressing from predictable to uncertain elements and gradually converging on the correct configuration. Together, these results suggest a computational bridge between apparently distinct functions of hippocampal-prefrontal circuitry and a role for generative replay in compositional inference and hypothesis testing.

Computational mechanisms underlying latent value updating of unchosen actions

Current studies suggest that individuals estimate the value of their choices based on observed feedback. Here, we ask whether individuals also update the value of their unchosen actions, even when the associated feedback remains unknown. One hundred seventy-eight individuals completed a multi-armed bandit task, making choices to gain rewards. We found robust evidence suggesting latent value updating of unchosen actions based on the chosen action's outcome. Computational modeling results suggested that this effect is mainly explained by a value updating mechanism whereby individuals integrate the outcome history for choosing an option with that of rejecting the alternative. Properties of the deliberation (i.e., duration/difficulty) did not moderate the latent value updating of unchosen actions, suggesting that memory traces generated during deliberation might take a smaller role in this specific phenomenon than previously thought. We discuss the mechanisms facilitating credit assignment to unchosen actions and their implications for human decision-making.

Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia-from thought disorder to delusions-are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to quantify the degree to which each participant's verbal behavior was guided by semantic similarity. Using MEG, we indexed neural replay and ripple power in a post-task rest session. Across all participants, word selection was strongly influenced by semantic similarity. The strength of this influence showed sensitivity to task demands (category > letter fluency) and predicted performance. In line with our hypothesis, the influence of semantic similarity on behavior was reduced in schizophrenia relative to controls, predicted negative psychotic symptoms, and correlated with an MEG signature of hippocampal ripple power (but not replay). The findings bridge a gap between phenomenological and neurocomputational accounts of schizophrenia.

Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.

Locating causal hubs of memory consolidation in spontaneous brain network in male mice

Memory consolidation after learning involves spontaneous, brain-wide network reorganization during rest and sleep, but how this is achieved is still poorly understood. Current theory suggests that the hippocampus is pivotal for this reshaping of connectivity. Using fMRI in male mice, we identify that a different set of spontaneous networks and their hubs are instrumental in consolidating memory during post-learning rest. We found that two types of spatial memory training invoke distinct functional connections, but that a network of the sensory cortex and subcortical areas is common for both tasks. Furthermore, learning increased brain-wide network integration, with the prefrontal, striatal and thalamic areas being influential for this network-level reconfiguration. Chemogenetic suppression of each hub identified after learning resulted in retrograde amnesia, confirming the behavioral significance. These results demonstrate the causal and functional roles of resting-state network hubs in memory consolidation and suggest that a distributed network beyond the hippocampus subserves this process.

Awake Hippocampal-Cortical Co-reactivation Is Associated with Forgetting

Systems consolidation theories posit that consolidation occurs primarily through a coordinated communication between hippocampus and neocortex [Moscovitch, M., & Gilboa, A. Systems consolidation, transformation and reorganization: Multiple trace theory, trace transformation theory and their competitors. PsyArXiv, 2021; Kumaran, D., Hassabis, D., & McClelland, J. L. What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends in Cognitive Sciences, 20, 512-534, 2016; McClelland, J. L., & O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419-457, 1995]. Recent sleep studies in rodents have shown that hippocampus and visual cortex replay the same information at temporal proximity ("co-replay"; Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biology, 7, e1000173, 2009; Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I., & Battaglia, F. P. Replay of rule-learning related neural patterns in the prefrontal cortex during sleep. Nature Neuroscience, 12, 919-926, 2009; Wierzynski, C. M., Lubenov, E. V., Gu, M., & Siapas, A. G. State-dependent spike-timing relationships between hippocampal and prefrontal circuits during sleep. Neuron, 61, 587-596, 2009; Ji, D., & Wilson, M. A. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature Neuroscience, 10, 100-107, 2007). We developed a novel repetition time (TR)-based co-reactivation analysis method to study hippocampal-cortical co-replays in humans using fMRI. Thirty-six young adults completed an image (face or scene) and location paired associate encoding task in the scanner, which were preceded and followed by resting state scans. We identified post-encoding rest TRs (± 1) that showed neural reactivation of each image-location trials in both hippocampus (HPC) and category-selective cortex (fusiform face area [FFA]). This allowed us to characterize temporally proximal coordinated reactivations ("co-reactivations") between HPC and FFA. Moreover, we found that increased HPC-FFA co-reactivations were associated with incorrectly recognized trials after a 1-week delay (p = .004). Finally, we found that these HPC-FFA co-reactivations were also associated with trials that were initially correctly recognized immediately after encoding but were later forgotten in 1-day (p = .043) and 1-week delay period (p = .031). We discuss these results from a trace transformation perspective [Sekeres, M. J., Winocur, G., & Moscovitch, M. The hippocampus and related neocortical structures in memory transformation. Neuroscience Letters, 680, 39-53, 2018; Winocur, G., & Moscovitch, M. Memory transformation and systems consolidation. Journal of the International Neuropsychological Society, 17, 766-780, 2011] and speculate that HPC-FFA co-reactivations may be integrating related events, at the expense of disrupting event-specific details, hence leading to forgetting.

Prediction errors drive dynamic changes in neural patterns that guide behavior

Learning describes the process by which our internal expectation models of the world are updated by surprising outcomes (prediction errors [PEs]) to improve predictions of future events. However, the mechanisms through which error signals dynamically influence existing neural representations are unknown. Here, we use functional magnetic resonance imaging (fMRI) in humans solving a two-step Markov decision task to investigate changes in neural activation patterns following PEs. Using a dynamic multivariate pattern analysis, we can show that PE-related fMRI responses in error-coding regions predict trial-by-trial changes in multivariate neural patterns in the orbitofrontal cortex, the precuneus, and the ventromedial prefrontal cortex (vmPFC). Importantly, the dynamics of these pattern changes in the vmPFC also predicted upcoming changes in choice strategies and thus highlight the importance of these pattern changes for behavior.

Widespread coding of navigational variables in prefrontal cortex

To navigate effectively, we must represent information about our location in the environment. Traditional research highlights the role of the hippocampal complex in this process. Spurred by recent research highlighting the widespread cortical encoding of cognitive and motor variables previously thought to have localized function, we hypothesized that navigational variables would be likewise encoded widely, especially in the prefrontal cortex, which is associated with volitional behavior. We recorded neural activity from six prefrontal regions while macaques performed a foraging task in an open enclosure. In all regions, we found strong encoding of allocentric position, allocentric head direction, boundary distance, and linear and angular velocity. These encodings were not accounted for by distance, time to reward, or motor factors. The strength of coding of all variables increased along a ventral-to-dorsal gradient. Together, these results argue that encoding of navigational variables is not localized to the hippocampus and support the hypothesis that navigation is continuous with other forms of flexible cognition in the service of action.

Psychological Momentum

Our mental experience is largely continuous on the scale of seconds and minutes. However, this continuity does not always arise from a volitional carrying forward of ideas. Instead, recent actions, thoughts, dispositions, and emotions can persist in mind, continually shaping our later experience. Aspects of this fundamental property of human cognition - psychological momentum - have been studied under the rubrics of memory, task set, mood, mind-wandering, and mindset. Reviewing these largely independent threads of research, we argue that psychological momentum is best understood from an integrated perspective, as an adaptation that helps us meet the current demands of our environment and to form lasting memories.

The Computational and Neural Bases of Context-Dependent Learning

Flexible behavior requires the creation, updating, and expression of memories to depend on context. While the neural underpinnings of each of these processes have been intensively studied, recent advances in computational modeling revealed a key challenge in context-dependent learning that had been largely ignored previously: Under naturalistic conditions, context is typically uncertain, necessitating contextual inference. We review a theoretical approach to formalizing context-dependent learning in the face of contextual uncertainty and the core computations it requires. We show how this approach begins to organize a large body of disparate experimental observations, from multiple levels of brain organization (including circuits, systems, and behavior) and multiple brain regions (most prominently the prefrontal cortex, the hippocampus, and motor cortices), into a coherent framework. We argue that contextual inference may also be key to understanding continual learning in the brain. This theory-driven perspective places contextual inference as a core component of learning.

GluN2B inhibition confers resilience against long-term cocaine-induced neurocognitive sequelae

Cocaine self-administration can disrupt the capacity of humans and rodents to flexibly modify familiar behavioral routines, even when they become maladaptive or unbeneficial. However, mechanistic factors, particularly those driving long-term behavioral changes, are still being determined. Here, we capitalized on individual differences in oral cocaine self-administration patterns in adolescent mice and revealed that the post-synaptic protein PSD-95 was reduced in the orbitofrontal cortex (OFC) of escalating, but not stable, responders, which corresponded with later deficits in flexible decision-making behavior. Meanwhile, NMDA receptor GluN2B subunit content was lower in the OFC of mice that were resilient to escalatory oral cocaine seeking. This discovery led us to next co-administer the GluN2B-selective antagonist ifenprodil with cocaine, blocking the later emergence of cocaine-induced decision-making abnormalities. GluN2B inhibition also prevented cocaine-induced dysregulation of neuronal structure and function in the OFC, preserving mature, mushroom-shaped dendritic spine densities on deep-layer pyramidal neurons, which were otherwise lower with cocaine, and safeguarding functional BLA→OFC connections necessary for action flexibility. We posit that cocaine potentiates GluN2B-dependent signaling, which triggers a series of durable adaptations that result in the dysregulation of post-synaptic neuronal structure in the OFC and disruption of BLA→OFC connections, ultimately weakening the capacity for flexible choice. And thus, inhibiting GluN2B-NMDARs promotes resilience to long-term cocaine-related sequelae.

The role of memory in counterfactual valuation

Value-based decisions are often guided by past experience. If a choice led to a good outcome, we are more likely to repeat it. This basic idea is well-captured by reinforcement-learning models. However, open questions remain about how we assign value to options we did not choose and which we therefore never had the chance to learn about directly. One solution to this problem is proposed by policy gradient reinforcement-learning models; these do not require direct learning of value, instead optimizing choices according to a behavioral policy. For example, a logistic policy predicts that if a chosen option was rewarded, the unchosen option would be deemed less desirable. Here, we test the relevance of these models to human behavior and explore the role of memory in this phenomenon. We hypothesize that a policy may emerge from an associative memory trace formed during deliberation between choice options. In a preregistered study (n = 315) we show that people tend to invert the value of unchosen options relative to the outcome of chosen options, a phenomenon we term inverse decision bias. The inverse decision bias is correlated with memory for the association between choice options; moreover, it is reduced when memory formation is experimentally interfered with. Finally, we present a new memory-based policy gradient model that predicts both the inverse decision bias and its dependence on memory. Our findings point to a significant role of associative memory in valuation of unchosen options and introduce a new perspective on the interaction between decision-making, memory, and counterfactual reasoning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Précis of What Babies Know

Where does human knowledge begin? Research on human infants, children, adults, and nonhuman animals, using diverse methods from the cognitive, brain, and computational sciences, provides evidence for six early emerging, domain-specific systems of core knowledge. These automatic, unconscious systems are situated between perceptual systems and systems of explicit concepts and beliefs. They emerge early in infancy, guide children's learning, and function throughout life.

Computational cross-species views of the hippocampal formation

The discovery of place cells and head direction cells in the hippocampal formation of freely foraging rodents has led to an emphasis of its role in encoding allocentric spatial relationships. In contrast, studies in head-fixed primates have additionally found representations of spatial views. We review recent experiments in freely moving monkeys that expand upon these findings and show that postural variables such as eye/head movements strongly influence neural activity in the hippocampal formation, suggesting that the function of the hippocampus depends on where the animal looks. We interpret these results in the light of recent studies in humans performing challenging navigation tasks which suggest that depending on the context, eye/head movements serve one of two roles-gathering information about the structure of the environment (active sensing) or externalizing the contents of internal beliefs/deliberation (embodied cognition). These findings prompt future experimental investigations into the information carried by signals flowing between the hippocampal formation and the brain regions controlling postural variables, and constitute a basis for updating computational theories of the hippocampal system to accommodate the influence of eye/head movements.

Sleep-A brain-state serving systems memory consolidation

Although long-term memory consolidation is supported by sleep, it is unclear how it differs from that during wakefulness. Our review, focusing on recent advances in the field, identifies the repeated replay of neuronal firing patterns as a basic mechanism triggering consolidation during sleep and wakefulness. During sleep, memory replay occurs during slow-wave sleep (SWS) in hippocampal assemblies together with ripples, thalamic spindles, neocortical slow oscillations, and noradrenergic activity. Here, hippocampal replay likely favors the transformation of hippocampus-dependent episodic memory into schema-like neocortical memory. REM sleep following SWS might balance local synaptic rescaling accompanying memory transformation with a sleep-dependent homeostatic process of global synaptic renormalization. Sleep-dependent memory transformation is intensified during early development despite the immaturity of the hippocampus. Overall, beyond its greater efficacy, sleep consolidation differs from wake consolidation mainly in that it is supported, rather than impaired, by spontaneous hippocampal replay activity possibly gating memory formation in neocortex.

Hippocampal spatio-predictive cognitive maps adaptively guide reward generalization

The brain forms cognitive maps of relational knowledge-an organizing principle thought to underlie our ability to generalize and make inferences. However, how can a relevant map be selected in situations where a stimulus is embedded in multiple relational structures? Here, we find that both spatial and predictive cognitive maps influence generalization in a choice task, where spatial location determines reward magnitude. Mirroring behavior, the hippocampus not only builds a map of spatial relationships but also encodes the experienced transition structure. As the task progresses, participants' choices become more influenced by spatial relationships, reflected in a strengthening of the spatial map and a weakening of the predictive map. This change is driven by orbitofrontal cortex, which represents the degree to which an outcome is consistent with the spatial rather than the predictive map and updates hippocampal representations accordingly. Taken together, this demonstrates how hippocampal cognitive maps are used and updated flexibly for inference.

Differential replay of reward and punishment paths predicts approach and avoidance

Neural replay is implicated in planning, where states relevant to a task goal are rapidly reactivated in sequence. It remains unclear whether, during planning, replay relates to an actual prospective choice. Here, using magnetoencephalography (MEG), we studied replay in human participants while they planned to either approach or avoid an uncertain environment containing paths leading to reward or punishment. We find evidence for forward sequential replay during planning, with rapid state-to-state transitions from 20 to 90 ms. Replay of rewarding paths was boosted, relative to aversive paths, before a decision to avoid and attenuated before a decision to approach. A trial-by-trial bias toward replaying prospective punishing paths predicted irrational decisions to approach riskier environments, an effect more pronounced in participants with higher trait anxiety. The findings indicate a coupling of replay with planned behavior, where replay prioritizes an online representation of a worst-case scenario for approaching or avoiding.

Spontaneous Activity Patterns in Human Attention Networks Code for Hand Movements

Recent evidence suggests that, in the absence of any task, spontaneous brain activity patterns and connectivity in the visual and motor cortex code for natural stimuli and actions, respectively. These "resting-state" activity patterns may underlie the maintenance and consolidation (replay) of information states coding for ecological stimuli and behaviors. In this study, we examine whether replay patterns occur in resting-state activity in association cortex grouped into high-order cognitive networks not directly processing sensory inputs or motor outputs. Fifteen participants (7 females) performed four hand movements during an fMRI study. Three movements were ecological. The fourth movement as control was less ecological. Before and after the task scans, we acquired resting-state fMRI scans. The analysis examined whether multivertex task activation patterns for the four movements computed at the cortical surface in different brain networks resembled spontaneous activity patterns measured at rest. For each movement, we computed a vector of r values indicating the strength of the similarity between the mean task activation pattern and frame-by-frame resting-state patterns. We computed a cumulative distribution function of r 2 values and used the 90th percentile cutoff value for comparison. In the dorsal attention network, resting-state patterns were more likely to match task patterns for the ecological movements than the control movement. In contrast, rest-task pattern correlation was more likely for less ecological movement in the ventral attention network. These findings show that spontaneous activity patterns in human attention networks code for hand movements.SIGNIFICANCE STATEMENT fMRI indirectly measures neural activity noninvasively. Resting-state (spontaneous) fMRI signals measured in the absence of any task resemble signals evoked by task performance both in topography and inter-regional (functional) connectivity. However, the function of spontaneous brain activity is unknown. We recently showed that spatial activity patterns evoked by visual and motor tasks in visual and motor cortex, respectively, occur at rest in the absence of any stimulus or response. Here we show that activity patterns related to hand movements replay at rest in frontoparietal regions of the human attention system. These findings show that spontaneous activity in the human cortex may mediate the maintenance and consolidation of information states coding for ecological stimuli and behaviors.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Social navigation modulates the anterior and posterior hippocampal circuits in the resting brain

Social navigation is a dynamic and complex process that requires the collaboration of multiple brain regions. However, the neural networks for navigation in a social space remain largely unknown. This study aimed to investigate the role of hippocampal circuit in social navigation from a resting-state fMRI data. Here, resting-state fMRI data were acquired before and after participants performed a social navigation task. By taking the anterior and posterior hippocampus (HPC) as the seeds, we calculated their connectivity with the whole brain using the seed-based static functional connectivity (sFC) and dynamic FC (dFC) approaches. We found that the sFC and dFC between the anterior HPC and supramarginal gyrus, sFC or dFC between posterior HPC and middle cingulate cortex, inferior parietal gyrus, angular gyrus, posterior cerebellum, medial superior frontal gyrus were increased after the social navigation task. These alterations were related to social cognition of tracking location in the social navigation. Moreover, participants who had more social support or less neuroticism showed a greater increase in hippocampal connectivity. These findings may highlight a more important role of the posterior hippocampal circuit in the social navigation, which is crucial for social cognition.

Distinct replay signatures for prospective decision-making and memory preservation

Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.

Successor-like representation guides the prediction of future events in human visual cortex and hippocampus

Human agents build models of their environment, which enable them to anticipate and plan upcoming events. However, little is known about the properties of such predictive models. Recently, it has been proposed that hippocampal representations take the form of a predictive map-like structure, the so-called successor representation (SR). Here, we used human functional magnetic resonance imaging to probe whether activity in the early visual cortex (V1) and hippocampus adhere to the postulated properties of the SR after visual sequence learning. Participants were exposed to an arbitrary spatiotemporal sequence consisting of four items (A-B-C-D). We found that after repeated exposure to the sequence, merely presenting single sequence items (e.g., - B - -) resulted in V1 activation at the successor locations of the full sequence (e.g., C-D), but not at the predecessor locations (e.g., A). This highlights that visual representations are skewed toward future states, in line with the SR. Similar results were also found in the hippocampus. Moreover, the hippocampus developed a coactivation profile that showed sensitivity to the temporal distance in sequence space, with fading representations for sequence events in the more distant past and future. V1, in contrast, showed a coactivation profile that was only sensitive to spatial distance in stimulus space. Taken together, these results provide empirical evidence for the proposition that both visual and hippocampal cortex represent a predictive map of the visual world akin to the SR.

Visual Plasticity in Adulthood: Perspectives from Hebbian and Homeostatic Plasticity

The visual system retains profound plastic potential in adulthood. In the current review, we summarize the evidence of preserved plasticity in the adult visual system during visual perceptual learning as well as both monocular and binocular visual deprivation. In each condition, we discuss how such evidence reflects two major cellular mechanisms of plasticity: Hebbian and homeostatic processes. We focus on how these two mechanisms work together to shape plasticity in the visual system. In addition, we discuss how these two mechanisms could be further revealed in future studies investigating cross-modal plasticity in the visual system.

Continuously changing memories: a framework for proactive and non-linear consolidation

The traditional view of long-term memory is that memory traces mature in a predetermined 'linear' process: their neural substrate shifts from rapidly plastic medial temporal regions towards stable neocortical networks. We propose that memories remain malleable, not by repeated reinstantiations of this linear process but instead via dynamic routes of proactive and non-linear consolidation: memories change, their trajectory is flexible and reversible, and their physical basis develops continuously according to anticipated demands. Studies demonstrating memory updating, increasing hippocampal dependence to support adaptive use, and rapid neocortical plasticity provide evidence for continued non-linear consolidation. Although anticipated demand can affect all stages of memory formation, the extent to which it shapes the physical memory trace repeatedly and proactively will require further dedicated research.

Goals, usefulness and abstraction in value-based choice

Colombian drug lord Pablo Escobar, while on the run, purportedly burned two million dollars in banknotes to keep his daughter warm. A stark reminder that, in life, circumstances and goals can quickly change, forcing us to reassess and modify our values on-the-fly. Studies in decision-making and neuroeconomics have often implicitly equated value to reward, emphasising the hedonic and automatic aspect of the value computation, while overlooking its functional (concept-like) nature. Here we outline the computational and biological principles that enable the brain to compute the usefulness of an option or action by creating abstractions that flexibly adapt to changing goals. We present different algorithmic architectures, comparing ideas from artificial intelligence (AI) and cognitive neuroscience with psychological theories and, when possible, drawing parallels.

Imagination as a fundamental function of the hippocampus

Imagination is a biological function that is vital to human experience and advanced cognition. Despite this importance, it remains unknown how imagination is realized in the brain. Substantial research focusing on the hippocampus, a brain structure traditionally linked to memory, indicates that firing patterns in spatially tuned neurons can represent previous and upcoming paths in space. This work has generally been interpreted under standard views that the hippocampus implements cognitive abilities primarily related to actual experience, whether in the past (e.g. recollection, consolidation), present (e.g. spatial mapping) or future (e.g. planning). However, relatively recent findings in rodents identify robust patterns of hippocampal firing corresponding to a variety of alternatives to actual experience, in many cases without overt reference to the past, present or future. Given these findings, and others on hippocampal contributions to human imagination, we suggest that a fundamental function of the hippocampus is to generate a wealth of hypothetical experiences and thoughts. Under this view, traditional accounts of hippocampal function in episodic memory and spatial navigation can be understood as particular applications of a more general system for imagination. This view also suggests that the hippocampus contributes to a wider range of cognitive abilities than previously thought. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Mesoscopic description of hippocampal replay and metastability in spiking neural networks with short-term plasticity

Bottom-up models of functionally relevant patterns of neural activity provide an explicit link between neuronal dynamics and computation. A prime example of functional activity patterns are propagating bursts of place-cell activities called hippocampal replay, which is critical for memory consolidation. The sudden and repeated occurrences of these burst states during ongoing neural activity suggest metastable neural circuit dynamics. As metastability has been attributed to noise and/or slow fatigue mechanisms, we propose a concise mesoscopic model which accounts for both. Crucially, our model is bottom-up: it is analytically derived from the dynamics of finite-size networks of Linear-Nonlinear Poisson neurons with short-term synaptic depression. As such, noise is explicitly linked to stochastic spiking and network size, and fatigue is explicitly linked to synaptic dynamics. To derive the mesoscopic model, we first consider a homogeneous spiking neural network and follow the temporal coarse-graining approach of Gillespie to obtain a "chemical Langevin equation", which can be naturally interpreted as a stochastic neural mass model. The Langevin equation is computationally inexpensive to simulate and enables a thorough study of metastable dynamics in classical setups (population spikes and Up-Down-states dynamics) by means of phase-plane analysis. An extension of the Langevin equation for small network sizes is also presented. The stochastic neural mass model constitutes the basic component of our mesoscopic model for replay. We show that the mesoscopic model faithfully captures the statistical structure of individual replayed trajectories in microscopic simulations and in previously reported experimental data. Moreover, compared to the deterministic Romani-Tsodyks model of place-cell dynamics, it exhibits a higher level of variability regarding order, direction and timing of replayed trajectories, which seems biologically more plausible and could be functionally desirable. This variability is the product of a new dynamical regime where metastability emerges from a complex interplay between finite-size fluctuations and local fatigue.

Rapid encoding of task regularities in the human hippocampus guides sensorimotor timing

The brain encodes the statistical regularities of the environment in a task-specific yet flexible and generalizable format. Here, we seek to understand this process by bridging two parallel lines of research, one centered on sensorimotor timing, and the other on cognitive mapping in the hippocampal system. By combining functional magnetic resonance imaging (fMRI) with a fast-paced time-to-contact (TTC) estimation task, we found that the hippocampus signaled behavioral feedback received in each trial as well as performance improvements across trials along with reward-processing regions. Critically, it signaled performance improvements independent from the tested intervals, and its activity accounted for the trial-wise regression-to-the-mean biases in TTC estimation. This is in line with the idea that the hippocampus supports the rapid encoding of temporal context even on short time scales in a behavior-dependent manner. Our results emphasize the central role of the hippocampus in statistical learning and position it at the core of a brain-wide network updating sensorimotor representations in real time for flexible behavior.

Dangerous Ground: One-Year-Old Infants are Sensitive to Peril in Other Agents' Action Plans

Do infants appreciate that other people's actions may fail, and that these failures endow risky actions with varying degrees of negative utility (i.e., danger)? Three experiments, including a pre-registered replication, addressed this question by presenting 12- to 15-month-old infants (N = 104, 52 female, majority White) with an animated agent who jumped over trenches of varying depth towards its goals. Infants expected the agent to minimize the danger of its actions, and they learned which goal the agent preferred by observing how much danger it risked to reach each goal, even though the agent's actions were physically identical and never failed. When we tested younger, 10-month-old infants (N = 102, 52 female, majority White) in a fourth experiment, they did not succeed consistently on the same tasks. These findings provide evidence that one-year-old infants use the height that other agents could fall from in order to explain and predict those agents' actions.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.