Solving the Credit Assignment Problem With the Prefrontal Cortex

Sound-evoked adenosine release in cooperation with neuromodulatory circuits permits auditory cortical plasticity and perceptual learning

Meaningful auditory memories are formed in adults when acoustic information is delivered to the auditory cortex during heightened states of attention, vigilance, or alertness, as mediated by neuromodulatory circuits. Here, we identify that, in awake mice, acoustic stimulation triggers auditory thalamocortical projections to release adenosine, which prevents cortical plasticity (i.e., selective expansion of neural representation of behaviorally relevant acoustic stimuli) and perceptual learning (i.e., experience-dependent improvement in frequency discrimination ability). This sound-evoked adenosine release (SEAR) becomes reduced within seconds when acoustic stimuli are tightly paired with the activation of neuromodulatory (cholinergic or dopaminergic) circuits or periods of attentive wakefulness. If thalamic adenosine production is enhanced, then SEAR elevates further, the neuromodulatory circuits are unable to sufficiently reduce SEAR, and associative cortical plasticity and perceptual learning are blocked. This suggests that transient low-adenosine periods triggered by neuromodulatory circuits permit associative cortical plasticity and auditory perceptual learning in adults to occur.

The relation between implicit statistical learning and proactivity as revealed by EEG

Environmental events often occur on a probabilistic basis but can sometimes be predicted based on specific cues and thus approached proactively. Incidental statistical learning enables the acquisition of knowledge about probabilistic cue-target contingencies. However, the neural mechanisms of statistical learning about contingencies (SLC), the required conditions for successful learning, and the role of implicit processes in the resultant proactive behavior are still debated. We examined changes in behavior and cortical activity during an SLC task in which subjects responded to visual targets. Unbeknown to them, there were three types of target cues associated with high-, low-, and zero target probabilities. About half of the subjects spontaneously gained explicit knowledge about the contingencies (contingency-aware group), and only they showed evidence of proactivity: shortened response times to predictable targets and enhanced event-related brain responses (cue-evoked P300 and contingent negative variation, CNV) to high probability cues. The behavioral and brain responses were strictly associated on a single-trial basis. Source reconstruction of the brain responses revealed activation of fronto-parietal brain regions associated with cognitive control, particularly the anterior cingulate cortex and precuneus. We also found neural correlates of SLC in the contingency-unaware group, but these were restricted to post-target latencies and visual association areas. Our results document a qualitative difference between explicit and implicit learning processes and suggest that in certain conditions, proactivity may require explicit knowledge about contingencies.

The Radically Embodied Conscious Cybernetic Bayesian Brain: From Free Energy to Free Will and Back Again

Drawing from both enactivist and cognitivist perspectives on mind, I propose that explaining teleological phenomena may require reappraising both "Cartesian theaters" and mental homunculi in terms of embodied self-models (ESMs), understood as body maps with agentic properties, functioning as predictive-memory systems and cybernetic controllers. Quasi-homuncular ESMs are suggested to constitute a major organizing principle for neural architectures due to their initial and ongoing significance for solutions to inference problems in cognitive (and affective) development. Embodied experiences provide foundational lessons in learning curriculums in which agents explore increasingly challenging problem spaces, so answering an unresolved question in Bayesian cognitive science: what are biologically plausible mechanisms for equipping learners with sufficiently powerful inductive biases to adequately constrain inference spaces? Drawing on models from neurophysiology, psychology, and developmental robotics, I describe how embodiment provides fundamental sources of empirical priors (as reliably learnable posterior expectations). If ESMs play this kind of foundational role in cognitive development, then bidirectional linkages will be found between all sensory modalities and frontal-parietal control hierarchies, so infusing all senses with somatic-motoric properties, thereby structuring all perception by relevant affordances, so solving frame problems for embodied agents. Drawing upon the Free Energy Principle and Active Inference framework, I describe a particular mechanism for intentional action selection via consciously imagined (and explicitly represented) goal realization, where contrasts between desired and present states influence ongoing policy selection via predictive coding mechanisms and backward-chained imaginings (as self-realizing predictions). This embodied developmental legacy suggests a mechanism by which imaginings can be intentionally shaped by (internalized) partially-expressed motor acts, so providing means of agentic control for attention, working memory, imagination, and behavior. I further describe the nature(s) of mental causation and self-control, and also provide an account of readiness potentials in Libet paradigms wherein conscious intentions shape causal streams leading to enaction. Finally, I provide neurophenomenological handlings of prototypical qualia including pleasure, pain, and desire in terms of self-annihilating free energy gradients via quasi-synesthetic interoceptive active inference. In brief, this manuscript is intended to illustrate how radically embodied minds may create foundations for intelligence (as capacity for learning and inference), consciousness (as somatically-grounded self-world modeling), and will (as deployment of predictive models for enacting valued goals).

The orbitofrontal cortex in temporal cognition

One of the most important factors in decision-making is estimating the value of available options. Subregions of the prefrontal cortex, including the orbitofrontal cortex (OFC), have been deemed essential for this process. Value computations require a complex integration across numerous dimensions, including, reward magnitude, effort, internal state, and time. The importance of the temporal dimension is well illustrated by temporal discounting tasks, in which subjects select between smaller-sooner versus larger-later rewards. The specific role of OFC in telling time and integrating temporal information into decision-making remains unclear. Based on the current literature, in this review we reevaluate current theories of OFC function, accounting for the influence of time. Incorporating temporal information into value estimation and decision-making requires distinct, yet interrelated, forms of temporal information including the ability to tell time, represent time, create temporal expectations, and the ability to use this information for optimal decision-making in a wide range of tasks, including temporal discounting and wagering. We use the term "temporal cognition" to refer to the integrated use of these different aspects of temporal information. We suggest that the OFC may be a critical site for the integration of reward magnitude and delay, and thus important for temporal cognition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Active maintenance of eligibility trace in rodent prefrontal cortex

Even though persistent neural activity has been proposed as a mechanism for maintaining eligibility trace, direct empirical evidence for active maintenance of eligibility trace has been lacking. We recorded neuronal activity in the medial prefrontal cortex (mPFC) in rats performing a dynamic foraging task in which a choice must be remembered until its outcome on the timescale of seconds for correct credit assignment. We found that mPFC neurons maintain significant choice signals during the time period between action selection and choice outcome. We also found that neural signals for choice, outcome, and action value converge in the mPFC when choice outcome was revealed. Our results indicate that the mPFC maintains choice signals necessary for temporal credit assignment in the form of persistent neural activity in our task. They also suggest that the mPFC might update action value by combining actively maintained eligibility trace with action value and outcome signals.

Reinforcement Learning based Decoding Using Internal Reward for Time Delayed Task in Brain Machine Interfaces

Reinforcement learning (RL) algorithm interprets neural signals into movement intentions with the guidance of the reward in Brain-machine interfaces (BMIs). Current RL algorithms generally work for the tasks with immediate rewards delivery, and lack of efficiency in delayed reward task. Prefrontal cortex, including medial prefrontal cortex(mPFC), has been demonstrated to assign credit to intermediate steps, which reinforces preceding action more efficiently. In this paper, we propose to simulate the functionality of mPFC activities as intermediate rewards to train a RL based decoder in a two-step movement task. A support vector machine (SVM) is adopted to verify if the subject expects a reward due to the accomplishment of a subtask from mPFC activity. Then this discrimination result will be utilized to guide the training of the RL decoder for each step respectively. Here, we apply the Sarsa-style attention-gated reinforcement learning (SAGREL) as the decoder to interpret motor cortex(M1) activity to action states. We test on in vivo primary motor cortex (M1) and mPFC data collected from rats, where the rats need to first trigger the start and then press lever for rewards using M1 signals. SAGREL using intermediate rewards from mPFC activities achieves a prediction accuracy of 66.8% ± 2.0.% (mean ± std) %, which is significantly better than the one using the reward by the end of trial (45.9.% ± 1.2%). This reveals the potentials of modelling mPFC activities as intermediate rewards for the delayed reward tasks.

Bittersweet: The Neuroscience of Ambivalent Affect

Ambivalent affective states, such as bittersweetness, nostalgia, and longing, which are felt as having both positive and negative aspects, are an important component of human experience but have received little attention. The most influential theoretical frameworks in affective neuroscience focus largely on univalenced states. As a result, it is not known whether ambivalent affect corresponds to a simultaneously positive and negative valenced state or whether it results from a rapid vacillation between positive and negative states. Here we hypothesize that ambivalent affect involves both mechanisms, that is, rapid vacillation and simultaneity of positive and negative affect, albeit at different neurobiological levels. Rapidly vacillating univalent emotions could give rise to an ambivalent feeling, a mechanism that depends on brainstem nuclei that facilitate rapid action programs of emotional behavior while inhibiting opposing behaviors. This reciprocal inhibition prevents organisms from simultaneously implementing responses to conflicting emotions but also allows for rapid switching between emotions triggered by counterfactual thinking and rapid reappraisal of situations. We propose that as these transitions occur and respective interoceptive information reaches the insular cortex, further processing of this "emotional moment" would allow separate emotional events to be experienced as one "mixed" and integrated feeling.

Signaling models for dopamine-dependent temporal contiguity in striatal synaptic plasticity

Animals remember temporal links between their actions and subsequent rewards. We previously discovered a synaptic mechanism underlying such reward learning in D1 receptor (D1R)-expressing spiny projection neurons (D1 SPN) of the striatum. Dopamine (DA) bursts promote dendritic spine enlargement in a time window of only a few seconds after paired pre- and post-synaptic spiking (pre-post pairing), which is termed as reinforcement plasticity (RP). The previous study has also identified underlying signaling pathways; however, it still remains unclear how the signaling dynamics results in RP. In the present study, we first developed a computational model of signaling dynamics of D1 SPNs. The D1 RP model successfully reproduced experimentally observed protein kinase A (PKA) activity, including its critical time window. In this model, adenylate cyclase type 1 (AC1) in the spines/thin dendrites played a pivotal role as a coincidence detector against pre-post pairing and DA burst. In particular, pre-post pairing (Ca2+ signal) stimulated AC1 with a delay, and the Ca2+-stimulated AC1 was activated by the DA burst for the asymmetric time window. Moreover, the smallness of the spines/thin dendrites is crucial to the short time window for the PKA activity. We then developed a RP model for D2 SPNs, which also predicted the critical time window for RP that depended on the timing of pre-post pairing and phasic DA dip. AC1 worked for the coincidence detector in the D2 RP model as well. We further simulated the signaling pathway leading to Ca2+/calmodulin-dependent protein kinase II (CaMKII) activation and clarified the role of the downstream molecules of AC1 as the integrators that turn transient input signals into persistent spine enlargement. Finally, we discuss how such timing windows guide animals' reward learning.