Surprise signals in the supplementary eye field: rectified prediction errors drive exploration-exploitation transitions

Neural mechanisms for executive control of speed-accuracy trade-off

The medial frontal cortex (MFC) plays an important but disputed role in speed-accuracy trade-off (SAT). In samples of neural spiking in the supplementary eye field (SEF) in the MFC simultaneous with the visuomotor frontal eye field and superior colliculus in macaques performing a visual search with instructed SAT, during accuracy emphasis, most SEF neurons discharge less from before stimulus presentation until response generation. Discharge rates adjust immediately and simultaneously across structures upon SAT cue changes. SEF neurons signal choice errors with stronger and earlier activity during accuracy emphasis. Other neurons signal timing errors, covarying with adjusting response time. Spike correlations between neurons in the SEF and visuomotor areas did not appear, disappear, or change sign across SAT conditions or trial outcomes. These results clarify findings with noninvasive measures, complement previous neurophysiological findings, and endorse the role of the MFC as a critic for the actor instantiated in visuomotor structures.

Functional architecture of executive control and associated event-related potentials in macaques

The medial frontal cortex (MFC) enables executive control by monitoring relevant information and using it to adapt behavior. In macaques performing a saccade countermanding (stop-signal) task, we simultaneously recorded electrical potentials over MFC and neural spiking across all layers of the supplementary eye field (SEF). We report the laminar organization of neurons enabling executive control by monitoring the conflict between incompatible responses, the timing of events, and sustaining goal maintenance. These neurons were a mix of narrow-spiking and broad-spiking found in all layers, but those predicting the duration of control and sustaining the task goal until the release of operant control were more commonly narrow-spiking neurons confined to layers 2 and 3 (L2/3). We complement these results with evidence for a monkey homolog of the N2/P3 event-related potential (ERP) complex associated with response inhibition. N2 polarization varied with error-likelihood and P3 polarization varied with the duration of expected control. The amplitude of the N2 and P3 were predicted by the spike rate of different classes of neurons located in L2/3 but not L5/6. These findings reveal features of the cortical microcircuitry supporting executive control and producing associated ERPs.

Reinforcement Learning Model With Dynamic State Space Tested on Target Search Tasks for Monkeys: Extension to Learning Task Events

Learning is a crucial basis for biological systems to adapt to environments. Environments include various states or episodes, and episode-dependent learning is essential in adaptation to such complex situations. Here, we developed a model for learning a two-target search task used in primate physiological experiments. In the task, the agent is required to gaze one of the four presented light spots. Two neighboring spots are served as the correct target alternately, and the correct target pair is switched after a certain number of consecutive successes. In order for the agent to obtain rewards with a high probability, it is necessary to make decisions based on the actions and results of the previous two trials. Our previous work achieved this by using a dynamic state space. However, to learn a task that includes events such as fixation to the initial central spot, the model framework should be extended. For this purpose, here we propose a "history-in-episode architecture." Specifically, we divide states into episodes and histories, and actions are selected based on the histories within each episode. When we compared the proposed model including the dynamic state space with the conventional SARSA method in the two-target search task, the former performed close to the theoretical optimum, while the latter never achieved target-pair switch because it had to re-learn each correct target each time. The reinforcement learning model including the proposed history-in-episode architecture and dynamic state scape enables episode-dependent learning and provides a basis for highly adaptable learning systems to complex environments.

Involuntary oculomotor inhibition markers of saliency and deviance in response to auditory sequences

Our eyes move constantly but are often inhibited momentarily in response to external stimuli (oculomotor inhibition [OMI]), depending on the stimulus saliency, anticipation, and attention. Previous studies have shown prolonged OMI for auditory oddballs; however, they required counting the oddballs, possibly reflecting voluntary attention. Here, we investigated whether the “passive” OMI response to auditory deviants can provide a quantitative measure of deviance strength (pitch difference) and studied its dependence on the inter-trial interval (ITI). Participants fixated centrally and passively listened to repeated short sequences of pure tones that contained a deviant tone either regularly or with 20% probability (oddballs). In an “active” control experiment, participants counted the deviant or the standard. As in previous studies, the results showed prolonged microsaccade inhibition and increased pupil dilation following the rare deviant tone. Earlier inhibition onset was found in proportion to the pitch deviance (the saliency effect), and a later release was found for oddballs, but only for ITI <2.5 seconds. The active control experiment showed similar results when counting the deviant but longer OMI for the standard when counting it. Taken together, these results suggest that OMI provides involuntary markers of saliency and deviance, which can be obtained without the participant's response.

Reinforcement Learning Model With Dynamic State Space Tested on Target Search Tasks for Monkeys: Self-Determination of Previous States Based on Experience Saturation and Decision Uniqueness

The real world is essentially an indefinite environment in which the probability space, i. e., what can happen, cannot be specified in advance. Conventional reinforcement learning models that learn under uncertain conditions are given the state space as prior knowledge. Here, we developed a reinforcement learning model with a dynamic state space and tested it on a two-target search task previously used for monkeys. In the task, two out of four neighboring spots were alternately correct, and the valid pair was switched after consecutive correct trials in the exploitation phase. The agent was required to find a new pair during the exploration phase, but it could not obtain the maximum reward by referring only to the single previous one trial; it needed to select an action based on the two previous trials. To adapt to this task structure without prior knowledge, the model expanded its state space so that it referred to more than one trial as the previous state, based on two explicit criteria for appropriateness of state expansion: experience saturation and decision uniqueness of action selection. The model not only performed comparably to the ideal model given prior knowledge of the task structure, but also performed well on a task that was not envisioned when the models were developed. Moreover, it learned how to search rationally without falling into the exploration–exploitation trade-off. For constructing a learning model that can adapt to an indefinite environment, the method of expanding the state space based on experience saturation and decision uniqueness of action selection used by our model is promising.

Meta-analysis of cognitive and behavioral tests in leptin- and leptin receptor-deficient mice

Leptin is a hormone produced by adipocytes that regulates food intake and metabolism. Leptin-related gene-deficient mice, such as db/db and ob/ob mice, are widely used to study diabetes and its related diseases. However, broad effects of leptin appear to cause variability in behavioral test results. We performed a meta-analysis of major behavioral tests in db/db and ob/ob mice. These mice exhibited significant impairments in the Morris water maze, forced swim, novel object recognition, Y-maze, tail suspension, and light-dark box tests, whereas the elevated plus maze and open field tests did not reveal significant changes. We also performed correlation and regression analyses between the animals' performances and the experimental protocols and conditions. The memory-related tests were characterized by the correlations of their results with animal age, while the performances in the elevated plus-maze and forced swim tests were affected by the width of the devices used. In conclusion, db/db and ob/ob mice mainly exhibit memory deficits and depression-like behavior, although experimenters should be aware of animal age and device size in conducting experiments.

Abstract goal representation in visual search by neurons in the human pre-supplementary motor area

The medial frontal cortex is important for goal-directed behaviours such as visual search. The pre-supplementary motor area (pre-SMA) plays a critical role in linking higher-level goals to actions, but little is known about the responses of individual cells in this area in humans. Pre-SMA dysfunction is thought to be a critical factor in the cognitive deficits that are observed in diseases such as Parkinson's disease and schizophrenia, making it important to develop a better mechanistic understanding of the pre-SMA's role in cognition. We simultaneously recorded single neurons in the human pre-SMA and eye movements while subjects performed goal-directed visual search tasks. We characterized two groups of neurons in the pre-SMA. First, 40% of neurons changed their firing rate whenever a fixation landed on the search target. These neurons responded to targets in an abstract manner across several conditions and tasks. Responses were invariant to motor output (i.e. button press or not), and to different ways of defining the search target (by instruction or pop-out). Second, ∼50% of neurons changed their response as a function of fixation order. Together, our results show that human pre-SMA neurons carry abstract signals during visual search that indicate whether a goal was reached in an action- and cue-independent manner. This suggests that the pre-SMA contributes to goal-directed behaviour by flexibly signalling goal detection and time elapsed since start of the search, and this process occurs regardless of task. These observations provide insights into how pre-SMA dysfunction might impact cognitive function.

Cortical microcircuitry of performance monitoring

Medial frontal cortex enables performance monitoring, indexed by the error-related negativity (ERN) and manifest by performance adaptations. In monkeys performing a saccade countermanding (stop signal) task, we recorded EEG over and neural spiking across all layers of the supplementary eye field (SEF), an agranular cortical area. Neurons signaling error production, feedback predicting reward gain or loss, and delivery of fluid reward had different spike widths and were concentrated differently across layers. Neurons signaling error or loss of reward were more common in layers 2 and 3 (L2/3), while neurons signaling gain of reward were more common in layers 5 and 6 (L5/6). Variation of error- and reinforcement-related spike rates in L2/3 but not L5/6 predicted response time adaptation. Variation in error-related spike rate in L2/3 but not L5/6 predicted ERN magnitude. These findings reveal novel features of cortical microcircuitry supporting performance monitoring and confirm one cortical source of the ERN.

Learning From Surprise: Harnessing a Metacognitive Surprise Signal to Build and Adapt Belief Networks

One's level of surprise can be thought of as a metacognitive signal indicating how well one can explain new information. We discuss literature on how this signal can be used adaptively to build, and, when necessary, reorganize belief networks. We present challenges in the use of a surprise signal, such as hindsight bias and the tendency to equate difficulty with implausibility, and point to evidence suggesting that one can overcome these challenges through consideration of alternative outcomes-especially before receiving feedback on actual outcomes-and by calibrating task difficulty with one's knowledge level. As such, we propose that a major function of education-broadly construed as the work of teachers, journalists, parents, etc.-is to assist learners in using their metacognitive surprise signals to facilitate the building and adaptation of belief networks.

Neuronal Correlates of Serial Decision-Making in the Supplementary Eye Field

Human behavior is influenced by serial decision-making: past decisions affect choices that set the context for selecting future options. A primate brain region that may be involved in linking decisions across time is the supplementary eye field (SEF), which, in addition to its well known visual responses and saccade-related activity, also signals the rules that govern flexible decisions and the outcomes of those decisions. Our hypotheses were that SEF neurons encode events during serial decision-making and link the sequential decisions with sustained activity. We recorded from neurons in the SEF of two rhesus monkeys (Macaca mulatta, one male, one female) that performed a serial decision-making task. The monkeys used saccades to select a rule that had to be applied later in the same trial to discriminate between visual stimuli. We found, first, that SEF neurons encoded the spatial parameters of saccades during rule selection but not during visual discrimination, suggesting a malleability to their movement-related tuning. Second, SEF activity linked the sequential decisions of rule selection and visual discrimination, but not continuously. Instead, rule-encoding activity appeared in a "just-in-time" manner before the visual discrimination. Third, SEF neurons encoded trial outcomes both prospectively, before decisions within a trial, and retrospectively, across multiple trials. The results thus identify neuronal correlates of rule selection and application in the SEF, including transient signals that link these sequential decisions. Its activity patterns suggest that the SEF participates in serial decision-making in a contextually dependent manner as part of a broader network.SIGNIFICANCE STATEMENT Much research has gone into studying the neurobiological basis of single, isolated decisions. An important next step is to understand how the brain links multiple decisions to generate a coherent stream of thought and behavior. We studied neural activity related to serial decision-making in an area of frontal cortex known as the supplementary eye field (SEF). Neural recordings were conducted in monkeys that performed a serial decision-making task in which they selected and applied rules. We found that SEF neurons convey signals for serial decision-making, including transient encoding of one decision at the time it is needed for the next one and longer-term representations of trial outcomes, suggesting that the region plays a role in continuity of cognition and behavior.

Exploration Disrupts Choice-Predictive Signals and Alters Dynamics in Prefrontal Cortex

In uncertain environments, decision-makers must balance two goals: they must "exploit" rewarding options but also "explore" in order to discover rewarding alternatives. Exploring and exploiting necessarily change how the brain responds to identical stimuli, but little is known about how these states, and transitions between them, change how the brain transforms sensory information into action. To address this question, we recorded neural activity in a prefrontal sensorimotor area while monkeys naturally switched between exploring and exploiting rewarding options. We found that exploration profoundly reduced spatially selective, choice-predictive activity in single neurons and delayed choice-predictive population dynamics. At the same time, reward learning was increased in brain and behavior. These results indicate that exploration is related to sudden disruptions in prefrontal sensorimotor control and rapid, reward-dependent reorganization of control dynamics. This may facilitate discovery through trial and error.