Reinforcing behavioral variability: An analysis of dopamine-receptor subtypes and intermittent reinforcement

Spontaneous behaviour is structured by reinforcement without explicit reward

Spontaneous animal behaviour is built from action modules that are concatenated by the brain into sequences1,2. However, the neural mechanisms that guide the composition of naturalistic, self-motivated behaviour remain unknown. Here we show that dopamine systematically fluctuates in the dorsolateral striatum (DLS) as mice spontaneously express sub-second behavioural modules, despite the absence of task structure, sensory cues or exogenous reward. Photometric recordings and calibrated closed-loop optogenetic manipulations during open field behaviour demonstrate that DLS dopamine fluctuations increase sequence variation over seconds, reinforce the use of associated behavioural modules over minutes, and modulate the vigour with which modules are expressed, without directly influencing movement initiation or moment-to-moment kinematics. Although the reinforcing effects of optogenetic DLS dopamine manipulations vary across behavioural modules and individual mice, these differences are well predicted by observed variation in the relationships between endogenous dopamine and module use. Consistent with the possibility that DLS dopamine fluctuations act as a teaching signal, mice build sequences during exploration as if to maximize dopamine. Together, these findings suggest a model in which the same circuits and computations that govern action choices in structured tasks have a key role in sculpting the content of unconstrained, high-dimensional, spontaneous behaviour.

An investigation of resurgence of reinforced behavioral variability in humans

The present study examined resurgence of reinforced variability in college students, who completed a 3-phase computer-based variability task. In the first phase, baseline, points were delivered for drawing rectangles that sufficiently differed from previous rectangles in terms of a target dimension (size or location, counterbalanced) but were sufficiently similar in terms of the alternative dimension. In the second phase, alternative, points were only delivered for rectangles that were sufficiently different in terms of the alternative dimension, but repetitive in terms of the target dimension. In the third phase, extinction, no points were delivered. In baseline, participants made rectangles that were highly varied in terms of the target dimension and less varied in terms of the alternative dimension, and vice versa in the alternative phase. During extinction, levels of variability increased for the target dimension, providing evidence for resurgence of reinforced variability of a specific dimension of behavior. However, levels of variability also remained high for the alternative dimension, indicating that extinction-induced response variability may also have impacted the results. Although future research is needed to explore other explanations, the results of this study replicate prior research with pigeons and provide some support for the notion of variability as an operant.

Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models

Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.

Differential effects of 3,4-methylenedioxymethamphetamine, methamphetamine, meta-Chlorophenylpiperazine, and scopolamine on behavioral repetition versus variation in rats

Acute administration of drugs of abuse, such as MDMA and methamphetamine, disrupts performance on many operant tasks, for example, those used to study memory. This might occur in part because drugs make behavior, in general, more repetitive or more variable, or because they produce a more global disruption to performance. The current study explored this across two experiments by employing Neuringer's 'reinforced variability' procedure. Varied behavior was reinforced at some times during the session and repetitive behavior at other times; lights signalled the behavior required. This procedure allowed an investigation of whether a particular drug made behavior more variable (affected behavior when repetition was required), more repetitive (affected behavior when variability was required), or produced a global disruption (affected both components). In Experiment 1, MDMA increased variability while methamphetamine affected both components. In Experiment 2, m-CPP affected both components while scopolamine affected both components at lower doses and increased variability at higher doses. These results indicate both that the reinforced variability procedure can be used to isolate the specific effects of drugs of abuse on the variability of behavior, and that the specific impact of a given drug needs to be considered when interpreting pharmacological disruptions to operant task performance.

Lack of reliability in the disruption of cognitive performance following exposure to protons

A series of three replications were run to determine the reliability with which exposure to protons produces a disruption of cognitive performance, using a novel object recognition task and operant responding on an ascending fixed-ratio task. For the first two replications, rats were exposed to head-only exposures to 1000 MeV/n protons at the NASA Space Radiation Laboratory. For the third replication, subjects were given head-only or whole-body exposures to both 1000 and 150 MeV/n protons. The results were characterized by a lack of consistency in the effects of exposure to protons on the performance of these cognitive tasks, both within and between replications. The factors that might influence the lack of consistency and the implications for exploratory class missions are discussed.

Reward-dependent modulation of movement variability

Movement variability is often considered an unwanted byproduct of a noisy nervous system. However, variability can signal a form of implicit exploration, indicating that the nervous system is intentionally varying the motor commands in search of actions that yield the greatest success. Here, we investigated the role of the human basal ganglia in controlling reward-dependent motor variability as measured by trial-to-trial changes in performance during a reaching task. We designed an experiment in which the only performance feedback was success or failure and quantified how reach variability was modulated as a function of the probability of reward. In healthy controls, reach variability increased as the probability of reward decreased. Control of variability depended on the history of past rewards, with the largest trial-to-trial changes occurring immediately after an unrewarded trial. In contrast, in participants with Parkinson's disease, a known example of basal ganglia dysfunction, reward was a poor modulator of variability; that is, the patients showed an impaired ability to increase variability in response to decreases in the probability of reward. This was despite the fact that, after rewarded trials, reach variability in the patients was comparable to healthy controls. In summary, we found that movement variability is partially a form of exploration driven by the recent history of rewards. When the function of the human basal ganglia is compromised, the reward-dependent control of movement variability is impaired, particularly affecting the ability to increase variability after unsuccessful outcomes.

Music training relates to the development of neural mechanisms of selective auditory attention

•

Does music training shape the development of neural mechanisms of auditory attention?

•

We compared cortical responses to attended speech in child and adult musicians and nonmusicians.

•

Musician children and adults had less prefrontal auditory response variability during attention.

Selective attention decreases trial-to-trial variability in cortical auditory-evoked activity. This effect increases over the course of maturation, potentially reflecting the gradual development of selective attention and inhibitory control. Work in adults indicates that music training may alter the development of this neural response characteristic, especially over brain regions associated with executive control: in adult musicians, attention decreases variability in auditory-evoked responses recorded over prefrontal cortex to a greater extent than in nonmusicians. We aimed to determine whether this musician-associated effect emerges during childhood, when selective attention and inhibitory control are under development. We compared cortical auditory-evoked variability to attended and ignored speech streams in musicians and nonmusicians across three age groups: preschoolers, school-aged children and young adults. Results reveal that childhood music training is associated with reduced auditory-evoked response variability recorded over prefrontal cortex during selective auditory attention in school-aged child and adult musicians. Preschoolers, on the other hand, demonstrate no impact of selective attention on cortical response variability and no musician distinctions. This finding is consistent with the gradual emergence of attention during this period and may suggest no pre-existing differences in this attention-related cortical metric between children who undergo music training and those who do not.

The dopamine D2/D3 receptor agonist quinpirole increases checking-like behaviour in an operant observing response task with uncertain reinforcement: a novel possible model of OCD

Excessive checking is a common, debilitating symptom of obsessive-compulsive disorder (OCD). In an established rodent model of OCD checking behaviour, quinpirole (dopamine D2/3-receptor agonist) increased checking in open-field tests, indicating dopaminergic modulation of checking-like behaviours. We designed a novel operant paradigm for rats (observing response task (ORT)) to further examine cognitive processes underpinning checking behaviour and clarify how and why checking develops. We investigated i) how quinpirole increases checking, ii) dependence of these effects on D2/3 receptor function (following treatment with D2/3 receptor antagonist sulpiride) and iii) effects of reward uncertainty. In the ORT, rats pressed an 'observing' lever for information about the location of an 'active' lever that provided food reinforcement. High- and low-checkers (defined from baseline observing) received quinpirole (0.5mg/kg, 10 treatments) or vehicle. Parametric task manipulations assessed observing/checking under increasing task demands relating to reinforcement uncertainty (variable response requirement and active-lever location switching). Treatment with sulpiride further probed the pharmacological basis of long-term behavioural changes. Quinpirole selectively increased checking, both functional observing lever presses (OLPs) and non-functional extra OLPs (EOLPs). The increase in OLPs and EOLPs was long-lasting, without further quinpirole administration. Quinpirole did not affect the immediate ability to use information from checking. Vehicle and quinpirole-treated rats (VEH and QNP respectively) were selectively sensitive to different forms of uncertainty. Sulpiride reduced non-functional EOLPs in QNP rats but had no effect on functional OLPs. These data have implications for treatment of compulsive checking in OCD, particularly for serotonin-reuptake-inhibitor treatment-refractory cases, where supplementation with dopamine receptor antagonists may be beneficial.

Punishment-induced behavioral and neurophysiological variability reveals dopamine-dependent selection of kinematic movement parameters

Action selection describes the high-level process that selects between competing movements. In animals, behavioral variability is critical for the motor exploration required to select the action that optimizes reward and minimizes cost/punishment and is guided by dopamine (DA). The aim of this study was to test in humans whether low-level movement parameters are affected by punishment and reward in ways similar to high-level action selection. Moreover, we addressed the proposed dependence of behavioral and neurophysiological variability on DA and whether this may underpin the exploration of kinematic parameters. Participants performed an out-and-back index finger movement and were instructed that monetary reward and punishment were based on its maximal acceleration (MA). In fact, the feedback was not contingent on the participant's behavior but predetermined. Blocks highly biased toward punishment were associated with increased MA variability relative to blocks either with reward or without feedback. This increase in behavioral variability was positively correlated with neurophysiological variability, as measured by changes in corticospinal excitability with transcranial magnetic stimulation over the primary motor cortex. Following the administration of a DA antagonist, the variability associated with punishment diminished and the correlation between behavioral and neurophysiological variability no longer existed. Similar changes in variability were not observed when participants executed a predetermined MA, nor did DA influence resting neurophysiological variability. Thus, under conditions of punishment, DA-dependent processes influence the selection of low-level movement parameters. We propose that the enhanced behavioral variability reflects the exploration of kinematic parameters for less punishing, or conversely more rewarding, outcomes.