The combination of appetitive and aversive reinforcers and the nature of their interaction during auditory learning

Assessing reinforcing versus aversive consequences in a real-time secondhand smoke intervention

Few studies have examined the relative effectiveness of reinforcing versus aversive consequences at changing behavior in real-world environments. Real-time sensing devices makes it easier to investigate such questions, offering the potential to improve both intervention outcomes and theory. This research aims to describe the development of a real-time, operant theory-based secondhand smoke (SHS) intervention and compare the efficacy of aversive versus aversive plus reinforcement contingency systems. Indoor air particle monitors were placed in the households of 253 smokers for approximately three months. Participants were assigned to a measurement-only control group (N = 129) or one of the following groups: 1.) aversive only (AO, N = 71), with aversive audio/visual consequences triggered by the detection of elevated air particle measurements, or 2.) aversive plus reinforcement (AP, N = 53), with reinforcing consequences contingent on the absence of SHS added to the AO intervention. Residualized change ANCOVA analysis compared particle concentrations over time and across groups. Post-hoc pairwise comparisons were also performed. After controlling for Baseline, Post-Baseline daily particle counts (F = 6.42, p = 0.002), % of time >15,000 counts (F = 7.72, p < 0.001), and daily particle events (F = 4.04, p = 0.02) significantly differed by study group. Nearly all control versus AO/AP pair-wise comparisons were statistically significant. No significant differences were found for AO versus AP groups. The aversive feedback system reduced SHS, but adding reinforcing consequences did not further improve outcomes. The complexity of real-world environments requires the nuances of these two contingency systems continue to be explored, with this study demonstrating that real-time sensing technology can serve as a platform for such research.

Appetitive conditioning task in a shuttle box and its comparison with the active avoidance paradigm

The main features of the Shuttle Box Active Avoidance paradigm (e.g., the use of simple locomotor response as an operant and electrical current as a primary reinforcer) make this task easily automated. However, learning in this paradigm cannot be easily separated from the specificity of fear motivation. Punishment and negative reinforcement highly affect behavior in this task and complicate learning. In the present study, we describe a novel computer-controlled appetitive task in a shuttle box and compare it with active avoidance. The appetitive task was performed in the same shuttle box apparatus, additionally equipped with food dispensers in each compartment, and using a similar protocol. The reinforced reaction included the transition to the feeder in the opposite compartment in response to a stimulus. Animals mastered the appetitive task faster than the active avoidance task in the shuttle box. Other major differences between the models were the number and dynamics of intertrial responses (ITRs). Whereas in active avoidance the number of ITRs was low during learning, in the appetitive task rates were higher and they persisted throughout learning. Overall, the findings demonstrate some benefits of the appetitive task as a control condition to active avoidance: the use of a similar reaction and apparatus, no prior habituation, and fast acquisition.

Mismatch Negativity in Rat Auditory Cortex Represents the Empirical Salience of Sounds

Mismatch negativity (MMN) is an N-methyl-D-aspartic acid-mediated component and thus has been widely considered a major candidate biomarker of schizophrenia. However, at present, no direct evidence has linked the MMN response and aberrant salience processing reported in schizophrenia patients, i.e., whether MMN represents empirical salience of the deviant stimulus. To test the eligibility of the MMN response as a biomarker of schizophrenia, we hypothesized and verified that the MMN in the rat auditory cortex encodes empirical salience, which depends on both empirical rarity of sound and association of sound with salient events through classical conditioning. We exposed rats to sound or appetitive or aversive classical conditioning and extinction training of aversive conditioning, then recorded MMN from the auditory cortex. We focused on the frequency-dependent asymmetry of the MMN amplitude; increment frequency changes elicit asymmetrically larger MMN amplitudes than do decremental frequency changes. We found that sound exposure reversed this asymmetry in rats, indicating that MMN encodes the empirical rarity of deviant sounds. The MMN amplitude was larger in the appetitive or aversive classical conditioning groups, and smaller after extinction training following aversive conditioning. These results indicate that the MMN response reflects the empirical salience of sound. The present work first linked MMN with empirical salience processing and expands the possibilities of MMN as a strong clinical biomarker of schizophrenia.

Sound preference test in animal models of addicts and phobias

Biased or too strong preference for a particular object is often problematic, resulting in addiction and phobia. In animal models, alternative forced-choice tasks have been routinely used, but such preference test is far from daily situations that addicts or phobic are facing. In the present study, we developed a behavioral assay to evaluate the preference of sounds in rodents. In the assay, several sounds were presented according to the position of free-moving rats, and quantified the sound preference based on the behavior. A particular tone was paired with microstimulation to the ventral tegmental area (VTA), which plays central roles in reward processing, to increase sound preference. The behaviors of rats were logged during the classical conditioning for six days. Consequently, some behavioral indices suggest that rats search for the conditioned sound. Thus, our data demonstrated that quantitative evaluation of preference in the behavioral assay is feasible.

Selective Increase of Auditory Cortico-Striatal Coherence during Auditory-Cued Go/NoGo Discrimination Learning

Goal directed behavior and associated learning processes are tightly linked to neuronal activity in the ventral striatum. Mechanisms that integrate task relevant sensory information into striatal processing during decision making and learning are implicitly assumed in current reinforcement models, yet they are still weakly understood. To identify the functional activation of cortico-striatal subpopulations of connections during auditory discrimination learning, we trained Mongolian gerbils in a two-way active avoidance task in a shuttlebox to discriminate between falling and rising frequency modulated tones with identical spectral properties. We assessed functional coupling by analyzing the field-field coherence between the auditory cortex and the ventral striatum of animals performing the task. During the course of training, we observed a selective increase of functional coupling during Go-stimulus presentations. These results suggest that the auditory cortex functionally interacts with the ventral striatum during auditory learning and that the strengthening of these functional connections is selectively goal-directed.

Dopaminergic impact on local and global cortical circuit processing during learning

We have learned to detect, predict and behaviorally respond to important changes in our environment on short and longer time scales. Therefore, brains of humans and higher animals build upon a perceptual and semantic salience stored in their memories mainly generated by associative reinforcement learning. Functionally, the brain needs to extract and amplify a small number of features of sensory input with behavioral relevance to a particular situation in order to guide behavior. In this review, I argue that dopamine action, particularly in sensory cortex, orchestrates layer-dependent local and long-range cortical circuits integrating sensory associated bottom-up and semantically relevant top-down information, respectively. Available evidence reveals that dopamine thereby controls both the selection of perceptually or semantically salient signals as well as feedback processing from higher-order areas in the brain. Sensory cortical dopamine thereby governs the integration of selected sensory information within a behavioral context. This review proposes that dopamine enfolds this function by temporally distinct actions on particular layer-dependent local and global cortical circuits underlying the integration of sensory, and non-sensory cognitive and behavioral variables.

Stimulation of the Dopaminergic Midbrain as a Behavioral Reward in Instrumentally Conditioned Monkeys

BACKGROUND

Since the mesocortical dopaminergic system of rodents has several differences to that found in primate species, including humans, there is the need for more exhaustively studying causative relationships between activation/stimulation of the ventral tegmental area (VTA) and substantia nigra (SN) and behavior in monkeys.

OBJECTIVE

To gain causative relationships between VTA/SN stimulation and behavior, we investigated whether monkeys perform audiovisual (AV) tasks using brain stimulation reward (BSR) as the reinforcer, and how reward intensity affects performance during self-stimulation.

METHODS

Monkeys were required to touch a bar freely when self-stimulating or when instructed by an AV stimulus, to receive BSR.

RESULTS

We were able to train monkeys to successfully perform the AV task for BSR within three days. Self-stimulation revealed an increase in the bar touch rate when using higher electrical currents, with no ceiling effects observed. During a training session the touch rate decreased, often before the monkeys had received 1000 deliveries of BSR, suggesting satiation.

CONCLUSIONS

When BSR is applied directly to the VTA/SN, it can motivate monkeys to perform detection tasks, exhibit operant actions, and may be used as a substitute for fluid or food rewards. Monkeys ceased self-stimulation during a training session by their own volition, in contrast to work on rodents. This may be an important safety aspect for consideration in the development of electrical stimulation procedures for patients with dysfunctions of the dopaminergic system; thus, satiation may avert additional compulsions to already existing compulsive behaviors in patients.

Positive and negative reinforcement activate human auditory cortex

Prior studies suggest that reward modulates neural activity in sensory cortices, but less is known about punishment. We used functional magnetic resonance imaging and an auditory discrimination task, where participants had to judge the duration of frequency modulated tones. In one session correct performance resulted in financial gains at the end of the trial, in a second session incorrect performance resulted in financial loss. Incorrect performance in the rewarded as well as correct performance in the punishment condition resulted in a neutral outcome. The size of gains and losses was either low or high (10 or 50 Euro cent) depending on the direction of frequency modulation. We analyzed neural activity at the end of the trial, during reinforcement, and found increased neural activity in auditory cortex when gaining a financial reward as compared to gaining no reward and when avoiding financial loss as compared to receiving a financial loss. This was independent on the size of gains and losses. A similar pattern of neural activity for both gaining a reward and avoiding a loss was also seen in right middle temporal gyrus, bilateral insula and pre-supplemental motor area, here however neural activity was lower after correct responses compared to incorrect responses. To summarize, this study shows that the activation of sensory cortices, as previously shown for gaining a reward is also seen during avoiding a loss.

Electrical stimulation of lateral habenula during learning: frequency-dependent effects on acquisition but not retrieval of a two-way active avoidance response

The lateral habenula (LHb) is an epithalamic structure involved in signaling reward omission and aversive stimuli, and it inhibits dopaminergic neurons during motivated behavior. Less is known about LHb involvement in the acquisition and retrieval of avoidance learning. Our previous studies indicated that brief electrical stimulation of the LHb, time-locked to the avoidance of aversive footshock (presumably during the positive affective “relief” state that occurs when an aversive outcome is averted), inhibited the acquisition of avoidance learning. In the present study, we used the same paradigm to investigate different frequencies of LHb stimulation. The effect of 20 Hz vs. 50 Hz vs. 100 Hz stimulation was investigated during two phases, either during acquisition or retrieval in Mongolian gerbils. The results indicated that 50 Hz, but not 20 Hz, was sufficient to produce a long-term impairment in avoidance learning, and was somewhat more effective than 100 Hz in this regard. None of the stimulation parameters led to any effects on retrieval of avoidance learning, nor did they affect general motor activity. This suggests that, at frequencies in excess of the observed tonic firing rates of LHb neurons (>1–20 Hz), LHb stimulation may serve to interrupt the consolidation of new avoidance memories. However, these stimulation parameters are not capable of modifying avoidance memories that have already undergone extensive consolidation.

Motivational salience and genetic variability of dopamine D2 receptor expression interact in the modulation of interference processing

Dopamine has been implicated in the fine-tuning of complex cognitive and motor function and also in the anticipation of future rewards. This dual function of dopamine suggests that dopamine might be involved in the generation of active motivated behavior. The DRD2 TaqIA polymorphism of the dopamine D2 receptor gene (rs1800497) has previously been suggested to affect striatal function with carriers of the less common A1 allele exhibiting reduced striatal D2 receptor density and increased risk for addiction. Here we aimed to investigate the influences of DRD2 TaqIA genotype on the modulation of interference processing by reward and punishment. Forty-six young, healthy volunteers participated in a behavioral experiment, and 32 underwent functional magnetic resonance imaging (fMRI). Participants performed a flanker task with a motivation manipulation (monetary reward, monetary loss, neither, or both). Reaction times (RTs) were shorter in motivated flanker trials, irrespective of congruency. In the fMRI experiment motivation was associated with reduced prefrontal activation during incongruent vs. congruent flanker trials, possibly reflecting increased processing efficiency. DRD2 TaqIA genotype did not affect overall RTs, but interacted with motivation on the congruency-related RT differences, with A1 carriers showing smaller interference effects to reward alone and A2 homozygotes exhibiting a specific interference reduction during combined reward (REW) and punishment trials (PUN). In fMRI, anterior cingulate activity showed a similar pattern of genotype-related modulation. Additionally, A1 carriers showed increased anterior insula activation relative to A2 homozygotes. Our results point to a role for genetic variations of the dopaminergic system in individual differences of cognition-motivation interaction.

Complexity and competition in appetitive and aversive neural circuits

Decision-making often involves using sensory cues to predict possible rewarding or punishing reinforcement outcomes before selecting a course of action. Recent work has revealed complexity in how the brain learns to predict rewards and punishments. Analysis of neural signaling during and after learning in the amygdala and orbitofrontal cortex, two brain areas that process appetitive and aversive stimuli, reveals a dynamic relationship between appetitive and aversive circuits. Specifically, the relationship between signaling in appetitive and aversive circuits in these areas shifts as a function of learning. Furthermore, although appetitive and aversive circuits may often drive opposite behaviors – approaching or avoiding reinforcement depending upon its valence – these circuits can also drive similar behaviors, such as enhanced arousal or attention; these processes also may influence choice behavior. These data highlight the formidable challenges ahead in dissecting how appetitive and aversive neural circuits interact to produce a complex and nuanced range of behaviors.

The role of dopamine in the context of aversive stimuli with particular reference to acoustically signaled avoidance learning

Learning from punishment is a powerful means for behavioral adaptation with high relevance for various mechanisms of self-protection. Several studies have explored the contribution of released dopamine (DA) or responses of DA neurons on reward seeking using rewards such as food, water, and sex. Phasic DA signals evoked by rewards or conditioned reward predictors are well documented, as are modulations of these signals by such parameters as reward magnitude, probability, and deviation of actually occurring from expected rewards. Less attention has been paid to DA neuron firing and DA release in response to aversive stimuli, and the prediction and avoidance of punishment. In this review, we first focus on DA changes in response to aversive stimuli as measured by microdialysis and voltammetry followed by the change in electrophysiological signatures by aversive stimuli and fearful events. We subsequently focus on the role of DA and effect of DA manipulations on signaled avoidance learning, which consists of learning the significance of a warning cue through Pavlovian associations and the execution of an instrumental avoidance response. We present a coherent framework utilizing the data on microdialysis, voltammetry, electrophysiological recording, electrical brain stimulation, and behavioral analysis. We end by outlining current gaps in the literature and proposing future directions aimed at incorporating technical and conceptual progress to understand the involvement of reward circuit on punishment based decisions.

Task reward structure shapes rapid receptive field plasticity in auditory cortex

As sensory stimuli and behavioral demands change, the attentive brain quickly identifies task-relevant stimuli and associates them with appropriate motor responses. The effects of attention on sensory processing vary across task paradigms, suggesting that the brain may use multiple strategies and mechanisms to highlight attended stimuli and link them to motor action. To better understand factors that contribute to these variable effects, we studied sensory representations in primary auditory cortex (A1) during two instrumental tasks that shared the same auditory discrimination but required different behavioral responses, either approach or avoidance. In the approach task, ferrets were rewarded for licking a spout when they heard a target tone amid a sequence of reference noise sounds. In the avoidance task, they were punished unless they inhibited licking to the target. To explore how these changes in task reward structure influenced attention-driven rapid plasticity in A1, we measured changes in sensory neural responses during behavior. Responses to the target changed selectively during both tasks but did so with opposite sign. Despite the differences in sign, both effects were consistent with a general neural coding strategy that maximizes discriminability between sound classes. The dependence of the direction of plasticity on task suggests that representations in A1 change not only to sharpen representations of task-relevant stimuli but also to amplify responses to stimuli that signal aversive outcomes and lead to behavioral inhibition. Thus, top-down control of sensory processing can be shaped by task reward structure in addition to the required sensory discrimination.

A behavioral framework to guide research on central auditory development and plasticity

The auditory CNS is influenced profoundly by sounds heard during development. Auditory deprivation and augmented sound exposure can each perturb the maturation of neural computations as well as their underlying synaptic properties. However, we have learned little about the emergence of perceptual skills in these same model systems, and especially how perception is influenced by early acoustic experience. Here, we argue that developmental studies must take greater advantage of behavioral benchmarks. We discuss quantitative measures of perceptual development and suggest how they can play a much larger role in guiding experimental design. Most importantly, including behavioral measures will allow us to establish empirical connections among environment, neural development, and perception.

Effects of ventral tegmental area stimulation on the acquisition and long-term retention of active avoidance learning

The development of avoidance learning depends on dopamine release in forebrain regions. Previous studies indicated that rewarding brain stimulation facilitated two-way active avoidance learning. However, it is not clear whether the temporal relationship of brain stimulation to the training session (before, during or after) is important. To investigate the role of stimulation condition (no stimulation, self-stimulation only, or self-stimulation plus avoidance stimulation) and sequence of self-stimulation training (before or after avoidance training), we used a 3×2 factorial design, in which every level of stimulation was paired with every level of sequence for a total of 6 different groups. The results suggest that self-stimulation either before or after avoidance learning improved acquisition performance, but acquisition was maximal when stimulation was also given during acquisition trials. Importantly, the sequence of self-stimulation (before or after each acquisition session) was irrelevant to this beneficial effect. However, stimulation had no apparent effect on long-term retention when tested 10 days later under conditions of no stimulation, except that the performance of the group that had previously received avoidance-contingent stimulation deteriorated over the course of 60 trials. This may reflect frustration from the omission of expected reward. These results are relevant for optimizing brain stimulation to improve learning.

Behavioral semantics of learning and crossmodal processing in auditory cortex: the semantic processor concept

Two phenomena of auditory cortex activity have recently attracted attention, namely that the primary field can show different types of learning-related changes of sound representation and that during learning even this early auditory cortex is under strong multimodal influence. Based on neuronal recordings in animal auditory cortex during instrumental tasks, in this review we put forward the hypothesis that these two phenomena serve to derive the task-specific meaning of sounds by associative learning. To understand the implications of this tenet, it is helpful to realize how a behavioral meaning is usually derived for novel environmental sounds. For this purpose, associations with other sensory, e.g. visual, information are mandatory to develop a connection between a sound and its behaviorally relevant cause and/or the context of sound occurrence. This makes it plausible that in instrumental tasks various non-auditory sensory and procedural contingencies of sound generation become co-represented by neuronal firing in auditory cortex. Information related to reward or to avoidance of discomfort during task learning, that is essentially non-auditory, is also co-represented. The reinforcement influence points to the dopaminergic internal reward system, the local role of which for memory consolidation in auditory cortex is well-established. Thus, during a trial of task performance, the neuronal responses to the sounds are embedded in a sequence of representations of such non-auditory information. The embedded auditory responses show task-related modulations of auditory responses falling into types that correspond to three basic logical classifications that may be performed with a perceptual item, i.e. from simple detection to discrimination, and categorization. This hierarchy of classifications determine the semantic "same-different" relationships among sounds. Different cognitive classifications appear to be a consequence of learning task and lead to a recruitment of different excitatory and inhibitory mechanisms and to distinct spatiotemporal metrics of map activation to represent a sound. The described non-auditory firing and modulations of auditory responses suggest that auditory cortex, by collecting all necessary information, functions as a "semantic processor" deducing the task-specific meaning of sounds by learning.