Why the carrot is more effective than the stick: different dynamics of punishment memory and reward memory and its possible biological basis

Aversive social learning with a dead conspecific is achieved by Pavlovian conditioning in crickets

Social learning, learning from other individuals, has been demonstrated in many animals, including insects, but its detailed neural mechanisms remain virtually unknown. We showed that crickets (Gryllus bimaculatus) exhibit aversive social learning with a dead conspecific. When a learner cricket was trained to observe a dead cricket on a drinking apparatus, the learner avoided the odor of that apparatus thereafter. Here we investigated the hypothesis that this social learning is achieved by first-order Pavlovian conditioning of an odor (conditioned stimulus) and a dead conspecific (unconditioned stimulus, US). Injection of a dopamine receptor antagonist (flupentixol) before training or testing impaired the learning or execution of the response to the learned odor, as we reported in aversive non-social Pavlovian conditioning in crickets. Moreover, crickets that were trained with a dead conspecific and then received revaluation of the dead conspecific by pairing it with water reward exhibited an appetitive conditioned response (CR) to the odor paired with the dead conspecific. This suggests that execution of the CR is governed by the current value of the US as in non-social Pavlovian conditioning. In addition, we previously suggested that appetitive social learning with a living conspecific is based on second-order conditioning (SOC), and here we showed that SOC is achieved when crickets experienced pairing of a dead conspecific with water reward before experiencing social learning training with a dead conspecific. We conclude that social learning with a dead conspecific is based on Pavlovian conditioning and that this learning can be extended to second-order social learning.

Visuo-spatial compound stimuli discrimination with (Gryllus pennsylvanicus) in two-choices rewarding learning tasks

This paper proposes an experimental protocol allowing Gryllus pennsylvanicus to discriminate an A-A and A-B motif pairs of compound visual stimuli. Specifically, this study consists in an operant conditioning procedure including a dichotomous Y-maze, two different pairs of compound visual colored cues and a water reward. Results are conclusive for this visuo-spatial regularities study,(Gryllus pennsylvanicus) were able to significantly discriminate between the two compound visual patterns and learned the association with the reinforcer.

Independent operations of appetitive and aversive conditioning systems lead to simultaneous production of conflicting memories in an insect

Pavlovian conditioning is a ubiquitous form of associative learning that enables animals to remember appetitive and aversive experiences. Animals possess appetitive and aversive conditioning systems that memorize and retrieve appetitive and aversive experiences. Here, we addressed a question of whether integration of competing appetitive and aversive information takes place during the encoding of the experience or during memory retrieval. We developed novel experimental procedures to address this question using crickets (Gryllus bimaculatus), which allowed selective blockade of the expression of appetitive and aversive memories by injecting octopamine and dopamine receptor antagonists. We conditioned an odour (conditioned stimulus 1, CS1) with water and then with sodium chloride solution. At 24 h after conditioning, crickets retained both appetitive and aversive memories, and the memories were integrated to produce a conditioned response (CR). Importantly, when a visual pattern (CS2) was conditioned with CS1, appetitive and aversive memories formed simultaneously. This indicates that appetitive and aversive second-order conditionings are achieved at the same time. The memories were integrated for producing a conditioned response. We conclude that appetitive and aversive conditioning systems operate independently to form parallel appetitive and aversive memories, which compete to produce learned behaviour in crickets.

The influence of psychological traits and prior experience on treatment expectations

BACKGROUND

Placebo and nocebo responses are modulated by the treatment expectations of participants and patients. However, interindividual differences predicting treatment expectations and placebo responses are unclear. In this large-scale pooled analysis, we aim to investigate the influence of psychological traits and prior experiences on treatment expectations.

METHODS

This paper analyses data from six different placebo studies (total n = 748). In all studies, participants' sociodemographic information, treatment expectations and prior treatment experiences and traits relating to stress, somatization, depression and anxiety, the Big Five and behavioral inhibition and approach tendencies were assessed using the same established questionnaires. Correlation coefficients and structural equation models were calculated to investigate the relationship between trait variables and expectations.

RESULTS

We found small positive correlations between side effect expectations and improvement expectations (r = 0.187), perceived stress (r = 0.154), somatization (r = 0.115), agitation (r = 0.108), anhedonia (r = 0.118), and dysthymia (r = 0.118). In the structural equation model previous experiences emerged as the strongest predictors of improvement (β = 0.32, p = .005), worsening (β = -0.24, p = .005) and side effect expectations (β = 0.47, p = .005). Traits related to positive affect (β = - 0.09; p = .007) and negative affect (β = 0.04; p = .014) were associated with side effect expectations.

DISCUSSION

This study is the first large analysis to investigate the relationship between traits, prior experiences and treatment expectations. Exploratory analyses indicate that experiences of symptom improvement are associated with improvement and worsening expectations, while previous negative experiences are only related to side effect expectations. Additionally, a proneness to experience negative affect may be a predictor for side effect expectation and thus mediate the occurrence of nocebo responses.

Dopaminergic signaling supports auditory social learning

Explicit rewards are commonly used to reinforce a behavior, a form of learning that engages the dopaminergic neuromodulatory system. In contrast, skill acquisition can display dramatic improvements from a social learning experience, even though the observer receives no explicit reward. Here, we test whether a dopaminergic signal contributes to social learning in naïve gerbils that are exposed to, and learn from, a skilled demonstrator performing an auditory discrimination task. Following five exposure sessions, naïve observer gerbils were allowed to practice the auditory task and their performance was assessed across days. We first tested the effect of an explicit food reward in the observer's compartment that was yoked to the demonstrator's performance during exposure sessions. Naïve observer gerbils with the yoked reward learned the discrimination task significantly faster, as compared to unrewarded observers. The effect of this explicit reward was abolished by administration of a D1/D5 dopamine receptor antagonist during the exposure sessions. Similarly, the D1/D5 antagonist reduced the rate of learning in unrewarded observers. To test whether a dopaminergic signal was sufficient to enhance social learning, we administered a D1/D5 receptor agonist during the exposure sessions in which no reward was present and found that the rate of learning occurred significantly faster. Finally, a quantitative analysis of vocalizations during the exposure sessions suggests one behavioral strategy that contributes to social learning. Together, these results are consistent with a dopamine-dependent reward signal during social learning.

What Is Learned in Pavlovian Conditioning in Crickets? Revisiting the S-S and S-R Learning Theories

In Pavlovian conditioning in mammals, two theories have been proposed for associations underlying conditioned responses (CRs). One theory, called S-S theory, assumes an association between a conditioned stimulus (CS) and internal representation of an unconditioned stimulus (US), allowing the animal to adjust the CR depending on the current value of the US. The other theory, called S-R theory, assumes an association or connection between the CS center and the CR center, allowing the CS to elicit the CR. Whether these theories account for Pavlovian conditioning in invertebrates has remained unclear. In this article, results of our studies in the cricket Gryllus bimaculatus are reviewed. We showed that after a standard amount of Pavlovian training, crickets exhibited no response to odor CS when water US was devalued by providing it until satiation, whereas after extended training, they exhibited a CR after US devaluation. An increase of behavioral automaticity by extended training has not been reported in Pavlovian conditioning in any other animals, but it has been documented in instrumental conditioning in mammals. Our pharmacological analysis suggested that octopamine neurons mediate US (water) value signals and control execution of the CR after standard training. The control, however, diminishes with extension of training and hence the CR becomes insensitive to the US value. We also found that the nature of the habitual response after extended Pavlovian training in crickets is not the same as that after extended instrumental training in mammals concerning the context specificity. Adaptive significance and evolutionary implications for our findings are discussed.

MicroRNA-277 regulates dopa decarboxylase to control larval-pupal and pupal-adult metamorphosis of Helicoverpa armigera

Insect metamorphosis is a complex process involving many metabolic pathways, such as juvenile hormones and molting hormones, bioamines, microRNAs (miRNAs), etc. However, relatively little is known about the biogenic amines and their miRNAs to regulate cotton bollworm metamorphosis. Here we show that one miRNA, miR-277 regulates larval-pupal and pupal-adult metamorphosis of cotton bollworm by targeting the 3'UTR of Dopa decarboxylase (DDC), a synthetic catalytic enzyme of dopamine. Injection of miR-277 agomir inhibited the expression of DDC at the mRNA and protein levels, leading to defects in the pupation and emergence of H. armigera that was consistent with the phenotype obtained by injection of DDC double-stranded RNA (dsRNA). Injection of miR-277 antagomir induced the mRNA and protein expression of DDC and rescued the phenotype of pupation failure caused by DDC gene silencing. Unexpectedly, miR-277 antagomir can also cause failure of emergence of H. armigera and both agomir and antagomir of miR-277 injection could cause abnormal phenotypes in wing veins. This study reveals that elaborate regulation of miRNA and its target gene expression is prerequisite for insect development, which provides a new insight to study the developmental mechanisms of insect wing veins.

Tyrosine hydroxylase-immunoreactive neurons in the mushroom body of the field cricket, Gryllus bimaculatus

The mushroom body of the insect brain participates in processing and integrating multimodal sensory information and in various forms of learning. In the field cricket, Gryllus bimaculatus, dopamine plays a crucial role in aversive memory formation. However, the morphologies of dopamine neurons projecting to the mushroom body and their potential target neurons, the Kenyon cells, have not been characterized. Golgi impregnations revealed two classes of Kenyon cells (types I and II) and five different types of extrinsic fibers in the mushroom body. Type I cells, which are further divided into two subtypes (types I core and I surface), extend their dendrites into the anterior calyx, whereas type II cells extend many bushy dendritic branches into the posterior calyx. Axons of the two classes bifurcate between the pedunculus and lobes to form the vertical, medial and γ lobes. Immunocytochemistry to tyrosine hydroxylase (TH), a rate-limiting enzyme in dopamine biosynthesis, revealed the following four distinct classes of neurons: (1) TH-SLP projecting to the distal vertical lobe; (2) TH-IP1 extending to the medial and γ lobes; (3) TH-IP2 projecting to the basal vertical lobe; and (4) a multiglomerular projection neuron invading the anterior calyx and the lateral horn (TH-MPN). We previously proposed a model in the field cricket in which the efficiency of synapses from Kenyon cells transmitting a relevant sensory stimulus to output neurons commanding an appropriate behavioral reaction can be modified by dopaminergic neurons mediating aversive signals and here, we provide putative neural substrates for the cricket's aversive learning. These will be instrumental in understanding the principle of aversive memory formation in this model species.

Application of a Prediction Error Theory to Pavlovian Conditioning in an Insect

Elucidation of the conditions in which associative learning occurs is a critical issue in neuroscience and comparative psychology. In Pavlovian conditioning in mammals, it is thought that the discrepancy, or error, between the actual reward and the predicted reward determines whether learning occurs. This theory stems from the finding of Kamin’s blocking effect, in which after pairing of a stimulus with an unconditioned stimulus (US), conditioning of a second stimulus is blocked when the two stimuli are presented in compound and paired with the same US. Whether this theory is applicable to any species of invertebrates, however, has remained unknown. We first showed blocking and one-trial blocking of Pavlovian conditioning in the cricket Gryllus bimaculatus, which supported the Rescorla–Wagner model but not attentional theories, the major competitive error-correction learning theories to account for blocking. To match the prediction error theory, a neural circuit model was proposed, and prediction from the model was tested: the results were consistent with the Rescorla–Wagner model but not with the retrieval theory, another competitive theory to account for blocking. The findings suggest that the Rescorla–Wagner model best accounts for Pavlovian conditioning in crickets and that the basic computation rule underlying Pavlovian conditioning in crickets is the same to those suggested in mammals. Moreover, results of pharmacological studies in crickets suggested that octopamine and dopamine mediate prediction error signals in appetitive and aversive conditioning, respectively. This was in contrast to the notion that dopamine mediates appetitive prediction error signals in mammals. The functional significance and evolutionary implications of these findings are discussed.

Signaling Pathways for Long-Term Memory Formation in the Cricket

Unraveling the molecular mechanisms underlying memory formation in insects and a comparison with those of mammals will contribute to a further understanding of the evolution of higher-brain functions. As it is for mammals, insect memory can be divided into at least two distinct phases: protein-independent short-term memory and protein-dependent long-term memory (LTM). We have been investigating the signaling pathway of LTM formation by behavioral-pharmacological experiments using the cricket Gryllus bimaculatus, whose olfactory learning and memory abilities are among the highest in insect species. Our studies revealed that the NO-cGMP signaling pathway, CaMKII and PKA play crucial roles in LTM formation in crickets. These LTM formation signaling pathways in crickets share a number of attributes with those of mammals, and thus we conclude that insects, with relatively simple brain structures and neural circuitry, will also be beneficial in exploratory experiments to predict the molecular mechanisms underlying memory formation in mammals.

Selection for associative learning of color stimuli reveals correlated evolution of this learning ability across multiple stimuli and rewards

We are only starting to understand how variation in cognitive ability can result from local adaptations to environmental conditions. A major question in this regard is to what extent selection on cognitive ability in a specific context affects that ability in general through correlated evolution. To address this question, we performed artificial selection on visual associative learning in female Nasonia vitripennis wasps. Using appetitive conditioning in which a visual stimulus was offered in association with a host reward, the ability to learn visual associations was enhanced within 10 generations of selection. To test for correlated evolution affecting this form of learning, the ability to readily form learned associations in females was also tested using an olfactory instead of a visual stimulus in the appetitive conditioning. Additionally, we assessed whether the improved associative learning ability was expressed across sexes by color-conditioning males with a mating reward. Both females and males from the selected lines consistently demonstrated an increased associative learning ability compared to the control lines, independent of learning context or conditioned stimulus. No difference in relative volume of brain neuropils was detected between the selected and control lines.

Visual discrimination transfer and modulation by biogenic amines in honeybees

For more than a century, visual learning and memory has been studied in the honeybee Apis mellifera using operant appetitive conditioning. Although honeybees show impressive visual learning capacities in this well-established protocol, operant training of free-flying animals can hardly be combined with invasive protocols for studying the neurobiological basis of visual learning. In view of that, different efforts have been made to develop new classical conditioning protocols for studying visual learning in harnessed honeybees, though learning performances remain considerably poorer than those obtained in free-flying animals. Here we investigated the ability of honeybees to use visual information acquired during classical conditioning in a new operant context. We performed differential visual conditioning of the proboscis extension reflex (PER) followed by visual orientation tests in Y-maze. Classical conditioning and Y-maze retention tests were performed using a same pair of perceptually isoluminant monochromatic stimuli, to avoid the influence of phototaxis during free-flying orientation. Visual discrimination transfer was clearly observed, with pre-trained honeybees significantly orienting their flights towards the former positive conditioned stimulus (CS+). We thus show that visual memories acquired by honeybees are resistant to context changes between conditioning and retention test. We combined this visual discrimination approach with selective pharmacological injections to evaluate the effect of dopamine and octopamine in appetitive visual learning. Both octopaminergic and dopaminergic antagonists impaired visual discrimination performances, suggesting that both these biogenic amines modulate appetitive visual learning in honeybees. Our study brings new insights into cognitive and neurobiological mechanisms underlying visual learning in honeybees.

Roles of Octopamine and Dopamine Neurons for Mediating Appetitive and Aversive Signals in Pavlovian Conditioning in Crickets

Revealing neural systems that mediate appetite and aversive signals in associative learning is critical for understanding the brain mechanisms controlling adaptive behavior in animals. In mammals, it has been shown that some classes of dopamine neurons in the midbrain mediate prediction error signals that govern the learning process, whereas other classes of dopamine neurons control execution of learned actions. In this review, based on the results of our studies on Pavlovian conditioning in the cricket Gryllus bimaculatus and by referring to the findings in honey bees and fruit-flies, we argue that comparable aminergic systems exist in the insect brain. We found that administrations of octopamine (the invertebrate counterpart of noradrenaline) and dopamine receptor antagonists impair conditioning to associate an olfactory or visual conditioned stimulus (CS) with water or sodium chloride solution (appetitive or aversive unconditioned stimulus, US), respectively, suggesting that specific octopamine and dopamine neurons mediate appetitive and aversive signals, respectively, in conditioning in crickets. These findings differ from findings in fruit-flies. In fruit-flies, appetitive and aversive signals are mediated by different dopamine neuron subsets, suggesting diversity in neurotransmitters mediating appetitive signals in insects. We also found evidences of “blocking” and “auto-blocking” phenomena, which suggested that the prediction error, the discrepancy between actual US and predicted US, governs the conditioning in crickets and that octopamine neurons mediate prediction error signals for appetitive US. Our studies also showed that activations of octopamine and dopamine neurons are needed for the execution of an appetitive conditioned response (CR) and an aversive CR, respectively, and we, thus, proposed that these neurons mediate US prediction signals that drive appetitive and aversive CRs. Our findings suggest that the basic principles of functioning of aminergic systems in associative learning, i.e., to transmit prediction error signals for conditioning and to convey US prediction signals for execution of CR, are conserved among insects and mammals, on account of the fact that the organization of the insect brain is much simpler than that of the mammalian brain. Further investigation of aminergic systems that govern associative learning in insects should lead to a better understanding of commonalities and diversities of computational rules underlying associative learning in animals.

Roles of dopamine neurons in mediating the prediction error in aversive learning in insects

In associative learning in mammals, it is widely accepted that the discrepancy, or error, between actual and predicted reward determines whether learning occurs. The prediction error theory has been proposed to account for the finding of a blocking phenomenon, in which pairing of a stimulus X with an unconditioned stimulus (US) could block subsequent association of a second stimulus Y to the US when the two stimuli were paired in compound with the same US. Evidence for this theory, however, has been imperfect since blocking can also be accounted for by competitive theories. We recently reported blocking in classical conditioning of an odor with water reward in crickets. We also reported an "auto-blocking" phenomenon in appetitive learning, which supported the prediction error theory and rejected alternative theories. The presence of auto-blocking also suggested that octopamine neurons mediate reward prediction error signals. Here we show that blocking and auto-blocking occur in aversive learning to associate an odor with salt water (US) in crickets, and our results suggest that dopamine neurons mediate aversive prediction error signals. We conclude that the prediction error theory is applicable to both appetitive learning and aversive learning in insects.

Development rate rather than social environment influences cognitive performance in Australian black field crickets, Teleogryllus commodus

Cognitive functioning is vital for enabling animals of all taxa to optimise their chances of survival and reproductive success. Learning and memory in particular are drivers of many evolutionary processes. In this study, we examine how developmental plasticity can affect cognitive ability by exploring the role the early social environment has on problem solving ability and learning of female black field crickets, Teleogryllus commodus. We used two learning paradigms, an analog of the Morris water maze and a novel linear maze, to examine cognitive differences between individuals reared in two acoustic treatments: silence or calling. Although there was no evidence of learning or memory, individuals that took longer to mature solved the Morris water maze more quickly. Our results suggest that increased investment into cognitive development is likely associated with increased development time during immature stages. Inconsistent individual performance and motivation during the novel linear maze task highlights the difficulties of designing ecologically relevant learning tasks within a lab setting. The role of experimental design in understanding cognitive ability and learning in more natural circumstances is discussed.

Factors influencing aversive learning in the oriental fruit fly, Bactrocera dorsalis

Parameters such as the intensity of conditioned and unconditioned stimuli, the inter-trial interval, and starvation time can influence learning. In this study, the parameters that govern aversive learning in the oriental fruit fly, Bactrocera dorsalis, a serious pest of fruits and vegetables, were examined. Male flies were trained to associate the attractive odorant methyl eugenol, a male lure, with a food punishment, sodium chloride solution, and the conditioned suppression of the proboscis-extension response was investigated. We found that high methyl eugenol concentrations support a stronger association. With increasing concentrations of sodium chloride solution, a steady decrease of proboscis-extension response during six training trials was observed. A high level of learning was achieved with an inter-trial interval of 1-10 min. However, extending the inter-trial interval to 15 min led to reduced learning. No effect of physiological status (starvation time) on learning performance was detected, nor was any non-associative learning effect induced by the repeat presentation of odor or punishment alone. The memory formed after six training trials could be retained for at least 3 h. Our results indicate that aversive learning by oriental fruit flies can be affected by odor, punishment concentration and inter-trial interval.

Activation of NO-cGMP Signaling Rescues Age-Related Memory Impairment in Crickets

Age-related memory impairment (AMI) is a common feature and a debilitating phenotype of brain aging in many animals. However, the molecular mechanisms underlying AMI are still largely unknown. The cricket Gryllus bimaculatus is a useful experimental animal for studying age-related changes in learning and memory capability; because the cricket has relatively short life-cycle and a high capability of olfactory learning and memory. Moreover, the molecular mechanisms underlying memory formation in crickets have been examined in detail. In the present study, we trained male crickets of different ages by multiple-trial olfactory conditioning to determine whether AMI occurs in crickets. Crickets 3 weeks after the final molt (3-week-old crickets) exhibited levels of retention similar to those of 1-week-old crickets at 30 min or 2 h after training; however they showed significantly decreased levels of 1-day retention, indicating AMI in long-term memory (LTM) but not in anesthesia-resistant memory (ARM) in olfactory learning of crickets. Furthermore, 3-week-old crickets injected with a nitric oxide (NO) donor, a cyclic GMP (cGMP) analog or a cyclic AMP (cAMP) analog into the hemolymph before conditioning exhibited a normal level of LTM, the same level as that in 1-week-old crickets. The rescue effect by NO donor or cGMP analog injection was absent when the crickets were injected after the conditioning. For the first time, an NO donor and a cGMP analog were found to antagonize the age-related impairment of LTM formation, suggesting that deterioration of NO synthase (NOS) or molecules upstream of NOS activation is involved in brain-aging processes.

Dopamine- and Tyrosine Hydroxylase-Immunoreactive Neurons in the Brain of the American Cockroach, Periplaneta americana

The catecholamine dopamine plays several vital roles in the central nervous system of many species, but its neural mechanisms remain elusive. Detailed neuroanatomical characterization of dopamine neurons is a prerequisite for elucidating dopamine’s actions in the brain. In the present study, we investigated the distribution of dopaminergic neurons in the brain of the American cockroach, Periplaneta americana, using two antisera: 1) an antiserum against dopamine, and 2) an antiserum against tyrosine hydroxylase (TH, an enzyme required for dopamine synthesis), and identified about 250 putatively dopaminergic neurons. The patterns of dopamine- and TH-immunoreactive neurons were strikingly similar, suggesting that both antisera recognize the same sets of “dopaminergic” neurons. The dopamine and TH antibodies intensively or moderately immunolabeled prominent brain neuropils, e.g. the mushroom body (memory center), antennal lobe (first-order olfactory center) and central complex (motor coordination center). All subdivisions of the mushroom body exhibit both dopamine and TH immunoreactivity. Comparison of immunolabeled neurons with those filled by dye injection revealed that a group of immunolabeled neurons with cell bodies near the calyx projects into a distal region of the vertical lobe, which is a plausible site for olfactory memory formation in insects. In the antennal lobe, ordinary glomeruli as well as macroglomeruli exhibit both dopamine and TH immunoreactivity. It is noteworthy that the dopamine antiserum labeled tiny granular structures inside the glomeruli whereas the TH antiserum labeled processes in the marginal regions of the glomeruli, suggesting a different origin. In the central complex, all subdivisions excluding part of the noduli and protocerebral bridge exhibit both dopamine and TH immunoreactivity. These anatomical findings will accelerate our understanding of dopaminergic systems, specifically in neural circuits underlying aversive memory formation and arousal, in insects.

Roles of OA1 octopamine receptor and Dop1 dopamine receptor in mediating appetitive and aversive reinforcement revealed by RNAi studies

Revealing reinforcing mechanisms in associative learning is important for elucidation of brain mechanisms of behavior. In mammals, dopamine neurons are thought to mediate both appetitive and aversive reinforcement signals. Studies using transgenic fruit-flies suggested that dopamine neurons mediate both appetitive and aversive reinforcements, through the Dop1 dopamine receptor, but our studies using octopamine and dopamine receptor antagonists and using Dop1 knockout crickets suggested that octopamine neurons mediate appetitive reinforcement and dopamine neurons mediate aversive reinforcement in associative learning in crickets. To fully resolve this issue, we examined the effects of silencing of expression of genes that code the OA1 octopamine receptor and Dop1 and Dop2 dopamine receptors by RNAi in crickets. OA1-silenced crickets exhibited impairment in appetitive learning with water but not in aversive learning with sodium chloride solution, while Dop1-silenced crickets exhibited impairment in aversive learning but not in appetitive learning. Dop2-silenced crickets showed normal scores in both appetitive learning and aversive learning. The results indicate that octopamine neurons mediate appetitive reinforcement via OA1 and that dopamine neurons mediate aversive reinforcement via Dop1 in crickets, providing decisive evidence that neurotransmitters and receptors that mediate appetitive reinforcement indeed differ among different species of insects.

Knockout crickets for the study of learning and memory: Dopamine receptor Dop1 mediates aversive but not appetitive reinforcement in crickets

Elucidation of reinforcement mechanisms in associative learning is an important subject in neuroscience. In mammals, dopamine neurons are thought to play critical roles in mediating both appetitive and aversive reinforcement. Our pharmacological studies suggested that octopamine and dopamine neurons mediate reward and punishment, respectively, in crickets, but recent studies in fruit-flies concluded that dopamine neurons mediates both reward and punishment, via the type 1 dopamine receptor Dop1. To resolve the discrepancy between studies in different insect species, we produced Dop1 knockout crickets using the CRISPR/Cas9 system and found that they are defective in aversive learning with sodium chloride punishment but not appetitive learning with water or sucrose reward. The results suggest that dopamine and octopamine neurons mediate aversive and appetitive reinforcement, respectively, in crickets. We suggest unexpected diversity in neurotransmitters mediating appetitive reinforcement between crickets and fruit-flies, although the neurotransmitter mediating aversive reinforcement is conserved. This study demonstrates usefulness of the CRISPR/Cas9 system for producing knockout animals for the study of learning and memory.

Long-term retention of a divided attention psycho-motor test combining choice reaction test and postural balance test: A preliminary study

Driving in degraded psychophysical conditions, such as under the influence of alcohol or drugs but also in a state of fatigue or drowsiness, is a growing problem. The current roadside tests used for detecting drugs from drivers suffer various limitations, while impairment is subjective and does not necessarily correlate with drug metabolite concentration found in body fluids. This work is a validation step towards the study of feasibility of a novel test conceived to assess psychophysical conditions of individuals performing at-risk activities. Motor gestures, long-term retention and learning phase related to the protocol are analysed in unimpaired subjects. The protocol is a divided attention test, which combines a critical tracking test achieved with postural movements and a visual choice reaction test. Ten healthy subjects participated in a first set of trials and in a second set after about six months. Each session required the carrying out of the test for ten times in order to investigate learning effect and performance over repetitions. In the first set the subjects showed a learning trend up to the third trial, whilst in the second set of trials they showed motor retention. Nevertheless, the overall performance did not significantly improve. Gestures are probably retained due to the type of tasks and the way in which the instructions are conveyed to the subjects. Moreover, motor retention after a short training suggests that the protocol is easy to learn and understand. Implications for roadside test usage and comparison with current tests are also discussed.

Roles of octopamine and dopamine in appetitive and aversive memory acquisition studied in olfactory conditioning of maxillary palpi extension response in crickets

Elucidation of reinforcing mechanisms for associative learning is an important subject in neuroscience. Based on results of our previous pharmacological studies in crickets, we suggested that octopamine and dopamine mediate reward and punishment signals, respectively, in associative learning. In fruit-flies, however, it was concluded that dopamine mediates both appetitive and aversive reinforcement, which differs from our suggestion in crickets. In our previous studies, the effect of conditioning was tested at 30 min after training or later, due to limitations of our experimental procedures, and thus the possibility that octopamine and dopamine were not needed for initial acquisition of learning was not ruled out. In this study we first established a conditioning procedure to enable us to evaluate acquisition performance in crickets. Crickets extended their maxillary palpi and vigorously swung them when they perceived some odors, and we found that crickets that received pairing of an odor with water reward or sodium chloride punishment exhibited an increase or decrease in percentages of maxillary palpi extension responses to the odor. Using this procedure, we found that octopamine and dopamine receptor antagonists impair acquisition of appetitive and aversive learning, respectively. This finding suggests that neurotransmitters mediating appetitive reinforcement differ in crickets and fruit-flies.

Critical evidence for the prediction error theory in associative learning

In associative learning in mammals, it is widely accepted that the discrepancy, or error, between actual and predicted reward determines whether learning occurs. Complete evidence for the prediction error theory, however, has not been obtained in any learning systems: Prediction error theory stems from the finding of a blocking phenomenon, but blocking can also be accounted for by other theories, such as the attentional theory. We demonstrated blocking in classical conditioning in crickets and obtained evidence to reject the attentional theory. To obtain further evidence supporting the prediction error theory and rejecting alternative theories, we constructed a neural model to match the prediction error theory, by modifying our previous model of learning in crickets, and we tested a prediction from the model: the model predicts that pharmacological intervention of octopaminergic transmission during appetitive conditioning impairs learning but not formation of reward prediction itself, and it thus predicts no learning in subsequent training. We observed such an "auto-blocking", which could be accounted for by the prediction error theory but not by other competitive theories to account for blocking. This study unambiguously demonstrates validity of the prediction error theory in associative learning.

Toward elucidating diversity of neural mechanisms underlying insect learning

Insects are widely used as models to study neural mechanisms of learning and memory. Our recent studies on crickets, together with reports on other insect species, suggest that some fundamental differences exist in neural and molecular mechanisms of learning and memory among different species of insects, particularly between crickets and fruit flies. First, we suggested that in crickets octopamine (OA) and dopamine (DA) neurons convey reward and punishment signals, respectively, in associated learning. On the other hand, it has been reported that in fruit flies different sets of DA neurons convey reward or punishment signals. Secondly, we have suggested that in crickets OA and DA neurons participate in the retrieval of appetitive and aversive memories, respectively, while this is not the case in fruit flies. Thirdly, cyclic AMP signaling is critical for short-term memory formation in fruit flies, but not in crickets. Finally, nitric oxide-cyclic GMP signaling and calcium-calmodulin signaling are critical for long-term memory (LTM) formation in crickets, but such roles have not been reported in fruit flies. Not all of these differences can be ascribed to different experimental methods used in studies. We thus suggest that there are unexpected diversities in basic mechanisms of learning and memory among different insect species, especially between crickets and fruit flies. Studies on a larger number of insect species will help clarify the diversity of learning and memory mechanisms in relation to functional adaptation to the environment and evolutionary history.

Cocaine affects foraging behaviour and biogenic amine modulated behavioural reflexes in honey bees

In humans and other mammals, drugs of abuse alter the function of biogenic amine pathways in the brain leading to the subjective experience of reward and euphoria. Biogenic amine pathways are involved in reward processing across diverse animal phyla, however whether cocaine acts on these neurochemical pathways to cause similar rewarding behavioural effects in animal phyla other than mammals is unclear. Previously, it has been shown that bees are more likely to dance (a signal of perceived reward) when returning from a sucrose feeder after cocaine treatment. Here we examined more broadly whether cocaine altered reward-related behaviour, and biogenic amine modulated behavioural responses in bees. Bees developed a preference for locations at which they received cocaine, and when foraging at low quality sucrose feeders increase their foraging rate in response to cocaine treatment. Cocaine also increased reflexive proboscis extension to sucrose, and sting extension to electric shock. Both of these simple reflexes are modulated by biogenic amines. This shows that systemic cocaine treatment alters behavioural responses that are modulated by biogenic amines in insects. Since insect reward responses involve both octopamine and dopamine signalling, we conclude that cocaine treatment altered diverse reward-related aspects of behaviour in bees. We discuss the implications of these results for understanding the ecology of cocaine as a plant defence compound. Our findings further validate the honey bee as a model system for understanding the behavioural impacts of cocaine, and potentially other drugs of abuse.

Conditioned sounds enhance visual processing

This psychophysics study investigated whether prior auditory conditioning influences how a sound interacts with visual perception. In the conditioning phase, subjects were presented with three pure tones ( =  conditioned stimuli, CS) that were paired with positive, negative or neutral unconditioned stimuli. As unconditioned reinforcers we employed pictures (highly pleasant, unpleasant and neutral) or monetary outcomes (+50 euro cents, −50 cents, 0 cents). In the subsequent visual selective attention paradigm, subjects were presented with near-threshold Gabors displayed in their left or right hemifield. Critically, the Gabors were presented in synchrony with one of the conditioned sounds. Subjects discriminated whether the Gabors were presented in their left or right hemifields. Participants determined the location more accurately when the Gabors were presented in synchrony with positive relative to neutral sounds irrespective of reinforcer type. Thus, previously rewarded relative to neutral sounds increased the bottom-up salience of the visual Gabors. Our results are the first demonstration that prior auditory conditioning is a potent mechanism to modulate the effect of sounds on visual perception.

Cortical signals for rewarded actions and strategic exploration

In stable environments, decision makers can exploit their previously learned strategies for optimal outcomes, while exploration might lead to better options in unstable environments. Here, to investigate the cortical contributions to exploratory behavior, we analyzed single-neuron activity recorded from four different cortical areas of monkeys performing a matching-pennies task and a visual search task, which encouraged and discouraged exploration, respectively. We found that neurons in multiple regions in the frontal and parietal cortex tended to encode signals related to previously rewarded actions more reliably than unrewarded actions. In addition, signals for rewarded choices in the supplementary eye field were attenuated during the visual search task and were correlated with the tendency to switch choices during the matching-pennies task. These results suggest that the supplementary eye field might play a unique role in encouraging animals to explore alternative decision-making strategies.

Roles of NO signaling in long-term memory formation in visual learning in an insect

Many insects exhibit excellent capability of visual learning, but the molecular and neural mechanisms are poorly understood. This is in contrast to accumulation of information on molecular and neural mechanisms of olfactory learning in insects. In olfactory learning in insects, it has been shown that cyclic AMP (cAMP) signaling critically participates in the formation of protein synthesis-dependent long-term memory (LTM) and, in some insects, nitric oxide (NO)-cyclic GMP (cGMP) signaling also plays roles in LTM formation. In this study, we examined the possible contribution of NO-cGMP signaling and cAMP signaling to LTM formation in visual pattern learning in crickets. Crickets that had been subjected to 8-trial conditioning to associate a visual pattern with water reward exhibited memory retention 1 day after conditioning, whereas those subjected to 4-trial conditioning exhibited 30-min memory retention but not 1-day retention. Injection of cycloheximide, a protein synthesis inhibitor, into the hemolymph prior to 8-trial conditioning blocked formation of 1-day memory, whereas it had no effect on 30-min memory formation, indicating that 1-day memory can be characterized as protein synthesis-dependent long-term memory (LTM). Injection of an inhibitor of the enzyme producing an NO or cAMP prior to 8-trial visual conditioning blocked LTM formation, whereas it had no effect on 30-min memory formation. Moreover, injection of an NO donor, cGMP analogue or cAMP analogue prior to 4-trial conditioning induced LTM. Induction of LTM by an NO donor was blocked by DDA, an inhibitor of adenylyl cyclase, an enzyme producing cAMP, but LTM induction by a cAMP analogue was not impaired by L-NAME, an inhibitor of NO synthase. The results indicate that cAMP signaling is downstream of NO signaling for visual LTM formation. We conclude that visual learning and olfactory learning share common biochemical cascades for LTM formation.

Analysis and modeling of neural processes underlying sensory preconditioning

Sensory preconditioning (SPC) is a procedure to demonstrate learning to associate between relatively neutral sensory stimuli in the absence of an external reinforcing stimulus, the underlying neural mechanisms of which have remained obscure. We address basic questions about neural processes underlying SPC, including whether neurons that mediate reward or punishment signals in reinforcement learning participate in association between neutral sensory stimuli. In crickets, we have suggested that octopaminergic (OA-ergic) or dopaminergic (DA-ergic) neurons participate in memory acquisition and retrieval in appetitive or aversive conditioning, respectively. Crickets that had been trained to associate an odor (CS2) with a visual pattern (CS1) (phase 1) and then to associate CS1 with water reward or quinine punishment (phase 2) exhibited a significantly increased or decreased preference for CS2 that had never been paired with the US, demonstrating successful SPC. Injection of an OA or DA receptor antagonist at different phases of the SPC training and testing showed that OA-ergic or DA-ergic neurons do not participate in learning of CS2-CS1 association in phase 1, but that OA-ergic neurons participate in learning in phase 2 and memory retrieval after appetitive SPC training. We also obtained evidence suggesting that association between CS2 and US, which should underlie conditioned response of crickets to CS2, is formed in phase 2, contrary to the standard theory of SPC assuming that it occurs in the final test. We propose models of SPC to account for these findings, by extending our model of classical conditioning.

High-throughput olfactory conditioning and memory retention test show variation in Nasonia parasitic wasps

Most of our knowledge on learning and memory formation results from extensive studies on a small number of animal species. Although features and cellular pathways of learning and memory are highly similar in this diverse group of species, there are also subtle differences. Closely related species of parasitic wasps display substantial variation in memory dynamics and can be instrumental to understanding both the adaptive benefit of and mechanisms underlying this variation. Parasitic wasps of the genus Nasonia offer excellent opportunities for multidisciplinary research on this topic. Genetic and genomic resources available for Nasonia are unrivaled among parasitic wasps, providing tools for genetic dissection of mechanisms that cause differences in learning. This study presents a robust, high-throughput method for olfactory conditioning of Nasonia using a host encounter as reward. A T-maze olfactometer facilitates high-throughput memory retention testing and employs standardized odors of equal detectability, as quantified by electroantennogram recordings. Using this setup, differences in memory retention between Nasonia species were shown. In both Nasonia vitripennis and Nasonia longicornis, memory was observed up to at least 5 days after a single conditioning trial, whereas Nasonia giraulti lost its memory after 2 days. This difference in learning may be an adaptation to species-specific differences in ecological factors, for example, host preference. The high-throughput methods for conditioning and memory retention testing are essential tools to study both ultimate and proximate factors that cause variation in learning and memory formation in Nasonia and other parasitic wasp species.

Distinct roles for direct and indirect pathway striatal neurons in reinforcement

Dopamine signaling is implicated in reinforcement learning, but the neural substrates targeted by dopamine are poorly understood. Here, we bypassed dopamine signaling itself and tested how optogenetic activation of dopamine D1- or D2-receptor-expressing striatal projection neurons influenced reinforcement learning in mice. Stimulating D1-expressing neurons induced persistent reinforcement, whereas stimulating D2-expressing neurons induced transient punishment, demonstrating that activation of these circuits is sufficient to modify the probability of performing future actions.

Striatal mechanisms underlying movement, reinforcement, and punishment

Direct and indirect pathway striatal neurons are known to exert opposing control over motor output. In this review, we discuss a hypothetical extension of this framework, in which direct pathway striatal neurons also mediate reinforcement and reward, and indirect pathway neurons mediate punishment and aversion.

Reward improves long-term retention of a motor memory through induction of offline memory gains

In humans, training in which good performance is rewarded or bad performance punished results in transient behavioral improvements. The relative effects of reward and punishment on consolidation and long-term retention, critical behavioral stages for successful learning, are not known. Here, we investigated the effects of reward and punishment on these different stages of human motor skill learning. We studied healthy subjects who trained on a motor task under rewarded, punished, or neutral control conditions. Performance was tested before and immediately, 6 hr, 24 hr, and 30 days after training in the absence of reward or punishment. Performance improvements immediately after training were comparable in the three groups. At 6 hr, the rewarded group maintained performance gains, whereas the other two groups experienced significant forgetting. At 24 hr, the reward group showed significant offline (posttraining) improvements, whereas the other two groups did not. At 30 days, the rewarded group retained the gains identified at 24 hr, whereas the other two groups experienced significant forgetting. We conclude that training under rewarded conditions is more effective than training under punished or neutral conditions in eliciting lasting motor learning, an advantage driven by offline memory gains that persist over time.

Herbal extracts and phytochemicals: plant secondary metabolites and the enhancement of human brain function

Humans consume a wide range of foods, drugs, and dietary supplements that are derived from plants and which modify the functioning of the central nervous sytem (CNS). The psychoactive properties of these substances are attributable to the presence of plant secondary metabolites, chemicals that are not required for the immediate survival of the plant but which are synthesized to increase the fitness of the plant to survive by allowing it to interact with its environment, including pathogens and herbivorous and symbiotic insects. In many cases, the effects of these phytochemicals on the human CNS might be linked either to their ecological roles in the life of the plant or to molecular and biochemical similarities in the biology of plants and higher animals. This review assesses the current evidence for the efficacy of a range of readily available plant-based extracts and chemicals that may improve brain function and which have attracted sufficient research in this regard to reach a conclusion as to their potential effectiveness as nootropics. Many of these candidate phytochemicals/extracts can be grouped by the chemical nature of their potentially active secondary metabolite constituents into alkaloids (caffeine, nicotine), terpenes (ginkgo, ginseng, valerian, Melissa officinalis, sage), and phenolic compounds (curcumin, resveratrol, epigallocatechin-3-gallate, Hypericum perforatum, soy isoflavones). They are discussed in terms of how an increased understanding of the relationship between their ecological roles and CNS effects might further the field of natural, phytochemical drug discovery.

Roles of aminergic neurons in formation and recall of associative memory in crickets

We review recent progress in the study of roles of octopaminergic (OA-ergic) and dopaminergic (DA-ergic) signaling in insect classical conditioning, focusing on our studies on crickets. Studies on olfactory learning in honey bees and fruit-flies have suggested that OA-ergic and DA-ergic neurons convey reinforcing signals of appetitive unconditioned stimulus (US) and aversive US, respectively. Our work suggested that this is applicable to olfactory, visual pattern, and color learning in crickets, indicating that this feature is ubiquitous in learning of various sensory stimuli. We also showed that aversive memory decayed much faster than did appetitive memory, and we proposed that this feature is common in insects and humans. Our study also suggested that activation of OA- or DA-ergic neurons is needed for appetitive or aversive memory recall, respectively. To account for this finding, we proposed a model in which it is assumed that two types of synaptic connections are strengthened by conditioning and are activated during memory recall, one type being connections from neurons representing conditioned stimulus (CS) to neurons inducing conditioned response and the other being connections from neurons representing CS to OA- or DA-ergic neurons representing appetitive or aversive US, respectively. The former is called stimulus–response (S–R) connection and the latter is called stimulus–stimulus (S–S) connection by theorists studying classical conditioning in vertebrates. Results of our studies using a second-order conditioning procedure supported our model. We propose that insect classical conditioning involves the formation of S–S connection and its activation for memory recall, which are often called cognitive processes.

Critical roles of mecamylamine-sensitive mushroom body neurons in insect olfactory learning

In insects, cholinergic neurons are thought to transmit olfactory conditioned stimulus (CS) to the sites for associating the CS with unconditioned stimulus (US), but the types of acetylcholine (ACh) receptor used by neurons participating in the association have not been determined. In cockroaches, a type of nicotinic ACh receptor specifically antagonized by mecamylamine (MEC) has been characterized. Here we investigated the roles of neurons possessing MEC-sensitive ACh receptors (MEC-sensitive neurons) in olfactory conditioning of salivation, monitored by changes in activities of salivary neurons, in cockroaches. Local and bilateral microinjection of MEC into each of the three olfactory centers, antennal lobes, calyces of the mushroom bodies and lateral protocerebra, impaired olfactory responses of salivary neurons, indicating that MEC-sensitive neurons in all olfactory centers participate in pathways mediating olfactory responses of salivary neurons. Conditioning of olfactory CS with sucrose US was impaired by injection of MEC into the antennal lobes or calyces, i.e., conditioned responses were absent even after recovery from MEC injection, suggesting that the CS-US association occurs in MEC-sensitive neurons in calyces (most probably Kenyon cells) or in neurons in downstream pathways. In contrast, conditioned responses appeared after recovery from MEC injection into the lateral protocerebra, suggesting that MEC-sensitive neurons in the lateral protocerebra are downstream of the association sites. Since lateral protocerebra are major termination areas of mushroom body efferent neurons, we suggest that input synapses of MEC-sensitive Kenyon cells, or their output synapses upon mushroom body efferent neurons, are the sites for CS-US association for conditioning of salivation.

Appetitive and aversive visual learning in freely moving Drosophila

To compare appetitive and aversive visual memories of the fruit fly Drosophila melanogaster, we developed a new paradigm for classical conditioning. Adult flies are trained en masse to differentially associate one of two visual conditioned stimuli (CS) (blue and green light as CS) with an appetitive or aversive chemical substance (unconditioned stimulus or US). In a test phase, flies are given a choice between the paired and the unpaired visual stimuli. Associative memory is measured based on altered visual preference in the test. If a group of flies has, for example, received a sugar reward with green light in the training, they show a significantly higher preference for the green stimulus during the test than another group of flies having received the same reward with blue light. We demonstrate critical parameters for the formation of visual appetitive memory, such as training repetition, order of reinforcement, starvation, and individual conditioning. Furthermore, we show that formic acid can act as an aversive chemical reinforcer, yielding weak, yet significant, aversive memory. These results provide a basis for future investigations into the cellular and molecular mechanisms underlying visual memory and perception in Drosophila.

Roles of octopaminergic and dopaminergic neurons in appetitive and aversive memory recall in an insect

Background

In insect classical conditioning, octopamine (the invertebrate counterpart of noradrenaline) or dopamine has been suggested to mediate reinforcing properties of appetitive or aversive unconditioned stimulus, respectively. However, the roles of octopaminergic and dopaminergic neurons in memory recall have remained unclear.

Results

We studied the roles of octopaminergic and dopaminergic neurons in appetitive and aversive memory recall in olfactory and visual conditioning in crickets. We found that pharmacological blockade of octopamine and dopamine receptors impaired aversive memory recall and appetitive memory recall, respectively, thereby suggesting that activation of octopaminergic and dopaminergic neurons and the resulting release of octopamine and dopamine are needed for appetitive and aversive memory recall, respectively. On the basis of this finding, we propose a new model in which it is assumed that two types of synaptic connections are formed by conditioning and are activated during memory recall, one type being connections from neurons representing conditioned stimulus to neurons inducing conditioned response and the other being connections from neurons representing conditioned stimulus to octopaminergic or dopaminergic neurons representing appetitive or aversive unconditioned stimulus, respectively. The former is called 'stimulus-response connection' and the latter is called 'stimulus-stimulus connection' by theorists studying classical conditioning in higher vertebrates. Our model predicts that pharmacological blockade of octopamine or dopamine receptors during the first stage of second-order conditioning does not impair second-order conditioning, because it impairs the formation of the stimulus-response connection but not the stimulus-stimulus connection. The results of our study with a cross-modal second-order conditioning were in full accordance with this prediction.

Conclusion

We suggest that insect classical conditioning involves the formation of two kinds of memory traces, which match to stimulus-stimulus connection and stimulus-response connection. This is the first study to suggest that classical conditioning in insects involves, as does classical conditioning in higher vertebrates, the formation of stimulus-stimulus connection and its activation for memory recall, which are often called cognitive processes.