Monkeys (Macaca mulatta) learn two-choice discriminations under displaced reinforcement

State-transition-free reinforcement learning in chimpanzees (Pan troglodytes)

The outcome of an action often occurs after a delay. One solution for learning appropriate actions from delayed outcomes is to rely on a chain of state transitions. Another solution, which does not rest on state transitions, is to use an eligibility trace (ET) that directly bridges a current outcome and multiple past actions via transient memories. Previous studies revealed that humans (Homo sapiens) learned appropriate actions in a behavioral task in which solutions based on the ET were effective but transition-based solutions were ineffective. This suggests that ET may be used in human learning systems. However, no studies have examined nonhuman animals with an equivalent behavioral task. We designed a task for nonhuman animals following a previous human study. In each trial, participants chose one of two stimuli that were randomly selected from three stimulus types: a stimulus associated with a food reward delivered immediately, a stimulus associated with a reward delivered after a few trials, and a stimulus associated with no reward. The presented stimuli did not vary according to the participants' choices. To maximize the total reward, participants had to learn the value of the stimulus associated with a delayed reward. Five chimpanzees (Pan troglodytes) performed the task using a touchscreen. Two chimpanzees were able to learn successfully, indicating that learning mechanisms that do not depend on state transitions were involved in the learning processes. The current study extends previous ET research by proposing a behavioral task and providing empirical data from chimpanzees.

1-Back reinforcement symbolic-matching by humans: How do they learn it?

For humans, a distinction has been made between implicit and explicit learning. Implicit learning is thought to involve automatic processes of the kind involved in much Pavlovian conditioning, while explicit learning is thought to involve conscious hypothesis testing and rule formation, in which the subject's statement of the rule has been taken as evidence of explicit learning. Various methods have been used to determine if nonverbal animals are able to learn a task explicitly - among these is the 1-back reinforcement task in which feedback from performance on the current conditional discrimination trial is provided only after completion of the following trial. We propose that it is not whether an organism can learn the task, but whether they learn it rapidly, all-or-none, that provides a better distinction between the two kinds of learning. We had humans learn a symbolic matching, 1-back reinforcement task. Almost half of the subjects failed to learn the task, and of those who did, none described the 1-back rule. Thus, it is possible to learn this task without learning the 1-back rule. Furthermore, the backward learning functions for humans differ from those of pigeons. Human subjects who learned the task did so all-or-none, suggesting explicit learning. In earlier research with pigeons, they too showed significant learning of this task; however, backward learning functions suggested that they did so gradually over the course of several sessions of training and to a lower level of asymptotic accuracy than the humans, a result suggesting implicit learning was involved.

Implicit learning of the one-back reinforcement matching-mismatching task by pigeons

Humans can learn tasks explicitly, as they can often describe the rules they have used to learn the task.^1,2,3 Animals, however, are thought to learn tasks implicitly (i.e., purely associatively).^2,3 That is, they gradually learn the correlation or association between the stimulus (or response) and the outcome. Both humans and pigeons can learn matching, where a sample stimulus indicates which one of two stimuli matches the sample. The 1-back reinforcement task is a difficult version of matching in which a correct response on trial N is rewarded only following a response on trial N + 1 (independent of the response on trial N + 1),⁴ and the correct response on trial N + 1 indicates whether a reward will occur on trial N + 2, and so forth. Humans do not appear to be able to learn the 1-back rule.⁵ Pigeons, however, do show 1-back reinforcement learning,^6,7 and they appear to do so implicitly by gradually learning the correlation between their response on one trial and the outcome on the next trial (because all other relations are uncorrelated with the outcome). They learn the task slowly and to a level below what would be expected had they learned it explicitly. The present results, together with research with humans,⁷ suggest that there are times when human explicit learning may interfere with the ability of humans to learn. Pigeons, however, are not "distracted" by attempts at explicit learning, and thus they are able to learn this and other similar tasks.^6,7,8.

1-Back reinforcement matching and mismatching by pigeons: Implicit or explicit learning?

In human learning a distinction has been made between implicit and explicit learning. Implicit learning is thought involve automatic processes of the kind involved in Pavlovian conditioning, while explicit learning is thought to involve conscious hypothesis testing and rule formation, in which the ability to report the rule used to learn the task is taken as evidence. Because non-verbal animals cannot provide such evidence, several indirect methods have been proposed. One of these methods is faster learning by humans of certain explicitly learned tasks than implicitly learned tasks, but pigeons do not show a similar difference. Another method involves the 1-back-reinforcement conditional discrimination (if A choose X, if B choose Y) in which feedback following the conditional response is delayed until the next trial. It has been argued that implicit learning cannot occur over the delay between the conditional response and the reinforcer on the next trial, yet, it has been found that monkeys can learn this 1-back reinforcement task. We have argued that such learning can occur implicitly. We have found that pigeons, a species not thought to learn explicitly, can show significant learning of both 1-back reinforcement matching and 1-back reinforcement mismatching, two versions of the 1-back-reinforcement conditional discrimination. We propose that the evidence for explicit learning by non-verbal animals suffers from alternative simpler accounts because the rationale for explicit learning is based on assumptions that likely are not correct.

The Biology and Pathobiology of Glutamatergic, Cholinergic, and Dopaminergic Signaling in the Aging Brain

The elderly population is growing worldwide, with important health and socioeconomic implications. Clinical and experimental studies on aging have uncovered numerous changes in the brain, such as decreased neurogenesis, increased synaptic defects, greater metabolic stress, and enhanced inflammation. These changes are associated with cognitive decline and neurobehavioral deficits. Although aging is not a disease, it is a significant risk factor for functional worsening, affective impairment, disease exaggeration, dementia, and general disease susceptibility. Conversely, life events related to mental stress and trauma can also lead to accelerated age-associated disorders and dementia. Here, we review human studies and studies on mice and rats, such as those modeling human neurodegenerative diseases, that have helped elucidate (1) the dynamics and mechanisms underlying the biological and pathological aging of the main projecting systems in the brain (glutamatergic, cholinergic, and dopaminergic) and (2) the effect of defective glutamatergic, cholinergic, and dopaminergic projection on disabilities associated with aging and neurodegenerative disorders, such as Alzheimer's and Parkinson's diseases. Detailed knowledge of the mechanisms of age-related diseases can be an important element in the development of effective ways of treatment. In this context, we briefly analyze which adverse changes associated with neurodegenerative diseases in the cholinergic, glutaminergic and dopaminergic systems could be targeted by therapeutic strategies developed as a result of our better understanding of these damaging mechanisms.

Pigeons acquire the 1-back task: Implications for implicit versus explicit learning?

In humans, a distinction can be made between implicit or procedural learning (involving stimulus-response associations) and explicit or declarative learning (involving verbalizable rules) that is relatively easy to make in verbal humans. According to several investigators, it is also possible to make such a distinction in nonverbal animals. One way is by training them on a conditional discrimination task (e.g., matching-to-sample) in which reinforcement for correct choice on the current trial is delayed until after a choice is made on the next trial - a method known as the 1-back procedure. According to Smith, Jackson, and Church ( Journal of Comparative Psychology, 134(4), 423-434, 2020), the delay between the sample-correct-comparison response on one trial and reinforcement obtained on the next trial is too long for implicit (associative) learning. Thus, according to this theory, learning must be explicit. In the present experiments we trained pigeons using the 1-back procedure. In Experiment 1, pigeons were trained on red/green 1-back matching using a non-correction procedure. Some of the pigeons showed significant learning. When a correction procedure was introduced, all the pigeons showed evidence of learning. In Experiment 2, new pigeons learned red/green 1-back matching with the correction procedure. In Experiment 3, new pigeons learned symbolic 1-back matching with yellow and blue conditional stimuli and red/green choice stimuli. Thus, pigeons can learn using 1-back reinforcement. Although it would appear that the pigeons acquired this task explicitly, we believe that this procedure does not adequately distinguish between implicit and explicit learning.

Exploring Explicit Learning Strategies: A Dissociative Framework for Research

To explain learning, comparative researchers invoke an associative construct by which immediate reinforcement strengthens animal's adaptive responses. In contrast, cognitive researchers freely acknowledge humans' explicit-learning capability to test and confirm hypotheses even lacking direct reinforcement. We describe a new dissociative framework that may stretch animals' learning toward the explicit pole of cognition. We discuss the neuroscience of reinforcement-based learning and suggest the possibility of disabling a dominant form of reinforcement-based discrimination learning. In that vacuum, researchers may have an opportunity to observe animals' explicit learning strategies (i.e., hypotheses, rules, task self-construals). We review initial research using this framework showing explicit learning by humans and perhaps by monkeys. Finally, we consider why complementary explicit and reinforcement-based learning systems might promote evolutionary and ecological fitness. Illuminating the evolution of parallel learning systems may also tell part of the story of the emergence of humans' extraordinary capacity for explicit-declarative cognition.