Midsession reversal learning: why do pigeons anticipate and perseverate?

Deliberately making miskates: Behavioural consistency under win maximization and loss maximization conditions

We argue that the feedback traditionally used to indicate negative outcomes causes future detrimental performance because of the default goal of win maximization. In gaming paradigms where participants intentionally performed as well (win maximization) and as poorly (loss maximization) as possible, we showed a double dissociation where actions following wins were more consistent during win maximization, but actions following losses were more consistent during loss maximization. This broader distinction between goal-congruent and goal-incongruent feedback suggests that individuals are able to flexibly redefine their definition of 'success', and provide a reconsideration of the way we think about 'losing'.

Kea (Nestor notabilis) show flexibility and individuality in within-session reversal learning tasks

The midsession reversal paradigm confronts an animal with a two-choice discrimination task where the reward contingencies are reversed at the midpoint of the session. Species react to the reversal with either win-stay/lose-shift, using local information of reinforcement, or reversal estimation, using global information, e.g. time, to estimate the point of reversal. Besides pigeons, only mammalian species were tested in this paradigm so far and analyses were conducted on pooled data, not considering possible individually different responses. We tested twelve kea parrots with a 40-trial midsession reversal test and additional shifted reversal tests with a variable point of reversal. Birds were tested in two groups on a touchscreen, with the discrimination task having either only visual or additional spatial information. We used Generalized Linear Mixed Models to control for individual differences when analysing the data. Our results demonstrate that kea can use win-stay/lose-shift independently of local information. The predictors group, session, and trial number as well as their interactions had a significant influence on the response. Furthermore, we discovered notable individual differences not only between birds but also between sessions of individual birds, including the ability to quite accurately estimate the reversal position in alternation to win-stay/lose-shift. Our findings of the kea’s quick and flexible responses contribute to the knowledge of diversity in avian cognitive abilities and emphasize the need to consider individuality as well as the limitation of pooling the data when analysing midsession reversal data.

Variability in competitive decision-making speed and quality against exploiting and exploitative opponents

A presumption in previous work has been that sub-optimality in competitive performance following loss is the result of a reduction in decision-making time (i.e., post-error speeding). The main goal of this paper is to test the relationship between decision-making speed and quality, with the hypothesis that slowing down decision-making should increase the likelihood of successful performance in cases where a model of opponent domination can be implemented. Across Experiments 1–3, the speed and quality of competitive decision-making was examined in a zero-sum game as a function of the nature of the opponent (unexploitable, exploiting, exploitable). Performance was also examined against the nature of a credit (or token) system used as a within-experimental manipulation (no credit, fixed credit, variable credit). To compliment reaction time variation as a function of outcome, both the fixed credit and variable credit conditions were designed to slow down decision-making, relative to a no credit condition where the game could be played in quick succession and without interruption. The data confirmed that (a) self-imposed reductions in processing time following losses (post-error speeding) were causal factors in determining poorer-quality behaviour, (b) the expression of lose-shift was less flexible than the expression of win-stay, and, (c) the use of a variable credit system may enhance the perceived control participants have against exploitable opponents. Future work should seek to disentangle temporal delay and response interruption as determinants of decision-making quality against numerous styles of opponency.

Step changes in the intertrial interval in the midsession reversal task: Predicting pigeons' performance with the learning-to-time model

Our goal was to assess the role of timing in pigeons' performance in the midsession reversal task. In discrete-trial sessions, pigeons learned to discriminate between 2 stimuli, S1 and S2. Choices of S1 were reinforced only in the first half of the session and choices of S2 were reinforced only in the second half. Typically, pigeons choose S2 before the contingency reverses (anticipatory errors) and S1 after (perseverative errors), suggesting that they time the interval from the beginning of the session to the contingency reversal. To test this hypothesis, we exposed pigeons to a midsession reversal task and, depending on the group, either increased or decreased the ITI duration. We then contrasted the pigeons' performance with the predictions of the Learning-to-Time (LeT) model: In both conditions, preference was expected to reverse at the same time as in the previous sessions. When the ITI was doubled, pigeons' preference reversal occurred at half the trial number but at the same time as in the previous sessions. When the ITI was halved, pigeons' preference reversal occurred at a later trial but at an earlier time than in the previous sessions. Hence, pigeons' performance was only partially consistent with the predictions of LeT, suggesting that besides timing, other sources of control, such as the outcome of previous trials, seem to influence choice.

Pigeons' midsession reversal: Greater magnitude of reinforcement on the first half of the session leads to improved accuracy

In the midsession reversal task, pigeons are trained on a simultaneous two-alternative discrimination in which S1 is correct for the first half of the session and S2 is correct for the second half of the session. Optimally, pigeons should choose S1 until it stops being correct and choose S2 afterward. Instead, pigeons anticipate S2 too early and continue choosing S1 even after the reversal. Research suggests that they attempt to time the reversal rather than use the feedback from the preceding response(s). Recently, there is evidence that performance is almost optimized by generating an asymmetry between S1 and S2. For example, pigeons' accuracy improves if correct S1 responses are reinforced 100% of the time, but correct S2 responses are reinforced only 20% of the time. Similarly, accuracy improves if S1 requires one peck but S2 requires 10 pecks. Accuracy does not improve, however, if the value of S1 is less than the value of S2. In the current experiment, we manipulated the magnitude of reinforcement. For the experimental group, correct responses to S1 were reinforced with five pellets of food and correct responses to S2 were reinforced with one pellet. For the control group, all correct responses were reinforced with three pellets. Consistent with the earlier findings, results indicated that there was a significant reduction in anticipatory errors in the experimental group compared with the control, and there was no significant increase in perseverative errors.

Behavioural and neural limits in competitive decision making: The roles of outcome, opponency and observation

To understand the boundaries we set for ourselves in terms of environmental responsibility during competition, we examined a neural index of outcome valence (feedback-related negativity; FRN) in relation to an early index of visual attention (N1), a later index of motivational significance (P3), and, eventual behaviour. In Experiment 1 (n = 36), participants either were (play) or were not (observe) responsible for action selection. In Experiment 2 (n = 36), opponents additionally either could (exploitable) or could not (unexploitable) be beaten. Various failures in reinforcement learning expression were revealed including large-scale approximations of random behaviour. Against unexploitable opponents, N1 determined the extent to which negative and positive outcomes were perceived as distinct categories by FRN. Against exploitable opponents, FRN determined the extent to which P3 generated neural gain for future events. Differential activation of the N1 - FRN - P3 processing chain provides a framework for understanding the behavioural dynamism observed during competitive decision making.

Evidence of non-circadian timing in a low response-cost daily Time-Place Learning task with pigeons Columba Livia

Previous research has shown that rats require high response cost in order to display circadian timing in daily Time-Place Learning (TPL) tasks. For many possible reasons, no explicit effort to explore the effects of response cost on the performance of other species in these tasks has been made. Therefore, the present paper explores the effects of response cost on pigeon's performance on a daily TPL task. Head entry responses were reinforced according to a Random Interval schedule of reinforcement on one feeder during morning sessions and on another feeder during afternoon sessions. Feeders were located 8 cm apart for one group of birds (Group Near) and 56 cm apart for another group (Group Far). After 50 training sessions, testing began. Test sessions consisted of skipping either the morning or the afternoon session. Results show that most birds in the near group respond primarily on the opposite feeder during the first 20 s of the test sessions and then they switch to the correct feeder. On the other hand, most birds in Group Far respond at the same rate on both opposite and correct feeders during 20 s, and then they respond primarily on the correct feeder. The possibility of these data revealing non circadian timing for birds in a low response-cost daily TPL task is discussed along with the implications of such a finding for previous literature that claims that this type of performance could be unique to rats.

Behavioural Isomorphism, Cognitive Economy and Recursive Thought in Non-Transitive Game Strategy

Game spaces in which an organism must repeatedly compete with an opponent for mutually exclusive outcomes are critical methodologies for understanding decision-making under pressure. In the non-transitive game rock, paper, scissors (RPS), the only technique that guarantees the lack of exploitation is to perform randomly in accordance with mixed-strategy. However, such behavior is thought to be outside bounded rationality and so decision-making can become deterministic, predictable, and ultimately exploitable. This review identifies similarities across economics, neuroscience, nonlinear dynamics, human, and animal cognition literatures, and provides a taxonomy of RPS strategy. RPS strategies are discussed in terms of (a) whether the relevant computations require sensitivity to item frequency, the cyclic relationships between responses, or the outcome of the previous trial, and (b) whether the strategy is framed around the self or other. The negative implication of this taxonomy is that despite the differences in cognitive economy and recursive thought, many of the identified strategies are behaviorally isomorphic. This makes it difficult to infer strategy from behavior. The positive implication is that this isomorphism can be used as a novel design feature in furthering our understanding of the attribution, agency, and acquisition of strategy in RPS and other game spaces.

Environment tracking and signal following in a reinforcer-ratio reversal procedure

Several studies suggest that the degree of control by reinforcer ratios (environment tracking) and by exteroceptive stimuli that signal future reinforcer availability (signal following) depends on environmental certainty: As reinforcers become more likely at one location, environmental contingencies exert stronger control and exteroceptive stimuli exert weaker control. This research has not yet been extended to environments in which reinforcer availability changes across time, even though such changes are present in most natural environments. Thus, in the present experiment, we examined environment tracking and signal following in a concurrent schedule in which the reinforcer ratio reversed to its reciprocal 30 s after a reinforcer delivery and keylight-color stimuli signaled the likely or definite time or location of the next reinforcer. Across conditions, we manipulated environmental certainty by varying the probability of reinforcer deliveries on the locally richer key. This made the location of future reinforcers at a particular time more or less certain, but did not change the overall reinforcer ratio. Changes in local environmental certainty had little to no effect on environment tracking and signal following; in all conditions, keylight-color stimuli strongly controlled choice and reinforcer ratios exerted weak control. The present findings suggest that the extent of environment tracking and signal following is primarily determined by global, not local, environmental certainty.

Midsession shifts in reward probability and the control of behavioral variability

Extensive research has shown that the variability of organismal behavior is great when contingent reinforcement is delayed, small, or improbable. This research has generally employed stable response-reinforcer relationships, and therefore is limited in its explanatory scope with respect to a dynamic environment. We conducted two experiments to investigate whether pigeons' conditioned pecking behavior shows anticipatory or perseverative patterns of behavioral variability when the reinforcement probability reliably changes within experimental sessions. In Experiment 1, three pigeons received alternating sessions in which the reinforcement probability (35% or 4.2%) was shifted at the midpoint of each session in the presence of the same discrete cue. Experiment 2 featured a similar design, but with the inclusion of a discrete visual cue that changed at the session midpoint, and thus unambiguously indicated reinforcement probability. Local reinforcement rates only reliably controlled response variability when a discrete visual cue was available. Without this, pigeons did not discriminate between trial types in the first halves of sessions, and showed evidence of perseveration of response variability following a within-session shift. Critically, this is the first experimental demonstration that the relationship between reinforcement probability and behavioral variability is moderated by another factor (i.e., trial position within a session). This study thus expands our understanding of the control of behavioral variability as a function of experiential factors.

Spatial midsession reversal learning in rats: Effects of egocentric Cue use and memory

The midsession reversal task has been used to investigate behavioral flexibility and cue use in non-human animals, with results indicating differences in the degree of control by environmental cues across species. For example, time-based control has been found in rats only when tested in a T-maze apparatus and under specific conditions in which position and orientation (i.e., egocentric) cues during the intertrial interval could not be used to aid performance. Other research in an operant setting has shown that rats often produce minimal errors around the reversal location, demonstrating response patterns similar to patterns exhibited by humans and primates in this task. The current study aimed to reduce, but not eliminate, the ability for rats to utilize egocentric cues by placing the response levers on the opposite wall of the chamber in relation to the pellet dispenser. Results showed that rats made minimal errors prior to the reversal, suggesting time-based cues were not controlling responses, and that they switched to the second correct stimulus within a few trials after the reversal event. Video recordings also revealed highly structured patterns of behavior by the majority of rats, which often differed depending on which response was reinforced. We interpret these findings as evidence that rats are adept at utilizing their own egocentric cues and that these cues, along with memory for the recent response-reinforcement contingencies, aid in maximizing reinforcement over the session.

Basic reversal-learning capacity in flies suggests rudiments of complex cognition

The most basic models of learning are reinforcement learning models (for instance, classical and operant conditioning) that posit a constant learning rate; however many animals change their learning rates with experience. This process is sometimes studied by reversing an existing association between cues and rewards, and measuring the rate of relearning. Augmented reversal-learning, where learning rates increase with practice, can be an important component of behavioral flexibility; and may provide insight into higher cognition. Previous studies of reversal-learning in Drosophila have not measured learning rates, but have tended to focus on measuring gross deficits in reversal-learning, as the ratio of two timepoints. These studies have uncovered a diversity of mechanisms underlying reversal-learning, but natural genetic variation in this trait has yet to be assessed. We conducted a reversal-learning regime on a diverse panel of Drosophila melanogaster genotypes. We found highly significant genetic variation in their baseline ability to learn. We also found that they have a consistent, and strong (1.3×), increase in their learning speed with reversal. We found no evidence, however, that there was genetic variation in their ability to increase their learning rates with experience. This may suggest that Drosophila have a hitherto unrecognized ability to integrate acquired information, and improve their decision making; but that their mechanisms for doing so are under strong constraints.

Change in the relative contributions of habit and working memory facilitates serial reversal learning expertise in rhesus monkeys

Functionally distinct memory systems likely evolved in response to incompatible demands placed on learning by distinct environmental conditions. Working memory appears adapted, in part, for conditions that change frequently, making rapid acquisition and brief retention of information appropriate. In contrast, habits form gradually over many experiences, adapting organisms to contingencies of reinforcement that are stable over relatively long intervals. Serial reversal learning provides an opportunity to simultaneously examine the processes involved in adapting to rapidly changing and relatively stable contingencies. In serial reversal learning, selecting one of the two simultaneously presented stimuli is positively reinforced, while selection of the other is not. After a preference for the positive stimulus develops, the contingencies of reinforcement reverse. Naïve subjects adapt to such reversals gradually, perseverating in selection of the previously rewarded stimulus. Experts reverse rapidly according to a win-stay, lose-shift response pattern. We assessed whether a change in the relative control of choice by habit and working memory accounts for the development of serial reversal learning expertise. Across three experiments, we applied manipulations intended to attenuate the contribution of working memory but leave the contribution of habit intact. We contrasted performance following long and short intervals in Experiments 1 and 2, and we interposed a competing cognitive load between trials in Experiment 3. These manipulations slowed the acquisition of reversals in expert subjects, but not naïve subjects, indicating that serial reversal learning expertise is facilitated by a shift in the control of choice from passively acquired habit to actively maintained working memory.

Rats' midsession reversal performance: the nature of the response

The midsession reversal task involves a simple simultaneous discrimination that predictably reverses midway through a session. Under various conditions, pigeons generally both anticipate the reversal and perseverate once it has occurred, whereas rats tend to make very few of either kind of error. In the present research, we investigated the hypothesis that the difference in performance between rats and pigeons is related to the nature of the responses made. We hypothesized that rats could have been better at bridging the intertrial interval by keeping the relevant paw close to the lever while eating, whereas the pigeons had to remove their beak from the response key and insert it into the feeder, thus making it difficult to mediate the response last made. In the present experiment, in successive phases, rats were trained to leverpress or nose-poke on a 40-trial midsession reversal, an 80-trial midsession reversal, and a variable-location reversal. The results showed that the leverpress group acquired the task faster than the nose-poke group, but that both groups reached comparable levels of performance. Thus, the difference in the natures of the responses cannot fully account for the differences in accuracy between rats and pigeons. Additionally, differences in the types of errors made by the two groups suggest that the nature of the response plays different roles in the performance of this task.

Midsession reversals with pigeons: visual versus spatial discriminations and the intertrial interval

Discrimination reversal learning has been used as a measure of species flexibility in dealing with changes in reinforcement contingency. In the simultaneous-discrimination, midsession-reversal task, one stimulus (S1) is correct for the first half of the session, and the other stimulus (S2) is correct for the second half. After training, pigeons show a curious pattern of choices: They begin to respond to S2 well before the reversal point (i.e., they make anticipatory errors), and they continue to respond to S1 well after the reversal (i.e., they make perseverative errors). That is, pigeons appear to be using the passage of time or the number of trials into the session as a cue to reverse, and are less sensitive to the feedback at the point of reversal. To determine whether the nature of the discrimination or a failure of memory for the stimulus chosen on the preceding trial contributed to the pigeons' less-than-optimal performance, we manipulated the nature of the discrimination (spatial or visual) and the duration of the intertrial interval (5.0 or 1.5 s), in order to determine the conditions under which pigeons would show efficient reversal learning. The major finding was that only when the discrimination was spatial and the intertrial interval was short did the pigeons perform optimally.

Everywhere and everything: The power and ubiquity of time

Anticipatory timing plays a critical role in many aspects of human and non-human animal behavior. Timing has been consistently observed in the range of milliseconds to hours, and demonstrates a powerful influence on the organization of behavior. Anticipatory timing is acquired early in associative learning and appears to guide association formation in important ways. Importantly, timing participates in regulating goal-directed behaviors in many schedules of reinforcements, and plays a critical role in value-based decision making under concurrent schedules. In addition to playing a key role in fundamental learning processes, timing often dominates when temporal cues are available concurrently with other stimulus dimensions. Such control by the passage of time has even been observed when other cues provide more accurate information and can lead to sub-optimal behaviors. The dominance of temporal cues in governing anticipatory behavior suggests that time may be inherently more salient than many other stimulus dimensions. Discussions of the interface of the timing system with other cognitive processes are provided to demonstrate the powerful and primitive nature of time as a stimulus dimension.

Further investigation of the Monty Hall Dilemma in pigeons and rats

In the Monty Hall Dilemma (MHD), three doors are presented with a prize behind one and participants are instructed to choose a door. One of the unchosen doors not containing the prize is revealed, following which the participant can choose to stay with their chosen door or switch to the other one. The optimal strategy is to switch. Herbranson and Schroeder (2010) found that humans performed poorly on this task, whereas pigeons learned to switch readily. We found that pigeons performed only slightly better than humans and that pigeons stayed nearly exclusively when staying and switching were reinforced equally and when staying was the optimal strategy (Stagner et al., 2013b). In Experiment 1 of the present research, rats were trained under these same conditions to observe if possible differences in foraging strategy would influence performance on this task. In Experiment 2, pigeons were trained in an analogous procedure to better compare the two species. We found that both species were sensitive to the overall probability of reinforcement, as both switched significantly more often than subjects that were reinforced equally for staying and switching or reinforced more often for staying. Overall, the two species performed very similarly within the parameters of the current procedure. "This article is part of a Special Issue entitled: Tribute to Tom Zentall."

A three-stimulus midsession reversal task in pigeons with visual and spatial discriminative stimuli

In a two-stimulus visual discrimination task with a reversal in reward contingencies midway through each session, pigeons produce a surprising number of both anticipatory (i.e., before the reversal) and perseverative (i.e., after the reversal) errors. In the current work, we examined pigeons' (Columba livia) patterns of responding on a 90-trial, three-stimulus visual or spatial discrimination task with two changes in reward contingency (one after Trial 30 and one after Trial 60) during each session. On probe sessions where pecking the first-correct stimulus was rewarded for the first 60 rather than 30 trials, pigeons on a spatial discrimination pecked the first-correct stimulus until it was no longer rewarded, while visual discrimination birds ceased responding to the first-correct stimulus even while it was still being rewarded. On probe sessions where the onset of the first trial was delayed 7 min, pigeons' performance on the visual discrimination was disrupted by the interval delay, but performance in the spatial condition was more similar to baseline. Pigeons use different strategies (temporal control vs. local reinforcement) on midsession reversal tasks with visual versus spatial stimuli, suggesting that they are selectively permeable to changes in information (global vs. local reinforcement rates).