What can associative learning do for planning?

Apparent statistical inference in crows may reflect simple reinforcement learning

Johnston et al. report results which they argue demonstrate that crows engage in statistical inference during decision-making. They trained two crows to associate a set of stimuli with different reward probabilities (from 10% to 90%) before choice tests between pairs of stimuli. Across most pairwise combinations, and in a control task in which the number of rewards was equated between probabilities, both crows preferred the stimulus associated with higher reward probability. The magnitude of this preference was affected by the absolute difference between the two probabilities, although (contrary to a claim made by Johnston et al. 2023) preference did not reflect the ratio of prior probabilities independently of absolute differences. Johnston et al. argue that preference for the stimulus with the higher reward probability is "the signature of true statistical inference" (p. 3238), implemented by an analogue magnitude system that represents the reward probability associated with each stimulus. Here, we show that a simple reinforcement learning model, with no explicit representation of reward probabilities, reproduces the critical features of crows' performance-and indeed better accounts for the observed empirical findings than the concept of statistical inference based on analogue magnitude representations, because it correctly predicts the absence of a ratio effect that would reflect magnitudes when absolute distance is controlled. Contrary to Johnston et al.'s claims, these patterns of behaviour do not necessitate retrieval of calculated reward probabilities from long-term memory and dynamic application of this information across contexts, or (more specifically) require the involvement of an analogue magnitude system in representing abstract probabilities.

Cost does not prevent pigeons from investing in the future

One of the simplest forms of behavior, operant behavior, appears fundamentally prospective, implying potential similarity to 'sophisticated' prospective behaviors like planning in terms of underlying mechanisms. But differences between paradigms for studying behavior resulting from 'simple' versus 'sophisticated' mechanisms prevent true comparison of underlying mechanisms. To aid development of an operant paradigm with more similarity to 'sophisticated' prospective paradigms, we replicated and extended Cowie and Davison's (2021) investing task. Pigeons were required to emit an investing response to ensure food at a different time and different response location. We asked if investing depended on whether the behavior was a single, discrete key peck (typical in operant paradigms) or an extended sequence of pecks (echoing behaviors in planning paradigms), and whether facilitative effects of an immediate stimulus change persisted when the stimulus change no longer occurred. Pigeons invested successfully whether investing required one or more responses, and for extended investing responses, performance did not worsen significantly with increasing response requirements. Experience investing with an immediate stimulus change did not enhance subsequent investing without the stimulus change. Findings show simple learning mechanisms can support extended activities with no immediate consequences. Further, they support the investing paradigm as a potential tool for investigations of overlap in mechanisms controlling 'simple' and 'sophisticated' behavior.

Assessment of positive experiences using associative learning in chickens

To determine whether differences in positive experiences can influence associative learning ability, 2 tasks were conducted with 90 laying hens at the peak of the laying period. The selected hens were reared in a larger flock under the same housing conditions without perches, so they had the same rearing experiences and were moved to either enriched cages or to the floor system at 16 wk of age. They belonged to 3 breeds (Slovenian barred hen: Ba; brown hen: Br; silver hen, S), with 30 hens per breed. The predictor signal, the sound of a clicker, with a 3-second delayed reinforcer (commercial layer feed) was used to mark the desired behaviour (pecking for feed). Hens that associated feed with a clicker (85.06%) were taught 2 tasks, the colour discrimination task (CD) and the target following task (TF). In the CD, the hens had to discriminate between yellow, red and blue colors and peck at a yellow magnet only. In the TF, the desired behaviour was to follow a target, a purple ball on a stick, from 1 perch to another and peck it at the end of the perch. The main results of the hens associating the signal with feed were that the Ba hens learned faster than the S hens (P = 0.006) and required fewer clicker sounds than the Br hens in the CD (P = 0.003). Floor hens that completed CD or TF or both took less time to complete the task (CD, P = 0.03, TF, P = 0.06; both tasks, P = 0.02) or with fewer clicker sounds (CD task; P = 0.02) than cage hens. Although these results suggest that Ba hens and floor hens showed better associate learning performance, probably because they perceived their environment as more rewarding and thus potentially more positive, caution is needed in interpretation considering social experiences and that the ability to perch and the ability to move quickly on a perch can be confounded in TF and both tasks can be confounded with foraging ability.

A sequence bottleneck for animal intelligence and language?

We discuss recent findings suggesting that non-human animals lack memory for stimulus sequences, and therefore do not represent the order of stimuli faithfully. These observations have far-reaching consequences for animal cognition, neuroscience, and studies of the evolution of language and culture. This is because, if non-human animals do not remember or process information about order faithfully, then it is unlikely that non-human animals perform mental simulations, construct mental world models, have episodic memory, or transmit culture faithfully. If this suggested sequence bottleneck proves to be a prevalent characteristic of animal memory systems, as suggested by recent work, it would require a re-examination of some influential concepts and ideas.

The role of hearing and listening in hypnotic responsiveness

The hypnosis literature emphasizes features of suggestion, induction, and communication that are described as evoking heightened responsiveness on the part of the subject or client. This article explores what is often overlooked: to have an effect, the subject must not only hear the suggestion but listen to it. The process of listening is described across multiple levels ranging from the acoustic signal to its transduction into a meaning-filled and motivationally enriched message that spurs action. That journey traverses challenging terrain, with numerous obstacles that serve to maintain past habits and response patterns, despite a client's stated desire for adaptive change. The article highlights those obstacles and then provides descriptions of five language structures that can reach the client consciously or non-consciously, but always in ways that increase the odds that the therapeutic messaging is packaged in a manner that optimizes it being "heard," absorbed, and enacted. Applications of each language structure are provided using relevant clinical case examples.

Limits of flexibility and associative learning in pigeons

In a recent study, Wasserman, Kain, and O'Donoghue (Current Biology, 33(6), 1112-1116, 2023) set out to resolve the associative learning paradox by showing that pigeons can solve a complex category learning task through associative learning. The present Outlook paper presents their findings, expands on this paradox, and discusses implications of their results.

Sequence representation as an early step in the evolution of language

Human language is unique in its compositional, open-ended, and sequential form, and its evolution is often solely explained by advantages of communication. However, it has proven challenging to identify an evolutionary trajectory from a world without language to a world with language, especially while at the same time explaining why such an advantageous phenomenon has not evolved in other animals. Decoding sequential information is necessary for language, making domain-general sequence representation a tentative basic requirement for the evolution of language and other uniquely human phenomena. Here, using formal evolutionary analyses of the utility of sequence representation we show that sequence representation is exceedingly costly and that current memory systems found in animals may prevent abilities necessary for language to emerge. For sequence representation to evolve, flexibility allowing for ignoring irrelevant information is necessary. Furthermore, an abundance of useful sequential information and extensive learning opportunities are required, two conditions that were likely fulfilled early in human evolution. Our results provide a novel, logically plausible trajectory for the evolution of uniquely human cognition and language, and support the hypothesis that human culture is rooted in sequential representational and processing abilities.

Why is the Rescorla-Wagner model so influential?

The influence of the Rescorla-Wagner model cannot be overestimated, despite that (1) the model does not differ much computationally from its predecessors and competitors, and (2) its shortcomings are well-known in the learning community. Here we discuss the reasons behind its widespread influence in the cognitive and neural sciences, and argue that it is the constant search for general-process theories by learning scholars which eventually produced a model whose application spans many different areas of research to this day. We focus on the theoretical and empirical background of the model, the theoretical connections that it has with later developments across Marr's levels of analysis, as well as the broad variety of research that it has guided and inspired.

A test of memory for stimulus sequences in great apes

Identifying cognitive capacities underlying the human evolutionary transition is challenging, and many hypotheses exist for what makes humans capable of, for example, producing and understanding language, preparing meals, and having culture on a grand scale. Instead of describing processes whereby information is processed, recent studies have suggested that there are key differences between humans and other animals in how information is recognized and remembered. Such constraints may act as a bottleneck for subsequent information processing and behavior, proving important for understanding differences between humans and other animals. We briefly discuss different sequential aspects of cognition and behavior and the importance of distinguishing between simultaneous and sequential input, and conclude that explicit tests on non-human great apes have been lacking. Here, we test the memory for stimulus sequences-hypothesis by carrying out three tests on bonobos and one test on humans. Our results show that bonobos' general working memory decays rapidly and that they fail to learn the difference between the order of two stimuli even after more than 2,000 trials, corroborating earlier findings in other animals. However, as expected, humans solve the same sequence discrimination almost immediately. The explicit test on whether bonobos represent stimulus sequences as an unstructured collection of memory traces was not informative as no differences were found between responses to the different probe tests. However, overall, this first empirical study of sequence discrimination on non-human great apes supports the idea that non-human animals, including the closest relatives to humans, lack a memory for stimulus sequences. This may be an ability that sets humans apart from other animals and could be one reason behind the origin of human culture.

Computational models of episodic-like memory in food-caching birds

Birds of the crow family adapt food-caching strategies to anticipated needs at the time of cache recovery and rely on memory of the what, where and when of previous caching events to recover their hidden food. It is unclear if this behavior can be explained by simple associative learning or if it relies on higher cognitive processes like mental time-travel. We present a computational model and propose a neural implementation of food-caching behavior. The model has hunger variables for motivational control, reward-modulated update of retrieval and caching policies and an associative neural network for remembering caching events with a memory consolidation mechanism for flexible decoding of the age of a memory. Our methodology of formalizing experimental protocols is transferable to other domains and facilitates model evaluation and experiment design. Here, we show that memory-augmented, associative reinforcement learning without mental time-travel is sufficient to explain the results of 28 behavioral experiments with food-caching birds.

Resolving the associative learning paradox by category learning in pigeons

A wealth of evidence indicates that humans can engage two types of mechanisms to solve category-learning tasks: declarative mechanisms, which involve forming and testing verbalizable decision rules, and associative mechanisms, which involve gradually linking stimuli to appropriate behavioral responses.1,2,3 In contrast to declarative mechanisms, associative mechanisms have received surprisingly little attention in the broader category-learning literature. Although various forms of associatively driven artificial intelligence (AI) have matched-and even surpassed-humans' performance on several challenging problems,3,4,5,6 associative learning is routinely dismissed as being too simple to power the impressive cognitive achievements of both humans and non-human species.6,7,8,9 Here, we attempt to resolve this paradox by demonstrating that pigeons-which appear to rely solely on associative learning mechanisms in several tasks that promote declarative rule use by humans3,10,11,12-succeed at learning a novel, highly demanding category structure that ought to hinder declarative rule use: the sectioned-rings task. Our findings highlight the power and flexibility that associative mechanisms afford in the realm of category learning.

No evidence for future planning in Canada jays (Perisoreus canadensis)

In the past 20 years, research in animal cognition has challenged the belief that complex cognitive processes are uniquely human. At the forefront of these challenges has been research on mental time travel and future planning in jays. We tested whether Canada jays (Perisoreus canadensis) demonstrated future planning, using a procedure that has produced evidence of future planning in California scrub-jays. Future planning in this procedure is caching in locations where the bird will predictably experience a lack of food in the future. Canada jays showed no evidence of future planning in this sense and instead cached in the location where food was usually available, opposite to the behaviour described for California scrub-jays. We provide potential explanations for these differing results adding to the recent debates about the role of complex cognition in corvid caching strategies.

Foraging efficiency in temporally predictable environments: is a long-term temporal memory really advantageous?

Cognitive abilities enabling animals that feed on ephemeral but yearly renewable resources to infer when resources are available may have been favoured by natural selection, but the magnitude of the benefits brought by these abilities remains poorly known. Using computer simulations, we compared the efficiencies of three main types of foragers with different abilities to process temporal information, in spatially and/or temporally homogeneous or heterogeneous environments. One was endowed with a sampling memory, which stores recent experience about the availability of the different food types. The other two were endowed with a chronological or associative memory, which stores long-term temporal information about absolute times of these availabilities or delays between them, respectively. To determine the range of possible efficiencies, we also simulated a forager without temporal cognition but which simply targeted the closest and possibly empty food sources, and a perfectly prescient forager, able to know at any time which food source was effectively providing food. The sampling, associative and chronological foragers were far more efficient than the forager without temporal cognition in temporally predictable environments, and interestingly, their efficiencies increased with the level of temporal heterogeneity. The use of a long-term temporal memory results in a foraging efficiency up to 1.16 times better (chronological memory) or 1.14 times worse (associative memory) than the use of a simple sampling memory. Our results thus show that, for everyday foraging, a long-term temporal memory did not provide a clear benefit over a simple short-term memory that keeps track of the current resource availability. Long-term temporal memories may therefore have emerged in contexts where short-term temporal cognition is useless, i.e. when the anticipation of future environmental changes is strongly needed.

The hidden side of animal cognition research: Scientists' attitudes toward bias, replicability and scientific practice

Animal cognition research aims to understand animal minds by using a diverse range of methods across an equally diverse range of species. Throughout its history, the field has sought to mitigate various biases that occur when studying animal minds, from experimenter effects to anthropomorphism. Recently, there has also been a focus on how common scientific practices might affect the reliability and validity of published research. Usually, these issues are discussed in the literature by a small group of scholars with a specific interest in the topics. This study aimed to survey a wider range of animal cognition researchers to ask about their attitudes towards classic and contemporary issues facing the field. Two-hundred and ten active animal cognition researchers completed our survey, and provided answers on questions relating to bias, replicability, statistics, publication, and belief in animal cognition. Collectively, researchers were wary of bias in the research field, but less so in their own work. Over 70% of researchers endorsed Morgan’s canon as a useful principle but many caveated this in their free-text responses. Researchers self-reported that most of their studies had been published, however they often reported that studies went unpublished because they had negative or inconclusive results, or results that questioned “preferred” theories. Researchers rarely reported having performed questionable research practices themselves—however they thought that other researchers sometimes (52.7% of responses) or often (27.9% of responses) perform them. Researchers near unanimously agreed that replication studies are important but too infrequently performed in animal cognition research, 73.0% of respondents suggested areas of animal cognition research could experience a ‘replication crisis’ if replication studies were performed. Consistently, participants’ free-text responses provided a nuanced picture of the challenges animal cognition research faces, which are available as part of an open dataset. However, many researchers appeared concerned with how to interpret negative results, publication bias, theoretical bias and reliability in areas of animal cognition research. Collectively, these data provide a candid overview of barriers to progress in animal cognition and can inform debates on how individual researchers, as well as organizations and journals, can facilitate robust scientific research in animal cognition.

Pigeons prefer to invest early for future reinforcers

Five pigeons were trained in a series of conditions in which food was delivered after 25 responses, but only when a different (Investing) response had been made before the 25 responses had been completed. If an Investing response was not made, the 25 responses ended in blackout. In various conditions, effective Investing responses either had to be made before the first of the 25 responses, or anywhere within the 25 responses; and effective Investing responses either resulted in a stimulus change or did not. Pigeons Invested even when the consequences were temporally and spatially distant, but Investing was most likely when it produced an immediate stimulus change. When given the choice, pigeons preferred to make Investing responses at the beginning of a trial. These findings again demonstrate that behavior may be maintained by events that are separated in time and space from the present.

Replications, Comparisons, Sampling and the Problem of Representativeness in Animal Cognition Research

Animal cognition research often involves small and idiosyncratic samples. This can constrain the generalizability and replicability of a study's results and prevent meaningful comparisons between samples. However, there is little consensus about what makes a strong replication or comparison in animal research. We apply a resampling definition of replication to answer these questions in Part 1 of this article, and, in Part 2, we focus on the problem of representativeness in animal research. Through a case study and a simulation study, we highlight how and when representativeness may be an issue in animal behavior and cognition research and show how the representativeness problems can be viewed through the lenses of, i) replicability, ii) generalizability and external validity, iii) pseudoreplication and, iv) theory testing. Next, we discuss when and how researchers can improve their ability to learn from small sample research through, i) increasing heterogeneity in experimental design, ii) increasing homogeneity in experimental design, and, iii) statistically modeling variation. Finally, we describe how the strongest solutions will vary depending on the goals and resources of individual research programs and discuss some barriers towards implementing them.

Cuttlefish exert self-control in a delay of gratification task

The ability to exert self-control varies within and across taxa. Some species can exert self-control for several seconds whereas others, such as large-brained vertebrates, can tolerate delays of up to several minutes. Advanced self-control has been linked to better performance in cognitive tasks and has been hypothesized to evolve in response to specific socio-ecological pressures. These pressures are difficult to uncouple because previously studied species face similar socio-ecological challenges. Here, we investigate self-control and learning performance in cuttlefish, an invertebrate that is thought to have evolved under partially different pressures to previously studied vertebrates. To test self-control, cuttlefish were presented with a delay maintenance task, which measures an individual's ability to forgo immediate gratification and sustain a delay for a better but delayed reward. Cuttlefish maintained delay durations for up to 50-130 s. To test learning performance, we used a reversal-learning task, whereby cuttlefish were required to learn to associate the reward with one of two stimuli and then subsequently learn to associate the reward with the alternative stimulus. Cuttlefish that delayed gratification for longer had better learning performance. Our results demonstrate that cuttlefish can tolerate delays to obtain food of higher quality comparable to that of some large-brained vertebrates.

Ants Can Anticipate the Following Quantity in an Arithmetic Sequence

Workers of the ant Myrmica sabuleti have been previously shown to be able to add and subtract numbers of elements and to expect the time and location of the next food delivery. We wanted to know if they could anticipate the following quantity of elements present near their food when the number of these elements increases or decreases over time according to an arithmetic sequence. Two experiments were therefore carried out, one with an increasing sequence, the other with a decreasing sequence. Each experiment consisted of two steps, one for the ants to learn the numbers of elements successively present near their food, the other to test their choice when they were simultaneously in the presence of the numbers from a previously learned sequence and the following quantity. The ants anticipated the following quantity in each presented numerical sequence. This forethinking of the next quantity applies to numerosity, thus, to concrete items. This anticipatory behavior may be explained by associative learning and by the ants’ ability to memorize events and to estimate the elapsing time.

New Caledonian crows plan for specific future tool use

The ability to plan for future events is one of the defining features of human intelligence. Whether non-human animals can plan for specific future situations remains contentious: despite a sustained research effort over the last two decades, there is still no consensus on this question. Here, we show that New Caledonian crows can use tools to plan for specific future events. Crows learned a temporal sequence where they were (a) shown a baited apparatus, (b) 5 min later given a choice of five objects and (c) 10 min later given access to the apparatus. At test, these crows were presented with one of two tool-apparatus combinations. For each combination, the crows chose the right tool for the right future task, while ignoring previously useful tools and a low-value food item. This study establishes that planning for specific future tool use can evolve via convergent evolution, given that corvids and humans shared a common ancestor over 300 million years ago, and offers a route to mapping the planning capacities of animals.

Building Thinking Machines by Solving Animal Cognition Tasks

In ‘Computing Machinery and Intelligence’, Turing, sceptical of the question ‘Can machines think?’, quickly replaces it with an experimentally verifiable test: the imitation game. I suggest that for such a move to be successful the test needs to be relevant, expansive, solvable by exemplars, unpredictable, and lead to actionable research. The Imitation Game is only partially successful in this regard and its reliance on language, whilst insightful for partially solving the problem, has put AI progress on the wrong foot, prescribing a top-down approach for building thinking machines. I argue that to fix shortcomings with modern AI systems a nonverbal operationalisation is required. This is provided by the recent Animal-AI Testbed, which translates animal cognition tests for AI and provides a bottom-up research pathway for building thinking machines that create predictive models of their environment from sensory input.

A-learning: A new formulation of associative learning theory

We present a new mathematical formulation of associative learning focused on non-human animals, which we call A-learning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

Macphail's Null Hypothesis of Vertebrate Intelligence: Insights From Avian Cognition

Macphail famously criticized two foundational assumptions that underlie the evolutionary approach to comparative psychology: that there are differences in intelligence across species, and that intelligent behavior in animals is based on more than associative learning. Here, we provide evidence from recent work in avian cognition that supports both these assumptions: intelligence across species varies, and animals can perform intelligent behaviors that are not guided solely by associative learning mechanisms. Finally, we reflect on the limitations of comparative psychology that led to Macphail's claims and suggest strategies researchers can use to make more advances in the field.

Temporal Junctures in the Mind

Humans can imagine what happened in the past and what will happen in the future, but also what did not happen and what might happen. We reflect on envisioned events from alternative timelines, while knowing that we only ever live on one timeline. Considering alternative timelines rests on representations of temporal junctures, or points in time at which possible versions of reality diverge. These representations become increasingly sophisticated over childhood, first enabling preparation for mutually exclusive future possibilities and later the experience of counterfactual emotions like regret. By contrast, it remains unclear whether non-human animals represent temporal junctures at all. The emergence of these representations may have been a prime mover in human evolution.

Thinking about thinking about time

Hoerl & McCormack (H&M) discuss the possible function of meta-representations in temporal cognition but ultimately take an agnostic stance. Here we outline the fundamental role that we believe meta-representations play. Because humans know that their representations of future events are just representations, they are in a position to compensate for the shortcomings of their own foresight and to prepare for multiple contingencies.

Social learning through associative processes: a computational theory

Social transmission of information is a key phenomenon in the evolution of behaviour and in the establishment of traditions and culture. The diversity of social learning phenomena has engendered a diverse terminology and numerous ideas about underlying learning mechanisms, at the same time that some researchers have called for a unitary analysis of social learning in terms of associative processes. Leveraging previous attempts and a recent computational formulation of associative learning, we analyse the following learning scenarios in some generality: learning responses to social stimuli, including learning to imitate; learning responses to non-social stimuli; learning sequences of actions; learning to avoid danger. We conceptualize social learning as situations in which stimuli that arise from other individuals have an important role in learning. This role is supported by genetic predispositions that either cause responses to social stimuli or enable social stimuli to reinforce specific responses. Simulations were performed using a new learning simulator program. The simulator is publicly available and can be used for further theoretical investigations and to guide empirical research of learning and behaviour. Our explorations show that, when guided by genetic predispositions, associative processes can give rise to a wide variety of social learning phenomena, such as stimulus and local enhancement, contextual imitation and simple production imitation, observational conditioning, and social and response facilitation. In addition, we clarify how associative mechanisms can result in transfer of information and behaviour from experienced to naive individuals.