The power of associative learning and the ontogeny of optimal behaviour

Limits of flexibility and associative learning in pigeons

In a recent study, Wasserman, Kain, and O'Donoghue (Current Biology, 33(6), 1112-1116, 2023) set out to resolve the associative learning paradox by showing that pigeons can solve a complex category learning task through associative learning. The present Outlook paper presents their findings, expands on this paradox, and discusses implications of their results.

Sequence representation as an early step in the evolution of language

Human language is unique in its compositional, open-ended, and sequential form, and its evolution is often solely explained by advantages of communication. However, it has proven challenging to identify an evolutionary trajectory from a world without language to a world with language, especially while at the same time explaining why such an advantageous phenomenon has not evolved in other animals. Decoding sequential information is necessary for language, making domain-general sequence representation a tentative basic requirement for the evolution of language and other uniquely human phenomena. Here, using formal evolutionary analyses of the utility of sequence representation we show that sequence representation is exceedingly costly and that current memory systems found in animals may prevent abilities necessary for language to emerge. For sequence representation to evolve, flexibility allowing for ignoring irrelevant information is necessary. Furthermore, an abundance of useful sequential information and extensive learning opportunities are required, two conditions that were likely fulfilled early in human evolution. Our results provide a novel, logically plausible trajectory for the evolution of uniquely human cognition and language, and support the hypothesis that human culture is rooted in sequential representational and processing abilities.

Conditions under which faithful cultural transmission through teaching promotes cumulative cultural evolution

It has been argued that teaching promotes the accurate transmission of cultural traits and eventually leads to cumulative cultural evolution (CCE). However, previous studies have questioned this argument. In this study, we modified the action sequences model into a network exploring model with reinforcement learning to examine the conditions under which teaching promotes CCE. Our model incorporates a time trade-off between innovation and teaching. Simulations revealed that the positive influence of teaching on CCE depends on task difficulty. When the task was too difficult and advanced, such that it could not be accomplished through individual learning within a limited time, spending more time on teaching-even at the expense of time for innovation-contributed to CCE. On the contrary, the easier the task, the more time was spent on innovation than on teaching, which contributed to the improvement of performance. These findings suggest that teaching becomes more valuable as cultures become more complex. Therefore, humanity must have co-evolved a complex cumulative culture and teaching that supports cultural fidelity.

A test of memory for stimulus sequences in great apes

Identifying cognitive capacities underlying the human evolutionary transition is challenging, and many hypotheses exist for what makes humans capable of, for example, producing and understanding language, preparing meals, and having culture on a grand scale. Instead of describing processes whereby information is processed, recent studies have suggested that there are key differences between humans and other animals in how information is recognized and remembered. Such constraints may act as a bottleneck for subsequent information processing and behavior, proving important for understanding differences between humans and other animals. We briefly discuss different sequential aspects of cognition and behavior and the importance of distinguishing between simultaneous and sequential input, and conclude that explicit tests on non-human great apes have been lacking. Here, we test the memory for stimulus sequences-hypothesis by carrying out three tests on bonobos and one test on humans. Our results show that bonobos' general working memory decays rapidly and that they fail to learn the difference between the order of two stimuli even after more than 2,000 trials, corroborating earlier findings in other animals. However, as expected, humans solve the same sequence discrimination almost immediately. The explicit test on whether bonobos represent stimulus sequences as an unstructured collection of memory traces was not informative as no differences were found between responses to the different probe tests. However, overall, this first empirical study of sequence discrimination on non-human great apes supports the idea that non-human animals, including the closest relatives to humans, lack a memory for stimulus sequences. This may be an ability that sets humans apart from other animals and could be one reason behind the origin of human culture.

The behavioral signature of stepwise learning strategy in male rats and its neural correlate in the basal forebrain

Studies of associative learning have commonly focused on how rewarding outcomes are predicted by either sensory stimuli or animals' actions. However, in many learning scenarios, reward delivery requires the occurrence of both sensory stimuli and animals' actions in a specific order, in the form of behavioral sequences. How such behavioral sequences are learned is much less understood. Here we provide behavioral and neurophysiological evidence to show that behavioral sequences are learned using a stepwise strategy. In male rats learning a new association, learning started from the behavioral event closest to the reward and sequentially incorporated earlier events. This led to the sequential refinement of reward-seeking behaviors, which was characterized by the stepwise elimination of ineffective and non-rewarded behavioral sequences. At the neuronal level, this stepwise learning process was mirrored by the sequential emergence of basal forebrain neuronal responses toward each event, which quantitatively conveyed a reward prediction error signal and promoted reward-seeking behaviors. Together, these behavioral and neural signatures revealed how behavioral sequences were learned in discrete steps and when each learning step took place.

From Prediction to Action: Dissociable Roles of Ventral Tegmental Area and Substantia Nigra Dopamine Neurons in Instrumental Reinforcement

Reward seeking requires the coordination of motor programs to achieve goals. Midbrain dopamine neurons are critical for reinforcement, and their activation is sufficient for learning about cues, actions, and outcomes. Here we examine in detail the mechanisms underlying the ability of ventral tegmental area (VTA) and substantia nigra (SNc) dopamine neurons to support instrumental learning. By exploiting numerous behavioral tasks in combination with time-limited optogenetic manipulations in male and female rats, we reveal that VTA and SNc dopamine neurons generate reinforcement through separable psychological processes. VTA dopamine neurons imbue actions and their associated cues with motivational value that allows flexible and persistent pursuit, whereas SNc dopamine neurons support time-limited, precise, action-specific learning that is nonscalable and inflexible. This architecture is reminiscent of actor-critic reinforcement learning models with VTA and SNc instructing the critic and actor, respectively. Our findings indicate that heterogeneous dopamine systems support unique forms of instrumental learning that ultimately result in disparate reward-seeking strategies.SIGNIFICANCE STATEMENT Dopamine neurons in the midbrain are essential for learning, motivation, and movement. Here we describe in detail the ability of VTA and SNc dopamine neurons to generate instrumental reinforcement, a process where an agent learns about actions they can emit to earn reward. While rats will avidly work and learn to respond for activation of VTA and SNc dopamine neurons, we find that only VTA dopamine neurons imbue actions and their associated cues with motivational value that spur continued pursuit of reward. Our data support a hypothesis that VTA and SNc dopamine neurons engage distinct psychological processes that have consequences for our understanding of these neurons in health and disease.

Models of Conditioned Reinforcement and Abnormal Behaviour in Captive Animals

Abnormal behaviours are common in captive animals, and despite a lot of research, the development, maintenance and alleviation of these behaviours are not fully understood. Here, we suggest that conditioned reinforcement can induce sequential dependencies in behaviour that are difficult to infer from direct observation. We develop this hypothesis using recent models of associative learning that include conditioned reinforcement and inborn facets of behaviour, such as predisposed responses and motivational systems. We explore three scenarios in which abnormal behaviour emerges from a combination of associative learning and a mismatch between the captive environment and inborn predispositions. The first model considers how abnormal behaviours, such as locomotor stereotypies, may arise from certain spatial locations acquiring conditioned reinforcement value. The second model shows that conditioned reinforcement can give rise to abnormal behaviour in response to stimuli that regularly precede food or other reinforcers. The third model shows that abnormal behaviour can result from motivational systems being adapted to natural environments that have different temporal structures than the captive environment. We conclude that models including conditioned reinforcement offer an important theoretical insight regarding the complex relationships between captive environments, inborn predispositions, and learning. In the future, this general framework could allow us to further understand and possibly alleviate abnormal behaviours.

The Popcorn Illusion

A popcorn popping is almost magical. And yet, the science of popcorn is safe and clear about the steps until the pop: the components, processes, and results of making popcorn. Nature has its own way to produce surprise in the form of "pops" (i.e., emergence, qualitative shifts). Emergent features spread throughout the life of taxa and individuals. A pop can be sudden and chaotic. And so is creativity. There is no incompatibility between creativity and naturalistic endeavors in science. Creativity is no god given gift blown inside humans. When creativity is defined by originality and spontaneity, it describes a feature with no past or present. I briefly summarize how one can see non-random innovation, no free occurring spontaneity, and non-heuristic effectiveness as features of behaviors that are not necessarily considered creative. Those three features reveal how traditional views of creativity undermine its real determiners and how it can be objectively defined and observed.

Midazolam Prevents the Adverse Outcome of Neonatal Asphyxia

OBJECTIVE

Birth asphyxia (BA) is the most frequent cause of neonatal death as well as central nervous system (CNS) injury. BA is often associated with neonatal seizures, which only poorly respond to anti-seizure medications and may contribute to the adverse neurodevelopmental outcome. Using a non-invasive rat model of BA, we have recently reported that the potent benzodiazepine, midazolam, prevents neonatal seizures in ~50% of rat pups. In addition to its anti-seizure effect, midazolam exerts anti-inflammatory actions, which is highly relevant for therapeutic intervention following BA. The 2 major aims of the present study were to examine (1) whether midazolam reduces the adverse outcome of BA, and (2) whether this effect is different in rats that did or did not exhibit neonatal seizures after drug treatment.

METHODS

Behavioral and cognitive tests were performed over 14 months after asphyxia, followed by immunohistochemical analyses.

RESULTS

All vehicle-treated rats had seizures after asphyxia and developed behavioral and cognitive abnormalities, neuroinflammation in gray and white matter, neurodegeneration in the hippocampus and thalamus, and hippocampal mossy fiber sprouting in subsequent months. Administration of midazolam (1 mg/kg i.p.) directly after asphyxia prevented post-asphyctic seizures in ~50% of the rats and resulted in the prevention or decrease of neuroinflammation and the behavioral, cognitive, and neurodegenerative consequences of asphyxia. Except for neurodegeneration in the thalamus, seizures did not seem to contribute to the adverse outcome of asphyxia.

INTERPRETATION

The disease-modifying effect of midazolam identified here strongly suggests that this drug provides a valuable option for improving the treatment and outcome of BA. ANN NEUROL 2023;93:226-243.

In search for consciousness in animals: Using working memory and voluntary attention as behavioral indicators

Whether animals have subjective experiences about the content of their sensory input, i.e., whether they are aware of stimuli, is a notoriously difficult question to answer. If consciousness is present in animals, it must share fundamental characteristics with human awareness. Working memory and voluntary/endogenous attention are suggested as diagnostic features of conscious awareness. Behavioral evidence shows clear signatures of both working memory and voluntary attention as minimal criterium for sensory consciousness in mammals and birds. In contrast, reptiles and amphibians show no sign of either working memory or volitional attention. Surprisingly, some species of teleost fishes exhibit elementary working memory and voluntary attention effects suggestive of possibly rudimentary forms of subjective experience. With the potential exception of honeybees, evidence for conscious processing is lacking in invertebrates. These findings suggest that consciousness is not ubiquitous in the animal kingdom but also not exclusive to humans. The phylogenetic gap between animal taxa argues that evolution does not rely on specific neural substrates to endow distantly related species with basic forms of consciousness.

Preliminary evidence for one-trial social learning of vervet monkey alarm calling

How do non-human primates learn to use their alarm calls? Social learning is a promising candidate, but its role in the acquisition of meaning and call usage has not been studied systematically, neither during ontogeny nor in adulthood. To investigate the role of social learning in alarm call comprehension and use, we exposed groups of wild vervet monkeys to two unfamiliar animal models in the presence or absence of conspecific alarm calls. To assess the learning outcome of these experiences, we then presented the models for a second time to the same monkeys, but now without additional alarm call information. In subjects previously exposed in conjunction with alarm calls, we found heightened predator inspection compared to control subjects exposed without alarm calls, indicating one-trial social learning of 'meaning'. Moreover, some juveniles (but not adults) produced the same alarm calls they heard during the initial exposure whereas the authenticity of the models had an additional effect. Our experiment provides preliminary evidence that, in non-human primates, call meaning can be acquired by one-trail social learning but that subject age and core knowledge about predators additionally moderate the acquisition of novel call-referent associations.

Preliminary evidence for one-trial social learning of vervet monkey alarm calling

How do non-human primates learn to use their alarm calls? Social learning is a promising candidate, but its role in the acquisition of meaning and call usage has not been studied systematically, neither during ontogeny nor in adulthood. To investigate the role of social learning in alarm call comprehension and use, we exposed groups of wild vervet monkeys to two unfamiliar animal models in the presence or absence of conspecific alarm calls. To assess the learning outcome of these experiences, we then presented the models for a second time to the same monkeys, but now without additional alarm call information. In subjects previously exposed in conjunction with alarm calls, we found heightened predator inspection compared to control subjects exposed without alarm calls, indicating one-trial social learning of 'meaning'. Moreover, some juveniles (but not adults) produced the same alarm calls they heard during the initial exposure whereas the authenticity of the models had an additional effect. Our experiment provides preliminary evidence that, in non-human primates, call meaning can be acquired by one-trail social learning but that subject age and core knowledge about predators additionally moderate the acquisition of novel call-referent associations.

Modelling how cleaner fish approach an ephemeral reward task demonstrates a role for ecologically tuned chunking in the evolution of advanced cognition

What makes cognition “advanced” is an open and not precisely defined question. One perspective involves increasing the complexity of associative learning, from conditioning to learning sequences of events (“chaining”) to representing various cue combinations as “chunks.” Here we develop a weighted graph model to study the mechanism enabling chunking ability and the conditions for its evolution and success, based on the ecology of the cleaner fish Labroides dimidiatus. In some environments, cleaners must learn to serve visitor clients before resident clients, because a visitor leaves if not attended while a resident waits for service. This challenge has been captured in various versions of the ephemeral reward task, which has been proven difficult for a range of cognitively capable species. We show that chaining is the minimal requirement for solving this task in its common simplified laboratory format that involves repeated simultaneous exposure to an ephemeral and permanent food source. Adding ephemeral–ephemeral and permanent–permanent combinations, as cleaners face in the wild, requires individuals to have chunking abilities to solve the task. Importantly, chunking parameters need to be calibrated to ecological conditions in order to produce adaptive decisions. Thus, it is the fine-tuning of this ability, which may be the major target of selection during the evolution of advanced associative learning.

What makes cognition ‘advanced’ is an open and not precisely defined question. In this study, a cognitive model of cleaner fish learning the ephemeral-reward task demonstrates how a critical step in cognitive evolution may be understood as the evolution of chunking and its tuning to fit ecological conditions.

Long-term outcome in a noninvasive rat model of birth asphyxia with neonatal seizures: Cognitive impairment, anxiety, epilepsy, and structural brain alterations

OBJECTIVE

Birth asphyxia is a major cause of hypoxic-ischemic encephalopathy (HIE) in neonates and often associated with mortality, neonatal seizures, brain damage, and later life motor, cognitive, and behavioral impairments and epilepsy. Preclinical studies on rodent models are needed to develop more effective therapies for preventing HIE and its consequences. Thus far, the most popular rodent models have used either exposure of intact animals to hypoxia-only, or a combination of hypoxia and carotid occlusion, for the induction of neonatal seizures and adverse outcomes. However, such models lack systemic hypercapnia, which is a fundamental constituent of birth asphyxia with major effects on neuronal excitability. Here, we use a recently developed noninvasive rat model of birth asphyxia with subsequent neonatal seizures to study later life adverse outcome.

METHODS

Intermittent asphyxia was induced for 30 min by exposing male and female postnatal day 11 rat pups to three 7 + 3-min cycles of 9% and 5% O₂ at constant 20% CO₂ . All pups exhibited convulsive seizures after asphyxia. A set of behavioral tests were performed systematically over 14 months following asphyxia, that is, a large part of the rat's life span. Video-electroencephalographic (EEG) monitoring was used to determine whether asphyxia led to the development of epilepsy. Finally, structural brain alterations were examined.

RESULTS

The animals showed impaired spatial learning and memory and increased anxiety when tested at an age of 3-14 months. Video-EEG at ~10 months showed an abundance of spontaneous seizures, which was paralleled by neurodegeneration in the hippocampus and thalamus, and by aberrant mossy fiber sprouting.

SIGNIFICANCE

The present model of birth asphyxia recapitulates several of the later life consequences associated with human HIE. This model thus allows evaluation of the efficacy of novel therapies designed to prevent HIE and seizures following asphyxia, and of how such therapies might alleviate long-term adverse consequences.

Deep associative learning approach for bio-medical sentiment analysis utilizing unsupervised representation from large-scale patients' narratives

Owing to the quick spread of minute health-related experiences, the distillation of knowledge from such unstructured narratives is an extremely challenging task. In spite of the success of neural networks methods in improving learning structural reliability, they result in inadequate accuracy of the bio-medical sentiment classification when employing less useful features sets. Therefore, they lack discriminatory potential. In this study, we propose to add a deep associative memory into neural networks for an effective sentiment decomposition, which emphasizes correctly on bio-medical entities related to the extraction of different data-object properties, and contextual-semantics dependencies for a given aspect. The underlying trust of these measures is behind the ability to compute the completion of unseen medical patterns, where comprehensive bio-medical distributed representations are used for representing the formal medical connections from PubMed databases. Experiments on a biomedical sentiment analysis task show that the model provides comprehensive embeddings with meaningful medical patterns. It achieved an average performance of 87% on varied large online datasets. It also outperforms baselines in discovering and identifying medical natural concepts. We provide meaningful support to bio-medical sentiment analysis applications in social networks. Indeed, the facets of this study might be used in many health concerns such as analyzing change in health status or unexpected situations.

Learning in single cell organisms

The survival of all species requires appropriate behavioral responses to environmental challenges. Learning is one of the key processes to acquire information about the environment and adapt to changing and uncertain conditions. Learning has long been acknowledged in animals from invertebrates to vertebrates but remains a subject of debate in non-animal systems such a plants and single cell organisms. In this review I will attempt to answer the following question: are single cell organisms capable of learning? I will first briefly discuss the concept of learning and argue that the ability to acquire and store information through learning is pervasive and may be found in single cell organisms. Second, by focusing on habituation, the simplest form of learning, I will review a series of experiments showing that single cell organisms such as slime molds and ciliates display habituation and follow most of the criteria adopted by neuroscientists to define habituation. Then I will discuss disputed evidence suggesting that single cell organisms might also undergo more sophisticated forms of learning such as associative learning. Finally, I will stress out that the challenge for the future is less about whether or not to single cell organisms fulfill the definition of learning established from extensive studies in animal systems and more about acknowledging and understanding the range of behavioral plasticity exhibited by such fascinating organisms.

Ants Can Anticipate the Following Quantity in an Arithmetic Sequence

Workers of the ant Myrmica sabuleti have been previously shown to be able to add and subtract numbers of elements and to expect the time and location of the next food delivery. We wanted to know if they could anticipate the following quantity of elements present near their food when the number of these elements increases or decreases over time according to an arithmetic sequence. Two experiments were therefore carried out, one with an increasing sequence, the other with a decreasing sequence. Each experiment consisted of two steps, one for the ants to learn the numbers of elements successively present near their food, the other to test their choice when they were simultaneously in the presence of the numbers from a previously learned sequence and the following quantity. The ants anticipated the following quantity in each presented numerical sequence. This forethinking of the next quantity applies to numerosity, thus, to concrete items. This anticipatory behavior may be explained by associative learning and by the ants’ ability to memorize events and to estimate the elapsing time.

Learning steers the ontogeny of an efficient hunting sequence in zebrafish larvae

Goal-directed behaviors may be poorly coordinated in young animals but, with age and experience, behavior progressively adapts to efficiently exploit the animal’s ecological niche. How experience impinges on the developing neural circuits of behavior is an open question. We have conducted a detailed study of the effects of experience on the ontogeny of hunting behavior in larval zebrafish. We report that larvae with prior experience of live prey consume considerably more prey than naive larvae. This is mainly due to increased capture success and a modest increase in hunt rate. We demonstrate that the initial turn to prey and the final capture manoeuvre of the hunting sequence were jointly modified by experience and that modification of these components predicted capture success. Our findings establish an ethologically relevant paradigm in zebrafish for studying how the brain is shaped by experience to drive the ontogeny of efficient behavior.

A-learning: A new formulation of associative learning theory

We present a new mathematical formulation of associative learning focused on non-human animals, which we call A-learning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

Learning to predict pain: differences in people with persistent neck pain and pain-free controls

Background

Learning to predict threatening events enables an organism to engage in protective behavior and prevent harm. Failure to differentiate between cues that truly predict danger and those that do not, however, may lead to indiscriminate fear and avoidance behaviors, which in turn may contribute to disability in people with persistent pain. We aimed to test whether people with persistent neck pain exhibit contingency learning deficits in predicting pain relative to pain-free, gender-and age-matched controls.

Method

We developed a differential predictive learning task with a neck pain-relevant scenario. During the acquisition phase, images displaying two distinct neck positions were presented and participants were asked to predict whether these neck positions would lead to pain in a fictive patient with persistent neck pain (see fictive patient scenario details in Appendix A). After participants gave their pain-expectancy judgment in the hypothetical scenario, the verbal outcome (PAIN or NO PAIN) was shown on the screen. One image (CS+) was followed by the outcome “PAIN”, while another image (CS−) was followed by the outcome “NO PAIN”. During the generalization phase, novel but related images depicting neck positions along a continuum between the CS+ and CS− images (generalization stimuli; GSs) were introduced to assess the generalization of acquired predictive learning to the novel images; the GSs were always followed by the verbal outcome “NOTES UNREADABLE” to prevent extinction learning. Finally, an extinction phase was included in which all images were followed by “NO PAIN” assessing the persistence of pain-expectancy judgments following disconfirming information.

Results

Differential pain-expectancy learning was reduced in people with neck pain relative to controls, resulting from patients giving significantly lower pain-expectancy judgments for the CS+, and significantly higher pain-expectancy judgments for the CS−. People with neck pain also demonstrated flatter generalization gradients relative to controls. No differences in extinction were noted.

Discussion

The results support the hypothesis that people with persistent neck pain exhibit reduced differential pain-expectancy learning and flatter generalization gradients, reflecting deficits in predictive learning. Contrary to our hypothesis, no differences in extinction were found. These findings may be relevant to understanding behavioral aspects of chronic pain.

Modeling the evolution of sensitive periods

In the past decade, there has been monumental progress in our understanding of the neurobiological basis of sensitive periods. Little is known, however, about the evolution of sensitive periods. Recent studies have started to address this gap. Biologists have built mathematical models exploring the environmental conditions in which sensitive periods are likely to evolve. These models investigate how mechanisms of plasticity can respond optimally to experience during an individual's lifetime. This paper discusses the central tenets, insights, and predictions of these models, in relation to empirical work on humans and other animals. We also discuss which future models are needed to improve the bridge between theory and data, advancing their synergy. Our paper is written in an accessible manner and for a broad audience. We hope our work will contribute to recently emerging connections between the fields of developmental neuroscience and evolutionary biology.

Computational optimization of associative learning experiments

With computational biology striving to provide more accurate theoretical accounts of biological systems, use of increasingly complex computational models seems inevitable. However, this trend engenders a challenge of optimal experimental design: due to the flexibility of complex models, it is difficult to intuitively design experiments that will efficiently expose differences between candidate models or allow accurate estimation of their parameters. This challenge is well exemplified in associative learning research. Associative learning theory has a rich tradition of computational modeling, resulting in a growing space of increasingly complex models, which in turn renders manual design of informative experiments difficult. Here we propose a novel method for computational optimization of associative learning experiments. We first formalize associative learning experiments using a low number of tunable design variables, to make optimization tractable. Next, we combine simulation-based Bayesian experimental design with Bayesian optimization to arrive at a flexible method of tuning design variables. Finally, we validate the proposed method through extensive simulations covering both the objectives of accurate parameter estimation and model selection. The validation results show that computationally optimized experimental designs have the potential to substantially improve upon manual designs drawn from the literature, even when prior information guiding the optimization is scarce. Computational optimization of experiments may help address recent concerns over reproducibility by increasing the expected utility of studies, and it may even incentivize practices such as study pre-registration, since optimization requires a pre-specified analysis plan. Moreover, design optimization has the potential not only to improve basic research in domains such as associative learning, but also to play an important role in translational research. For example, design of behavioral and physiological diagnostic tests in the nascent field of computational psychiatry could benefit from an optimization-based approach, similar to the one presented here.

Learning leads to bounded rationality and the evolution of cognitive bias in public goods games

In social interactions, including cooperation and conflict, individuals can adjust their behaviour over the shorter term through learning within a generation, and natural selection can change behaviour over the longer term of many generations. Here we investigate the evolution of cognitive bias by individuals investing into a project that delivers joint benefits. For members of a group that learn how much to invest using the costs and benefits they experience in repeated interactions, we show that overestimation of the cost of investing can evolve. The bias causes individuals to invest less into the project. Our explanation is that learning responds to immediate rather than longer-term rewards. There are thus cognitive limitations in learning, which can be seen as bounded rationality. Over a time horizon of several rounds of interaction, individuals respond to each other’s investments, for instance by partially compensating for another’s shortfall. However, learning individuals fail to strategically take into account that social partners respond in this way. Learning instead converges to a one-shot Nash equilibrium of a game with perceived rewards as payoffs. Evolution of bias can then compensate for the cognitive limitations of learning.

Social learning through associative processes: a computational theory

Social transmission of information is a key phenomenon in the evolution of behaviour and in the establishment of traditions and culture. The diversity of social learning phenomena has engendered a diverse terminology and numerous ideas about underlying learning mechanisms, at the same time that some researchers have called for a unitary analysis of social learning in terms of associative processes. Leveraging previous attempts and a recent computational formulation of associative learning, we analyse the following learning scenarios in some generality: learning responses to social stimuli, including learning to imitate; learning responses to non-social stimuli; learning sequences of actions; learning to avoid danger. We conceptualize social learning as situations in which stimuli that arise from other individuals have an important role in learning. This role is supported by genetic predispositions that either cause responses to social stimuli or enable social stimuli to reinforce specific responses. Simulations were performed using a new learning simulator program. The simulator is publicly available and can be used for further theoretical investigations and to guide empirical research of learning and behaviour. Our explorations show that, when guided by genetic predispositions, associative processes can give rise to a wide variety of social learning phenomena, such as stimulus and local enhancement, contextual imitation and simple production imitation, observational conditioning, and social and response facilitation. In addition, we clarify how associative mechanisms can result in transfer of information and behaviour from experienced to naive individuals.

Portia’s capacity to decide whether a detour is necessary

Proficiency at planning is known to be part of the exceptionally complex predatory repertoire of Portia, a genus of jumping spiders (Salticidae) that specialize at preying on other spiders. This includes proficiency at choosing between two detour routes, with only one leading to otherwise inaccessible prey. Less is known about Portia’s proficiency at making strategic decisions pertaining to whether a detour is required or not. By using Portia africana, we investigated this by having lures (prey or leaf pieces) visible at the beginning of a trial but not later, and by using water for restricting Portia’s freedom of movement. A detour path was always present, but sometimes a causeway was also present, allowing for direct access to lures. After seeing prey, Portia more often took the causeway when present and, when absent, more often took the detour path. After seeing leaf pieces, Portia never took the detour path.

What can associative learning do for planning?

There is a new associative learning paradox. The power of associative learning for producing flexible behaviour in non-human animals is downplayed or ignored by researchers in animal cognition, whereas artificial intelligence research shows that associative learning models can beat humans in chess. One phenomenon in which associative learning often is ruled out as an explanation for animal behaviour is flexible planning. However, planning studies have been criticized and questions have been raised regarding both methodological validity and interpretations of results. Due to the power of associative learning and the uncertainty of what causes planning behaviour in non-human animals, I explored what associative learning can do for planning. A previously published sequence learning model which combines Pavlovian and instrumental conditioning was used to simulate two planning studies, namely Mulcahy & Call 2006 'Apes save tools for future use.' Science 312, 1038-1040 and Kabadayi & Osvath 2017 'Ravens parallel great apes in flexible planning for tool-use and bartering.' Science 357, 202-204. Simulations show that behaviour matching current definitions of flexible planning can emerge through associative learning. Through conditioned reinforcement, the learning model gives rise to planning behaviour by learning that a behaviour towards a current stimulus will produce high value food at a later stage; it can make decisions about future states not within current sensory scope. The simulations tracked key patterns both between and within studies. It is concluded that one cannot rule out that these studies of flexible planning in apes and corvids can be completely accounted for by associative learning. Future empirical studies of flexible planning in non-human animals can benefit from theoretical developments within artificial intelligence and animal learning.

Toward a Unifying Account of Impulsivity and the Development of Self-Control

Impulsivity has traditionally been thought to involve various behavioral traits that can be measured using different laboratory protocols. Whereas some authors regard different measures of impulsivity as reflecting fundamentally distinct and unrelated behavioral tendencies (fragmentation approach), others regard those different indexes as analogue forms of the same behavioral tendency, only superficially different (unification approach). Unifying accounts range from mere intuitions to more sophisticated theoretical systems. Some of the more complete attempts at unifying are intriguing but have validity weaknesses. We propose a new unifying attempt based on theoretical points posed by other authors and supplemented by theory and research on associative learning. We then apply these assumptions to characterize the paradigms used to study impulsivity in laboratory settings and evaluate their scope as an attempt at unification. We argue that our approach possesses a good balance of parsimony and empirical and theoretical grounding, as well as a more encompassing scope, and is more suitable for experimental testing than previous theoretical frameworks. In addition, the proposed approach is capable of generating a new definition of impulsivity and outlines a hypothesis of how self-control can be developed. Finally, we examine the fragmentation approach from a different perspective, emphasizing the importance of finding similarities among seemingly different phenomena.

Enriching behavioral ecology with reinforcement learning methods

This article focuses on the division of labor between evolution and development in solving sequential, state-dependent decision problems. Currently, behavioral ecologists tend to use dynamic programming methods to study such problems. These methods are successful at predicting animal behavior in a variety of contexts. However, they depend on a distinct set of assumptions. Here, we argue that behavioral ecology will benefit from drawing more than it currently does on a complementary collection of tools, called reinforcement learning methods. These methods allow for the study of behavior in highly complex environments, which conventional dynamic programming methods do not feasibly address. In addition, reinforcement learning methods are well-suited to studying how biological mechanisms solve developmental and learning problems. For instance, we can use them to study simple rules that perform well in complex environments. Or to investigate under what conditions natural selection favors fixed, non-plastic traits (which do not vary across individuals), cue-driven-switch plasticity (innate instructions for adaptive behavioral development based on experience), or developmental selection (the incremental acquisition of adaptive behavior based on experience). If natural selection favors developmental selection, which includes learning from environmental feedback, we can also make predictions about the design of reward systems. Our paper is written in an accessible manner and for a broad audience, though we believe some novel insights can be drawn from our discussion. We hope our paper will help advance the emerging bridge connecting the fields of behavioral ecology and reinforcement learning.

The evolution of cognitive mechanisms in response to cultural innovations

When humans and other animals make cultural innovations, they also change their environment, thereby imposing new selective pressures that can modify their biological traits. For example, there is evidence that dairy farming by humans favored alleles for adult lactose tolerance. Similarly, the invention of cooking possibly affected the evolution of jaw and tooth morphology. However, when it comes to cognitive traits and learning mechanisms, it is much more difficult to determine whether and how their evolution was affected by culture or by their use in cultural transmission. Here we argue that, excluding very recent cultural innovations, the assumption that culture shaped the evolution of cognition is both more parsimonious and more productive than assuming the opposite. In considering how culture shapes cognition, we suggest that a process-level model of cognitive evolution is necessary and offer such a model. The model employs relatively simple coevolving mechanisms of learning and data acquisition that jointly construct a complex network of a type previously shown to be capable of supporting a range of cognitive abilities. The evolution of cognition, and thus the effect of culture on cognitive evolution, is captured through small modifications of these coevolving learning and data-acquisition mechanisms, whose coordinated action is critical for building an effective network. We use the model to show how these mechanisms are likely to evolve in response to cultural phenomena, such as language and tool-making, which are associated with major changes in data patterns and with new computational and statistical challenges.

Time Does Not Help Orangutans Pongo abelii Solve Physical Problems

Many questions in animal intelligence and cognition research are challenging. One challenge is to identify mechanisms underlying reasoning in experiments. Here, we provide a way to design such tests in non-human animals. We know from research in skill acquisition in humans that reasoning and thinking can take time because some problems are processed in multiple steps before a solution is reached (e.g., during mental arithmetics). If animals are able to learn through similar processes their decision making can be time consuming, and most importantly improve if more time to process information is allowed. We tested if performance of two Sumatran orangutans (Pongo abelii) increased in a two-choice experiment when they were allowed extra time before making their decisions, compared to when they were forced to decide immediately. We found that the performance of the orangutans did not depend on the time they were allowed to process the information before making their decisions. This methodology provides a potential avenue for empirical tests of mechanisms underlying reasoning in non-human animals.