Embodied intelligence via learning and evolution

Simultaneous Optimization of Discrete and Continuous Parameters Defining a Robot Morphology and Controller

The morphology and controller design of robots is often a labor-intensive task performed by experienced and intuitive engineers. Automatic robot design using machine learning is attracting increasing attention in the hope that it will reduce the design workload and result in better-performing robots. Most robots are created by joining several rigid parts and then mounting actuators and their controllers. Many studies limit the possible types of rigid parts to a finite set to reduce the computational burden. However, this not only limits the search space, but also prohibits the use of powerful optimization techniques. To find a robot closer to the global optimal design, a method that explores a richer set of robots is desirable. In this article, we propose a novel method to efficiently search for various robot designs. The method combines three different optimization methods with different characteristics. We apply proximal policy optimization (PPO) or soft actor-critic (SAC) as the controller, the REINFORCE algorithm to determine the lengths and other numerical parameters of the rigid parts, and a newly proposed method to determine the number and layout of the rigid parts and joints. Experiments with physical simulations confirm that when this method is used to handle two types of tasks-walking and manipulation-it performs better than simple combinations of existing methods. The source code and videos of our experiments are available online (https://github.com/r-koike/eagent).

Artificial intelligence for geoscience: Progress, challenges, and perspectives

This paper explores the evolution of geoscientific inquiry, tracing the progression from traditional physics-based models to modern data-driven approaches facilitated by significant advancements in artificial intelligence (AI) and data collection techniques. Traditional models, which are grounded in physical and numerical frameworks, provide robust explanations by explicitly reconstructing underlying physical processes. However, their limitations in comprehensively capturing Earth's complexities and uncertainties pose challenges in optimization and real-world applicability. In contrast, contemporary data-driven models, particularly those utilizing machine learning (ML) and deep learning (DL), leverage extensive geoscience data to glean insights without requiring exhaustive theoretical knowledge. ML techniques have shown promise in addressing Earth science-related questions. Nevertheless, challenges such as data scarcity, computational demands, data privacy concerns, and the "black-box" nature of AI models hinder their seamless integration into geoscience. The integration of physics-based and data-driven methodologies into hybrid models presents an alternative paradigm. These models, which incorporate domain knowledge to guide AI methodologies, demonstrate enhanced efficiency and performance with reduced training data requirements. This review provides a comprehensive overview of geoscientific research paradigms, emphasizing untapped opportunities at the intersection of advanced AI techniques and geoscience. It examines major methodologies, showcases advances in large-scale models, and discusses the challenges and prospects that will shape the future landscape of AI in geoscience. The paper outlines a dynamic field ripe with possibilities, poised to unlock new understandings of Earth's complexities and further advance geoscience exploration.

Is artificial consciousness achievable? Lessons from the human brain

We here analyse the question of developing artificial consciousness from an evolutionary perspective, taking the evolution of the human brain and its relation with consciousness as a reference model or as a benchmark. This kind of analysis reveals several structural and functional features of the human brain that appear to be key for reaching human-like complex conscious experience and that current research on Artificial Intelligence (AI) should take into account in its attempt to develop systems capable of human-like conscious processing. We argue that, even if AI is limited in its ability to emulate human consciousness for both intrinsic (i.e., structural and architectural) and extrinsic (i.e., related to the current stage of scientific and technological knowledge) reasons, taking inspiration from those characteristics of the brain that make human-like conscious processing possible and/or modulate it, is a potentially promising strategy towards developing conscious AI. Also, it cannot be theoretically excluded that AI research can develop partial or potentially alternative forms of consciousness that are qualitatively different from the human form, and that may be either more or less sophisticated depending on the perspectives. Therefore, we recommend neuroscience-inspired caution in talking about artificial consciousness: since the use of the same word "consciousness" for humans and AI becomes ambiguous and potentially misleading, we propose to clearly specify which level and/or type of consciousness AI research aims to develop, as well as what would be common versus differ in AI conscious processing compared to human conscious experience.

Integration of cognitive tasks into artificial general intelligence test for large models

During the evolution of large models, performance evaluation is necessary for assessing their capabilities. However, current model evaluations mainly rely on specific tasks and datasets, lacking a united framework for assessing the multidimensional intelligence of large models. In this perspective, we advocate for a comprehensive framework of cognitive science-inspired artificial general intelligence (AGI) tests, including crystallized, fluid, social, and embodied intelligence. The AGI tests consist of well-designed cognitive tests adopted from human intelligence tests, and then naturally encapsulates into an immersive virtual community. We propose increasing the complexity of AGI testing tasks commensurate with advancements in large models and emphasizing the necessity for the interpretation of test results to avoid false negatives and false positives. We believe that cognitive science-inspired AGI tests will effectively guide the targeted improvement of large models in specific dimensions of intelligence and accelerate the integration of large models into human society.

Exploring Embodied Intelligence in Soft Robotics: A Review

Soft robotics is closely related to embodied intelligence in the joint exploration of the means to achieve more natural and effective robotic behaviors via physical forms and intelligent interactions. Embodied intelligence emphasizes that intelligence is affected by the synergy of the brain, body, and environment, focusing on the interaction between agents and the environment. Under this framework, the design and control strategies of soft robotics depend on their physical forms and material properties, as well as algorithms and data processing, which enable them to interact with the environment in a natural and adaptable manner. At present, embodied intelligence has comprehensively integrated related research results on the evolution, learning, perception, decision making in the field of intelligent algorithms, as well as on the behaviors and controls in the field of robotics. From this perspective, the relevant branches of the embodied intelligence in the context of soft robotics were studied, covering the computation of embodied morphology; the evolution of embodied AI; and the perception, control, and decision making of soft robotics. Moreover, on this basis, important research progress was summarized, and related scientific problems were discussed. This study can provide a reference for the research of embodied intelligence in the context of soft robotics.

Body size as a metric for the affordable world

The physical body of an organism serves as a vital interface for interactions with its environment. Here, we investigated the impact of human body size on the perception of action possibilities (affordances) offered by the environment. We found that the body size delineated a distinct boundary on affordances, dividing objects of continuous real-world sizes into two discrete categories with each affording distinct action sets. Additionally, the boundary shifted with imagined body sizes, suggesting a causal link between body size and affordance perception. Intriguingly, ChatGPT, a large language model lacking physical embodiment, exhibited a modest yet comparable affordance boundary at the scale of human body size, suggesting the boundary is not exclusively derived from organism-environment interactions. A subsequent fMRI experiment offered preliminary evidence of affordance processing exclusively for objects within the body size range, but not for those beyond. This suggests that only objects capable of being manipulated are the objects capable of offering affordance in the eyes of an organism. In summary, our study suggests a novel definition of object-ness in an affordance-based context, advocating the concept of embodied cognition in understanding the emergence of intelligence constrained by an organism's physical attributes.

Effects of caudal fin stiffness on optimized forward swimming and turning maneuver in a robotic swimmer

In animal and robot swimmers of body and caudal fin (BCF) form, hydrodynamic thrust is mainly produced by their caudal fins, the stiffness of which has profound effects on both thrust and efficiency of swimming. Caudal fin stiffness also affects the motor control and resulting swimming gaits that correspond to optimal swimming performance; however, their relationship remains scarcely explored. Here using magnetic, modular, undulatory robots (μBots), we tested the effects of caudal fin stiffness on both forward swimming and turning maneuver. We developed six caudal fins with stiffness of more than three orders of difference. For aμBot equipped with each caudal fin (andμBot absent of caudal fin), we applied reinforcement learning in experiments to optimize the motor control for maximizing forward swimming speed or final heading change. The motor control ofμBot was generated by a central pattern generator for forward swimming or by a series of parameterized square waves for turning maneuver. In forward swimming, the variations in caudal fin stiffness gave rise to three modes of optimized motor frequencies and swimming gaits including no caudal fin (4.6 Hz), stiffness <10-4Pa m4(∼10.6 Hz) and stiffness >10-4Pa m4(∼8.4 Hz). Swimming speed, however, varied independently with the modes of swimming gaits, and reached maximal at stiffness of 0.23 × 10-4Pa m4, with theμBot without caudal fin achieving the lowest speed. In turning maneuver, caudal fin stiffness had considerable effects on the amplitudes of both initial head steering and subsequent recoil, as well as the final heading change. It had relatively minor effect on the turning motor program except for theμBots without caudal fin. Optimized forward swimming and turning maneuver shared an identical caudal fin stiffness and similar patterns of peduncle and caudal fin motion, suggesting simplicity in the form and function relationship inμBot swimming.

Enhancing robot evolution through Lamarckian principles

Evolutionary robot systems offer two principal advantages: an advanced way of developing robots through evolutionary optimization and a special research platform to conduct what-if experiments regarding questions about evolution. Our study sits at the intersection of these. We investigate the question "What if the 18th-century biologist Lamarck was not completely wrong and individual traits learned during a lifetime could be passed on to offspring through inheritance?" We research this issue through simulations with an evolutionary robot framework where morphologies (bodies) and controllers (brains) of robots are evolvable and robots also can improve their controllers through learning during their lifetime. Within this framework, we compare a Lamarckian system, where learned bits of the brain are inheritable, with a Darwinian system, where they are not. Analyzing simulations based on these systems, we obtain new insights about Lamarckian evolution dynamics and the interaction between evolution and learning. Specifically, we show that Lamarckism amplifies the emergence of 'morphological intelligence', the ability of a given robot body to acquire a good brain by learning, and identify the source of this success: newborn robots have a higher fitness because their inherited brains match their bodies better than those in a Darwinian system.

Artificial Collective Intelligence Engineering: A Survey of Concepts and Perspectives

Collectiveness is an important property of many systems-both natural and artificial. By exploiting a large number of individuals, it is often possible to produce effects that go far beyond the capabilities of the smartest individuals or even to produce intelligent collective behavior out of not-so-intelligent individuals. Indeed, collective intelligence, namely, the capability of a group to act collectively in a seemingly intelligent way, is increasingly often a design goal of engineered computational systems-motivated by recent technoscientific trends like the Internet of Things, swarm robotics, and crowd computing, to name only a few. For several years, the collective intelligence observed in natural and artificial systems has served as a source of inspiration for engineering ideas, models, and mechanisms. Today, artificial and computational collective intelligence are recognized research topics, spanning various techniques, kinds of target systems, and application domains. However, there is still a lot of fragmentation in the research panorama of the topic within computer science, and the verticality of most communities and contributions makes it difficult to extract the core underlying ideas and frames of reference. The challenge is to identify, place in a common structure, and ultimately connect the different areas and methods addressing intelligent collectives. To address this gap, this article considers a set of broad scoping questions providing a map of collective intelligence research, mostly by the point of view of computer scientists and engineers. Accordingly, it covers preliminary notions, fundamental concepts, and the main research perspectives, identifying opportunities and challenges for researchers on artificial and computational collective intelligence engineering.

Efficient automatic design of robots

Robots are notoriously difficult to design because of complex interdependencies between their physical structure, sensory and motor layouts, and behavior. Despite this, almost every detail of every robot built to date has been manually determined by a human designer after several months or years of iterative ideation, prototyping, and testing. Inspired by evolutionary design in nature, the automated design of robots using evolutionary algorithms has been attempted for two decades, but it too remains inefficient: days of supercomputing are required to design robots in simulation that, when manufactured, exhibit desired behavior. Here we show de novo optimization of a robot's structure to exhibit a desired behavior, within seconds on a single consumer-grade computer, and the manufactured robot's retention of that behavior. Unlike other gradient-based robot design methods, this algorithm does not presuppose any particular anatomical form; starting instead from a randomly-generated apodous body plan, it consistently discovers legged locomotion, the most efficient known form of terrestrial movement. If combined with automated fabrication and scaled up to more challenging tasks, this advance promises near-instantaneous design, manufacture, and deployment of unique and useful machines for medical, environmental, vehicular, and space-based tasks.

Learning to learn by using nonequilibrium training protocols for adaptable materials

Evolution in time-varying environments naturally leads to adaptable biological systems that can easily switch functionalities. Advances in the synthesis of environmentally responsive materials therefore open up the possibility of creating a wide range of synthetic materials which can also be trained for adaptability. We consider high-dimensional inverse problems for materials where any particular functionality can be realized by numerous equivalent choices of design parameters. By periodically switching targets in a given design algorithm, we can teach a material to perform incompatible functionalities with minimal changes in design parameters. We exhibit this learning strategy for adaptability in two simulated settings: elastic networks that are designed to switch deformation modes with minimal bond changes and heteropolymers whose folding pathway selections are controlled by a minimal set of monomer affinities. The resulting designs can reveal physical principles, such as nucleation-controlled folding, that enable such adaptability.

Catalyzing next-generation Artificial Intelligence through NeuroAI

Neuroscience has long been an essential driver of progress in artificial intelligence (AI). We propose that to accelerate progress in AI, we must invest in fundamental research in NeuroAI. A core component of this is the embodied Turing test, which challenges AI animal models to interact with the sensorimotor world at skill levels akin to their living counterparts. The embodied Turing test shifts the focus from those capabilities like game playing and language that are especially well-developed or uniquely human to those capabilities - inherited from over 500 million years of evolution - that are shared with all animals. Building models that can pass the embodied Turing test will provide a roadmap for the next generation of AI.

Understanding Use Intention of mHealth Applications Based on the Unified Theory of Acceptance and Use of Technology 2 (UTAUT-2) Model in China

The COVID-19 pandemic has significantly impacted the healthcare industry, especially public health resources and resource allocation. With the change in people's lifestyles and increased demand for medical and health care in the post-pandemic era, the Internet and home healthcare have rapidly developed. As an essential part of Internet healthcare, mobile health (mHealth) applications help to fundamentally address the lack of medical resources and meet people's healthcare needs. In this mixed-method study, we conducted in-depth interviews with 20 users in China (mean age = 26.13, SD = 2.80, all born in China) during the pandemic, based on the unified theory of acceptance and use of technology 2 (UTAUT-2) mode, and identified four dimensions of user needs in mHealth scenarios: convenience, control, trust, and emotionality. Based on the interview results, we adjusted the independent variables, deleted the hedonic motivation and the habit, and added the perceived trust and perceived risk as the variables. Using a structural equation model (SEM), we designed the questionnaire according to the qualitative results and collected data from 371 participants (above 18 years old, 43.9% male) online to examine the interrelationships these variables. The results show that performance expectancy (β = 0.40, p < 0.001), effort expectancy (β = 0.40, p < 0.001), social influence (β = 0.14, p < 0.05), facilitating condition (β = 0.15, p < 0.001), and perceived trust (β = 0.31, p < 0.001) had positive effects on use intention. Perceived risk (β = -0.31, p < 0.001) harmed use intention, and price value (β = 0.10, p > 0.5) had no significant effects on use intention. Finally, we discussed design and development guidelines that can enhance user experience of mHealth applications. This research combines the actual needs and the main factors affecting the use intention of users, solves the problems of low satisfaction of user experience, and provides better strategic suggestions for developing mHealth applications in the future.

Emergent behaviour and neural dynamics in artificial agents tracking odour plumes

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a complementary in silico approach to develop an integrated understanding of their behaviour and neural computations. Specifically, we train artificial recurrent neural network agents using deep reinforcement learning to locate the source of simulated odour plumes that mimic features of plumes in a turbulent flow. Interestingly, the agents' emergent behaviours resemble those of flying insects, and the recurrent neural networks learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioural hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in odour plume tracking.

Co-optimization of robotic design and skill inspired by human hand evolution

During evolution of the human hand, evolutionary morphology has been closely related to behavior in complicated environments. Numerous researchers have revealed that learned skills have affected hand evolution. Inspired by this phenomenon, a co-optimization approach for underactuated hands is proposed that takes grasping skills and structural parameters into consideration. In our proposal, hand design, especially the underactuated mechanism, can be parameterized and shared with all the local agents. These mechanical parameters can be updated globally by the independent agents. In addition, we also train human-like 'feeling' of grasping: grasping stability is estimated in advance before the object drops, which can speed up grasping training. In this paper, our method is instantiated to address the optimization problem for the torsion spring mechanical parameters of an underactuated robotic hand with multi-actuators, and then the optimized results are transferred to the actual physical robotic hand to test the improvement of grasping. This collaborative evolution process leverages the dexterity of the multi-actuators and the adaptivity of the underactuated mechanism.

Memristive Circuit Implementation of Operant Cascaded With Classical Conditioning

Classical conditioning (CC) and operant conditioning (OC), also known as associative memory, are two of the most fundamental and critical learning mechanisms in the biological brain. However, the existing designs of associative memory memristive circuits mainly focus on CC, and few studies have used memristors to imitate OC at the behavioral level, as well as the OC-CC cascaded associative memories that are widespread in biological learning processes. This work proposes an OC-CC cascaded circuit composed of OC and CC circuits. With the OC memristive circuit, bio-like functions such as random exploration, feedback learning, experience memory, and experience-based decision-making are achieved, which enables the circuit to continuously reshape its own memories and actions to adapt to changing environments. By cascading it with the CC memristive circuit that has the functions of associative learning, forgetting, generalization, and differentiation, the OC-CC cascaded circuit implements richer associative memories and has stronger environmental adaptability. Finally, the proposed circuits can perform on-line in-situ learning and in-memory computing. This is a more brain-like processing method, which is different from the von Neumann architecture. The simulation results of the proposed circuits in PSPICE show that they can simulate the above functions and have advantages in power consumption and hardware overhead. This work provides a possible realization idea for large-scale bionic learning.

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Artificial neural networks (ANNs) have been successfully trained to perform a wide range of sensory-motor behaviors. In contrast, the performance of spiking neuronal network (SNN) models trained to perform similar behaviors remains relatively suboptimal. In this work, we aimed to push the field of SNNs forward by exploring the potential of different learning mechanisms to achieve optimal performance. We trained SNNs to solve the CartPole reinforcement learning (RL) control problem using two learning mechanisms operating at different timescales: (1) spike-timing-dependent reinforcement learning (STDP-RL) and (2) evolutionary strategy (EVOL). Though the role of STDP-RL in biological systems is well established, several other mechanisms, though not fully understood, work in concert during learning in vivo. Recreating accurate models that capture the interaction of STDP-RL with these diverse learning mechanisms is extremely difficult. EVOL is an alternative method and has been successfully used in many studies to fit model neural responsiveness to electrophysiological recordings and, in some cases, for classification problems. One advantage of EVOL is that it may not need to capture all interacting components of synaptic plasticity and thus provides a better alternative to STDP-RL. Here, we compared the performance of each algorithm after training, which revealed EVOL as a powerful method for training SNNs to perform sensory-motor behaviors. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.

The Effects of Learning in Morphologically Evolving Robot Systems

Simultaneously evolving morphologies (bodies) and controllers (brains) of robots can cause a mismatch between the inherited body and brain in the offspring. To mitigate this problem, the addition of an infant learning period has been proposed relatively long ago by the so-called Triangle of Life approach. However, an empirical assessment is still lacking to-date. In this paper, we investigate the effects of such a learning mechanism from different perspectives. Using extensive simulations we show that learning can greatly increase task performance and reduce the number of generations required to reach a certain fitness level compared to the purely evolutionary approach. Furthermore, we demonstrate that the evolved morphologies will be also different, even though learning only directly affects the controllers. This provides a quantitative demonstration that changes in the brain can induce changes in the body. Finally, we examine the learning delta defined as the performance difference between the inherited and the learned brain, and find that it is growing throughout the evolutionary process. This shows that evolution produces robots with an increasing plasticity, that is, consecutive generations become better learners and, consequently, they perform better at the given task. Moreover, our results demonstrate that the Triangle of Life is not only a concept of theoretical interest, but a system methodology with practical benefits.

Evolution of Brains and Computers: The Roads Not Taken

When computers started to become a dominant part of technology around the 1950s, fundamental questions about reliable designs and robustness were of great relevance. Their development gave rise to the exploration of new questions, such as what made brains reliable (since neurons can die) and how computers could get inspiration from neural systems. In parallel, the first artificial neural networks came to life. Since then, the comparative view between brains and computers has been developed in new, sometimes unexpected directions. With the rise of deep learning and the development of connectomics, an evolutionary look at how both hardware and neural complexity have evolved or designed is required. In this paper, we argue that important similarities have resulted both from convergent evolution (the inevitable outcome of architectural constraints) and inspiration of hardware and software principles guided by toy pictures of neurobiology. Moreover, dissimilarities and gaps originate from the lack of major innovations that have paved the way to biological computing (including brains) that are completely absent within the artificial domain. As it occurs within synthetic biocomputation, we can also ask whether alternative minds can emerge from A.I. designs. Here, we take an evolutionary view of the problem and discuss the remarkable convergences between living and artificial designs and what are the pre-conditions to achieve artificial intelligence.

Social Neuro AI: Social Interaction as the “Dark Matter” of AI

This article introduces a three-axis framework indicating how AI can be informed by biological examples of social learning mechanisms. We argue that the complex human cognitive architecture owes a large portion of its expressive power to its ability to engage in social and cultural learning. However, the field of AI has mostly embraced a solipsistic perspective on intelligence. We thus argue that social interactions not only are largely unexplored in this field but also are an essential element of advanced cognitive ability, and therefore constitute metaphorically the “dark matter” of AI. In the first section, we discuss how social learning plays a key role in the development of intelligence. We do so by discussing social and cultural learning theories and empirical findings from social neuroscience. Then, we discuss three lines of research that fall under the umbrella of Social NeuroAI and can contribute to developing socially intelligent embodied agents in complex environments. First, neuroscientific theories of cognitive architecture, such as the global workspace theory and the attention schema theory, can enhance biological plausibility and help us understand how we could bridge individual and social theories of intelligence. Second, intelligence occurs in time as opposed to over time, and this is naturally incorporated by dynamical systems. Third, embodiment has been demonstrated to provide more sophisticated array of communicative signals. To conclude, we discuss the example of active inference, which offers powerful insights for developing agents that possess biological realism, can self-organize in time, and are socially embodied.

Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks

Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are recurrent, dynamic, and non-linear and models of neural circuits employed in neuroscience and neuroethology tend to involve, accordingly, continuous-time, non-linear, and recurrently interconnected components. Currently, the main approach for finding configurations of dynamical recurrent neural networks that demonstrate behaviors of interest is using stochastic search techniques, such as evolutionary algorithms. In an evolutionary algorithm, these dynamic recurrent neural networks are evolved to perform the behavior over multiple generations, through selection, inheritance, and mutation, across a population of solutions. Although, these systems can be evolved to exhibit lifetime learning behavior, there are no explicit rules built into these dynamic recurrent neural networks that facilitate learning during their lifetime (e.g., reward signals). In this work, we examine a biologically plausible lifetime learning mechanism for dynamical recurrent neural networks. We focus on a recently proposed reinforcement learning mechanism inspired by neuromodulatory reward signals and ongoing fluctuations in synaptic strengths. Specifically, we extend one of the best-studied and most-commonly used dynamic recurrent neural networks to incorporate the reinforcement learning mechanism. First, we demonstrate that this extended dynamical system (model and learning mechanism) can autonomously learn to perform a central pattern generation task. Second, we compare the robustness and efficiency of the reinforcement learning rules in relation to two baseline models, a random walk and a hill-climbing walk through parameter space. Third, we systematically study the effect of the different meta-parameters of the learning mechanism on the behavioral learning performance. Finally, we report on preliminary results exploring the generality and scalability of this learning mechanism for dynamical neural networks as well as directions for future work.

Artificial evolution of robot bodies and control: on the interaction between evolution, learning and culture

We survey and reflect on how learning (in the form of individual learning and/or culture) can augment evolutionary approaches to the joint optimization of the body and control of a robot. We focus on a class of applications where the goal is to evolve the body and brain of a single robot to optimize performance on a specified task. The review is grounded in a general framework for evolution which permits the interaction of artificial evolution acting on a population with individual and cultural learning mechanisms. We discuss examples of variations of the general scheme of 'evolution plus learning' from a broad range of robotic systems, and reflect on how the interaction of the two paradigms influences diversity, performance and rate of improvement. Finally, we suggest a number of avenues for future work as a result of the insights that arise from the review. This article is part of a discussion meeting issue 'The emergence of collective knowledge and cumulative culture in animals, humans and machines'.