The neuronal replicator hypothesis

Search and insight processes in card sorting games

Insight problems are particularly interesting, because problems which require restructuring allow researchers to investigate the underpinnings of the Aha-experience, creativity and out of the box thinking. There is a need for new insight tasks to probe and extend the limits of existing theories and cognitive frameworks. To shed more light on this fascinating issue, we addressed the question: Is it possible to convey a well-known card sorting game into an insight task? We introduced different conditions and tested them via two online experiments (N = 546). Between the conditions we systematically varied the available perceptual features, and the existence of non-obvious rules. We found that our card sorting game elicited insight experience. In the first experiment, our data revealed that solution strategies and insight experience varied by the availability and saliency of perceptual features. The discovery of a non-obvious rule, which is not hinted at by perceptual features, was most difficult. With our new paradigm, we were able to construe ambiguous problems which allowed participants to find more than one solution strategy. Interestingly, we realized interindividual preferences for different strategies. The same problem drove strategies which either relied on feature integration or on more deliberate strategies. The second experiment varied the degree of independence of a sorting rule from the standard rules which were in accordance with prior knowledge. It was shown that the more independent the hidden rule was, the more difficult the task became. In sum, we demonstrated a new insight task which extended the available task domains and shed light on sequential and multi-step rule learning problems. Finally, we provided a first sketch of a cognitive model that should help to integrate the data within the existing literature on cognitive models and speculated about the generalizability of the interplay of prior knowledge modification and variation for problem solving.

Novelty and imitation within the brain: a Darwinian neurodynamic approach to combinatorial problems

Efficient search in vast combinatorial spaces, such as those of possible action sequences, linguistic structures, or causal explanations, is an essential component of intelligence. Is there any computational domain that is flexible enough to provide solutions to such diverse problems and can be robustly implemented over neural substrates? Based on previous accounts, we propose that a Darwinian process, operating over sequential cycles of imperfect copying and selection of neural informational patterns, is a promising candidate. Here we implement imperfect information copying through one reservoir computing unit teaching another. Teacher and learner roles are assigned dynamically based on evaluation of the readout signal. We demonstrate that the emerging Darwinian population of readout activity patterns is capable of maintaining and continually improving upon existing solutions over rugged combinatorial reward landscapes. We also demonstrate the existence of a sharp error threshold, a neural noise level beyond which information accumulated by an evolutionary process cannot be maintained. We introduce a novel analysis method, neural phylogenies, that displays the unfolding of the neural-evolutionary process.

Proximodistal exploration in motor learning as an emergent property of optimization

To harness the complexity of their high-dimensional bodies during sensorimotor development, infants are guided by patterns of freezing and freeing of degrees of freedom. For instance, when learning to reach, infants free the degrees of freedom in their arm proximodistally, that is, from joints that are closer to the body to those that are more distant. Here, we formulate and study computationally the hypothesis that such patterns can emerge spontaneously as the result of a family of stochastic optimization processes, without an innate encoding of a maturational schedule. In particular, we present simulated experiments with an arm where a computational learner progressively acquires reaching skills through adaptive exploration, and we show that a proximodistal organization appears spontaneously, which we denote PDFF (Proximo Distal Freezing and Freeing of degrees of freedom). We also compare this emergent organization between different arm morphologies-from human-like to quite unnatural ones-to study the effect of different kinematic structures on the emergence of PDFF.

Expansion and Renormalization of Human Brain Structure During Skill Acquisition

Research on human brain changes during skill acquisition has revealed brain volume expansion in task-relevant areas. However, the large number of skills that humans acquire during ontogeny militates against plasticity as a perpetual process of volume growth. Building on animal models and available theories, we promote the expansion-renormalization model for plastic changes in humans. The model predicts an initial increase of gray matter structure, potentially reflecting growth of neural resources like neurons, synapses, and glial cells, which is followed by a selection process operating on this new tissue leading to a complete or partial return to baseline of the overall volume after selection has ended. The model sheds new light on available evidence and current debates and fosters the search for mechanistic explanations.

The evolutionary dynamics of language

The well-established framework of evolutionary dynamics can be applied to the fascinating open problems how human brains are able to acquire and adapt language and how languages change in a population. Schemas for handling grammatical constructions are the replicating unit. They emerge and multiply with variation in the brains of individuals and undergo selection based on their contribution to needed expressive power, communicative success and the reduction of cognitive effort. Adopting this perspective has two major benefits. (i) It makes a bridge to neurobiological models of the brain that have also adopted an evolutionary dynamics point of view, thus opening a new horizon for studying how human brains achieve the remarkably complex competence for language. And (ii) it suggests a new foundation for studying cultural language change as an evolutionary dynamics process. The paper sketches this novel perspective, provides references to empirical data and computational experiments, and points to open problems.

Insight into the ten-penny problem: guiding search by constraints and maximization

For a long time, insight problem solving has been either understood as nothing special or as a particular class of problem solving. The first view implicates the necessity to find efficient heuristics that restrict the search space, the second, the necessity to overcome self-imposed constraints. Recently, promising hybrid cognitive models attempt to merge both approaches. In this vein, we were interested in the interplay of constraints and heuristic search, when problem solvers were asked to solve a difficult multi-step problem, the ten-penny problem. In three experimental groups and one control group (N = 4 × 30) we aimed at revealing, what constraints drive problem difficulty in this problem, and how relaxing constraints, and providing an efficient search criterion facilitates the solution. We also investigated how the search behavior of successful problem solvers and non-solvers differ. We found that relaxing constraints was necessary but not sufficient to solve the problem. Without efficient heuristics that facilitate the restriction of the search space, and testing the progress of the problem solving process, the relaxation of constraints was not effective. Relaxing constraints and applying the search criterion are both necessary to effectively increase solution rates. We also found that successful solvers showed promising moves earlier and had a higher maximization and variation rate across solution attempts. We propose that this finding sheds light on how different strategies contribute to solving difficult problems. Finally, we speculate about the implications of our findings for insight problem solving.

Cognitive Architecture with Evolutionary Dynamics Solves Insight Problem

In this paper, we show that a neurally implemented a cognitive architecture with evolutionary dynamics can solve the four-tree problem. Our model, called Darwinian Neurodynamics, assumes that the unconscious mechanism of problem solving during insight tasks is a Darwinian process. It is based on the evolution of patterns that represent candidate solutions to a problem, and are stored and reproduced by a population of attractor networks. In our first experiment, we used human data as a benchmark and showed that the model behaves comparably to humans: it shows an improvement in performance if it is pretrained and primed appropriately, just like human participants in Kershaw et al. (2013)'s experiment. In the second experiment, we further investigated the effects of pretraining and priming in a two-by-two design and found a beginner's luck type of effect: solution rate was highest in the condition that was primed, but not pretrained with patterns relevant for the task. In the third experiment, we showed that deficits in computational capacity and learning abilities decreased the performance of the model, as expected. We conclude that Darwinian Neurodynamics is a promising model of human problem solving that deserves further investigation.

Can Contraries Prompt Intuition in Insight Problem Solving?

This paper aims to test whether the use of contraries can facilitate spatial problem solving. Specifically, we examined whether a training session which included explicit guidance on thinking in contraries would improve problem solving abilities. In our study, the participants in the experimental condition were exposed to a brief training session before being presented with seven visuo-spatial problems to solve. During training it was suggested that it would help them to find the solution to the problems if they systematically transformed the spatial features of each problem into their contraries. Their performance was compared to that of a control group (who had no training). Two participation conditions were considered: small groups and individuals. Higher success rates were found in the groups exposed to training as compared to the individuals (in both the training and no training conditions), even though the time required to find a solution was longer. In general, participants made more attempts (i.e., drawings) when participating in groups than individually. The number of drawings done while the participants were trying to solve the problems did not increase after training. In order to explore if the quality (if not the number) of drawings was modified, we sampled one problem out of the seven we had used in the experiment (the “pigs in a pen” problem) and examined the drawings in detail. Differences between the training and no training conditions emerged in terms of properties focused on and transformed in the drawings. Based on these results, in the final discussion possible explanations are suggested as to why training had positive effects specifically in the group condition.

The Transition to Minimal Consciousness through the Evolution of Associative Learning

The minimal state of consciousness is sentience. This includes any phenomenal sensory experience - exteroceptive, such as vision and olfaction; interoceptive, such as pain and hunger; or proprioceptive, such as the sense of bodily position and movement. We propose unlimited associative learning (UAL) as the marker of the evolutionary transition to minimal consciousness (or sentience), its phylogenetically earliest sustainable manifestation and the driver of its evolution. We define and describe UAL at the behavioral and functional level and argue that the structural-anatomical implementations of this mode of learning in different taxa entail subjective feelings (sentience). We end with a discussion of the implications of our proposal for the distribution of consciousness in the animal kingdom, suggesting testable predictions, and revisiting the ongoing debate about the function of minimal consciousness in light of our approach.

Breeding novel solutions in the brain: a model of Darwinian neurodynamics

Background: The fact that surplus connections and neurons are pruned during development is well established. We complement this selectionist picture by a proof-of-principle model of evolutionary search in the brain, that accounts for new variations in theory space. We present a model for Darwinian evolutionary search for candidate solutions in the brain.

Methods: We combine known components of the brain – recurrent neural networks (acting as attractors), the action selection loop and implicit working memory – to provide the appropriate Darwinian architecture. We employ a population of attractor networks with palimpsest memory. The action selection loop is employed with winners-share-all dynamics to select for candidate solutions that are transiently stored in implicit working memory.

Results: We document two processes: selection of stored solutions and evolutionary search for novel solutions. During the replication of candidate solutions attractor networks occasionally produce recombinant patterns, increasing variation on which selection can act. Combinatorial search acts on multiplying units (activity patterns) with hereditary variation and novel variants appear due to (i) noisy recall of patterns from the attractor networks, (ii) noise during transmission of candidate solutions as messages between networks, and, (iii) spontaneously generated, untrained patterns in spurious attractors.

Conclusions: Attractor dynamics of recurrent neural networks can be used to model Darwinian search. The proposed architecture can be used for fast search among stored solutions (by selection) and for evolutionary search when novel candidate solutions are generated in successive iterations. Since all the suggested components are present in advanced nervous systems, we hypothesize that the brain could implement a truly evolutionary combinatorial search system, capable of generating novel variants.

Breeding novel solutions in the brain: a model of Darwinian neurodynamics

Background: The fact that surplus connections and neurons are pruned during development is well established. We complement this selectionist picture by a proof-of-principle model of evolutionary search in the brain, that accounts for new variations in theory space. We present a model for Darwinian evolutionary search for candidate solutions in the brain. Methods: We combine known components of the brain - recurrent neural networks (acting as attractors), the action selection loop and implicit working memory - to provide the appropriate Darwinian architecture. We employ a population of attractor networks with palimpsest memory. The action selection loop is employed with winners-share-all dynamics to select for candidate solutions that are transiently stored in implicit working memory. Results: We document two processes: selection of stored solutions and evolutionary search for novel solutions. During the replication of candidate solutions attractor networks occasionally produce recombinant patterns, increasing variation on which selection can act. Combinatorial search acts on multiplying units (activity patterns) with hereditary variation and novel variants appear due to (i) noisy recall of patterns from the attractor networks, (ii) noise during transmission of candidate solutions as messages between networks, and, (iii) spontaneously generated, untrained patterns in spurious attractors. Conclusions: Attractor dynamics of recurrent neural networks can be used to model Darwinian search. The proposed architecture can be used for fast search among stored solutions (by selection) and for evolutionary search when novel candidate solutions are generated in successive iterations. Since all the suggested components are present in advanced nervous systems, we hypothesize that the brain could implement a truly evolutionary combinatorial search system, capable of generating novel variants.

Neuronal boost to evolutionary dynamics

Standard evolutionary dynamics is limited by the constraints of the genetic system. A central message of evolutionary neurodynamics is that evolutionary dynamics in the brain can happen in a neuronal niche in real time, despite the fact that neurons do not reproduce. We show that Hebbian learning and structural synaptic plasticity broaden the capacity for informational replication and guided variability provided a neuronally plausible mechanism of replication is in place. The synergy between learning and selection is more efficient than the equivalent search by mutation selection. We also consider asymmetric landscapes and show that the learning weights become correlated with the fitness gradient. That is, the neuronal complexes learn the local properties of the fitness landscape, resulting in the generation of variability directed towards the direction of fitness increase, as if mutations in a genetic pool were drawn such that they would increase reproductive success. Evolution might thus be more efficient within evolved brains than among organisms out in the wild.

Problem solving stages in the five square problem

According to the restructuring hypothesis, insight problem solving typically progresses through consecutive stages of search, impasse, insight, and search again for someone, who solves the task. The order of these stages was determined through self-reports of problem solvers and has never been verified behaviorally. We asked whether individual analysis of problem solving attempts of participants revealed the same order of problem solving stages as defined by the theory and whether their subjective feelings corresponded to the problem solving stages they were in. Our participants tried to solve the Five-Square problem in an online task, while we recorded the time and trajectory of their stick movements. After the task they were asked about their feelings related to insight and some of them also had the possibility of reporting impasse while working on the task. We found that the majority of participants did not follow the classic four-stage model of insight, but had more complex sequences of problem solving stages, with search and impasse recurring several times. This means that the classic four-stage model is not sufficient to describe variability on the individual level. We revised the classic model and we provide a new model that can generate all sequences found. Solvers reported insight more often than non-solvers and non-solvers reported impasse more often than solvers, as expected; but participants did not report impasse more often during behaviorally defined impasse stages than during other stages. This shows that impasse reports might be unreliable indicators of impasse. Our study highlights the importance of individual analysis of problem solving behavior to verify insight theory.

Input statistics and Hebbian cross-talk effects

As an extension of prior work, we studied inspecific Hebbian learning using the classical Oja model. We used a combination of analytical tools and numerical simulations to investigate how the effects of synaptic cross talk (which we also refer to as synaptic inspecificity) depend on the input statistics. We investigated a variety of patterns that appear in dimensions higher than two (and classified them based on covariance type and input bias). We found that the effects of cross talk on learning dynamics and outcome is highly dependent on the input statistics and that cross talk may lead in some cases to catastrophic effects on learning or development. Arbitrarily small levels of cross talk are able to trigger bifurcations in learning dynamics, or bring the system in close enough proximity to a critical state, to make the effects indistinguishable from a real bifurcation. We also investigated how cross talk behaves toward unbiased ("competitive") inputs and in which circumstances it can help the system productively resolve the competition. Finally, we discuss the idea that sophisticated neocortical learning requires accurate synaptic updates (similar to polynucleotide copying, which requires highly accurate replication). Since it is unlikely that the brain can completely eliminate cross talk, we support the proposal that is uses a neural mechanism that "proofreads" the accuracy of the updates, much as DNA proofreading lowers copying error rate.

Hebbian crosstalk and input segregation

Hebbian synapses respond to input/output correlations, and thus to input statistical structure. However, recent evidence suggests that strength adjustments are not completely connection-specific, and this "crosstalk" could distort, or even prevent, learning processes. Crosstalk would then be a form of adjustment mistake, analogous to mistakes in polynucleotide copying. The mutation rate must be extremely low for successful evolution (which is a type of learning process), and similarly neural learning might require minimal crosstalk. We analyze aspects of the effect of crosstalk in Hebbian learning from pairwise input correlations, using the classical Oja model. In previous work we showed that crosstalk leads to learning of the principal eigenvector of EC (the input covariance matrix pre-multiplied by an error matrix that describes the crosstalk pattern), and found that, with positive input correlations, increasing crosstalk smoothly degrades performance. However, the Oja model requires negative input correlations to account for biological ocular segregation. Although this assumption is biologically somewhat implausible, it captures features that are seen in more complex models. Here, we analyze how crosstalk would affect such segregation. We show that, for statistically unbiased inputs, crosstalk induces a bifurcation from segregating to non-segregating outcomes at a critical value which depends on correlations. We also investigate the behavior in the vicinity of this critical state and for weakly biased inputs. Our results show that crosstalk can induce a bifurcation under special conditions even in the simplest Hebbian models, and that even the low levels of crosstalk observed in the brain could prevent normal development. However, during learning pairwise input statistics are more complex, and crosstalk-induced bifurcations may not occur in the Oja model. Such bifurcations would be analogous to "error catastrophes" in genetic models, and we argue that they are usually absent for simple linear Hebbian learning because such learning is only driven by pairwise correlations.

Using reinforcement learning to examine dynamic attention allocation during reading

A fundamental question in reading research concerns whether attention is allocated strictly serially, supporting lexical processing of one word at a time, or in parallel, supporting concurrent lexical processing of two or more words (Reichle, Liversedge, Pollatsek, & Rayner, 2009). The origins of this debate are reviewed. We then report three simulations to address this question using artificial reading agents (Liu & Reichle, 2010; Reichle & Laurent, 2006) that learn to dynamically allocate attention to 1-4 words to "read" as efficiently as possible. These simulation results indicate that the agents strongly preferred serial word processing, although they occasionally attended to more than one word concurrently. The reason for this preference is discussed, along with implications for the debate about how humans allocate attention during reading.

Harnessing plasticity to understand learning and treat disease

A large body of evidence suggests that neural plasticity contributes to learning and disease. Recent studies suggest that cortical map plasticity is typically a transient phase that improves learning by increasing the pool of task-relevant responses. Here, I discuss a new perspective on neural plasticity and suggest how plasticity might be targeted to reset dysfunctional circuits. Specifically, a new model is proposed in which map expansion provides a form of replication with variation that supports a Darwinian mechanism to select the most behaviorally useful circuits. Precisely targeted neural plasticity provides a new avenue for the treatment of neurological and psychiatric disorders and is a powerful tool to test the neural mechanisms of learning and memory.

Selectionist and evolutionary approaches to brain function: a critical appraisal

We consider approaches to brain dynamics and function that have been claimed to be Darwinian. These include Edelman’s theory of neuronal group selection, Changeux’s theory of synaptic selection and selective stabilization of pre-representations, Seung’s Darwinian synapse, Loewenstein’s synaptic melioration, Adam’s selfish synapse, and Calvin’s replicating activity patterns. Except for the last two, the proposed mechanisms are selectionist but not truly Darwinian, because no replicators with information transfer to copies and hereditary variation can be identified in them. All of them fit, however, a generalized selectionist framework conforming to the picture of Price’s covariance formulation, which deliberately was not specific even to selection in biology, and therefore does not imply an algorithmic picture of biological evolution. Bayesian models and reinforcement learning are formally in agreement with selection dynamics. A classification of search algorithms is shown to include Darwinian replicators (evolutionary units with multiplication, heredity, and variability) as the most powerful mechanism for search in a sparsely occupied search space. Examples are given of cases where parallel competitive search with information transfer among the units is more efficient than search without information transfer between units. Finally, we review our recent attempts to construct and analyze simple models of true Darwinian evolutionary units in the brain in terms of connectivity and activity copying of neuronal groups. Although none of the proposed neuronal replicators include miraculous mechanisms, their identification remains a challenge but also a great promise.

Evolvable neuronal paths: a novel basis for information and search in the brain

We propose a previously unrecognized kind of informational entity in the brain that is capable of acting as the basis for unlimited hereditary variation in neuronal networks. This unit is a path of activity through a network of neurons, analogous to a path taken through a hidden Markov model. To prove in principle the capabilities of this new kind of informational substrate, we show how a population of paths can be used as the hereditary material for a neuronally implemented genetic algorithm, (the swiss-army knife of black-box optimization techniques) which we have proposed elsewhere could operate at somatic timescales in the brain. We compare this to the same genetic algorithm that uses a standard 'genetic' informational substrate, i.e. non-overlapping discrete genotypes, on a range of optimization problems. A path evolution algorithm (PEA) is defined as any algorithm that implements natural selection of paths in a network substrate. A PEA is a previously unrecognized type of natural selection that is well suited for implementation by biological neuronal networks with structural plasticity. The important similarities and differences between a standard genetic algorithm and a PEA are considered. Whilst most experiments are conducted on an abstract network model, at the conclusion of the paper a slightly more realistic neuronal implementation of a PEA is outlined based on Izhikevich spiking neurons. Finally, experimental predictions are made for the identification of such informational paths in the brain.

Synchronization and redundancy: implications for robustness of neural learning and decision making

Learning and decision making in the brain are key processes critical to survival, and yet are processes implemented by nonideal biological building blocks that can impose significant error. We explore quantitatively how the brain might cope with this inherent source of error by taking advantage of two ubiquitous mechanisms, redundancy and synchronization. In particular we consider a neural process whose goal is to learn a decision function by implementing a nonlinear gradient dynamics. The dynamics, however, are assumed to be corrupted by perturbations modeling the error, which might be incurred due to limitations of the biology, intrinsic neuronal noise, and imperfect measurements. We show that error, and the associated uncertainty surrounding a learned solution, can be controlled in large part by trading off synchronization strength among multiple redundant neural systems against the noise amplitude. The impact of the coupling between such redundant systems is quantified by the spectrum of the network Laplacian, and we discuss the role of network topology in synchronization and in reducing the effect of noise. We discuss range of situations in which the mechanisms we model arise in brain science and draw attention to experimental evidence suggesting that cortical circuits capable of implementing the computations of interest here can be found on several scales. Finally, simulations comparing theoretical bounds to the relevant empirical quantities show that the theoretical estimates we derive can be tight.

Symbol manipulation and rule learning in spiking neuronal networks

It has been claimed that the productivity, systematicity and compositionality of human language and thought necessitate the existence of a physical symbol system (PSS) in the brain. Recent discoveries about temporal coding suggest a novel type of neuronal implementation of a physical symbol system. Furthermore, learning classifier systems provide a plausible algorithmic basis by which symbol re-write rules could be trained to undertake behaviors exhibiting systematicity and compositionality, using a kind of natural selection of re-write rules in the brain, We show how the core operation of a learning classifier system, namely, the replication with variation of symbol re-write rules, can be implemented using spike-time dependent plasticity based supervised learning. As a whole, the aim of this paper is to integrate an algorithmic and an implementation level description of a neuronal symbol system capable of sustaining systematic and compositional behaviors. Previously proposed neuronal implementations of symbolic representations are compared with this new proposal.