Exploration versus exploitation in space, mind, and society

Rationality, perception, and the all-seeing eye

Seeing-perception and vision-is implicitly the fundamental building block of the literature on rationality and cognition. Herbert Simon and Daniel Kahneman's arguments against the omniscience of economic agents-and the concept of bounded rationality-depend critically on a particular view of the nature of perception and vision. We propose that this framework of rationality merely replaces economic omniscience with perceptual omniscience. We show how the cognitive and social sciences feature a pervasive but problematic meta-assumption that is characterized by an "all-seeing eye." We raise concerns about this assumption and discuss different ways in which the all-seeing eye manifests itself in existing research on (bounded) rationality. We first consider the centrality of vision and perception in Simon's pioneering work. We then point to Kahneman's work-particularly his article "Maps of Bounded Rationality"-to illustrate the pervasiveness of an all-seeing view of perception, as manifested in the extensive use of visual examples and illusions. Similar assumptions about perception can be found across a large literature in the cognitive sciences. The central problem is the present emphasis on inverse optics-the objective nature of objects and environments, e.g., size, contrast, and color. This framework ignores the nature of the organism and perceiver. We argue instead that reality is constructed and expressed, and we discuss the species-specificity of perception, as well as perception as a user interface. We draw on vision science as well as the arts to develop an alternative understanding of rationality in the cognitive and social sciences. We conclude with a discussion of the implications of our arguments for the rationality and decision-making literature in cognitive psychology and behavioral economics, along with suggesting some ways forward.

How ecology shapes exploitation: a framework to predict the behavioural response of human and animal foragers along exploration-exploitation trade-offs

Understanding how humans and other animals behave in response to changes in their environments is vital for predicting population dynamics and the trajectory of coupled social-ecological systems. Here, we present a novel framework for identifying emergent social behaviours in foragers (including humans engaged in fishing or hunting) in predator-prey contexts based on the exploration difficulty and exploitation potential of a renewable natural resource. A qualitative framework is introduced that predicts when foragers should behave territorially, search collectively, act independently or switch among these states. To validate it, we derived quantitative predictions from two models of different structure: a generic mathematical model, and a lattice-based evolutionary model emphasising exploitation and exclusion costs. These models independently identified that the exploration difficulty and exploitation potential of the natural resource controls the social behaviour of resource exploiters. Our theoretical predictions were finally compared to a diverse set of empirical cases focusing on fisheries and aquatic organisms across a range of taxa, substantiating the framework's predictions. Understanding social behaviour for given social-ecological characteristics has important implications, particularly for the design of governance structures and regulations to move exploited systems, such as fisheries, towards sustainability. Our framework provides concrete steps in this direction.

Reaching Into the Unknown: Actions, Goal Hierarchies, and Explorative Agency

Action is widely characterized as possessing a teleological dimension. The dominant way of describing goal-directed action and agency is in terms of exploitation, i.e., pursuing pre-specified goals using existing strategies. Recent theoretical developments emphasize the place of exploration, i.e., discovering new goals or acquiring new strategies. The exploitation-exploration distinction poses questions with regard to goals and agency: Should exploration, as some authors have suggested, be regarded as acting without a goal? We argue that recognizing the hierarchical nature of goals is crucial in distinguishing the two kinds of activity, because this recognition prevents the claim that exploration is goal-free, while allowing for a homogeneous account of both exploitative and explorative actions. An action typically causes relatively low-level/proximal (i.e., sensorimotor, immediate) and relatively high-level/distal (i.e., in the environment, at a wider timescale) outcomes. In exploitation, one relies on existing associations between low- and high-level states, whereas in exploration one does not have the ability or intention to control high-level/distal states. We argue that explorative action entails the capacity to exercise control within the low-level/proximal states, which enables the pursuit of indeterminate goals at the higher levels of a goal hierarchy, and the possibility of acquiring new goals and reorganization of goal hierarchies. We consider how the dominant models of agency might accommodate this capacity for explorative action.

Dopamine and Proximity in Motivation and Cognitive Control

Cognitive control - the ability to override a salient or prepotent action to execute a more deliberate one - is required for flexible, goal-directed behavior, and yet it is subjectively costly: decision-makers avoid allocating control resources, even when doing so affords more valuable outcomes. Dopamine likely offsets effort costs just as it does for physical effort. And yet, dopamine can also promote impulsive action, undermining control. We propose a novel hypothesis that reconciles opposing effects of dopamine on cognitive control: during action selection, striatal dopamine biases benefits relative to costs, but does so preferentially for "proximal" motor and cognitive actions. Considering the nature of instrumental affordances and their dynamics during action selection facilitates a parsimonious interpretation and conserved corticostriatal mechanisms across physical and cognitive domains.

Neural field model of memory-guided search

Many organisms can remember locations they have previously visited during a search. Visual search experiments have shown exploration is guided away from these locations, reducing redundancies in the search path before finding a hidden target. We develop and analyze a two-layer neural field model that encodes positional information during a search task. A position-encoding layer sustains a bump attractor corresponding to the searching agent's current location, and search is modeled by velocity input that propagates the bump. A memory layer sustains persistent activity bounded by a wave front, whose edges expand in response to excitatory input from the position layer. Search can then be biased in response to remembered locations, influencing velocity inputs to the position layer. Asymptotic techniques are used to reduce the dynamics of our model to a low-dimensional system of equations that track the bump position and front boundary. Performance is compared for different target-finding tasks.

Exploration and recency as the main proximate causes of probability matching: a reinforcement learning analysis

Research has not yet reached a consensus on why humans match probabilities instead of maximise in a probability learning task. The most influential explanation is that they search for patterns in the random sequence of outcomes. Other explanations, such as expectation matching, are plausible, but do not consider how reinforcement learning shapes people's choices. We aimed to quantify how human performance in a probability learning task is affected by pattern search and reinforcement learning. We collected behavioural data from 84 young adult participants who performed a probability learning task wherein the majority outcome was rewarded with 0.7 probability, and analysed the data using a reinforcement learning model that searches for patterns. Model simulations indicated that pattern search, exploration, recency (discounting early experiences), and forgetting may impair performance. Our analysis estimated that 85% (95% HDI [76, 94]) of participants searched for patterns and believed that each trial outcome depended on one or two previous ones. The estimated impact of pattern search on performance was, however, only 6%, while those of exploration and recency were 19% and 13% respectively. This suggests that probability matching is caused by uncertainty about how outcomes are generated, which leads to pattern search, exploration, and recency.

Choosing attentional control settings in a dynamically changing environment

Goal-directed attentional control supports efficient visual search by prioritizing relevant stimuli in the environment. Previous research has shown that goal-directed control can be configured in many ways, and often multiple control settings can be used to achieve the same goal. However, little is known about how control settings are selected. We explored the extent to which the configuration of goal-directed control is driven by performance maximization (optimally configuring settings to maximize speed and accuracy) and effort minimization (selecting the least effortful settings). We used a new paradigm, adaptive choice visual search, which allows participants to choose one of two available targets (a red or a blue square) on each trial. Distractor colors vary predictively across trials, such that the optimal target switches back and forth throughout the experiment. Results (N = 43) show that participants chose the optimal target most often, updating to the new target when the environment changed, supporting performance maximization. However, individuals were sluggish to update to the optimal color, consistent with effort minimization. Additionally, we found a surprisingly high rate of nonoptimal choices and switching between targets, which could not be explained by either factor. Analysis of participants' self-reported search strategy revealed substantial individual differences in the control strategies used. In sum, the adaptive choice visual search enables a fresh approach to studying goal-directed control. The results contribute new evidence that control is partly determined by both performance maximization and effort minimization, as well as at least one additional factor, which we speculate to include novelty seeking.

A causal role for right frontopolar cortex in directed, but not random, exploration

The explore-exploit dilemma occurs anytime we must choose between exploring unknown options for information and exploiting known resources for reward. Previous work suggests that people use two different strategies to solve the explore-exploit dilemma: directed exploration, driven by information seeking, and random exploration, driven by decision noise. Here, we show that these two strategies rely on different neural systems. Using transcranial magnetic stimulation to inhibit the right frontopolar cortex, we were able to selectively inhibit directed exploration while leaving random exploration intact. This suggests a causal role for right frontopolar cortex in directed, but not random, exploration and that directed and random exploration rely on (at least partially) dissociable neural systems.

Energetic cost determines voluntary movement speed only in familiar environments

Locomotor performance is closely related to fitness. However, in many ecological contexts, animals do not move at their maximal locomotor capacity, but adopt a voluntary speed that is lower than maximal. It is important to understand the mechanisms that underlie voluntary speed, because these determine movement patterns of animals across natural environments. We show that voluntary speed is a stable trait in zebrafish (Danio rerio), but there were pronounced differences between individuals in maximal sustained speed, voluntary speed and metabolic cost of locomotion. We accept the hypothesis that voluntary speed scales positively with maximal sustained swimming performance (Ucrit), but only in unfamiliar environments (1st minute in an open-field arena versus 10th minute) at high temperature (30°C). There was no significant effect of metabolic scope on Ucrit Contrary to expectation, we rejected the hypothesis that voluntary speed decreases with increasing metabolic cost of movement, except in familiar spatial (after 10 min of exploration) and thermal (24°C but not 18 or 30°C) environments. The implications of these data are that the energetic costs of exploration and dispersal in novel environments are higher than those for movement within familiar home ranges.

Tonic noradrenergic activity modulates explorative behavior and attentional set shifting: Evidence from pupillometry and gaze pattern analysis

A constant task for every living organism is to decide whether to exploit rewards associated with current behavior or to explore the environment for more rewarding options. Current empirical evidence indicates that exploitation is related to phasic whereas exploration is related to tonic firing mode of noradrenergic neurons in the locus coeruleus. In humans, this exploration-exploitation trade-off is subserved by the ability to flexibly switch attention between task-related and task-irrelevant information. Here, we investigated whether this function, called attentional set shifting, is related to exploration and tonic noradrenergic discharge. We measured pretrial baseline pupil dilation, proved to be strongly correlated with the activity of the locus coeruleus, while human participants took part in well-known tasks of attentional set shifting. Study 1 used the Wisconsin Card Sorting Task, whereas in Study 2, the Intra/Extradimensional Set Shifting Task was used. Both tasks require participants to choose between different compound stimuli based on feedback provided for their previous decisions. During the task, stimulus-reward contingencies change periodically, thus participants are repeatedly required to reassess which stimulus features are relevant (i.e., they shift their attentional set). Our results showed that baseline pupil diameter steadily decreased when the stimulus-reward contingencies were stable, whereas they suddenly increased when these contingencies changed. Analysis of looking patterns also confirmed the presence of exploratory behavior during attentional set shifting. Thus, our results suggest that tonic firing mode of noradrenergic neurons in the locus coeruleus is implicated in attentional set shifting, as it regulates the amount of exploration.

Cohesive network reconfiguration accompanies extended training

Human behavior is supported by flexible neurophysiological processes that enable the fine‐scale manipulation of information across distributed neural circuits. Yet, approaches for understanding the dynamics of these circuit interactions have been limited. One promising avenue for quantifying and describing these dynamics lies in multilayer network models. Here, networks are composed of nodes (which represent brain regions) and time‐dependent edges (which represent statistical similarities in activity time series). We use this approach to examine functional connectivity measured by non‐invasive neuroimaging techniques. These multilayer network models facilitate the examination of changes in the pattern of statistical interactions between large‐scale brain regions that might facilitate behavior. In this study, we define and exercise two novel measures of network reconfiguration, and demonstrate their utility in neuroimaging data acquired as healthy adult human subjects learn a new motor skill. In particular, we identify putative functional modules in multilayer networks and characterize the degree to which nodes switch between modules. Next, we define cohesive switches, in which a set of nodes moves between modules together as a group, and we define disjoint switches, in which a single node moves between modules independently from other nodes. Together, these two concepts offer complementary yet distinct insights into the changes in functional connectivity that accompany motor learning. More generally, our work offers statistical tools that other researchers can use to better understand the reconfiguration patterns of functional connectivity over time. Hum Brain Mapp 38:4744–4759, 2017. © 2017 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc.

Exploration and Exploitation in Natural Viewing Behavior

Many eye-tracking studies investigate visual behavior with a focus on image features and the semantic content of a scene. A wealth of results on these aspects is available, and our understanding of the decision process where to look has reached a mature stage. However, the temporal aspect, whether to stay and further scrutinize a region (exploitation) or to move on and explore image regions that were yet not in the focus of attention (exploration) is less well understood. Here, we investigate the trade-off between these two processes across stimuli with varying properties and sizes. In a free viewing task, we examined gaze parameters in humans, involving the central tendency, entropy, saccadic amplitudes, number of fixations and duration of fixations. The results revealed that the central tendency and entropy scaled with stimulus size. The mean saccadic amplitudes showed a linear increase that originated from an interaction between the distribution of saccades and the spatial bias. Further, larger images led to spatially more extensive sampling as indicated by a higher number of fixations at the expense of reduced fixation durations. These results demonstrate a profound shift from exploitation to exploration as an adaptation of main gaze parameters with increasing image size.

The nearly universal link between the age of past knowledge and tomorrow's breakthroughs in science and technology: The hotspot

Scientists and inventors can draw on an ever-expanding literature for the building blocks of tomorrow's ideas, yet little is known about how combinations of past work are related to future discoveries. Our analysis parameterizes the age distribution of a work's references and revealed three links between the age of prior knowledge and hit papers and patents. First, works that cite literature with a low mean age and high age variance are in a citation "hotspot"; these works double their likelihood of being in the top 5% or better of citations. Second, the hotspot is nearly universal in all branches of science and technology and is increasingly predictive of a work's future citation impact. Third, a scientist or inventor is significantly more likely to write a paper in the hotspot when they are coauthoring than whey they are working alone. Our findings are based on all 28,426,345 scientific papers in the Web of Science, 1945-2013, and all 5,382,833 U.S. patents, 1950-2010, and reveal new antecedents of high-impact science and the link between prior literature and tomorrow's breakthrough ideas.

Precuneus proportions and cortical folding: A morphometric evaluation on a racially diverse human sample

Recent analyses have suggested that the size and proportions of the precuneus are remarkably variable among adult humans, representing a major source of geometrical difference in midsagittal brain morphology. The same area also represents the main midsagittal brain difference between humans and chimpanzees, being more expanded in our species. Enlargement of the upper parietal surface is a specific feature of Homo sapiens, when compared with other fossil hominids, suggesting the involvement of these cortical areas in recent modern human evolution. Here, we provide a survey on midsagittal brain morphology by investigating whether precuneus size represents the largest component of variance within a larger and racially diverse sample of 265 adult humans. Additionally, we investigate the relationship between precuneus shape variation and folding patterns. Precuneus proportions are confirmed to be a major source of human brain variation even when racial variability is considered. Larger precuneus size is associated with additional precuneal gyri, generally in its anterior district. Spatial variation is most pronounced in the dorsal areas, with no apparent differences between hemispheres, between sexes, or among different racial groups. These dorsal areas integrate somatic and visual information together with the lateral elements of the parietal cortex, representing a crucial node for self-centered mental imagery. The histological basis and functional significance of this intra-specific variation in the upper precuneus remains to be evaluated.

Habit formation generates secondary modules that emulate the efficiency of evolved behavior

We discuss the evolutionary implications of connections drawn between the authors' learned “secondary modules” and the habit-formation system that appears to be ubiquitous among vertebrates. Prior to any subsequent coevolution with social learning, we suggest that aspects of general intelligence likely arose in tandem with mechanisms of adaptive motor control that rely on basal ganglia circuitry.

Exploration and exploitation of Victorian science in Darwin's reading notebooks

Search in an environment with an uncertain distribution of resources involves a trade-off between exploitation of past discoveries and further exploration. This extends to information foraging, where a knowledge-seeker shifts between reading in depth and studying new domains. To study this decision-making process, we examine the reading choices made by one of the most celebrated scientists of the modern era: Charles Darwin. From the full-text of books listed in his chronologically-organized reading journals, we generate topic models to quantify his local (text-to-text) and global (text-to-past) reading decisions using Kullback-Liebler Divergence, a cognitively-validated, information-theoretic measure of relative surprise. Rather than a pattern of surprise-minimization, corresponding to a pure exploitation strategy, Darwin's behavior shifts from early exploitation to later exploration, seeking unusually high levels of cognitive surprise relative to previous eras. These shifts, detected by an unsupervised Bayesian model, correlate with major intellectual epochs of his career as identified both by qualitative scholarship and Darwin's own self-commentary. Our methods allow us to compare his consumption of texts with their publication order. We find Darwin's consumption more exploratory than the culture's production, suggesting that underneath gradual societal changes are the explorations of individual synthesis and discovery. Our quantitative methods advance the study of cognitive search through a framework for testing interactions between individual and collective behavior and between short- and long-term consumption choices. This novel application of topic modeling to characterize individual reading complements widespread studies of collective scientific behavior.

Variability in individual activity bursts improves ant foraging success

Using experimental and computational methods, we study the role of behavioural variability in activity bursts (or temporal activity patterns) for individual and collective regulation of foraging in A. senilis ants. First, foraging experiments were carried out under special conditions (low densities of ants and food and absence of external cues or stimuli) where individual-based strategies are most prevalent. By using marked individuals and recording all foraging trajectories, we were then able to precisely quantify behavioural variability among individuals. Our main conclusions are that (i) variability of ant trajectories (turning angles, speed, etc.) is low compared with variability of temporal activity profiles, and (ii) this variability seems to be driven by plasticity of individual behaviour through time, rather than the presence of fixed behavioural stereotypes or specialists within the group. The statistical measures obtained from these experimental foraging patterns are then used to build a general agent-based model (ABM) which includes the most relevant properties of ant foraging under natural conditions, including recruitment through pheromone communication. Using the ABM, we are able to provide computational evidence that the characteristics of individual variability observed in our experiments can provide a functional advantage (in terms of foraging success) to the group; thus, we propose the biological basis underpinning our observations. Altogether, our study reveals the potential utility of experiments under simplified (laboratory) conditions for understanding information-gathering in biological systems.

Risk and Ambiguity in Information Seeking: Eye Gaze Patterns Reveal Contextual Behavior in Dealing with Uncertainty

Information foraging connects optimal foraging theory in ecology with how humans search for information. The theory suggests that, following an information scent, the information seeker must optimize the tradeoff between exploration by repeated steps in the search space vs. exploitation, using the resources encountered. We conjecture that this tradeoff characterizes how a user deals with uncertainty and its two aspects, risk and ambiguity in economic theory. Risk is related to the perceived quality of the actually visited patch of information, and can be reduced by exploiting and understanding the patch to a better extent. Ambiguity, on the other hand, is the opportunity cost of having higher quality patches elsewhere in the search space. The aforementioned tradeoff depends on many attributes, including traits of the user: at the two extreme ends of the spectrum, analytic and wholistic searchers employ entirely different strategies. The former type focuses on exploitation first, interspersed with bouts of exploration, whereas the latter type prefers to explore the search space first and consume later. Our findings from an eye-tracking study of experts' interactions with novel search interfaces in the biomedical domain suggest that user traits of cognitive styles and perceived search task difficulty are significantly correlated with eye gaze and search behavior. We also demonstrate that perceived risk shifts the balance between exploration and exploitation in either type of users, tilting it against vs. in favor of ambiguity minimization. Since the pattern of behavior in information foraging is quintessentially sequential, risk and ambiguity minimization cannot happen simultaneously, leading to a fundamental limit on how good such a tradeoff can be. This in turn connects information seeking with the emergent field of quantum decision theory.

Understanding visual attention in childhood: Insights from a new visual foraging task

A recently developed visual foraging task, involving multiple targets of different types, can provide a rich and dynamic picture of visual attention performance. We measured the foraging performance of 66 children aged 4–7 years, along with measures of two conceptually related constructs, self-regulation and verbal working memory. Our results show that foraging patterns of young children differ from adult patterns. Children have difficulty with foraging for two target types, not only when they are defined by a conjunction of features but, unlike adults, also when they forage simultaneously for two target types that are distinguished from distractors by a single feature. Importantly, such feature/conjunction differences between adults and children are not seen in more traditional single-target visual search tasks. Interestingly, the foraging patterns of the youngest children were slightly more adult-like than of the oldest ones, which may suggest that older children attempt to use strategies that they have not yet fully mastered. The older children were, however, able to complete more trials, during both feature and conjunction foraging. Self-regulation and verbal working memory did not seem to affect foraging strategies, but both were connected with faster and more efficient foraging. We propose that our visual foraging paradigm is a promising avenue for studying the development of visual cognitive abilities.

Internal states drive nutrient homeostasis by modulating exploration-exploitation trade-off

Internal states can profoundly alter the behavior of animals. A quantitative understanding of the behavioral changes upon metabolic challenges is key to a mechanistic dissection of how animals maintain nutritional homeostasis. We used an automated video tracking setup to characterize how amino acid and reproductive states interact to shape exploitation and exploration decisions taken by adult Drosophila melanogaster. We find that these two states have specific effects on the decisions to stop at and leave proteinaceous food patches. Furthermore, the internal nutrient state defines the exploration-exploitation trade-off: nutrient-deprived flies focus on specific patches while satiated flies explore more globally. Finally, we show that olfaction mediates the efficient recognition of yeast as an appropriate protein source in mated females and that octopamine is specifically required to mediate homeostatic postmating responses without affecting internal nutrient sensing. Internal states therefore modulate specific aspects of exploitation and exploration to change nutrient selection.

DOI:http://dx.doi.org/10.7554/eLife.19920.001

When making decisions, animals, including humans, do not always choose the same option. One reason for this is that their “internal state” changes the value of different options. This is particularly evident when deciding what type of food to eat. Depending on which nutrients the animal needs, it will choose to eat different foods.

Amino acids are key nutrients that affect health, lifespan and reproduction. Female fruit flies that have recently mated, for example, eat more amino acids in order to obtain the raw materials required to produce eggs. Despite the importance of amino acids, little was known about how animal behavior changes in response to a lack of this nutrient.

Corrales-Carvajal et al. used a video tracking system to measure the time that fruit flies – some of which had a need for amino acids – spent feeding on patches of yeast (which are rich in amino acids) versus patches of sucrose. Recently mated females – and virgins that had been fed a diet lacking in amino acids – consumed more yeast than sucrose, whereas virgin females that were not amino acid deficient showed the opposite pattern. To bias the fly toward eating the right food for their needs, several aspects of the fly’s behavior changed, including the number and length of individual feeding bouts. These different behaviors did not all change at the same time.

The pattern of exploration taken by the flies also depended on their need for amino acids. Amino acid deficient flies spent most of their time near known yeast patches. By contrast, fully fed flies adopted a riskier foraging strategy, moving away from known sources of food to explore their environment more widely. In common with humans, the flies relied upon their sense of smell to efficiently identify different types of food.

Overall, the results presented by Corrales-Carvajal et al. provide us with a detailed understanding about how changes to the internal state of the fly affect its behavior. The next step will be to use the powerful genetic tools available for studying fruit flies to reveal the neural circuits and molecular mechanisms that help animals find the types of food that they need.

DOI:http://dx.doi.org/10.7554/eLife.19920.002

T Cell Interstitial Migration: Motility Cues from the Inflamed Tissue for Micro- and Macro-Positioning

Effector T cells exit the inflamed vasculature into an environment shaped by tissue-specific structural configurations and inflammation-imposed extrinsic modifications. Once within interstitial spaces of non-lymphoid tissues, T cells migrate in an apparent random, non-directional, fashion. Efficient T cell scanning of the tissue environment is essential for successful location of infected target cells or encounter with antigen-presenting cells that activate the T cell's antimicrobial effector functions. The mechanisms of interstitial T cell motility and the environmental cues that may promote or hinder efficient tissue scanning are poorly understood. The extracellular matrix (ECM) appears to play an important scaffolding role in guidance of T cell migration and likely provides a platform for the display of chemotactic factors that may help to direct the positioning of T cells. Here, we discuss how intravital imaging has provided insight into the motility patterns and cellular machinery that facilitates T cell interstitial migration and the critical environmental factors that may optimize the efficiency of effector T cell scanning of the inflamed tissue. Specifically, we highlight the local micro-positioning cues T cells encounter as they migrate within inflamed tissues, from surrounding ECM and signaling molecules, as well as a requirement for appropriate long-range macro-positioning within distinct tissue compartments or at discrete foci of infection or tissue damage. The central nervous system (CNS) responds to injury and infection by extensively remodeling the ECM and with the de novo generation of a fibroblastic reticular network that likely influences T cell motility. We examine how inflammation-induced changes to the CNS landscape may regulate T cell tissue exploration and modulate function.

Foraging success under uncertainty: search tradeoffs and optimal space use

Understanding the structural complexity and the main drivers of animal search behaviour is pivotal to foraging ecology. Yet, the role of uncertainty as a generative mechanism of movement patterns is poorly understood. Novel insights from search theory suggest that organisms should collect and assess new information from the environment by producing complex exploratory strategies. Based on an extension of the first passage time theory, and using simple equations and simulations, we unveil the elementary heuristics behind search behaviour. In particular, we show that normal diffusion is not enough for determining optimal exploratory behaviour but anomalous diffusion is required. Searching organisms go through two critical sequential phases (approach and detection) and experience fundamental search tradeoffs that may limit their encounter rates. Using experimental data, we show that biological search includes elements not fully considered in contemporary physical search theory. In particular, the need to consider search movement as a non-stationary process that brings the organism from one informational state to another. For example, the transition from remaining in an area to departing from it may occur through an exploratory state where cognitive search is challenged. Therefore, a more comprehensive view of foraging ecology requires including current perspectives about movement under uncertainty.

Active Inference, epistemic value, and vicarious trial and error

Balancing habitual and deliberate forms of choice entails a comparison of their respective merits—the former being faster but inflexible, and the latter slower but more versatile. Here, we show that arbitration between these two forms of control can be derived from first principles within an Active Inference scheme. We illustrate our arguments with simulations that reproduce rodent spatial decisions in T-mazes. In this context, deliberation has been associated with vicarious trial and error (VTE) behavior (i.e., the fact that rodents sometimes stop at decision points as if deliberating between choice alternatives), whose neurophysiological correlates are “forward sweeps” of hippocampal place cells in the arms of the maze under consideration. Crucially, forward sweeps arise early in learning and disappear shortly after, marking a transition from deliberative to habitual choice. Our simulations show that this transition emerges as the optimal solution to the trade-off between policies that maximize reward or extrinsic value (habitual policies) and those that also consider the epistemic value of exploratory behavior (deliberative or epistemic policies)—the latter requiring VTE and the retrieval of episodic information via forward sweeps. We thus offer a novel perspective on the optimality principles that engender forward sweeps and VTE, and on their role on deliberate choice.

Morphological differences between habitats are associated with physiological and behavioural trade-offs in stickleback (Gasterosteus aculeatus)

Local specialization can be advantageous for individuals and may increase the resilience of the species to environmental change. However, there may be trade-offs between morphological responses and physiological performance and behaviour. Our aim was to test whether habitat-specific morphology of stickleback (Gasterosteus aculeatus) interacts with physiological performance and behaviour at different salinities. We rejected the hypothesis that deeper body shape of fish from habitats with high predation pressure led to decreases in locomotor performance. However, there was a trade-off between deeper body shape and muscle quality. Muscle of deeper-bodied fish produced less force than that of shallow-bodied saltmarsh fish. Nonetheless, saltmarsh fish had lower swimming performance, presumably because of lower muscle mass overall coupled with smaller caudal peduncles and larger heads. Saltmarsh fish performed better in saline water (20 ppt) relative to freshwater and relative to fish from freshwater habitats. However, exposure to salinity affected shoaling behaviour of fish from all habitats and shoals moved faster and closer together compared with freshwater. We show that habitat modification can alter phenotypes of native species, but local morphological specialization is associated with trade-offs that may reduce its benefits.

Evolution of protolinguistic abilities as a by-product of learning to forage in structured environments

The skills required for the learning and use of language are the focus of extensive research, and their evolutionary origins are widely debated. Using agent-based simulations in a range of virtual environments, we demonstrate that challenges of foraging for food can select for cognitive mechanisms supporting complex, hierarchical, sequential learning, the need for which arises in language acquisition. Building on previous work, where we explored the conditions under which reinforcement learning is out-competed by seldom-reinforced continuous learning that constructs a network model of the environment, we now show that realistic features of the foraging environment can select for two critical advances: (i) chunking of meaningful sequences found in the data, leading to representations composed of units that better fit the prevalent statistical patterns in the environment; and (ii) generalization across units based on their contextual similarity. Importantly, these learning processes, which in our framework evolved for making better foraging decisions, had been earlier shown to reproduce a range of findings in language learning in humans. Thus, our results suggest a possible evolutionary trajectory that may have led from basic learning mechanisms to complex hierarchical sequential learning that can support advanced cognitive abilities of the kind needed for language acquisition.

Slow Lévy flights

Among Markovian processes, the hallmark of Lévy flights is superdiffusion, or faster-than-Brownian dynamics. Here we show that Lévy laws, as well as Gaussian distributions, can also be the limit distributions of processes with long-range memory that exhibit very slow diffusion, logarithmic in time. These processes are path dependent and anomalous motion emerges from frequent relocations to already visited sites. We show how the central limit theorem is modified in this context, keeping the usual distinction between analytic and nonanalytic characteristic functions. A fluctuation-dissipation relation is also derived. Our results may have important applications in the study of animal and human displacements.

T cell migration, search strategies and mechanisms

T cell migration is essential for T cell responses; it allows for the detection of cognate antigen at the surface of antigen-presenting cells and for interactions with other cells involved in the immune response. Although appearing random, growing evidence suggests that T cell motility patterns are strategic and governed by mechanisms that are optimized for both the activation stage of the cell and for environment-specific cues. In this Opinion article, we discuss how the combined effects of T cell-intrinsic and -extrinsic forces influence T cell motility patterns in the context of highly complex tissues that are filled with other cells involved in parallel motility. In particular, we examine how insights from 'search theory' can be used to describe T cell movement across an 'exploitation-exploration trade-off' in the context of activation versus effector function and lymph nodes versus peripheral tissues.

Who Knows? Metacognitive Social Learning Strategies

To make good use of learning from others (social learning), we need to learn from the right others; from agents who know better than we do. Research on social learning strategies (SLSs) has identified rules that focus social learning on the right agents, and has shown that the behaviour of many animals conforms to these rules. However, it has not asked what the rules are made of, that is, about the cognitive processes implementing SLSs. Here, I suggest that most SLSs depend on domain-general, sensorimotor processes. However, some SLSs have the characteristics tacitly ascribed to all of them. These metacognitive SLSs represent 'who knows' in a conscious, reportable way, and have the power to promote cultural evolution.

Evidence for expansion of the precuneus in human evolution

The evolution of neurocranial morphology in Homo sapiens is characterized by bulging of the parietal region, a feature unique to our species. In modern humans, expansion of the parietal surface occurs during the first year of life, in a morphogenetic stage which is absent in chimpanzees and Neandertals. A similar variation in brain shape among living adult humans is associated with expansion of the precuneus. Using MRI-derived structural brain templates, we compare medial brain morphology between humans and chimpanzees through shape analysis and geometrical modeling. We find that the main spatial difference is a prominent expansion of the precuneus in our species, providing further evidence of evolutionary changes associated with this area. The precuneus is a major hub of brain organization, a central node of the default-mode network, and plays an essential role in visuospatial integration. Together, the comparative neuroanatomical and paleontological evidence suggest that precuneus expansion is a neurological specialization of H. sapiens that evolved in the last 150,000 years that may be associated with recent human cognitive specializations.

Hidden processes in structural representations: A reply to Abbott, Austerweil, and Griffiths (2015)

In recent work exploring the semantic fluency task, we found evidence indicative of optimal foraging policies in memory search that mirror search in physical environments. We determined that a 2-stage cue-switching model applied to a memory representation from a semantic space model best explained the human data. Abbott, Austerweil, and Griffiths demonstrate how these patterns could also emerge from a random walk applied to a network representation of memory based on human free-association norms. However, a major representational issue limits any conclusions that can be drawn about the process model comparison: Our process model operated on a memory space constructed from a learning model, whereas their model used human behavioral data from a task that is quite similar to the behavior they attempt to explain. Predicting semantic fluency (e.g., how likely it is to say cat after dog in a sequence of animals) from free association (how likely it is to say cat when given dog as a cue) should be possible with a relatively simple retrieval mechanism. The 2 tasks both tap memory, but they also share a common process of retrieval. Assuming that semantic memory is a network from free-association behavior embeds variance due to the shared retrieval process directly into the representation. A simple process mechanism is then sufficient to simulate semantic fluency because much of the requisite process complexity may already be hidden in the representation. (PsycINFO Database Record

Interfacing Mind and Environment: The Central Role of Search in Cognition

Search can be found in almost every cognitive activity, ranging across vision, memory retrieval, problem solving, decision making, foraging, and social interaction. Because of its ubiquity, research on search has a tendency to fragment into multiple areas of cognitive science. The proposed topic aims at providing integrative discussion of the central role of search from multiple perspectives. We focus on controlled search processes, which require (a) a goal, (b) uncertainty about the nature, location, or acquisition method of the objects to be searched for, and (c) a method for sampling through the search environment. While this definition of search is general and applicable to different domains, the specific mechanisms in the search process will likely differ. The goal of this issue is to compare and contrast how these search processes are similar and differ in different cognitive activities, with the goal of understanding the general nature of search in terms of the three characteristics stated above. We expect that given its cross-domain nature, the topic on search will be of broad interest to cognitive scientists, including psychologists, behavioral ecologists, computer scientists, neuroscientists, linguists, and sociologists.

A frontal dopamine system for reflective exploratory behavior

The COMT gene modulates dopamine levels in prefrontal cortex with Met allele carriers having lower COMT enzyme activity and, therefore, higher dopamine levels compared to Val/Val homozygotes. Concordantly, Val/Val homozygotes tend to perform worse and display increased (interpreted as inefficient) frontal activation in certain cognitive tasks. In a sample of 209 participants, we test the hypothesis that Met carriers will be advantaged in a decision-making task that demands sequencing exploratory and exploitive choices to minimize uncertainty about the reward structure in the environment. Previous work suggests that optimal performance depends on limited cognitive resources supported by prefrontal systems. If so, Met carriers should outperform Val/Val homozygotes, particularly under dual-task conditions that tax limited cognitive resources. In accord with these a priori predictions, Met carriers were more resilient in the face of cognitive load, continuing to explore in a sophisticated manner. We fit computational models that embody sophisticated reflective and simple reflexive strategies to further evaluate participants' exploration behavior. The Ideal Actor model reflectively updates beliefs and plans ahead, taking into account the information gained by each choice and making choices that maximize long-term payoffs. In contrast, the Naïve Reinforcement Learning (RL) model instantiates the reflexive account of choice, in which the values of actions are based only on the rewards experienced so far. Its beliefs are updated reflexively in response to observed changes in rewards. Converging with standard analyses, Met carriers were best characterized by the Ideal Actor model, whereas Val/Val homozygotes were best characterized by the Naive RL model, particularly under dual-task conditions.