Exploration versus exploitation in space, mind, and society

From exploration to exploitation: a shifting mental mode in late life development

Changes in cognition, affect, and brain function combine to promote a shift in the nature of mentation in older adulthood, favoring exploitation of prior knowledge over exploratory search as the starting point for thought and action. Age-related exploitation biases result from the accumulation of prior knowledge, reduced cognitive control, and a shift toward affective goals. These are accompanied by changes in cortical networks, as well as attention and reward circuits. By incorporating these factors into a unified account, the exploration-to-exploitation shift offers an integrative model of cognitive, affective, and brain aging. Here, we review evidence for this model, identify determinants and consequences, and survey the challenges and opportunities posed by an exploitation-biased mental mode in later life.

Characterization of exploratory patterns and hippocampal-prefrontal network oscillations during the emergence of free exploration

During free exploration, the emergence of patterned and sequential behavioral responses to an unknown environment reflects exploration traits and adaptation. However, the behavioral dynamics and neural substrates underlying the exploratory behavior remain poorly understood. We developed computational tools to quantify the exploratory behavior and performed in vivo electrophysiological recordings in a large arena in which mice made sequential excursions into unknown territory. Occupancy entropy was calculated to characterize the cumulative and moment-to-moment behavioral dynamics in explored and unexplored territories. Local field potential analysis revealed that the theta activity in the dorsal hippocampus (dHPC) was highly correlated with the occupancy entropy. Individual dHPC and prefrontal cortex (PFC) oscillatory activities could classify various aspects of free exploration. Initiation of exploration was accompanied by a coordinated decrease and increase in theta activity in PFC and dHPC, respectively. Our results indicate that dHPC and PFC work synergistically in shaping free exploration by modulating exploratory traits during emergence and visits to an unknown environment.

On the Effect of Practice on Exploration and Exploitation of Options and Strategies

Insufficient exploration of one's surroundings is at the root of many real-life problems, as demonstrated by many famous biases (e.g., the status quo bias, learned helplessness). The current work focuses on the emergence of this phenomenon at the strategy level: the tendency to under-explore the set of available choice strategies. We demonstrate that insufficient exploration of strategies can also manifest as excessive exploration between options. In such cases, interventions aimed at improving choices by reducing the costs of exploration of options are likely to fail. In Study 1, participants faced an exploration task that implies an infinite number of choice strategies and a small sub-set of (near) optimal solutions. We manipulated the amount of practice participants underwent during the first, shorter game and compared their performance in a second, longer game with an identical payoff structure. Our results show that regardless of the amount of practice, participants in all experimental groups tended to under-explore the strategy space and relied on a specific strategy that implied over-exploration of the option space. That is, under-exploration of strategies was manifested as over-exploration of options. In Study 2, we added a constraint that, on a subset of practice trials, forced participants to exploit familiar options. This manipulation almost doubled the per-trial average outcome on the comparable longer second game. This suggests that forcing participants to experience the effects of different (underexplored) strategy components during practice can greatly increase the chance they make better choices later on.

The past, present, and future of selection history

The last ten years of attention research have witnessed a revolution, replacing a theoretical dichotomy (top-down vs. bottom-up control) with a trichotomy (biased by current goals, physical salience, and selection history). This third new mechanism of attentional control, selection history, is multifaceted. Some aspects of selection history must be learned over time whereas others reflect much more transient influences. A variety of different learning experiences can shape the attention system, including reward, aversive outcomes, past experience searching for a target, target‒non-target relations, and more. In this review, we provide an overview of the historical forces that led to the proposal of selection history as a distinct mechanism of attentional control. We then propose a formal definition of selection history, with concrete criteria, and identify different components of experience-driven attention that fit within this definition. The bulk of the review is devoted to exploring how these different components relate to one another. We conclude by proposing an integrative account of selection history centered on underlying themes that emerge from our review.

Children are more exploratory and learn more than adults in an approach-avoid task

Intuitively, children appear to be more exploratory than adults, and this exploration seems to help children learn,. However, there have been few clear tests of these ideas. We test whether exploration and learning change across development using a task that presents a "learning trap." In this task, exploitation-maximizing immediate reward and avoiding costs-may lead the learner to draw incorrect conclusions, while exploration may lead to better learning but be more costly. In Studies 1, 2, and 3 we find that preschoolers and early school-aged children explore more than adults and learn the true structure of the environment better. Study 3 demonstrates that children explore more than adults even though they, like adults, predict that exploration will be costly, and it shows that exploration and learning are correlated. Study 4 shows that children's and adults' learning depends on the evidence they generate during exploration: children exposed to adult-like evidence learn like adults, and adults exposed to child-like evidence learn like children. Together, these studies support the idea that children may be more exploratory than adults, and this increased exploration influences learning.

The influence of exposure to randomness on lateral thinking in divergent, convergent, and creative search

Exposure to random stimuli has often been suggested to help unlock problem-solving abilities and creativity, helping us to see problems differently and imagine new possibilities. Equally, randomness is widely used in computer science to escape local maxima and find effective solutions to intractable problems. However, randomness has rarely been used as a formal aid in human decision making or investigated in controlled experimental settings. In this pre-registered study, we tested the effect of extraneous random stimuli using Wikipedia's random page generator on 592 British participants' performance across three online tasks: one 'convergent' forecasting task and two 'divergent' fluency tasks. We found no improvement associated with the treatment and often significant impairment. A Bayesian meta-analysis of the tasks finds strong support for the null hypothesis. We conclude that stimulating lateral thinking through random stimuli is non-trivial and may require such stimuli to be sufficiently task-related or 'optimally random'.

Sources of variation in search and foraging: A theoretical perspective

Search-the problem of exploring a space of alternatives to identify target goals-is a fundamental behaviour for many species. Although its foundation lies in foraging, most studies of human search behaviour have been directed towards understanding the attentional mechanisms that underlie the efficient visual exploration of two-dimensional (2D) scenes. With this review, we aim to characterise how search behaviour can be explained across a wide range of contexts, environments, spatial scales, and populations, both typical and atypical. We first consider the generality of search processes across psychological domains. We then review studies of interspecies differences in search. Finally, we explore in detail the individual and contextual variables that affect visual search and related behaviours in established experimental psychology paradigms. Despite the heterogeneity of the findings discussed, we identify that variations in control processes, along with the ability to regulate behaviour as a function of the structure of search space and the sampling processes adopted, to be central to explanations of variations in search behaviour. We propose a tentative theoretical model aimed at integrating these notions and close by exploring questions that remain unaddressed.

Detours increase local knowledge-Exploring the hidden benefits of self-control failure

Self-control enables people to override momentary thoughts, emotions, or impulses in order to pursue long-term goals. Good self-control is a predictor for health, success, and subjective well-being, as bad self-control is for the opposite. Therefore, the question arises why evolution has not endowed us with perfect self-control. In this article, we draw some attention to the hidden benefits of self-control failure and present a new experimental paradigm that captures both costs and benefits of self-control failure. In an experiment, participants worked on three consecutive tasks: 1) In a transcription task, we manipulated how much effortful self-control two groups of participants had to exert. 2) In a number-comparison task, participants of both groups were asked to compare numbers and ignore distracting neutral versus reward-related pictures. 3) After a pause for recreation, participants were confronted with an unannounced recognition task measuring whether they had incidentally encoded the distracting pictures during the previous number-comparison task. The results showed that participants who exerted a high amount of effortful self-control during the first task shifted their priorities and attention toward the distractors during the second self-control demanding task: The cost of self-control failure was reflected in worse performance in the number-comparison task. Moreover, the group which had exerted a high amount of self-control during the first task and showed self-control failure during the second task was better in the unannounced third task. The benefit of self-control failure during number comparison was reflected in better performance during the recognition task. However, costs and benefits were not specific for reward-related distractors but also occurred with neutral pictures. We propose that the hidden benefit of self-control failure lies in the exploration of distractors present during goal pursuit, i.e. the collection of information about the environment and the potential discovery of new sources of reward. Detours increase local knowledge.

Network analysis of intra- and interspecific freshwater fish interactions using year-around tracking

A long-term, yet detailed view into the social patterns of aquatic animals has been elusive. With advances in reality mining tracking technologies, a proximity-based social network (PBSN) can capture detailed spatio-temporal underwater interactions. We collected and analysed a large dataset of 108 freshwater fish from four species, tracked every few seconds over 1 year in their natural environment. We calculated the clustering coefficient of minute-by-minute PBSNs to measure social interactions, which can happen among fish sharing resources or habitat preferences (positive/neutral interactions) or in predator and prey during foraging interactions (agonistic interactions). A statistically significant coefficient compared to an equivalent random network suggests interactions, while a significant aggregated clustering across PBSNs indicates prolonged, purposeful social behaviour. Carp (Cyprinus carpio) displayed within- and among-species interactions, especially during the day and in the winter, while tench (Tinca tinca) and catfish (Silurus glanis) were solitary. Perch (Perca fluviatilis) did not exhibit significant social behaviour (except in autumn) despite being usually described as a predator using social facilitation to increase prey intake. Our work illustrates how methods for building a PBSN can affect the network's structure and highlights challenges (e.g. missing signals, different burst frequencies) in deriving a PBSN from reality mining technologies.

Biosociological ethodiversity in the social system

A comprehensive understanding of human sociality needs to embrace the coevolution of genes and culture. Recent advances in biological research about niche construction by organisms, and the development of the concepts of social niche and ethodiversity, can be integrated into a common approach to understand this coevolution, which implies the interaction between sociology and ecology in an integrative framework of knowledge. In this paper the authors propose such inclusive biosociological and heuristic framework to improve the understanding of the evolution of social niche construction. In addition, it allows a better understanding of the concept of sociotype in non-human organisms and explains some aspects of the social or presocial behavior through the concept of ethodiversity.

Potential-driven random walks on interconnected systems

Interconnected systems have to route information to function properly: At the lowest scale neural cells exchange electrochemical signals to communicate, while at larger scales animals and humans move between distinct spatial patches and machines exchange information via the Internet through communication protocols. Nontrivial patterns emerge from the analysis of information flows, which are not captured either by broadcasting, such as in random walks, or by geodesic routing, such as shortest paths. In fact, alternative models between those extreme protocols are still eluding us. Here we propose a class of stochastic processes, based on biased random walks, where agents are driven by a physical potential pervading the underlying network topology. By considering a generalized Coulomb dependence on the distance on destination(s), we show that it is possible to interpolate between random walk and geodesic routing in a simple and effective way. We demonstrate that it is not possible to find a one-size-fit-all solution to efficient navigation and that network heterogeneity or modularity has measurable effects. We illustrate how our framework can describe the movements of animals and humans, capturing with a stylized model some measurable features of the latter. From a methodological perspective, our potential-driven random walks open the doors to a broad spectrum of analytical tools, ranging from random-walk centralities to geometry induced by potential-driven network processes.

The role of great ape behavioral ecology in One Health: Implications for captive welfare and re-habilitation success

Behavior is the interface through which animals interact with their environments, and therefore has potentially cascading impacts on the health of individuals, populations, their habitats, and the humans that share them. Evolution has shaped the interaction between species and their environments. Thus, alterations to the species-typical "wild-type" behavioral repertoire (and the ability of the individual to adapt flexibly which elements of the repertoire it employs) may disrupt the relationship between the organism and its environment, creating cascading One Health effects. A good example is rehabilitant orangutans where, for example, seemingly minor differences from wild conspecifics in the time spent traveling on the ground rather than in the forest canopy can affect an individual's musculoskeletal and nutritional health, as well as social integration. It can also increase two-way transmission of infectious diseases and/or pathogens with local human populations, or potentially with neighboring wild populations if there are no geographical barriers and rehabilitants travel far enough to leave their release area. Primates are well known ecosystem engineers, reshaping plant communities and maintaining biodiversity through seed dispersal, consuming plants, and creating canopy gaps and trails. From the habitat perspective, a rehabilitant orangutan which does not behave like a wild orangutan is unlikely to fulfill these same ecosystem services. Despite the importance of the diversity of an ape's behavioral repertoire, how it compares to that of wild conspecifics and how it alters in response to habitat variation, behavior is an often under-appreciated aspect of One Health. In this review, focusing on orangutans as an example of the kinds of problems faced by all captive great apes, we examine the ways in which understanding and facilitating the expression of wild-type behavior can improve their health, their ability to thrive, and the robustness of local One Health systems.

Simple Threshold Rules Solve Explore/Exploit Trade-offs in a Resource Accumulation Search Task

How, and how well, do people switch between exploration and exploitation to search for and accumulate resources? We study the decision processes underlying such exploration/exploitation trade-offs using a novel card selection task that captures the common situation of searching among multiple resources (e.g., jobs) that can be exploited without depleting. With experience, participants learn to switch appropriately between exploration and exploitation and approach optimal performance. We model participants' behavior on this task with random, threshold, and sampling strategies, and find that a linear decreasing threshold rule best fits participants' results. Further evidence that participants use decreasing threshold-based strategies comes from reaction time differences between exploration and exploitation; however, participants themselves report non-decreasing thresholds. Decreasing threshold strategies that "front-load" exploration and switch quickly to exploitation are particularly effective in resource accumulation tasks, in contrast to optimal stopping problems like the Secretary Problem requiring longer exploration.

Priming exploration across domains: does search in a spatial environment influence search in a cognitive environment?

Is there a general tendency to explore that connects search behaviour across different domains? Although the experimental evidence collected so far suggests an affirmative answer, this fundamental question about human behaviour remains open. A feasible way to test the domain-generality hypothesis is that of testing the so-called priming hypothesis: priming explorative behaviour in one domain should subsequently influence explorative behaviour in another domain. However, only a limited number of studies have experimentally tested this priming hypothesis, and the evidence is mixed. We tested the priming hypothesis in a registered report. We manipulated explorative behaviour in a spatial search task by randomly allocating people to search environments with resources that were either clustered together or dispersedly distributed. We hypothesized that, in a subsequent anagram task, participants who searched in clustered spatial environments would search for words in a more clustered way than participants who searched in the dispersed spatial environments. The pre-registered hypothesis was not supported. An equivalence test showed that the difference between conditions was smaller than the smallest effect size of interest (d = 0.36). Out of several exploratory analyses, we found only one inferential result in favour of priming. We discuss implications of these findings for the theory and propose future tests of the hypothesis.

Active Inferants: An Active Inference Framework for Ant Colony Behavior

In this paper, we introduce an active inference model of ant colony foraging behavior, and implement the model in a series of in silico experiments. Active inference is a multiscale approach to behavioral modeling that is being applied across settings in theoretical biology and ethology. The ant colony is a classic case system in the function of distributed systems in terms of stigmergic decision-making and information sharing. Here we specify and simulate a Markov decision process (MDP) model for ant colony foraging. We investigate a well-known paradigm from laboratory ant colony behavioral experiments, the alternating T-maze paradigm, to illustrate the ability of the model to recover basic colony phenomena such as trail formation after food location discovery. We conclude by outlining how the active inference ant colony foraging behavioral model can be extended and situated within a nested multiscale framework and systems approaches to biology more generally.

Flexible modulation of sequence generation in the entorhinal-hippocampal system

Exploration, consolidation and planning depend on the generation of sequential state representations. However, these algorithms require disparate forms of sampling dynamics for optimal performance. We theorize how the brain should adapt internally generated sequences for particular cognitive functions and propose a neural mechanism by which this may be accomplished within the entorhinal-hippocampal circuit. Specifically, we demonstrate that the systematic modulation along the medial entorhinal cortex dorsoventral axis of grid population input into the hippocampus facilitates a flexible generative process that can interpolate between qualitatively distinct regimes of sequential hippocampal reactivations. By relating the emergent hippocampal activity patterns drawn from our model to empirical data, we explain and reconcile a diversity of recently observed, but apparently unrelated, phenomena such as generative cycling, diffusive hippocampal reactivations and jumping trajectory events.

"Is it Reasonable to Study Decision-Making Quantitatively?"

Scientists studying decision-making often provide a set of choices, each specified with values or distributions of values, and probabilities or distributions of probabilities. For example, "Would you prefer $100 with probability 1.0 or $1 with probability .9 and $1,000 with probability 0.1?" Other decision research examines choices made in the absence of most quantitative information; for example, "Would you prefer a Ford now or a Porsche a year from now?," "Which food would you prefer," but models the findings with precise quantitative assumptions. Yet other research does neither; for example, modeling verbally stated choices with verbally stated heuristics. This article asks about the relevance of the first two research approaches for much of the decision-making made in life. The use of quantitative research and modeling is unsurprising, given that this approach underlies most of science. In life, values and probabilities are almost always partly or wholly vague and qualitative rather than quantitative. For example, when deciding which house to buy, there are relevant features such as size, color, neighborhood schools, construction materials, attractiveness, and many more, but the decision-maker finds it difficult and of little use to assign these precise values or weights. Nonetheless, humans have evolved to make decisions in such vaguely specified settings. I provide an example showing how a very high degree of uncertainty can defeat the application of quantitative decision-making, but such a demonstration is not critical if quantitative research and modeling produce a good understanding of and a good approximation to decision-making in the natural environment. This perspective addresses these issues.

The Predation Game: Does dividing attention affect patterns of human foraging?

Attention is known to play an important role in shaping the behaviour of both human and animal foragers. Here, in three experiments, we built on previous interactive tasks to create an online foraging game for studying divided attention in human participants exposed to the (simulated) risk of predation. Participants used a "sheep" icon to collect items from different target categories randomly distributed across the display. Each trial also contained "wolf" objects, whose movement was inspired by classic studies of multiple object tracking. When participants needed to physically avoid the wolves, foraging patterns changed, with an increased tendency to switch between target categories and a decreased ability to prioritise high reward targets, relative to participants who could safely ignore them. However, when the wolves became dangerous by periodically changing form (briefly having big eyes) instead of by approaching the sheep, foraging patterns were unaffected. Spatial disruption caused by the need to rapidly shift position-rather the cost of reallocating attention-therefore appears to influence foraging in this context. These results thus confirm that participants can efficiently alternate between target selection and tracking moving objects, replicating earlier single-target search findings. Future studies may need to increase the perceived risk or potential costs associated with simulated danger, in order to elicit the extended run behaviour predicted by animal models of foraging, but absent in the current data.

Two variations and one similarity in memory functions deployed by mice and humans to support foraging

Assessing variations in cognitive function between humans and animals is vital for understanding the idiosyncrasies of human cognition and for refining animal models of human brain function and disease. We determined memory functions deployed by mice and humans to support foraging with a search task acting as a test battery. Mice searched for food from the top of poles within an open arena. Poles were divided into groups based on visual cues and baited according to different schedules. White and black poles were baited in alternate trials. Striped poles were never baited. The requirement of the task was to find all baits in each trial. Mice's foraging efficiency, defined as the number of poles visited before all baits were retrieved, improved with practice. Mice learnt to avoid visiting unbaited poles across trials (long-term memory) and revisits to poles within each trial (working memory). Humans tested with a virtual reality version of the task outperformed mice in foraging efficiency, working memory, and exploitation of the temporal pattern of rewards across trials. Moreover, humans, but not mice, reduced the number of possible movement sequences used to search the set of poles. For these measures, interspecies differences were maintained throughout the 3 weeks of testing. By contrast, long-term memory for never-rewarded poles was similar in mice and humans after the first week of testing. These results indicate that human cognitive functions relying on archaic brain structures may be adequately modelled in mice. Conversely, modelling in mice fluid skills likely to have developed specifically in primates requires caution.

Balancing exploration and exploitation with information and randomization

Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.

Bayesian collective learning emerges from heuristic social learning

Researchers across cognitive science, economics, and evolutionary biology have studied the ubiquitous phenomenon of social learning-the use of information about other people's decisions to make your own. Decision-making with the benefit of the accumulated knowledge of a community can result in superior decisions compared to what people can achieve alone. However, groups of people face two coupled challenges in accumulating knowledge to make good decisions: (1) aggregating information and (2) addressing an informational public goods problem known as the exploration-exploitation dilemma. Here, we show how a Bayesian social sampling model can in principle simultaneously optimally aggregate information and nearly optimally solve the exploration-exploitation dilemma. The key idea we explore is that Bayesian rationality at the level of a population can be implemented through a more simplistic heuristic social learning mechanism at the individual level. This simple individual-level behavioral rule in the context of a group of decision-makers functions as a distributed algorithm that tracks a Bayesian posterior in population-level statistics. We test this model using a large-scale dataset from an online financial trading platform.

Foraging behavior in visual search: A review of theoretical and mathematical models in humans and animals

Visual search (VS) is a fundamental task in daily life widely studied for over half a century. A variant of the classic paradigm-searching one target among distractors-requires the observer to look for several (undetermined) instances of a target (so-called foraging) or several targets that may appear an undefined number of times (recently named as hybrid foraging). In these searches, besides looking for targets, the observer must decide how much time is needed to exploit the area, and when to quit the search to eventually explore new search options. In fact, visual foraging is a very common search task in the real world, probably involving additional cognitive functions than typical VS. It has been widely studied in natural animal environments, for which several mathematical models have been proposed, and just recently applied to humans: Lévy processes, composite and area-restricted search models, marginal value theorem, and Bayesian learning (among others). We conducted a systematic search in the literature to understand those mathematical models and study its applicability in human visual foraging. The review suggests that these models might be the first step, but they seem to be limited to fully comprehend foraging in visual search. There are essential variables involving human visual foraging still to be established and understood. Indeed, a jointly theoretical interpretation based on the different models reviewed could better account for its understanding. In addition, some other relevant variables, such as certain individual differences or time perception might be crucial to understanding visual foraging in humans.

Development of directed and random exploration in children

Are young children just random explorers who learn serendipitously? Or are even young children guided by uncertainty-directed sampling, seeking to explore in a systematic fashion? We study how children between the ages of 4 and 9 search in an explore-exploit task with spatially correlated rewards, where exhaustive exploration is infeasible and not all options can be experienced. By combining behavioral data with a computational model that decomposes search into similarity-based generalization, uncertainty-directed exploration, and random exploration, we map out developmental trajectories of generalization and exploration. The behavioral data show strong developmental differences in children's capability to exploit environmental structure, with performance and adaptiveness of sampling decisions increasing with age. Through model-based analyses, we disentangle different forms of exploration, finding signature of both uncertainty-directed and random exploration. The amount of random exploration strongly decreases as children get older, supporting the notion of a developmental "cooling off" process that modulates the randomness in sampling. However, even at the youngest age range, children do not solely rely on random exploration. Even as random exploration begins to taper off, children are actively seeking out options with high uncertainty in a goal-directed fashion, and using inductive inferences to generalize their experience to novel options. Our findings provide critical insights into the behavioral and computational principles underlying the developmental trajectory of learning and exploration.

Individual difference in serial dependence results from opposite influences of perceptual choices and motor responses

Natural image statistics exhibit temporal regularities of slow changes and short-term correlations and visual perception, too, is biased toward recently seen stimuli, i.e., a positive serial dependence. Some studies report strong individual differences in serial dependence in perceptual decision-making: some observers show positive serial effects, others repulsive effects, and some show no bias. To understand these contrasting results, this study separates the influences of physical stimuli per se, perceptual choices, and motor responses on serial dependence in perceptual decision making. In two experiments, human observers reported which orientation (45° or −45°, at threshold contrast) they perceived. Experiment 1, used a consistent mapping between stimulus and response buttons whereas in Experiment 2, observers did two tasks: one with a consistent stimulus-response mapping, the other with a random stimulus-response mapping (perceptual choice and motor response unrelated). Results show that the stimulus percept (not the physical stimulus per se) affected subsequent perceptual choices in an attractive way and that motor responses produced a repulsive serial effect. When the choice-response mapping was consistent (inseparable choice and response, typical of most experiments), individual differences in the overall serial effect was observed: some were positive, some repulsive, and some were bias-free. The multiple regression analysis revealed that observers’ overall serial effects in the consistent choice-response mapping task could be predicted by their serial effects for choices and motor responses in the random mapping task. These individual differences likely reflect relative weightings of a positive choice bias and a repulsive motor bias.

Out of the blue: understanding abrupt and wayward transitions in thought using probability and predictive processing

Thoughts that appear to come to us ‘out of the blue’ or ‘out of nowhere’ are a familiar aspect of mental experience. Such thoughts tend to elicit feelings of surprise and spontaneity. Although we are beginning to understand the neural processes that underlie the arising of such thoughts, little is known about what accounts for their peculiar phenomenology. Here, we focus on one central aspect of this phenomenology—the experience of surprise at their occurrence, as it relates to internal probabilistic predictions regarding mental states. We introduce a distinction between two phenomenologically different types of transitions in thought content: (i) abrupt transitions, which occur at surprising times but lead to unsurprising thought content, and (ii) wayward transitions, which occur at surprising times and also lead to surprising thought content. We examine these two types of transitions using a novel approach that combines probabilistic and predictive processing concepts and principles. We employ two different probability metrics—transition and occurrence probability—to characterize and differentiate between abrupt and wayward transitions. We close by discussing some potentially beneficial ways in which these two kinds of transitions in thought content may contribute to mental function, and how they may be implemented at the neural level.

This article is part of the theme issue ‘Offline perception: voluntary and spontaneous perceptual experiences without matching external stimulation’.

Dogmatism manifests in lowered information search under uncertainty

When knowledge is scarce, it is adaptive to seek further information to resolve uncertainty and obtain a more accurate worldview. Biases in such information-seeking behavior can contribute to the maintenance of inaccurate views. Here, we investigate whether predispositions for uncertainty-guided information seeking relate to individual differences in dogmatism, a phenomenon linked to entrenched beliefs in political, scientific, and religious discourse. We addressed this question in a perceptual decision-making task, allowing us to rule out motivational factors and isolate the role of uncertainty. In two independent general population samples (n = 370 and n = 364), we show that more dogmatic participants are less likely to seek out new information to refine an initial perceptual decision, leading to a reduction in overall belief accuracy despite similar initial decision performance. Trial-by-trial modeling revealed that dogmatic participants placed less reliance on internal signals of uncertainty (confidence) to guide information search, rendering them less likely to seek additional information to update beliefs derived from weak or uncertain initial evidence. Together, our results highlight a cognitive mechanism that may contribute to the formation of dogmatic worldviews.

Linking ecology and cognition: does ecological specialisation predict cognitive test performance?

Variation in cognitive abilities is thought to be linked to variation in brain size, which varies across species with either social factors (Social Intelligence Hypothesis) or ecological challenges (Ecological Intelligence Hypothesis). However, the nature of the ecological processes invoked by the Ecological Intelligence Hypothesis, like adaptations to certain habitat characteristics or dietary requirements, remains relatively poorly known. Here, we review comparative studies that experimentally investigated interspecific variation in cognitive performance in relation to a species’ degree of ecological specialisation. Overall, the relevant literature was biased towards studies of mammals and birds as well as studies focusing on ecological challenges related to diet. We separated ecological challenges into those related to searching for food, accessing a food item and memorising food locations. We found interspecific variation in cognitive performance that can be explained by adaptations to different foraging styles. Species-specific adaptations to certain ecological conditions, like food patch distribution, characteristics of food items or seasonality also broadly predicted variation in cognitive abilities. A species’ innovative problem-solving and spatial processing ability, for example, could be explained by its use of specific foraging techniques or search strategies, respectively. Further, habitat generalists were more likely to outperform habitat specialists. Hence, we found evidence that ecological adaptations and cognitive performance are linked and that the classification concept of ecological specialisation can explain variation in cognitive performance only with regard to habitat, but not dietary specialisation.

Investigating information seeking in physical and online environments with escape room and web search

Searching and interacting with information is one of the most fundamental behaviours of human beings – something that takes place in both online and physical environments. Yet, most studies of information interaction have focused on only one of these sides. This work aims to connect them by investigating one’s information interaction behaviours in different physical and online contexts as well as different types of tasks. During Web search (online searching) and Escape Room (physical searching), 31 participants’ behavioural data during web search (online searching) and escape room (physical searching) were collected through eye-tracker, web browser logs, and wearable video recorder. Analysis of the behavioural data suggests that individuals have a preferred search strategy that they adopt across different tasks and environments. The behavioural pattern, however, was found to be affected by the task type (e.g. problem searching vs exploratory search) and the way information is structured within the environments.

Internal state transition to switch behavioral strategies in cricket phonotaxis

Animals employ multiple behavioral strategies for exploring food and mating partners based on both their internal state and external environment. Here, we examined how cricket phonotaxis, which was considered an innate reactive behavior of females to approach the calling song of conspecific males, depended on these internal and external conditions. Our observation revealed that the phonotaxis process consisted of two distinctive phases: wandering and approaching. In the latter phase, crickets moved directly towards the sound source. The transition into this phase, referred to as the 'approach phase', was based on changes in the animal's internal state. Moreover, retention of the approach phase required recognition of the calling song, while song loss downregulated cricket mobility and induced frequent stopping. This is a typical movement in local search behaviors. Our results indicate that phonotaxis is not only a reactive response but a complicated process including multiple behavioral strategies.

Inferring Behavior From Partial Social Information Plays Little or No Role in the Cultural Transmission of Adaptive Traits

Many human cultural traits become increasingly beneficial as they are repeatedly transmitted, thanks to an accumulation of modifications made by successive generations. But how do later generations typically avoid modifications which revert traits to less beneficial forms already sampled and rejected by earlier generations? And how can later generations do so without direct exposure to their predecessors' behavior? One possibility is that learners are sensitive to cues of non-random production in others' behavior, and that particular variants (e.g., those containing structural regularities unlikely to occur spontaneously) have been produced deliberately and with some effort. If this non-random behavior is attributed to an informed strategy, then the learner may infer that apparent avoidance of certain possibilities indicates that these have already been sampled and rejected. This could potentially prevent performance plateaus resulting from learners modifying inherited behaviors randomly. We test this hypothesis in four experiments in which participants, either individually or in interacting dyads, attempt to locate rewards in a search grid, guided by partial information about another individual's experience of the task. We find that in some contexts, valid inferences about another's behavior can be made from partial information, and these inferences can be used in a way which facilitates trait adaptation. However, the benefit of these inferences appears to be limited, and in many contexts-including some which have the potential to make inferring the experience of another individual easier-there appears to be no benefit at all. We suggest that inferring previous behavior from partial social information plays a minimal role in the adaptation of cultural traits.

The growth and form of knowledge networks by kinesthetic curiosity

Throughout life, we might seek a calling, companions, skills, entertainment, truth, self-knowledge, beauty, and edification. The practice of curiosity can be viewed as an extended and open-ended search for valuable information with hidden identity and location in a complex space of interconnected information. Despite its importance, curiosity has been challenging to computationally model because the practice of curiosity often flourishes without specific goals, external reward, or immediate feedback. Here, we show how network science, statistical physics, and philosophy can be integrated into an approach that coheres with and expands the psychological taxonomies of specific-diversive and perceptual-epistemic curiosity. Using this interdisciplinary approach, we distill functional modes of curious information seeking as searching movements in information space. The kinesthetic model of curiosity offers a vibrant counterpart to the deliberative predictions of model-based reinforcement learning. In doing so, this model unearths new computational opportunities for identifying what makes curiosity curious.

Structure in the stream of consciousness: Evidence from a verbalized thought protocol and automated text analytic methods

A key question about the spontaneous stream of thought (SST), often called the stream of consciousness, concerns its serial structure: How are thoughts in an extended sequence related to each other? In this study, we used a verbalized thought protocol to investigate "clump-and-jump" structure in SST-clusters of related thoughts about a topic followed by a jump to a new topic, in a repeating pattern. Several lines of evidence convergently supported the presence of clump-and-jump structure: high interrater agreement in identifying jumps, corroboration of rater-assigned jumps by automated text analytic methods, identification of clumps and jumps by a data-driven algorithm, and the inferred presence of clumps and jumps in unverbalized SST. We also found evidence that jumps involve a discontinuous shift in which a new clump is only modestly related to the previous one. These results illuminate serial structure in SST and invite research into the processes that generate the clump-and-jump pattern.

Functional advantages of Lévy walks emerging near a critical point

A special class of random walks, so-called Lévy walks, has been observed in a variety of organisms ranging from cells, insects, fishes, and birds to mammals, including humans. Although their prevalence is considered to be a consequence of natural selection for higher search efficiency, some findings suggest that Lévy walks might also be epiphenomena that arise from interactions with the environment. Therefore, why they are common in biological movements remains an open question. Based on some evidence that Lévy walks are spontaneously generated in the brain and the fact that power-law distributions in Lévy walks can emerge at a critical point, we hypothesized that the advantages of Lévy walks might be enhanced by criticality. However, the functional advantages of Lévy walks are poorly understood. Here, we modeled nonlinear systems for the generation of locomotion and showed that Lévy walks emerging near a critical point had optimal dynamic ranges for coding information. This discovery suggested that Lévy walks could change movement trajectories based on the magnitude of environmental stimuli. We then showed that the high flexibility of Lévy walks enabled switching exploitation/exploration based on the nature of external cues. Finally, we analyzed the movement trajectories of freely moving Drosophila larvae and showed empirically that the Lévy walks may emerge near a critical point and have large dynamic range and high flexibility. Our results suggest that the commonly observed Lévy walks emerge near a critical point and could be explained on the basis of these functional advantages.

Science-Driven Societal Transformation, Part I: Worldview

Humanity faces serious social and environmental problems, including climate change and biodiversity loss. Increasingly, scientists, global policy experts, and the general public conclude that incremental approaches to reduce risk are insufficient and transformative change is needed across all sectors of society. However, the meaning of transformation is still unsettled in the literature, as is the proper role of science in fostering it. This paper is the first in a three-part series that adds to the discussion by proposing a novel science-driven research-and-development program aimed at societal transformation. More than a proposal, it offers a perspective and conceptual framework from which societal transformation might be approached. As part of this, it advances a formal mechanics with which to model and understand self-organizing societies of individuals. While acknowledging the necessity of reform to existing societal systems (e.g., governance, economic, and financial systems), the focus of the series is on transformation understood as systems change or systems migration—the de novo development of and migration to new societal systems. The series provides definitions, aims, reasoning, worldview, and a theory of change, and discusses fitness metrics and design principles for new systems. This first paper proposes a worldview, built using ideas from evolutionary biology, complex systems science, cognitive sciences, and information theory, which is intended to serve as the foundation for the R&D program. Subsequent papers in the series build on the worldview to address fitness metrics, system design, and other topics.

Albatrosses can memorize locations of predictable fishing boats but favour natural foraging

Human activities generate food attracting many animals worldwide, causing major conservation issues. The spatio-temporal predictability of anthropogenic resources could reduce search costs for animals and mediate their attractiveness. We investigated this through GPS tracking in breeding black-browed albatrosses attracted to fishing boats. We tested for answers to the following questions. (i) Can future boat locations be anticipated from cues available to birds? (ii) Are birds able to appropriately use these cues to increase encounters? (iii) How frequently do birds use these cues? Boats were spatially persistent: birds searching in the direction where they previously attended boats would encounter twice as many boats compared with following a random direction strategy. A large proportion of birds did not use this cue: across pairs of consecutive trips (n = 85), 51% of birds switched their foraging direction irrespective of previous boat encounters. Still, 15 birds (27%) were observed to closely approach (approx. 0.1-1 km) where they previously attended a boat while boats were no longer there. This is less than the distance expected by chance (approx. 10-100 km), based on permutation control procedures accounting for individual-specific spatial consistency, suggesting individuals could memorize where they encountered boats across consecutive trips. We conclude albatrosses were able to exploit predictive cues from recent boat encounters but most favoured alternative resources.

Systematic exploration and uncertainty dominate young children's choices

Organisms need to constantly balance the competing demands of gathering information and using previously acquired information to obtain rewarding outcomes (i.e., the "exploration-exploitation" dilemma). Exploration is critical to obtain information to discover how the world works, which should be particularly important for young children. While studies have shown that young children explore in response to surprising events, little is known about how they balance exploration and exploitation across multiple decisions or about how this process changes with development. In this study, we compare decision-making patterns of children and adults and evaluate the relative influences of reward seeking, random exploration, and systematic switching (which approximates uncertainty-directed exploration). In a second experiment, we directly test the effect of uncertainty on children's choices. Influential models of decision-making generally describe systematic exploration as a computationally refined capacity that relies on top-down cognitive control. We demonstrate that (a) systematic patterns dominate young children's behavior (facilitating exploration), despite protracted development of cognitive control; and (b) that uncertainty plays a major, but complicated, role in determining children's choices. We conclude that while young children's immature top-down control should hinder adult-like systematic exploration, other mechanisms may pick up the slack, facilitating broad information gathering in a systematic fashion to build a foundation of knowledge for use later in life.

Explaining individual differences in infant visual sensory seeking

Individual differences in infants’ engagement with their environment manifest early in development and are noticed by parents. Three views have been advanced to explain differences in seeking novel stimulation. The optimal stimulation hypothesis suggests that individuals seek further stimulation when they are under‐responsive to current sensory input. The processing speed hypothesis proposes that those capable of processing information faster are driven to seek stimulation more frequently. The information prioritization hypothesis suggests the differences in stimulation seeking index variation in the prioritization of incoming relative to ongoing information processing. Ten‐month‐old infants saw 10 repetitions of a video clip and changes in frontal theta oscillatory amplitude were measured as an index of information processing speed. Stimulus‐locked P1 peak amplitude in response to checkerboards briefly overlaid on the video at random points during its presentation indexed processing of incoming stimulation. Parental report of higher visual seeking did not relate to reduced P1 peak amplitude or to a stronger decrease in frontal theta amplitude with repetition, thus not supporting either the optimal stimulation or the processing speed hypotheses. Higher visual seeking occurred in those infants whose P1 peak amplitude was greater than expected based on their theta amplitude. These findings indicate that visual sensory seeking in infancy is explained by a bias toward novel stimulation, thus supporting the information prioritization hypothesis.

The Downsides of Cognitive Enhancement

Cognitive enhancement is becoming progressively popular as a subject of scientific investigation and by the public, although possible adverse effects are not sufficiently understood. We call for cognitive enhancement to build on more specific, mechanistic theories given that a-theoretical approaches to cognitive enhancement are both a cause and a consequence of a strong, if not exclusive focus on the benefits of procedures suited to enhance human cognition. We focus on downsides of cognitive enhancement and suggest that every attempt to enhance human cognition needs to deal with two basic principles: the neuro-competition principle and the nonlinearity principle. We discuss the possibility of both principles in light of recent attempts to improve human cognition by means of transcranial direct current stimulation, a well-established brain stimulation method, and clinically relevant nootropic drugs. We propose that much stronger emphasis on mechanistic theorizing is necessary in guiding future research on both the upsides and the downsides of cognitive enhancement.

Learning From Success or Failure? - Positivity Biases Revisited

The purpose of this study was to reexamine positivity learning biases through a Q learning computation model and relate them to behavioral characteristics of exploitation and exploration. It was found that while the positivity learning biases existed in the simple asymmetric Q learning model, they completely disappeared once the time-varying nature of learning rates was incorporated. In the time-varying model, learning rates depended on the magnitudes of success and failure. The corresponding positive and negative learning rates were related to high and low performance, respectively, indicating that successes and failures were accounted for by positive and negative learning rates. Moreover, these learning rates were related to both exploitation and exploration in somewhat balanced ways. In contrast, under the constant learning parameter model, positivity biases were associated only with exploration. Therefore, the results in the time-varying model are more intuitively appealing than the simple asymmetric model. However, the statistical tests indicated that participants eclectically selected between the asymmetric learning model and its time-varying version, a frequency of which differed across participants.

Individual learning phenotypes drive collective behavior

Individual differences in learning can influence how animals respond to and communicate about their environment, which may nonlinearly shape how a social group accomplishes a collective task. There are few empirical examples of how differences in collective dynamics emerge from variation among individuals in cognition. Here, we use a naturally variable and heritable learning behavior called latent inhibition (LI) to show that interactions among individuals that differ in this cognitive ability drive collective foraging behavior in honey bee colonies. We artificially selected two distinct phenotypes: high-LI bees that ignore previously familiar stimuli in favor of novel ones and low-LI bees that learn familiar and novel stimuli equally well. We then provided colonies differentially composed of different ratios of these phenotypes with a choice between familiar and novel feeders. Colonies of predominantly high-LI individuals preferred to visit familiar food locations, while low-LI colonies visited novel and familiar food locations equally. Interestingly, in colonies of mixed learning phenotypes, the low-LI individuals showed a preference to visiting familiar feeders, which contrasts with their behavior when in a uniform low-LI group. We show that the shift in feeder preference of low-LI bees is driven by foragers of the high-LI phenotype dancing more intensely and attracting more followers. Our results reveal that cognitive abilities of individuals and their social interactions, which we argue relate to differences in attention, drive emergent collective outcomes.

How decisions and the desire for coherency shape subjective preferences over time

Recent findings suggest a bidirectional relationship between preferences and choices such that what is chosen can become preferred. Yet, it is still commonly held that preferences for individual items are maintained, such as caching a separate value estimate for each experienced option. Instead, we propose that all possible choice options and preferences are represented in a shared, continuous, multidimensional space that supports generalization. Decision making is cast as a learning process that seeks to align choices and preferences to maintain coherency. We formalized an error-driven learning model that updates preferences to align with past choices, which makes repeating those and related choices more likely in the future. The model correctly predicts that making a free choice increases preferences along related attributes. For example, after choosing a political candidate based on trivial information (e.g., they like cats), voters' views on abortion, immigration, and trade subsequently shifted to match their chosen candidate.

Exploratory dynamics of vocal foraging during infant-caregiver communication

We investigated the hypothesis that infants search in an acoustic space for vocalisations that elicit adult utterances and vice versa, inspired by research on animal and human foraging. Infant-worn recorders were used to collect day-long audio recordings, and infant speech-related and adult vocalisation onsets and offsets were automatically identified. We examined vocalisation-to-vocalisation steps, focusing on inter-vocalisation time intervals and distances in an acoustic space defined by mean pitch and mean amplitude, measured from the child's perspective. Infant inter-vocalisation intervals were shorter immediately following a vocal response from an adult. Adult intervals were shorter following an infant response and adult inter-vocalisation pitch differences were smaller following the receipt of a vocal response from the infant. These findings are consistent with the hypothesis that infants and caregivers are foraging vocally for social input. Increasing infant age was associated with changes in inter-vocalisation step sizes for both infants and adults, and we found associations between response likelihood and acoustic characteristics. Future work is needed to determine the impact of different labelling methods and of automatic labelling errors on the results. The study represents a novel application of foraging theory, demonstrating how infant behaviour and infant-caregiver interaction can be characterised as foraging processes.

Multiscale Computation and Dynamic Attention in Biological and Artificial Intelligence

Biological and artificial intelligence (AI) are often defined by their capacity to achieve a hierarchy of short-term and long-term goals that require incorporating information over time and space at both local and global scales. More advanced forms of this capacity involve the adaptive modulation of integration across scales, which resolve computational inefficiency and explore-exploit dilemmas at the same time. Research in neuroscience and AI have both made progress towards understanding architectures that achieve this. Insight into biological computations come from phenomena such as decision inertia, habit formation, information search, risky choices and foraging. Across these domains, the brain is equipped with mechanisms (such as the dorsal anterior cingulate and dorsolateral prefrontal cortex) that can represent and modulate across scales, both with top-down control processes and by local to global consolidation as information progresses from sensory to prefrontal areas. Paralleling these biological architectures, progress in AI is marked by innovations in dynamic multiscale modulation, moving from recurrent and convolutional neural networks-with fixed scalings-to attention, transformers, dynamic convolutions, and consciousness priors-which modulate scale to input and increase scale breadth. The use and development of these multiscale innovations in robotic agents, game AI, and natural language processing (NLP) are pushing the boundaries of AI achievements. By juxtaposing biological and artificial intelligence, the present work underscores the critical importance of multiscale processing to general intelligence, as well as highlighting innovations and differences between the future of biological and artificial intelligence.

Dopamine neurons do not constitute an obligatory stage in the final common path for the evaluation and pursuit of brain stimulation reward

The neurobiological study of reward was launched by the discovery of intracranial self-stimulation (ICSS). Subsequent investigation of this phenomenon provided the initial link between reward-seeking behavior and dopaminergic neurotransmission. We re-evaluated this relationship by psychophysical, pharmacological, optogenetic, and computational means. In rats working for direct, optical activation of midbrain dopamine neurons, we varied the strength and opportunity cost of the stimulation and measured time allocation, the proportion of trial time devoted to reward pursuit. We found that the dependence of time allocation on the strength and cost of stimulation was similar formally to that observed when electrical stimulation of the medial forebrain bundle served as the reward. When the stimulation is strong and cheap, the rats devote almost all their time to reward pursuit; time allocation falls off as stimulation strength is decreased and/or its opportunity cost is increased. A 3D plot of time allocation versus stimulation strength and cost produces a surface resembling the corner of a plateau (the “reward mountain”). We show that dopamine-transporter blockade shifts the mountain along both the strength and cost axes in rats working for optical activation of midbrain dopamine neurons. In contrast, the same drug shifted the mountain uniquely along the opportunity-cost axis when rats worked for electrical MFB stimulation in a prior study. Dopamine neurons are an obligatory stage in the dominant model of ICSS, which positions them at a key nexus in the final common path for reward seeking. This model fails to provide a cogent account for the differential effect of dopamine transporter blockade on the reward mountain. Instead, we propose that midbrain dopamine neurons and neurons with non-dopaminergic, MFB axons constitute parallel limbs of brain-reward circuitry that ultimately converge on the final-common path for the evaluation and pursuit of rewards.