Exploration versus exploitation in space, mind, and society

Tonic activity in lateral habenula neurons acts as a neutral valence brake on reward-seeking behavior

Survival requires both the ability to persistently pursue goals and the ability to determine when it is time to stop, an adaptive balance of perseverance and disengagement. Neural activity in the lateral habenula (LHb) has been linked to negative valence, but its role in regulating the balance between engaged reward seeking and disengaged behavioral states remains unclear. Here, we show that LHb neural activity is tonically elevated during minutes-long periods of disengagement from reward-seeking behavior, both when due to repeated reward omission (negative valence) and when sufficient reward has been consumed (positive valence). Furthermore, we show that LHb inhibition extends ongoing reward-seeking behavioral states but does not prompt task re-engagement. We find no evidence for similar tonic activity changes in ventral tegmental area dopamine neurons. Our findings support a framework in which tonic activity in LHb neurons suppresses engagement in reward-seeking behavior in response to both negatively and positively valenced factors.

Assessing Team Effectiveness by How Players Structure Their Search in a First-Person Multiplayer Video Game

People working as a team can achieve more than when working alone due to a team's ability to parallelize the completion of tasks. In collaborative search tasks, this necessitates the formation of effective division of labor strategies to minimize redundancies in search. For such strategies to be developed, team members need to perceive the task's relevant components and how they evolve over time, as well as an understanding of what others will do so that they can structure their own behavior to contribute to the team's goal. This study explored whether the capacity for team members to coordinate effectively can be related to how participants structure their search behaviors in an online multiplayer collaborative search task. Our results demonstrated that the structure of search behavior, quantified using detrended fluctuation analysis, was sensitive to contextual factors that limit a participant's ability to gather information. Further, increases in the persistence of movement fluctuations during search behavior were found as teams developed more effective coordinative strategies and were associated with better task performance.

Human-in-the-loop machine learning: a state of the art

Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in which the system remains in control; interactive machine learning, in which there is a closer interaction between users and learning systems; and machine teaching, where human domain experts have control over the learning process. Aside from control, humans can also be involved in the learning process in other ways. In curriculum learning human domain experts try to impose some structure on the examples presented to improve the learning; in explainable AI the focus is on the ability of the model to explain to humans why a given solution was chosen. This collaboration between AI models and humans should not be limited only to the learning process; if we go further, we can see other terms that arise such as Usable and Useful AI. In this paper we review the state of the art of the techniques involved in the new forms of relationship between humans and ML algorithms. Our contribution is not merely listing the different approaches, but to provide definitions clarifying confusing, varied and sometimes contradictory terms; to elucidate and determine the boundaries between the different methods; and to correlate all the techniques searching for the connections and influences between them.

Mapping Phenotypic Plasticity upon the Cancer Cell State Landscape Using Manifold Learning

ABSTRACT

Phenotypic plasticity describes the ability of cancer cells to undergo dynamic, nongenetic cell state changes that amplify cancer heterogeneity to promote metastasis and therapy evasion. Thus, cancer cells occupy a continuous spectrum of phenotypic states connected by trajectories defining dynamic transitions upon a cancer cell state landscape. With technologies proliferating to systematically record molecular mechanisms at single-cell resolution, we illuminate manifold learning techniques as emerging computational tools to effectively model cell state dynamics in a way that mimics our understanding of the cell state landscape. We anticipate that "state-gating" therapies targeting phenotypic plasticity will limit cancer heterogeneity, metastasis, and therapy resistance.

SIGNIFICANCE

Nongenetic mechanisms underlying phenotypic plasticity have emerged as significant drivers of tumor heterogeneity, metastasis, and therapy resistance. Herein, we discuss new experimental and computational techniques to define phenotypic plasticity as a scaffold to guide accelerated progress in uncovering new vulnerabilities for therapeutic exploitation.

Closing the mechanistic gap: the value of microarchitecture in understanding cognitive networks

Cognitive neuroscience aims to provide biologically relevant accounts of cognition. Contemporary research linking spatial patterns of neural activity to psychological constructs describes 'where' hypothesised functions occur, but not 'how' these regions contribute to cognition. Technological, empirical, and conceptual advances allow this mechanistic gap to be closed by embedding patterns of functional activity in macro- and microscale descriptions of brain organisation. Recent work on the default mode network (DMN) and the multiple demand network (MDN), for example, highlights a microarchitectural landscape that may explain how activity in these networks integrates varied information, thus providing an anatomical foundation that will help to explain how these networks contribute to many different cognitive states. This perspective highlights emerging insights into how microarchitecture can constrain network accounts of human cognition.

Algorithms of adaptation in inductive inference

We investigate the idea that human concept inference utilizes local adaptive search within a compositional mental theory space. To explore this, we study human judgments in a challenging task that involves actively gathering evidence about a symbolic rule governing the behavior of a simulated environment. Participants learn by performing mini-experiments before making generalizations and explicit guesses about a hidden rule. They then collect additional evidence themselves (Experiment 1) or observe evidence gathered by someone else (Experiment 2) before revising their own generalizations and guesses. In each case, we focus on the relationship between participants' initial and revised guesses about the hidden rule concept. We find an order effect whereby revised guesses are anchored to idiosyncratic elements of the earlier guess. To explain this pattern, we develop a family of process accounts that combine program induction ideas with local (MCMC-like) adaptation mechanisms. A particularly local variant of this adaptive account captures participants' hypothesis revisions better than a range of alternative explanations. We take this as suggestive that people deal with the inherent complexity of concept inference partly through use of local adaptive search in a latent compositional theory space.

Alternative model systems for cognitive variation: eusocial-insect colonies

Understanding the origins and maintenance of cognitive variation in animal populations is central to the study of the evolution of cognition. However, the brain is itself a complex, hierarchical network of heterogeneous components, from diverse cell types to diverse neuropils, each of which may be of limited use to study in isolation or prohibitively challenging to manipulate in situ. Consequently, highly tractable alternative model systems may be valuable tools. Eusocial-insect colonies display emergent cognitive-like properties from relatively simple social interactions between diverse subunits that can be observed and manipulated while operating collectively. Here, we review the individual-scale mechanisms that cause group-level variation in how colonies solve problems analogous to cognitive challenges faced by brains, like decision-making, attention, and search.

Horizons in Human Aging Neuroscience: From Normal Neural Aging to Mental (Fr)Agility

While aging is an important risk factor for neurodegenerative disorders such as Alzheimer’s disease and Parkinson’s disease, age-related cognitive decline can also manifest without apparent neurodegenerative changes. In this review, we discuss molecular, cellular, and network changes that occur during normal aging in the absence of neurodegenerative disease. Emerging findings reveal that these changes include metabolic alterations, oxidative stress, DNA damage, inflammation, calcium dyshomeostasis, and several other hallmarks of age-related neural changes that do not act on their own, but are often interconnected and together may underlie age-related alterations in brain plasticity and cognitive function. Importantly, age-related cognitive decline may not be reduced to a single neurobiological cause, but should instead be considered in terms of a densely connected system that underlies age-related cognitive alterations. We speculate that a decline in one hallmark of neural aging may trigger a decline in other, otherwise thus far stable subsystems, thereby triggering a cascade that may at some point also incur a decline of cognitive functions and mental well-being. Beyond studying the effects of these factors in isolation, considerable insight may be gained by studying the larger picture that entails a representative collection of such factors and their interactions, ranging from molecules to neural networks. Finally, we discuss some potential interventions that may help to prevent these alterations, thereby reducing cognitive decline and mental fragility, and enhancing mental well-being, and healthy aging.

Developmental Dyslexia: Disorder or Specialization in Exploration?

We raise the new possibility that people diagnosed with developmental dyslexia (DD) are specialized in explorative cognitive search, and rather than having a neurocognitive disorder, play an essential role in human adaptation. Most DD research has studied educational difficulties, with theories framing differences in neurocognitive processes as deficits. However, people with DD are also often proposed to have certain strengths - particularly in realms like discovery, invention, and creativity - that deficit-centered theories cannot explain. We investigate whether these strengths reflect an underlying explorative specialization. We re-examine experimental studies in psychology and neuroscience using the framework of cognitive search, whereby many psychological processes involve a trade-off between exploration and exploitation. We report evidence of an explorative bias in DD-associated cognitive strategies. High DD prevalence and an attendant explorative bias across multiple areas of cognition suggest the existence of explorative specialization. An evolutionary perspective explains the combination of findings and challenges the view that individuals with DD have a disorder. In cooperating groups, individual specialization is favored when features that confer fitness benefits are functionally incompatible. Evidence for search specialization suggests that, as with some other social organisms, humans mediate the exploration-exploitation trade-off by specializing in complementary strategies. The existence of a system of collective cognitive search that emerges through collaboration would help to explain our species' exceptional adaptiveness. It also aligns with evidence for substantial variability during our evolutionary history and the notion that humans are adapted not to a particular habitat but to variability itself. Specialization creates interdependence and necessitates balancing complementary strategies. Reframing DD therefore underscores the urgency of changing certain cultural practices to ensure we do not inhibit adaptation. Key improvements would remove cultural barriers to exploration and nurture explorative learning in education, academia, and the workplace, as well as emphasize collaboration over competition. Specialization in complementary search abilities represents a meta-adaptation; through collaboration, this likely enables human groups (as a species and as cultural systems) to successfully adapt. Cultural change to support this system of collaborative search may therefore be essential in confronting the challenges humanity now faces.

An investigation of the cognitive and neural correlates of semantic memory search related to creative ability

Creative ideas likely result from searching and combining semantic memory knowledge, yet the mechanisms acting on memory to yield creative ideas remain unclear. Here, we identified the neurocognitive correlates of semantic search components related to creative abilities. We designed an associative fluency task based on polysemous words and distinguished two search components related to clustering and switching between the different meanings of the polysemous words. Clustering correlated with divergent thinking, while switching correlated with the ability to combine remote associates. Furthermore, switching correlated with semantic memory structure and executive abilities, and was predicted by connectivity between the default, control, and salience neural networks. In contrast, clustering relied on interactions between control, salience, and attentional neural networks. Our results suggest that switching captures interactions between memory structure and control processes guiding the search whereas clustering may capture attentional controlled processes for persistent search, and that alternations between exploratory search and focused attention support creativity.

Competency in Navigating Arbitrary Spaces as an Invariant for Analyzing Cognition in Diverse Embodiments

One of the most salient features of life is its capacity to handle novelty and namely to thrive and adapt to new circumstances and changes in both the environment and internal components. An understanding of this capacity is central to several fields: the evolution of form and function, the design of effective strategies for biomedicine, and the creation of novel life forms via chimeric and bioengineering technologies. Here, we review instructive examples of living organisms solving diverse problems and propose competent navigation in arbitrary spaces as an invariant for thinking about the scaling of cognition during evolution. We argue that our innate capacity to recognize agency and intelligence in unfamiliar guises lags far behind our ability to detect it in familiar behavioral contexts. The multi-scale competency of life is essential to adaptive function, potentiating evolution and providing strategies for top-down control (not micromanagement) to address complex disease and injury. We propose an observer-focused viewpoint that is agnostic about scale and implementation, illustrating how evolution pivoted similar strategies to explore and exploit metabolic, transcriptional, morphological, and finally 3D motion spaces. By generalizing the concept of behavior, we gain novel perspectives on evolution, strategies for system-level biomedical interventions, and the construction of bioengineered intelligences. This framework is a first step toward relating to intelligence in highly unfamiliar embodiments, which will be essential for progress in artificial intelligence and regenerative medicine and for thriving in a world increasingly populated by synthetic, bio-robotic, and hybrid beings.

Insights about the common generative rule underlying an information foraging task can be facilitated via collective search

Social learning is beneficial for efficient information search in unfamiliar environments ("within-task" learning). In the real world, however, possible search spaces are often so large that decision makers are incapable of covering all options, even if they pool their information collectively. One strategy to handle such overload is developing generalizable knowledge that extends to multiple related environments ("across-task" learning). However, it is unknown whether and how social information may facilitate such across-task learning. Here, we investigated participants' social learning processes across multiple laboratory foraging sessions in spatially correlated reward landscapes that were generated according to a common rule. The results showed that paired participants were able to improve efficiency in information search across sessions more than solo participants. Computational analysis of participants' choice-behaviors revealed that such improvement across sessions was related to better understanding of the common generative rule. Rule understanding was correlated within a pair, suggesting that social interaction is a key to the improvement of across-task learning.

Computational models of exploration and exploitation characterise onset and efficacy of treatment in methamphetamine use disorder

People with Methamphetamine Use Disorder (PwMUD) spend substantial time and resources on substance use, which hinders their ability to explore alternate reinforcers. Gold‐standard behavioural treatments attempt to remedy this by encouraging action towards non‐drug reinforcers, but substance use often persists. We aimed to unravel the mechanistic drivers of this behaviour by applying a computational model of explore/exploit behaviour to decision‐making data (Iowa Gambling Task) from 106 PwMUD and 48 controls. We then examined the longitudinal link between explore/exploit mechanisms and changes in methamphetamine use 6 weeks later. Exploitation parameters included reinforcement sensitivity and inverse decay (i.e., number of past outcomes used to guide choices). Exploration parameters included maximum directed exploration value (i.e., value of trying novel actions). The Timeline Follow Back measured changes in methamphetamine use. Compared to controls, PwMUD showed deficits in exploitative decision‐making, characterised by reduced reinforcement sensitivity, U = 3065, p = 0.009, and less use of previous choice outcomes, U = 3062, p = 0.010. This was accompanied by a behavioural pattern of frequent shifting between choices, which appeared consistent with random exploration. Furthermore, PwMUD with greater reductions of methamphetamine use at 6 weeks had increased directed exploration (β = 0.22, p = 0.045); greater use of past choice outcomes (β = −0.39, p = 0.002) and greater choice consistency (β = −0.39, p = 0.002). Therefore, limited computational exploitation and increased behavioural exploration characterise PwMUD's presentation to treatment, while increased directed exploration, use of past choice outcomes and choice consistency predict greater reductions of methamphetamine use.

Individual exploration and selective social learning: balancing exploration-exploitation trade-offs in collective foraging

Search requires balancing exploring for more options and exploiting the ones previously found. Individuals foraging in a group face another trade-off: whether to engage in social learning to exploit the solutions found by others or to solitarily search for unexplored solutions. Social learning can better exploit learned information and decrease the costs of finding new resources, but excessive social learning can lead to over-exploitation and too little exploration for new solutions. We study how these two trade-offs interact to influence search efficiency in a model of collective foraging under conditions of varying resource abundance, resource density and group size. We modelled individual search strategies as Lévy walks, where a power-law exponent (μ) controlled the trade-off between exploitative and explorative movements in individual search. We modulated the trade-off between individual search and social learning using a selectivity parameter that determined how agents responded to social cues in terms of distance and likely opportunity costs. Our results show that social learning is favoured in rich and clustered environments, but also that the benefits of exploiting social information are maximized by engaging in high levels of individual exploration. We show that selective use of social information can modulate the disadvantages of excessive social learning, especially in larger groups and when individual exploration is limited. Finally, we found that the optimal combination of individual exploration and social learning gave rise to trajectories with μ ≈ 2 and provide support for the general optimality of such patterns in search. Our work sheds light on the interplay between individual search and social learning, and has broader implications for collective search and problem-solving.

A spatiotemporal mechanism of visual attention: Superdiffusive motion and theta oscillations of neural population activity patterns

Recent evidence has demonstrated that during visual spatial attention sampling, neural activity and behavioral performance exhibit large fluctuations. To understand the origin of these fluctuations and their functional role, here, we introduce a mechanism based on the dynamical activity pattern (attention spotlight) emerging from neural circuit models in the transition regime between different dynamical states. This attention activity pattern with rich spatiotemporal dynamics flexibly samples from different stimulus locations, explaining many key aspects of temporal fluctuations such as variable theta oscillations of visual spatial attention. Moreover, the mechanism expands our understanding of how visual attention exploits spatially complex fluctuations characterized by superdiffusive motion in space and makes experimentally testable predictions. We further illustrate that attention sampling based on such spatiotemporal fluctuations provides profound functional advantages such as adaptive switching between exploitation and exploration activities and is particularly efficient at sampling natural scenes with multiple salient objects.

Prediction in the aging brain: Merging cognitive, neurological, and evolutionary perspectives

Although the aging brain is typically characterized by declines in a variety of cognitive functions, there has been growing attention to cognitive functions that may stabilize or improve with age. We integrate evidence from behavioral, computational, and neurological domains under the hypothesis that over the life span the brain becomes more effective at predicting (i.e., utilizing knowledge) compared to learning. Moving beyond mere description of the empirical literature—with the aim of arriving at a deeper understanding of cognitive aging—we provide potential explanations for a learning-to-prediction shift based on evolutionary models and principles of senescence and plasticity. The proposed explanations explore whether the occurrence of a learning-to-prediction shift can be explained by (changes in) the fitness effects of learning and prediction over the life span. Prediction may optimize (a) the allocation of limited resources across the life span, and/or (b) late-life knowledge transfer (social learning). Alternatively, late-life prediction may reflect a slower decline in prediction compared to learning. By discussing these hypotheses, we aim to provide a foundation for an integrative neurocognitive–evolutionary perspective on aging and to stimulate further theoretical and empirical work.

Strength of Belief Guides Information Foraging

Theories of how people value and search for information share the assumption that beliefs give rise to the perceived value of information. However, few studies have directly addressed the pre-search processes that influence information-foraging behavior. This experiment examined the influence of pre-search belief updating on the perceived value of information sources. A sample of college students completed a hypothesis-testing, medical-diagnosis task. The experiment used medical tests with equal objective informative value before unveiling a presenting symptom intended to alter the strength of belief in different disease hypotheses. The observed patterns of test selection suggest that changes in beliefs about disease hypotheses result in systematic and predictable changes in test preference-a notion we refer to as the principle of hypothesis-guided search. We also present a simulation of how pre-search processes (e.g., hypothesis generation and working memory capacity) and task variables (e.g., time pressure) influence subsequent information search.

The conceptual building blocks of everyday thought: Tracking the emergence and dynamics of ruminative and nonruminative thinking

How do thoughts arise, unfold, and change over time? Are the contents and dynamics of everyday thought rooted in conceptual associations within one's semantic networks? To address these questions, we developed the Free Association Semantic task (FAST), whereby participants generate dynamic chains of conceptual associations in response to seed words that vary in valence. Ninety-four adults from a community sample completed the FAST task and additionally described and rated six of their most frequently occurring everyday thoughts. Text analysis and valence ratings revealed similarities in thematic and affective content between FAST concept chains and recurrent autobiographical thoughts. Dynamic analyses revealed that individuals higher in rumination were more strongly attracted to negative conceptual spaces and more likely to remain there longer. Overall, these findings provide quantitative evidence that conceptual associations may act as a semantic scaffold for more complex everyday thoughts, and that more negative and less dynamic conceptual associations in ruminative individuals mirror maladaptive repetitive thoughts in daily life. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Evolution of behavioural control from chordates to primates

This article outlines a hypothetical sequence of evolutionary innovations, along the lineage that produced humans, which extended behavioural control from simple feedback loops to sophisticated control of diverse species-typical actions. I begin with basic feedback mechanisms of ancient mobile animals and follow the major niche transitions from aquatic to terrestrial life, the retreat into nocturnality in early mammals, the transition to arboreal life and the return to diurnality. Along the way, I propose a sequence of elaboration and diversification of the behavioural repertoire and associated neuroanatomical substrates. This includes midbrain control of approach versus escape actions, telencephalic control of local versus long-range foraging, detection of affordances by the dorsal pallium, diversified control of nocturnal foraging in the mammalian neocortex and expansion of primate frontal, temporal and parietal cortex to support a wide variety of primate-specific behavioural strategies. The result is a proposed functional architecture consisting of parallel control systems, each dedicated to specifying the affordances for guiding particular species-typical actions, which compete against each other through a hierarchy of selection mechanisms.

This article is part of the theme issue ‘Systems neuroscience through the lens of evolutionary theory’.

Increased associative interference under high cognitive load

Associative processing is central for human cognition, perception and memory. But while associations often facilitate performance, processing irrelevant associations can interfere with performance, for example when learning new information. The aim of this study was to explore whether associative interference is influenced by contextual factors such as resources availability. Experiments 1–3 show that associative interference increases under high cognitive load. This result generalized to both long-term and short-term memory associations, and to both explicitly learned as well as incidentally learned associations in the linguistic and pictorial domains. Experiment 4 further revealed that attention to associative information can delay one’s perceptual processing when lacking resources. Taken together, when resources diminish associative interference increases, and additionally, processing novel and ambiguous information is hindered. These findings bare relevance to other domains as well (e.g., social, educational), in which increased load or stress may prompt an undesirable bias towards prior, misleading information.

The role of food odor in invertebrate foraging

Foraging for food is an integral part of animal survival. In small insects and invertebrates, multisensory information and optimized locomotion strategies are used to effectively forage in patchy and complex environments. Here, the importance of olfactory cues for effective invertebrate foraging is discussed in detail. We review how odors are used by foragers to move toward a likely food source and the recent models that describe this sensory-driven behavior. We argue that smell serves a second function by priming an organism for the efficient exploitation of food. By appraising food odors, invertebrates can establish preferences and better adapt to their ecological niches, thereby promoting survival. The smell of food pre-prepares the gastrointestinal system and primes feeding motor programs for more effective ingestion as well. Optimizing resource utilization affects longevity and reproduction as a result, leading to drastic changes in survival. We propose that models of foraging behavior should include odor priming, and illustrate this with a simple toy model based on the marginal value theorem. Lastly, we discuss the novel techniques and assays in invertebrate research that could investigate the interactions between odor sensing and food intake. Overall, the sense of smell is indispensable for efficient foraging and influences not only locomotion, but also organismal physiology, which should be reflected in behavioral modeling.

Tonal Emergence: An agent-based model of tonal coordination

Humans have a remarkable capacity for coordination. Our ability to interact and act jointly in groups is crucial to our success as a species. Joint Action (JA) research has often concerned itself with simplistic behaviors in highly constrained laboratory tasks. But there has been a growing interest in understanding complex coordination in more open-ended contexts. In this regard, collective music improvisation has emerged as a fascinating model domain for studying basic JA mechanisms in an unconstrained and highly sophisticated setting. A number of empirical studies have begun to elucidate coordination mechanisms underlying joint musical improvisation, but these findings have yet to be cached out in a working computational model. The present work fills this gap by presenting Tonal Emergence, an idealized agent-based model of improvised musical coordination. Tonal Emergence models the coordination of notes played by improvisers to generate harmony (i.e., tonality), by simulating agents that stochastically generate notes biased towards maximizing harmonic consonance given their partner's previous notes. The model replicates an interesting empirical result from a previous study of professional jazz pianists: feedback loops of mutual adaptation between interacting agents support the production of consonant harmony. The model is further explored to show how complex tonal dynamics, such as the production and dissolution of stable tonal centers, are supported by agents that are characterized by (i) a tendency to strive toward consonance, (ii) stochasticity, and (iii) a limited memory for previously played notes. Tonal Emergence thus provides a grounded computational model to simulate and probe the coordination mechanisms underpinning one of the more remarkable feats of human cognition: collective music improvisation.

Naïve individuals promote collective exploration in homing pigeons

Group-living animals that rely on stable foraging or migratory routes can develop behavioural traditions to pass route information down to inexperienced individuals. Striking a balance between exploitation of social information and exploration for better alternatives is essential to prevent the spread of maladaptive traditions. We investigated this balance during cumulative route development in the homing pigeon Columba livia. We quantified information transfer within pairs of birds in a transmission-chain experiment and determined how birds with different levels of experience contributed to the exploration-exploitation trade-off. Newly introduced naïve individuals were initially more likely to initiate exploration than experienced birds, but the pair soon settled into a pattern of alternating leadership with both birds contributing equally. Experimental pairs showed an oscillating pattern of exploration over generations that might facilitate the discovery of more efficient routes. Our results introduce a new perspective on the roles of leadership and information pooling in the context of collective learning.

From exploration to exploitation: a shifting mental mode in late life development

Changes in cognition, affect, and brain function combine to promote a shift in the nature of mentation in older adulthood, favoring exploitation of prior knowledge over exploratory search as the starting point for thought and action. Age-related exploitation biases result from the accumulation of prior knowledge, reduced cognitive control, and a shift toward affective goals. These are accompanied by changes in cortical networks, as well as attention and reward circuits. By incorporating these factors into a unified account, the exploration-to-exploitation shift offers an integrative model of cognitive, affective, and brain aging. Here, we review evidence for this model, identify determinants and consequences, and survey the challenges and opportunities posed by an exploitation-biased mental mode in later life.

Characterization of exploratory patterns and hippocampal-prefrontal network oscillations during the emergence of free exploration

During free exploration, the emergence of patterned and sequential behavioral responses to an unknown environment reflects exploration traits and adaptation. However, the behavioral dynamics and neural substrates underlying the exploratory behavior remain poorly understood. We developed computational tools to quantify the exploratory behavior and performed in vivo electrophysiological recordings in a large arena in which mice made sequential excursions into unknown territory. Occupancy entropy was calculated to characterize the cumulative and moment-to-moment behavioral dynamics in explored and unexplored territories. Local field potential analysis revealed that the theta activity in the dorsal hippocampus (dHPC) was highly correlated with the occupancy entropy. Individual dHPC and prefrontal cortex (PFC) oscillatory activities could classify various aspects of free exploration. Initiation of exploration was accompanied by a coordinated decrease and increase in theta activity in PFC and dHPC, respectively. Our results indicate that dHPC and PFC work synergistically in shaping free exploration by modulating exploratory traits during emergence and visits to an unknown environment.

On the Effect of Practice on Exploration and Exploitation of Options and Strategies

Insufficient exploration of one's surroundings is at the root of many real-life problems, as demonstrated by many famous biases (e.g., the status quo bias, learned helplessness). The current work focuses on the emergence of this phenomenon at the strategy level: the tendency to under-explore the set of available choice strategies. We demonstrate that insufficient exploration of strategies can also manifest as excessive exploration between options. In such cases, interventions aimed at improving choices by reducing the costs of exploration of options are likely to fail. In Study 1, participants faced an exploration task that implies an infinite number of choice strategies and a small sub-set of (near) optimal solutions. We manipulated the amount of practice participants underwent during the first, shorter game and compared their performance in a second, longer game with an identical payoff structure. Our results show that regardless of the amount of practice, participants in all experimental groups tended to under-explore the strategy space and relied on a specific strategy that implied over-exploration of the option space. That is, under-exploration of strategies was manifested as over-exploration of options. In Study 2, we added a constraint that, on a subset of practice trials, forced participants to exploit familiar options. This manipulation almost doubled the per-trial average outcome on the comparable longer second game. This suggests that forcing participants to experience the effects of different (underexplored) strategy components during practice can greatly increase the chance they make better choices later on.

The past, present, and future of selection history

The last ten years of attention research have witnessed a revolution, replacing a theoretical dichotomy (top-down vs. bottom-up control) with a trichotomy (biased by current goals, physical salience, and selection history). This third new mechanism of attentional control, selection history, is multifaceted. Some aspects of selection history must be learned over time whereas others reflect much more transient influences. A variety of different learning experiences can shape the attention system, including reward, aversive outcomes, past experience searching for a target, target‒non-target relations, and more. In this review, we provide an overview of the historical forces that led to the proposal of selection history as a distinct mechanism of attentional control. We then propose a formal definition of selection history, with concrete criteria, and identify different components of experience-driven attention that fit within this definition. The bulk of the review is devoted to exploring how these different components relate to one another. We conclude by proposing an integrative account of selection history centered on underlying themes that emerge from our review.

Children are more exploratory and learn more than adults in an approach-avoid task

Intuitively, children appear to be more exploratory than adults, and this exploration seems to help children learn,. However, there have been few clear tests of these ideas. We test whether exploration and learning change across development using a task that presents a "learning trap." In this task, exploitation-maximizing immediate reward and avoiding costs-may lead the learner to draw incorrect conclusions, while exploration may lead to better learning but be more costly. In Studies 1, 2, and 3 we find that preschoolers and early school-aged children explore more than adults and learn the true structure of the environment better. Study 3 demonstrates that children explore more than adults even though they, like adults, predict that exploration will be costly, and it shows that exploration and learning are correlated. Study 4 shows that children's and adults' learning depends on the evidence they generate during exploration: children exposed to adult-like evidence learn like adults, and adults exposed to child-like evidence learn like children. Together, these studies support the idea that children may be more exploratory than adults, and this increased exploration influences learning.

The influence of exposure to randomness on lateral thinking in divergent, convergent, and creative search

Exposure to random stimuli has often been suggested to help unlock problem-solving abilities and creativity, helping us to see problems differently and imagine new possibilities. Equally, randomness is widely used in computer science to escape local maxima and find effective solutions to intractable problems. However, randomness has rarely been used as a formal aid in human decision making or investigated in controlled experimental settings. In this pre-registered study, we tested the effect of extraneous random stimuli using Wikipedia's random page generator on 592 British participants' performance across three online tasks: one 'convergent' forecasting task and two 'divergent' fluency tasks. We found no improvement associated with the treatment and often significant impairment. A Bayesian meta-analysis of the tasks finds strong support for the null hypothesis. We conclude that stimulating lateral thinking through random stimuli is non-trivial and may require such stimuli to be sufficiently task-related or 'optimally random'.

Sources of variation in search and foraging: A theoretical perspective

Search-the problem of exploring a space of alternatives to identify target goals-is a fundamental behaviour for many species. Although its foundation lies in foraging, most studies of human search behaviour have been directed towards understanding the attentional mechanisms that underlie the efficient visual exploration of two-dimensional (2D) scenes. With this review, we aim to characterise how search behaviour can be explained across a wide range of contexts, environments, spatial scales, and populations, both typical and atypical. We first consider the generality of search processes across psychological domains. We then review studies of interspecies differences in search. Finally, we explore in detail the individual and contextual variables that affect visual search and related behaviours in established experimental psychology paradigms. Despite the heterogeneity of the findings discussed, we identify that variations in control processes, along with the ability to regulate behaviour as a function of the structure of search space and the sampling processes adopted, to be central to explanations of variations in search behaviour. We propose a tentative theoretical model aimed at integrating these notions and close by exploring questions that remain unaddressed.

Detours increase local knowledge-Exploring the hidden benefits of self-control failure

Self-control enables people to override momentary thoughts, emotions, or impulses in order to pursue long-term goals. Good self-control is a predictor for health, success, and subjective well-being, as bad self-control is for the opposite. Therefore, the question arises why evolution has not endowed us with perfect self-control. In this article, we draw some attention to the hidden benefits of self-control failure and present a new experimental paradigm that captures both costs and benefits of self-control failure. In an experiment, participants worked on three consecutive tasks: 1) In a transcription task, we manipulated how much effortful self-control two groups of participants had to exert. 2) In a number-comparison task, participants of both groups were asked to compare numbers and ignore distracting neutral versus reward-related pictures. 3) After a pause for recreation, participants were confronted with an unannounced recognition task measuring whether they had incidentally encoded the distracting pictures during the previous number-comparison task. The results showed that participants who exerted a high amount of effortful self-control during the first task shifted their priorities and attention toward the distractors during the second self-control demanding task: The cost of self-control failure was reflected in worse performance in the number-comparison task. Moreover, the group which had exerted a high amount of self-control during the first task and showed self-control failure during the second task was better in the unannounced third task. The benefit of self-control failure during number comparison was reflected in better performance during the recognition task. However, costs and benefits were not specific for reward-related distractors but also occurred with neutral pictures. We propose that the hidden benefit of self-control failure lies in the exploration of distractors present during goal pursuit, i.e. the collection of information about the environment and the potential discovery of new sources of reward. Detours increase local knowledge.

Network analysis of intra- and interspecific freshwater fish interactions using year-around tracking

A long-term, yet detailed view into the social patterns of aquatic animals has been elusive. With advances in reality mining tracking technologies, a proximity-based social network (PBSN) can capture detailed spatio-temporal underwater interactions. We collected and analysed a large dataset of 108 freshwater fish from four species, tracked every few seconds over 1 year in their natural environment. We calculated the clustering coefficient of minute-by-minute PBSNs to measure social interactions, which can happen among fish sharing resources or habitat preferences (positive/neutral interactions) or in predator and prey during foraging interactions (agonistic interactions). A statistically significant coefficient compared to an equivalent random network suggests interactions, while a significant aggregated clustering across PBSNs indicates prolonged, purposeful social behaviour. Carp (Cyprinus carpio) displayed within- and among-species interactions, especially during the day and in the winter, while tench (Tinca tinca) and catfish (Silurus glanis) were solitary. Perch (Perca fluviatilis) did not exhibit significant social behaviour (except in autumn) despite being usually described as a predator using social facilitation to increase prey intake. Our work illustrates how methods for building a PBSN can affect the network's structure and highlights challenges (e.g. missing signals, different burst frequencies) in deriving a PBSN from reality mining technologies.

Biosociological ethodiversity in the social system

A comprehensive understanding of human sociality needs to embrace the coevolution of genes and culture. Recent advances in biological research about niche construction by organisms, and the development of the concepts of social niche and ethodiversity, can be integrated into a common approach to understand this coevolution, which implies the interaction between sociology and ecology in an integrative framework of knowledge. In this paper the authors propose such inclusive biosociological and heuristic framework to improve the understanding of the evolution of social niche construction. In addition, it allows a better understanding of the concept of sociotype in non-human organisms and explains some aspects of the social or presocial behavior through the concept of ethodiversity.

Potential-driven random walks on interconnected systems

Interconnected systems have to route information to function properly: At the lowest scale neural cells exchange electrochemical signals to communicate, while at larger scales animals and humans move between distinct spatial patches and machines exchange information via the Internet through communication protocols. Nontrivial patterns emerge from the analysis of information flows, which are not captured either by broadcasting, such as in random walks, or by geodesic routing, such as shortest paths. In fact, alternative models between those extreme protocols are still eluding us. Here we propose a class of stochastic processes, based on biased random walks, where agents are driven by a physical potential pervading the underlying network topology. By considering a generalized Coulomb dependence on the distance on destination(s), we show that it is possible to interpolate between random walk and geodesic routing in a simple and effective way. We demonstrate that it is not possible to find a one-size-fit-all solution to efficient navigation and that network heterogeneity or modularity has measurable effects. We illustrate how our framework can describe the movements of animals and humans, capturing with a stylized model some measurable features of the latter. From a methodological perspective, our potential-driven random walks open the doors to a broad spectrum of analytical tools, ranging from random-walk centralities to geometry induced by potential-driven network processes.

The role of great ape behavioral ecology in One Health: Implications for captive welfare and re-habilitation success

Behavior is the interface through which animals interact with their environments, and therefore has potentially cascading impacts on the health of individuals, populations, their habitats, and the humans that share them. Evolution has shaped the interaction between species and their environments. Thus, alterations to the species-typical "wild-type" behavioral repertoire (and the ability of the individual to adapt flexibly which elements of the repertoire it employs) may disrupt the relationship between the organism and its environment, creating cascading One Health effects. A good example is rehabilitant orangutans where, for example, seemingly minor differences from wild conspecifics in the time spent traveling on the ground rather than in the forest canopy can affect an individual's musculoskeletal and nutritional health, as well as social integration. It can also increase two-way transmission of infectious diseases and/or pathogens with local human populations, or potentially with neighboring wild populations if there are no geographical barriers and rehabilitants travel far enough to leave their release area. Primates are well known ecosystem engineers, reshaping plant communities and maintaining biodiversity through seed dispersal, consuming plants, and creating canopy gaps and trails. From the habitat perspective, a rehabilitant orangutan which does not behave like a wild orangutan is unlikely to fulfill these same ecosystem services. Despite the importance of the diversity of an ape's behavioral repertoire, how it compares to that of wild conspecifics and how it alters in response to habitat variation, behavior is an often under-appreciated aspect of One Health. In this review, focusing on orangutans as an example of the kinds of problems faced by all captive great apes, we examine the ways in which understanding and facilitating the expression of wild-type behavior can improve their health, their ability to thrive, and the robustness of local One Health systems.

Simple Threshold Rules Solve Explore/Exploit Trade-offs in a Resource Accumulation Search Task

How, and how well, do people switch between exploration and exploitation to search for and accumulate resources? We study the decision processes underlying such exploration/exploitation trade-offs using a novel card selection task that captures the common situation of searching among multiple resources (e.g., jobs) that can be exploited without depleting. With experience, participants learn to switch appropriately between exploration and exploitation and approach optimal performance. We model participants' behavior on this task with random, threshold, and sampling strategies, and find that a linear decreasing threshold rule best fits participants' results. Further evidence that participants use decreasing threshold-based strategies comes from reaction time differences between exploration and exploitation; however, participants themselves report non-decreasing thresholds. Decreasing threshold strategies that "front-load" exploration and switch quickly to exploitation are particularly effective in resource accumulation tasks, in contrast to optimal stopping problems like the Secretary Problem requiring longer exploration.

Priming exploration across domains: does search in a spatial environment influence search in a cognitive environment?

Is there a general tendency to explore that connects search behaviour across different domains? Although the experimental evidence collected so far suggests an affirmative answer, this fundamental question about human behaviour remains open. A feasible way to test the domain-generality hypothesis is that of testing the so-called priming hypothesis: priming explorative behaviour in one domain should subsequently influence explorative behaviour in another domain. However, only a limited number of studies have experimentally tested this priming hypothesis, and the evidence is mixed. We tested the priming hypothesis in a registered report. We manipulated explorative behaviour in a spatial search task by randomly allocating people to search environments with resources that were either clustered together or dispersedly distributed. We hypothesized that, in a subsequent anagram task, participants who searched in clustered spatial environments would search for words in a more clustered way than participants who searched in the dispersed spatial environments. The pre-registered hypothesis was not supported. An equivalence test showed that the difference between conditions was smaller than the smallest effect size of interest (d = 0.36). Out of several exploratory analyses, we found only one inferential result in favour of priming. We discuss implications of these findings for the theory and propose future tests of the hypothesis.

Active Inferants: An Active Inference Framework for Ant Colony Behavior

In this paper, we introduce an active inference model of ant colony foraging behavior, and implement the model in a series of in silico experiments. Active inference is a multiscale approach to behavioral modeling that is being applied across settings in theoretical biology and ethology. The ant colony is a classic case system in the function of distributed systems in terms of stigmergic decision-making and information sharing. Here we specify and simulate a Markov decision process (MDP) model for ant colony foraging. We investigate a well-known paradigm from laboratory ant colony behavioral experiments, the alternating T-maze paradigm, to illustrate the ability of the model to recover basic colony phenomena such as trail formation after food location discovery. We conclude by outlining how the active inference ant colony foraging behavioral model can be extended and situated within a nested multiscale framework and systems approaches to biology more generally.

Flexible modulation of sequence generation in the entorhinal-hippocampal system

Exploration, consolidation and planning depend on the generation of sequential state representations. However, these algorithms require disparate forms of sampling dynamics for optimal performance. We theorize how the brain should adapt internally generated sequences for particular cognitive functions and propose a neural mechanism by which this may be accomplished within the entorhinal-hippocampal circuit. Specifically, we demonstrate that the systematic modulation along the medial entorhinal cortex dorsoventral axis of grid population input into the hippocampus facilitates a flexible generative process that can interpolate between qualitatively distinct regimes of sequential hippocampal reactivations. By relating the emergent hippocampal activity patterns drawn from our model to empirical data, we explain and reconcile a diversity of recently observed, but apparently unrelated, phenomena such as generative cycling, diffusive hippocampal reactivations and jumping trajectory events.

"Is it Reasonable to Study Decision-Making Quantitatively?"

Scientists studying decision-making often provide a set of choices, each specified with values or distributions of values, and probabilities or distributions of probabilities. For example, "Would you prefer $100 with probability 1.0 or $1 with probability .9 and $1,000 with probability 0.1?" Other decision research examines choices made in the absence of most quantitative information; for example, "Would you prefer a Ford now or a Porsche a year from now?," "Which food would you prefer," but models the findings with precise quantitative assumptions. Yet other research does neither; for example, modeling verbally stated choices with verbally stated heuristics. This article asks about the relevance of the first two research approaches for much of the decision-making made in life. The use of quantitative research and modeling is unsurprising, given that this approach underlies most of science. In life, values and probabilities are almost always partly or wholly vague and qualitative rather than quantitative. For example, when deciding which house to buy, there are relevant features such as size, color, neighborhood schools, construction materials, attractiveness, and many more, but the decision-maker finds it difficult and of little use to assign these precise values or weights. Nonetheless, humans have evolved to make decisions in such vaguely specified settings. I provide an example showing how a very high degree of uncertainty can defeat the application of quantitative decision-making, but such a demonstration is not critical if quantitative research and modeling produce a good understanding of and a good approximation to decision-making in the natural environment. This perspective addresses these issues.

The Predation Game: Does dividing attention affect patterns of human foraging?

Attention is known to play an important role in shaping the behaviour of both human and animal foragers. Here, in three experiments, we built on previous interactive tasks to create an online foraging game for studying divided attention in human participants exposed to the (simulated) risk of predation. Participants used a "sheep" icon to collect items from different target categories randomly distributed across the display. Each trial also contained "wolf" objects, whose movement was inspired by classic studies of multiple object tracking. When participants needed to physically avoid the wolves, foraging patterns changed, with an increased tendency to switch between target categories and a decreased ability to prioritise high reward targets, relative to participants who could safely ignore them. However, when the wolves became dangerous by periodically changing form (briefly having big eyes) instead of by approaching the sheep, foraging patterns were unaffected. Spatial disruption caused by the need to rapidly shift position-rather the cost of reallocating attention-therefore appears to influence foraging in this context. These results thus confirm that participants can efficiently alternate between target selection and tracking moving objects, replicating earlier single-target search findings. Future studies may need to increase the perceived risk or potential costs associated with simulated danger, in order to elicit the extended run behaviour predicted by animal models of foraging, but absent in the current data.

Two variations and one similarity in memory functions deployed by mice and humans to support foraging

Assessing variations in cognitive function between humans and animals is vital for understanding the idiosyncrasies of human cognition and for refining animal models of human brain function and disease. We determined memory functions deployed by mice and humans to support foraging with a search task acting as a test battery. Mice searched for food from the top of poles within an open arena. Poles were divided into groups based on visual cues and baited according to different schedules. White and black poles were baited in alternate trials. Striped poles were never baited. The requirement of the task was to find all baits in each trial. Mice's foraging efficiency, defined as the number of poles visited before all baits were retrieved, improved with practice. Mice learnt to avoid visiting unbaited poles across trials (long-term memory) and revisits to poles within each trial (working memory). Humans tested with a virtual reality version of the task outperformed mice in foraging efficiency, working memory, and exploitation of the temporal pattern of rewards across trials. Moreover, humans, but not mice, reduced the number of possible movement sequences used to search the set of poles. For these measures, interspecies differences were maintained throughout the 3 weeks of testing. By contrast, long-term memory for never-rewarded poles was similar in mice and humans after the first week of testing. These results indicate that human cognitive functions relying on archaic brain structures may be adequately modelled in mice. Conversely, modelling in mice fluid skills likely to have developed specifically in primates requires caution.

Balancing exploration and exploitation with information and randomization

Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.