Temporal discounting correlates with directed exploration but not with random exploration

Meta-Analysis Reveals That Explore-Exploit Decisions are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex and the Anterior Cingulate Cortex

Explore-exploit research has challenges in generalizability due to a limited theoretical basis of exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions use an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N=23 studies) where we found activation in the dorsal lateral prefrontal cortex and anterior cingulate cortex during exploration versus exploitation, providing some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the dorsal anterior cingulate cortex, dorsal medial prefrontal cortex, and anterior insula, suggesting that these brain regions do not engage in opponent processing. Further, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation in exploration and exploitation decisions is generally more similar than it is different suggests there remain significant challenges toward characterizing explore-exploit decision making. Nonetheless, dlPFC and ACC activation differentiate explore and exploit decisions and identifying these responses can help in targeted interventions aimed at manipulating these decisions.

Differential impacts of healthy cognitive aging on directed and random exploration

Deciding whether to explore unknown opportunities or exploit well-known options is a ubiquitous part of our everyday lives. Extensive work in college students suggests that young people make explore-exploit decisions using a mixture of information seeking and random behavioral variability. Whether, and to what extent, older adults use the same strategies is unknown. To address this question, 51 older adults (ages 65-74) and 32 younger adults (ages 18-25) completed the Horizon Task, a gambling task that quantifies information seeking and behavioral variability as well as how these strategies are controlled for the purposes of exploration. Qualitatively, we found that older adults performed similar to younger adults on this task, increasing both their information seeking and behavioral variability when it was adaptive to explore. Quantitively, however, there were substantial differences between the age groups, with older adults showing less information seeking overall and less reliance on variability as a means to explore. In addition, we found a subset of approximately 26% of older adults whose information seeking was close to zero, avoiding informative options even when they were clearly the better choice. Unsurprisingly, these "information avoiders" performed worse on the task. In contrast, task performance in the remaining "information seeking" older adults was comparable to that of younger adults suggesting that age-related differences in explore-exploit decision making may be adaptive except when they are taken to extremes. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Changes in adaptation to time horizons across development

When making decisions, the amount of time remaining matters. When time horizons are long, exploring unknown options can inform later decisions, but when time horizons are short, exploiting known options should be prioritized. While adults and adolescents adapt their exploration in this way, it is unclear when such adaptation emerges and how individuals behave when time horizons are ambiguous, as in many real-life situations. We examined these questions by having 5- to 6-year-olds (N = 43), 11- to 12-year-olds (N = 40), and adult college students (N = 49) in the United States complete a Simplified Horizons Task under short, long, and ambiguous time horizons. Adaptation to time horizons increased with age: older children and adults explored more when horizons were long than when short, and while some younger children adapted to time horizons, younger children overall did not show strong evidence of adapting. Under ambiguous horizons, older children and adults preferred to exploit over explore, while younger children did not show this preference. Thus, adaptation to time horizons is evident by ages 11-12 and may begin to emerge around 5-6 years, and children decrease their tendencies to explore under short and ambiguous time horizons with development. This developmental shift may lead to less learning but more adaptive decision making. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Instructed motivational states bias reinforcement learning and memory formation

Motivation influences goals, decisions, and memory formation. Imperative motivation links urgent goals to actions, narrowing the focus of attention and memory. Conversely, interrogative motivation integrates goals over time and space, supporting rich memory encoding for flexible future use. We manipulated motivational states via cover stories for a reinforcement learning task: The imperative group imagined executing a museum heist, whereas the interrogative group imagined planning a future heist. Participants repeatedly chose among four doors, representing different museum rooms, to sample trial-unique paintings with variable rewards (later converted to bonus payments). The next day, participants performed a surprise memory test. Crucially, only the cover stories differed between the imperative and interrogative groups; the reinforcement learning task was identical, and all participants had the same expectations about how and when bonus payments would be awarded. In an initial sample and a preregistered replication, we demonstrated that imperative motivation increased exploitation during reinforcement learning. Conversely, interrogative motivation increased directed (but not random) exploration, despite the cost to participants' earnings. At test, the interrogative group was more accurate at recognizing paintings and recalling associated values. In the interrogative group, higher value paintings were more likely to be remembered; imperative motivation disrupted this effect of reward modulating memory. Overall, we demonstrate that a prelearning motivational manipulation can bias learning and memory, bearing implications for education, behavior change, clinical interventions, and communication.

Exploratory preferences explain the human fascination for imaginary worlds in fictional stories

Imaginary worlds are present and often central in many of the most culturally successful modern narrative fictions, be it in novels (e.g., Harry Potter), movies (e.g., Star Wars), video games (e.g., The Legend of Zelda), graphic novels (e.g., One Piece) and TV series (e.g., Game of Thrones). We propose that imaginary worlds are popular because they activate exploratory preferences that evolved to help us navigate the real world and find new fitness-relevant information. Therefore, we hypothesize that the attraction to imaginary worlds is intrinsically linked to the desire to explore novel environments and that both are influenced by the same underlying factors. Notably, the inter-individual and cross-cultural variability of the preference for imaginary worlds should follow the inter-individual and cross-cultural variability of exploratory preferences (with the personality trait Openness-to-experience, age, sex, and ecological conditions). We test these predictions with both experimental and computational methods. For experimental tests, we run a pre-registered online experiment about movie preferences (N = 230). For computational tests, we leverage two large cultural datasets, namely the Internet Movie Database (N = 9424 movies) and the Movie Personality Dataset (N = 3.5 million participants), and use machine-learning algorithms (i.e., random forest and topic modeling). In all, consistent with how the human preference for spatial exploration adaptively varies, we provide empirical evidence that imaginary worlds appeal more to more explorative people, people higher in Openness-to-experience, younger individuals, males, and individuals living in more affluent environments. We discuss the implications of these findings for our understanding of the cultural evolution of narrative fiction and, more broadly, the evolution of human exploratory preferences.

Sequential delay and probability discounting tasks in mice reveal anchoring effects partially attributable to decision noise

Delay discounting and probability discounting decision making tasks in rodent models have high translational potential. However, it is unclear whether the discounted value of the large reward option is the main contributor to variability in animals' choices in either task, which may limit translation to humans. Male and female mice underwent sessions of delay and probability discounting in sequence to assess how choice behavior adapts over experience with each task. To control for "anchoring" (persistent choices based on the initial delay or probability), mice experienced "Worsening" schedules where the large reward was offered under initially favorable conditions that became less favorable during testing, followed by "Improving" schedules where the large reward was offered under initially unfavorable conditions that improved over a session. During delay discounting, both male and female mice showed elimination of anchoring effects over training. In probability discounting, both sexes of mice continued to show some anchoring even after months of training. One possibility is that "noisy", exploratory choices could contribute to these persistent anchoring effects, rather than constant fluctuations in value discounting. We fit choice behavior in individual animals using models that included both a value-based discounting parameter and a decision noise parameter that captured variability in choices deviating from value maximization. Changes in anchoring behavior over time were tracked by changes in both the value and decision noise parameters in delay discounting, but by the decision noise parameter in probability discounting. Exploratory decision making was also reflected in choice response times that tracked the degree of conflict caused by both uncertainty and temporal cost, but was not linked with differences in locomotor activity reflecting chamber exploration. Thus, variable discounting behavior in mice can result from changes in exploration of the decision options rather than changes in reward valuation.

Value-free random exploration is linked to impulsivity

Deciding whether to forgo a good choice in favour of exploring a potentially more rewarding alternative is one of the most challenging arbitrations both in human reasoning and in artificial intelligence. Humans show substantial variability in their exploration, and theoretical (but only limited empirical) work has suggested that excessive exploration is a critical mechanism underlying the psychiatric dimension of impulsivity. In this registered report, we put these theories to test using large online samples, dimensional analyses, and computational modelling. Capitalising on recent advances in disentangling distinct human exploration strategies, we not only demonstrate that impulsivity is associated with a specific form of exploration—value-free random exploration—but also explore links between exploration and other psychiatric dimensions.

The Stage 1 protocol for this Registered Report was accepted in principle on 19/03/2021. The protocol, as accepted by the journal, can be found at 10.6084/m9.figshare.14346506.v1.

Deciding between known rewarding options and exploring novel avenues is central to decision making. Humans show variability in their exploration. Here, the authors show that impulsivity is associated to an increased usage of a cognitively cheap (and sometimes sub-optimal) exploration strategy.

Insights about the common generative rule underlying an information foraging task can be facilitated via collective search

Social learning is beneficial for efficient information search in unfamiliar environments ("within-task" learning). In the real world, however, possible search spaces are often so large that decision makers are incapable of covering all options, even if they pool their information collectively. One strategy to handle such overload is developing generalizable knowledge that extends to multiple related environments ("across-task" learning). However, it is unknown whether and how social information may facilitate such across-task learning. Here, we investigated participants' social learning processes across multiple laboratory foraging sessions in spatially correlated reward landscapes that were generated according to a common rule. The results showed that paired participants were able to improve efficiency in information search across sessions more than solo participants. Computational analysis of participants' choice-behaviors revealed that such improvement across sessions was related to better understanding of the common generative rule. Rule understanding was correlated within a pair, suggesting that social interaction is a key to the improvement of across-task learning.

Individuals with adverse childhood experiences explore less and underweight reward feedback

Adverse childhood experiences (ACEs) are extreme stressors that lead to negative psychosocial outcomes in adulthood. Nonhuman animals explore less after exposure to early stress. Therefore, in this preregistered study, we hypothesized that reduced exploration following ACEs would also be evident in human adults. Further, we predicted that adults with ACEs, in a foraging task, would adopt a decision-making policy that relies on the most-recent reward feedback, a rational strategy for unstable environments. We analyzed data from 145 adult participants, 47 with four or more ACEs and 98 with fewer than four ACEs. In the foraging task, participants evaluated the trade-off between exploiting a known patch with diminishing rewards and exploring a novel one with a fresh distribution of rewards. Using computational modeling, we quantified the degree to which participants' decisions weighted recent feedback. As predicted, participants with ACEs explored less. However, contrary to our hypothesis, they underweighted recent feedback. These unexpected findings indicate that early adversity may dampen reward sensitivity. Our results may help to identify cognitive mechanisms that link childhood trauma to the onset of psychopathology.

Dynamic decision policy reconfiguration under outcome uncertainty

In uncertain or unstable environments, sometimes the best decision is to change your mind. To shed light on this flexibility, we evaluated how the underlying decision policy adapts when the most rewarding action changes. Human participants performed a dynamic two-armed bandit task that manipulated the certainty in relative reward (conflict) and the reliability of action-outcomes (volatility). Continuous estimates of conflict and volatility contributed to shifts in exploratory states by changing both the rate of evidence accumulation (drift rate) and the amount of evidence needed to make a decision (boundary height), respectively. At the trialwise level, following a switch in the optimal choice, the drift rate plummets and the boundary height weakly spikes, leading to a slow exploratory state. We find that the drift rate drives most of this response, with an unreliable contribution of boundary height across experiments. Surprisingly, we find no evidence that pupillary responses associated with decision policy changes. We conclude that humans show a stereotypical shift in their decision policies in response to environmental changes.

From exploration to exploitation: a shifting mental mode in late life development

Changes in cognition, affect, and brain function combine to promote a shift in the nature of mentation in older adulthood, favoring exploitation of prior knowledge over exploratory search as the starting point for thought and action. Age-related exploitation biases result from the accumulation of prior knowledge, reduced cognitive control, and a shift toward affective goals. These are accompanied by changes in cortical networks, as well as attention and reward circuits. By incorporating these factors into a unified account, the exploration-to-exploitation shift offers an integrative model of cognitive, affective, and brain aging. Here, we review evidence for this model, identify determinants and consequences, and survey the challenges and opportunities posed by an exploitation-biased mental mode in later life.

On the Psychology of the Psychology Subject Pool

Abstract. Many psychology researches are performed through “psychology subject pools” which give participants considerable flexibility when they participate. This “participant degree-of-freedom” has led to concern that the characteristics of subject pool participants may change with time, with the most engaged students signing up at the start of the semester and the least engaged students leaving it all to the end. In this paper, we performed an exploratory analysis to look for evidence of this “good student effect.” Consistent with previous work, we find support for the good student effect with earlier participants scoring higher on the Big-Five subscales of Achievement-Striving and Cooperation, as well as Grit and Empathic-Concern. In addition, we found a non-linear effect of time-of-semester on Sensation-Seeking, with this measure peaking in the middle of the semester as well as the end. However, the vast majority of the measures we tested, including measures of personality, cognition, decision-making, and social interaction, did not correlate with time-of-semester or time-of-day at all. Thus, we conclude that, while some studies directly related to measures of Grit and Sensation-Seeking would do well to recruit throughout the semester, in most cases any bias introduced by the good student effect is likely to be small.