Searching for Rewards Like a Child Means Less Generalization and More Directed Exploration

Play in Cognitive Development: From Rational Constructivism to Predictive Processing

It is widely believed that play and curiosity are key ingredients as children develop models of the world. There is also an emerging consensus that children are Bayesian learners who combine their structured prior beliefs with estimations of the likelihood of new evidence to infer the most probable model of the world. An influential school of thought within developmental psychology, rational constructivism, combines these two ideas to propose that children learn intuitive theories of how the world works in part by engaging in play activities that allow them to gather new information for testing their theories. There are still, however, at least two pieces missing from rational constructivist theories of development. First, rational constructivism has so far devoted little attention to explaining why children's preferred form of learning, play, feels so fun, enjoyable, and rewarding. Rational constructivism may suggest that children are curious and like to play because reducing uncertainty and learning better theories of the causal workings of the world is enjoyable. What remains unclear, however, is why reducing uncertainty in play is interesting, fun, and joyful, while doing so in other forms of learning can be frustrating or boring. Second, rational constructivism may have overlooked how children, during play, will take control of and manipulate their environment, sometimes in an effort to create ideal niches for surprise-extraction, sometimes for developing strategies for making the world fit with their predictions. These missing elements from rational constructivism can be provided by understanding the contribution of play to development in terms of predictive processing, an influential framework in cognitive neuroscience that models many of the brain's cognitive functions as processes of model-based, probabilistic prediction.

Exploration, Distributed Attention, and Development of Category Learning

Category learning is a crucial aspect of cognition that involves organizing entities into equivalence classes. Whereas adults tend to focus on category-relevant features, young children often distribute attention between relevant and irrelevant ones. The reasons for children's distributed attention are not fully understood. In two category-learning experiments with adults and with children aged 4, 5, and 6 (N = 201), we examined potential drivers of distributed attention, including (a) immature filtering of distractors and (b) the general tendency for exploration or broad information sampling. By eliminating distractor competition, we reduced filtering demands. Despite identifying the features critical for accurate categorization, children, regardless of their categorization performance, continued sampling more information than was necessary. These results indicate that the tendency to sample information extensively contributes to distributed attention in young children. We identify candidate drivers of this tendency that need to be examined in future research.

Balancing exploration and exploitation? The impact of cost and inhibitory control on information gathering in early childhood

The information-seeking behaviour of adults focuses on optimizing the gathering and utilizing information to minimize search costs. In contrast, children tend to engage in information search during decision-making with less consideration for costs. This difference in behaviour is believed to be linked to the development of executive functions. Therefore, the present study aimed to investigate the relationship between executive function and cost-related information-gathering behaviour. We assessed 56 children aged 4-6 years, involving three tasks: an information-gathering task, an inhibitory control and a working memory task. In the information-gathering task, children participated in both non-cost and cost conditions, where they were given the opportunity to freely gather information or incur a cost to acquire information. The findings revealed that children with higher inhibitory control tended to gather less information when a cost was involved. This highlights the important role of inhibitory control in shaping information-seeking behaviour in early childhood.

Protocol and preregistration for the CODEC project: measuring, modelling and mechanistically understanding the nature of cognitive variability in early childhood

BACKGROUND

Children's cognitive performance fluctuates across multiple timescales. However, fluctuations have often been neglected in favour of research into average cognitive performance, limiting the unique insights into cognitive abilities and development that cognitive variability may afford. Preliminary evidence suggests that greater variability is associated with increased symptoms of neurodevelopmental disorders, and differences in behavioural and neural functioning. The relative dearth of empirical work on variability, historically limited due to a lack of suitable data and quantitative methodology, has left crucial questions unanswered, which the CODEC (COgnitive Dynamics in Early Childhood) study aims to address.

METHOD

The CODEC cohort is an accelerated 3-year longitudinal study which encompasses 600 7-to-10-year-old children. Each year includes a 'burst' week (3 times per day, 5 days per week) of cognitive measurements on five cognitive domains (reasoning, working memory, processing speed, vocabulary, exploration), conducted both in classrooms and at home through experience sampling assessments. We also measure academic outcomes and external factors hypothesised to predict cognitive variability, including sleep, mood, motivation and background noise. A subset of 200 children (CODEC-MRI) are invited for two deep phenotyping sessions (in year 1 and year 3 of the study), including structural and functional magnetic resonance imaging, eye-tracking, parental measurements and questionnaire-based demographic and psychosocial measures. We will quantify developmental differences and changes in variability using Dynamic Structural Equation Modelling, allowing us to simultaneously capture variability and the multilevel structure of trials nested in sessions, days, children and classrooms.

DISCUSSION

CODEC's unique design allows us to measure variability across a range of different cognitive domains, ages, and temporal resolutions. The deep-phenotyping arm allows us to test hypotheses concerning variability, including the role of mind wandering, strategy exploration, mood, sleep, and brain structure. Due to CODEC's longitudinal nature, we are able to quantify which measures of variability at baseline predict long-term outcomes. In summary, the CODEC study is a unique longitudinal study combining experience sampling, an accelerated longitudinal 'burst' design, deep phenotyping, and cutting-edge statistical methodologies to better understand the nature, causes, and consequences of cognitive variability in children.

TRIAL REGISTRATION

ClinicalTrials.gov - NCT06330090.

Age-dependent predictors of effective reinforcement motor learning across childhood

Across development, children must learn motor skills such as eating with a spoon and drawing with a crayon. Reinforcement learning, driven by success and failure, is fundamental to such sensorimotor learning. It typically requires a child to explore movement options along a continuum (grip location on a crayon) and learn from probabilistic rewards (whether the crayon draws or breaks). Here, we studied the development of reinforcement motor learning using online motor tasks to engage children aged 3 to 17 and adults (cross-sectional sample, N=385). Participants moved a cartoon penguin across a scene and were rewarded (animated cartoon clip) based on their final movement position. Learning followed a clear developmental trajectory when participants could choose to move anywhere along a continuum and the reward probability depended on final movement position. Learning was incomplete or absent in 3 to 8-year-olds and gradually improved to adult-like levels by adolescence. A reinforcement learning model fit to each participant identified three age-dependent factors underlying improvement: amount of exploration after a failed movement, learning rate, and level of motor noise. We predicted, and confirmed, that switching to discrete targets and deterministic reward would improve 3 to 8-year-olds' learning to adult-like levels by increasing exploration after failed movements. Overall, we show a robust developmental trajectory of reinforcement motor learning abilities under ecologically relevant conditions i.e., continuous movement options mapped to probabilistic reward. This learning appears to be limited by immature spatial processing and probabilistic reasoning abilities in young children and can be rescued by reducing the demands in these domains.

Exploration, exploitation, and development: Developmental shifts in decision-making

Decision-making requires balancing exploration with exploitation, yet children are highly exploratory, with exploration decreasing with development. Less is known about what drives these changes. We examined the development of decision-making in 188 three- to eight-year-old children (M = 64 months; 98 girls) and 26 adults (M = 19 years; 13 women). Children were recruited from ethnically diverse suburban middle-class neighborhoods of Columbus, Ohio, USA. Results indicate that mature reward-based choices emerge relatively late in development, with children tending to over-explore. Computational modeling suggests that this exploration is systematic rather than random, as children tend to avoid repeating choices made on the previous trial. This pattern of exploration (reminiscent of novelty preference) decreased with development, whereas the tendency to exploit increased.

Dynamic computational phenotyping of human cognition

Computational phenotyping has emerged as a powerful tool for characterizing individual variability across a variety of cognitive domains. An individual's computational phenotype is defined as a set of mechanistically interpretable parameters obtained from fitting computational models to behavioural data. However, the interpretation of these parameters hinges critically on their psychometric properties, which are rarely studied. To identify the sources governing the temporal variability of the computational phenotype, we carried out a 12-week longitudinal study using a battery of seven tasks that measure aspects of human learning, memory, perception and decision making. To examine the influence of state effects, each week, participants provided reports tracking their mood, habits and daily activities. We developed a dynamic computational phenotyping framework, which allowed us to tease apart the time-varying effects of practice and internal states such as affective valence and arousal. Our results show that many phenotype dimensions covary with practice and affective factors, indicating that what appears to be unreliability may reflect previously unmeasured structure. These results support a fundamentally dynamic understanding of cognitive variability within an individual.

The structure and development of explore-exploit decision making

A critical component of human learning reflects the balance people must achieve between focusing on the utility of what they know versus openness to what they have yet to experience. How individuals decide whether to explore new options versus exploit known options has garnered growing interest in recent years. Yet, the component processes underlying decisions to explore and whether these processes change across development remain poorly understood. By contrasting a variety of tasks that measure exploration in slightly different ways, we found that decisions about whether to explore reflect (a) random exploration that is not explicitly goal-directed and (b) directed exploration to purposefully reduce uncertainty. While these components similarly characterized the decision-making of both youth and adults, younger participants made decisions that were less strategic, but more exploratory and flexible, than those of adults. These findings are discussed in terms of how people adapt to and learn from changing environments over time.Data has been made available in the Open Science Foundation platform (osf.io).

Children's subjective uncertainty-driven sampling behaviour

Are children and adults sensitive to gaps in their knowledge, and do they actively elicit information to resolve such knowledge gaps? In a cross-situational word learning task, we asked 5-year-olds, 6- to 9-year-olds and adults to estimate their knowledge of newly learned word-object associations. We then examined whether participants preferentially sampled objects they reported not knowing the label in order to hear their labels again. We also examined whether such uncertainty-driven sampling behaviour led to improved learning. We found that all age groups were sensitive to gaps in their knowledge of the word-object associations, i.e. were more likely to say they had correctly indicated the label of an object when they were correct, relative to when they were incorrect. Furthermore, 6- to 9-year-olds and adults-but not 5-year-olds-were more likely to sample objects whose labels they reported not knowing. In other words, older children and adults displayed sampling behaviour directed at reducing knowledge gaps and uncertainty, while younger children did not. However, participants who displayed more uncertainty-driven sampling behaviour were not more accurate at test. Our findings underscore the role of uncertainty in driving 6- to 9-year-olds' and adults' sampling behaviour and speak to the mechanisms underlying previously reported performance boosts in active learning.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Preschoolers search longer when there is more information to be gained

What drives children to explore and learn when external rewards are uncertain or absent? Across three studies, we tested whether information gain itself acts as an internal reward and suffices to motivate children's actions. We measured 24-56-month-olds' persistence in a game where they had to search for an object (animal or toy), which they never find, hidden behind a series of doors, manipulating the degree of uncertainty about which specific object was hidden. We found that children were more persistent in their search when there was higher uncertainty, and therefore, more information to be gained with each action, highlighting the importance of research on artificial intelligence to invest in curiosity-driven algorithms. RESEARCH HIGHLIGHTS: Across three studies, we tested whether information gain itself acts as an internal reward and suffices to motivate preschoolers' actions. We measured preschoolers' persistence when searching for an object behind a series of doors, manipulating the uncertainty about which specific object was hidden. We found that preschoolers were more persistent when there was higher uncertainty, and therefore, more information to be gained with each action. Our results highlight the importance of research on artificial intelligence to invest in curiosity-driven algorithms.

Childhood unpredictability and the development of exploration

Early in development, the process of exploration helps children gather new information that fosters learning about the world. Yet, it is unclear how childhood experiences may influence the way humans approach new learning. What influences decisions to exploit known, familiar options versus trying a novel alternative? We found that childhood unpredictability, characterized by unpredictable caregiving and unstable living environments, was associated with reduced exploratory behavior. This effect holds while controlling for individual differences, including anxiety and stress. Individuals who perceived their childhoods as unpredictable explored less and were instead more likely to repeat previous choices (habitual responding). They were also more sensitive to uncertainty than to potential rewards, even when the familiar options yielded lower rewards. We examined these effects across multiple task contexts and via both in-person (N = 78) and online replication (N = 84) studies among 10- to 13-y-olds. Results are discussed in terms of the potential cascading effects of unpredictable environments on the development of decision-making and the effects of early experience on subsequent learning.

Environmental statistics and experience shape risk-taking across adolescence

Adolescents are often portrayed as reckless risk-takers because of their immature brains. Recent research has cast doubt on this portrayal, identifying the environment as a moderator of risk-taking. However, the key features of environments that drive risk-taking behaviors are often underspecified. We call for greater attention to the environment by drawing on research showing that its statistical structure impacts future risk-taking as people learn from outcomes they experience after taking a risk. This opinion shows that adolescents are unlikely to experience harm from many risks because environmental statistics are skewed and favor safe experiences. Environmental statistics and experience suggest entry points for policy interventions by carefully timing risk warnings and leveraging peers' potential to shape the statistics of rewarding experiences.

Children Use Teachers' Beliefs About Their Abilities to Calibrate Explore-Exploit Decisions

Models of the explore-exploit problem have explained how children's decision making is weighed by a bias for information (directed exploration), randomness, and generalization. These behaviors are often tested in domains where a choice to explore (or exploit) is guaranteed to reveal an outcome. An often overlooked but critical component of the assessment of explore-exploit decisions lies in the expected success of taking actions in the first place-and, crucially, how such decisions might be carried out when learning from others. Here, we examine how children consider an informal teacher's beliefs about the child's competence when deciding how difficult a task they want to pursue. We present a simple model of this problem that predicts that while learners should follow the recommendation of an accurate teacher, they should exploit easier games when a teacher overestimates their abilities, and explore harder games when she underestimates them. We tested these predictions in two experiments with adults (Experiment 1) and 6- to 8-year-old children (Experiment 2). In our task, participants' performance on a picture-matching game was either overestimated, underestimated, or accurately represented by a confederate (the "Teacher"), who then presented three new matching games of varying assessed difficulty (too easy, too hard, just right) at varying potential reward (low, medium, high). In line with our model's predictions, we found that both adults and children calibrated their choices to the teacher's representation of their competence. That is, to maximize expected reward, when she underestimated them, participants chose games the teacher evaluated as being too hard for them; when she overestimated them, they chose games she evaluated as being too easy; and when she was accurate, they chose games she assessed as being just right. This work provides insight into the early-emerging ability to calibrate explore-exploit decisions to others' knowledge when learning in informal pedagogical contexts.

Developmental changes in exploration resemble stochastic optimization

Human development is often described as a 'cooling off' process, analogous to stochastic optimization algorithms that implement a gradual reduction in randomness over time. Yet there is ambiguity in how to interpret this analogy, due to a lack of concrete empirical comparisons. Using data from n = 281 participants ages 5 to 55, we show that cooling off does not only apply to the single dimension of randomness. Rather, human development resembles an optimization process of multiple learning parameters, for example, reward generalization, uncertainty-directed exploration and random temperature. Rapid changes in parameters occur during childhood, but these changes plateau and converge to efficient values in adulthood. We show that while the developmental trajectory of human parameters is strikingly similar to several stochastic optimization algorithms, there are important differences in convergence. None of the optimization algorithms tested were able to discover reliably better regions of the strategy space than adult participants on this task.

Understanding patch foraging strategies across development

Patch foraging is a near-ubiquitous behaviour across the animal kingdom and characterises many decision-making domains encountered by humans. We review how a disposition to explore in adolescence may reflect the evolutionary conditions under which hunter-gatherers foraged for resources. We propose that neurocomputational mechanisms responsible for reward processing, learning, and cognitive control facilitate the transition from exploratory strategies in adolescence to exploitative strategies in adulthood - where individuals capitalise on known resources. This developmental transition may be disrupted by psychopathology, as there is emerging evidence of biases in explore/exploit choices in mental health problems. Explore/exploit choices may be an informative marker for mental health across development and future research should consider this feature of decision-making as a target for clinical intervention.

What's in the Box? Preschoolers Consider Ambiguity, Expected Value, and Information for Future Decisions in Explore-Exploit Tasks

Self-directed exploration in childhood appears driven by a desire to resolve uncertainties in order to learn more about the world. However, in adult decision-making, the choice to explore new information rather than exploit what is already known takes many factors beyond uncertainty (such as expected utilities and costs) into account. The evidence for whether young children are sensitive to complex, contextual factors in making exploration decisions is limited and mixed. Here, we investigate whether modifying uncertain options influences explore-exploit behavior in preschool-aged children (48-68 months). Over the course of three experiments, we manipulate uncertain options' ambiguity, expected value, and potential to improve epistemic state for future exploration in a novel forced-choice design. We find evidence that young children are influenced by each of these factors, suggesting that early, self-directed exploration involves sophisticated, context-sensitive decision-making under uncertainty.

Multifaceted information-seeking motives in children

From an early age, children need to gather information to learn about their environment. Deciding which knowledge to pursue can be difficult because information can serve several, sometimes competing, purposes. Here, we examine the developmental trajectories of such diverse information-seeking motives. Over five experiments involving 521 children (aged 4-12), we find that school-age children integrate three key factors into their information-seeking choices: whether information reduces uncertainty, is useful in directing action, and is likely to be positive. Choices that likely reveal positive information and are useful for action emerge as early as age 4, followed by choices that reduce uncertainty (at ~age 5). Our results suggest that motives related to usefulness and uncertainty reduction become stronger with age, while the tendency to seek positive news does not show a statistically significant change throughout development. This study reveals how the relative importance of diverging, sometimes conflicting, information-seeking motives emerges throughout development.

Empowerment contributes to exploration behaviour in a creative video game

Studies of human exploration frequently cast people as serendipitously stumbling upon good options. Yet these studies may not capture the richness of exploration strategies that people exhibit in more complex environments. Here we study behaviour in a large dataset of 29,493 players of the richly structured online game 'Little Alchemy 2'. In this game, players start with four elements, which they can combine to create up to 720 complex objects. We find that players are driven not only by external reward signals, such as an attempt to produce successful outcomes, but also by an intrinsic motivation to create objects that empower them to create even more objects. We find that this drive for empowerment is eliminated when playing a game variant that lacks recognizable semantics, indicating that people use their knowledge about the world and its possibilities to guide their exploration. Our results suggest that the drive for empowerment may be a potent source of intrinsic motivation in richly structured domains, particularly those that lack explicit reward signals.

Novelty and uncertainty differentially drive exploration across development

Across the lifespan, individuals frequently choose between exploiting known rewarding options or exploring unknown alternatives. A large body of work has suggested that children may explore more than adults. However, because novelty and reward uncertainty are often correlated, it is unclear how they differentially influence decision-making across development. Here, children, adolescents, and adults (ages 8-27 years, N = 122) completed an adapted version of a recently developed value-guided decision-making task that decouples novelty and uncertainty. In line with prior studies, we found that exploration decreased with increasing age. Critically, participants of all ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion to reward uncertainty increased into adulthood. Computational modeling of participant choices revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for more novel choice options, children's choices were not influenced by reward uncertainty.

Comparative curiosity: How do great apes and children deal with uncertainty?

Humans are perhaps the most curious animals on earth, but to what extent our innate motivations for discovering new information are shared with our closest relatives remain poorly understood. To shed light on this question, we presented great apes with two experimental paradigms in which they had to initially choose between an empty opaque cup and a baited opaque cup with rewards invisible to the ape in study 1, or to choose between a transparent cup with rewards or a baited opaque cup with rewards invisible to the ape in studies 2 and 3. We also presented young children with scenarios comparable to the second paradigm (studies 4 and 5). Notably, after the initial choice phase, we presented participants with potential alternatives providing better rewards than the previously secured options. Importantly, those alternatives shared some features with the uncertain options, giving subjects the possibility to relate both options through analogical reasoning. We found that most great apes were not curious about the uncertain options. They only explored those options after they were presented with the alternatives. Children, instead, explored the uncertain options before the alternatives were presented, showing a higher degree of curiosity than the great apes. We argue that differences between children and apes mostly lay in motivational dispositions to explore the unknown.

Exploratory preferences explain the human fascination for imaginary worlds in fictional stories

Imaginary worlds are present and often central in many of the most culturally successful modern narrative fictions, be it in novels (e.g., Harry Potter), movies (e.g., Star Wars), video games (e.g., The Legend of Zelda), graphic novels (e.g., One Piece) and TV series (e.g., Game of Thrones). We propose that imaginary worlds are popular because they activate exploratory preferences that evolved to help us navigate the real world and find new fitness-relevant information. Therefore, we hypothesize that the attraction to imaginary worlds is intrinsically linked to the desire to explore novel environments and that both are influenced by the same underlying factors. Notably, the inter-individual and cross-cultural variability of the preference for imaginary worlds should follow the inter-individual and cross-cultural variability of exploratory preferences (with the personality trait Openness-to-experience, age, sex, and ecological conditions). We test these predictions with both experimental and computational methods. For experimental tests, we run a pre-registered online experiment about movie preferences (N = 230). For computational tests, we leverage two large cultural datasets, namely the Internet Movie Database (N = 9424 movies) and the Movie Personality Dataset (N = 3.5 million participants), and use machine-learning algorithms (i.e., random forest and topic modeling). In all, consistent with how the human preference for spatial exploration adaptively varies, we provide empirical evidence that imaginary worlds appeal more to more explorative people, people higher in Openness-to-experience, younger individuals, males, and individuals living in more affluent environments. We discuss the implications of these findings for our understanding of the cultural evolution of narrative fiction and, more broadly, the evolution of human exploratory preferences.

Early adversity and the development of explore-exploit tradeoffs

Childhood adversity can have wide-ranging and long-lasting effects on later life. But what are the mechanisms that are responsible for these effects? This article brings together the cognitive science literature on explore-exploit tradeoffs, the empirical literature on early adversity, and the literature in evolutionary biology on 'life history' to explain how early experience influences later life. We propose one potential mechanism: early experiences influence 'hyperparameters' that determine the balance between exploration and exploitation. Adversity might accelerate a shift from exploration to exploitation, with broad and enduring effects on the adult brain and mind. These effects may be produced by life-history adaptations that use early experience to tailor development and learning to the likely future states of an organism and its environment.

Exploration patterns shape cognitive map learning

Spontaneous, volitional spatial exploration is crucial for building up a cognitive map of the environment. However, decades of research have primarily measured the fidelity of cognitive maps after discrete, controlled learning episodes. We know little about how cognitive maps are formed during naturalistic free exploration. Here, we investigated whether exploration trajectories predicted cognitive map accuracy, and how these patterns were shaped by environmental structure. In two experiments, participants freely explored a previously unfamiliar virtual environment. We related their exploration trajectories to a measure of how long they spent in areas with high global environmental connectivity (integration, as assessed by space syntax). In both experiments, we found that participants who spent more time on paths that offered opportunities for integration formed more accurate cognitive maps. Interestingly, we found no support for our pre-registered hypothesis that self-reported trait differences in navigation ability would mediate this relationship. Our findings suggest that exploration patterns predict cognitive map accuracy, even for people who self-report low ability, and highlight the importance of considering both environmental structure and individual variability in formal theory- and model-building.

Maladaptive explore/exploit trade-offs in schizophrenia

Schizophrenia is a complex disorder that remains poorly understood, particularly at the systems level. In this opinion article we argue that the explore/exploit trade-off concept provides a holistic and ecologically valid framework to resolve some of the apparent paradoxes that have emerged within schizophrenia research. We review recent evidence suggesting that fundamental explore/exploit behaviors may be maladaptive in schizophrenia during physical, visual, and cognitive foraging. We also describe how theories from the broader optimal foraging literature, such as the marginal value theorem (MVT), could provide valuable insight into how aberrant processing of reward, context, and cost/effort evaluations interact to produce maladaptive responses.

Striatal dopamine supports reward expectation and learning: A simultaneous PET/fMRI study

Converging evidence from both human neuroimaging and animal studies has supported a model of mesolimbic processing underlying reward learning behaviors, based on the computation of reward prediction errors. However, competing evidence supports human dopamine signaling in the basal ganglia as also contributing to the generation of higher order learning heuristics. Here, we present data from a large (N = 81, 18-30yo), multi-modal neuroimaging study using simultaneously acquired task fMRI, affording temporal resolution of reward system function, and PET imaging with [¹¹C]Raclopride (RAC), assessing striatal dopamine (DA) D2/3 receptor binding, during performance of a probabilistic reward learning task. Both fMRI activation and PET DA measures showed ventral striatum involvement for signaling rewards. However, greater DA release was uniquely associated with learning strategies (i.e., learning rates) that were more task-optimal within the best fitting reinforcement learning model. This DA response was associated with BOLD activation of a network of regions including anterior cingulate cortex, medial prefrontal cortex, thalamus and posterior parietal cortex, primarily during expectation, rather than prediction error, task epochs. Together, these data provide novel, human in vivo evidence that striatal dopaminergic signaling interacts with a network of cortical regions to generate task-optimal learning strategies, rather than representing reward outcomes in isolation.

Microfoundations of Adaptive Search in Complex Tasks: The Role of Cognitive Abilities and Styles

Problem-solving in complex environments requires a cognitively demanding search for task solutions. Managing this search process presents a major challenge in organizations. We contribute to the literature on this topic by providing new evidence on the cognitive antecedents that shape how individuals search when engaged in complex problem-solving tasks. We present results from three laboratory studies, wherein 335 individuals solved a complex task. In doing so, they generated behavioral data coupled with survey-based measurements of the individuals’ cognitive styles and performance-based tests of their cognitive abilities. Our data analysis contributes to the current literature by documenting systematic heterogeneity in the persistence and distance of search that can be explained by the participants’ level of creativity, attention to detail, and executive functions. We extend the research on the microfoundations of adaptive search by linking cognitive antecedents with a complex search task, widening our insight into what search behavior certain cognitive microfoundations lead to, and showing how managers can more effectively shape organizational search.

History: This paper has been accepted for the Organization Science Special Issue on Experiments in Organizational Theory.

Supplemental Material: The online appendix is available at https://doi.org/10.1287/orsc.2023.1654 .

Balancing novelty and appropriateness leads to creative associations in children

Creative problem solving is a fundamental skill of human cognition and is conceived as a search process whereby a novel and appropriate solution is generated. However, it is unclear whether children are able to balance novelty and appropriateness to generate creative solutions and what are the underlying computational mechanisms. Here, we asked children, ranging from 10 to 11 years old, to perform a word association task according to three instructions, which triggered a more appropriate (ordinary), novel (random), or balanced (creative) response. Results revealed that children exhibited greater cognitive flexibility in the creative condition compared to the control conditions, as revealed by the structure and resiliency of the semantic networks. Moreover, responses' word embeddings extracted from pretrained deep neural networks showed that semantic distance and category switching index increased in the creative condition with respect to the ordinary condition and decreased compared to the random condition. Critically, we showed how children efficiently solved the exploration/exploitation trade-off to generate creative associations by fitting a computational reinforcement learning (RL) model that simulates semantic search strategies. Our findings provide compelling evidence that children balance novelty and appropriateness to generate creative associations by optimally regulating the level of exploration in the semantic search. This corroborates previous findings on the adult population and highlights the crucial contribution of both components to the overall creative process. In conclusion, these results shed light on the connections between theoretical concepts such as bottom-up/top-down modes of thinking in creativity research and the exploration/exploitation trade-off in human RL research.

Contributions of expected learning progress and perceptual novelty to curiosity-driven exploration

Exploration is curiosity-driven when it relies on the intrinsic motivation to know rather than on extrinsic rewards. Recent evidence shows that artificial agents perform better on a variety of tasks when their learning is curiosity-driven, and humans often engage in curiosity-driven learning when sampling information from the environment. However, which mechanisms underlie curiosity is still unclear. Here, we let participants freely explore different unknown environments that contained learnable sequences of events with varying degrees of noise and volatility. A hierarchical reinforcement learning model captured how participants were learning in these different kinds of unknown environments, and it also tracked the errors they expected to make and the learning opportunities they were planning to seek. With this computational approach, we show that participants' exploratory behavior is guided by learning progress and perceptual novelty. Moreover, we demonstrate an overall tendency of participants to avoid extreme forms of uncertainty. These findings elucidate the cognitive mechanisms that underlie curiosity-driven exploration of unknown environments. Implications of this novel way of quantifying curiosity within a reinforcement learning framework are discussed.

Spontaneous discovery of novel task solutions in children

Children often perform worse than adults on tasks that require focused attention. While this is commonly regarded as a sign of incomplete cognitive development, a broader attentional focus could also endow children with the ability to find novel solutions to a given task. To test this idea, we investigated children's ability to discover and use novel aspects of the environment that allowed them to improve their decision-making strategy. Participants were given a simple choice task in which the possibility of strategy improvement was neither mentioned by instructions nor encouraged by explicit error feedback. Among 47 children (8-10 years of age) who were instructed to perform the choice task across two experiments, 27.5% showed a full strategy change. This closely matched the proportion of adults who had the same insight (28.2% of n = 39). The amount of erroneous choices, working memory capacity and inhibitory control, in contrast, indicated substantial disadvantages of children in task execution and cognitive control. A task difficulty manipulation did not affect the results. The stark contrast between age-differences in different aspects of cognitive performance might offer a unique opportunity for educators in fostering learning in children.

Novelty preferences depend on goals

People are sometimes drawn to novel items, but other times prefer familiar ones. In the present research we show, though, that both children’s and adults’ preferences for novel versus familiar items depend on their goals. Across four experiments, we showed 4- to 7-year-olds (total N = 498) and adults (total N = 659) pairs of artifacts where one was familiar and the other was novel (e.g., a four-legged chair and ten-legged chair). In Experiment 1, children wanted to have familiar artifacts, but to learn about novel ones. Experiment 2 replicated this pattern using a simpler procedure, and found the same pattern in adults. In Experiment 3, 4- to 6-year-olds and adults more strongly preferred familiar items when choosing which they would rather have than when choosing which they would rather try using. Finally, Experiment 4 replicated adults’ preferences to have familiar items and learn about novel ones with an additional set of items. Together these findings show that preferences for novelty depend on people’s goals. We suggest these effects arise because children and adults are motivated both by the promise of information and the desire for safe options in high commitment decisions that entail risk.

Insights about the common generative rule underlying an information foraging task can be facilitated via collective search

Social learning is beneficial for efficient information search in unfamiliar environments ("within-task" learning). In the real world, however, possible search spaces are often so large that decision makers are incapable of covering all options, even if they pool their information collectively. One strategy to handle such overload is developing generalizable knowledge that extends to multiple related environments ("across-task" learning). However, it is unknown whether and how social information may facilitate such across-task learning. Here, we investigated participants' social learning processes across multiple laboratory foraging sessions in spatially correlated reward landscapes that were generated according to a common rule. The results showed that paired participants were able to improve efficiency in information search across sessions more than solo participants. Computational analysis of participants' choice-behaviors revealed that such improvement across sessions was related to better understanding of the common generative rule. Rule understanding was correlated within a pair, suggesting that social interaction is a key to the improvement of across-task learning.

Uncertainty about others' trustworthiness increases during adolescence and guides social information sampling

Adolescence is a key life phase for developing well-adjusted social behaviour. An essential component of well-adjusted social behaviour is the ability to update our beliefs about the trustworthiness of others based on gathered information. Here, we examined how adolescents (n = 157, 10-24 years) sequentially sampled information about the trustworthiness of peers and how they used this information to update their beliefs about others' trustworthiness. Our Bayesian computational modelling approach revealed an adolescence-emergent increase in uncertainty of prior beliefs about others' trustworthiness. As a consequence, early to mid-adolescents (ages 10-16) gradually relied less on their prior beliefs and more on the gathered evidence when deciding to sample more information, and when deciding to trust. We propose that these age-related differences could be adaptive to the rapidly changing social environment of early and mid-adolescents. Together, these findings contribute to the understanding of adolescent social development by revealing adolescent-emergent flexibility in prior beliefs about others that drives adolescents' information sampling and trust decisions.

The Sweet Spot: When Children’s Developing Abilities, Brains, and Knowledge Make Them Better Learners Than Adults

Cognitive development is marked by age-related improvements across a number of domains, as young children perform worse than their older counterparts on most tasks. However, there are cases in which young children, and even infants, outperform older children and adults. So when, and why, does being young sometimes confer an advantage? This article provides a comprehensive examination of the peculiar cases in which younger children perform better. First, we outline the specific instances in which younger is better across domains, including mastering language, using probabilistic information, detecting causal relations, remembering certain information, and even solving problems. We then examine how children’s reduced cognitive abilities, ongoing brain development, more limited prior knowledge, and heightened tendency to explore benefits their learning, reasoning, perception, and memory from a mechanistic perspective. We hold that considering all of these factors together is essential for understanding the ways in which children’s learning is unique and that science has much to learn from a careful consideration of childhood.

Parental praise and children's exploration: a virtual reality experiment

When children practice a new skill and fail, it is critical for them to explore new strategies to succeed. How can parents encourage children's exploration? Bridging insights from developmental psychology and the neuroscience of motor control, we examined the effects of parental praise on children's motor exploration. We theorize that modest praise can spark exploration. Unlike inflated praise, modest praise acknowledges children's performance, without setting a high standard for future performance. This may be reassuring to children with lower levels of self-esteem, who often doubt their ability. We conducted a novel virtual-reality experiment. Children (N = 202, ages 8-12) reported self-esteem and performed a virtual-reality 3D trajectory-matching task, with success/failure feedback after each trial. Children received modest praise ("You did well!"), inflated praise ("You did incredibly well!"), or no praise from their parent. We measured motor exploration as children's tendency to vary their movements following failure. Relative to no praise, modest praise-unlike inflated praise-encouraged exploration in children with lower levels of self-esteem. By contrast, modest praise discouraged exploration in children with higher levels of self-esteem. Effects were small yet robust. This experiment demonstrates that modest praise can spark exploration in children with lower levels of self-esteem.

Restricted Access to Working Memory Does Not Prevent Cumulative Score Improvement in a Cultural Evolution Task

Some theories propose that human cumulative culture is dependent on explicit, system-2, metacognitive processes. To test this, we investigated whether access to working memory is required for cumulative cultural evolution. We restricted access to adults' working-memory (WM) via a dual-task paradigm, to assess whether this reduced performance in a cultural evolution task, and a metacognitive monitoring task. In total, 247 participants completed either a grid search task or a metacognitive monitoring task in conjunction with a WM task and a matched control. Participants' behaviour in the grid search task was then used to simulate the outcome of iterating the task over multiple generations. Participants in the grid search task scored higher after observing higher-scoring examples, but could only beat the scores of low-scoring example trials. Scores did not differ significantly between the control and WM distractor blocks, although more errors were made when under WM load. The simulation showed similar levels of cumulative score improvement across conditions. However, scores plateaued without reaching the maximum. Metacognitive efficiency was low in both blocks, with no indication of dual-task interference. Overall, we found that taxing working-memory resources did not prevent cumulative score improvement on this task, but impeded it slightly relative to a control distractor task. However, we found no evidence that the dual-task manipulation impacted participants' ability to use explicit metacognition. Although we found minimal evidence in support of the explicit metacognition theory of cumulative culture, our results provide valuable insights into empirical approaches that could be used to further test predictions arising from this account.

Exploration heuristics decrease during youth

Deciding between exploring new avenues and exploiting known choices is central to learning, and this exploration-exploitation trade-off changes during development. Exploration is not a unitary concept, and humans deploy multiple distinct mechanisms, but little is known about their specific emergence during development. Using a previously validated task in adults, changes in exploration mechanisms were investigated between childhood (8-9 y/o, N = 26; 16 females), early (12-13 y/o, N = 38; 21 females), and late adolescence (16-17 y/o, N = 33; 19 females) in ethnically and socially diverse schools from disadvantaged areas. We find an increased usage of a computationally light exploration heuristic in younger groups, effectively accommodating their limited neurocognitive resources. Moreover, this heuristic was associated with self-reported, attention-deficit/hyperactivity disorder symptoms in this population-based sample. This study enriches our mechanistic understanding about how exploration strategies mature during development.

Children are more exploratory and learn more than adults in an approach-avoid task

Intuitively, children appear to be more exploratory than adults, and this exploration seems to help children learn,. However, there have been few clear tests of these ideas. We test whether exploration and learning change across development using a task that presents a "learning trap." In this task, exploitation-maximizing immediate reward and avoiding costs-may lead the learner to draw incorrect conclusions, while exploration may lead to better learning but be more costly. In Studies 1, 2, and 3 we find that preschoolers and early school-aged children explore more than adults and learn the true structure of the environment better. Study 3 demonstrates that children explore more than adults even though they, like adults, predict that exploration will be costly, and it shows that exploration and learning are correlated. Study 4 shows that children's and adults' learning depends on the evidence they generate during exploration: children exposed to adult-like evidence learn like adults, and adults exposed to child-like evidence learn like children. Together, these studies support the idea that children may be more exploratory than adults, and this increased exploration influences learning.

Sources of variation in search and foraging: A theoretical perspective

Search-the problem of exploring a space of alternatives to identify target goals-is a fundamental behaviour for many species. Although its foundation lies in foraging, most studies of human search behaviour have been directed towards understanding the attentional mechanisms that underlie the efficient visual exploration of two-dimensional (2D) scenes. With this review, we aim to characterise how search behaviour can be explained across a wide range of contexts, environments, spatial scales, and populations, both typical and atypical. We first consider the generality of search processes across psychological domains. We then review studies of interspecies differences in search. Finally, we explore in detail the individual and contextual variables that affect visual search and related behaviours in established experimental psychology paradigms. Despite the heterogeneity of the findings discussed, we identify that variations in control processes, along with the ability to regulate behaviour as a function of the structure of search space and the sampling processes adopted, to be central to explanations of variations in search behaviour. We propose a tentative theoretical model aimed at integrating these notions and close by exploring questions that remain unaddressed.

Context and attention control determine whether attending to competing information helps or hinders learning in school-aged children

Attention control regulates efficient processing of goal-relevant information by suppressing interference from irrelevant competing inputs while also flexibly allocating attention across relevant inputs according to task demands. Research has established that developing attention control skills promote effective learning by minimizing distractions from task-irrelevant competing information. Additional research also suggests that competing contextual information can provide meaningful input for learning and should not always be ignored. Instead, attending to competing information that is relevant to task goals can facilitate and broaden the scope of children's learning. We review this past research examining effects of attending to task-relevant and task-irrelevant competing information on learning outcomes, focusing on relations between visual attention and learning in childhood. We then present a synthesis argument that complex interactions across learning goals, the contexts of learning environments and tasks, and developing attention control mechanisms will determine whether attending to competing information helps or hinders learning. This article is categorized under: Psychology > Attention Psychology > Learning Psychology > Development and Aging.

Balancing exploration and exploitation with information and randomization

Explore-exploit decisions require us to trade off the benefits of exploring unknown options to learn more about them, with exploiting known options, for immediate reward. Such decisions are ubiquitous in nature, but from a computational perspective, they are notoriously hard. There is therefore much interest in how humans and animals make these decisions and recently there has been an explosion of research in this area. Here we provide a biased and incomplete snapshot of this field focusing on the major finding that many organisms use two distinct strategies to solve the explore-exploit dilemma: a bias for information ('directed exploration') and the randomization of choice ('random exploration'). We review evidence for the existence of these strategies, their computational properties, their neural implementations, as well as how directed and random exploration vary over the lifespan. We conclude by highlighting open questions in this field that are ripe to both explore and exploit.

In Pursuit of Knowledge: Preschoolers Expect Agents to Weigh Information Gain and Information Cost When Deciding Whether to Explore

When deciding whether to explore, agents must consider both their need for information and its cost. Do children recognize that exploration reflects a trade-off between action costs and expected information gain, inferring epistemic states accordingly? In two experiments, 4- and 5-year-olds (N = 144; of diverse race and ethnicity) judge that an agent who refuses to obtain low-cost information must have already known it, and an agent who incurs a greater cost to gain information must have a greater epistemic desire. Two control studies suggest that these findings cannot be explained by low-level associations between competence and knowledge. Our results suggest that preschoolers' theory of mind includes expectations about how costs interact with epistemic desires and states to produce exploratory action.

Children perform extensive information gathering when it is not costly

Humans often face decisions where little is known about the choice options. Gathering information prior to making a choice is an important strategy to improve decision making under uncertainty. This is of particular importance during childhood and adolescence, when knowledge about the world is still limited. To examine how much information youths gather, we asked 107 children (8-9 years, N = 30), early (12-13 years, N = 41) and late adolescents (16-17 years, N = 36) to perform an information sampling task. We find that children gather significantly more information before making a decision compared to adolescents, but only if it does not come with explicit costs. Using computational modelling, we find that this is because children have reduced subjective costs for gathering information. Our findings thus demonstrate how children overcome their limited knowledge and neurocognitive constraints by deploying excessive information gathering, a developmental feature that could inform aberrant information gathering in psychiatric disorders.

Development of directed and random exploration in children

Are young children just random explorers who learn serendipitously? Or are even young children guided by uncertainty-directed sampling, seeking to explore in a systematic fashion? We study how children between the ages of 4 and 9 search in an explore-exploit task with spatially correlated rewards, where exhaustive exploration is infeasible and not all options can be experienced. By combining behavioral data with a computational model that decomposes search into similarity-based generalization, uncertainty-directed exploration, and random exploration, we map out developmental trajectories of generalization and exploration. The behavioral data show strong developmental differences in children's capability to exploit environmental structure, with performance and adaptiveness of sampling decisions increasing with age. Through model-based analyses, we disentangle different forms of exploration, finding signature of both uncertainty-directed and random exploration. The amount of random exploration strongly decreases as children get older, supporting the notion of a developmental "cooling off" process that modulates the randomness in sampling. However, even at the youngest age range, children do not solely rely on random exploration. Even as random exploration begins to taper off, children are actively seeking out options with high uncertainty in a goal-directed fashion, and using inductive inferences to generalize their experience to novel options. Our findings provide critical insights into the behavioral and computational principles underlying the developmental trajectory of learning and exploration.

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore-exploit dilemma

When making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Here we report that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation on a reinforcement learning task with a spatially structured reward function. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.

Similarities and differences in spatial and non-spatial cognitive maps

Learning and generalization in spatial domains is often thought to rely on a "cognitive map", representing relationships between spatial locations. Recent research suggests that this same neural machinery is also recruited for reasoning about more abstract, conceptual forms of knowledge. Yet, to what extent do spatial and conceptual reasoning share common computational principles, and what are the implications for behavior? Using a within-subject design we studied how participants used spatial or conceptual distances to generalize and search for correlated rewards in successive multi-armed bandit tasks. Participant behavior indicated sensitivity to both spatial and conceptual distance, and was best captured using a Bayesian model of generalization that formalized distance-dependent generalization and uncertainty-guided exploration as a Gaussian Process regression with a radial basis function kernel. The same Gaussian Process model best captured human search decisions and judgments in both domains, and could simulate realistic learning curves, where we found equivalent levels of generalization in spatial and conceptual tasks. At the same time, we also find characteristic differences between domains. Relative to the spatial domain, participants showed reduced levels of uncertainty-directed exploration and increased levels of random exploration in the conceptual domain. Participants also displayed a one-directional transfer effect, where experience in the spatial task boosted performance in the conceptual task, but not vice versa. While confidence judgments indicated that participants were sensitive to the uncertainty of their knowledge in both tasks, they did not or could not leverage their estimates of uncertainty to guide exploration in the conceptual task. These results support the notion that value-guided learning and generalization recruit cognitive-map dependent computational mechanisms in spatial and conceptual domains. Yet both behavioral and model-based analyses suggest domain specific differences in how these representations map onto actions.

Attentional mechanisms drive systematic exploration in young children

Exploration is critical for discovering how the world works. Exploration should be particularly valuable for young children, who have little knowledge about the world. Theories of decision-making describe systematic exploration as being primarily driven by top-down cognitive control, which is immature in young children. Recent research suggests that a type of systematic exploration predominates in young children's choices, despite immature control, suggesting that it may be driven by different mechanisms. We hypothesize that young children's tendency to distribute attention widely promotes elevated exploration, and that interrupting distributed attention allocation through bottom-up attentional capture would also disrupt systematic exploration. We test this hypothesis by manipulating saliency of the options in a simple choice task. Saliency disrupted systematic exploration, thus indicating that attentional mechanisms may drive children's systematic exploratory behavior. We suggest that both may be part of a larger tendency toward broad information gathering in young children.