Manifesto for a new (computational) cognitive revolution

Using games to understand the mind

Board, card or video games have been played by virtually every individual in the world. Games are popular because they are intuitive and fun. These distinctive qualities of games also make them ideal for studying the mind. By being intuitive, games provide a unique vantage point for understanding the inductive biases that support behaviour in more complex, ecological settings than traditional laboratory experiments. By being fun, games allow researchers to study new questions in cognition such as the meaning of 'play' and intrinsic motivation, while also supporting more extensive and diverse data collection by attracting many more participants. We describe the advantages and drawbacks of using games relative to standard laboratory-based experiments and lay out a set of recommendations on how to gain the most from using games to study cognition. We hope this Perspective will lead to a wider use of games as experimental paradigms, elevating the ecological validity, scale and robustness of research on the mind.

Designing optimal behavioral experiments using machine learning

Computational models are powerful tools for understanding human cognition and behavior. They let us express our theories clearly and precisely and offer predictions that can be subtle and often counter-intuitive. However, this same richness and ability to surprise means our scientific intuitions and traditional tools are ill-suited to designing experiments to test and compare these models. To avoid these pitfalls and realize the full potential of computational modeling, we require tools to design experiments that provide clear answers about what models explain human behavior and the auxiliary assumptions those models must make. Bayesian optimal experimental design (BOED) formalizes the search for optimal experimental designs by identifying experiments that are expected to yield informative data. In this work, we provide a tutorial on leveraging recent advances in BOED and machine learning to find optimal experiments for any kind of model that we can simulate data from, and show how by-products of this procedure allow for quick and straightforward evaluation of models and their parameters against real experimental data. As a case study, we consider theories of how people balance exploration and exploitation in multi-armed bandit decision-making tasks. We validate the presented approach using simulations and a real-world experiment. As compared to experimental designs commonly used in the literature, we show that our optimal designs more efficiently determine which of a set of models best account for individual human behavior, and more efficiently characterize behavior given a preferred model. At the same time, formalizing a scientific question such that it can be adequately addressed with BOED can be challenging and we discuss several potential caveats and pitfalls that practitioners should be aware of. We provide code to replicate all analyses as well as tutorial notebooks and pointers to adapt the methodology to different experimental settings.

The cognitive (lateral) hypothalamus

Despite the physiological complexity of the hypothalamus, its role is typically restricted to initiation or cessation of innate behaviors. For example, theories of lateral hypothalamus argue that it is a switch to turn feeding 'on' and 'off' as dictated by higher-order structures that render when feeding is appropriate. However, recent data demonstrate that the lateral hypothalamus is critical for learning about food-related cues. Furthermore, the lateral hypothalamus opposes learning about information that is neutral or distal to food. This reveals the lateral hypothalamus as a unique arbitrator of learning capable of shifting behavior toward or away from important events. This has relevance for disorders characterized by changes in this balance, including addiction and schizophrenia. Generally, this suggests that hypothalamic function is more complex than increasing or decreasing innate behaviors.

Commonsense psychology in human infants and machines

Human infants are fascinated by other people. They bring to this fascination a constellation of rich and flexible expectations about the intentions motivating people's actions. Here we test 11-month-old infants and state-of-the-art learning-driven neural-network models on the "Baby Intuitions Benchmark (BIB)," a suite of tasks challenging both infants and machines to make high-level predictions about the underlying causes of agents' actions. Infants expected agents' actions to be directed towards objects, not locations, and infants demonstrated default expectations about agents' rationally efficient actions towards goals. The neural-network models failed to capture infants' knowledge. Our work provides a comprehensive framework in which to characterize infants' commonsense psychology and takes the first step in testing whether human knowledge and human-like artificial intelligence can be built from the foundations cognitive and developmental theories postulate.

Individual differences in naturalistic learning link negative emotionality to the development of anxiety

Organisms learn from prediction errors (PEs) to predict the future. Laboratory studies using small financial outcomes find that humans use PEs to update expectations and link individual differences in PE-based learning to internalizing disorders. Because of the low-stakes outcomes in most tasks, it is unclear whether PE learning emerges in naturalistic, high-stakes contexts and whether individual differences in PE learning predict psychopathology risk. Using experience sampling to assess 625 college students' expected exam grades, we found evidence of PE-based learning and a general tendency to discount negative PEs, an "optimism bias." However, individuals with elevated negative emotionality, a personality trait linked to the development of anxiety disorders, displayed a global pessimism and learning differences that impeded accurate expectations and predicted future anxiety symptoms. A sensitivity to PEs combined with an aversion to negative PEs may result in a pessimistic and inaccurate model of the world, leading to anxiety.

Conversation dynamics in a multiplayer video game with knowledge asymmetry

Despite the challenges associated with virtually mediated communication, remote collaboration is a defining characteristic of online multiplayer gaming communities. Inspired by the teamwork exhibited by players in first-person shooter games, this study investigated the verbal and behavioral coordination of four-player teams playing a cooperative online video game. The game, Desert Herding, involved teams consisting of three ground players and one drone operator tasked to locate, corral, and contain evasive robot agents scattered across a large desert environment. Ground players could move throughout the environment, while the drone operator's role was akin to that of a "spectator" with a bird's-eye view, with access to veridical information of the locations of teammates and the to-be-corralled agents. Categorical recurrence quantification analysis (catRQA) was used to measure the communication dynamics of teams as they completed the task. Demands on coordination were manipulated by varying the ground players' ability to observe the environment with the use of game "fog." Results show that catRQA was sensitive to changes to task visibility, with reductions in task visibility reorganizing how participants conversed during the game to maintain team situation awareness. The results are discussed in the context of future work that can address how team coordination can be augmented with the inclusion of artificial agents, as synthetic teammates.

Context-dependent choice and evaluation in real-world consumer behavior

A body of work spanning neuroscience, economics, and psychology indicates that decision-making is context-dependent, which means that the value of an option depends not only on the option in question, but also on the other options in the choice set-or the 'context'. While context effects have been observed primarily in small-scale laboratory studies with tightly constrained, artificially constructed choice sets, it remains to be determined whether these context effects take hold in real-world choice problems, where choice sets are large and decisions driven by rich histories of direct experience. Here, we investigate whether valuations are context-dependent in real-world choice by analyzing a massive restaurant rating dataset as well as two independent replication datasets which provide complementary operationalizations of restaurant choice. We find that users make fewer ratings-maximizing choices in choice sets with higher-rated options-a hallmark of context-dependent choice-and that post-choice restaurant ratings also varied systematically with the ratings of unchosen restaurants. Furthermore, in a follow-up laboratory experiment using hypothetical choice sets matched to the real-world data, we find further support for the idea that subjective valuations of restaurants are scaled in accordance with the choice context, providing corroborating evidence for a general mechanistic-level account of these effects. Taken together, our results provide a potent demonstration of context-dependent choice in real-world choice settings, manifesting both in decisions and subjective valuation of options.

Cognition-Enhanced Machine Learning for Better Predictions with Limited Data

The fields of machine learning (ML) and cognitive science have developed complementary approaches to computationally modeling human behavior. ML's primary concern is maximizing prediction accuracy; cognitive science's primary concern is explaining the underlying mechanisms. Cross-talk between these disciplines is limited, likely because the tasks and goals usually differ. The domain of e-learning and knowledge acquisition constitutes a fruitful intersection for the two fields' methodologies to be integrated because accurately tracking learning and forgetting over time and predicting future performance based on learning histories are central to developing effective, personalized learning tools. Here, we show how a state-of-the-art ML model can be enhanced by incorporating insights from a cognitive model of human memory. This was done by exploiting the predictive performance equation's (PPE) narrow but highly specialized domain knowledge with regard to the temporal dynamics of learning and forgetting. Specifically, the PPE was used to engineer timing-related input features for a gradient-boosted decision trees (GBDT) model. The resulting PPE-enhanced GBDT outperformed the default GBDT, especially under conditions in which limited data were available for training. Results suggest that integrating cognitive and ML models could be particularly productive if the available data are too high-dimensional to be explained by a cognitive model but not sufficiently large to effectively train a modern ML algorithm. Here, the cognitive model's insights pertaining to only one aspect of the data were enough to jump-start the ML model's ability to make predictions-a finding that holds promise for future explorations.

SpeedyIBL: A comprehensive, precise, and fast implementation of instance-based learning theory

Instance-based learning theory (IBLT) is a comprehensive account of how humans make decisions from experience during dynamic tasks. Since it was first proposed almost two decades ago, multiple computational models have been constructed based on IBLT (i.e., IBL models). These models have been demonstrated to be very successful in explaining and predicting human decisions in multiple decision-making contexts. However, as IBLT has evolved, the initial description of the theory has become less precise, and it is unclear how its demonstration can be expanded to more complex, dynamic, and multi-agent environments. This paper presents an updated version of the current theoretical components of IBLT in a comprehensive and precise form. It also provides an advanced implementation of the full set of theoretical mechanisms, SpeedyIBL, to unlock the capabilities of IBLT to handle a diverse taxonomy of individual and multi-agent decision-making problems. SpeedyIBL addresses a practical computational issue in past implementations of IBL models, the curse of exponential growth, that emerges from memory-based tabular computations. When more observations accumulate over time, there is an exponential growth of the memory of instances that leads directly to an exponential slowdown of the computational time. Thus, SpeedyIBL leverages parallel computation with vectorization to speed up the execution time of IBL models. We evaluate the robustness of SpeedyIBL over an existing implementation of IBLT in decision games of increased complexity. The results not only demonstrate the applicability of IBLT through a wide range of decision-making tasks, but also highlight the improvement of SpeedyIBL over its prior implementation as the complexity of decision features the of agents increase. The library is open sourced for the use of the broad research community.

Sequential consumer choice as multi-cued retrieval

Whether adding songs to a playlist or groceries during an online shop, how do we decide what to choose next? We develop a model that predicts such open-ended, sequential choices using a process of cued retrieval from long-term memory. Using the past choice to cue subsequent retrievals, this model predicts the sequential purchases and response times of nearly 5 million grocery purchases made by more than 100,000 online shoppers. Products can be associated in different ways, such as by their episodic association or semantic overlap, and we find that consumers query multiple forms of associative knowledge when retrieving options. Attending to certain knowledge sources, as estimated by our model, predicts important retrieval errors, such as the propensity to forget or add unwanted products. Our results demonstrate how basic memory retrieval mechanisms shape choices in real-world, goal-directed tasks.

From convolutional neural networks to models of higher-level cognition (and back again)

The remarkable successes of convolutional neural networks (CNNs) in modern computer vision are by now well known, and they are increasingly being explored as computational models of the human visual system. In this paper, we ask whether CNNs might also provide a basis for modeling higher-level cognition, focusing on the core phenomena of similarity and categorization. The most important advance comes from the ability of CNNs to learn high-dimensional representations of complex naturalistic images, substantially extending the scope of traditional cognitive models that were previously only evaluated with simple artificial stimuli. In all cases, the most successful combinations arise when CNN representations are used with cognitive models that have the capacity to transform them to better fit human behavior. One consequence of these insights is a toolkit for the integration of cognitively motivated constraints back into CNN training paradigms in computer vision and machine learning, and we review cases where this leads to improved performance. A second consequence is a roadmap for how CNNs and cognitive models can be more fully integrated in the future, allowing for flexible end-to-end algorithms that can learn representations from data while still retaining the structured behavior characteristic of human cognition.

Using large-scale experiments and machine learning to discover theories of human decision-making

Predicting and understanding how people make decisions has been a long-standing goal in many fields, with quantitative models of human decision-making informing research in both the social sciences and engineering. We show how progress toward this goal can be accelerated by using large datasets to power machine-learning algorithms that are constrained to produce interpretable psychological theories. Conducting the largest experiment on risky choice to date and analyzing the results using gradient-based optimization of differentiable decision theories implemented through artificial neural networks, we were able to recapitulate historical discoveries, establish that there is room to improve on existing theories, and discover a new, more accurate model of human decision-making in a form that preserves the insights from centuries of research.

The More Who Die, the Less We Care: Evidence from Natural Language Analysis of Online News Articles and Social Media Posts

Considerable amount of laboratory and survey-based research finds that people show disproportional compassionate and affective response to the scope of human mortality risk. According to research on "psychic numbing," it is often the case that the more who die, the less we care. In the present article, we examine the extent of this phenomenon in verbal behavior, using large corpora of natural language to quantify the affective reactions to loss of life. We analyze valence, arousal, and specific emotional content of over 100,000 mentions of death in news articles and social media posts, and find that language shows an increase in valence (i.e., decreased negative affect) and a decrease in arousal when describing mortality of larger numbers of people. These patterns are most clearly reflected in specific emotions of joy and (in a reverse fashion) of fear and anger. Our results showcase a novel methodology for studying affective decision making, and highlight the robustness and real-world relevance of psychic numbing. They also offer new insights regarding the psychological underpinnings of psychic numbing, as well as possible interventions for reducing psychic numbing and overcoming social and psychological barriers to action in the face of the world's most serious threats.

On the Nature of Explanations Offered by Network Science: A Perspective From and for Practicing Neuroscientists

Network neuroscience represents the brain as a collection of regions and inter-regional connections. Given its ability to formalize systems-level models, network neuroscience has generated unique explanations of neural function and behavior. The mechanistic status of these explanations and how they can contribute to and fit within the field of neuroscience as a whole has received careful treatment from philosophers. However, these philosophical contributions have not yet reached many neuroscientists. Here we complement formal philosophical efforts by providing an applied perspective from and for neuroscientists. We discuss the mechanistic status of the explanations offered by network neuroscience and how they contribute to, enhance, and interdigitate with other types of explanations in neuroscience. In doing so, we rely on philosophical work concerning the role of causality, scale, and mechanisms in scientific explanations. In particular, we make the distinction between an explanation and the evidence supporting that explanation, and we argue for a scale-free nature of mechanistic explanations. In the course of these discussions, we hope to provide a useful applied framework in which network neuroscience explanations can be exercised across scales and combined with other fields of neuroscience to gain deeper insights into the brain and behavior.

Inferring latent learning factors in large-scale cognitive training data

The flexibility to learn diverse tasks is a hallmark of human cognition. To improve our understanding of individual differences and dynamics of learning across tasks, we analyse the latent structure of learning trajectories from 36,297 individuals as they learned 51 different tasks on the Lumosity online cognitive training platform. Through a data-driven modelling approach using probabilistic dimensionality reduction, we investigate covariation across learning trajectories with few assumptions about learning curve form or relationships between tasks. Modelling results show substantial covariation across tasks, such that an entirely unobserved learning trajectory can be predicted by observing trajectories on other tasks. The latent learning factors from the model include a general ability factor that is expressed mostly at later stages of practice and additional task-specific factors that carry information capable of accounting for manually defined task features and task domains such as attention, spatial processing, language and math.

Scaling up psychology via Scientific Regret Minimization

Do large datasets provide value to psychologists? Without a systematic methodology for working with such datasets, there is a valid concern that analyses will produce noise artifacts rather than true effects. In this paper, we offer a way to enable researchers to systematically build models and identify novel phenomena in large datasets. One traditional approach is to analyze the residuals of models-the biggest errors they make in predicting the data-to discover what might be missing from those models. However, once a dataset is sufficiently large, machine learning algorithms approximate the true underlying function better than the data, suggesting, instead, that the predictions of these data-driven models should be used to guide model building. We call this approach "Scientific Regret Minimization" (SRM), as it focuses on minimizing errors for cases that we know should have been predictable. We apply this exploratory method on a subset of the Moral Machine dataset, a public collection of roughly 40 million moral decisions. Using SRM, we find that incorporating a set of deontological principles that capture dimensions along which groups of agents can vary (e.g., sex and age) improves a computational model of human moral judgment. Furthermore, we are able to identify and independently validate three interesting moral phenomena: criminal dehumanization, age of responsibility, and asymmetric notions of responsibility.

A large-scale analysis of task switching practice effects across the lifespan

In modern life, we engage with many sources of information concurrently. To do this, we must continuously switch between different tasks, but this comes at a cost to performance, especially in older adults. Using a large dataset from an online cognitive-training platform, we develop a computational model of task switching that defines distinct latent measures of activating the relevant task, deactivating the irrelevant task, and making a decision. This model shows that, although task practice can improve task-switching performance, persistent costs remain even after extensive practice, and more so in older adults. We show that, with extensive task practice, older people can become functionally similar to less-practiced younger people.

An important feature of human cognition is the ability to flexibly and efficiently adapt behavior in response to continuously changing contextual demands. We leverage a large-scale dataset from Lumosity, an online cognitive-training platform, to investigate how cognitive processes involved in cued switching between tasks are affected by level of task practice across the adult lifespan. We develop a computational account of task switching that specifies the temporal dynamics of activating task-relevant representations and inhibiting task-irrelevant representations and how they vary with extended task practice across a number of age groups. Practice modulates the level of activation of the task-relevant representation and improves the rate at which this information becomes available, but has little effect on the task-irrelevant representation. While long-term practice improves performance across all age groups, it has a greater effect on older adults. Indeed, extensive task practice can make older individuals functionally similar to less-practiced younger individuals, especially for cognitive measures that focus on the rate at which task-relevant information becomes available.

Structured, uncertainty-driven exploration in real-world consumer choice

Making good decisions requires people to appropriately explore their available options and generalize what they have learned. While computational models can explain exploratory behavior in constrained laboratory tasks, it is unclear to what extent these models generalize to real-world choice problems. We investigate the factors guiding exploratory behavior in a dataset consisting of 195,333 customers placing 1,613,967 orders from a large online food delivery service. We find important hallmarks of adaptive exploration and generalization, which we analyze using computational models. In particular, customers seem to engage in uncertainty-directed exploration and use feature-based generalization to guide their exploration. Our results provide evidence that people use sophisticated strategies to explore complex, real-world environments.

The Flatland Fallacy: Moving Beyond Low-Dimensional Thinking

Psychology is a complicated science. It has no general axioms or mathematical proofs, is rarely directly observable, and is the only discipline in which the subject matter (i.e., human psychological phenomena) is also the tool of investigation. Like the Flatlanders in Edwin Abbot's famous short story (), we may be led to believe that the parsimony offered by our low-dimensional theories reflects the reality of a much higher-dimensional problem. Here we contend that this "Flatland fallacy" leads us to seek out simplified explanations of complex phenomena, limiting our capacity as scientists to build and communicate useful models of human psychology. We suggest that this fallacy can be overcome through (a) the use of quantitative models, which force researchers to formalize their theories to overcome this fallacy, and (b) improved quantitative training, which can build new norms for conducting psychological research.

Affective responses to uncertain real-world outcomes: Sentiment change on Twitter

We use data from Twitter.com to study the interplay between affect and expectations about uncertain outcomes. In two studies, we obtained tweets about candidates in the 2014 US Senate elections and tweets about National Football League (NFL) teams in the 2014/2015 NFL season. We chose these events because a) their outcomes are highly uncertain and b) they attract a lot of attention and feature heavily in the communication on social media. We also obtained a priori expectations for the events from political forecasting and sport betting websites. Using this quasi-experimental design, we found that unexpected events are associated with more intense affect than expected events. Moreover, the effect of expectations is larger for outcomes that fall below expectations than outcomes that exceed expectations. Our results are consistent with fundamental principles in psychological science, such as reference-dependence in experienced affect. We discuss how naturally occurring online data can be used to test psychological predictions and develop novel psychological insights.

Retooling Computational Techniques for EEG-Based Neurocognitive Modeling of Children's Data, Validity and Prospects for Learning and Education

This paper describes continuing research on the building of neurocognitive models of the internal mental and brain processes of children using a novel adapted combination of existing computational approaches and tools, and using electro-encephalographic (EEG) data to validate the models. The guiding working model which was pragmatically selected for investigation was the established and widely used Adaptive Control of Thought-Rational (ACT-R) modeling architecture from cognitive science. The anatomo-functional circuitry covered by ACT-R is validated by MRI-based neuroscience research. The present experimental data was obtained from a cognitive neuropsychology study involving preschool children (aged 4-6), which measured their visual selective attention and word comprehension behaviors. The collection and analysis of Event-Related Potentials (ERPs) from the EEG data allowed for the identification of sources of electrical activity known as dipoles within the cortex, using a combination of computational tools (Independent Component Analysis, FASTICA; EEG-Lab DIPFIT). The results were then used to build neurocognitive models based on Python ACT-R such that the patterns and the timings of the measured EEG could be reproduced as simplified symbolic representations of spikes, built through simplified electric-field simulations. The models simulated ultimately accounted for more than three-quarters of variations spatially and temporally in all electrical potential measurements (fit of model to dipole data expressed as R 2 ranged between 0.75 and 0.98; P < 0.0001). Implications for practical uses of the present work are discussed for learning and educational applications in non-clinical and special needs children's populations, and for the possible use of non-experts (teachers and parents).

What makes rhythms hard to perform? An investigation using Steve Reich's Clapping Music

Clapping Music is a minimalist work by Steve Reich based on twelve phased variations of a rhythmic pattern. It has been reimagined as a game-based mobile application, designed with a dual purpose. First, to introduce new audiences to the Minimalist genre through interaction with the piece presented as an engaging game. Second, to use large-scale data collection within the app to address research questions about the factors determining rhythm production performance. The twelve patterns can be differentiated using existing theories of rhythmic complexity. Using performance indicators from the game such as tap accuracy we can determine which patterns players found most challenging and so assess hypotheses from theoretical models with empirical evidence. The app has been downloaded over 140,000 times since the launch in July 2015, and over 46 million rows of gameplay data have been collected, requiring a big data approach to analysis. The results shed light on the rhythmic factors contributing to performance difficulty and show that the effect of making a transition from one pattern to the next is as significant, in terms of pattern difficulty, as the inherent complexity of the pattern itself. Challenges that arose in applying this novel approach are discussed.

Association and response accuracy in the wild

We studied contestant accuracy and error in a popular television quiz show, "Jeopardy!" Using vector-based knowledge representations obtained from distributional models of semantic memory, we computed the strength of association between clues and responses in over 5,000 televised games. Such representations have been shown to play a key role in memory and judgment, and consistent with this work, we find that contestants are more likely to provide correct responses when these responses are strongly associated with their clues, and more likely to provide incorrect responses when correct responses are weakly or negatively associated with their clues. This effect is stronger for easier questions with low monetary values, and for questions in which contestants compete to respond quickly. Our results show how distributional models of semantic memory can be used to predict human behavior in naturalistic high-level judgment tasks with skilled participants and significant monetary and social incentives.

Compression in cultural evolution: Homogeneity and structure in the emergence and evolution of a large-scale online collaborative art project

Cultural evolutionary theory provides a framework for explaining change in population-level distributions. A consistent finding in the literature is that multiple transmission episodes shape a distribution of cultural traits to become more compressible, i.e., a set of derived traits are more compressed than their ancestral forms. Importantly, this amplification of compressible patterns can become manifest in two ways, either via the homogenisation of variation or through the organisation of variation into structured and specialised patterns. Using a novel, large-scale dataset from Reddit Place, an online collaborative art project, we investigate the emergence and evolution of compressible patterns on a 1000x1000 pixel canvas. Here, all Reddit users could select a coloured pixel, place it on the canvas, and then wait for a fixed period before placing another pixel. By analysing all 16.5 million pixel placements by over 1 million individuals, we found that compression follows a quadratic trajectory through time. From a non-structured state, where individual artworks exist relatively independently from one another, Place gradually transitions to a structured state where pixel placements form specialised, interdependent patterns.

Cognitive computational neuroscience

To learn how cognition is implemented in the brain, we must build computational models that can perform cognitive tasks, and test such models with brain and behavioral experiments. Cognitive science has developed computational models that decompose cognition into functional components. Computational neuroscience has modeled how interacting neurons can implement elementary components of cognition. It is time to assemble the pieces of the puzzle of brain computation and to better integrate these separate disciplines. Modern technologies enable us to measure and manipulate brain activity in unprecedentedly rich ways in animals and humans. However, experiments will yield theoretical insight only when employed to test brain-computational models. Here we review recent work in the intersection of cognitive science, computational neuroscience and artificial intelligence. Computational models that mimic brain information processing during perceptual, cognitive and control tasks are beginning to be developed and tested with brain and behavioral data.

Latent topics resonance in scientific literature and commentaries: evidences from natural language processing approach

Resonance is generally used as a metaphor to describe the manner how the information from different sources is combined. Although it is an attractive and fundamental phenomenon in human behavior studies, most studies observed semantic resonances in well-controlled experimental settings at word level. To make up the missing link between word and document level resonances, we devoted our contributions to topic resonances in a novel and natural setting: academic commentaries. Ninety-three academic commentaries from ninety-three authors, along with their references and original papers, are analyzed by a latent Dirichlet allocation based natural language processing approach. This approach can decompose a corpus written and read by an author into several topics with different weights, which can reveal the phenomena ignored at word or document level. We found that (1) topic resonances commonly exist between commenters' fundamental input and output topics; (2) output words are re-allocated by commenters to echo salient input topics; (3) commenters are more prone to associate references which focus on the non-dominant input topics; and (4) topic resonance can even be predicted by a Hebbian-like model which matches the aforementioned findings. These findings will continue to enrich our understanding on the relationship among probe, feedback and context.

Finding the traces of behavioral and cognitive processes in big data and naturally occurring datasets

Today, people generate and store more data than ever before as they interact with both real and virtual environments. These digital traces of behavior and cognition offer cognitive scientists and psychologists an unprecedented opportunity to test theories outside the laboratory. Despite general excitement about big data and naturally occurring datasets among researchers, three "gaps" stand in the way of their wider adoption in theory-driven research: the imagination gap, the skills gap, and the culture gap. We outline an approach to bridging these three gaps while respecting our responsibilities to the public as participants in and consumers of the resulting research. To that end, we introduce Data on the Mind ( http://www.dataonthemind.org ), a community-focused initiative aimed at meeting the unprecedented challenges and opportunities of theory-driven research with big data and naturally occurring datasets. We argue that big data and naturally occurring datasets are most powerfully used to supplement-not supplant-traditional experimental paradigms in order to understand human behavior and cognition, and we highlight emerging ethical issues related to the collection, sharing, and use of these powerful datasets.

Promises and pitfalls of Web-based experimentation in the advance of replicable psychological science: A reply to Plant (2015)

In a recent letter, Plant (2015) reminded us that proper calibration of our laboratory experiments is important for the progress of psychological science. Therefore, carefully controlled laboratory studies are argued to be preferred over Web-based experimentation, in which timing is usually more imprecise. Here we argue that there are many situations in which the timing of Web-based experimentation is acceptable and that online experimentation provides a very useful and promising complementary toolbox to available lab-based approaches. We discuss examples in which stimulus calibration or calibration against response criteria is necessary and situations in which this is not critical. We also discuss how online labor markets, such as Amazon's Mechanical Turk, allow researchers to acquire data in more diverse populations and to test theories along more psychological dimensions. Recent methodological advances that have produced more accurate browser-based stimulus presentation are also discussed. In our view, online experimentation is one of the most promising avenues to advance replicable psychological science in the near future.

Efficient n-gram analysis in R with cmscu

We present a new R package, cmscu, which implements a Count-Min-Sketch with conservative updating (Cormode and Muthukrishnan Journal of Algorithms, 55(1), 58-75, 2005), and its application to n-gram analyses (Goyal et al. 2012). By writing the core implementation in C++ and exposing it to R via Rcpp, we are able to provide a memory-efficient, high-throughput, and easy-to-use library. As a proof of concept, we implemented the computationally challenging (Heafield et al. 2013) modified Kneser-Ney n-gram smoothing algorithm using cmscu as the querying engine. We then explore information density measures (Jaeger Cognitive Psychology, 61(1), 23-62, 2010) from n-gram frequencies (for n=2,3) derived from a corpus of over 2.2 million reviews provided by a Yelp, Inc. dataset. We demonstrate that these text data are at a scale beyond the reach of other more common, more general-purpose libraries available through CRAN. Using the cmscu library and the smoothing implementation, we find a positive relationship between review information density and reader review ratings. We end by highlighting the important use of new efficient tools to explore behavioral phenomena in large, relatively noisy data sets.

Hidden processes in structural representations: A reply to Abbott, Austerweil, and Griffiths (2015)

In recent work exploring the semantic fluency task, we found evidence indicative of optimal foraging policies in memory search that mirror search in physical environments. We determined that a 2-stage cue-switching model applied to a memory representation from a semantic space model best explained the human data. Abbott, Austerweil, and Griffiths demonstrate how these patterns could also emerge from a random walk applied to a network representation of memory based on human free-association norms. However, a major representational issue limits any conclusions that can be drawn about the process model comparison: Our process model operated on a memory space constructed from a learning model, whereas their model used human behavioral data from a task that is quite similar to the behavior they attempt to explain. Predicting semantic fluency (e.g., how likely it is to say cat after dog in a sequence of animals) from free association (how likely it is to say cat when given dog as a cue) should be possible with a relatively simple retrieval mechanism. The 2 tasks both tap memory, but they also share a common process of retrieval. Assuming that semantic memory is a network from free-association behavior embeds variance due to the shared retrieval process directly into the representation. A simple process mechanism is then sufficient to simulate semantic fluency because much of the requisite process complexity may already be hidden in the representation. (PsycINFO Database Record