Game‐XP: Action Games as Experimental Paradigms for Cognitive Science

Tracking the Cognitive Band in an Open-Ended Task

Open-ended tasks can be decomposed into the three levels of Newell's Cognitive Band: the Unit-Task level, the Operation level, and the Deliberate-Act level. We analyzed the video game Co-op Space Fortress at these levels, reporting both the match of a cognitive model to subject behavior and the use of electroencephalogram (EEG) to track subject cognition. The Unit Task level in this game involves coordinating with a partner to kill a fortress. At this highest level of the Cognitive Band, there is a good match between subject behavior and the model. The EEG signals were also strong enough to track when Unit Tasks succeeded or failed. The intermediate Operation level in this task involves legs of flight to achieve a kill. The EEG signals associated with these operations are much weaker than the signals associated with the Unit Tasks. Still, it was possible to reconstruct subject play with much better than chance success. There were significant differences in the leg behavior of subjects and models. Model behavior did not provide a good basis for interpreting a subject's behavior at this level. At the lowest Deliberate-Act level, we observed overlapping key actions, which the model did not display. Such overlapping key actions also frustrated efforts to identify EEG signals of motor actions. We conclude that the Unit-task level is the appropriate level both for understanding open-ended tasks and for using EEG to track the performance of open-ended tasks.

Open design of a reproducible videogame controller for MRI and MEG

Videogames are emerging as a promising experimental paradigm in neuroimaging. Acquiring gameplay in a scanner remains challenging due to the lack of a scanner-compatible videogame controller that provides a similar experience to standard, commercial devices. In this paper, we introduce a videogame controller designed for use in the functional magnetic resonance imaging as well as magnetoencephalography. The controller is made exclusively of 3D-printed and commercially available parts. We evaluated the quality of our controller by comparing it to a non-MRI compatible controller that was kept outside the scanner. The comparison of response latencies showed reliable button press accuracies of adequate precision. Comparison of the subjects' motion during fMRI recordings of various tasks showed that the use of our controller did not increase the amount of motion produced compared to a regular MR compatible button press box. Motion levels during an ecological videogame task were of moderate amplitude. In addition, we found that the controller only had marginal effect on temporal SNR in fMRI, as well as on covariance between sensors in MEG, as expected due to the use of non-magnetic building materials. Finally, the reproducibility of the controller was demonstrated by having team members who were not involved in the design build a reproduction using only the documentation. This new videogame controller opens new avenues for ecological tasks in fMRI, including challenging videogames and more generally tasks with complex responses. The detailed controller documentation and build instructions are released under an Open Source Hardware license to increase accessibility, and reproducibility and enable the neuroimaging research community to improve or modify the controller for future experiments.

Difficulty-skill balance does not affect engagement and enjoyment: a pre-registered study using artificial intelligence-controlled difficulty

How does the difficulty of a task affect people's enjoyment and engagement? Intrinsic motivation and flow theories posit a 'goldilocks' optimum where task difficulty matches performer skill, yet current work is confounded by questionable measurement practices and lacks scalable methods to manipulate objective difficulty-skill ratios. We developed a two-player tactical game test suite with an artificial intelligence (AI)-controlled opponent that uses a variant of the Monte Carlo Tree Search algorithm to precisely manipulate difficulty-skill ratios. A pre-registered study (n = 311) showed that our AI produced targeted difficulty-skill ratios without participants noticing the manipulation, yet different ratios had no significant impact on enjoyment or engagement. This indicates that difficulty-skill balance does not always affect engagement and enjoyment, but that games with AI-controlled difficulty provide a useful paradigm for rigorous future work on this issue.

Assessing evidence accumulation and rule learning in humans with an online game

Evidence accumulation, an essential component of perception and decision making, is frequently studied with psychophysical tasks involving noisy or ambiguous stimuli. In these tasks, participants typically receive verbal or written instructions that describe the strategy that should be used to guide decisions. Although convenient and effective, explicit instructions can influence learning and decision making strategies and can limit comparisons with animal models, in which behaviors are reinforced through feedback. Here, we developed an online video game and nonverbal training pipeline, inspired by pulse-based tasks for rodents, as an alternative to traditional psychophysical tasks used to study evidence accumulation. Using this game, we collected behavioral data from hundreds of participants trained with an explicit description of the decision rule or with experiential feedback. Participants trained with feedback alone learned the game rules rapidly and used strategies and displayed biases similar to those who received explicit instructions. Finally, by leveraging data across hundreds of participants, we show that perceptual judgments were well described by an accumulation process in which noise scaled nonlinearly with evidence, consistent with previous animal studies but inconsistent with diffusion models widely used to describe perceptual decisions in humans. These results challenge the conventional description of the accumulation process and suggest that online games provide a valuable platform to examine perceptual decision making and learning in humans. In addition, the feedback-based training pipeline developed for this game may be useful for evaluating perceptual decision making in human populations with difficulty following verbal instructions.NEW & NOTEWORTHY Perceptual uncertainty sets critical constraints on our ability to accumulate evidence and make decisions; however, its sources remain unclear. We developed a video game, and feedback-based training pipeline, to study uncertainty during decision making. Leveraging choices from hundreds of subjects, we demonstrate that human choices are inconsistent with popular diffusion models of human decision making and instead are best fit by models in which perceptual uncertainty scales nonlinearly with the strength of sensory evidence.

Conversation dynamics in a multiplayer video game with knowledge asymmetry

Despite the challenges associated with virtually mediated communication, remote collaboration is a defining characteristic of online multiplayer gaming communities. Inspired by the teamwork exhibited by players in first-person shooter games, this study investigated the verbal and behavioral coordination of four-player teams playing a cooperative online video game. The game, Desert Herding, involved teams consisting of three ground players and one drone operator tasked to locate, corral, and contain evasive robot agents scattered across a large desert environment. Ground players could move throughout the environment, while the drone operator's role was akin to that of a "spectator" with a bird's-eye view, with access to veridical information of the locations of teammates and the to-be-corralled agents. Categorical recurrence quantification analysis (catRQA) was used to measure the communication dynamics of teams as they completed the task. Demands on coordination were manipulated by varying the ground players' ability to observe the environment with the use of game "fog." Results show that catRQA was sensitive to changes to task visibility, with reductions in task visibility reorganizing how participants conversed during the game to maintain team situation awareness. The results are discussed in the context of future work that can address how team coordination can be augmented with the inclusion of artificial agents, as synthetic teammates.

Mind the gap: Distributed practice enhances performance in a MOBA game

Understanding how humans master complex skills has the potential for wide-reaching societal benefit. Research has shown that one important aspect of effective skill learning is the temporal distribution of practice episodes (i.e., distributed practice). Using a large observational sample of players (n = 162,417) drawn from a competitive and popular online game (League of Legends), we analysed the relationship between practice distribution and performance through time. We compared groups of players who exhibited different play schedules using data slicing and machine learning techniques, to show that players who cluster gameplay into shorter time frames ultimately achieve lower performance levels than those who space their games across longer time windows. Additionally, we found that the timing of intensive play periods does not affect final performance-it is the overall amount of spacing that matters. These results extend some of the key findings in the literature on practice and learning to an ecologically valid environment with huge n. We discuss our work in relation to recent studies that have examined practice effects using Big Data and suggest solutions for salient confounds.

Investigating the behavioral mechanisms of action video game effects in a complex transfer task

Over the past few decades, researchers have argued that playing action video games can substantially improve cognitive abilities and enhance learning. However, consensus has not been reached regarding the mechanisms through which action game experience facilitates superior performance on untrained perceptually and cognitively demanding transfer tasks. We argue that analysis of behaviors engaged in during transfer task performance may provide key insights into answering this question. In the current investigation, we examined potential action game effects in the context of a complex psychomotor task, the Space Fortress (SF) game, that allows for the detailed examination of player behaviors beyond aggregate score reports. Performance (game score) was compared between action video game players (VGPs) and non-gamers (nVGPs) in two different control interface conditions (keyboard or joystick), followed by analyses of behaviors associated with superior performance. Against expectations, VGPs displayed superior performance only in the keyboard condition, suggesting that the action gamer advantage may not generalize to less-familiar control interfaces. Performance advantages were specifically associated with more efficient ship control behaviors by VGPs. Findings highlight how process-tracing approaches may provide insight into the nature of, and mechanisms producing, action gamers' advantages on learning untrained tasks.

Visuospatial working memory and attention control make the difference between experts, regulars and non-players of the videogame League of Legends

Video games have been postulated as an emerging field for studying the cognition-expertise relationship. Despite this, some methodological practices hinder scientific advance (e.g., heterogeneous samples, an ambiguous definition of expertise, etc.). League of Legends (LOL) is a massively played video game with a moderately defined structure that meets the requirements to overcome current study limitations. The aim of this study was to analyze cognitive differences among expert LOL players, regular LOL players, and non-videogame players. A sample of 80 participants was enrolled in three different groups of expertise. Participants were evaluated with behavioral tests of working memory, attention, cognitive flexibility, and inhibition. Kruskal-Wallis tests for group comparison showed that the experts performed significantly better than regular players and non-videogame players in the working memory test. Significant differences were also found between players and non-videogame players in the attention test. Methodological implications for future research in neuroscience and human-computer interaction are discussed.

Cognitive & motor skill transfer across speeds: A video game study

We examined the detailed behavioral characteristics of transfer of skill and the ability of the adaptive control of thought rational (ACT-R) architecture to account for this with its new Controller module. We employed a simple action video game called Auto Orbit and investigated the control tuning of timing skills across speed perturbations of the environment. In Auto Orbit, players needed to learn to alternate turn and shot actions to blow and burst balloons under time constraints imposed by balloon resets and deflations. Cognitive and motor skill transfer was assessed both in terms of game performance and in terms of the details of their motor actions. We found that skill transfer across speeds necessitated the recalibration of action timing skills. In addition, we found that acquiring skill in Auto Orbit involved a progressive decrease in variability of behavior. Finally, we found that players with higher skill levels tended to be less variable in terms of action chunking and action timing. These findings further shed light on the complex cognitive and motor mechanisms of skill transfer across speeds in complex task environments.

Coordination Cost and Super-Efficiency in Teamwork: The Role of Communication, Psychological States, Cardiovascular Responses, and Brain Rhythms

To advance knowledge on the psychophysiological markers of "coordination cost" in team settings, we explored differences in meta-communication patterns (i.e., silence, speaking, listening, and overlap), perceived psychological states (i.e., core affect, attention, efficacy beliefs), heart rate variability (i.e., RMSSD), and brain rhythms (i.e., alpha, beta and theta absolute power) across three studies involving 48 male dyads (Mage = 21.30; SD = 2.03). Skilled participants cooperatively played three consecutive FIFA-17 (Xbox) games in a dyad against the computer, or competed against the computer in a solo condition and a dyad condition. We observed that playing in a team, in contrast to playing alone, was associated with higher alpha peak and global efficiency in the brain and, at the same time, led to an increase in focused attention as evidenced by participants' higher theta activity in the frontal lobe. Moreover, we observed that overtime participants' brain dynamics moved towards a state of "neural-efficiency", characterized by increased theta and beta activity in the frontal lobe, and high alpha activity across the whole brain. Our findings advance the literature by demonstrating that (1) the notion of coordination cost can be captured at the neural level in the initial stages of team development; (2) by decreasing the costs of switching between tasks, teamwork increases both individuals' attentional focus and global neural efficiency; and (3) communication dynamics become more proficient and individuals' brain patterns change towards neural efficiency over time, likely due to team learning and decreases in intra-team conflict.

The Impact of Domain-Specific Experience on Chess Skill: Reanalysis of a Key Study

How important are training and other forms of domain-relevant experience in predicting individual differences in expertise? To answer this question, we used structural equation modeling to reanalyze data from a study of chess by Charness, Tuffiash, Krampe, Reingold, and Vasyukova (2005). Latent variables reflecting serious chess activity and formal instruction, along with a manifest variable indexing serious starting age, accounted for 63% of the variance in peak rating. Serious starting age had a significant negative effect on peak rating (β = –.15), even after we controlled for domain-specific experience, indicating an advantage for starting earlier. We also tested the prediction that formal instruction increases the effectiveness of serious study (Ericsson & Charness, 1994) using moderated regression. This claim was not supported. Overall, the results affirm that serious study and other forms of domain-specific experience are important pieces of the expertise puzzle, but other factors must matter too.

Supplemental materials are available at https://www.press.uillinois.edu/journals/ajp/media/chess_skill

Distinguishing experts from novices by the Mind's Hand and Mind's Eye

Tetris is a complex task notable for the increasingly substantial demands it makes on perception, decision-making, and action as the game is played. To investigate these issues, we collected data on 39 features of Tetris play for each Tetris zoid (piece), for up to 16 levels of difficulty, as each of 240 players played an hour of Tetris under laboratory conditions. Using only early (level 1) data, we conducted a Principle Component Analysis which found intriguing differences among its three, statistically significant, principle components. Each of these components captures different combinations of perception, decision-making, and action which suggests differing higher level skills, tactics, and strategies. Each component is presented and discussed, and then used in a series of principle component regression analyses on subsets of these data (a) from different Tetris levels, as well as (b) from players of different levels of expertise. We validate these models with data collected at a locally held Tetris tournament. These components represent elements of expertise; namely, correlations among perceptual, decision-making, and motor features that represent processing stages and hierarchical control and which distinguish expert from novice Tetris players. These components provide evidence for an integrated complex of processes - the Mind's Hand and the Mind's Eye - that are the essence of expertise in the real-time, sequential-decision-making task of Tetris.

Looking behind the score: Skill structure explains sex differences in skilled video game performance

Some have explained large sex differences in visuospatial abilities by genetic adaptations to different roles in primitive hunter-gatherer societies and the interaction of innate biological differences and environmental factors. We explored the extent to which variations in behavior and acquired skills can provide alternative accounts for sex differences in the performance of a complex spatially-demanding video game (Space Fortress). Men and women with limited video game experience were given 30 hours of training, and latent curve analyses examined the development of their ship control performance and behavior. Men had significantly better control performance than women before and after training, but differences diminished substantially over the training period. An analysis of participants’ joystick behaviors revealed that initially men and women relied on different patterns of control behaviors, but changes in these behaviors over time accounted for the reduced sex differences in performance. When we controlled for these differences in behavior, sex effects after training were no longer significant. Finally, examining the development of control performance and control behaviors of men and women categorized as initially high and low performers revealed the lower-performing women may have been controlling their ship using an approach that was very different from the men and higher-performing women. The potential problems of analyzing men and women’s spatial performance as homogenous groups are discussed, as well as how these issues may account for sex differences in skilled video game performance and perhaps other domains involving spatial abilities.

Six Suggestions for Research on Games in Cognitive Science

Games are more varied and occupy more of daily life than ever before. At the same time, the tools available to study game play and players are more powerful than ever, especially massive data sets from online platforms and computational engines that can accurately evaluate human decisions. This essay offers six suggestions for future cognitive science research on games: (1) Don't forget about chess, (2) Look beyond action games and chess, (3) Use (near)-optimal play to understand human play and players, (4) Investigate social phenomena, (5) Raise the standards for studies of games as treatments, (6) Talk to real experts.