A Mobile App Delivering a Gamified Battery of Cognitive Tests Designed for Repeated Play (OU Brainwave): App Design and Cohort Study

Measuring the Reliability of a Gamified Stroop Task: Quantitative Experiment

BACKGROUND

Few gamified cognitive tasks are subjected to rigorous examination of psychometric properties, despite their use in experimental and clinical settings. Even small manipulations to cognitive tasks require extensive research to understand their effects.

OBJECTIVE

This study aims to investigate how game elements can affect the reliability of scores on a Stroop task. We specifically investigated performance consistency within and across sessions.

METHODS

We created 2 versions of the Stroop task, with and without game elements, and then tested each task with participants at 2 time points. The gamified task used points and feedback as game elements. In this paper, we report on the reliability of the gamified Stroop task in terms of internal consistency and test-retest reliability, compared with the control task. We used a permutation approach to evaluate internal consistency. For test-retest reliability, we calculated the Pearson correlation and intraclass correlation coefficients between each time point. We also descriptively compared the reliability of scores on a trial-by-trial basis, considering the different trial types.

RESULTS

At the first time point, the Stroop effect was reduced in the game condition, indicating an increase in performance. Participants in the game condition had faster reaction times (P=.005) and lower error rates (P=.04) than those in the basic task condition. Furthermore, the game condition led to higher measures of internal consistency at both time points for reaction times and error rates, which indicates a more consistent response pattern. For reaction time in the basic task condition, at time 1, rSpearman-Brown=0.78, 95% CI 0.64-0.89. At time 2, rSpearman-Brown=0.64, 95% CI 0.40-0.81. For reaction time, in the game condition, at time 1, rSpearman-Brown=0.83, 95% CI 0.71-0.91. At time 2, rSpearman-Brown=0.76, 95% CI 0.60-0.88. Similarly, for error rates in the basic task condition, at time 1, rSpearman-Brown=0.76, 95% CI 0.62-0.87. At time 2, rSpearman-Brown=0.74, 95% CI 0.58-0.86. For error rates in the game condition, at time 1, rSpearman-Brown=0.76, 95% CI 0.62-0.87. At time 2, rSpearman-Brown=0.74, 95% CI 0.58-0.86. Test-retest reliability analysis revealed a distinctive performance pattern depending on the trial type, which may be reflective of motivational differences between task versions. In short, especially in the incongruent trials where cognitive conflict occurs, performance in the game condition reaches peak consistency after 100 trials, whereas performance consistency drops after 50 trials for the basic version and only catches up to the game after 250 trials.

CONCLUSIONS

Even subtle gamification can impact task performance albeit not only in terms of a direct difference in performance between conditions. People playing the game reach peak performance sooner, and their performance is more consistent within and across sessions. We advocate for a closer examination of the impact of game elements on performance.

Conceptual Ambiguity Surrounding Gamification and Serious Games in Health Care: Literature Review and Development of Game-Based Intervention Reporting Guidelines (GAMING)

BACKGROUND

In health care, the use of game-based interventions to increase motivation, engagement, and overall sustainability of health behaviors is steadily becoming more common. The most prevalent types of game-based interventions in health care research are gamification and serious games. Various researchers have discussed substantial conceptual differences between these 2 concepts, supported by empirical studies showing differences in the effects on specific health behaviors. However, researchers also frequently report cases in which terms related to these 2 concepts are used ambiguously or even interchangeably. It remains unclear to what extent existing health care research explicitly distinguishes between gamification and serious games and whether it draws on existing conceptual considerations to do so.

OBJECTIVE

This study aims to address this lack of knowledge by capturing the current state of conceptualizations of gamification and serious games in health care research. Furthermore, we aim to provide tools for researchers to disambiguate the reporting of game-based interventions.

METHODS

We used a 2-step research approach. First, we conducted a systematic literature review of 206 studies, published in the Journal of Medical Internet Research and its sister journals, containing terms related to gamification, serious games, or both. We analyzed their conceptualizations of gamification and serious games, as well as the distinctions between the two concepts. Second, based on the literature review findings, we developed a set of guidelines for researchers reporting on game-based interventions and evaluated them with a group of 9 experts from the field.

RESULTS

Our results show that less than half of the concept mentions are accompanied by an explicit definition. To distinguish between the 2 concepts, we identified four common approaches: implicit distinction, synonymous use of terms, serious games as a type of gamified system, and distinction based on the full game dimension. Our Game-Based Intervention Reporting Guidelines (GAMING) consist of 25 items grouped into four topics: conceptual focus, contribution, mindfulness about related concepts, and individual concept definitions.

CONCLUSIONS

Conceptualizations of gamification and serious games in health care literature are strongly heterogeneous, leading to conceptual ambiguity. Following the GAMING can support authors in rigorous reporting on study results of game-based interventions.

The Making and Evaluation of Digital Games Used for the Assessment of Attention: Systematic Review

Background

Serious games are now widely used in many contexts, including psychological research and clinical use. One area of growing interest is that of cognitive assessment, which seeks to measure different cognitive functions such as memory, attention, and perception. Measuring these functions at both the population and individual levels can inform research and indicate health issues. Attention is an important function to assess, as an accurate measure of attention can help diagnose many common disorders, such as attention-deficit/hyperactivity disorder and dementia. However, using games to assess attention poses unique problems, as games inherently manipulate attention through elements such as sound effects, graphics, and rewards, and research on adding game elements to assessments (ie, gamification) has shown mixed results. The process for developing cognitive tasks is robust, with high psychometric standards that must be met before these tasks are used for assessment. Although games offer more diverse approaches for assessment, there is no standard for how they should be developed or evaluated.

Objective

To better understand the field and provide guidance to interdisciplinary researchers, we aim to answer the question: How are digital games used for the cognitive assessment of attention made and measured?

Methods

We searched several databases for papers that described a digital game used to assess attention that could be deployed remotely without specialized hardware. We used Rayyan, a systematic review software, to screen the records before conducting a systematic review.

Results

The initial database search returned 49,365 papers. Our screening process resulted in a total of 74 papers that used a digital game to measure cognitive functions related to attention. Across the studies in our review, we found three approaches to making assessment games: gamifying cognitive tasks, creating custom games based on theories of cognition, and exploring potential assessment properties of commercial games. With regard to measuring the assessment properties of these games (eg, how accurately they assess attention), we found three approaches: comparison to a traditional cognitive task, comparison to a clinical diagnosis, and comparison to knowledge of cognition; however, most studies in our review did not evaluate the game’s properties (eg, if participants enjoyed the game).

Conclusions

Our review provides an overview of how games used for the assessment of attention are developed and evaluated. We further identified three barriers to advancing the field: reliance on assumptions, lack of evaluation, and lack of integration and standardization. We then recommend the best practices to address these barriers. Our review can act as a resource to help guide the field toward more standardized approaches and rigorous evaluation required for the widespread adoption of assessment games.