Can we reliably measure the general factor of intelligence (g) through commercial video games? Yes, we can!

Assessment of executive functions using a 3D-video game in children and adolescents with ADHD

Objective

Executive functions are important factors that affect the well-being of children with ADHD. Therefore, inclusion of a convenient assessment of executive dysfunction in diagnosis and treatment of ADHD patients is warranted. However, executive dysfunction assessment presently relies on lab-based neuropsychological tests and symptom rating scales. The present study examined the potential of a 3-D action puzzle video game to reflect ecologically valid executive functioning in pediatric ADHD patients.

Methods

Participant gameplay metrics were compared to both their Cambridge Neuropsychological Test Automated Battery (CANTAB) and Conners 3 Parent Form's executive functioning subscale scores. Participants consisted of 33 ADHD and non-ADHD patients aged 8-21.

Results

Metrics from gameplay were associated with distinct CANTAB test scores, and a composite score from gameplay was significantly correlated with executive dysfunction from Conners 3.

Conclusion

For children with ADHD, cognitive domains related to executive function and overall real-life executive functioning appear to both be measurable via video games. It may be possible to develop individualized behavioral therapy based on the quantitative data obtained from the video game used in this study.

Checking Different Video Game Mechanics to Assess Cognitive Abilities in Groups with and without Emotional Problems

This study assesses cognitive abilities through video games for entertainment (Blek, Edge, and Unpossible) that were programmed from scratch to record players' behavior and the levels achieved in a sample without emotional problems and in one with emotional problems. The non-emotional-problem sample was recruited from three universities and two bachelor's degree programs. The emotional-problem sample was recruited from two outpatient centers. The participants in the emotional-problem sample completed reduced versions of the ability tests and video games, as required by their emotional problems. Three subtests of the Differential Aptitude Test that assessed abstract reasoning, visuospatial reasoning, and perceptual speed were selected as ability tests. All participants were required to complete a mental health questionnaire (PROMIS) and a brief questionnaire on their gaming habits and previous experience with the video games used. The results that were obtained showed good convergent validity of the video games as measures of cognitive abilities, and they showed that the behavior of players in the sample without emotional problems while playing predicted the level achieved in the Blek and Unpossible game fragments, but this was only true for Unpossible in the emotional-problem sample; finally, shorter versions of the Blek and Edge game fragments can be used because they maintain their good psychometric properties.

Game based assessments of cognitive ability in recruitment: Validity, fairness and test-taking experience

Gamification and machine learning are emergent technologies in recruitment, promising to improve the user experience and fairness of assessments. We test this by validating a game based assessment of cognitive ability with a machine learning based scoring algorithm optimised for validity and fairness. We use applied data from 11,574 assessment completions. The assessment has convergent validity (r = 0.5) and test-retest reliability (r = 0.68). It maintains fairness in a separate sample of 3,107 job applicants, showing that fairness-optimised machine learning can improve outcome parity issues with cognitive ability tests in recruitment settings. We show that there are no significant gender differences in test taking anxiety resulting from the games, and that anxiety does not directly predict game performance, supporting the notion that game based assessments help with test taking anxiety. Interactions between anxiety, gender and performance are explored. Feedback from 4,778 job applicants reveals a Net Promoter score of 58, indicating more applicants support than dislike the assessment, and that games deliver a positive applicant experience in practise. Satisfaction with the format is high, but applicants raise face validity concerns over the abstract games. We encourage the use of gamification and machine learning to improve the fairness and user experience of psychometric tests.

Cognitive Dyadic Measurements: A Game-Changer? Construction and First Validation of Three Cognitively Demanding Competitive Tasks

Competition among individuals is a natural mode of determining who is fittest. While in nature, economics, and sports, it is common to infer ability or aptitude from the outcome of competitions, our knowledge on its effects in regard to psychological/educational assessment is scarce. In the present pilot study, we explore a measurement approach for assessing individual differences in interpersonal, face-to-face competitions, based on a set of cognitively demanding, competitive, fast-paced, two-opponent tasks. For initial task evaluation, we conducted comprehensive reliability and construct validation analyses, considering cognitive ability, motivation, and personality measures. Moreover, using structural equation models we conducted a simultaneous factorization of the tasks with the other validation measures. The results suggest that the newly developed tasks measure both cognitive ability (intelligence) as well as a competition-specific component. The competition-specific component was positively associated with experience in competitive gaming and negatively correlated with neuroticism. While the pattern of validities was promising, the measurements’ reliabilities were yet unsatisfactory. Implications for future research as well as the design of competition-based measurements are discussed.

Novel 3-D action video game mechanics reveal differentiable cognitive constructs in young players, but not in old

Video game research predominantly uses a “one game-one function” approach—researchers deploy a constellation of task-like minigames to span multiple domains or consider a complex video game to essentially represent one cognitive construct. To profile cognitive functioning in a more ecologically valid setting, we developed a novel 3-D action shooter video game explicitly designed to engage multiple cognitive domains. We compared gameplay data with results from a web-based cognitive battery (WebCNP) for 158 participants (aged 18–74). There were significant negative main effects on game performance from age and gender, even when controlling for prior video game exposure. Among younger players, game mechanics displayed significant and unique correlations to cognitive constructs such as aim accuracy with attention and stealth with abstract thinking within the same session. Among older players the relation between game components and cognitive domains was unclear. Findings suggest that while game mechanics within a single game can be deconstructed to correspond to existing cognitive metrics, how game mechanics are understood and utilized likely differs between the young and old. We argue that while complex games can be utilized to measure distinct cognitive functions, the translation scheme of gameplay to cognitive function should not be one-size-fits-all across all demographics.

Predicting verbal reasoning from virtual community membership in a sample of Russian young adults

Predicting personality traits from social networking site profiles can help to assess individual differences in verbal reasoning without using long questionnaires. Inspired by earlier studies, which investigated whether abstract-thinking ability are predictable by social networking sites data, we used supervised machine learning to predict verbal-reasoning ability based on a proposed set of features extracted from virtual community membership. A large sample (N = 3,646) of Russian young adults aged 18-22 years approved access to the data from their social networking accounts and completed an online test on verbal reasoning. We experimented with binary classification machine-learning models for verbal-reasoning prediction. Prediction performance was tested on isolated control subsamples for men and women. The results of prediction on AUC-ROC metrics for control subsamples over 0.7 indicated reasonably good performance on predicting verbal-reasoning level. We also investigated the contribution of virtual community's genres to verbal reasoning level prediction for male and female participants. Theoretical interpretations of results stemming from both Vygotsky's sociocultural theory and behavioural genomics are discussed, including the implication that virtual communities make up a non-shared environment that can cause variance in verbal reasoning. We intend to conduct studies to explore the implications of the results further.

Student Characteristics, Institutional Factors, and Outcomes in Higher Education and Beyond: An Analysis of Standardized Test Scores and Other Factors at the Institutional Level with School Rankings and Salary

When seeking to explain the eventual outcomes of a higher education experience, do the personal attributes and background factors students bring to college matter more than what the college is able to contribute to the development of the student through education or other institutional factors? Most education studies tend to simply ignore cognitive aptitudes and other student characteristics—in particular the long history of research on this topic—since the focus is on trying to assess the impact of education. Thus, the role of student characteristics has in many ways been underappreciated in even highly sophisticated quantitative education research. Conversely, educational and institutional factors are not as prominent in studies focused on cognitive aptitudes, as these fields focus first on reasoning capacity, and secondarily on other factors. We examine the variance in student outcomes due to student (e.g., cognitive aptitudes) versus institutional characteristics (e.g., teachers, schools). At the level of universities, two contemporary U.S. datasets are used to examine the proportion of variance accounted for in various university rankings and long-run salary by student cognitive characteristics and institutional factors. We find that depending upon the ways the variables are entered into regression models, the findings are somewhat different. We suggest some fruitful paths forward which might integrate the methods and findings showing that teachers and schools matter, along with the broader developmental bounds within which these effects take place.

Measuring Personality through Images: Validating a Forced-Choice Image-Based Assessment of the Big Five Personality Traits

Selection methods are commonly used in talent acquisition to predict future job performance and to find the best candidates, but questionnaire-based assessments can be lengthy and lead to candidate fatigue and poor engagement, affecting completion rates and producing poor data. Gamification can mitigate some of these issues through greater engagement and shorter testing times. One avenue of gamification is image-based tests. Although such assessments are starting to gain traction in personnel selection, few studies describing their validity and psychometric properties exist. The current study explores the potential of a five-minute, forced-choice, image-based assessment of the Big Five personality traits to be used in selection. Study 1 describes the creation of the image pairs and the selection of the 150 best-performing items based on a sample of 300 respondents. Study 2 describes the creation of machine-learning-based scoring algorithms and tests of their convergent and discriminate validity and adverse impact based on a sample of 431 respondents. All models showed good levels of convergent validity with the IPIP-NEO-120 (openness r = 0.71, conscientiousness r = 0.70, extraversion r = 0.78, agreeableness r = 0.60, and emotional stability r = 0.70) and were largely free from potential adverse impact. The implications for recruitment policy and practice and the need for further validation are discussed.

Dissecting Digital Card Games to Yield Digital Biomarkers for the Assessment of Mild Cognitive Impairment: Methodological Approach and Exploratory Study

Background

Mild cognitive impairment (MCI), the intermediate cognitive status between normal cognitive decline and pathological decline, is an important clinical construct for signaling possible prodromes of dementia. However, this condition is underdiagnosed. To assist monitoring and screening, digital biomarkers derived from commercial off-the-shelf video games may be of interest. These games maintain player engagement over a longer period of time and support longitudinal measurements of cognitive performance.

Objective

This paper aims to explore how the player actions of Klondike Solitaire relate to cognitive functions and to what extent the digital biomarkers derived from these player actions are indicative of MCI.

Methods

First, 11 experts in the domain of cognitive impairments were asked to correlate 21 player actions to 11 cognitive functions. Expert agreement was verified through intraclass correlation, based on a 2-way, fully crossed design with type consistency. On the basis of these player actions, 23 potential digital biomarkers of performance for Klondike Solitaire were defined. Next, 23 healthy participants and 23 participants living with MCI were asked to play 3 rounds of Klondike Solitaire, which took 17 minutes on average to complete. A generalized linear mixed model analysis was conducted to explore the differences in digital biomarkers between the healthy participants and those living with MCI, while controlling for age, tablet experience, and Klondike Solitaire experience.

Results

All intraclass correlations for player actions and cognitive functions scored higher than 0.75, indicating good to excellent reliability. Furthermore, all player actions had, according to the experts, at least one cognitive function that was on average moderately to strongly correlated to a cognitive function. Of the 23 potential digital biomarkers, 12 (52%) were revealed by the generalized linear mixed model analysis to have sizeable effects and significance levels. The analysis indicates sensitivity of the derived digital biomarkers to MCI.

Conclusions

Commercial off-the-shelf games such as digital card games show potential as a complementary tool for screening and monitoring cognition.

Trial Registration

ClinicalTrials.gov NCT02971124; https://clinicaltrials.gov/ct2/show/NCT02971124

Examining the Use of Game-Based Assessments for Hiring Autistic Job Seekers

Although people with autism are protected under the Americans with Disabilities Act of 1990, there is little theoretical or practical effort to determine whether traditional pre-employment assessments unfairly impact autistic job seekers. Due to the lack of emphasis on social communication, game-based assessments (GBAs) may offer a way of assessing candidate ability without disadvantaging autistic candidates. A total of 263 autistic job seekers took one of two game-based assessment packages designed to measure cognitive ability. After comparing their results to 323 college-graduate job seekers in the general population, we found that performance on the GBAs was generally similar in both populations, although some small differences were detected. Implications for hiring decisions are discussed.

The potential of game- and video-based assessments for social attributes: examples from practice

Purpose

The purpose of this study is to describe the development and psychometric properties of a novel game- and video-based assessment of social attributes. Despite their increasing adaption, little research is available on the suitability of games and video analytics for measuring noncognitive attributes in the selection context.

Design/methodology/approach

The authors describe three novel assessments and their psychometric properties in a sample of 1,300 participants: a game-based adaptation of an Emotion Recognition Task, a chatbot-based situational judgment test for emotion management and a video-based conscientiousness assessment.

Findings

The novel assessments show good to moderate convergent validity for Emotional Recognition (r = 0.42), Emotion Management (r = 0.39) and Conscientiousness (r = 0.21). The video-based assessment demonstrates preliminary predictive validity for self-reported work performance. Novel game-based assessments (GBAs) are perceived as better designed and more immersive than traditional questionnaires. Adverse impact analysis indicates small group differences by age, gender and ethnicity.

Research limitations/implications

Predictive validity findings need to be replicated using objective measures of performance, such as performance ratings by supervisors and extended to the GBAs. Adverse impact should be evaluated using a real-life applicant pool and extended to additional groups.

Practical implications

Evidence for the psychometric validity of novel assessment formats supports their adoption in selection and recruitment. Improved user experience and shortened assessment times open up new areas of application.

Originality/value

This study gives first insights into psychometric properties of video- and game-based assessments of social attributes.

Gamification in human resource management—Status quo and quo vadis

Gamification has recently been presented as a promising opportunity to improve human resource management (HRM) practices and tools. However, while the number of publications on gamification has been increasing in recent years, an overview of the current landscape of HRM-related literature of gamification is missing so far. Intending to support and ease the understanding of prior research findings in this field, this article conducts a systematic literature review. This study contributes to the field of human resources research by examining 45 research papers, aiming to explore areas of application and outcomes of the use of gamification in HRM. Propositions are outlined along with elaborating risks and approaches on how to mitigate the risks of using game design elements in HRM.

Stronger Together: Personality, Intelligence and the Assessment of Career Potential

Personality and intelligence have a long history in applied psychology, with research dating back more than 100 years. In line, early developments in industrial-organizational psychology were largely founded on the predictive power of personality and intelligence measures vis-à-vis career-related outcomes. However, despite a wealth of evidence in support of their utility, the concepts, theories, and measures of personality and intelligence are still widely underutilized in organizations, even when these express a commitment to making data-driven decisions about employees and leaders. This paper discusses the value of personality and intelligence to understand individual differences in career potential, and how to increase the adoption of theories and tools for evaluating personality and intelligence in real-world organizational contexts. Although personality and intelligence are distinct constructs, the assessment of career potential is incomplete without both.

Exploring the relationship between video game expertise and fluid intelligence

Hundreds of millions of people play intellectually-demanding video games every day. What does individual performance on these games tell us about cognition? Here, we describe two studies that examine the potential link between intelligence and performance in one of the most popular video games genres in the world (Multiplayer Online Battle Arenas: MOBAs). In the first study, we show that performance in the popular MOBA League of Legends’ correlates with fluid intelligence as measured under controlled laboratory conditions. In the second study, we also show that the age profile of performance in the two most widely-played MOBAs (League of Legends and DOTA II) matches that of raw fluid intelligence. We discuss and extend previous videogame literature on intelligence and videogames and suggest that commercial video games can be useful as 'proxy' tests of cognitive performance at a global population level.

What Older People Like to Play: Genre Preferences and Acceptance of Casual Games

BACKGROUND

In recent computerized cognitive training studies, video games have emerged as a promising tool that can benefit cognitive function and well-being. Whereas most video game training studies have used first-person shooter (FPS) action video games, subsequent studies found that older adults dislike this type of game and generally prefer casual video games (CVGs), which are a subtype of video games that are easy to learn and use simple rules and interfaces. Like other video games, CVGs are organized into genres (eg, puzzle games) based on the rule-directed interaction with the game. Importantly, game genre not only influences the ease of interaction and cognitive abilities CVGs demand, but also affects whether older adults are willing to play any particular genre. To date, studies looking at how different CVG genres resonate with older adults are lacking.

OBJECTIVE

The aim of this study was to investigate how much older adults enjoy different CVG genres and how favorably their CVG characteristics are rated.

METHODS

A total of 16 healthy adults aged 65 years and above playtested 7 CVGs from 4 genres: casual action, puzzle, simulation, and strategy video games. Thereafter, they rated casual game preference and acceptance of casual game characteristics using 4 scales from the Core Elements of the Gaming Experience Questionnaire (CEGEQ). For this, participants rated how much they liked the game (enjoyment), understood the rules of the game (game-play), learned to manipulate the game (control), and make the game their own (ownership).

RESULTS

Overall, enjoyment and acceptance of casual game characteristics was high and significantly above the midpoint of the rating scale for all CVG genres. Mixed model analyses revealed that ratings of enjoyment and casual game characteristics were significantly influenced by CVG genre. Participants' mean enjoyment of casual puzzle games (mean 0.95 out of 1.00) was significantly higher than that for casual simulation games (mean 0.75 and 0.73). For casual game characteristics, casual puzzle and simulation games were given significantly higher game-play ratings than casual action games. Similarly, participants' control ratings for casual puzzle games were significantly higher than that for casual action and simulation games. Finally, ownership was rated significantly higher for casual puzzle and strategy games than for casual action games.

CONCLUSIONS

The findings of this study show that CVGs have characteristics that are suitable and enjoyable for older adults. In addition, genre was found to influence enjoyment and ratings of CVG characteristics, indicating that puzzle games are particularly easy to understand, learn, and play, and are enjoyable. Future studies should continue exploring the potential of CVG interventions for older adults in improving cognitive function, everyday functioning, and well-being. We see particular potential for CVGs in people suffering from cognitive impairment due to dementia or brain injury.

Examining the Roles of Reasoning and Working Memory in Predicting Casual Game Performance across Extended Gameplay

The variety and availability of casual video games presents an exciting opportunity for applications such as cognitive training. Casual games have been associated with fluid abilities such as working memory (WM) and reasoning, but the importance of these cognitive constructs in predicting performance may change across extended gameplay and vary with game structure. The current investigation examined the relationship between cognitive abilities and casual game performance over time by analyzing first and final session performance over 4-5 weeks of game play. We focused on two groups of subjects who played different types of casual games previously shown to relate to WM and reasoning when played for a single session: (1) puzzle-based games played adaptively across sessions and (2) speeded switching games played non-adaptively across sessions. Reasoning uniquely predicted first session casual game scores for both groups and accounted for much of the relationship with WM. Furthermore, over time, WM became uniquely important for predicting casual game performance for the puzzle-based adaptive games but not for the speeded switching non-adaptive games. These results extend the burgeoning literature on cognitive abilities involved in video games by showing differential relationships of fluid abilities across different game types and extended play. More broadly, the current study illustrates the usefulness of using multiple cognitive measures in predicting performance, and provides potential directions for game-based cognitive training research.

Advances in Intelligence Research: What Should be Expected in the XXI Century (Questions & Answers)

Here I briefly delineate my view about the main question of this International Seminar, namely, what should we expecting from the XXI Century regarding the advancements in intelligence research. This view can be summarized as 'The Brain Connection' (TBC), meaning that neuroscience will be of paramount relevance for increasing our current knowledge related to the key question: why are some people smarter than others? We need answers to the issue of what happens in our brains when the genotype and the environment are integrated. The scientific community has devoted great research efforts, ranging from observable behavior to hidden genetics, but we are still far from having a clear general picture of what it means to be more or less intelligent. After the discussion held with the panel of experts participating in the seminar, it is concluded that advancements will be more solid and safe increasing the collaboration of scientists with shared research interests worldwide. Paralleling current sophisticated analyses of how the brain computes, nowadays science may embrace a network approach.

The Measurement of Intelligence in the XXI Century using Video Games

This paper reviews the use of video games for measuring intelligence differences and reports two studies analyzing the relationship between intelligence and performance on a leisure video game. In the first study, the main focus was to design an Intelligence Test using puzzles from the video game. Forty-seven young participants played "Professor Layton and the curious village"® for a maximum of 15 hours and completed a set of intelligence standardized tests. Results show that the time required for completing the game interacts with intelligence differences: the higher the intelligence, the lower the time (d = .91). Furthermore, a set of 41 puzzles showed excellent psychometric properties. The second study, done seven years later, confirmed the previous findings. We finally discuss the pros and cons of video games as tools for measuring cognitive abilities with commercial video games, underscoring that psychologists must develop their own intelligence video games and delineate their key features for the measurement devices of next generation.