The Social Shapes Test as a Self-Administered, Online Measure of Social Intelligence: Two Studies with Typically Developing Adults and Adults with Autism Spectrum Disorder

The Social Shapes Test (SST) is a measure of social intelligence which does not use human faces or rely on extensive verbal ability. The SST has shown promising validity among adults without autism spectrum disorder (ASD), but it is uncertain whether it is suitable for adults with ASD. We find measurement invariance between adults with (n = 229) or without ASD (n = 1,049) on the 23-item SST. We also find that adults without ASD score higher on the SST than adults with ASD (d = 0.21). We also provide two, 14-item versions which demonstrated good parallel test-retest reliability and are positively related to scores on the Frith-Happé task. The SST is suitable for remote, online research studies.

Development of a serious gaming approach for cyber aptitude assessment

Numerous traditional assessments have been developed to determine suitability of US military recruits for cyber careers. Cyber career field managers expressed a concern there may be well-qualified candidates that lack cyber knowledge, and therefore are not identified with knowledge-based tests. Technological advances such as serious gaming may provide opportunities to assess constructs traditional methods do not effectively measure. The purpose of this effort was to identify potential gains in validity that could be achieved beyond traditional methods through the use of serious games for several cyber jobs (both for enlisted and officer positions). Throughout this phase of research, an extensive literature review of military and civilian assessments targeted cyber occupations. Then, military subject matter experts in these career fields provided input and guidance (e.g., focus on aptitudes and traits as knowledge and skill are rapidly outdated). A gap analysis between all measures of such constructs identified a short list of candidates for measurement in a serious game. A survey of 800 airmen in the 1N4X1A, 3D1X2 and 17DEX/SX career fields was conducted; 290 respondents identified six constructs to be the focus for serious game assessment. The game was developed, and constructs validated on a sample chosen to model Air Force enlisted recruits. Additional psychometric data from enlistees and cyber trainees will be gathered once COVID-19 restrictions are lifted.

Computerized Cognitive Test Batteries for Children and Adolescents-A Scoping Review of Tools For Lab- and Web-Based Settings From 2000 to 2021

OBJECTIVE

Cognitive functioning is essential to well-being. Since cognitive difficulties are common in many disorders, their early identification is critical, notably during childhood and adolescence. This scoping review aims to provide a comprehensive literature overview of computerized cognitive test batteries (CCTB) that have been developed and used in children and adolescents over the past 22 years and to evaluate their psychometric properties.

METHOD

Among 3192 records identified from three databases (PubMed, PsycNET, and Web of Science) between 2000 and 2021, 564 peer-reviewed articles conducted in children and adolescents aged 3 to 18 years met inclusion criteria. Twenty main CCTBs were identified and further reviewed following PRISMA guidelines. Relevant study details (sample information, topic, location, setting, norms, and psychometrics) were extracted, as well as administration and instrument characteristics for the main CCTBs.

RESULTS

Findings suggest that CCTB use varies according to age, location, and topic, with eight tools accounting for 85% of studies, and the Cambridge Neuropsychological Test Automated Battery (CANTAB) being most frequently used. Few instruments were applied in web-based settings or include social cognition tasks. Only 13% of studies reported psychometric properties.

CONCLUSIONS

Over the past two decades, a high number of computerized cognitive batteries have been developed. Among these, more validation studies are needed, particularly across diverse cultural contexts. This review offers a comprehensive synthesis of CCTBs to aid both researchers and clinicians to conduct cognitive assessments in children in either a lab- or web-based setting.

Game based assessments of cognitive ability in recruitment: Validity, fairness and test-taking experience

Gamification and machine learning are emergent technologies in recruitment, promising to improve the user experience and fairness of assessments. We test this by validating a game based assessment of cognitive ability with a machine learning based scoring algorithm optimised for validity and fairness. We use applied data from 11,574 assessment completions. The assessment has convergent validity (r = 0.5) and test-retest reliability (r = 0.68). It maintains fairness in a separate sample of 3,107 job applicants, showing that fairness-optimised machine learning can improve outcome parity issues with cognitive ability tests in recruitment settings. We show that there are no significant gender differences in test taking anxiety resulting from the games, and that anxiety does not directly predict game performance, supporting the notion that game based assessments help with test taking anxiety. Interactions between anxiety, gender and performance are explored. Feedback from 4,778 job applicants reveals a Net Promoter score of 58, indicating more applicants support than dislike the assessment, and that games deliver a positive applicant experience in practise. Satisfaction with the format is high, but applicants raise face validity concerns over the abstract games. We encourage the use of gamification and machine learning to improve the fairness and user experience of psychometric tests.

Game-related assessments for personnel selection: A systematic review

Industrial development in recent decades has led to using information and communication technologies (ICT) to support personnel selection processes. One of the most notable examples is game-related assessments (GRA), supposedly as accurate as conventional tests but which generate better applicant reactions and reduce the likelihood of adverse impact and faking. However, such claims still lack scientific support. Given practitioners’ increasing use of GRA, this article reviews the scientific literature on gamification applied to personnel selection to determine whether the current state of the art supports their use in professional practice and identify specific aspects on which future research should focus. Following the PRISMA model, a search was carried out in the Web of Science and Scopus databases, identifying 34 valid articles, of which 85.3% are empirical studies that analyze five areas: (1) validity; (2) applicant reactions; (3) design of GRA; (4) personal characteristics and GRA; and (5) adverse impact and faking. Together, these studies show that GRA can be used in personnel selection but that the supposed advantages of GRA over conventional tests are fewer than imagined. The results also suggest several aspects on which research should focus (e.g., construct validity, differences depending on the type of game, prediction of different job performance dimensions), which could help define the situations in which the use of GRA may be recommended.

Assessing Interests in the Twenty-First-Century Workforce: Building on a Century of Interest Measurement

Recent research has re-emphasized the importance of vocational interests for understanding workplace attitudes and behavior. As a result, there is a renewed interest in the assessment of vocational interests in organizations. Numerous interest assessments have been developed over the past century, and they are now administered to millions of people throughout the world. Nevertheless, there is still work to be done, particularly as interest assessments are increasingly being used in organizational settings. This article reviews developments in interest assessments and discusses the implications of their use for both research and practice. It discusses the advantages and disadvantages of examining vocational interests in organizational contexts and proposes future research directions.

Theory and Technology in Organizational Psychology: A Review of Technology Integration Paradigms and Their Effects on the Validity of Theory

Despite the centrality of technology to understanding how humans in organizations think, feel, and behave, researchers in organizational psychology and organizational behavior even now often avoid theorizing about it. In our review, we identify four major paradigmatic approaches in theoretical approaches to technology, which typically occur in sequence: technology-as-context, technology-as-causal, technology-as-instrumental, and technology-as-designed. Each paradigm describes a typically implicit philosophical orientation toward technology as demonstrated through choices about theory development and research design. Of these approaches, one is unnecessarily limited and two are harmful oversimplifications that we contend have systematically weakened the quality of theory across our discipline. As such, we argue that to avoid creating impractical and even inaccurate theory, researchers must explicitly model technology design. To facilitate this shift, we define technology, present our paradigmatic framework, explain the framework's importance, and provide recommendations across five key domains: personnel selection, training and development, performance management and motivation, groups and teams, and leadership.

Cheating on Unproctored Internet Test Applications: An Analysis of a Verification Test in a Real Personnel Selection Context

This study analyses the extent to which cheating occurs in a real selection setting. A two-stage, unproctored and proctored, test administration was considered. Test score inconsistencies were concluded by applying a verification test (Guo and Drasgow Z-test). An initial simulation study showed that the Z-test has adequate Type I error and power rates in the specific selection settings explored. A second study applied the Z-test statistic verification procedure to a sample of 954 employment candidates. Additional external evidence based on item time response to the verification items was gathered. The results revealed a good performance of the Z-test statistic and a relatively low, but non-negligible, number of suspected cheaters that showed higher distorted ability estimates. The study with real data provided additional information on the presence of suspected cheating in unproctored applications and the viability of using item response times as an additional evidence of cheating. In the verification test, suspected cheaters spent 5.78 seconds per item more than expected considering the item difficulty and their assumed ability in the unproctored stage. We found that the percentage of suspected cheaters in the empirical study could be estimated at 13.84%. In summary, the study provides evidence of the usefulness of the Z-test in the detection of cheating in a specific setting, in which a computerized adaptive test for assessing English grammar knowledge was used for personnel selection.

Stronger Together: Personality, Intelligence and the Assessment of Career Potential

Personality and intelligence have a long history in applied psychology, with research dating back more than 100 years. In line, early developments in industrial-organizational psychology were largely founded on the predictive power of personality and intelligence measures vis-à-vis career-related outcomes. However, despite a wealth of evidence in support of their utility, the concepts, theories, and measures of personality and intelligence are still widely underutilized in organizations, even when these express a commitment to making data-driven decisions about employees and leaders. This paper discusses the value of personality and intelligence to understand individual differences in career potential, and how to increase the adoption of theories and tools for evaluating personality and intelligence in real-world organizational contexts. Although personality and intelligence are distinct constructs, the assessment of career potential is incomplete without both.

Socially Desirable Responding in Web-Based Questionnaires: A Meta-Analytic Review of the Candor Hypothesis

Unproctored, web-based assessments supposedly reduce social desirability distortions in self-report questionnaires because of an increased sense of privacy among participants. Three random-effects meta-analyses focusing either on social desirability ( k = 30, total N = 3,746), the Big Five of personality ( k = 66, total N = 2,951), or psychopathology ( k = 96, total N = 16,034) compared social desirability distortions of self-reports across computerized and paper-and-pencil administration modes. Overall, a near-zero effect, Δ = 0.01, was obtained that did not indicate less socially desirable responding in computerized assessments. Moreover, moderator analyses did not identify differential effects for proctored and unproctored procedures. Thus, paper-and-pencil and computerized administrations of self-report scales yield comparable mean scores. Unproctored web-based surveys do not offer an advantage with regard to socially desirable responding in self-report questionnaires.