Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects

Proctored remote testing of cognitive abilities in the private homes of test-takers is becoming an increasingly popular alternative to standard psychological assessments in test centers or classrooms. Because these tests are administered under less standardized conditions, differences in computer devices or situational contexts might contribute to measurement biases that impede fair comparisons between test-takers. Because it is unclear whether cognitive remote testing might be a feasible assessment approach for young children, the present study (N = 1,590) evaluated a test of reading comprehension administered to children at the age of 8 years. To disentangle mode from setting effects, the children finished the test either in the classroom on paper or computer or remotely on tablets or laptops. Analyses of differential response functioning found notable differences between assessment conditions for selected items. However, biases in test scores were largely negligible. Only for children with below-average reading comprehension small setting effects between on-site and remote testing were observed. Moreover, response effort was higher in the three computerized test versions, among which, reading on tablets most strongly resembled the paper condition. Overall, these results suggest that, on average, even for young children remote testing introduces little measurement bias.

Attitudes towards AI: measurement and associations with personality

Artificial intelligence (AI) has become an integral part of many contemporary technologies, such as social media platforms, smart devices, and global logistics systems. At the same time, research on the public acceptance of AI shows that many people feel quite apprehensive about the potential of such technologies-an observation that has been connected to both demographic and sociocultural user variables (e.g., age, previous media exposure). Yet, due to divergent and often ad-hoc measurements of AI-related attitudes, the current body of evidence remains inconclusive. Likewise, it is still unclear if attitudes towards AI are also affected by users' personality traits. In response to these research gaps, we offer a two-fold contribution. First, we present a novel, psychologically informed questionnaire (ATTARI-12) that captures attitudes towards AI as a single construct, independent of specific contexts or applications. Having observed good reliability and validity for our new measure across two studies (N1 = 490; N2 = 150), we examine several personality traits-the Big Five, the Dark Triad, and conspiracy mentality-as potential predictors of AI-related attitudes in a third study (N3 = 298). We find that agreeableness and younger age predict a more positive view towards artificially intelligent technology, whereas the susceptibility to conspiracy beliefs connects to a more negative attitude. Our findings are discussed considering potential limitations and future directions for research and practice.

Accuracy and precision of fixed and random effects in meta-analyses of randomized control trials for continuous outcomes

Meta-analyses of treatment effects in randomized control trials are often faced with the problem of missing information required to calculate effect sizes and their sampling variances. Particularly, correlations between pre- and posttest scores are frequently not available. As an ad-hoc solution, researchers impute a constant value for the missing correlation. As an alternative, we propose adopting a multivariate meta-regression approach that models independent group effect sizes and accounts for the dependency structure using robust variance estimation or three-level modeling. A comprehensive simulation study mimicking realistic conditions of meta-analyses in clinical and educational psychology suggested that imputing a fixed correlation 0.8 or adopting a multivariate meta-regression with robust variance estimation work well for estimating the pooled effect but lead to slightly distorted between-study heterogeneity estimates. In contrast, three-level meta-regressions resulted in largely unbiased fixed effects but more inconsistent prediction intervals. Based on these results recommendations for meta-analytic practice and future meta-analytic developments are provided.

The Predictive Validity of Item Effect Variables in the Satisfaction With Life Scale for Psychological and Physical Health

Although the Satisfaction with Life Scale strives to capture a single dimension, describing respondents' satisfaction with life as a whole, individual items might also capture unique aspects of life satisfaction leading to some form of multidimensionality. Such systematic item-specific variance can be viewed as a content-laden secondary trait. Information on the nomological net and predictive validity can be useful to aid the interpretation of these item-specific effects. Therefore, the present study on N = 2,543 Dutch respondents adopts revised latent state-trait theory to disentangle common construct variance, random measurement error, and person-specific item effects in the Satisfaction with Life Scale across three measurement occasions. The reported analyses not only demonstrate how to examine item-specific multidimensionality in longitudinal data but also emphasize how different identification constraints for the latent variable lead to different interpretations. Moreover, the predictive validity of item effect variables for the prediction of psychological and physical health is examined. A cross-validation with the same sample at a later measurement period and robustness checks with incomplete data, support our findings on the substantive value of a multidimensional specification of the Satisfaction with Life Scale for substantive analyses. Finally, the contributions of person-specific item effects for psychological assessments are discussed.

Studying item-effect variables and their correlation patterns with multi-construct multi-state models

Method effects on the item level can be modeled as latent difference variables in longitudinal data. These item-effect variables represent interindividual differences associated with responses to a specific item when assessing a common construct with multi-item scales. In latent variable analyses, their inclusion substantially improves model fits in comparison to classical unidimensional measurement models. More importantly, covariations between different item-effect variables and with other constructs can provide valuable insights, for example, into the structure of the studied instrument or the response process. Therefore, we introduce a multi-construct multi-state model with item-effect variables for systematic investigations of these correlation patterns within and between constructs. The implementation of this model is demonstrated using a sample of N = 2,529 Dutch respondents that provided measures of life satisfaction and positive affect at five measurement occasions. Our results confirm non-negligible item effects in two ostensibly unidimensional scales, indicating the importance of modeling interindividual differences on the item level. The correlation pattern between constructs indicated rather specific effects for individual items and no common causes, but the correlations within a construct align with the item content and support a substantive meaning. These analyses exemplify how multi-construct multi-state models allow the systematic examination of item effects to improve substantive and psychometric research.

The Dimensionality of the Brief COPE Before and During the COVID-19 Pandemic

The Brief COPE (Coping Orientation to Problems Experienced) is a frequently used questionnaire assessing 14 theoretically derived coping mechanisms, but psychometric research has suggested inconsistent results concerning its factor structure. The aim of this study was to investigate primary and secondary order factor structures of the Brief COPE during the COVID-19 pandemic by testing 11 different models by confirmatory factor analyses and to assess differences between sex, age groups, and relationship status. Altogether, 529 respondents from Austria and Germany participated in a web-based survey. Results supported the originally hypothesized 14-factor structure but did not support previously described higher-order structures. However, bass-ackwards analyses suggested systematic overlap between different factors, which might have contributed to different factor solutions in previous research. Measurement invariance across sex, age groups, and relationship status could be confirmed. Findings suggest that cultural and situational aspects as well as the functional level should be considered in research on theoretical framing of coping behavior.

Meta-Analytic Structural Equation Modeling With Fallible Measurements

Abstract. Meta-analytic structural equation modeling (MASEM) combines the strengths of meta-analysis with the flexibility of path models to address multivariate research questions using summary statistics. Because many research questions refer to latent constructs, measurement error can distort effect estimates in MASEMs if the unreliability of study variables is not properly acknowledged. Therefore, a comprehensive Monte Carlo simulation evaluated the impact of measurement error on MASEM results for different mediation models. These analyses showed that point estimates in MASEM were distorted by up to a third of the true effect, while confidence intervals exhibited undercoverage that were less than 10% in some situations. However, the use of adjustments for attenuation facilitated recovering largely undistorted point and interval estimates in MASEMs. These findings emphasize that MASEMs with fallible measurements can often yield highly distorted results. We encourage applied researchers to regularly adopt adjustment methods that account for attenuation in MASEMs.

Out-of-Level Cognitive Testing of Children with Special Educational Needs

Abstract. Children with special educational needs in the area of learning (SEN-L) have severe learning disabilities and often exhibit substantial cognitive impairments. Therefore, standard assessment instruments of basic cognitive abilities designed for regular school children are frequently too complex for them and, thus, unable to provide reliable proficiency estimates. The present study evaluated whether out-of-level testing with the German version of the Cognitive Abilities Test using test versions developed for younger age groups might suit the needs of these children. Therefore, N = 511 children with SEN-L and N = 573 low achieving children without SEN-L attending fifth grades in Germany were administered four tests measuring reasoning and verbal comprehension that were designed for fourth graders. The results showed that children with SEN-L exhibited significantly more missing responses than children without SEN-L. Moreover, three of the four tests were still too difficult for them. Importantly, no substantial differential response functioning was found for children with and without SEN-L. Thus, out-of-level testing might represent a feasible strategy to assess basic cognitive functioning in children with SEN-L. However, comparative interpretations would require additional norms or linked test versions that place results from out-of-level tests on a common metric.

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

The Impact of Different Methods to Correct for Response Styles on the External Validity of Self-Reports

Abstract. Response styles (RSs) such as acquiescence represent systematic respondent behaviors in self-report questionnaires beyond the actual item content. They distort trait estimates and contribute to measurement bias in questionnaire-based research. Although various approaches were proposed to correct the influence of RSs, little is known about their relative performance. Because different correction methods formalize the latent traits differently, it is unclear how model choice affects the external validity of the corrected measures. Therefore, the present study on N = 1,000 Dutch respondents investigated the impact of correcting responses to measures of self-esteem and the need for cognition using structural equation models with structured residuals, multidimensional generalized partial credit models, and multinomial processing trees. The study considered three RSs: extreme, midpoint, and acquiescence RS. The results showed homogeneous correlation patterns among the modeled latent and external variables, especially if they were not themselves subject to RSs. In that case, the IRT-based models, including an uncorrected model, still yielded consistent results. Nevertheless, the strength of the effect sizes showed variation.

The Web-Based Assessment of Mental Speed

Abstract. Although web-based cognitive assessments have gained increasing attention in recent decades, it is still debated whether unstandardized test settings allow for comparable measurements as compared to proctored testing, particularly for speeded cognitive tests. Therefore, two within-subject experiments ( N = 73 and N = 72) compared differences in means, criterion correlations with measures of intelligence, and subjective test quality perceptions of a trail-making test between a proctored paper-based, a proctored computerized, and an unproctored web-based administration mode. The results in both samples showed equivalent means between the two computerized modes, equivalent criterion correlations between the three modes, and no differential item functioning. However, the web-based tests were rated as having an inferior measurement quality as compared to the proctored assessments. Thus, web-based testing allows for comparable measurements of mental speed as compared to traditional computerized tests, although it is still regarded as a lower quality medium by test takers.

Parent and Teacher Assessments of Social-Emotional Competence in Three-Year-Old Children: Does Sibling Status Matter?

Valid information on early social-emotional competence is essential to diagnose, treat, and prevent behavioral problems in children and adolescents. Particularly in young children, social-emotional competence is frequently measured using parent and teacher ratings that frequently exhibit low agreement. Therefore, the present study on n = 532 three-year-olds (47% girls) examined whether sibling status might explain discrepancies between the two informant groups. First, multi-trait multi-informant analyses explored the construct validity of a short measure of three facets of social-emotional competence. Then, group comparisons evaluated the size of the observed method effects for only children and children with siblings. Results showed low convergent validity between parent and teacher ratings for aggressive behavior, cooperative behavior, and emotional self-regulation. Sibling status in the family contributed little to the observed discrepancies between parents and teachers. Thus, a comprehensive assessment of social-emotional competence in children requires a multi-informant approach to capture the construct breadth.

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Human-Like Robots and the Uncanny Valley

Abstract. In the field of human-robot interaction, the well-known uncanny valley hypothesis proposes a curvilinear relationship between a robot’s degree of human likeness and the observers’ responses to the robot. While low to medium human likeness should be associated with increased positive responses, a shift to negative responses is expected for highly anthropomorphic robots. As empirical findings on the uncanny valley hypothesis are inconclusive, we conducted a random-effects meta-analysis of 49 studies (total N = 3,556) that reported 131 evaluations of robots based on the Godspeed scales for anthropomorphism (i.e., human likeness) and likeability. Our results confirm more positive responses for more human-like robots at low to medium anthropomorphism, with moving robots rated as more human-like but not necessarily more likable than static ones. However, because highly anthropomorphic robots were sparsely utilized in previous studies, no conclusions regarding proposed adverse effects at higher levels of human likeness can be made at this stage.

Setting a standard for low reading proficiency: A comparison of the bookmark procedure and constrained mixture Rasch model

In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach (i.e., Bookmark procedure) and a model-based one (i.e., constrained mixture Rasch model). This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade (N = 13,897) as well as adults (Ns = 5,335 and 3,145). The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample.

Determinants of profiles of competence development in mathematics and reading in upper secondary education in Germany

The registered report was targeted at identifying latent profiles of competence development in reading and mathematics among N = 15,012 German students in upper secondary education sampled in a multi-stage stratified cluster design across German schools. These students were initially assessed in grade 9 and provided competence assessments on three measurement occasions across six years using tests especially developed for the German National Educational Panel Study (NEPS). Using Latent Growth Mixture Models, Using Latent Growth Mixture Models, we aimed at identifying multiple profiles of competence development. Specifically, we expected to find at least one generalized (i.e., reading and mathematical competence develop similarly) and two specialized profiles (i.e., one of the domains develops faster) of competence development and that these profiles are explained by the specialization of interest and of vocational education of students. Contrary to our expectations, we did not find multiple latent profiles of competence development. The model describing our data best was a single-group latent growth model confirming a competence development profile, which can be described as specializing in mathematical competences, indicating a higher increase in mathematical competences as compared to reading competences in upper secondary school. Since only one latent profile was identified, potential predictors (specialization of vocational education and interest) for different profiles of competence development were not examined.

The Structure of the Toronto Alexithymia Scale (TAS-20): A Meta-Analytic Confirmatory Factor Analysis

Alexithymia is defined as the inability of persons to describe their emotional states, to identify the feelings of others, and a utilitarian type of thinking. The most popular instrument to assess alexithymia is the Toronto Alexithymia Scale (TAS-20). Despite its widespread use, an ongoing controversy pertains to its internal structure. The TAS-20 was originally constructed to capture three different factors, but several studies suggested different factor solutions, including bifactor models and models with a method factor for the reversely keyed items. The present study examined the dimensionality of the TAS-20 using summary data of 88 samples from 62 studies (total N = 69,722) with meta-analytic structural equation modeling. We found support for the originally proposed three-dimensional solution, whereas more complex models produced inconsistent factor loadings. Because a major source of misfit stems from translated versions, the results are discussed with respect to generalizations across languages and cultural contexts.

Linking of Rasch-Scaled Tests: Consequences of Limited Item Pools and Model Misfit

In the context of item response theory (IRT), linking the scales of two measurement points is a prerequisite to examine a change in competence over time. In educational large-scale assessments, non-identical test forms sharing a number of anchor-items are frequently scaled and linked using two− or three-parametric item response models. However, if item pools are limited and/or sample sizes are small to medium, the sparser Rasch model is a suitable alternative regarding the precision of parameter estimation. As the Rasch model implies stricter assumptions about the response process, a violation of these assumptions may manifest as model misfit in form of item discrimination parameters empirically deviating from their fixed value of one. The present simulation study investigated the performance of four IRT linking methods—fixed parameter calibration, mean/mean linking, weighted mean/mean linking, and concurrent calibration—applied to Rasch-scaled data with a small item pool. Moreover, the number of anchor items required in the absence/presence of moderate model misfit was investigated in small to medium sample sizes. Effects on the link outcome were operationalized as bias, relative bias, and root mean square error of the estimated sample mean and variance of the latent variable. In the light of this limited context, concurrent calibration had substantial convergence issues, while the other methods resulted in an overall satisfying and similar parameter recovery—even in the presence of moderate model misfit. Our findings suggest that in case of model misfit, the share of anchor items should exceed 20% as is currently proposed in the literature. Future studies should further investigate the effects of anchor item composition regarding unbalanced model misfit.

Profiles of competence development in upper secondary education and their predictors

This registered report protocol elaborates on the theory, methods, and material of a study to identify latent profiles of competence development in reading and mathematics among German students in upper secondary education. It is expected that generalized (reading and mathematical competence develop similarly) and specialized (one of the domains develops faster) competence profiles will be identified. Moreover, it is hypothesized that students' domain-specific interest and educational history will predict membership of these latent profiles as these factors influence the students' learning environments. For this study, we will use data from the German National Educational Panel Study, including students from ninth grade in secondary schools (expected N = 14,500). These students were tracked across six years and provided competence assessments on three occasions. The latent profiles based on the students' reading and mathematical competences will be identified using latent growth mixture modeling. If different types of profiles can be identified, multinomial regression will be utilized to analyze whether the likelihood of belonging to a certain competence development profile is influenced by students' domain-specific interest or educational history. As this protocol is submitted before any analyses were conducted, it will provide neither results nor conclusions.

Airborne human papillomavirus (HPV) transmission risk during ablation procedures: A systematic review and meta-analysis

BACKGROUND

Human Papillomavirus (HPV) is associated with development of oropharyngeal cancer. Aim of this review was to assess airborne transmission risk of infectious particles from HPV lesions to airway mucosa of medical staff during established ablation procedures.

METHODS

A systematic review of human and animal studies, published before 09/2020, relevant to airborne HPV transmission. Controlled studies reporting prevalence of HPV-associated upper airway (nasal/oral/pharyngeal) disease in staff performing ablation procedures (laser, loop electrosurgical excision [LEEP], cryosurgery) on HPV lesions were included in meta-analysis. Additionally, we aimed for a comprehensive systematic overview of studies regarding occupational risk of airborne HPV transmission and safety measures during ablation procedures.

RESULTS

A total of n = 30 original studies report outcomes related to HPV transmission risk in medical staff conducting ablation procedures. HPV DNA detection in ablation smoke (n = 7), matching HPV genotypes on ablated HPV lesions and face/airways of medical staff after ablation (n = 2), and evidence for infectivity of papillomavirus in ablation smoke (n = 3, animal models only) were reported. Three case reports describe occupational HPV disease of upper airway mucosa. Three controlled studies assessed warts (in CO₂ laser-users only); when pooling all controls (general population, non-laser users), nasal/oral/pharyngeal lesion sites were more common amongst laser-users (OR = 5.75; 95%CI[1.55, 21.38]; p < .001).

DISCUSSION

Airborne HPV dispersal with matching "high-risk" HPV-genotypes in airways of medical staff after ablations (LEEP and CO₂-laser) and cases of HPV-associated upper airways neoplasms based on exposure to laser and LEEP smoke are documented. Upper airway mucosa is a more common anatomical site for warts in CO₂ laser users compared to controls. Simple safety measures greatly reduce HPV contamination and transmission risk.

The Day of the Week Effect on Subjective Well-Being in the European Social Survey

Abstract. In large-scale social surveys, respondents are typically interviewed on different days of the week. Because previous research established systematic daily fluctuations of people’s mood, it was hypothesized that subjective well-being ratings might be similarly affected by the day the interview takes place. Therefore, an individual-participant meta-analysis of 221 representative samples from the European Social Survey including 408,637 participants is presented. The random-effects meta-analysis found a negligible day of the week effect on life satisfaction and happiness ratings, even after accounting for selection and interviewer effects. Although significantly different ratings were observed on Sundays, the size of the obtained effects was trivial. These findings provide little evidence that the interview day has a meaningful impact on subjective well-being research in cross-sectional, large-scale studies.

On the Structure of Affect

Abstract. The Positive and Negative Affect Schedule (PANAS; Watson et al., 1988 ) is a popular self-report questionnaire that is administered all over the world. Though originally developed to measure two independent factors, different models have been proposed in the literature. Comparisons among alternative models as well as analyses concerning their robustness in cross-national research have left an inconclusive picture. Therefore, the present study evaluates the dimensionality of the PANAS and differences between English and translated versions of the PANAS using a meta-analytic structural equation modeling approach. Correlation matrices from 57 independent samples ( N = 54,043) were pooled across subsamples. For both English and non-English samples, a correlated two-factor model including correlated uniquenesses provided the best fit. However, measurement invariance analyses indicated differences in factor loadings between subsamples. Thus, cross-national application of the PANAS might only be justified if measurement equivalence was explicitly tested for the countries at hand.

Limited evidence for the effect of red color on cognitive performance: A meta-analysis

Red color supposedly affects cognitive functioning in achievement situations and impairs test performance. Although this has been shown for different cognitive domains in different populations and cultural contexts, recent studies including close replications failed to corroborate this effect. Reported here is a random-effects meta-analysis of 67 effect sizes (38 samples) that compared test performance after viewing red or a control color. For anagram tests and knowledge tests no significant difference between color conditions was found (Cohen's d of -0.06 and -0.04); for reasoning tests the pooled effect of d = -0.34, 95% CI [-0.61, -0.06] indicated significantly lower scores in the red condition. The cumulative meta-analysis revealed substantially larger effects in initial studies as compared to subsequent research. After correcting for publication bias no evidential value for an effect of red color on intellectual performance was available. The review casts doubt on the existence of a robust color-priming effect in achievement situations.

Degrees of Freedom in Multigroup Confirmatory Factor Analyses

Abstract. Measurement invaraiance is a key concept in psychological assessment and a fundamental prerequisite for meaningful comparisons across groups. In the prevalent approach, multigroup confirmatory factor analysis (MGCFA), specific measurement parameters are constrained to equality across groups. The degrees of freedom ( df) for these models readily follow from the hypothesized measurement model and the invariance constraints. In light of research questioning the soundness of statistical reporting in psychology, we examined how often reported df match with the df recalcualted based on information given in the publications. More specifically, we reviewed 128 studies from six leading peer-reviewed journals focusing on psychological assessment and recalculated the df for 302 measurement invariance testing procedures. Overall, about a quarter of all articles included at least one discrepancy with metric and scalar invariance being more frequently affected. We discuss moderators of these discrepancies and identify typical pitfalls in measurement invariance testing. Moreover, we provide example syntax for different methods of scaling latent variables and introduce a tool that allows for the recalculation of df in common MGCFA models to improve the statistical soundness of invariance testing in psychological research.

A Meta-Analysis of Test Scores in Proctored and Unproctored Ability Assessments

Abstract. Unproctored, web-based assessments are frequently compromised by a lack of control over the participants’ test-taking behavior. It is likely that participants cheat if personal consequences are high. This meta-analysis summarizes findings on context effects in unproctored and proctored ability assessments and examines mean score differences and correlations between both assessment contexts. As potential moderators, we consider (a) the perceived consequences of the assessment, (b) countermeasures against cheating, (c) the susceptibility to cheating of the measure itself, and (d) the use of different test media. For standardized mean differences, a three-level random-effects meta-analysis based on 109 effect sizes from 49 studies (total N = 100,434) identified a pooled effect of Δ = 0.20, 95% CI [0.10, 0.31], indicating higher scores in unproctored assessments. Moderator analyses revealed significantly smaller effects for measures that are difficult to research on the Internet. These results demonstrate that unproctored ability assessments are biased by cheating. Unproctored assessments may be most suitable for tasks that are difficult to search on the Internet.

Processing the Word Red and Intellectual Performance: Four Replication Attempts

Colors convey meaning and can impair intellectual performance in achievement situations. Even the processing of color words can exert similar detrimental effects. In four experiments, we tried to replicate previous findings regarding the processing of the word “red” (as compared to a control color) on cognitive test scores. Experiments 1 and 2 (Ns = 69 and 104) are direct replications of Lichtenfeld, Maier, Elliot, and Pekrun (2009). Both experiments failed to uncover a red color effect on verbal reasoning scores among high school students and undergraduates (Cohen’s d = 0.04 and –0.23). Experiments 3 and 4 (N = 103 and 1,149) failed to identify an effect of processing red on general knowledge test scores (Cohen’sd = 0.19) and 0.01) among undergraduates and adults. Together, these results do not corroborate the assumption that processing the word red impairs intellectual performance.

Are Social Media Ruining Our Lives? A Review of Meta-Analytic Evidence

A growing number of studies have examined the psychological corollaries of using social networking sites (SNSs) such as Facebook, Instagram, or Twitter (often called social media). The interdisciplinary research area and conflicting evidence from primary studies complicate the assessment of current scholarly knowledge in this field of high public attention. We review meta-analytic evidence on three hotly debated topics regarding the effects of SNSs: well-being, academic achievement, and narcissism. Meta-analyses from different laboratories draw a rather equivocal picture. They show small associations in the r = .10 range between the intensity of SNS use and loneliness, self-esteem, life satisfaction, or self-reported depression, and somewhat stronger links to a thin body ideal and higher social capital. There is no indication for potential devastating effects of social media on school achievement; social media use and school grades are unrelated for adolescents. The meta-analyses revealed small to moderate associations between narcissism and SNS use. In sum, meta-analytic evidence is not in support of dramatic claims relating social media use to mischief.

Development of a Shortened Version of the Latent and Manifest Benefits of Work (LAMB) Scale

Abstract. Paid work offers latent benefits (e.g., social contact, time structure) that go beyond mere access to financial resources in predicting individuals’ psychological well-being. Despite the importance of the concept for organizational research and practice, available instruments measuring these latent benefits suffer either from psychometric deficiencies or from a scale length that makes integrating them into large-scale work/life surveys cumbersome. Thus, the current two studies ( N = 1,054 and N = 677) report on the development of the Short Latent and Manifest Benefits of Work scale (LAMB-S; cf. Muller, Creed, Waters, & Machin, 2005 ). The new 18-item instrument showed a clear factor structure, appropriate external validities, and even slight improvements in content and criterion validity for some subscales. Overall, the LAMB-S represents an economical instrument with satisfactory psychometric properties, making it an attractive alternative in situations where participant time is limited.

Blood lead levels and cognitive functioning: A meta-analysis

OBJECTIVE

The neurotoxicity of lead is well-known and even low exposure levels potentially impact neurocognitive abilities. This metaanalysis aimed to investigate associations between neurocognitive performance and lead exposure in adults and further assess potential effect thresholds.

MATERIAL AND METHODS

Articles indexed in Medline published until January 2017 reporting associations between lead exposure, blood lead, cognitive abilities and sensomotoric performance were included. The unbiased, standardized mean differences g between lead exposure and control groups extracted from the primary studies were pooled using a three-level, random-effects meta-analytic model with a restricted maximum likelihood estimator. Moderator analyses were conducted using weighted, mixedeffects regression analyses.

RESULTS

We identified 22 articles (n=3,849 participants, mean age 39.94±7.87years) published between 1976 and 2014 reporting lead exposure effects on cognitive and sensomotoric parameters (verbal abilities, visuospatial abilities, memory, attention, psychomotor function). On average, blood lead concentrations were 21.09±6.44 μg/dl higher in exposed than in control subjects. After exclusion of outliers, the random-effects three-level meta-analysis identified a significant (p<.001) pooled mean difference between exposure and control groups. Except for a smaller effect in the digit symbol test (p<.05), lead exposure did not result in different outcomes across the examined cognitive measures. Based on a marginally significant (p=.06) effect of difference in exposure levels, a blood lead increase of 10 μg/dl translated into a decline in cognitive abilities of Hedges g=.09.

CONCLUSIONS

Neurocognitive performance in adults with occupational or environmental lead exposure was significant impaired with regard to the examined parameters. However, further studies are needed for the determination of effect thresholds and reversibility.

Exploring the myth of the chubby gamer: A meta-analysis on sedentary video gaming and body mass

RATIONALE

High body mass and obesity are frequently linked to the use of sedentary media, like television (TV) or non-active video games. Empirical evidence regarding video gaming, however, has been mixed, and theoretical considerations explaining a relationship between general screen time and body mass may not generalize to non-active video gaming.

OBJECTIVE

The current meta-analysis had two main goals. First, we wanted to provide an estimate of the average effect size of the relationship between sedentary video gaming and body mass. In doing so we acknowledged several context variables to gauge the stability of the average effect. Second, to provide additional evidence on processes, we tested the displacement effect of physical activity by video gaming time with the help of a meta-analytic structural equation model (MASEM).

METHOD

Published and unpublished studies were identified through keyword searches in different databases and references in relevant reports were inspected for further studies. We present a random-effects, three-level meta-analysis based on 20 studies (total N = 38,097) with 32 effect sizes.

RESULTS

The analyses revealed a small positive relationship between non-active video game use and body mass, ρˆ=.09, 95% CI [0.03, 0.14], indicating that they shared less than 1% in variance. The studies showed significant heterogeneity, Q (31) = 593.03, p < .001, I² = 95.13. Moderator analyses revealed that the relationship was more pronounced for adults, ρˆ=.22, 95% CI [0.04, 0.40], as compared to adolescents, ρˆ=.01, 95% CI [-0.21, 0.23], or children, ρˆ=.09, 95% CI [-0.07, 0.25]. Meta-analytic structural equation modeling found little evidence for a displacement of physical activity through time spent on video gaming.

CONCLUSION

These results do not corroborate the assumption of a strong link between video gaming and body mass as respective associations are small and primarily observed among adults.

The Longitudinal Measurement of Reasoning Abilities in Students With Special Educational Needs

Students with special educational needs in the area of learning (SEN-L) have learning disabilities that can lead to academic difficulties in regular schools. In Germany, these students are frequently enrolled in special schools providing specific training and support for these students. Because of their cognitive difficulties, it is unclear whether standard achievement tests that are typically administered in educational large-scale assessments (LSA) are suitable of students with SEN-L. The present study evaluated the psychometric properties of a short instrument for the assessment of reasoning abilities that was administered as part of a longitudinal LSA to German students from special schools (N = 324) and basic secondary schools (N = 338) twice within 6 years. Item response modeling demonstrated an essentially unidimensional scale for both school types. Few items exhibited systematic differential item functioning (DIF) between students with and without SEN-L, allowing for valid cross-group comparisons. However, change analyses across the two time points needed to account for longitudinal DIF among students with SEN-L. Overall, the cognitive test allowed for a valid measurement of reasoning abilities in students with SEN-L and comparative analyses regarding students without SEN-L. These results demonstrate the feasibility of incorporating students with SEN-L into educational LSAs.

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance ( p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion ( p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Modeling competence development in the presence of selection bias

A major challenge for representative longitudinal studies is panel attrition, because some respondents refuse to continue participating across all measurement waves. Depending on the nature of this selection process, statistical inferences based on the observed sample can be biased. Therefore, statistical analyses need to consider a missing-data mechanism. Because each missing-data model hinges on frequently untestable assumptions, sensitivity analyses are indispensable to gauging the robustness of statistical inferences. This article highlights contemporary approaches for applied researchers to acknowledge missing data in longitudinal, multilevel modeling and shows how sensitivity analyses can guide their interpretation. Using a representative sample of N = 13,417 German students, the development of mathematical competence across three years was examined by contrasting seven missing-data models, including listwise deletion, full-information maximum likelihood estimation, inverse probability weighting, multiple imputation, selection models, and pattern mixture models. These analyses identified strong selection effects related to various individual and context factors. Comparative analyses revealed that inverse probability weighting performed rather poorly in growth curve modeling. Moreover, school-specific effects should be acknowledged in missing-data models for educational data. Finally, we demonstrated how sensitivity analyses can be used to gauge the robustness of the identified effects.

The structure of the General Health Questionnaire (GHQ-12): two meta-analytic factor analyses

The General Health Questionnaire (GHQ-12) is a popular measure of psychological distress. Despite its widespread use, an ongoing controversy pertains to its internal structure. Although the GHQ-12 was originally constructed to capture a unitary construct, empirical studies identified different factor structures. Therefore, this study examined the dimensionality of the GHQ-12 in two independent meta-analyses. The first meta-analysis used summary data published in 38 primary studies (total N = 76,473). Meta-analytic exploratory factor analyses identified two factors formed by negatively and positively worded items. The second meta-analysis included individual responses of 410,640 participants from 84 independent samples. Meta-analytic confirmatory factor analyses corroborated the two-dimensional structure of the GHQ-12. However, bifactor modelling showed that most of the variance was explained by a general factor. Therefore, subscale scores reflected rather limited unique variance. Overall, the two meta-analyses demonstrated that the GHQ-12 is essentially unidimensional. It is not recommended to use and interpret subscale scores because they primarily reflect general mental health rather than distinct constructs.

The Structure of the Rosenberg Self-Esteem Scale

Abstract. The Rosenberg Self-Esteem Scale (RSES; Rosenberg, 1965 ) intends to measure a single dominant factor representing global self-esteem. However, several studies have identified some form of multidimensionality for the RSES. Therefore, we examined the factor structure of the RSES with a fixed-effects meta-analytic structural equation modeling approach including 113 independent samples (N = 140,671). A confirmatory bifactor model with specific factors for positively and negatively worded items and a general self-esteem factor fitted best. However, the general factor captured most of the explained common variance in the RSES, whereas the specific factors accounted for less than 15%. The general factor loadings were invariant across samples from the United States and other highly individualistic countries, but lower for less individualistic countries. Thus, although the RSES essentially represents a unidimensional scale, cross-cultural comparisons might not be justified because the cultural background of the respondents affects the interpretation of the items.

Cognitive Abilities Explain Wording Effects in the Rosenberg Self-Esteem Scale

There is consensus that the 10 items of the Rosenberg Self-Esteem Scale (RSES) reflect wording effects resulting from positively and negatively keyed items. The present study examined the effects of cognitive abilities on the factor structure of the RSES with a novel, nonparametric latent variable technique called local structural equation models. In a nationally representative German large-scale assessment including 12,437 students competing measurement models for the RSES were compared: a bifactor model with a common factor and a specific factor for all negatively worded items had an optimal fit. Local structural equation models showed that the unidimensionality of the scale increased with higher levels of reading competence and reasoning, while the proportion of variance attributed to the negatively keyed items declined. Wording effects on the factor structure of the RSES seem to represent a response style artifact associated with cognitive abilities.

Opinion leadership types or continuous opinion leadership traits?

Opinion leadership is typically conceptualised as a continuous personality trait. However, many authors adhere to the view of qualitatively different opinion leadership types and apply arbitrary criteria to split continuous trait scores into two groups (i.e., opinion leaders vs. non-leaders). The present study is the first to empirically evaluate this approach. A sample of N = 3812 adults (67% women) was administered a validated opinion leadership scale. Finite mixture models examined whether the latent trait distribution can be represented by a set of discrete trait levels that reflected distinct opinion leadership types. The results did not give support to a discrete typology that distinguished leaders from non-leaders. Rather, opinion leadership was best characterised as a continuous trait.

The Measurement of Variability and Change in Life Satisfaction

Abstract. In many large-scale surveys, life satisfaction is measured with a single item. Little is known about how sensitive single-item instruments react to changes across time or in the face of critical life events. The present study draws on a Dutch sample of N = 4,034 participants who were administered three different single items and an established multi-item instrument at four annual surveys within 6 years. An indicator-specific latent state-trait growth model was employed to estimate the agreement in change coefficients for the different instruments. The results indicated that single items estimated a similar developmental decline in life satisfaction across the life course as multi-item scales, but they were less sensitive to the loss of paid work. Interindividual differences in intraindividual change were better captured with multiple items. Moreover, single items reflected occasion-specific variance to a larger degree than longer scales. In addition, occasion-specific effects were larger among younger respondents as compared to older age groups. Therefore, single items appear to represent valid measures for the estimation of the average change in life satisfaction within a sample. However, they might exhibit a low power to detect moderating influences on the estimated change trajectories and capture less trait variance than multi-item scales.

Narcissism and Social Networking Behavior: A Meta-Analysis

OBJECTIVE

The increasing popularity of social networking sites (SNS) such as Facebook and Twitter has given rise to speculations that the intensity of using these platforms is associated with narcissistic tendencies. However, recent research on this issue has been all but conclusive.

METHOD

We present a three-level, random effects meta-analysis including 289 effect sizes from 57 studies (total N = 25,631) on the association between trait narcissism and social networking behavior.

RESULTS

The meta-analysis identified a small to moderate effect of ρ = .17 (τ = .11), 95% CI [.13, .21], for grandiose narcissism that replicated across different social networking platforms, respondent characteristics, and time. Moderator analyses revealed pronounced cultural differences, with stronger associations in power-distant cultures. Moreover, social networking behaviors geared toward self-presentation and the number of SNS friends exhibited stronger effects than usage durations.

CONCLUSIONS

Overall, the study not only supported but also refined the notion of a relationship between engaging in social networking sites and narcissistic personality traits.

Human Capital and Reemployment Success: The Role of Cognitive Abilities and Personality

Involuntary periods of unemployment represent major negative experiences for many individuals. Therefore, it is important to identify factors determining the speed job seekers are able to find new employment. The present study focused on cognitive and non-cognitive abilities of job seekers that determine their reemployment success. A sample of German adults (N = 1366) reported on their employment histories over the course of six years and provided measures on their fluid and crystallized intelligence, mathematical and reading competence, and the Big Five of personality. Proportional hazard regression analyses modeled the conditional probability of finding a new job at a given time dependent on the cognitive and personality scores. The results showed that fluid and crystallized intelligence as well as reading competence increased the probability of reemployment. Moreover, emotionally stable job seekers had higher odds of finding new employment. Other personality traits of the Big Five were less relevant for reemployment success. Finally, crystallized intelligence and emotional stability exhibited unique predictive power after controlling for the other traits and showed incremental effects with regard to age, education, and job type. These findings highlight that stable individual differences have a systematic, albeit rather small, effect on unemployment durations.