Rhetoric of psychological measurement theory and practice

Metascience scholars have long been concerned with tracking the use of rhetorical language in scientific discourse, oftentimes to analyze the legitimacy and validity of scientific claim-making. Psychology, however, has only recently become the explicit target of such metascientific scholarship, much of which has been in response to the recent crises surrounding replicability of quantitative research findings and questionable research practices. The focus of this paper is on the rhetoric of psychological measurement and validity scholarship, in both the theoretical and methodological and empirical literatures. We examine various discourse practices in published psychological measurement and validity literature, including: (a) clear instances of rhetoric (i.e., persuasion or performance); (b) common or rote expressions and tropes (e.g., perfunctory claims or declarations); (c) metaphors and other "literary" styles; and (d) ambiguous, confusing, or unjustifiable claims. The methodological approach we use is informed by a combination of conceptual analysis and exploratory grounded theory, the latter of which we used to identify relevant themes within the published psychological discourse. Examples of both constructive and useful or misleading and potentially harmful discourse practices will be given. Our objectives are both to contribute to the critical methodological literature on psychological measurement and connect metascience in psychology to broader interdisciplinary examinations of science discourse.

Team Efficacy Profiles: Congruence Predicts Objective Performance of Athlete Pairs

Efficacy beliefs targeting personal (self-efficacy), teammates' (other-efficacy), and conjoint (collective efficacy) abilities are each associated with performance of athlete pairs. The purpose of this study was to examine (a) congruence/incongruence of efficacy beliefs between athletes in a pair as a predictor of quality of individual and team performance and (b) quality of performance relative to efficacy congruence at high, moderate, and low levels of efficacy. Eighty-two cheerleading pairs, composed of one base and one flyer, completed questionnaires assessing self-, other, and collective efficacy prior to a national collegiate competition. Individual and team performances were assessed using objective criteria. Polynomial regression analyses indicated that team performance was predicted by congruence of (a) both athletes' collective efficacy beliefs and (b) base self-efficacy and flyer other efficacy. Findings supported that congruence at moderate to high levels of efficacy was associated with better performance relative to incongruent efficacy beliefs across the two athletes.

Instructors' Gestural Accuracy Affects Geology Learning in Interaction with Students' Spatial Skills

Complex and often unobservable STEM constructs and processes are represented using a variety of representations, including iconic gestures in which the body is configured or moved to resemble a referent's spatial properties or actions. Earlier researchers have suggested links between gesturing and expertise, leading some to recommend instructional gestures. Earlier research, however, has been largely correlational; furthermore, some gestures may be made with misleading positions or movements. Using the illustrative topic of strike in structural geology, we investigated the existence and impact of inaccurate instructional gestures. In Study 1, we examined videotapes of participants who had been asked to explain strikes to another person. We observed inaccurate (non-horizontal) strike gestures not only among novices (first introduced to strike during the study itself, n = 68) but also among participants who had greater expertise in geology (n = 21). In Study 2, we randomly assigned novices (N = 167) to watch video lessons in which the instructor accompanied verbal explanations of strikes with accurate, inaccurate, or no iconic gestures and tested students' learning on a strike-mapping task. Students with low spatial-perception skills showed no impact of their gestural condition on performance. Students with high spatial-perception skills showed no advantage from accurate gestures but performed significantly worse in the inaccurate-gesture condition. Findings suggest that recommendations to use gestures during instruction should include professional development programs that reduce the occurrence of inaccurate gestures.

The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices: Five years later

Changes in statistical practices and reporting have been documented by Giofrè et al. PLOS ONE 12(4), e0175583 (2017), who investigated ten statistical and open practices in two high-ranking journals (Psychological Science [PS] and Journal of Experimental Psychology-General [JEPG]): null hypothesis significance testing; confidence or credible intervals; meta-analysis of the results of multiple experiments; confidence interval interpretation; effect size interpretation; sample size determination; data exclusion; data availability; materials availability; and preregistered design and analysis plan. The investigation was based on an analysis of all papers published in these journals between 2013 and 2015. The aim of the present study was to follow up changes in both PS and JEPG in subsequent years, from 2016 to 2020, adding code availability as a further open practice. We found improvement in most practices, with some exceptions (i.e., confidence interval interpretation and meta-analysis). Despite these positive changes, our results indicate a need for further improvements in statistical practices and adoption of open practices.

Evaluating the Evidence for Enclothed Cognition: Z-Curve and Meta-Analyses

Enclothed cognition refers to the systematic influence that clothes can have on the wearer's feelings, thoughts, and behaviors through their symbolic meaning. It has attracted considerable academic and nonacademic interest, with the 2012 article that coined the phrase cited more than 600 times and covered in more than 160 news outlets. However, a recent high-powered replication failed to replicate one of the original effects. To determine whether the larger body of research on enclothed cognition possesses evidential value and replicable effects, we performed z-curve and meta-analyses using 105 effects from 40 studies across 24 articles (N = 3,789). Underscoring the marked improvement of psychological research practices in the mid-2010s, our results raise concerns about the replicability of early enclothed cognition studies but affirm the evidential value for effects published after 2015. These later studies support the core principle of enclothed cognition-what we wear influences how we think, feel, and act.

Pornography Consumption and Extramarital Sex Attitudes Among Married U.S. Adults: Longitudinal Replication

Social scientific interest in pornography use and effects dates back to at least the mid-twentieth century. Despite this, recent meta-analyses reveal a need for additional longitudinal studies, in general; a need for attitudinal studies, specifically; and a need for studies of U.S. consumers, in particular. In response to these needs and recent calls for the fields of communication and psychological science to prioritize replication, the present study probed whether Wright et al. (Psychol Pop Media 3(2):97-109, 2014) novel longitudinal findings on pornography consumption and extramarital sex attitudes among married U.S. adults were replicable. As in Wright et al., a distal assessment of extramarital sex attitudes did not predict interindividual increases in the likelihood of pornography consumption. Contrary to Wright et al., a distal assessment of pornography consumption also failed to predict interindividual increases in positive attitudes toward extramarital sex. However, more proximal measures of extramarital sex attitudes and pornography consumption did predict over time interindividual change in pornography use and attitudinal positivity, respectively, even after adjusting for participants' age, divorce history, education, race, sex, general unhappiness, martial unhappiness, liberal-conservative political orientation, and religiosity. These results are consistent with prior panel studies in the pornography literature in the macro, but also highlight a need for theoretical development (and testing) on the duration and time-course of selection and socialization effects in the context of pornography use and sexual attitudes.

The Clinical Phenotype of Binge Eating Disorder among Postmenopausal Women: A Pilot Study

Binge eating disorder (BED), a form of overnutrition, may impact healthy aging for postmenopausal women. In community samples, 12-26% of older women (ages 60+) engage in binge eating. In younger adults, BED is comorbid with physical and psychological morbidities. However, little is known regarding the clinical phenotype, including medical and psychiatric comorbidities, of BED in postmenopausal women. This pilot study sought to identify psychosomatic, cardiometabolic, body composition, and physical function characteristics of postmenopausal, older adult (age ≥60 years) women with BED. Participants (N = 21, ages 60-75) completed a battery of physical assessments and surveys assessing psychosomatic health. Overall, 62% of women reported BE onset during peri- or post-menopause. Rates of comorbid depression, anxiety, sleep problems, and a history of severe menopausal symptoms were high. Cardiometabolic health was poor, and 42.9% met the criteria for metabolic syndrome. Additionally, 71.4% met the BMI criteria for obesity, and 40% of this sample met the criteria for sarcopenic obesity. Almost half of the sample presented with at least one mobility limitation; 85.7% had poor endurance. Evidence suggests that BED is highly comorbid with other chronic health conditions and may complicate treatment of these conditions, warranting further investigation and increased attention from healthcare providers serving postmenopausal women.

Case-by-case: neural markers of emotion and task stimulus significance

The present study assessed the hypothesis that electrophysiological markers of emotional and task stimulus significance can be demonstrated in concert at the level of the individual case. Participants (n = 18, 9 females) viewed low and high-arousing pictures selected from behavior systems of sexual reproduction, disease avoidance, and predator fear. Furthermore, to concurrently manipulate task relevance, participants performed an explicit emotion categorization task with either low or high-arousing pictures alternating as target stimuli in separate experimental blocks. Pooled across behavior systems, event-related components sensitive to emotional significance reached statistical significance in 100% of the tests for the early posterior negativity and in 96% of the tests for the late positive potential. Regarding explicit task relevance, the target P3 effect was significant in 96% of the tests. These findings demonstrate that neural markers of stimulus significance driven by emotional picture content and explicit task demands can be assessed at the individual level. Replicating an effect case-after-case provides strong support for an effect common-to-all and may support individual inferences. Contributions of the case-by-case approach to reveal reproducible effects and implications for the development of neural biomarkers for specific affective and cognitive component processes are discussed.

A critique of using the labels confirmatory and exploratory in modern psychological research

Psychological science is experiencing a rise in the application of complex statistical models and, simultaneously, a renewed focus on applying research in a confirmatory manner. This presents a fundamental conflict for psychological researchers as more complex forms of modeling necessarily eschew as stringent of theoretical constraints. In this paper, I argue that this is less of a conflict, and more a result of a continued adherence to applying the overly simplistic labels of exploratory and confirmatory. These terms mask a distinction between exploratory/confirmatory research practices and modeling. Further, while many researchers recognize that this dichotomous distinction is better represented as a continuum, this only creates additional problems. Finally, I argue that while a focus on preregistration helps clarify the distinction, psychological research would be better off replacing the terms exploratory and confirmatory with additional levels of detail regarding the goals of the study, modeling details, and scientific method.

Do Open-Science Badges Increase Trust in Scientists Among Undergraduates, Scientists, and the Public?

In three experimental studies, we investigated whether badges for open-science practices have the potential to affect trust in scientists and topic-specific epistemic beliefs by student teachers (n = 270), social scientists (n = 250), or the public (n = 257), all of whom were at least 16 years old. Furthermore, we analyzed the moderating role of epistemic beliefs for badges and trust. Each participant was randomly assigned to two of three conditions: badges awarded, badges not awarded, and no badges (control). In all samples, our Bayesian analyses indicated that badges influence trust as expected, with one exception in the public sample: An additional positive effect of awarded badges (compared with no badges) was not supported. For students and scientists, we found evidence for the relation of badges and epistemic beliefs as well as epistemic beliefs and trust. Further, we found evidence for the absence of moderation by epistemic beliefs.

Empirical facts from search for replicable associations between cortical thickness and psychometric variables in healthy adults

The study of associations between inter-individual differences in brain structure and behaviour has a long history in psychology and neuroscience. Many associations between psychometric data, particularly intelligence and personality measures and local variations of brain structure have been reported. While the impact of such reported associations often goes beyond scientific communities, resonating in the public mind, their replicability is rarely evidenced. Previously, we have shown that associations between psychometric measures and estimates of grey matter volume (GMV) result in rarely replicated findings across large samples of healthy adults. However, the question remains if these observations are at least partly linked to the multidetermined nature of the variations in GMV, particularly within samples with wide age-range. Therefore, here we extended those evaluations and empirically investigated the replicability of associations of a broad range of psychometric variables and cortical thickness in a large cohort of healthy young adults. In line with our observations with GMV, our current analyses revealed low likelihood of significant associations and their rare replication across independent samples. We here discuss the implications of these findings within the context of accumulating evidence of the general poor replicability of structural-brain-behaviour associations, and more broadly of the replication crisis.

The Reproducibility Movement in Psychology: Does Researcher Gender Affect How People Perceive Scientists With a Failed Replication?

The reproducibility movement in psychology has resulted in numerous highly publicized instances of replication failures. The goal of the present work was to investigate people’s reactions to a psychology replication failure vs. success, and to test whether a failure elicits harsher reactions when the researcher is a woman vs. a man. We examined these questions in a pre-registered experiment with a working adult sample, a conceptual replication of that experiment with a student sample, and an analysis of data compiled and posted by a psychology researcher on their public weblog with the stated goal to improve research replicability by rank-ordering psychology researchers by their “estimated false discovery risk.” Participants in the experiments were randomly assigned to read a news article describing a successful vs. failed replication attempt of original work from a male vs. female psychological scientist, and then completed measures of researcher competence, likability, integrity, perceptions of the research, and behavioral intentions for future interactions with the researcher. In both working adult and student samples, analyses consistently yielded large main effects of replication outcome, but no interaction with researcher gender. Likewise, the coding of weblog data posted in July 2021 indicated that 66.3% of the researchers scrutinized were men and 33.8% were women, and their rank-ordering was not correlated with researcher gender. The lack of support for our pre-registered gender-replication hypothesis is, at first glance, encouraging for women researchers’ careers; however, the substantial effect sizes we observed for replication outcome underscore the tremendous negative impact the reproducibility movement can have on psychologists’ careers. We discuss the implications of such negative perceptions and the possible downstream consequences for women in the field that are essential for future study.

Open Science Practices in Gambling Research Publications (2016-2019): A Scoping Review

The replication crisis has stimulated researchers around the world to adopt open science research practices intended to reduce publication bias and improve research quality. Open science practices include study pre-registration, open data, open access, and avoiding methods that can lead to publication bias and low replication rates. Although gambling studies uses similar research methods as behavioral research fields that have struggled with replication, we know little about the uptake of open science research practices in gambling-focused research. We conducted a scoping review of 500 recent (1/1/2016–12/1/2019) studies focused on gambling and problem gambling to examine the use of open science and transparent research practices. Our results showed that a small percentage of studies used most practices: whereas 54.6% (95% CI: [50.2, 58.9]) of studies used at least one of nine open science practices, each practice’s prevalence was: 1.6% for pre-registration (95% CI: [0.8, 3.1]), 3.2% for open data (95% CI: [2.0, 5.1]), 0% for open notebook, 35.2% for open access (95% CI: [31.1, 39.5]), 7.8% for open materials (95% CI: [5.8, 10.5]), 1.4% for open code (95% CI: [0.7, 2.9]), and 15.0% for preprint posting (95% CI: [12.1, 18.4]). In all, 6.4% (95% CI: [4.6, 8.9]) of the studies included a power analysis and 2.4% (95% CI: [1.4, 4.2]) were replication studies. Exploratory analyses showed that studies that used any open science practice, and open access in particular, had higher citation counts. We suggest several practical ways to enhance the uptake of open science principles and practices both within gambling studies and in science more generally.

Justify Your Alpha: A Primer on Two Practical Approaches

The default use of an alpha level of .05 is suboptimal for two reasons. First, decisions based on data can be made more efficiently by choosing an alpha level that minimizes the combined Type 1 and Type 2 error rate. Second, it is possible that in studies with very high statistical power, p values lower than the alpha level can be more likely when the null hypothesis is true than when the alternative hypothesis is true (i.e., Lindley’s paradox). In this article, we explain two approaches that can be used to justify a better choice of an alpha level than relying on the default threshold of .05. The first approach is based on the idea to either minimize or balance Type 1 and Type 2 error rates. The second approach lowers the alpha level as a function of the sample size to prevent Lindley’s paradox. An R package and Shiny app are provided to perform the required calculations. Both approaches have their limitations (e.g., the challenge of specifying relative costs and priors) but can offer an improvement to current practices, especially when sample sizes are large. The use of alpha levels that are better justified should improve statistical inferences and can increase the efficiency and informativeness of scientific research.

How Cross-Discipline Understanding and Communication Can Improve Research on Multiracial Populations

One of the strengths of Critical Mixed Race Studies is that it represents research methodologies and frameworks from multiple disciplines across the social sciences and humanities. However, if these disciplines are not in dialogue with each other, that benefit may be lost. Here, we use psychological and sociological research on Multiracial populations as examples to argue how strict disciplinarity and methodological trends may limit scientific production. We propose that reading and citing work across disciplines, expanding methodological training, and rejecting hegemonic “white logic” assumptions about what is “publishable” can enhance Multiracial research. First, the ability to cite effectively across disciplines will shorten the time it takes for new theories to be developed that focus on empirically underrepresented populations. Secondly, increasing understanding of both quantitative and qualitative methods will allow more effective reading between disciplines while also creating opportunities to engage with both causality and the richness of experiences that comprise being Multiracial. Finally, these changes would then situate scholars to be more effective reviewers, thereby enhancing the peer-reviewed publication process to one that routinely rejects color evasive racist practices that privilege work on majority populations.

Validation of a battery of inhibitory control tasks reveals a multifaceted structure in non-human primates

Inhibitory control, the ability to override an inappropriate prepotent response, is crucial in many aspects of everyday life. However, the various paradigms designed to measure inhibitory control often suffer from a lack of systematic validation and have yielded mixed results. Thus the nature of this ability remains unclear, is it a general construct or a family of distinct sub-components? Therefore, the aim of this study was first to demonstrate the content validity and the temporal repeatability of a battery of inhibitory control tasks. Then we wanted to assess the contextual consistency of performances between these tasks to better understand the structure of inhibitory control. We tested 21 rhesus macaques (Macaca mulatta, 12 males, nine females) in a battery of touchscreen tasks assessing three main components of inhibitory control: inhibition of a distraction (using a Distraction task), inhibition of an impulsive action (using a Go/No-go task) and inhibition of a cognitive set (using a Reversal learning task). All tasks were reliable and effective at measuring the inhibition of a prepotent response. However, while there was consistency of performance between the inhibition of a distraction and the inhibition of an action, representing a response-driven basic form of inhibition, this was not found for the inhibition of a cognitive set. We argue that the inhibition of a cognitive set is a more cognitively demanding form of inhibition. This study gives a new insight in the multifaceted structure of inhibitory control and highlights the importance of a systematic validation of cognitive tasks in animal cognition.

When the p Value Doesn't Cut It: The Fragility Index Applied to Randomized Controlled Trials in Colorectal Surgery

BACKGROUND

The American Statistical Association, among others, has called for the use of statistical methods beyond p ≤ 0.05. The fragility index is a statistical metric defined as the minimum number of patients for whom if an event rather than a nonevent occurred, then the p value would increase to ≥0.05. Previous reviews have demonstrated that many randomized controlled trials have a low fragility index, suggesting they may not be robust.

OBJECTIVE

The purpose of this study was to review the fragility indices of randomized controlled trials in colorectal surgery.

DATA SOURCES

A PubMed search was performed.

STUDY SELECTION

Colorectal surgery randomized controlled trials with a dichotomous primary outcome p ≤ 0.05 and publication between 2016 and 2018 were systematically identified.

INTERVENTIONS

All procedural interventions related to colorectal surgery were included.

MAIN OUTCOME MEASURES

The main measures were the fragility index and the number of patients lost to follow-up for each trial. The percentage of trials with the number of patients lost to follow-up greater than the fragility index was calculated.

RESULTS

In total, 712 abstracts were reviewed, with 90 trials meeting the inclusion criteria. The median fragility index was 3 (interquartile range of 1 to 10). In 51 of the 90 trials (57%), the number of patients lost to follow-up was greater than the fragility index.

LIMITATIONS

The fragility index is only one measure of the robustness of a randomized clinical trial.

CONCLUSIONS

Most colorectal surgery randomized controlled trials have a low fragility index. In 57% of trials, more patients were lost to follow-up than would be required to change the outcome of the trial from "significant" to "nonsignificant" based on the p value. This emphasizes the importance of assessing the robustness of clinical trials when considering their clinical application, rather than relying solely on the p value. See Video Abstract at http://links.lww.com/DCR/B741.CUANDO EL VALOR-P ES INSUFICIENTE: ÍNDICE DE FRAGILIDAD APLICADO EN ESTUDIOS ALEATORIOS CONTROLADOS EN CIRUGÍA COLORECTAL.

ANTECEDENTES

La Sociedad Estadounidense de Estadística, entre otros, ha pedido el uso de métodos estadísticos más allá de p <0,05. El índice de fragilidad es una medida estadística definida como el número de desenlaces que podrían cambiar para revertir, o conseguir, la significación estadística, así el valor p aumentaría a ≥ 0,05. Las revisiones anteriores han demostrado que muchos estudios aleatorios controlados tienen un índice de fragilidad bajo, lo que sugiere que pueden poco sólidos.

OBJETIVO

El propósito de la présente investigación fué de revisar los índices de fragilidad de los estudios aleatorios controlados en cirugía colorrectal.

FUENTES DE DATOS

PubMed.

SELECCIN DE ESTUDIOS

Se identificaron sistemáticamente estudios aleatorios controlados de cirugía colorrectal con un resultado primario dicotómico, valor de p ≤ 0,05 y publicados entre 2016-2018.

INTERVENCIONES

Se incluyeron todas aquellas intervenciones con procedimientos relacionados con la cirugía colorrectal.

PRINCIPALES MEDIDAS DE RESULTADO

Las principales medidas fueron: el índice de fragilidad y el número de pacientes perdidos durante el seguimiento en cada estudio. Se calculó el el índice de fragilidad en porcentaje de estudios con el mayor número de pacientes perdidos durante el seguimiento mas prolongado.

RESULTADOS

En total, se revisaron 712 resúmenes con 90 ensayos que cumplieron con los criterios de inclusión. La mediana del índice de fragilidad fue de 3 (rango intercuartíl de 1 a 10). En 51 de los 90 estudios (57%), el número de pacientes perdidos durante el seguimiento fue mayor que el índice de fragilidad.

LIMITACIONES

El índice de fragilidad es solo una medida de la robustez de un estúdio clínico aleatorio.

CONCLUSIONES

La mayoría de los estudios aleatorios y controlados en cirugía colorrectal tienen un índice de fragilidad bajo. En el 57% de los estudios, se perdieron más pacientes durante el seguimiento de los que se necesitarían para cambiar el resultado del estudios de grado "significativo" a un grado "no significativo" según el valor-p. Este concepto enfatiza la importancia de evaluar la robustez de los estudios clínicos al considerar su aplicación verdadera aplicación clínica, en lugar de depender únicamente del valor-p. Consulte Video Resumen en http://links.lww.com/DCR/B741. (Traducción-Dr. Xavier Delgadillo).

Measuring Math Anxiety Among Predominantly Underrepresented Minority Undergraduates Using the Abbreviated Math Anxiety Scale

Many students experience math anxiety, which can negatively affect their academic achievement and even their career choices. One of the most commonly used methods to assess math anxiety is the Abbreviated Math Anxiety Scale (AMAS). Despite the ubiquity of this measure, there is a lack of studies assessing its validity. Moreover, almost all studies have administered the AMAS to a predominantly (>75%) White sample. The present study assessed the validity of the AMAS in a largely underrepresented minority-student (Hispanics and Blacks) sample. The construct validity of the scale was established by showing that students’ AMAS was predictive of both their actual (objective) and predicted (subjective) performance on an arithmetic task. The factor structure of the measure was validated by confirming that the two-factor model was a better fit than the one-factor model. Gender invariance was supported using multi-group confirmatory factor analyses. Overall, the results of the present study provide further evidence on the validity of the AMAS and its use among underrepresented minority college students.

Binge eating among older women: prevalence rates and health correlates across three independent samples

BACKGROUND

Emerging research indicates that binge eating (BE; consuming unusually large amounts of food in one siting while feeling a loss of control) is prevalent among older women. Yet, health correlates of BE in older adult populations are poorly understood. The original study aimed to investigate BE prevalence, frequency, and health correlates in a sample of older adult women. Based on results from this first study, we then sought to replicate findings in two additional samples of older adult women from separate studies.

METHOD

Using self-reported frequencies of BE from three separate samples of older women with very different demographics, we compared BE prevalence, frequency, and health correlates among older women. Study 1 (N = 185) includes data collected online (86% White; 59% overweight/obese status). Study 2 (N = 64) was conducted in person at a local food pantry (65% Hispanic; 47% household income < $10,000/year). Study 3 (N = 100) comprises data collected online (72% White; 50% Masters/Doctoral Degree).

RESULTS

Per DSM-5 frequency criterion of BE at least weekly, we found prevalence rates ranging from 19 to 26% across the three samples. Correlates of BE frequency included elevated negative mood, worry, BMI, and less nutritious food consumption.

CONCLUSIONS

Across three very different samples in terms of race/ethnicity, education, food security status, measurements, and sampling methodology, we found fairly consistent rates of self-reported BE at least weekly (19-26%). Results suggest that BE is related to negative health indices among older women and support the need for more research in this population.

A machine learning approach for the factorization of psychometric data with application to the Delis Kaplan Executive Function System

While a replicability crisis has shaken psychological sciences, the replicability of multivariate approaches for psychometric data factorization has received little attention. In particular, Exploratory Factor Analysis (EFA) is frequently promoted as the gold standard in psychological sciences. However, the application of EFA to executive functioning, a core concept in psychology and cognitive neuroscience, has led to divergent conceptual models. This heterogeneity severely limits the generalizability and replicability of findings. To tackle this issue, in this study, we propose to capitalize on a machine learning approach, OPNMF (Orthonormal Projective Non-Negative Factorization), and leverage internal cross-validation to promote generalizability to an independent dataset. We examined its application on the scores of 334 adults at the Delis-Kaplan Executive Function System (D-KEFS), while comparing to standard EFA and Principal Component Analysis (PCA). We further evaluated the replicability of the derived factorization across specific gender and age subsamples. Overall, OPNMF and PCA both converge towards a two-factor model as the best data-fit model. The derived factorization suggests a division between low-level and high-level executive functioning measures, a model further supported in subsamples. In contrast, EFA, highlighted a five-factor model which reflects the segregation of the D-KEFS battery into its main tasks while still clustering higher-level tasks together. However, this model was poorly supported in the subsamples. Thus, the parsimonious two-factors model revealed by OPNMF encompasses the more complex factorization yielded by EFA while enjoying higher generalizability. Hence, OPNMF provides a conceptually meaningful, technically robust, and generalizable factorization for psychometric tools.

Can the color red trick you into drinking less? A replication study

This replication of the study of Genschow et al. (2012) Genschow et al. (2012) examines the effect of the color red on beverage consumption. In total, 148 men were asked to consume drinks in either red- or blue-labeled cups. Cup labels were assigned at random. Unlike in the previous study, the findings in our replication study did not provide empirical support for the hypothesis that people will drink less from red-labeled cups than blue-labeled cups. The difference between groups in drink consumption was non-significant. Thus, the red color did not have an inhibitory effect on drink intake.

Developmental Trauma Disorder: A Legacy of Attachment Trauma in Victimized Children

Developmental trauma disorder (DTD) and posttraumatic stress disorder (PTSD) have been found to have both shared and unique traumatic antecedents. The present study was an independent replication, with the DTD Structured Interview and the Traumatic Events Screening Instrument administered to 271 children in mental health treatment in six U.S. sites. On an unadjusted basis, DTD (27.3% prevalence, N = 74) and PTSD (40.2% prevalence, N = 109) both were associated with traumatic physical assault or abuse, family violence, emotional abuse, caregiver separation or impairment, and polyvictimization. After controlling for PTSD, DTD was associated emotional abuse, OR = 2.9, 95% CI [1.19, 6.95], and traumatic separation from a primary caregiver, OR = 2.2, 95% CI [1.04. 4.60], both of which also were associated with caregiver impairment, physical assault/abuse, and witnessing family/community violence. Three traumatic antecedents associated with PTSD were not associated with DTD: noninterpersonal trauma, sexual trauma, and traumatic loss. Children exposed to both traumatic victimization and attachment trauma (36.2%) or attachment trauma alone (32.5%) were more likely than children exposed only to victimization (17.5%) or those with no history of victimization or attachment trauma (8.1%) to meet the symptom criteria for DTD, χ²(3, N = 271) = 17.68, p < .001. Study findings replicate and extend prior DTD field trial study results, showing that, although PTSD and DTD share traumatic antecedents, DTD is uniquely associated with traumatic emotional abuse and caregiver separation. Further research is needed to examine how specific trauma types contribute to the risk, course, and severity of DTD.

Educating the future generation of researchers: A cross-disciplinary survey of trends in analysis methods

Methods for data analysis in the biomedical, life, and social (BLS) sciences are developing at a rapid pace. At the same time, there is increasing concern that education in quantitative methods is failing to adequately prepare students for contemporary research. These trends have led to calls for educational reform to undergraduate and graduate quantitative research method curricula. We argue that such reform should be based on data-driven insights into within- and cross-disciplinary use of analytic methods. Our survey of peer-reviewed literature analyzed approximately 1.3 million openly available research articles to monitor the cross-disciplinary mentions of analytic methods in the past decade. We applied data-driven text mining analyses to the "Methods" and "Results" sections of a large subset of this corpus to identify trends in analytic method mentions shared across disciplines, as well as those unique to each discipline. We found that the t test, analysis of variance (ANOVA), linear regression, chi-squared test, and other classical statistical methods have been and remain the most mentioned analytic methods in biomedical, life science, and social science research articles. However, mentions of these methods have declined as a percentage of the published literature between 2009 and 2020. On the other hand, multivariate statistical and machine learning approaches, such as artificial neural networks (ANNs), have seen a significant increase in the total share of scientific publications. We also found unique groupings of analytic methods associated with each BLS science discipline, such as the use of structural equation modeling (SEM) in psychology, survival models in oncology, and manifold learning in ecology. We discuss the implications of these findings for education in statistics and research methods, as well as within- and cross-disciplinary collaboration.

Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

Diet pill and laxative use for weight control predicts first-time receipt of an eating disorder diagnosis within the next 5 years among female adolescents and young adults

OBJECTIVE

To replicate findings from a prior study which identified prospective associations between use of products for weight control and subsequent receipt of a first-time eating disorder (ED) diagnosis among female adolescents and young adults.

METHOD

Data from a prospective cohort study, Project EAT (Eating and Activity in Teens and Young Adults), were used to examine prospective associations between self-reported past-year diet pill and laxative use for weight control and self-reported receipt of an ED diagnosis among females without prior receipt of an ED diagnosis (N = 1,015). Participants were followed from early/middle adolescence (EAT-I; M_age  = 14.9 years) into late adolescence/emerging adulthood (EAT-II; M_age  = 19.5 years) and young adulthood (EAT-III; M_age  = 24.8 years).

RESULTS

First-time receipt of an ED diagnosis was reported by 2.4% of participants at EAT-II and 4.0% at EAT-III. After adjusting for demographics and weight status, participants using diet pills (risk ratio [RR] = 3.58, 95% confidence interval [CI]: 1.96-6.54) and laxatives (RR = 2.77, 95% CI: 1.01-7.64) had greater risk of receiving a first-time ED diagnosis within 5 years than those not using these products.

DISCUSSION

The present study replicated prior findings, providing further evidence for a prospective link between use of products for weight control and subsequent receipt of an ED diagnosis.

Prevalence of Research Misconduct and Questionable Research Practices: A Systematic Review and Meta-Analysis

Irresponsible research practices damaging the value of science has been an increasing concern among researchers, but previous work failed to estimate the prevalence of all forms of irresponsible research behavior. Additionally, these analyses have not included articles published in the last decade from 2011 to 2020. This meta-analysis provides an updated meta-analysis that calculates the pooled estimates of research misconduct (RM) and questionable research practices (QRPs), and explores the factors associated with the prevalence of these issues. The estimates, committing RM concern at least 1 of FFP (falsification, fabrication, plagiarism) and (unspecified) QRPs concern 1 or more QRPs, were 2.9% (95% CI 2.1-3.8%) and 12.5% (95% CI 10.5-14.7%), respectively. In addition, 15.5% (95% CI 12.4-19.2%) of researchers witnessed others who had committed at least 1 RM, while 39.7% (95% CI 35.6-44.0%) were aware of others who had used at least 1 QRP. The results document that response proportion, limited recall period, career level, disciplinary background and locations all affect significantly the prevalence of these issues. This meta-analysis addresses a gap in existing meta-analyses and estimates the prevalence of all forms of RM and QRPs, thus providing a better understanding of irresponsible research behaviors.

Functional Connectivity Within and Between n-Back Modulated Regions: An Adult Lifespan Psychophysiological Interaction Investigation

Introduction: Working memory (WM) and its blood-oxygen-level-dependent-related parametric modulation under load decrease with age. Functional connectivity (FC) generally increases with WM load; however, how aging impacts connectivity and whether this is load-dependent, region-dependent, or associated with cognitive performance is unclear. Methods: This study examines these questions in 170 healthy adults (meanage = 52.99 ± 19.18) who completed functional magnetic resonance imaging scanning during an n-back task (0-, 2-, 3-, and 4-back). The FC was estimated by utilizing a modified generalized psychophysiological interaction approach with seeds from fronto-parietal (FP) and default mode (DM) regions that modulated to n-back difficulty. The FC analyses focused on both connectivity during WM engagement (task vs. control) and connectivity in response to increased WM load (linear slope across conditions). Each analysis utilized within- and between-region FC, predicted by age (linear or quadratic), and its associations with in- and out-of-scanner task performance. Results: Engaging in WM either generally (task vs. control) or as a function of difficulty strengthened integration within- and between-FP and DM regions. Notably, these task-sensitive functional connections were robust to the effects of age. Stronger negative FC between FP and DM regions was also associated with better WM performance in an age-dependent manner, occurring selectively in middle-aged and older adults. Discussion: These results suggest that FC is critical for engaging in cognitively demanding tasks, and its lack of sensitivity to healthy aging may provide a means to maintain cognition across the adult lifespan. Thus, this study highlights the contribution of maintenance in brain function to support working memory processing with aging.

A Primer for Conducting Experiments in Human–Robot Interaction

We provide guidelines for planning, executing, analyzing, and reporting hypothesis-driven experiments in Human–Robot Interaction (HRI). The intended audience are researchers in the field of HRI who are not trained in empirical research but who are interested in conducting rigorous human-participant studies to support their research. Following the chronological order of research activities and grounded in updated research practices in psychological and behavioral sciences, this primer covers recommended methods and common pitfalls for defining research questions, identifying constructs and hypotheses, choosing appropriate study designs, operationalizing constructs as variables, planning and executing studies, sampling, choosing statistical tools for data analysis, and reporting results.

Memory in multiple sclerosis: A reappraisal using the item specific deficit approach

OBJECTIVE

As many as 65% of people with multiple sclerosis (MS) have clinically significant memory impairment, but the nature of this deficit is controversial. Some investigations suggest that an inability to retrieve newly learned information from memory is prominent, whereas others imply that compromised acquisition accounts for impairment. Prior research has not simultaneously evaluated acquisition and retrieval processes in MS, and fewer have attempted to account for initial acquisition when studying retrieval. The Item Specific Deficit Approach (ISDA) offers a method of quantifying acquisition, retrieval, and retention processes, with the latter two mechanisms being adjusted for initial acquisition. To simultaneously quantify acquisition and retrieval abilities, the ISDA was applied to list learning performance in two independent samples of people with MS and corresponding healthy comparison groups.

PARTICIPANTS AND METHODS

Study 1 included 85 people with MS and 47 healthy individuals. Study 2 involved a separate sample of 79 people with MS and 22 healthy people. They were administered neuropsychological batteries, and participants with MS were classified as globally impaired or unimpaired. The California Verbal Learning Test-II was administered to assess new-learning in both studies, and responses were scored using the ISDA.

RESULTS

Both studies revealed that cognitively impaired people with MS manifest weaknesses involving acquisition and retrieval. Nearly identical effect sizes emerged across samples, with cognitive impairment achieving a medium effect upon acquisition and a large effect upon retrieval.

CONCLUSIONS

These findings accord well with previous research showing diminished acquisition and retrieval among people with MS. The results may also reconcile contradictory findings in the extant literature by showing that memory impairment in MS is not exclusively attributable to either acquisition or retrieval. Rather, both processes may manifest across people with MS. The replication across samples with nearly identical effect sizes implies that these effects are reliable and possess external validity. These data hold implications for memory rehabilitation interventions involving people with MS, and suggest that acquisition and retrieval processes should be addressed in treatment. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

How does social distancing during COVID-19 affect negative moods and memory?

In the absence of an effective vaccine or treatment, the current best defence against COVID-19 is social distancing - staying at home as much as possible, keeping distance from others, and avoiding large gatherings. Although social distancing improves physical health in terms of helping to reduce viral transmission, its psychological consequences are less clear, particularly its effects on memory. In this research, we investigated the effect of social distancing duration on negative moods and memory. The relation between social distancing duration and both negative mood and memory errors followed the same U-shaped function: negative moods and memory errors initially decreased as social distancing duration increased, and then at approximately 30 days, they began to increase. Subsequent analyses indicated that memory errors were mediated by lonely mood in particular. Thus, short-term social distancing might benefit psychological well-being and memory performance, but extended social distancing has a negative impact on mood and memory.

Reflections on a registered report replicating a body of dyadic cross-sectional research

This article reflects on a new kind of registered report (RR) that replicated the work of an early career researcher. The research items targeted in this RR were peer-reviewed, cross-sectional, dyadic studies to which the first author of this RR had contributed. The findings being replicated are not noteworthy for their prestige or representativeness of the wider field. Instead, this method of replication may have several benefits and less desirable qualities for the researcher and research team whose work is being replicated, for science more broadly, and for relationship science specifically, reviewed herein. The authors hope that this reflection inspires researchers to improve upon their methodology by incorporating replication of their work early and often into their own research process.

Assessing evidence for replication: A likelihood-based approach

How to evaluate replications is a fundamental issue in experimental methodology. We develop a likelihood-based approach to assessing evidence for replication. In this approach, the design of the original study is used to derive an estimate of a theoretically interesting effect size. A likelihood ratio is then calculated to contrast the match of two models to the data from the replication attempt: (1) a model based on the derived theoretically interesting effect size, and (2) a null model. This approach provides new insights not available with existing methods of assessing replication. When applied to data from the Replication Project (Open Science Collaboration, 2015), the procedure indicates that a large portion of the replications failed to find evidence for a theoretically interesting effect.

Letters to our Future Selves? High-Powered Replication Attempts Question Effects on Future Orientation, Delinquent Decisions, and Risky Investments

Enhancing people's future orientation, in particular continuity with their future selves, has been proposed as promising to mitigate self-control–related problem behavior. In two pre-registered, direct replication studies, we tested a subtle manipulation, that is, writing a letter to one's future self, in order to reduce delinquent decisions (van Gelder et al., 2013, Study 1) and risky investments (Monroe et al., 2017, Study 1). With samples of n = 314 and n = 463, which is 2.5 times the original studies' sample sizes, the results suggested that the expected effects are either non-existent or smaller than originally reported, and/or dependent on factors not examined. Vividness of the future self was successfully manipulated in Study 2, but manipulation checks overall indicated that the letter task is not reliable to alter future orientation. We discuss ideas to integrate self-affirmation approaches and to test less subtle manipulations in samples with substantial, myopia-related self-control deficits.

Social Robots on a Global Stage: Establishing a Role for Culture During Human–Robot Interaction

Robotic agents designed to assist people across a variety of social and service settings are becoming increasingly prevalent across the world. Here we synthesise two decades of empirical evidence from human–robot interaction (HRI) research to focus on cultural influences on expectations towards and responses to social robots, as well as the utility of robots displaying culturally specific social cues for improving human engagement. Findings suggest complex and intricate relationships between culture and human cognition in the context of HRI. The studies reviewed here transcend the often-studied and prototypical east–west dichotomy of cultures, and explore how people’s perceptions of robots are informed by their national culture as well as their experiences with robots. Many of the findings presented in this review raise intriguing questions concerning future directions for robotics designers and cultural psychologists, in terms of conceptualising and delivering culturally sensitive robots. We point out that such development is currently limited by heterogenous methods and low statistical power, which contribute to a concerning lack of generalisability. We also propose several avenues through which future work may begin to address these shortcomings. In sum, we highlight the critical role of culture in mediating efforts to develop robots aligned with human users’ cultural backgrounds, and argue for further research into the role of culturally-informed robotic development in facilitating human–robot interaction.

Enhancing same-gender imitation by highlighting gender norms in Chinese pre-school children

Children selectively imitate in-group over outgroup individuals under certain experimental conditions. We investigated whether this bias applies to gender in-groups in China. Three- and five-year-olds were shown how to operate novel objects by same-gender and opposite-gender models. Results indicate that the combination of verbally highlighting the gender identity of the model (e.g., 'I am a girl') and making gender norms explicit (e.g., 'girls play this way') significantly enhances high-fidelity imitation. This 'double social effect' was more robust in 5-year-olds than 3-year-olds. Our results underscore how language about gender and the norms for gender-based groups influence behavioural imitation. The pattern of findings enhances our knowledge about pre-schoolers' social learning and imitation as well as the powerful influence of language and group norms on children's voluntary actions and learning.

Interactive situations reveal more about children's emotional knowledge

Research examining children's emotion judgments has generally used nonsocial tasks that do not resemble children's daily experiences in judging others' emotions. Here, younger children (4- to 6-year-olds) and older children (7- to 9-year-olds) participated in a socially interactive task where an experimenter opened boxes and made an expression (happy, sad, scared, or disgust) based on the object inside. Children guessed which of four objects (a sticker, a broken toy car, a spider, or toy poop) was in the box. Subsequently, children opened a set of boxes and generated facial expressions for the experimenter. Children also labeled the emotion elicited by the objects and static facial expressions. Children's ability to guess which object caused the experimenter's expression increased with age but did not predict their ability to generate a recognizable expression. Children's demonstration of emotion knowledge also varied across tasks, suggesting that when emotion judgment tasks more closely mimic their daily experiences, children demonstrate broader emotion knowledge.

The Eight Steps of Data Analysis: A Graphical Framework to Promote Sound Statistical Analysis

Data analysis is a risky endeavor, particularly among people who are unaware of its dangers. According to some researchers, "statistical conclusions validity" threatens all research subjected to the dark arts of statistical magic. Although traditional statistics classes may advise against certain practices (e.g., multiple comparisons, small sample sizes, violating normality), they may fail to cover others (e.g., outlier detection and violating linearity). More common, perhaps, is that researchers may fail to remember them. In this article, rather than rehashing old warnings and diatribes against this practice or that, I instead advocate a general statistical-analysis strategy. This graphic-based eight-step strategy promises to resolve the majority of statistical traps researchers may fall into-without having to remember large lists of problematic statistical practices. These steps will assist in preventing both false positives and false negatives and yield critical insights about the data that would have otherwise been missed. I conclude with an applied example that shows how the eight steps reveal interesting insights that would not be detected with standard statistical practices.

Compound Bias due to Measurement Error When Comparing Regression Coefficients

Multiple regression is often used to compare the importance of two or more predictors. When the predictors being compared are measured with error, the estimated coefficients can be biased and Type I error rates can be inflated. This study explores the impact of measurement error on comparing predictors when one is measured with error, followed by a simulation study to help quantify the bias and Type I error rates for common research situations. Two methods used to adjust for measurement error are demonstrated using a real data example. This study adds to the literature documenting the impact of measurement error on regression modeling, identifying issues particular to the use of multiple regression for comparing predictors, and offers recommendations for researchers conducting such studies.

Avatar Creation in Videogaming: Between Compensation and Constraint

Objective: We examine the extent that videogame avatars provide players with opportunities for identity exploration, aiming to test the findings of self-discrepancy theory research on the user/avatar relationship with novel cognitive anthropological methods. Specifically, we examine if avatar traits are idealized (more representative of players' ideal rather than actual self) or actualized (more representative of players' actual self) as a function of players' self-esteem. Materials and Methods: Utilizing cognitive anthropological methods, we examine the relationship between actual, avatar, and ideal selves. We first asked 21 respondents to list traits they associated with their various selves. We then asked 57 new respondents to perform four pile sorts of the salient items from these lists (1 unconstrained sort of like-traits, and 3 sorts of terms indicative of respondents' ideal/actual/avatar self). Analysis of this "free list" and "pile sort" data allowed us to clarify (in a manner sensitive to gamer culture) relationships between respondents' various conceptions of self, including how those relationships were modified by self-esteem. Illustrative quotes from the interviews further clarified these relationships. Results: Paired t-test analysis shows that informants as a whole describe their avatar compared with actual selves with fewer negative terms (idealization). Low-esteem players actualize what they deem as positive traits onto their avatars, while simultaneously idealizing avatars' negative traits by minimizing them. Compared with low-esteem gamers, high-esteem players associate significantly more positive attributes with all their various selves-actual, avatar, and ideal-while describing avatar compared with actual selves with fewer positive terms and comparable numbers of negative terms (the latter a process of actualization). Conclusion: Results point to the necessity of theoretical accounts that recognize that avatars may reflect a complex relationship with the user's actual and ideal self, without assuming that avatar play frees gamers from offline social, psychological, or bodily constraints.

How Different Indicator-Dimension Ratios in Assessment Center Ratings Affect Evidence for Dimension Factors

Previous research on the construct validity of assessment center (AC) ratings has usually struggled to find support for dimension factors as an underlying source of variance of these ratings. Confirmatory factor analysis (CFA) remains the most widely used method to specify and validate the internal structure of AC ratings. However, the research support for dimension effects in AC ratings remains mixed. In addition, competing CFA models (e.g., correlated dimensions-correlated exercises models) are often plagued by non-convergence and estimation problems. Recently, it has been proposed that increasing the number of indicators per dimension and exercise combination might help to find support for dimension factors, in addition to exercise factors, in CFAs of AC ratings. Furthermore, it was also suggested that the increased ratio of indicators to dimensions may also solve some of the methodological problems associated with CFA models used to model AC ratings. However, in this research it remained unclear whether the support for dimension factors was solely due to the use of a larger indicator-dimension ratio or due to parceling that combines several behavioral indicators per dimension and exercise combination into more reliable measures of the targeted dimension. These are important empirical questions that have been left unanswered in the literature but can be potentially meaningful in seeking more balanced support for dimension effects in AC research. Using data from N = 213 participants from a 1-day AC, we aimed to investigate the impact of using different indicator-dimension ratios when specifying CFA models of AC ratings. Therefore, we investigated the impact of using different indicator-dimension ratios in the form of item parcels with data from an operational AC. On average, using three parcels eventually led to support for dimension factors in CFAs. However, exercise-based CFA models still performed better than dimension-based models. Thus, the present results point out potential limits concerning the generalizability of recent results that provided support for dimension factors in ACs.