Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

Power to Detect What? Considerations for Planning and Evaluating Sample Size

ACADEMIC ABSTRACT

In the wake of the replication crisis, social and personality psychologists have increased attention to power analysis and the adequacy of sample sizes. In this article, we analyze current controversies in this area, including choosing effect sizes, why and whether power analyses should be conducted on already-collected data, how to mitigate the negative effects of sample size criteria on specific kinds of research, and which power criterion to use. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings), given the limitations of interest-based minimums or field-wide effect sizes. We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

PUBLIC ABSTRACT

Recently, social-personality psychology has been criticized for basing some of its conclusions on studies with low numbers of participants. As a result, power analysis, a mathematical way to ensure that a study has enough participants to reliably "detect" a given size of psychological effect, has become popular. This article describes power analysis and discusses some controversies about it, including how researchers should derive assumptions about effect size, and how the requirements of power analysis can be applied without harming research on hard-to-reach and marginalized communities. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings). We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

Which leadership style do more narcissistic subordinates prefer in supervisors?

BACKGROUND AND OBJECTIVE

Subordinates in Western cultures generally prefer supervisors with a democratic rather than autocratic leadership style. It is unclear, however, whether more narcissistic subordinates share or challenge this prodemocratic default attitude. On the one hand, more narcissistic individuals strive for power and thus may favor a democratic supervisor, who grants them power through participation. On the other hand, similarity attracts and, thus, more narcissistic subordinates may favor an autocratic supervisor, who exhibits the same leadership style that they would adopt in a leadership position.

METHOD

Four studies (Ntotal = 1284) tested these competing hypotheses with two narcissism dimensions: admiration and rivalry. Participants indicated the leadership style they generally prefer in a supervisor (Study 1), rated their own supervisor's leadership style (Study 2a: individual ratings; Study 2b: team ratings), and evaluated profiles of democratic and autocratic supervisors (Study 3).

RESULTS

We found a significantly weaker prodemocratic default attitude among more narcissistic subordinates: Subordinates' narcissism was negatively related to endorsement of democratic supervisors and positively related to endorsement of autocratic supervisors. Those relations were mostly driven by narcissistic rivalry rather than narcissistic admiration.

CONCLUSION

The results help clarify the narcissistic personality and, in particular, how more narcissistic subordinates prefer to be led.

Effects of Message-Sidedness on Perceived Source Bias: When Presenting Two Sides Does Versus Does Not Alleviate Concerns About Bias

Communicators commonly present two-sided messages to avoid being perceived as biased. This approach equates bias with one-sidedness rather than divergence from the position supported by available data. Messages often concern topics with mixed qualities: a product is exceptional but expensive; a politician is inexperienced but ethical. For these topics, providing a two-sided message should reduce perceived bias according to both views of bias as one-sidedness and divergence from available data. However, if perceived bias follows divergence from available data, for topics viewed as one-sided (univalent), a two-sided message should not reduce perceived bias. Across five studies, acknowledging two sides reduced perceived bias for novel topics. In two of the studies, two-sidedness no longer reduced perceived bias for topics viewed as univalent. This work clarifies that people conceptualize bias as a divergence from available data, not simply one-sidedness. It also clarifies when and how to leverage message-sidedness to reduce perceived bias.

Expanding Workplace Inclusion of Employees Who Are Parents of Children with Disabilities through Diversity Training

Employed parents raising children with disabilities manage exceptional care responsibilities along with their work careers. This study examines the effects of targeted diversity training on human resource (HR) professionals' knowledge of the work-family experiences of these parents, and on their self-efficacy in providing workplace supports. Using computer-based training in field settings, 64 U.S. human resource professionals in an international company participated in two diversity training sessions. Data related to knowledge and efficacy of dependent and disability care were collected before the first training and immediately after the second. HR participants demonstrated significant increases from pretest to posttest on the trained items: knowledge of dependent and disability care and self-efficacy regarding provision of workplace supports. There was no change in relevant but untrained variables over time. Training HR professionals on parents' exceptional care responsibilities, specific community resources, and heightened self-efficacy promoted their likelihood to grant flexible work arrangements. Results suggest HR self-efficacy is developmental, building on prior knowledge of dependent care and tenure in HR positions. This is one of the first studies that address the effects of HR diversity training regarding employees providing exceptional care. Theoretical developments and implications for inclusive practices are discussed.

How Attitudes Impact the Continued Influence Effect of Misinformation: The Mediating Role of Discomfort

Past research suggests that people continue believing retracted misinformation more when it is consistent versus inconsistent with their attitudes. However, the psychological mechanism responsible for this phenomenon remains unclear. We predicted that retractions of attitude-consistent misinformation produce greater feelings of discomfort than retractions of attitude-inconsistent misinformation and that this discomfort predicts continued belief in and use of the misinformation. We report combined analyses across 10 studies testing these predictions. Seven studies (total N = 1,323) used a mediational framework and found that the more consistent misinformation was with participants' attitudes, the more discomfort was elicited by a retraction of the misinformation. Greater discomfort then predicted greater continued belief in the misinformation, which, in turn, predicted greater use of the misinformation when participants made relevant inferences. Three additional studies (total N = 574) utilized misattribution paradigms to demonstrate that the relation between discomfort and belief in misinformation is causal in nature.