Conceptualizing and evaluating the replication of research results

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

Ambivalent attitudes promote support for extreme political actions

Political extremism varies across people and contexts, but which beliefs will a person support through extreme actions? We propose that ambivalent attitudes, despite reducing normative political actions like voting, increase support for extreme political actions. We demonstrate this hypothesized reversal using dozens of measures across six studies (N = 13,055). The effect was robust to relevant covariates and numerous methodological variations and was magnified when people's attitudinal or ideological positions were more polarized. It appears to occur because being conflicted about political issues can feel psychologically uncomfortable, making extreme actions more appealing. Notably, this emerged when people thought ambivalence was justified, whereas leading them to consider ambivalence unjustified suppressed the effect, suggesting that ambivalent people are coping with but not necessarily trying to reduce their ambivalence. These results highlight the interplay of affective and cognitive influences in extreme behavior, showing that beliefs people feel justifiably conflicted about can promote extremism.

The Relationship Between Perceived Uncontrollable Mortality Risk and Health Effort: Replication, Secondary Analysis, and Mini Meta-analysis

BACKGROUND

The Uncontrollable Mortality Risk Hypothesis (UMRH) states that those who are more likely to die due to factors beyond their control should be less motivated to invest in preventative health behaviors. Greater levels of perceived uncontrollable mortality risk (PUMR) have been associated with lower health effort in previous research, but the topic remains understudied.

PURPOSE

To examine the evidence for the UMRH by replicating a previous study investigating the effects of PUMR on social gradients in health effort, and conducting a mini meta-analysis of the overall relationship between PUMR and health effort.

METHODS

We replicated Pepper and Nettle (2014), who reported a negative relationship between PUMR and health effort, and that the positive effect of subjective socioeconomic position on health effort was explained away by PUMR. We also compared the predictive effect of PUMR on health effort with that of dimensions from the Multidimensional Health Locus of Control scale-a well-used measure of a similar construct, which is frequently found to be associated with health behavior. Finally, we conducted a mini meta-analysis of the relationship between PUMR and health effort from the available research.

RESULTS

PUMR was negatively associated with health effort, and mediated 24% of the total effect of subjective socioeconomic position on health effort, though this mediation effect was weaker than in Pepper and Nettle (2014). PUMR was shown to be a substantially stronger predictor of health effort than the relevant dimensions of the MHLC scale. Finally, our mini meta-analysis indicated a medium-sized negative relationship between PUMR and health effort.

CONCLUSIONS

Our findings offer support for the role of PUMR in mediating the relationship between subjective socioeconomic position and health effort. The results highlight the importance of measuring and understanding PUMR in studying socioeconomic inequalities in health behaviors. We discuss potential areas for future research, including determining the accuracy of PUMR, investigating influential cues, examining the role of media in shaping risk perceptions, and understanding individuals' awareness of their own perceptions of mortality risk.

The relationship between metamotivational knowledge and performance

Self-regulation research highlights the performance trade-offs of different motivational states. For instance, within the context of regulatory focus theory, promotion motivation enhances performance on eager tasks and prevention motivation enhances performance on vigilant tasks (i.e., regulatory focus task-motivation fit). Work on metamotivation-people's understanding and regulation of their motivational states-reveals that, on average, people demonstrate knowledge of how to create such task-motivation fit; at the same time, there is substantial variability in this normative accuracy. The present research examines whether having accurate normative metamotivational knowledge predicts performance. Results revealed that more accurate metamotivational knowledge predicts better performance on brief, single-shot tasks (Study 1) and in a consequential setting (course grades; Study 2). The effect was more robust in Study 2; potential implications of this variability are discussed for understanding when and why knowledge may be associated with performance.

Can Inflammation Predict Social Media Use? Linking a Biological Marker of Systemic Inflammation with Social Media Use Among College Students and Middle-Aged Adults

Drawing on recent evidence that inflammation may promote social affiliative motivation, the present research proposes a novel perspective that inflammation may be associated with more social media use. In a cross-sectional analysis of a nationally representative sample, Study 1 (N = 863) found a positive association between C-reactive protein (CRP), a biomarker of systemic inflammation, and the amount of social media use by middle-aged adults. Study 2 (N = 228) showed that among college students CRP was prospectively associated with more social media use 6 weeks later. Providing stronger evidence of the directionality of this effect, Study 3 (N = 171) showed that in college students CRP predicted increased social media use in the subsequent week even after controlling for current week's use. Additionally, in exploratory analyses of CRP and different types of social media use in the same week, CRP was only associated with using social media for social interaction and not for other purposes (e.g., entertainment). The present research sheds light on the social effects of inflammation and highlights potential benefits of using social media as a context for studying the impact of inflammation on social motivation and behavior.

Replication in Evidence-Based Aphasia Treatments

PURPOSE

A critical review was completed to evaluate replication of aphasia treatments that have been vetted and accepted on the American Speech-Language-Hearing Association Practice Portal.

METHOD

The electronic databases Academic Search Premier, ProQuest Central, CINAHL Complete, and ERIC were searched for relevant articles using treatment names as keywords. Coders compared stimuli, material, design, and statistical analysis to pilot treatment approaches. Each study was coded as direct, conceptual, failed, or no replication.

RESULTS

Eighteen treatment approaches were selected for this review. A total of 188 articles met the inclusion criteria. Results revealed that 14 out of 18 treatment approaches were somehow replicated. Direct replications as the most valuable replication type for affirming previously found results were represented in only 1.5% of all studies (3/188). Failed direct replication were at 2% overall (4/188). Conceptual replications were more common across treatment approaches, but also represented at a low level with 22.8% (43/188). The majority of studies were coded as no replication attempt with 73.4%.

DISCUSSION

A critical factor in developing an evidence-based practice is the existence of replicated results for treatment. Replication evaluates the reproducibility of an author's or authors' research methodology and resulting outcomes and helps to ensure that observed treatment effects are reproducible. For an evidence-based treatment to be implemented or used in any clinical setting, it must be one that can be replicated. Direct and conceptual replications of aphasia treatment approaches were found to be alarmingly low considering the importance of replication in our field. It is recommended that replication should become more valued and mainstreamed in aphasia research. A replication database that compiles and maintains treatment manuals for replication purposes can increase the accessibility and acceptability of replications for researchers.

A Question of Method and Subjective Beliefs

Abstract. Need for Cognition (NFC) describes the relatively stable intrinsic motivation to engage in cognitive endeavors. Recent research has revealed the importance of NFC for affective adjustment, especially in combination with self-control. We followed up on those findings by addressing methodological issues as well as processes that may underlie relations of NFC to self-control. Study 1 ( N = 102) examined whether NFC is associated with self-control independently of the measure or facet considered. Implicit willpower theories, that is, subjective beliefs concerning the limitation of self-control resources, were examined as a mediator for NFC predicting self-control. Higher NFC was associated with increased trait self-control but also with believing in unlimited self-control resources. The relation of NFC to willpower theories also mediated the prediction of trait self-control. Study 2 ( N = 188) replicated relations of NFC to self-control. We further pursued the explanatory approach from Study 1 and experimentally manipulated willpower theories to provide insight into their association with NFC. Willpower theories were related to NFC but had no mediating role in predicting self-control. The experimental manipulation had no impact on situation-specific NFC. Altogether, both studies provided first evidence that relations of NFC to self-control depend on the self-control measure and that willpower theories may be crucial for explaining the association with self-control.

Kinds of Replication: Examining the Meanings of "Conceptual Replication" and "Direct Replication"

Although psychology's recent crisis has been attributed to various scientific practices, it has come to be called a "replication crisis," prompting extensive appraisals of this putatively crucial scientific practice. These have yielded disagreements over what kind of replication is to be preferred and what phenomena are being explored, yet the proposals are all grounded in a conventional philosophy of science. This article proposes another avenue that invites moving beyond a discovery metaphor of science to rethink research as enabling realities and to consider how empirical findings enact or perform a reality. An enactment perspective appreciates multiple, dynamic realities and science as producing different entities, enactments that ever encounter differences, uncertainties, and precariousness. The axioms of an enactment perspective are described and employed to more fully understand the two kinds of replication that predominate in the crisis disputes. Although the enactment perspective described here is a relatively recent development in philosophy of science and science studies, some of its core axioms are not new to psychology, and the article concludes by revisiting psychologists' previous calls to apprehend the dynamism of psychological reality to appreciate how scientific practices actively and unavoidably participate in performativity of reality.

Evaluating validity properties of 25 race-related scales

Racial attitudes, beliefs, and motivations lie at the center of many influential theories of prejudice and discrimination. The extent to which such theories can meaningfully explain behavior hinges on accurate measurement of these latent constructs. We evaluated the validity properties of 25 race-related scales in a sample of 910,066 respondents using various tools, including dynamic fit indices, item response theory, and nomological nets. Despite showing adequate internal reliability, many scales demonstrated poor model fit and had latent score distributions showing clear floor or ceiling effects, results that illustrate deficiencies in these measures' ability to capture their intended latent construct. Nomological nets further suggested that the theoretical space of "racial prejudice" is crowded with scales that may not capture meaningfully distinct latent constructs. We provide concrete recommendations for both scale selection and scale renovation and outline implications for overlooking measurement issues in the study of prejudice and discrimination.

Business Goal Difficulty and Socially Irresponsible Executive Behavior: The Mediating Role of Focalism

Executive social irresponsibility has received increasing research attention in recent years, following the consensus for a broader stakeholder approach to managerial decision making. Despite the importance of the subject, there remains insufficient research on contextual factors that mold executives’ orientation toward social responsibility. Through three studies, we demonstrate that difficult business goals can reduce executives’ tendency to consider social responsibility in their decision making. Further, we find that focalism—a cognitive bias based on affective forecasting theory—can mediate positive relationships between business goal difficulty and socially irresponsible executive behavior. Our findings also suggest that, expanding executives’ thought processes beyond the narrow focus of a business goal achievement can be a good strategy in reducing socially irresponsible executive behavior, even in the presence of difficult goals.

How Attitudes Impact the Continued Influence Effect of Misinformation: The Mediating Role of Discomfort

Past research suggests that people continue believing retracted misinformation more when it is consistent versus inconsistent with their attitudes. However, the psychological mechanism responsible for this phenomenon remains unclear. We predicted that retractions of attitude-consistent misinformation produce greater feelings of discomfort than retractions of attitude-inconsistent misinformation and that this discomfort predicts continued belief in and use of the misinformation. We report combined analyses across 10 studies testing these predictions. Seven studies (total N = 1,323) used a mediational framework and found that the more consistent misinformation was with participants' attitudes, the more discomfort was elicited by a retraction of the misinformation. Greater discomfort then predicted greater continued belief in the misinformation, which, in turn, predicted greater use of the misinformation when participants made relevant inferences. Three additional studies (total N = 574) utilized misattribution paradigms to demonstrate that the relation between discomfort and belief in misinformation is causal in nature.

How to True Psychology’s Objects

Psychology’s current crisis attends most visibly to perceived problems with statistical models, methods, publication practices, and career incentives. Rarely is close attention given to the objects of inquiry—to ontological matters—yet the crisis-related literature does features statements about the nature of psychology’s objects. Close analysis of the ontological claims reveals discrepant understandings: some researchers assume objects to be stable and singular while others posit them to be dynamic and complex. Nevertheless, both views presume the objects under scrutiny to be real. The analysis also finds each of these ontological claims to be associated not only with particular method prescriptions but also with distinct notions of the scientific self. Though both take the scientific self to be objective, one figures the scientist as not always a rational actor and, therefore, requiring some behavior regulation, while the other sees the scientist as largely capable of self-governing sustained through painstakingly acquired expertise and self-control. The fate of these prevalent assemblages of object, method, and scientific self remains to be determined, yet as conditions of possibility they portend quite different futures. Following description of the assemblages, the article ventures a futuristic portrayal of the scientific practices they each might engender.

Mind-Body Practices and Self-Enhancement: Direct Replications of Gebauer et al.'s (2018) Experiments 1 and 2

Mind-body practices such as yoga and meditation are often believed to instill a "quiet ego," entailing less self-enhancement. In two experiments, however, Gebauer et al. (2018) demonstrated that mind-body practices may actually increase self-enhancement, particularly because such practices become self-central bases for self-esteem. We conducted preregistered replications of both of Gebauer et al.'s experiments. Experiment 1 was a field study of Canadian yoga students (N = 97), and Experiment 2 was a multiwave meditation intervention among Canadian university students (N = 300). Our results supported Gebauer et al.'s original conclusions that mind-body practices increase self-enhancement. Although the self-centrality effects were not clearly replicated in either experiment, we found evidence that measurement and sampling differences may explain this discrepancy. Moreover, an integrative data analysis of the original and the replication data strongly supported all of Gebauer et al.'s conclusions. In short, we provide new evidence against the ego-quieting perspective and in support of the self-centrality interpretation of mind-body practices.

Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

Context Dependency as a Predictor of Replicability

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Effects of acetaminophen on risk taking

Acetaminophen, an analgesic and antipyretic available over-the-counter and used in over 600 medicines, is one of the most consumed drugs in the USA. Recent research has suggested that acetaminophen’s effects extend to the blunting of negative as well as positive affect. Because affect is a determinant of risk perception and risk taking, we tested the hypothesis that acute acetaminophen consumption (1000 mg) could influence these important judgments and decisions. In three double-blind, placebo-controlled studies, healthy young adults completed a laboratory measure of risk taking (Balloon Analog Risk Task) and in Studies 1 and 2 completed self-report measures of risk perception. Across all studies (total n = 545), acetaminophen increased risk-taking behavior. On the more affectively stimulating risk perception measure used in Study 2, acetaminophen reduced self-reported perceived risk and this reduction statistically mediated increased risk-taking behavior. These results indicate that acetaminophen can increase risk taking, which may be due to reductions in risk perceptions, particularly those that are highly affect laden.

Will an Implementation of "Joy of Life in Nursing Homes" Have Positive Effect for the Work Culture? A Comparison Between Two Norwegian Municipalities

Background

Currently, we are facing a demographic shift to an older population and its consequences worldwide: in the years to come, several older people will need nursing home (NH) care. The work culture is important for care quality in NHs. Some Norwegian municipalities have implemented the Joy of Life Nursing Home (JoLNH) strategy, representing a resource-oriented health-promoting approach. Knowledge about how implementation of the JoLNH approach impacts the work culture is scarce.

Aimss

We hypothesized that the JoLNH strategy impacts positively on the work culture: (1) when comparing measurements at two time points (T1 and T2) and (2) when comparing two municipalities, among which one has implemented the JoLNH and the other has not.

Method

With a 1-year interval, healthcare personnel in 43 NHs located in two large Norwegian municipalities responded to a survey in two waves (T1 and T2). In total, 558 healthcare personnel participated at T1 and 515 at T2. Work culture was assessed by the Systematizing Person-Group Relations Instrument (SPGR).

Findings

The municipality implementing the JOLNH strategy experienced significant increases in SPGR dimensions, representing positive orientations towards a better work culture, whereas those working in the non-JoLNH municipality reported an increase in SPGR dimensions, signifying a worsened work culture.

Discussion

The data indicate that the implemented JoLNH strategy has endorsed positive effects in the work culture. The JoLNH strategy emphasizes on the quality of patient care, which is what healthcare personnel, in general, are much concerned about. Furthermore, attention to task orientation and independent work seem to encourage a better work culture in NHs.

Conclusion

This study suggests that the JoLNH strategy impacts positively on the work culture in Norwegian NHs.

The Impact of Complexity on Methods and Findings in Psychological Science

The study of human behavior is severely hampered by logistical problems, ethical and legal constraints, and funding shortfalls. However, the biggest difficulty of conducting social and behavioral research is the extraordinary complexity of the study phenomena. In this article, we review the impact of complexity on research design, hypothesis testing, measurement, data analyses, reproducibility, and the communication of findings in psychological science. The systematic investigation of the world often requires different approaches because of the variability in complexity. Confirmatory testing, multi-factorial designs, survey methods, large samples, and modeling are frequently needed to study complex social and behavioral topics. Complexity impedes the measurement of general constructs, the reproducibility of results and scientific reporting, and the general rigor of research. Many of the benchmarks established by classic work in physical science are not attainable in studies of more complex phenomena. Consequently, the standards used to evaluate scientific research should be tethered to the complexity of the study topic.

The Role of Replication Studies in Theory Building

At least since Meehl's (in)famous 1978 article, the state of theorizing in psychology has often been lamented. Replication studies have been presented as a way of directly supporting theory development by enabling researchers to more confidently and precisely test and update theoretical claims. In this article I use contemporary work from philosophy of science to make explicit and emphasize just how much theory development is required before "good" replication studies can be carried out and show just how little theoretical payoff even good conceptual replications offer. I suggest that in many areas of psychology aiming at replication is misplaced and that instead replication attempts are better seen as exploratory studies that can be used in the cumulative development of theory and measurement procedures.

Influences of Position Justification on Perceived Bias: Immediate Effects and Carryover Across Persuasive Messages

The current research examined how people infer whether novel sources are biased based on their ability to justify their position. Across nine studies, when sources provided weak versus strong arguments, message recipients perceived the source as more biased. This effect held controlling for other possible inferences, such as lack of expertise or untrustworthiness. This research also examined whether perceived source bias on one message can carry over to ambiguously related future persuasive messages. Studies 6 to 8 demonstrated that perceivers use both the perceived bias from an initial message and the argument quality of the second message to determine a source's bias on the new topic. Finally, perceived bias carried over from an initial message can influence persuasion on a second topic (Study 9). Ultimately, the present work provides insight into factors that affect perceived bias and the dynamic consequences of those perceptions.

Changing the logic of replication: A case from infant studies

Among infant researchers there is growing concern regarding the widespread practice of undertaking studies that have small sample sizes and employ tests with low statistical power (to detect a wide range of possible effects). For many researchers, issues of confidence may be partially resolved by relying on replications. Here, we bring further evidence that the classical logic of confirmation, according to which the result of a replication study confirms the original finding when it reaches statistical significance, could be usefully abandoned. With real examples taken from the infant literature and Monte Carlo simulations, we show that a very wide range of possible replication results would in a formal statistical sense constitute confirmation as they can be explained simply due to sampling error. Thus, often no useful conclusion can be derived from a single or small number of replication studies. We suggest that, in order to accumulate and generate new knowledge, the dichotomous view of replication as confirmatory/disconfirmatory can be replaced by an approach that emphasizes the estimation of effect sizes via meta-analysis. Moreover, we discuss possible solutions for reducing problems affecting the validity of conclusions drawn from meta-analyses in infant research.

Understanding When Similarity-Induced Affective Attraction Predicts Willingness to Affiliate: An Attitude Strength Perspective

Individuals reliably feel more attracted to those with whom they share similar attitudes. However, this affective liking does not always predict affiliative behavior, such as pursuing a friendship. The present research examined factors that influence the extent to which similarity-based affective attraction increases willingness to affiliate (i.e., behavioral attraction) – one potential step toward engaging in affiliative behavior. Research on attitude strength has identified attitude properties, such as confidence, that predict when an attitude is likely to impact relevant outcomes. We propose that when one’s attitudes possess these attitude strength-related properties, affective attraction to those who share that attitude will be more likely to spark willingness to affiliate. Across four studies on a variety of topics, participants (N = 428) reported their attitudes and various attitude properties regarding a topic. They were introduced to a target and learned the target’s stance on the issue. Participants reported their affective attraction and willingness to pursue friendship with the target. Consistent with past research, attitude similarity predicted affective attraction. More importantly, the relation between affective attraction and willingness to affiliate with the target was moderated by the attitude strength-related properties. A mini meta-analysis found this effect to be consistent across the four studies.

A Validity-Based Framework for Understanding Replication in Psychology

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication “failure.” Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

Multicultural Experiences and the Secondary Transfer Effect of Intercultural Attitudes

Abstract. This research investigates whether multicultural experiences (MCEs) (1) improve attitudes toward primary outgroups, (2) improve attitudes toward secondary outgroups (the “secondary transfer effect”) – depending on the perceived similarity versus dissimilarity of the target group, and (3) affect ingroup reappraisal. The deprovincialization and attitude generalization hypotheses were also investigated as underlying mechanisms. A meta-analysis of effects across studies ( N = 633) revealed MCEs (1) improve primary attitudes ( r = .19), (2) improve, albeit more weakly, similar secondary attitudes ( r = .10), but have no significant effect on dissimilar secondary attitudes ( r = .07); (3) have no effect on ingroup reappraisal ( r = .04), and (4) only improve secondary attitudes through attitude generalization. Contributions, limitations, and emerging questions regarding deprovincialization are discussed.

Average Power: A Cautionary Note

Replication is an important contemporary issue in psychological research, and there is great interest in ways of assessing replicability, in particular, retrospectively via prior studies. The average power of a set of prior studies is a quantity that has attracted considerable attention for this purpose, and techniques to estimate this quantity via a meta-analytic approach have recently been proposed. In this article, we have two aims. First, we clarify the nature of average power and its implications for replicability. We explain that average power is not relevant to the replicability of actual prospective replication studies. Instead, it relates to efforts in the history of science to catalogue the power of prior studies. Second, we evaluate the statistical properties of point estimates and interval estimates of average power obtained via the meta-analytic approach. We find that point estimates of average power are too variable and inaccurate for use in application. We also find that the width of interval estimates of average power depends on the corresponding point estimates; consequently, the width of an interval estimate of average power cannot serve as an independent measure of the precision of the point estimate. Our findings resolve a seeming puzzle posed by three estimates of the average power of the power-posing literature obtained via the meta-analytic approach.

Can Psychological Interventions Improve Intergroup Attitudes Post Terror Attacks?

This research concurrently investigated the effectiveness of three established bias-reducing interventions (i.e., positive affirmation, secure attachment, and cognitive dissonance) in the wake of the Paris and Brussels terror attacks. Using frequentist and Bayesian analyses, Study 1 ( N = 1,676), launched within days of the attacks, found that compared to a control condition, the interventions did not significantly improve intergroup attitudes. Instead, the data showed strong support of the null hypotheses that there were no intervention effects. Proximity to the attacks did not moderate the effect. Study 2 ( N = 285) reexamined the effects of the three interventions 2.5 years after the attacks, generally replicating the pattern of findings in Study 1. Together, this research highlights the challenge of intergroup bias reduction following terror attacks. We conclude by discussing several recommendations for how psychological interventions could play a more impactful role in contexts of heightened conflict.

Having pity on our victims to save ourselves: Compassion reduces self-critical emotions and self-blame about past harmful behavior among those who highly identify with their past self

Previous research has shown that people often separate the present self from past selves. Applying knowledge gained from intergroup research to the interpersonal domain, we argue that the degree to which people identify with their past self (self-identification) influences their reaction when recalling a past event during which they harmed another person. Because they feel close to their past self, we expected this to be threatening for high self-identifiers, and expected them to be motivated to avoid self-critical emotions and blame. Using four meta-analyses, conducted on a set of seven experimental studies, we investigated four ways in which high self-identifiers can distance themselves from the event: by feeling compassion, by taking a third-person rather than first-person perspective, by emphasizing ways in which their present self is different to their past self, and by disidentifying with the past self altogether. We found the strongest interaction effects for compassion: whereas a compassion manipulation increased self-critical emotions and self-blame about the past event for low self-identifiers, it decreased them for high self-identifiers. We argue that this occurs because the other-focused nature of compassion allows high self-identifiers subtly to shift the focus away from their harmful behavior. Our concept of past self-identification had stronger effects than a measure of self-continuity beliefs. It also correlated only moderately with the latter, suggesting they are distinct concepts. Our findings suggest that, ironically, the most effective way to protect the self against reminders of an undesirable past, may be to have compassion for our victims.

Perceived Knowledge Moderates the Relation Between Subjective Ambivalence and the “Impact” of Attitudes: An Attitude Strength Perspective

Previous work has reliably demonstrated that when people experience more subjective ambivalence about an attitude object, their attitudes have less impact on strength-related outcomes such as attitude-related thinking, judging, or behaving. However, previous research has not considered whether the amount of perceived knowledge a person has about the topic might moderate these effects. Across eight studies on different topics using a variety of outcome measures, the current research demonstrates that perceived knowledge can moderate the relation between ambivalence and the impact of attitudes on related thinking, judging, and behaving. Although the typical Attitude × Ambivalence effect emerged when participants had relatively high perceived knowledge, this interaction did not emerge when participants were lower in perceived knowledge. This work provides a more nuanced view of the effects of subjective ambivalence on attitude impact and highlights the importance of understanding the combined impact of attitude strength antecedents.

Same Test, Better Scores: Boosting the Reliability of Short Online Intelligence Recruitment Tests with Nested Logit Item Response Theory Models

Assessing job applicants’ general mental ability online poses psychometric challenges due to the necessity of having brief but accurate tests. Recent research (Myszkowski & Storme, 2018) suggests that recovering distractor information through Nested Logit Models (NLM; Suh & Bolt, 2010) increases the reliability of ability estimates in reasoning matrix-type tests. In the present research, we extended this result to a different context (online intelligence testing for recruitment) and in a larger sample (N=2949 job applicants). We found that the NLMs outperformed the Nominal Response Model (Bock, 1970) and provided significant reliability gains compared with their binary logistic counterparts. In line with previous research, the gain in reliability was especially obtained at low ability levels. Implications and practical recommendations are discussed.

When Sources Honestly Provide Their Biased Opinion: Bias as a Distinct Source Perception With Independent Effects on Credibility and Persuasion

Anecdotally, attributions that others are biased pervade many domains. Yet, research examining the effects of perceptions of bias is sparse, possibly due to some prior researchers conflating bias with untrustworthiness. We sought to demonstrate that perceptions of bias and untrustworthiness are separable and have independent effects. The current work examines these differences in the persuasion domain, but this distinction has implications for other domains as well. Two experiments clarify the conceptual distinction between bias (skewed perception) and untrustworthiness (dishonesty) and three studies demonstrate that source bias can have a negative effect on persuasion and source credibility beyond any parallel effects of untrustworthiness, lack of expertise, and dislikability. The current work suggests that bias is an independent, but understudied source characteristic.

Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods

Publication bias and questionable research practices in primary research can lead to badly overestimated effects in meta-analysis. Methodologists have proposed a variety of statistical approaches to correct for such overestimation. However, it is not clear which methods work best for data typically seen in psychology. Here, we present a comprehensive simulation study in which we examined how some of the most promising meta-analytic methods perform on data that might realistically be produced by research in psychology. We simulated several levels of questionable research practices, publication bias, and heterogeneity, and used study sample sizes empirically derived from the literature. Our results clearly indicated that no single meta-analytic method consistently outperformed all the others. Therefore, we recommend that meta-analysts in psychology focus on sensitivity analyses—that is, report on a variety of methods, consider the conditions under which these methods fail (as indicated by simulation studies such as ours), and then report how conclusions might change depending on which conditions are most plausible. Moreover, given the dependence of meta-analytic methods on untestable assumptions, we strongly recommend that researchers in psychology continue their efforts to improve the primary literature and conduct large-scale, preregistered replications. We provide detailed results and simulation code at https://osf.io/rf3ys and interactive figures at http://www.shinyapps.org/apps/metaExplorer/ .

Improving Psychological Science through Transparency and Openness: An Overview

The ability to independently verify and replicate observations made by other researchers is a hallmark of science. In this article, we provide an overview of recent discussions concerning replicability and best practices in mainstream psychology with an emphasis on the practical benefists to both researchers and the field as a whole. We first review challenges individual researchers face in producing research that is both publishable and reliable. We then suggest methods for producing more accurate research claims, such as transparently disclosing how results were obtained and analyzed, preregistering analysis plans, and publicly posting original data and materials. We also discuss ongoing changes at the institutional level to incentivize stronger research. These include officially recognizing open science practices at the journal level, disconnecting the publication decision from the results of a study, training students to conduct replications, and publishing replications. We conclude that these open science practices afford exciting low-cost opportunities to improve the quality of psychological science.

Conducting Replication Studies With Confidence

Although essential to the development of a robust evidence base for nurse educators, the concepts of replication and reproducibility have received little attention in the nursing education literature. In this Methodology Corner installment, the concepts of study replication and reproducibility are explored in depth. In designing, conducting, and documenting the findings of studies in nursing education, researchers are encouraged to make design choices that improve study replicability and reproducibility of study findings. [J Nurs Educ. 2018;57(11):638-640.].

Replication and Reproducibility in Cross-Cultural Psychology

Replication is the scientific gold standard that enables the confirmation of research findings. Concerns related to publication bias, flexibility in data analysis, and high-profile cases of academic misconduct have led to recent calls for more replication and systematic accumulation of scientific knowledge in psychological science. This renewed emphasis on replication may pose specific challenges to cross-cultural research due to inherent practical difficulties in emulating an original study in other cultural groups. The purpose of the present article is to discuss how the core concepts of this replication debate apply to cross-cultural psychology. Distinct to replications in cross-cultural research are examinations of bias and equivalence in manipulations and procedures, and that targeted research populations may differ in meaningful ways. We identify issues in current psychological research (analytic flexibility, low power) and possible solutions (preregistration, power analysis), and discuss ways to implement best practices in cross-cultural replication attempts.

Conditions for the clean slate effect after success or failure

The act of physically cleaning one's hands may reduce the impact of past experiences, termed clean-slate effect. Cleaning was found to affect negative, neutral, and mildly positive states. We extend this influence to success, a self-serving state. We manipulated success vs. failure and measured changes in optimism (Experiment 1) or self-esteem (Experiment 2). Moreover, we examined boundary conditions for the clean-slate effect. Experiment 1 indicates that the influence of performance on optimism diminishes if participants knew (compared to did not know) they were cleaning their hands. Experiment 2 indicates that the influence of performance on self-esteem diminishes if participants cleaned themselves (compared to an object). These results suggest that the clean-slate effect requires both awareness and self-reference of the cleaning act. Thus, the clean-slate effect seems to depend on both conscious inferences and automatic processes. A meta-analysis across the experiments confirms a moderate-sized clean-slate effect.

Metacognitive Reflection as a Moderator of Attitude Strength Versus Attitude Bolstering: Implications for Attitude Similarity and Attraction

"Strong" attitudes often have greater impact than "weak" attitudes. However, emerging research suggests that weak (uncertain) attitudes can substantially influence thinking or behavior. We propose metacognitive reflection as a moderator between traditional strength patterns and these emerging attitude bolstering patterns. Across six studies, research participants encountered a target person who agreed or disagreed with participants' attitudes. When focused on evaluating the target, attitudes predicted target evaluations better when the attitude was held with certainty (Study 1A), or after certainty had been primed (Studies 2A and 3; strength effects). However, when engaged in attitudinal social comparison (metacognitive reflection), attitudes better predicted target evaluation when the attitudes were held with doubt (Study 1B), or after doubt had been primed (Studies 2B and 3; bolstering effects). Expected change in certainty served as a mediator of attitude effects in metacognitive reflection but not target-focus conditions when doubt had been primed (Study 4).

Holding replication studies to mainstream standards of evidence

Replications can make theoretical contributions, but are unlikely to do so if their findings are open to multiple interpretations (especially violations of psychometric invariance). Thus, just as studies demonstrating novel effects are often expected to empirically evaluate competing explanations, replications should be held to similar standards. Unfortunately, this is rarely done, thereby undermining the value of replication research.