Understanding Data When Interactions Are Present or Hypothesized

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking

Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

Interaction effect: Are you doing the right thing?

How to correctly interpret interaction effects has been largely discussed in scientific literature. Nevertheless, misinterpretations are still frequently observed, and neuroscience is not exempt from this trend. We reviewed 645 papers published from 2019 to 2020 and found that, in the 93.2% of studies reporting a statistically significant interaction effect (N = 221), post-hoc pairwise comparisons were the designated method adopted to interpret its results. Given the widespread use of this approach, we aim to: (1) highlight its limitations and how it can lead to misinterpretations of the interaction effect; (2) discuss more effective and powerful ways to correctly interpret interaction effects, including both explorative and model selection procedures. The paper provides practical examples and freely accessible online materials to reproduce all analyses.

School Selection and the Social Class Divide: How Tracking Contributes to the Reproduction of Inequalities

Selection practices in education, such as tracking, may represent a structural obstacle that contributes to the social class achievement gap. We hypothesized that school’s function of selection leads evaluators to reproduce social inequalities in tracking decisions, even when performance is equal. In two studies, participants (students playing the role of teachers, N = 99, or preservice and in-service teachers, N = 70) decided which school track was suitable for a pupil whose socioeconomic status (SES) was manipulated. Although pupils’ achievement was identical, participants considered a lower track more suitable for lower SES than higher SES pupils, and the higher track more suitable for higher SES than lower SES pupils. A third study ( N = 160) revealed that when the selection function of school was salient, rather than its educational function, the gap in tracking between social classes was larger. The selection function of tracking appears to encourage evaluators to artificially create social class inequalities.

Performing Contrast Analysis in Factorial Designs: From NHST to Confidence Intervals and Beyond

Because of the continuing debates about statistics, many researchers may feel confused about how to analyze and interpret data. Current guidelines in psychology advocate the use of effect sizes and confidence intervals (CIs). However, researchers may be unsure about how to extract effect sizes from factorial designs. Contrast analysis is helpful because it can be used to test specific questions of central interest in studies with factorial designs. It weighs several means and combines them into one or two sets that can be tested with t tests. The effect size produced by a contrast analysis is simply the difference between means. The CI of the effect size informs directly about direction, hypothesis exclusion, and the relevance of the effects of interest. However, any interpretation in terms of precision or likelihood requires the use of likelihood intervals or credible intervals (Bayesian). These various intervals and even a Bayesian t test can be obtained easily with free software. This tutorial reviews these methods to guide researchers in answering the following questions: When I analyze mean differences in factorial designs, where can I find the effects of central interest, and what can I learn about their effect sizes?

An Equity Theory Analysis of the Impact of Forgiveness and Retribution on Transgressor Compliance

Forgiveness, when thought of as an unsolicited gift, may increase the perceived debt of the transgressor to the victim whereas retribution should reduce it. Male undergraduates participated in a study designed to test this equity interpretation of forgiveness and retribution. Participants were induced to break a piece of electronic equipment during an ostensible memory study; the reaction of the experimenter served as the experimental manipulation. Participants experienced one of forgiveness, retribution, both retribution and forgiveness, or neither, and were then asked to comply with a request from the experimenter as an indirect measure of perceived inequity. Consistent with an equity analysis, a planned contrast analysis indicated that forgiveness alone yielded the most compliance and retribution yielded the least compliance.

When It’s Okay That I Don’t Play

Being excluded and ignored has been shown to threaten fundamental human needs and cause pain. Such reflexive reactions to social exclusion have been conceptualized as direct and unmoderated (temporal need threat model of ostracism). Here, we propose an extension and argue that reflexive reactions depend on how social exclusion situations are construed. If being excluded is understood as a violation of an inclusion norm, individuals will react with pain and threat. In contrast, if being excluded is consistent with the prevailing norm, the exclusion situation is interpreted as less threatening, and negative reflexive reactions to ostracism should be attenuated. Four studies empirically support this conceptual model. Studies 3 and 4 further show that to guide situated construal, the norm has to be endorsed by the individual. In both Studies 1 and 3, the effect of the norm is mediated by the objective situation’s subjective construal.

Vicissitudes of Desire

Recent research on subliminal persuasion has documented effects primarily when people have a preexisting need related to the target of influence. Based on the situated inference model of priming effects (Loersch & Payne, 2011), we propose a novel matching mechanism and describe how it expands the circumstances under which subliminal primes can produce persuasive effects, doing so without a consideration of preexisting need states. In two studies, we alter the desirability of various products by selecting subliminal primes that address the basic questions participants consider while judging product desirability. Subliminal persuasion depends on the precise match between the subliminal primes and the question under consideration. These results are evident when the question participants consider varies naturally due to the type of product that is judged, and when the core question is directly manipulated by altering the aspect of a product on which participants focus.

Does Effort Suppress Cognition After Traumatic Brain Injury? A Re-Examination of the Evidence for the Word Memory Test

Green, Rohling, Lees-Haley, and Allen (2001) suggested that scores on a test of "effort," the Word Memory Test (WMT), explains more variance in outcome after brain injury than does injury severity. As a consequence, Green and colleagues recommend using the WMT to control for sub-optimal effort in neuropsychological evaluations and group research. We re-examine the evidence for their conclusions and argue that identifying a larger proportion of explained variance is not in itself evidence of validity unless the premise to be proven is already assumed, namely, that the test is a valid measure of effort. Instead, the crux of Green and colleagues claim for the validity of the WMT implies an interaction between effort and injury severity on outcome scores, although the specific interaction has not been tested in their previous research. We failed to find any evidence for this interaction in a sample of 100 Australian litigants. We conclude that our data do not support the view that effort, as measured by the WMT, interacts with injury severity to suppress cognition after brain injury.

Positive associations among interpersonal contact, motivation, and implicit and explicit attitudes toward gay men

A correlational study explored the role of intergroup contact and motivation to respond without prejudice on heterosexuals' expression of explicit and implicit (unconscious) bias against gay men. Participants who reported having more relationships and closer relationships with gay, lesbian, or bisexual people tended to exhibit more favorable attitudes toward gay men on implicit as well as explicit attitude measures. Attitudes were also related to self-reported motivation to be non-prejudiced, including motivation stemming from sources internal as well as external to the individual. Multiple regression analyses showed that contact and motivation explain unique variance in attitude but that motivation is a relatively stronger predictor. The results are interpreted to suggest that implicit and explicit prejudice may be reduced through motivation coupled with positive contact experiences.