Commentary: Misguided Effort with Elusive Implications, and Sifting Signal from Noise with Replication Science

A Multisite Preregistered Paradigmatic Test of the Ego-Depletion Effect

We conducted a preregistered multilaboratory project (k = 36; N = 3,531) to assess the size and robustness of ego-depletion effects using a novel replication method, termed the paradigmatic replication approach. Each laboratory implemented one of two procedures that was intended to manipulate self-control and tested performance on a subsequent measure of self-control. Confirmatory tests found a nonsignificant result (d = 0.06). Confirmatory Bayesian meta-analyses using an informed-prior hypothesis (δ = 0.30, SD = 0.15) found that the data were 4 times more likely under the null than the alternative hypothesis. Hence, preregistered analyses did not find evidence for a depletion effect. Exploratory analyses on the full sample (i.e., ignoring exclusion criteria) found a statistically significant effect (d = 0.08); Bayesian analyses showed that the data were about equally likely under the null and informed-prior hypotheses. Exploratory moderator tests suggested that the depletion effect was larger for participants who reported more fatigue but was not moderated by trait self-control, willpower beliefs, or action orientation.

Detrimental effects of prior self-control exertion on subsequent sporting skill performance

The prior exertion of self‐control has previously been shown to negatively affect physical performance, yet the effects on complex sporting skill performance have not been examined. Therefore, this study examined whether prior self‐control exertion influences performance on a field hockey task, alongside measuring plasma cortisol concentration and attention as potential mechanisms to explain any effects.

Following familiarization, 13 male hockey players (20 ± 1 years) participated in a randomized, order‐balanced, crossover design. For the manipulation of self‐control, participants completed an incongruent (self‐control exertion trial) or a congruent (control trial) Stroop task. Skill performance was assessed using a field hockey skills task. Capillary blood samples, for the determination of plasma cortisol concentration, were taken at baseline, post‐Stroop task, and post‐field hockey skills task. Cognitive tests of attention (RVIP and Flanker tasks) were completed following the field hockey skills task.

Participants made more errors in the latter stages of the field hockey skills task following self‐control exertion (trial*time interaction, p = 0.041). Participants also made more errors on the RVIP task following self‐control exertion (p = 0.035); yet the time taken to complete the hockey skills task, performance on the flanker task, and plasma cortisol concentrations were unaffected (all p > 0.05).

Overall, these findings suggest that prior self‐control exertion has detrimental effects on subsequent sporting skill performance (more errors made on the field hockey task), which may be explained by poorer sustained attention (lower accuracy on the RVIP task). This suggests that athletes should aim to avoid self‐control exertion before a competitive match to optimize performance.

A Multilab Replication of the Ego Depletion Effect

There is an active debate regarding whether the ego depletion effect is real. A recent preregistered experiment with the Stroop task as the depleting task and the antisaccade task as the outcome task found a medium-level effect size. In the current research, we conducted a preregistered multilab replication of that experiment. Data from 12 labs across the globe (N = 1,775) revealed a small and significant ego depletion effect, d = 0.10. After excluding participants who might have responded randomly during the outcome task, the effect size increased to d = 0.16. By adding an informative, unbiased data point to the literature, our findings contribute to clarifying the existence, size, and generality of ego depletion.

Reflections on a registered report replicating a body of dyadic cross-sectional research

This article reflects on a new kind of registered report (RR) that replicated the work of an early career researcher. The research items targeted in this RR were peer-reviewed, cross-sectional, dyadic studies to which the first author of this RR had contributed. The findings being replicated are not noteworthy for their prestige or representativeness of the wider field. Instead, this method of replication may have several benefits and less desirable qualities for the researcher and research team whose work is being replicated, for science more broadly, and for relationship science specifically, reviewed herein. The authors hope that this reflection inspires researchers to improve upon their methodology by incorporating replication of their work early and often into their own research process.

Manipulation of the Duration of the Initial Self-Control Task Within the Sequential-Task Paradigm: Effect on Exercise Performance

Self-control exertion on an initial task has been associated with impaired performance on subsequent physical tasks also requiring self-control; an effect suggested to be mediated by changes in perceptions of pain and motivation. However, the effects of spending longer on the initial self-control task are unknown. This study, therefore, explored the potential for the duration of the initial self-control task to influence subsequent physical performance, perceptions of pain, and perceived motivation; particularly during the early stages of the physical task. In a within-subject design, 29 participants (11 male, 18 female) completed a wall-sit task until volitional exhaustion, on four separate occasions. Prior to each wall-sit, participants completed either a non-self-control task (congruent Stroop task) for 4 min, or a self-control task (incongruent Stroop task) for 4 (short duration), 8 (medium duration), or 16 (long duration) min. Participant's perceptions of pain and motivation were recorded every 30 s during the wall-sit. Wall-sit performance time was analyzed using one-way ANOVA and perceptions of pain and motivation analyzed using multi-level modeling. Wall-sit performance time was significantly longer on the non-self-control exertion trial compared to all other trials (all p < 0.01), as well as longer on both the short duration and medium duration self-control exertion trials compared to the long duration self-control exertion trial (both p < 0.001). Perceptions of initial (at 30 s) pain and motivation were different between the trials (main effect of trial: pain, p = 0.001; motivation, p < 0.001); whereby longer durations of self-control exertion increased perceptions of pain and decreased motivation. The decrease in motivation during the wall-sit task was greater on the long duration self-control exertion trial compared to all other trials (trial^∗time interactions, all p < 0.05). The present study provides novel evidence that spending longer on the initial self-control task led to greater detrimental effects on subsequent wall-sit performance time. Furthermore, longer duration self-control exertion tasks led to increased perceptions of pain and decreased motivation within the first 30 s of the wall-sit task, as well as a greater decrease in motivation across the wall-sit task. These attentional and motivational shifts may explain performance decrements following the exertion of self-control.

Are Difficult-To-Study Populations too Difficult to Study in a Reliable Way?

Abstract. Replication studies, pre-registration, and increases in statistical power will likely improve the reliability of scientific evidence. However, these measures face critical limitations in populations that are inherently difficult to study. Members of difficult-to-study populations (e.g., patients, children, non-human animals) are less accessible to researchers, which typically results in small-sample studies that are infeasible to replicate. Nevertheless, meta-analyses on clinical neuropsychological data suggest that difficult-to-study populations can be studied in a reliable way. These analyses often produce unbiased effect-size estimates despite aggregating across severely underpowered original studies. This finding can be attributed to a neuropsychological research culture involving the non-selective reporting of results from standardized and validated test procedures. Consensus guidelines, test manuals, and psychometric evidence constrain the methodological choices made by neuropsychologists, who regularly report the results from neuropsychological test batteries irrespective of their statistical significance or novelty. Comparable shifts toward more standardization and validation, complete result reports, and between-lab collaborations can allow for a meaningful and reliable study of psychological phenomena in other difficult-to-study populations.

Self-Control and Exercise: A Review of the Bi-Directional Relationship

Self-control has been defined as the mental capacity of an individual to alter, modify, change or override their impulses, desires, and habitual responses. In this review, we will discuss the bi-directional nature of the relationship between self-control and exercise. In brief, higher levels of trait self-control have been associated with greater exercise performance and adherence; whilst the depletion of state self-control has been shown to decrease performance and persistence on subsequent exercise tasks requiring self-control. In the opposite direction, long-term participation in exercise (and improved physical fitness) has been demonstrated to enhance self-control. Furthermore, an acute bout of exercise has been shown to enhance subsequent self-control, particularly when the exercise is of a moderate intensity and requires some degree of cognitive engagement. Throughout, when discussing each of these relationships, evidence will be drawn from other aspects of the review, where appropriate, to enhance our understanding of the observed effects. Finally, recommendations for future research will be made; including the importance of considering the bi-directional nature of the relationship, given that this has implications for our understanding of both self-control and exercise performance and adherence.

A Validation Study of the German Complex-Span Tasks and Some General Considerations on Task Translation Procedures in Cognitive Psychology

Abstract. Automated complex-span tasks are widely used to assess working-memory capacity and the English versions show good psychometric properties ( Unsworth, Heitz, Schrock, & Engle, 2005 ). However, it is generally an open question whether translated task versions have the same properties as the original versions and whether results obtained with translated tasks can be interpreted equivalently to those obtained with the original tasks. We translated the complex-span tasks and had a sample of German participants perform these tasks as well as a running-memory-span task and a reasoning test. We assessed the reliabilities of the German complex-span tasks and their construct and criterion-related validities. Extrapolating from cross-cultural literature, we also employed a test of measurement invariance to compare the correlational patterns as well as the construct structure between the German sample and a similar North-American sample. Results show that the German complex-span tasks are reliable and valid indicators of working-memory capacity and that they are metrically and functionally equivalent to the original versions. As measurement equivalence is an important but often neglected topic in basic cognitive psychology, we also highlight the general benefits of using equivalence tests when translating cognitive tasks.

Task duration and task order do not matter: no effect on self-control performance

The strength model of self-control proposes that all acts of self-control are energized by one global limited resource that becomes temporarily depleted by a primary self-control task, leading to impaired self-control performance in secondary self-control tasks. However, failed replications have cast doubt on the existence of this so-called ego depletion effect. Here, we investigated between-task (i.e., variation in self-control tasks) and within-task variation (i.e., task duration) as possible explanations for the conflicting literature on ego depletion effects. In a high-powered experiment (N = 709 participants), we used two established self-control tasks (Stroop task, transcription task) to test how variations in the duration of primary and secondary self-control tasks (2, 4, 8, or 16 min per task) affect the occurrence of an ego depletion effect (i.e., impaired performance in the secondary task). In line with the ego depletion hypothesis, subjects perceived longer lasting secondary tasks as more self-control demanding. Contrary to the ego depletion hypothesis, however, performance did neither suffer from prior self-control exertion, nor as a function of task duration. If anything, performance tended to improve when the primary self-control task lasted longer. These effects did not differ between the two self-control tasks, suggesting that the observed null findings were independent of task type.

Testing the validity of the attention control video: An eye-tracking approach of the ego depletion effect

The attention control video has been frequently applied to test the ego depletion effect. However, its validity has never been tested, a shortcoming we address in this preregistered study. In the first task, self-control strength was temporarily depleted in the depletion condition (n = 56) but remained intact in the control condition (n = 56). The attention control video served as the secondary task, and we assumed that the depletion condition would perform significantly worse compared to the control condition. Attention regulation was measured with an eye-tracking device. The results revealed that the gaze behavior in the two conditions differed statistically significantly; however, the actual difference was small, indicating that the attention control video may not be an optimal measure of self-control.

Searching for the bottom of the ego well: failure to uncover ego depletion in Many Labs 3

According to a popular model of self-control, willpower depends on a limited resource that can be depleted when we perform a task demanding self-control. This theory has been put to the test in hundreds of experiments showing that completing a task that demands high self-control usually hinders performance in any secondary task that subsequently taxes self-control. Over the last 5 years, the reliability of the empirical evidence supporting this model has been questioned. In the present study, we reanalysed data from a large-scale study-Many Labs 3-to test whether performing a depleting task has any effect on a secondary task that also relies on self-control. Although we used a large sample of more than 2000 participants for our analyses, we did not find any significant evidence of ego depletion: persistence on an anagram-solving task (a typical measure of self-control) was not affected by previous completion of a Stroop task (a typical depleting task in this literature). Our results suggest that either ego depletion is not a real effect or, alternatively, persistence in anagram solving may not be an optimal measure to test it.

Self-reports from behind the scenes: Questionable research practices and rates of replication in ego depletion research

The strength model of self-control is one of the most influential and well-established models of self-regulation in social psychology. However, recent attempts to replicate the ego depletion effect have sometimes failed. The goal of this study is to investigate self-reported replication rates and the frequency of a set of questionable research practices (QRP) in ego depletion research. A literature search resulted in 1721 researchers who had previously published on ego depletion. They were invited to participate in an anonymous online survey. The respondents (n = 277), on average, had published over three papers on ego depletion, and had completed more than two additional, unpublished studies. Respondents indicated that in more than 40% of their studies, results were similar in magnitude to those reported in the existing literature, and more than 60% reported conducting a priori power analyses. 39.2% of respondents were aware of other researchers who engaged in the surveyed QRP's, while 37.7% affirmed to have employed said QRP's. These results underline the importance of reducing QRP's to reliably test the validity of the ego depletion effect.

On Nomological Validity and Auxiliary Assumptions: The Importance of Simultaneously Testing Effects in Social Cognitive Theories Applied to Health Behavior and Some Guidelines

Tests of social cognitive theories provide informative data on the factors that relate to health behavior, and the processes and mechanisms involved. In the present article, we contend that tests of social cognitive theories should adhere to the principles of nomological validity, defined as the degree to which predictions in a formal theoretical network are confirmed. We highlight the importance of nomological validity tests to ensure theory predictions can be disconfirmed through observation. We argue that researchers should be explicit on the conditions that lead to theory disconfirmation, and identify any auxiliary assumptions on which theory effects may be conditional. We contend that few researchers formally test the nomological validity of theories, or outline conditions that lead to model rejection and the auxiliary assumptions that may explain findings that run counter to hypotheses, raising potential for ‘falsification evasion.’ We present a brief analysis of studies (k = 122) testing four key social cognitive theories in health behavior to illustrate deficiencies in reporting theory tests and evaluations of nomological validity. Our analysis revealed that few articles report explicit statements suggesting that their findings support or reject the hypotheses of the theories tested, even when findings point to rejection. We illustrate the importance of explicit a priori specification of fundamental theory hypotheses and associated auxiliary assumptions, and identification of the conditions which would lead to rejection of theory predictions. We also demonstrate the value of confirmatory analytic techniques, meta-analytic structural equation modeling, and Bayesian analyses in providing robust converging evidence for nomological validity. We provide a set of guidelines for researchers on how to adopt and apply the nomological validity approach to testing health behavior models.

Depletion, moral identity, and unethical behavior: Why people behave unethically after self-control exertion

Self-control enables people to resist short-term temptations in the service of long-term goals. Previous exertion of self-control leads to a state of ego depletion. Three studies demonstrated that ego depletion leads to a high level of unethical behavior. These studies also hypothesized and confirmed that depleted individuals behave unethically because of low moral identity. Study 1 found that depleted participants were more likely to over-report their performance than non-depleted participants. Study 2 revealed that depletion reduced people's moral identity, which in turn increased their propensity to engage in unethical behavior. Study 3 proved that priming moral identity eliminated the effect of depletion on cheating. Findings suggest that reduced moral identity accounts for the effect of self-control depletion on unethical behavior.

Using meta-analytic path analysis to test theoretical predictions in health behavior: An illustration based on meta-analyses of the theory of planned behavior

OBJECTIVE

Synthesizing research on social cognitive theories applied to health behavior is an important step in the development of an evidence base of psychological factors as targets for effective behavioral interventions. However, few meta-analyses of research on social cognitive theories in health contexts have conducted simultaneous tests of theoretically-stipulated pattern effects using path analysis. We argue that conducting path analyses of meta-analytic effects among constructs from social cognitive theories is important to test nomological validity, account for mediation effects, and evaluate unique effects of theory constructs independent of past behavior. We illustrate our points by conducting new analyses of two meta-analyses of a popular theory applied to health behaviors, the theory of planned behavior.

METHOD

We conducted meta-analytic path analyses of the theory in two behavioral contexts (alcohol and dietary behaviors) using data from the primary studies included in the original meta-analyses augmented to include intercorrelations among constructs and relations with past behavior missing from the original analysis.

RESULTS

Findings supported the nomological validity of the theory and its hypotheses for both behaviors, confirmed important model processes through mediation analysis, demonstrated the attenuating effect of past behavior on theory relations, and provided estimates of the unique effects of theory constructs independent of past behavior.

CONCLUSIONS

Our analysis illustrates the importance of conducting a simultaneous test of theory-stipulated effects in meta-analyses of social cognitive theories applied to health behavior. We recommend researchers adopt this analytic procedure when synthesizing evidence across primary tests of social cognitive theories in health.