Primary Study Quality in Psychological Meta-Analyses: An Empirical Assessment of Recent Practice

Power to Detect What? Considerations for Planning and Evaluating Sample Size

ACADEMIC ABSTRACT

In the wake of the replication crisis, social and personality psychologists have increased attention to power analysis and the adequacy of sample sizes. In this article, we analyze current controversies in this area, including choosing effect sizes, why and whether power analyses should be conducted on already-collected data, how to mitigate the negative effects of sample size criteria on specific kinds of research, and which power criterion to use. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings), given the limitations of interest-based minimums or field-wide effect sizes. We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

PUBLIC ABSTRACT

Recently, social-personality psychology has been criticized for basing some of its conclusions on studies with low numbers of participants. As a result, power analysis, a mathematical way to ensure that a study has enough participants to reliably "detect" a given size of psychological effect, has become popular. This article describes power analysis and discusses some controversies about it, including how researchers should derive assumptions about effect size, and how the requirements of power analysis can be applied without harming research on hard-to-reach and marginalized communities. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings). We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

A systematic review of perception of affordances for the person-plus-object system

Human behavior often involves the use of an object held by or attached to the body, which modifies the individual's action capabilities. Moreover, most everyday behaviors consist of sets of behaviors that are nested over multiple spatial and temporal scales, which require perceiving and acting on nested affordances for the person-plus-object system. This systematic review investigates how individuals attune to information about affordances involving the person-plus-object system and how they (re)calibrate their actions to relevant information. We analyzed 71 articles-34 on attunement and 37 on (re)calibration with healthy participants-that experimentally investigated the processes involved in the perception of affordances for the person-plus-object system (including attunement, calibration, and recalibration). With respect to attunement, objects attached to the body create a multiplicity of affordances for the person-plus-object system, and individuals learned (1) to detect information about affordances of (and for) the person-plus-object system in a task and (2) to choose whether, when, and how to exploit those affordances to perform that task. Concerning (re)calibration, individuals were able (1) to quickly scale their actions in relation to the (changed) action capabilities of the person-plus-object system and (2) to perceive multiple functionally equivalent ways to exploit the affordances of that system, and these abilities improved with practice. Perceiving affordances for the person-plus-object system involves learning to detect the information about such affordances (attunement) and the scaling of behaviors to such information (calibration). These processes imply a general ability to incorporate an object attached to the body into an integrated person-plus-object system.

Emotion regulation strategies among individuals with borderline personality disorder relative to other groups: A review

Borderline personality disorder (BPD) is associated with difficulties in emotion regulation (ER) abilities. Investigations of ER strategies in BPD have been less robust. This systematic review identified 55 studies comparing ER strategy use between individuals diagnosed with BPD versus psychiatric and non-psychiatric comparison groups. Individuals with BPD reported more frequent maladaptive and less frequent adaptive ER strategy use than non-psychiatric controls. Results were less consistent relative to psychiatric comparison groups, though individuals with BPD reported greater self-criticism and avoidance. Groups responded comparably to instructed use of adaptive (but not maladaptive) ER strategies. This body of research would benefit from further examination of the roles of psychiatric comorbidity and problematic behaviours in the relationship between BPD and ER strategies.

A mega-analysis of personality prediction: Robustness and boundary conditions

Decades of studies identify personality traits as prospectively associated with life outcomes. However, previous investigations of personality characteristic-outcome associations have not taken a principled approach to covariate use or other sampling strategies to ensure the robustness of personality-outcome associations. The result is that it is unclear (a) whether personality characteristics are associated with important outcomes after accounting for a range of background variables, (b) for whom and when personality-outcome associations hold, and (c) that background variables are most important to account for. The present study examines the robustness and boundary conditions of personality-outcome associations using prospective Big Five associations with 14 health, social, education/work, and societal outcomes across eight different person- and study-level moderators using individual participant data from 171,395 individuals across 10 longitudinal panel studies in a mega-analytic framework. Robustness and boundary conditions were systematically tested using two approaches: propensity score matching and specification curve analysis. Three findings emerged: First, personality characteristics remain robustly associated with later life outcomes. Second, the effects generalize, as there are few moderators of personality-outcome associations. Third, robustness was differential across covariate choice in nearly half of the tested models, with the inclusion or exclusion of some of these flipping the direction of association. In summary, personality characteristics are robustly associated with later life outcomes with few moderated associations. However, researchers still need to be careful in their choices of covariates. We discuss how these findings can inform studies of personality-outcome associations, as well as recommendations for covariate inclusion. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The effect of sleep deprivation and restriction on mood, emotion, and emotion regulation: three meta-analyses in one

STUDY OBJECTIVES

New theory and measurement approaches have facilitated nuanced investigation of how sleep loss impacts dimensions of affective functioning. To provide a quantitative summary of this literature, three conceptually related meta-analyses examined the effect of sleep restriction and sleep deprivation on mood, emotion, and emotion regulation across the lifespan (i.e. from early childhood to late adulthood).

METHODS

A total of 241 effect sizes from 64 studies were selected for inclusion, and multilevel meta-analytic techniques were used when applicable.

RESULTS

There was a moderate, positive effect of sleep loss on negative mood (g = 0.45), which was stronger for studies with younger samples, as well as a large, negative effect of sleep loss on positive mood (g = -0.94). For negative mood only, studies that used total sleep deprivation had larger effect sizes than studies that restricted sleep. After correcting for publication bias, a modest but significant negative effect for sleep loss on emotion (g = -0.11) was found; the valence of emotional stimuli did not change the direction of this effect, and type of sleep manipulation was also not a significant moderator. Finally, sleep restriction had a small, negative effect on adaptive emotion regulation (g = -0.32), but no significant impact on maladaptive emotion regulation (g = 0.14); all studies on adaptive emotion regulation were conducted with youth samples.

CONCLUSIONS

Sleep loss compromises optimal affective functioning, though the magnitude of effects varies across components. Findings underscore the importance of sleep for healthy affective outcomes.

Community-Augmented Meta-Analyses (CAMAs) in Psychology

Abstract. The limits of static snapshot meta-analyses and the relevance of reproducibility and data accessibility for cumulative meta-analytic research are outlined. A publication format to meet these requirements is presented: Community-augmented meta-analyses (CAMA). We give an overview of existing systems implementing this approach and compare these in terms of scope, technical implementation, data collection and augmentation, data curation, tools available for analysis, and methodological flexibility.

The psychology of obesity: An umbrella review and evidence-based map of the psychological correlates of heavier body weight

Psychological factors may explain why some people develop obesity and others remain a normal weight during their life course. We use an umbrella review approach to build an evidence-based map of the psychological correlates of heavier body weight. Synthesising findings from 42 meta-analyses that have examined associations between psychological factors and heavier body weight, we assessed level of evidence for a range of cognitive, psychosocial and mental health individual difference factors. There is convincing evidence that impaired mental health is associated with heavier body weight and highly suggestive evidence that numerous cognitive factors are associated with heavier body weight. However, the relatively low methodological quality of meta-analyses resulted in lower evidential certainty for most psychosocial factors. Psychological correlates of heavier body weight tended to be small in statistical size and on average, people with obesity were likely to be more psychologically similar than different to people with normal weight. We consider implications for understanding the development of heavier body weight and identifying effective public health interventions to reduce obesity.

Psychological adjustment outcomes among pediatric brain tumor survivors: A meta-analysis

BACKGROUND

Pediatric brain tumor survivors (PBTS) are at significant risk for psychological adjustment difficulties, including greater depressive and anxious symptomology. Systematic reviews have identified this heightened risk among youth with medical conditions, but these reviews have not been specific to PBTS. Therefore, the current study aimed to directly examine the psychological adjustment of PBTS as compared to healthy peers.

PROCEDURE

A systematic review and meta-analysis was conducted using PubMed, PsychInfo, and Academic Search Premier databases. The search yielded 2833 articles, with 22 articles meeting inclusion criteria.

RESULTS

A statistically significant overall medium effect size (Hedge's g = 0.32) indicated that PBTS exhibited poorer overall psychological adjustment relative to healthy comparison groups. Studies that included younger children were associated with larger between-group differences. When evaluating specific outcomes, PBTS had relatively higher levels of depressive symptoms (Hedge's g = 0.36), anxious symptoms (Hedge's g = 0.11), and general distress (Hedge's g = 0.22), but not more externalizing problems.

CONCLUSIONS

The present study confirmed that PBTS are indeed at greater risk for psychological adjustment difficulties relative to healthy comparison groups. These findings highlight the importance of psychosocial screening among this population. Given that depressive symptoms were the most elevated relative to healthy peers, investigation of such symptomatology among PBTS is particularly important.

Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

OBJECTIVES

Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.

DESIGN

A meta-epidemiological study.

PARTICIPANTS

We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.

PRIMARY AND SECONDARY OUTCOMES

Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.

RESULTS

According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.

CONCLUSION

The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.

Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

OBJECTIVES

Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.

DESIGN

A meta-epidemiological study.

PARTICIPANTS

We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.

PRIMARY AND SECONDARY OUTCOMES

Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.

RESULTS

According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.

CONCLUSION

The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.