Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology

Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%-38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%-12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2-3 times (Type M errors) and variability by 4-10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts. Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experiments and non-manipulative observations had very similar results in terms of their power, Type M and S errors. Therefore, the previous assumption about the superiority of manipulative experiments in terms of power is overstated. These results call for highly powered field studies to reliably inform theory building and policymaking, via more collaboration and team science, and large-scale ecosystem facilities. Future studies also require transparent reporting and open science practices to approach reproducible and reliable empirical work and evidence synthesis.

Analyzing Spin in Abstracts of Orthopaedic Randomized Controlled Trials with Statistically Insignificant Primary Endpoints

PURPOSE

Our primary aim of our study is to evaluate the prevalence of spin among abstracts in orthopaedic randomized controlled trials (RCTs) with nonsignificant primary endpoints.

METHODS

This study was conducted in accordance with a previously written protocol publicly available via the Open Science Framework. PubMed (which includes Medline) was searched for RCTs in orthopaedic surgery. The articles that were identified were then uploaded to Rayyan, and the abstracts were screened for inclusion. To be included, a trial had to have randomized the patients for intervention, statistically compare multiple groups, and had a primary endpoint that was not significant. Odds ratios and summary statistics (frequencies and proportions) were then calculated for spin in the abstracts.

RESULTS

Of the 780 articles retrieved from our search string, 250 articles met the inclusion criteria. Analysis resulted in 112/250 (44.80%; [95% CI, 38.64-50.96]) RCTs that containing spin within the abstract. Of the 112 RCTs, 52 (46.43%; [95% CI, 37.19-55.66]) had spin in the results, and 89 (79.46%; [95% CI, 71.98-86.95]) had spin in the conclusion of the abstract. The Journal of Bone and Joint Surgery was found to have the highest prevalence of spin (21/37, 56.76% [95% CI, 40.79%-72.72%]) while Arthroscopy: The Journal of Arthroscopic & Related Surgery had the lowest prevalence of spin (5/15, 33.33% [95% CI, 9.48%-57.19%]). No correlation was found between industry funding and increased odds of spin in the abstract (uOR, 1.10; 95% CI, 0.45-2.63). Discrepancies for our primary endpoint, prevalence of spin among abstracts, were analyzed with Gwet's AC1 inter-rater statistic and found to be 81% [95% CI, 0.75-0.87].

CONCLUSION

Spin was found in 44.8% of the abstracts within our sample of orthopaedic RCTs. Nonsignificant primary data were often represented to seem significant, many orthopaedic RCTs did not indicate primary endpoints, and orthopaedic RCTs infrequently reported trial registration.

Managing the incidence of selective reporting bias: a survey of Cochrane review groups

BACKGROUND

Selective reporting bias (SRB), the incomplete publication of outcomes measured or of analyses performed in a study, may lead to the over- or underestimation of treatment effects or harms. Cochrane systematic reviews of interventions are required to assess the risk of SRB, achieved in part by applying the Cochrane risk of bias tool to each included randomised trial. The Cochrane Handbook outlines strategies for a comprehensive risk of bias assessment, but the extent to which these are followed by Cochrane review groups (CRGs) has not been assessed to date. The objective of this study was to determine the methods which CRGs require of their authors to address SRB within systematic reviews, and how SRB risk assessments are verified.

METHODS

A cross-sectional survey was developed and distributed electronically to the 52 CRGs involved in intervention reviews.

RESULTS

Responses from 42 CRGs show that the majority refer their authors to the Cochrane Handbook for specific instruction regarding assessments of SRB. The handbook strategies remain variably enforced, with 57 % (24/42) of CRGs not requiring review authors to search for included trial protocols and 31 % (13/42) not requiring that contact with individual study authors be attempted. Only half (48 %, 20/42) of the groups consistently verify review authors' assessments of the risk of SRB to ensure completeness.

CONCLUSIONS

A range of practices are used by CRGs for addressing SRB, with many steps outlined in the Cochrane Handbook being encouraged but not required. The majority of CRGs do not consider their review authors to be sufficiently competent to assess for SRB, yet risk of bias assessments are not always verified by editors before publication. The implications of SRB may not be fully appreciated by all CRGs, and resolving the identified issues may require an approach targeting several steps in the systematic review process.

Using network meta-analysis to evaluate the existence of small-study effects in a network of interventions

Suggested methods for exploring the presence of small-study effects in a meta-analysis and the possibility of publication bias are associated with important limitations. When a meta-analysis comprises only a few studies, funnel plots are difficult to interpret, and regression-based approaches to test and account for small-study effects have low power. Assuming that the cause of funnel plot asymmetry is likely to affect an entire research field rather than only a particular comparison of interventions, we suggest that network meta-regression is employed to account for small-study effects in a set of related meta-analyses. We present several possible models for the direction and distribution of small-study effects and we describe the methods by re-analysing two published networks. Copyright © 2012 John Wiley & Sons, Ltd.