Estimation in meta-analyses of response ratios

Standardization and other approaches to meta-analyze differences in means

Meta-analysts often use standardized mean differences (SMD) to combine mean effects from studies in which the dependent variable has been measured with different instruments or scales. In this tutorial we show how the SMD is properly calculated as the difference in means divided by a between-subject reference-group, control-group, or pooled pre-intervention SD, usually free of measurement error. When combining mean effects from controlled trials and crossovers, most meta-analysts have divided by either the pooled SD of change scores, the pooled SD of post-intervention scores, or the pooled SD of pre- and post-intervention scores, resulting in SMDs that are biased and difficult to interpret. The frequent use of such inappropriate standardizing SDs by meta-analysts in three medical journals we surveyed is due to misleading advice in peer-reviewed publications and meta-analysis packages. Even with an appropriate standardizing SD, meta-analysis of SMDs increases heterogeneity artifactually via differences in the standardizing SD between settings. Furthermore, the usual magnitude thresholds for standardized mean effects are not thresholds for clinically important differences. We therefore explain how to use other approaches to combining mean effects of disparate measures: log transformation of factor effects (response ratios) and of percent effects converted to factors; rescaling of psychometrics to percent of maximum range; and rescaling with minimum clinically important differences. In the absence of clinically important differences, we explain how standardization after meta-analysis with appropriately transformed or rescaled pre-intervention SDs can be used to assess magnitudes of a meta-analyzed mean effect in different settings.

The effect of livestock grazing on plant diversity and productivity of mountainous grasslands in South America - A meta-analysis

Mountainous grasslands in South America, characterized by their high diversity, provide a wide range of contributions to people, including water regulation, soil erosion prevention, livestock feed provision, and preservation of cultural heritage. Prior research has highlighted the significant role of grazing in shaping the diversity and productivity of grassland ecosystems, especially in highly productive, eutrophic systems. In such environments, grazing has been demonstrated to restore grassland plant diversity by reducing primary productivity. However, it remains unclear whether these findings are applicable to South American mountainous grasslands, where plants are adapted to different environmental conditions. To address this uncertainty, we conducted a meta-analysis of experiments excluding livestock grazing to assess its impact on plant diversity and productivity across mountainous grasslands in South America. In alignment with studies in temperate grasslands, our findings indicated that herbivore exclusion resulted in increased aboveground biomass but reduced species richness and Shannon diversity. The effects of grazing exclusion became more pronounced with longer durations of exclusion; nevertheless, they remained resilient to various climatic conditions, including mean annual precipitation and mean annual temperature, as well as the evolutionary history of grazing. In contrast to results observed in temperate grasslands, the reduction in species richness due to herbivore exclusion was not associated with increased aboveground biomass. This suggests that the processes governing (sub)tropical grassland plant diversity may differ from those in temperate grasslands. Consequently, further research is necessary to better understand the specific factors influencing plant diversity and productivity in South American montane grasslands and to elucidate the ecological implications of herbivore exclusion in these unique ecosystems.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

A robust and readily implementable method for the meta-analysis of response ratios with and without missing standard deviations

The log response ratio, lnRR, is the most frequently used effect size statistic for meta-analysis in ecology. However, often missing standard deviations (SDs) prevent estimation of the sampling variance of lnRR. We propose new methods to deal with missing SDs via a weighted average coefficient of variation (CV) estimated from studies in the dataset that do report SDs. Across a suite of simulated conditions, we find that using the average CV to estimate sampling variances for all observations, regardless of missingness, performs with minimal bias. Surprisingly, even with missing SDs, this simple method outperforms the conventional approach (basing each effect size on its individual study-specific CV) with complete data. This is because the conventional method ultimately yields less precise estimates of the sampling variances than using the pooled CV from multiple studies. Our approach is broadly applicable and can be implemented in all meta-analyses of lnRR, regardless of 'missingness'.