Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology

Developing isotopic proxies to reconstruct the metabolic rates and thermal histories of octopus

Understanding an animal's metabolic rate and thermal history is pivotal for ecological research. Recent studies have proposed the use of stable carbon and oxygen isotopes (δ13C and δ18O) in biogenic carbonates as proxies of metabolic rate and experienced temperature, respectively, to overcome the challenges of directly measuring these data in the field. Our study represents the first experimental investigation to develop δ13C and δ18O proxies in octopus. Octopus berrima hatchlings were raised in captivity, at varying water temperatures, for up to 110 days. O. berrima statoliths were then subsequently analysed for δ13C and δ18O values. The proportion of metabolically derived carbon, or respired carbon (Cresp), increased as the octopus grew (slope = 0.076, R2 = 0.72), suggesting an influence of somatic growth rate and body mass on δ13C values. Additionally, we identified an inverse correlation between δ18O values and environmental temperature (slope = -0.163, R2 = 0.91), which was subsequently used to develop a thermal reconstruction model. Our experiment aids in interpreting stable isotopic values in statoliths and their application as temperature and metabolic proxies in wild-caught octopus. Such proxies will increase our monitoring capabilities of these ecologically and commercially significant cephalopods and contribute to their conservation and effective management.

Responses of marine trophic levels to the combined effects of ocean acidification and warming

Marine organisms are simultaneously exposed to anthropogenic stressors associated with ocean acidification and ocean warming, with expected interactive effects. Species from different trophic levels with dissimilar characteristics and evolutionary histories are likely to respond differently. Here, we perform a meta-analysis of controlled experiments including both ocean acidification and ocean warming factors to investigate single and interactive effects of these stressors on marine species. Contrary to expectations, we find that synergistic interactions are less common (16%) than additive (40%) and antagonistic (44%) interactions overall and their proportion decreases with increasing trophic level. Predators are the most tolerant trophic level to both individual and combined effects. For interactive effects, calcifying and non-calcifying species show similar patterns. We also identify climate region-specific patterns, with interactive effects ranging from synergistic in temperate regions to compensatory in subtropical regions, to positive in tropical regions. Our findings improve understanding of how ocean warming, and acidification affect marine trophic levels and highlight the need for deeper consideration of multiple stressors in conservation efforts.

A systematic evidence map and bibliometric analysis of the behavioural impacts of pesticide exposure on zebrafish

Pesticides are indispensable in agriculture and have become ubiquitous in aquatic environments. Pesticides in natural environments can cause many negative impacts on aquatic species, ranging from mortality to sub-lethal physiological and behavioural changes. The complex sub-lethal impacts of pesticides are routinely tested on model species, with zebrafish (Danio rerio) being regularly used as a behavioural model. Although behavioural ecotoxicology research using zebrafish is increasing rapidly, we lack quantitative evidence to support which pesticides have been tested and how study designs are carried out. This shortcoming not only limits the deliberate planning for future primary studies to fill the knowledge gaps but also hinders evidence synthesis. To provide quantitative evidence of what pesticides are currently studied and what study designs are used, we combined a systematic evidence map approach and bibliometric analysis. This novel method has been coined research weaving and allows us to elicit gaps and clusters in our evidence base, whilst showing connections between authors and institutions. The methodology can be summarised in five primary steps: literature searching, screening, extraction, data analysis and bibliometric analysis. We identified four areas where research on the sub-lethal effects of pesticide exposure on zebrafish is lacking. First, some widely used pesticides, such as neonicotinoids, are understudied. Second, most studies do not report important elements of the study design, namely the sex and the life-stage of the zebrafish. Third, some behaviours, such as impacts of pesticide exposure on zebrafish cognition, are underexplored. And last, we revealed through the bibliometric analysis that most of the research is conducted in developed countries and there is limited cross country co-authorships. Upon identifying these gaps, we offer solutions for each limitation, emphasizing the importance of diverse global research output and cross-country co-authorships. Our systematic evidence map and bibliometric analysis provide valuable insights for helping to guide future research, which can be used to help support evidence-based policy decisions.

JF, Calvo DA, Carbognani M, Cardoni A, Carlyle CN, Castillo-Garcia M, Chang SX, Chieppa J, Cianciaruso MV, Cohen O, Cordeiro AL, Cusack DF, Dahlke S, Daleo P, D'Antonio CM, Dietterich LH, S. Doherty T, Dubbert M, Ebeling A, Eisenhauer N, Fischer FM, Forte TGW, Gebauer T, Gozalo B, Greenville AC, Guidoni-Martins KG, Hannusch HJ, Vatsø Haugum S, Hautier Y, Hefting M, Henry HAL, Hoss D, Ingrisch J, Iribarne O, Isbell F, Johnson Y, Jordan S, Kelly EF, Kimmel K, Kreyling J, Kröel-Dulay G, Kröpfl A, Kübert A, Kulmatiski A, Lamb EG, Larsen KS, Larson J, Lawson J, Leder CV, Linstädter A, Liu J, Liu S, Lodge AG, Longo G, Loydi A, Luan J, Curtis Lubbe F, Macfarlane C, Mackie-Haas K, Malyshev AV, Maturano-Ruiz A, Merchant T, Metcalfe DB, Mori AS, Mudongo E, Newman GS, Nielsen UN, Nimmo D, Niu Y, Nobre P, O'Connor RC, Ogaya R, Oñatibia GR, Orbán I, Osborne B, Otfinowski R, Pärtel M, Penuelas J, Peri PL, Peter G, Petraglia A, Picon-Cochard C, Pillar VD, Piñeiro-Guerra JM, Ploughe LW, Plowes RM, Portales-Reyes C, Prober SM, Pueyo Y, Reed SC, Ritchie EG, Rodríguez DA, Rogers WE, Roscher C, Sánchez AM, Santos BA, Cecilia Scarfó M, Seabloom EW, Shi B, Souza L, Stampfli A, Standish RJ, Sternberg M, Sun W, Sünnemann M, Tedder M, Thorvaldsen P, Tian D, Tielbörger K, Valdecantos A, van den Brink L, Vandvik V, Vankoughnett MR, Guri Velle L, Wang C, Wang Y, Wardle GM, Werner C, Wei C, Wiehl G, Williams JL, Wolf AA, Zeiter M, Zhang F, Zhu J, Zong N, Zuo X. Extreme drought impacts have been underestimated in grasslands and shrublands globally

Climate change is increasing the frequency and severity of short-term (~1 y) drought events-the most common duration of drought-globally. Yet the impact of this intensification of drought on ecosystem functioning remains poorly resolved. This is due in part to the widely disparate approaches ecologists have employed to study drought, variation in the severity and duration of drought studied, and differences among ecosystems in vegetation, edaphic and climatic attributes that can mediate drought impacts. To overcome these problems and better identify the factors that modulate drought responses, we used a coordinated distributed experiment to quantify the impact of short-term drought on grassland and shrubland ecosystems. With a standardized approach, we imposed ~a single year of drought at 100 sites on six continents. Here we show that loss of a foundational ecosystem function-aboveground net primary production (ANPP)-was 60% greater at sites that experienced statistically extreme drought (1-in-100-y event) vs. those sites where drought was nominal (historically more common) in magnitude (35% vs. 21%, respectively). This reduction in a key carbon cycle process with a single year of extreme drought greatly exceeds previously reported losses for grasslands and shrublands. Our global experiment also revealed high variability in drought response but that relative reductions in ANPP were greater in drier ecosystems and those with fewer plant species. Overall, our results demonstrate with unprecedented rigor that the global impacts of projected increases in drought severity have been significantly underestimated and that drier and less diverse sites are likely to be most vulnerable to extreme drought.

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

Surfactant-Mediated Effects on Hydrological and Physical Soil Properties: Data Synthesis

Soils are under the threat of a multitude of anthropogenic factors affecting the complex interplay of various physical and hydrological soil processes and properties. One such factor is the group of surface-active compounds. Surfactants have a broad range of applications and can reduce solid-liquid interfacial forces and increase wettability and dispersion of particles. Surfactant effects are context-dependent, giving rise to a wide range of reported effects on different soil processes and properties. Here, we evaluate the evidence base of surfactant research on 11 hydrological and physical soil variables. Our goal was to identify knowledge gaps and test the robustness of the proposed surfactant effects. We found that the current knowledge base is insufficient to reach strong data-backed conclusions about the effects of surfactants in soils. We identified a unique case of bias in the data as a result of conflated patterns from laboratory and field studies. We could not support the hypothesis that the surfactant charge determines soil effects for any of the tested soil variables. We believe that further experiments on surfactant-mediated effects on soil properties and processes are urgently required, paying attention, in particular, to improving experimental design and data reporting standards.

Runoff, sediment, organic carbon, and nutrient loads from a Canadian prairie micro-watershed under climate variability and land management practices

This study conducted a spatio-temporal analysis of runoff, total suspended sediment, suspended particulate carbon, nitrogen, and phosphorus loadings within the 2.06 km2 Steppler subwatershed in southern Manitoba of Canada based on 11 years of field monitoring data collected at nine stations. Results showed that the nutrient losses were very small because of the implementation of multiple BMPs in the study area. However, a high spatio-temporal variation of runoff and water quality parameters was found for the nine fields within the subwatershed. The average runoff coefficient was 0.19 at the subwatershed outlet with sediment, suspended particulate carbon, total nitrogen, and total phosphorus losses of 73.8, 6.10, 4.54, and 0.76 kg/ha respectively. Spring snowmelt runoff was about 74.5% of the annual runoff at the subwatershed outlet, while for sediment, suspended particulate carbon, total nitrogen, and total phosphorus, the proportions were 61.1%, 63.6%, 74.9%, and 81.2% respectively during the monitoring period, which suggests that BMPs designed for reducing nutrient loadings from snowmelt runoff would be more effective than BMPs designed for reducing pollutant loading from rainfall storms in the study area. Research findings from this study will benefit the enhancement of current BMPs and the development of new BMPs in the region to minimize soil and nutrient losses from agricultural fields and improve water quality in receiving water bodies.

Meta-analysis of variation in sport and exercise science: Examples of application within resistance training research

Meta-analysis has become commonplace within sport and exercise science for synthesising and summarising empirical studies. However, most research in the field focuses upon mean effects, particularly the effects of interventions to improve outcomes such as fitness or performance. It is thought that individual responses to interventions vary considerably. Hence, interest has increased in exploring precision or personalised exercise approaches. Not only is the mean often affected by interventions, but variation may also be impacted. Exploration of variation in studies such as randomised controlled trials (RCTs) can yield insight into interindividual heterogeneity in response to interventions and help determine generalisability of effects. Yet, larger samples sizes than those used for typical mean effects are required when probing variation. Thus, in a field with small samples such as sport and exercise science, exploration of variation through a meta-analytic framework is appealing. Despite the value of embracing and exploring variation alongside mean effects in sport and exercise science, it is rarely applied to research synthesis through meta-analysis. We introduce and evaluate different effect size calculations along with models for meta-analysis of variation using relatable examples from resistance training RCTs.

Understanding 'it depends' in ecology: a guide to hypothesising, visualising and interpreting statistical interactions

Ecologists routinely use statistical models to detect and explain interactions among ecological drivers, with a goal to evaluate whether an effect of interest changes in sign or magnitude in different contexts. Two fundamental properties of interactions are often overlooked during the process of hypothesising, visualising and interpreting interactions between drivers: the measurement scale - whether a response is analysed on an additive or multiplicative scale, such as a ratio or logarithmic scale; and the symmetry - whether dependencies are considered in both directions. Overlooking these properties can lead to one or more of three inferential errors: misinterpretation of (i) the detection and magnitude (Type-D error), and (ii) the sign of effect modification (Type-S error); and (iii) misidentification of the underlying processes (Type-A error). We illustrate each of these errors with a broad range of ecological questions applied to empirical and simulated data sets. We demonstrate how meta-analysis, a widely used approach that seeks explicitly to characterise context dependence, is especially prone to all three errors. Based on these insights, we propose guidelines to improve hypothesis generation, testing, visualisation and interpretation of interactions in ecology.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Advanced methods and implementations for the meta-analyses of animal models: Current practices and future recommendations

Meta-analytic techniques have been widely used to synthesize data from animal models of human diseases and conditions, but these analyses often face two statistical challenges due to complex nature of animal data (e.g., multiple effect sizes and multiple species): statistical dependency and confounding heterogeneity. These challenges can lead to unreliable and less informative evidence, which hinders the translation of findings from animal to human studies. We present a literature survey of meta-analysis using animal models (animal meta-analysis), showing that these issues are not adequately addressed in current practice. To address these challenges, we propose a meta-analytic framework based on multilevel (linear mixed-effects) models. Through conceptualization, formulations, and worked examples, we illustrate how this framework can appropriately address these issues while allowing for testing new questions. Additionally, we introduce other advanced techniques such as multivariate models, robust variance estimation, and meta-analysis of emergent effect sizes, which can deliver robust inferences and novel biological insights. We also provide a tutorial with annotated R code to demonstrate the implementation of these techniques.

High frequency root dynamics: sampling and interpretation using replicated robotic minirhizotrons

Automating dynamic fine root data collection in the field is a longstanding challenge with multiple applications for co-interpretation and synthesis for ecosystem understanding. High frequency root data are only achievable with paired automated sampling and processing. However, automatic minirhizotron (root camera) instruments are still rare and data are often not collected in natural soils or analysed at high temporal resolution. Instruments must also be affordable for replication and robust under variable natural conditions. Here, we show a system built with off-the-shelf parts which samples at sub-daily resolution. We paired this with a neural network to analyse all images collected. We performed two mesocosm studies and two field trials alongside ancillary data collection (soil CO2 efflux, temperature, and moisture content, and 'PhenoCam'-derived above-ground dynamics). We produce robust and replicated daily time series of root dynamics under all conditions. Temporal root changes were a stronger driver than absolute biomass on soil CO2 efflux in the mesocosm. Proximal sensed above-ground dynamics and below-ground dynamics from minirhizotron data were not synchronized. Root properties extracted were sensitive to soil moisture and occasionally to time of day (potentially relating to soil moisture). This may only affect high frequency imagery and should be considered in interpreting such data.

A robust and readily implementable method for the meta-analysis of response ratios with and without missing standard deviations

The log response ratio, lnRR, is the most frequently used effect size statistic for meta-analysis in ecology. However, often missing standard deviations (SDs) prevent estimation of the sampling variance of lnRR. We propose new methods to deal with missing SDs via a weighted average coefficient of variation (CV) estimated from studies in the dataset that do report SDs. Across a suite of simulated conditions, we find that using the average CV to estimate sampling variances for all observations, regardless of missingness, performs with minimal bias. Surprisingly, even with missing SDs, this simple method outperforms the conventional approach (basing each effect size on its individual study-specific CV) with complete data. This is because the conventional method ultimately yields less precise estimates of the sampling variances than using the pooled CV from multiple studies. Our approach is broadly applicable and can be implemented in all meta-analyses of lnRR, regardless of 'missingness'.

Are experiment sample sizes adequate to detect biologically important interactions between multiple stressors?

As most ecosystems are being challenged by multiple, co-occurring stressors, an important challenge is to understand and predict how stressors interact to affect biological responses. A popular approach is to design factorial experiments that measure biological responses to pairs of stressors and compare the observed response to a null model expectation. Unfortunately, we believe experiment sample sizes are inadequate to detect most non-null stressor interaction responses, greatly hindering progress. Using both real and simulated data, we show sample sizes typical of many experiments (<6) can (i) only detect very large deviations from the additive null model, implying many important non-null stressor-pair interactions are being missed, and (ii) potentially lead to mostly statistical outliers being reported. Computer code that simulates data under either additive or multiplicative null models is provided to estimate statistical power for user-defined responses and sample sizes, and we recommend this is used to aid experimental design and interpretation of results. We suspect that most experiments may require 20 or more replicates per treatment to have adequate power to detect nonadditive. However, estimates of power need to be made while considering the smallest interaction of interest, i.e., the lower limit for a biologically important interaction, which is likely to be system-specific, meaning a general guide is unavailable. We discuss ways in which the smallest interaction of interest can be chosen, and how sample sizes can be increased. Our main analyses relate to the additive null model, but we show similar problems occur for the multiplicative null model, and we encourage similar investigations into the statistical power of other null models and inference methods. Without knowledge of the detection abilities of the statistical tools at hand or the definition of the smallest meaningful interaction, we will undoubtedly continue to miss important ecosystem stressor interactions.

Are experiment sample sizes adequate to detect biologically important interactions between multiple stressors?

As most ecosystems are being challenged by multiple, co‐occurring stressors, an important challenge is to understand and predict how stressors interact to affect biological responses. A popular approach is to design factorial experiments that measure biological responses to pairs of stressors and compare the observed response to a null model expectation. Unfortunately, we believe experiment sample sizes are inadequate to detect most non‐null stressor interaction responses, greatly hindering progress. Using both real and simulated data, we show sample sizes typical of many experiments (<6) can (i) only detect very large deviations from the additive null model, implying many important non‐null stressor‐pair interactions are being missed, and (ii) potentially lead to mostly statistical outliers being reported. Computer code that simulates data under either additive or multiplicative null models is provided to estimate statistical power for user‐defined responses and sample sizes, and we recommend this is used to aid experimental design and interpretation of results. We suspect that most experiments may require 20 or more replicates per treatment to have adequate power to detect nonadditive. However, estimates of power need to be made while considering the smallest interaction of interest, i.e., the lower limit for a biologically important interaction, which is likely to be system‐specific, meaning a general guide is unavailable. We discuss ways in which the smallest interaction of interest can be chosen, and how sample sizes can be increased. Our main analyses relate to the additive null model, but we show similar problems occur for the multiplicative null model, and we encourage similar investigations into the statistical power of other null models and inference methods. Without knowledge of the detection abilities of the statistical tools at hand or the definition of the smallest meaningful interaction, we will undoubtedly continue to miss important ecosystem stressor interactions.

The Relative Benefits of Environmental Enrichment on Learning and Memory are Greater When Stressed: A Meta-analysis of Interactions in Rodents

Environmental enrichment ("EE") is expected to alleviate the negative effects of stress on cognitive performance. However, there are complexities associated with interpreting interactions that obscure determining the benefit EE may play in mitigating the negative effects of stress. To clarify these complexities, we conducted a systematic review with meta-analysis on the main and interactive effects of EE and stress on learning and memory in rodents. We show that EE and stress interact 'synergistically' where EE provides a greater relative benefit to stressed individuals compared to those reared in conventional housing. Importantly, EE can fully-compensate for the negative effects of stress where stressed individuals with EE performed equally to enriched individuals without a stress manipulation. Additionally, we show the importance of other mediating factors, including the order of treatment exposure, duration and type of stress, type of EE, and type of cognitive assays used. This study not only quantifies the interactions between EE and stress, but also provides a clear example for how to conduct and interpret meta-analysis of interactions.

Meta-analysis reveals an extreme "decline effect" in the impacts of ocean acidification on fish behavior

Ocean acidification-decreasing oceanic pH resulting from the uptake of excess atmospheric CO2-has the potential to affect marine life in the future. Among the possible consequences, a series of studies on coral reef fish suggested that the direct effects of acidification on fish behavior may be extreme and have broad ecological ramifications. Recent studies documenting a lack of effect of experimental ocean acidification on fish behavior, however, call this prediction into question. Indeed, the phenomenon of decreasing effect sizes over time is not uncommon and is typically referred to as the "decline effect." Here, we explore the consistency and robustness of scientific evidence over the past decade regarding direct effects of ocean acidification on fish behavior. Using a systematic review and meta-analysis of 91 studies empirically testing effects of ocean acidification on fish behavior, we provide quantitative evidence that the research to date on this topic is characterized by a decline effect, where large effects in initial studies have all but disappeared in subsequent studies over a decade. The decline effect in this field cannot be explained by 3 likely biological explanations, including increasing proportions of studies examining (1) cold-water species; (2) nonolfactory-associated behaviors; and (3) nonlarval life stages. Furthermore, the vast majority of studies with large effect sizes in this field tend to be characterized by low sample sizes, yet are published in high-impact journals and have a disproportionate influence on the field in terms of citations. We contend that ocean acidification has a negligible direct impact on fish behavior, and we advocate for improved approaches to minimize the potential for a decline effect in future avenues of research.

Meta-analysis reveals an extreme "decline effect" in the impacts of ocean acidification on fish behavior

Ocean acidification-decreasing oceanic pH resulting from the uptake of excess atmospheric CO2-has the potential to affect marine life in the future. Among the possible consequences, a series of studies on coral reef fish suggested that the direct effects of acidification on fish behavior may be extreme and have broad ecological ramifications. Recent studies documenting a lack of effect of experimental ocean acidification on fish behavior, however, call this prediction into question. Indeed, the phenomenon of decreasing effect sizes over time is not uncommon and is typically referred to as the "decline effect." Here, we explore the consistency and robustness of scientific evidence over the past decade regarding direct effects of ocean acidification on fish behavior. Using a systematic review and meta-analysis of 91 studies empirically testing effects of ocean acidification on fish behavior, we provide quantitative evidence that the research to date on this topic is characterized by a decline effect, where large effects in initial studies have all but disappeared in subsequent studies over a decade. The decline effect in this field cannot be explained by 3 likely biological explanations, including increasing proportions of studies examining (1) cold-water species; (2) nonolfactory-associated behaviors; and (3) nonlarval life stages. Furthermore, the vast majority of studies with large effect sizes in this field tend to be characterized by low sample sizes, yet are published in high-impact journals and have a disproportionate influence on the field in terms of citations. We contend that ocean acidification has a negligible direct impact on fish behavior, and we advocate for improved approaches to minimize the potential for a decline effect in future avenues of research.