Assessment of temporal instability in the applied ecology and conservation evidence base

Outcomes of meta-analyses are increasingly used to inform evidence-based decision making in various research fields. However, a number of recent studies have reported rapid temporal changes in magnitude and significance of the reported effects which could make policy-relevant recommendations from meta-analyses to quickly go out of date. We assessed the extent and patterns of temporal trends in magnitude and statistical significance of the cumulative effects in meta-analyses in applied ecology and conservation published between 2004 and 2018. Of the 121 meta-analyses analysed, 93% showed a temporal trend in cumulative effect magnitude or significance with 27% of the datasets exhibiting temporal trends in both. The most common trend was the early study effect when at least one of the first 5 years effect size estimates exhibited more than 50% magnitude difference to the subsequent estimate. The observed temporal trends persisted in majority of datasets once moderators were accounted for. Only 5 datasets showed significant changes in sample size over time which could potentially explain the observed temporal change in the cumulative effects. Year of publication of meta-analysis had no significant effect on presence of temporal trends in cumulative effects. Our results show that temporal changes in magnitude and statistical significance in applied ecology are widespread and represent a serious potential threat to use of meta-analyses for decision-making in conservation and environmental management. We recommend use of cumulative meta-analyses and call for more studies exploring the causes of the temporal effects.

Decline effects are rare in ecology

The scientific evidence base on any given topic changes over time as more studies are published. Currently, there is widespread concern about non-random, directional changes over time in the scientific evidence base associated with many topics. In particular, if studies finding large effects (e.g., large differences between treatment and control means) tend to get published quickly, while small effects tend to get published slowly, the net result will be a decrease over time in the estimated magnitude of the mean effect size, known as a "decline effect". If decline effects are common, then the published scientific literature will provide a biased and misleading guide to management decisions, and to the allocation of future research effort. We compiled data from 466 meta-analyses in ecology to look for evidence of decline effects. We found that decline effects are rare. Only ~5% of ecological meta-analyses truly exhibit a directional change in mean effect size over time arising for some reason other than random chance, usually but not always in the direction of decline. Most apparent directional changes in mean effect size over time are attributable to regression to the mean, consistent with primary studies being published in random order with respect to the effect sizes they report. Our results are good news: decline effects are the exception to the rule in ecology. Identifying and rectifying rare cases of true decline effects remains an important task, but ecologists should not overgeneralize from anecdotal reports of decline effects.

Does Resilience Help in Reducing Burnout Symptoms Among Chinese Students? A Meta-Analysis

As society has evolved, student burnout has become a common problem in schools around the world, including in China. Therefore, the purpose of the current study is to explore whether resilience is related to student burnout in China and to examine the changing trend of resilience and student burnout. Moreover, we will assess gender differences and possible biases, including publication biases, small-study biases, gray literature biases, and decline effects. This meta-analysis included 34 studies, with a total of 81 effect sizes and a total sample size of 22,474. We found that resilience was negatively correlated with student burnout in the Chinese context. We also found evidence of gray literature bias in student burnout, which needs to be verified by subsequent studies. However, we found that there were decline effects in resilience, possibly because, as culture evolves, people become more focused on themselves; thus, their collective behaviors decline, leading to a decrease in their ability to adapt to the collective and the environment. We also found similar decline effects at the individual level; that is, resilience might decrease with individual age stages (from the primary to college stage), which might be related to the use of immature defense mechanisms against stress by students.

Cumulative meta-analysis identifies declining but negative impacts of invasive species on richness after 20 yr

A principal impact of invasive species is that they reduce local species richness. However, it is unknown whether the magnitude of the richness decrease has been consistent over the past two decades of published research. We used cumulative meta-analysis to synthesize evidence from 240 articles evaluating whether this cumulative evidence base generally supports, or refutes, the association between invasive species presence and richness declines. First, we determined whether evidence accumulation lowered the mean effect size of invasive species on local native richness through time; termed the "decline effect." Then, as mean effect sizes changed over time, we identified when accumulated evidence reached sufficiency, indicating that the mean effect direction (positive or negative) was unlikely to be reversed by unpublished research. We also assessed whether the mean effect size reached a threshold of stability over publication years. To date, no research has tested mechanisms of the decline effect, and here we determine whether publication bias, sample size, time since invasion, or invader trophic position are driving a decline effect in the published evidence base. We found a clear decline in the cumulative mean effect of invasive species on local native species richness as published evidence accumulated between 1999 and 2016. Despite this decline, an average negative association was stable and sufficiently robust to unpublished studies by 2007, showing a 21% mean richness decrease by 2016. Contrary to our expectation, the decline effect manifested consistently regardless of invasive species trophic position, time since invasion, or journal rank. Within taxonomic subgroups, trees, insects, and herbaceous plants exhibit a decline effect, yet still show sufficient and stable negative impacts on richness. However, many other taxonomic subgroups (e.g., crustaceans, fish, mammals) lack evidence for average negative impacts on richness, or have not met sufficiency or stability thresholds.

Effect Declines Are Systematic, Strong, and Ubiquitous: A Meta-Meta-Analysis of the Decline Effect in Intelligence Research

Empirical sciences in general and psychological science in particular are plagued by replicability problems and biased published effect sizes. Although dissemination bias-related phenomena such as publication bias, time-lag bias, or visibility bias are well-known and have been intensively studied, another variant of effect distorting mechanisms, so-called decline effects, have not. Conceptually, decline effects are rooted in low initial (exploratory) study power due to strategic researcher behavior and can be expected to yield overproportional effect declines. Although decline effects have been documented in individual meta-analytic investigations, systematic evidence for decline effects in the psychological literature remains to date unavailable. Therefore, we present in this meta-meta-analysis a systematic investigation of the decline effect in intelligence research. In all, data from 22 meta-analyses comprising 36 meta-analytical and 1,391 primary effect sizes (N = 697,000+) that have been published in the journal Intelligence were included in our analyses. Two different analytic approaches showed consistent evidence for a higher prevalence of cross-temporal effect declines compared to effect increases, yielding a ratio of about 2:1. Moreover, effect declines were considerably stronger when referenced to the initial primary study within a meta-analysis, yielding about twice the magnitude of effect increases. Effect misestimations were more substantial when initial studies had smaller sample sizes and reported larger effects, thus indicating suboptimal initial study power as the main driver of effect misestimations in initial studies. Post hoc study power comparisons of initial versus subsequent studies were consistent with this interpretation, showing substantially lower initial study power of declining, than of increasing effects. Our findings add another facet to the ever accumulating evidence about non-trivial effect misestimations in the scientific literature. We therefore stress the necessity for more rigorous protocols when it comes to designing and conducting primary research as well as reporting findings in exploratory and replication studies. Increasing transparency in scientific processes such as data sharing, (exploratory) study preregistration, but also self- (or independent) replication preceding the publication of exploratory findings may be suitable approaches to strengthen the credibility of empirical research in general and psychological science in particular.

Changes in Dosing and Dose Timing of D-Cycloserine Explain Its Apparent Declining Efficacy for Augmenting Exposure Therapy for Anxiety-related Disorders: An Individual Participant-data Meta-analysis

The apparent efficacy of d-cycloserine (DCS) for enhancing exposure treatment for anxiety disorders appears to have declined over the past 14 years. We examined whether variations in how DCS has been administered can account for this "declining effect". We also investigated the association between DCS administration characteristics and treatment outcome to find optimal dosing parameters. We conducted a secondary analysis of individual participant data obtained from 1047 participants in 21 studies testing the efficacy of DCS-augmented exposure treatments. Different outcome measures in different studies were harmonized to a 0-100 scale. Intent-to-treat analyses showed that, in participants randomized to DCS augmentation (n = 523), fewer DCS doses, later timing of DCS dose, and lower baseline severity appear to account for this decline effect. More DCS doses were related to better outcomes, but this advantage leveled-off at nine doses. Administering DCS more than 60 minutes before exposures was also related to better outcomes. These predictors were not significant in the placebo arm (n = 521). Results suggested that optimal DCS administration could increase pre-to-follow-up DCS effect size by 50%. In conclusion, the apparent declining effectiveness of DCS over time may be accounted for by how it has been administered. Optimal DCS administration may substantially improve outcomes. Registration: The analysis plan for this manuscript was registered on Open Science Framework (https://osf.io/c39p8/).

Turning the Lens of Science on Itself: Verbal Overshadowing, Replication, and Metascience

This issue of Perspectives on Psychological Science reports an unprecedented replication effort entailing numerous independent laboratories conducting two versions of the verbal overshadowing paradigm (Schooler & Engstler-Schooler, 1990) using different timing intervals. The results (Alogna et al., 2014, this issue) provide unequivocal support for the existence of verbal overshadowing--the finding that describing a previously seen face can impair its subsequent recognition--while simultaneously revealing a number of factors that may have contributed to challenges in replicating verbal overshadowing in the past. In this commentary, I review my participation in this process and consider the implications of the results of this replication effort for verbal overshadowing, the decline effect, and the general goal of metascience: turning the lens of science onto itself.