Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

Evidence of HARKing in mouse behavioural tests of anxiety

Over the last decades, behavioural tests in animals, especially rodents, have been a standard screening method to determine the mechanisms of action and efficacy of psychopharmacological compounds. Yet, recently the reproducibility of some of these tests has been questioned. Based on a systematic review of the sensitivity of mouse behavioural tests to anxiolytic drugs, we analysed behavioural outcomes extracted from 206 studies testing the effect of diazepam in either the open-field test or the hole-board test. Surprisingly, we found that both the rationale given for using the test, whether to detect anxiolytic or sedative effects, and the predicted effect of diazepam, anxiolytic or sedative, strongly depended on the reported test results. The most likely explanation for such strong dependency is post hoc reasoning, also called hypothesizing after the results are known (HARKing). HARKing can invalidate study outcomes and hampers evidence synthesis by inflating effect sizes. It may also lead researchers into blind alleys, and waste animals, time and resources for inconclusive research.

P-hacking in meta-analyses: A formalization and new meta-analytic methods

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.

Toxicity of glyphosate to animals: A meta-analytical approach

Glyphosate (GLY)-based herbicides (GBHs) are the most commonly applied pesticide worldwide, and non-target organisms (e.g., animals) are now regularly exposed to GLY and GBHs due to the accumulation of these chemicals in many environments. Although GLY/GBH was previously considered to be non-toxic, growing evidence indicates that GLY/GBH negatively affects some animal taxa. However, there has been no systematic analysis quantifying its toxicity to animals. Therefore, we used a meta-analytical approach to determine whether there is a demonstrable effect of GLY/GBH toxicity across animals. We further addressed whether the effects of GLY/GBH vary due to (1) taxon (invertebrate vs. vertebrate), (2) habitat (aquatic vs. terrestrial), (3) type of biological response (behavior vs. physiology vs. survival), and (4) dosage or concentration of GLY/GBH. Using this approach, we also determined whether adjuvants (e.g., surfactants) in commercial formulations of GBHs increased toxicity for animals relative to exposure to GLY alone. We analyzed 1282 observations from 121 articles. We conclude that GLY is generally sub-lethally toxic for animals, particularly for animals in aquatic or marine habitats, and that toxicity did not exhibit dose-dependency. Yet, our analyses detected evidence for widespread publication bias so we encourage continued experimental investigations to better understand factors influencing GLY/GBH toxicity to animals.

Evidence for Infant-directed Speech Preference Is Consistent Across Large-scale, Multi-site Replication and Meta-analysis

There is substantial evidence that infants prefer infant-directed speech (IDS) to adult-directed speech (ADS). The strongest evidence for this claim has come from two large-scale investigations: i) a community-augmented meta-analysis of published behavioral studies and ii) a large-scale multi-lab replication study. In this paper, we aim to improve our understanding of the IDS preference and its boundary conditions by combining and comparing these two data sources across key population and design characteristics of the underlying studies. Our analyses reveal that both the meta-analysis and multi-lab replication show moderate effect sizes (d ≈ 0.35 for each estimate) and that both of these effects persist when relevant study-level moderators are added to the models (i.e., experimental methods, infant ages, and native languages). However, while the overall effect size estimates were similar, the two sources diverged in the effects of key moderators: both infant age and experimental method predicted IDS preference in the multi-lab replication study, but showed no effect in the meta-analysis. These results demonstrate that the IDS preference generalizes across a variety of experimental conditions and sampling characteristics, while simultaneously identifying key differences in the empirical picture offered by each source individually and pinpointing areas where substantial uncertainty remains about the influence of theoretically central moderators on IDS preference. Overall, our results show how meta-analyses and multi-lab replications can be used in tandem to understand the robustness and generalizability of developmental phenomena.

Sensitivity analysis for the interactive effects of internal bias and publication bias in meta-analyses

Meta-analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst-case" form of publication bias. Robust estimation methods accommodate non-normal effects, small meta-analyses, and clustered estimates. As we illustrate by re-analyzing published meta-analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).

Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values

The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to settle this question empirically. Here we propose a new way to estimate the false positive risk and apply the method to the results of (randomized) clinical trials in top medical journals. Contrary to claims that most published results are false, we find that the traditional significance criterion of α = .05 produces a false positive risk of 13%. Adjusting α to.01 lowers the false positive risk to less than 5%. However, our method does provide clear evidence of publication bias that leads to inflated effect size estimates. These results provide a solid empirical foundation for evaluations of the trustworthiness of medical research.

Is There Evidence of P-Hacking in Imaging Research?

BACKGROUND

P-hacking, the tendency to run selective analyses until they become significant, is prevalent in many scientific disciplines.

PURPOSE

This study aims to assess if p-hacking exists in imaging research.

METHODS

Protocol, data, and code available here https://osf.io/xz9ku/?view_only=a9f7c2d841684cb7a3616f567db273fa. We searched imaging journals Ovid MEDLINE from 1972 to 2021. Text mining using Python script was used to collect metadata: journal, publication year, title, abstract, and P-values from abstracts. One P-value was randomly sampled per abstract. We assessed for evidence of p-hacking using a p-curve, by evaluating for a concentration of P-values just below .05. We conducted a one-tailed binomial test (α = .05 level of significance) to assess whether there were more P-values falling in the upper range (e.g., .045 < P < .05) than in the lower range (e.g., .04 < P < .045). To assess variation in results introduced by our random sampling of a single P-value per abstract, we repeated the random sampling process 1000 times and pooled results across the samples. Analysis was done (divided into 10-year periods) to determine if p-hacking practices evolved over time.

RESULTS

Our search of 136 journals identified 967,981 abstracts. Text mining identified 293,687 P-values, and a total of 4105 randomly sampled P-values were included in the p-hacking analysis. The number of journals and abstracts that were included in the analysis as a fraction and percentage of the total number was, respectively, 108/136 (80%) and 4105/967,981 (.4%). P-values did not concentrate just under .05; in fact, there were more P-values falling in the lower range (e.g., .04 < P < .045) than falling just below .05 (e.g., .045 < P < .05), indicating lack of evidence for p-hacking. Time trend analysis did not identify p-hacking in any of the five 10-year periods.

CONCLUSION

We did not identify evidence of p-hacking in abstracts published in over 100 imaging journals since 1972. These analyses cannot detect all forms of p-hacking, and other forms of bias may exist in imaging research such as publication bias and selective outcome reporting.

The effect of ambient ozone exposure on three types of diabetes: a meta-analysis

BACKGROUND

Ozone as an air pollutant is gradually becoming a threat to people's health. However, the effect of ozone exposure on risk of developing diabetes, a fast-growing global metabolic disease, remains controversial.

OBJECTIVE

To evaluate the impact of ambient ozone exposure on the incidence rate of type 1, type 2 and gestational diabetes mellitus.

METHOD

We systematically searched PubMed, Web of Science, and Cochrane Library databases before July 9, 2022, to determine relevant literature. Data were extracted after quality evaluation according to the Newcastle Ottawa Scale (NOS) and the agency for healthcare research and quality (AHRQ) standards, and a meta-analysis was used to evaluate the correlation between ozone exposure and type 1 diabetes mellitus (T1D), type 2 diabetes mellitus (T2D), and gestational diabetes mellitus (GDM). The heterogeneity test, sensitivity analysis, and publication bias were performed using Stata 16.0.

RESULTS

Our search identified 667 studies from three databases, 19 of which were included in our analysis after removing duplicate and ineligible studies. Among the remaining studies, three were on T1D, five were on T2D, and eleven were on GDM. The result showed that ozone exposure was positively correlated with T2D [effect size (ES) = 1.06, 95% CI: 1.02, 1.11] and GDM [pooled odds ratio (OR) = 1.01, 95% CI: 1.00, 1.03]. Subgroup analysis demonstrated that ozone exposure in the first trimester of pregnancy might raise the risk of GDM. However, no significant association was observed between ozone exposure and T1D.

CONCLUSION

Long-term exposure to ozone may increase the risk of T2D, and daily ozone exposure during pregnancy was a hazard factor for developing GDM. Decreasing ambient ozone pollution may reduce the burden of both diseases.

Interindividual Variability in Mental Fatigue-Related Impairments in Endurance Performance: A Systematic Review and Multiple Meta-regression

BACKGROUND

The negative effect of mental fatigue (MF) on physical performance has recently been questioned. One reason behind this could lie in the interindividual differences in MF-susceptibility and the individual features influencing them. However, the range of individual differences in mental fatigue-susceptibility is not known, and there is no clear consensus on which individual features could be responsible for these differences.

OBJECTIVE

To give an overview of interindividual differences in the effects of MF on whole-body endurance performance, and individual features influencing this effect.

METHODS

The review was registered on the PROSPERO database (CRD42022293242). PubMed, Web of Science, SPORTDiscus and PsycINFO were searched until the 16th of June 2022 for studies detailing the effect of MF on dynamic maximal whole-body endurance performance. Studies needed to include healthy participants, describe at least one individual feature in participant characteristics, and apply at least one manipulation check. The Cochrane crossover risk of bias tool was used to assess risk of bias. The meta-analysis and regression were conducted in R.

RESULTS

Twenty-eight studies were included, with 23 added to the meta-analysis. Overall risk of bias of the included studies was high, with only three presenting an unclear or low rating. The meta-analysis shows the effect of MF on endurance performance was on average slightly negative (g = - 0.32, [95% CI - 0.46; - 0.18], p < 0.001). The multiple meta-regression showed no significant influences of the included features (i.e. age, sex, body mass index and physical fitness level) on MF-susceptibility.

CONCLUSIONS

The present review confirmed the negative impact of MF on endurance performance. However, no individual features influencing MF-susceptibility were identified. This can partially be explained by the multiple methodological limitations such as underreporting of participant characteristics, lack of standardization across studies, and the restricted inclusion of potentially relevant variables. Future research should include a rigorous description of multiple different individual features (e.g., performance level, diet, etc.) to further elucidate MF mechanisms.

Publication bias, time-lag bias, and place-of-publication bias in social intervention research: An exploratory study of 527 Swedish articles published between 1990-2019

Publication and related biases constitute serious threats to the validity of research synthesis. If research syntheses are based on a biased selection of the available research, there is an increased risk of producing misleading results. The purpose fo this study is to explore the extent of positive outcome bias, time-lag bias, and place-of-publication bias in published research on the effects of psychological, social, and behavioral interventions. The results are based on 527 Swedish outcome trials published in peer-reviewed journals between 1990 and 2019. We found no difference in the number of studies reporting significant compared to non-significant findings or in the number of studies reporting strong effect sizes in the published literature. We found no evidence of time-lag bias or place-of-publication bias in our results. The average reported effect size remained constant over time as did the proportion of studies reporting significant effects.

Open science, the replication crisis, and environmental public health

Concerns about a crisis of mass irreplicability across scientific fields ("the replication crisis") have stimulated a movement for open science, encouraging or even requiring researchers to publish their raw data and analysis code. Recently, a rule at the US Environmental Protection Agency (US EPA) would have imposed a strong open data requirement. The rule prompted significant public discussion about whether open science practices are appropriate for fields of environmental public health. The aims of this paper are to assess (1) whether the replication crisis extends to fields of environmental public health; and (2) in general whether open science requirements can address the replication crisis. There is little empirical evidence for or against mass irreplicability in environmental public health specifically. Without such evidence, strong claims about whether the replication crisis extends to environmental public health - or not - seem premature. By distinguishing three concepts - reproducibility, replicability, and robustness - it is clear that open data initiatives can promote reproducibility and robustness but do little to promote replicability. I conclude by reviewing some of the other benefits of open science, and offer some suggestions for funding streams to mitigate the costs of adoption of open science practices in environmental public health.

A Systematic Literature Review and Meta-Analysis of Studies on Online Fake News Detection

The ubiquitous access and exponential growth of information available on social media networks have facilitated the spread of fake news, complicating the task of distinguishing between this and real news. Fake news is a significant social barrier that has a profoundly negative impact on society. Despite the large number of studies on fake news detection, they have not yet been combined to offer coherent insight on trends and advancements in this domain. Hence, the primary objective of this study was to fill this knowledge gap. The method for selecting the pertinent articles for extraction was created using the preferred reporting items for systematic reviews and meta-analyses (PRISMA). This study reviewed deep learning, machine learning, and ensemble-based fake news detection methods by a meta-analysis of 125 studies to aggregate their results quantitatively. The meta-analysis primarily focused on statistics and the quantitative analysis of data from numerous separate primary investigations to identify overall trends. The results of the meta-analysis were reported by the spatial distribution, the approaches adopted, the sample size, and the performance of methods in terms of accuracy. According to the statistics of between-study variance high heterogeneity was found with τ2 = 3.441; the ratio of true heterogeneity to total observed variation was I2 = 75.27% with the heterogeneity chi-square (Q) = 501.34, the degree of freedom = 124, and p ≤ 0.001. A p-value of 0.912 from the Egger statistical test confirmed the absence of a publication bias. The findings of the meta-analysis demonstrated satisfaction with the effectiveness of the recommended approaches from the primary studies on fake news detection that were included. Furthermore, the findings can inform researchers about various approaches they can use to detect online fake news.

The benefits of simultaneous tDCS and working memory training on transfer outcomes: A systematic review and meta-analysis

BACKGROUND

Transcranial direct current stimulation (tDCS) has shown potential as an effective aid to facilitate learning. A popular application of this technology has been in combination with working memory training (WMT) in order to enhance transfer effects to other cognitive measures after training.

OBJECTIVE

This meta-analytic review aims to synthesize the existing literature on tDCS-enhanced WMT to quantify the extent to which tDCS can improve performance on transfer tasks after training. Furthermore, we were interested to evaluate the moderating effects of assessment time point (immediate post-test vs. follow-up) and transfer distance, i.e., the degree of similarity between transfer and training tasks.

METHODS

Using robust variance estimation, we performed a systematic meta-analysis of all studies to date that compared WMT with tDCS to WMT with sham in healthy adults. All procedures conformed to PRISMA guidelines.

RESULTS

Across 265 transfer measures in 18 studies, we found a small positive net effect of tDCS on improving overall performance on transfer measures after WMT. These effects were sustained at follow-up, which ranged from 1 week to one year after training, with a median of 1 month. Additionally, although there were no significant differences as a function of transfer distance, effects were most pronounced for non-trained working memory tasks.

CONCLUSIONS

This review provides evidence that tDCS can be effective in promoting learning over and above WMT alone, and can durably improve performance on trained and untrained measures for weeks to months after the initial training and stimulation period. In particular, boosting performance on dissimilar working memory tasks may present the most promising target for tDCS-augmented WMT.

How much does the typical ecological meta-analysis overestimate the true mean effect size?

Many primary research studies in ecology are underpowered, providing very imprecise estimates of effect size. Meta-analyses partially mitigate this imprecision by combining data from different studies. But meta-analytic estimates of mean effect size may still remain imprecise, particularly if the meta-analysis includes a small number of studies. Imprecise, large-magnitude estimates of mean effect size from small meta-analyses likely would shrink if additional studies were conducted (regression towards the mean). Here, I propose a way to estimate and correct this regression to the mean, using meta-meta-analysis (meta-analysis of meta-analyses). Hierarchical random effects meta-meta-analysis shrinks estimated mean effect sizes from different meta-analyses towards the grand mean, bringing those estimated means closer on average to their unknown true values. The intuition is that, if a meta-analysis reports a mean effect size much larger in magnitude than that reported by other meta-analyses, that large mean effect size likely is an overestimate. This intuition holds even if different meta-analyses of different topics have different true mean effect sizes. Drawing on a compilation of data from hundreds of ecological meta-analyses, I find that the typical (median) ecological meta-analysis overestimates the absolute magnitude of the true mean effect size by ~10%. Some small ecological meta-analyses overestimate the magnitude of the true mean effect size by >50%. Meta-meta-analysis is a promising tool for improving the accuracy of meta-analytic estimates of mean effect size, particularly estimates based on just a few studies.

Local Tramadol Improves the Anesthetic Success in Patients with Symptomatic Irreversible Pulpitis: A Meta-Analysis

Symptomatic irreversible pulpitis is a painful clinical condition with a broad inflammatory component. Dental anesthesia in these patients is affected by the inflammatory process, reporting a high incidence of anesthesia failure. The aim of this systematic review and meta-analytical evaluation was to determine the effect of pre-treatment with tramadol in patients with symptomatic irreversible pulpitis, as well as for pain control and adverse effects. This study was registered in PROSPERO (ID: CRD42021279262). PubMed was consulted to identify clinical investigations comparing tramadol and placebo/local anesthetics in patients with symptomatic irreversible pulpitis. Data about the anesthesia, pain control, and adverse effects were extracted. Both the anesthetic success index and the adverse effects of local tramadol and placebo were compared with the Mantel−Haenszel test and odds ratio. Data analysis showed that the local administration of tramadol increased the anesthetic success rate when compared to placebo in patients with symptomatic irreversible pulpitis (n = 228; I2 = 0; OR = 2.2; 95% CIs: 1.30 to 3.79; p < 0.004). However, local administration of tramadol increased the risk of adverse effects when compared to placebo/local anesthetics (n = 288; I2 = 0; OR = 7.72; 95% CIs: 1.37 to 43.46; p < 0.02). In conclusion, this study shows that the local administration of tramadol increases the anesthetic success index when compared to placebo in patients with symptomatic irreversible pulpitis.

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

In meta-analyses, it is critical to assess the extent to which publication bias might have compromised the results. Classical methods based on the funnel plot, including Egger's test and Trim-and-Fill, have become the de facto default methods to do so, with a large majority of recent meta-analyses in top medical journals (85%) assessing for publication bias exclusively using these methods. However, these classical funnel plot methods have important limitations when used as the sole means of assessing publication bias: they essentially assume that the publication process favors large point estimates for small studies and does not affect the largest studies, and they can perform poorly when effects are heterogeneous. In light of these limitations, we recommend that meta-analyses routinely apply other publication bias methods in addition to or instead of classical funnel plot methods. To this end, we describe how to use and interpret selection models. These methods make the often more realistic assumption that publication bias favors "statistically significant" results, and the methods also directly accommodate effect heterogeneity. Selection models have been established for decades in the statistics literature and are supported by user-friendly software, yet remain rarely reported in many disciplines. We use a previously published meta-analysis to demonstrate that selection models can yield insights that extend beyond those provided by funnel plot methods, suggesting the importance of establishing more comprehensive reporting practices for publication bias assessment.

Identifying a brain network for musical rhythm: A functional neuroimaging meta-analysis and systematic review

We conducted a systematic review and meta-analysis of 30 functional magnetic resonance imaging studies investigating processing of musical rhythms in neurotypical adults. First, we identified a general network for musical rhythm, encompassing all relevant sensory and motor processes (Beat-based, rest baseline, 12 contrasts) which revealed a large network involving auditory and motor regions. This network included the bilateral superior temporal cortices, supplementary motor area (SMA), putamen, and cerebellum. Second, we identified more precise loci for beat-based musical rhythms (Beat-based, audio-motor control, 8 contrasts) in the bilateral putamen. Third, we identified regions modulated by beat based rhythmic complexity (Complexity, 16 contrasts) which included the bilateral SMA-proper/pre-SMA, cerebellum, inferior parietal regions, and right temporal areas. This meta-analysis suggests that musical rhythm is largely represented in a bilateral cortico-subcortical network. Our findings align with existing theoretical frameworks about auditory-motor coupling to a musical beat and provide a foundation for studying how the neural bases of musical rhythm may overlap with other cognitive domains.

Ten easy steps to conducting transparent, reproducible meta-analyses for infant researchers

Meta-analyses provide researchers with an overview of the body of evidence in a topic, with quantified estimates of effect sizes and the role of moderators, and weighting studies according to their precision. We provide a guide for conducting a transparent and reproducible meta-analysis in the field of developmental psychology within the framework of the MetaLab platform, in 10 steps: (1) Choose a topic for your meta-analysis, (2) Formulate your research question and specify inclusion criteria, (3) Preregister and document all stages of your meta-analysis, (4) Conduct the literature search, (5) Collect and screen records, (6) Extract data from eligible studies, (7) Read the data into analysis software and compute effect sizes, (8) Visualize your data, (9) Create meta-analytic models to assess the strength of the effect and investigate possible moderators, (10) Write up and promote your meta-analysis. Meta-analyses can inform future studies, through power calculations, by identifying robust methods and exposing research gaps. By adding a new meta-analysis to MetaLab, datasets across multiple topics of developmental psychology can be synthesized, and the dataset can be maintained as a living, community-augmented meta-analysis to which researchers add new data, allowing for a cumulative approach to evidence synthesis.

The puzzling relationship between multi-laboratory replications and meta-analyses of the published literature

What is the best way to estimate the size of important effects? Should we aggregate across disparate findings using statistical meta-analysis, or instead run large, multi-laboratory replications (MLR)? A recent paper by Kvarven, Strømland and Johannesson (Kvarven et al. 2020 Nat. Hum. Behav. 4, 423-434. (doi:10.1038/s41562-019-0787-z)) compared effect size estimates derived from these two different methods for 15 different psychological phenomena. The authors reported that, for the same phenomenon, the meta-analytic estimate tended to be about three times larger than the MLR estimate. These results are a specific example of a broader question: What is the relationship between meta-analysis and MLR estimates? Kvarven et al. suggested that their results undermine the value of meta-analysis. By contrast, we argue that both meta-analysis and MLR are informative, and that the discrepancy between the two estimates that they observed is in fact still largely unexplained. Informed by re-analyses of Kvarven et al.'s data and by other empirical evidence, we discuss possible sources of this discrepancy and argue that understanding the relationship between estimates obtained from these two methods is an important puzzle for future meta-scientific research.

Methods to Address Confounding and Other Biases in Meta-Analyses: Review and Recommendations

Meta-analyses contribute critically to cumulative science, but they can produce misleading conclusions if their constituent primary studies are biased, for example by unmeasured confounding in nonrandomized studies. We provide practical guidance on how meta-analysts can address confounding and other biases that affect studies' internal validity, focusing primarily on sensitivity analyses that help quantify how biased the meta-analysis estimates might be. We review a number of sensitivity analysis methods to do so, especially recent developments that are straightforward to implement and interpret and that use somewhat less stringent statistical assumptions than do earlier methods. We give recommendations for how these newer methods could be applied in practice and illustrate using a previously published meta-analysis. Sensitivity analyses can provide informative quantitative summaries of evidence strength, and we suggest reporting them routinely in meta-analyses of potentially biased studies. This recommendation in no way diminishes the importance of defining study eligibility criteria that reduce bias and of characterizing studies' risks of bias qualitatively. Expected final online publication date for the Annual Review of Public Health, Volume 43 is April 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Neglect of publication bias compromises meta-analyses of educational research

Because negative findings have less chance of getting published, available studies tend to be a biased sample. This leads to an inflation of effect size estimates to an unknown degree. To see how meta-analyses in education account for publication bias, we surveyed all meta-analyses published in the last five years in the Review of Educational Research and Educational Research Review. The results show that meta-analyses usually neglect publication bias adjustment. In the minority of meta-analyses adjusting for bias, mostly non-principled adjustment methods were used, and only rarely were the conclusions based on corrected estimates, rendering the adjustment inconsequential. It is argued that appropriate state-of-the-art adjustment (e.g., selection models) should be attempted by default, yet one needs to take into account the uncertainty inherent in any meta-analytic inference under bias. We conclude by providing practical recommendations on dealing with publication bias.