The Perils of Misinterpreting and Misusing "Publication Bias" in Meta-analyses: An Education Review on Funnel Plot-Based Methods

Publication bias refers to a systematic deviation from the truth in the results of a meta-analysis due to the higher likelihood for published studies to be included in meta-analyses than unpublished studies. Publication bias can lead to misleading recommendations for decision and policy making. In this education review, we introduce, explain, and provide solutions to the pervasive misuses and misinterpretations of publication bias that afflict evidence syntheses in sport and exercise medicine, with a focus on the commonly used funnel-plot based methods. Publication bias is more routinely assessed by visually inspecting funnel plot asymmetry, although it has been consistently deemed unreliable, leading to the development of statistical tests to assess publication bias. However, most statistical tests of publication bias (i) cannot rule out alternative explanations for funnel plot asymmetry (e.g., between-study heterogeneity, choice of metric, chance) and (ii) are grossly underpowered, even when using an arbitrary minimum threshold of ten or more studies. We performed a cross-sectional meta-research investigation of how publication bias was assessed in systematic reviews with meta-analyses published in the top two sport and exercise medicine journals throughout 2021. This analysis highlights that publication bias is frequently misused and misinterpreted, even in top tier journals. Because of conceptual and methodological problems when assessing and interpreting publication bias, preventive strategies (e.g., pre-registration, registered reports, disclosing protocol deviations, and reporting all study findings regardless of direction or magnitude) offer the best and most efficient solution to mitigate the misuse and misinterpretation of publication bias. Because true publication bias is very difficult to determine, we recommend that future publications use the term "risk of publication bias".

Lifestyle, WCRF/AICR Recommendations, and Esophageal Adenocarcinoma Risk: A Systematic Review of the Literature

One of the most notable changes in the epidemiology of esophageal cancer (EC) is the rising incidence and prevalence of esophageal adenocarcinoma (EAC) in developed countries. The aim of this systematic review was to collect and summarize all the available evidence regarding lifestyle, diet, and EAC risk. We searched the PubMed and Scopus databases in January 2021 for studies providing information about lifestyle, diet, WCRF/AICR recommendations, and EAC risk; published in English; without a time filter. The Newcastle-Ottawa Scale was used to assess risk of bias. The results are stratified by risk factor. A total of 106 publications were included. Half of the case-control studies were judged as high quality, whilst practically all cohort studies were judged as high quality. Body mass index and waist circumference were associated with increased EAC risk. Physical activity did not appear to have a significant direct role in EAC risk. A diet rich in fruit, vegetables, and whole grains appeared to be more protective than a Western diet. Alcohol does not seem to be related to EAC, whereas smokers, particularly heavy smokers, have an increased risk of EAC. Prevention remains the best option to avert EAC. Comprehensible and easy to follow recommendations should be provided to all subjects. Protocol ID number: CRD-42021228762, no funds received.

Published intimate partner violence studies often differ from their trial registration records

INTRODUCTION

Registering study protocols in a trial registry is important for methodologic transparency and reducing selective reporting bias. The objective of this investigation was to determine whether published studies of intimate partner violence (IPV) that had been registered matched the registration record on key study design elements.

METHODS

We systematically searched three trial registries to identify registered IPV studies and the published literature for the associated publication. Two authors independently determined for each study whether key study elements in the registry matched those in the published paper.

RESULTS

We included 66 studies published between 2006 and 2017. Nearly half (29/66, 44%) were registered after study completion. Many (26/66, 39%) had discrepancies regarding the primary outcome, and nearly two-thirds (42/66, 64%) had discrepancies in secondary outcomes. Discrepancies in study design were less frequent (13/66, 20%). However, large changes in sample size (26/66, 39%) and discrepancies in funding source (28/66, 42%) were frequently observed.

CONCLUSIONS

Trial registries are important tools for research transparency and identifying and preventing outcome switching and selective outcome reporting bias. Published IPV studies often differ from their records in trial registries. Researchers should pay close attention to the accuracy of trial registry records.

Surveillance of arteriovenous hemodialysis access: a systematic review and meta-analysis

OBJECTIVES

Hemodialysis centers regularly survey arteriovenous (AV) accesses for signs of dysfunction. In this review, we synthesize the available evidence to determine to what extent proactive vascular access monitoring affects the incidence of AV access thrombosis and abandonment compared with clinical monitoring.

METHODS

We searched electronic databases (MEDLINE, EMBASE, Cochrane CENTRAL, Web of Science, and SCOPUS) and sought references from experts, bibliographies of included trials, and articles that cited included studies. Two reviewers independently assessed trial quality and extracted data. We used random effects meta-analysis to estimate the pooled relative risk (RR) and 95% confidence interval (CI) across studies and conducted subgroup analyses to explain heterogeneity. The I(2) statistic was used to assess heterogeneity of treatment effect among trials.

RESULTS

Nine studies (1363 patients) compared a strategy of surveillance vs clinical monitoring. A vascular intervention to maintain or restore patency was provided to both groups if needed. Surveillance followed by intervention led to a nonsignificant reduction of the risk of access thrombosis (RR, 0.82; 95% CI, 0.58-1.16; I(2) = 37%) and access abandonment (RR, 0.80; 95% CI, 0.51-1.25; I(2) = 60%). Three studies (207 patients) compared the effect of vascular interventions vs observation in patients with abnormal surveillance result. Vascular interventions after an abnormal AV access surveillance led to a significant reduction of the risk of access thrombosis (RR, 0.53; 95% CI, 0.36-0.76) and a nonsignificant reduction of the risk of access abandonment (RR, 0.76; 95% CI, 0.43-1.37).

CONCLUSION

Very low quality evidence yielding imprecise results suggests a potentially beneficial effect of AV access surveillance followed by interventions to restore patency. This inference, however, is weak and will require randomized trials of AV access surveillance vs clinical monitoring for rejection or confirmation.

A case for clarity, consistency, and helpfulness: state-of-the-art clinical practice guidelines in endocrinology using the grading of recommendations, assessment, development, and evaluation system

CONTEXT

The Endocrine Society, and a growing number of other organizations, have adopted the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) system to develop clinical practice guidelines and grade the strength of recommendations and the quality of the evidence. Despite the use of GRADE in several of The Endocrine Society's clinical practice guidelines, endocrinologists have not had access to a context-specific discussion of this system and its merits.

EVIDENCE ACQUISITION

The authors are involved in the development of the GRADE standard and its application to The Endocrine Society clinical practice guidelines. Examples were extracted from these guidelines to illustrate how this grading system enhances the quality of practice guidelines.

EVIDENCE SYNTHESIS

We summarized and described the components of the GRADE system, and discussed the features of GRADE that help bring clarity and consistency to guideline documents, making them more helpful to practicing clinicians and their patients with endocrine disorders.

CONCLUSIONS

GRADE describes the quality of the evidence using four levels: very low, low, moderate, and high quality. Recommendations can be either strong ("we recommend") or weak ("we suggest"), and this strength reflects the confidence that guideline panel members have that patients who receive recommended care will be better off. The separation of the quality of the evidence from the strength of the recommendation recognizes the role that values and preferences, as well as clinical and social circumstances, play in formulating practice recommendations.