Do simple screening statistical tools help to detect reporting bias?

The Perils of Misinterpreting and Misusing "Publication Bias" in Meta-analyses: An Education Review on Funnel Plot-Based Methods

Publication bias refers to a systematic deviation from the truth in the results of a meta-analysis due to the higher likelihood for published studies to be included in meta-analyses than unpublished studies. Publication bias can lead to misleading recommendations for decision and policy making. In this education review, we introduce, explain, and provide solutions to the pervasive misuses and misinterpretations of publication bias that afflict evidence syntheses in sport and exercise medicine, with a focus on the commonly used funnel-plot based methods. Publication bias is more routinely assessed by visually inspecting funnel plot asymmetry, although it has been consistently deemed unreliable, leading to the development of statistical tests to assess publication bias. However, most statistical tests of publication bias (i) cannot rule out alternative explanations for funnel plot asymmetry (e.g., between-study heterogeneity, choice of metric, chance) and (ii) are grossly underpowered, even when using an arbitrary minimum threshold of ten or more studies. We performed a cross-sectional meta-research investigation of how publication bias was assessed in systematic reviews with meta-analyses published in the top two sport and exercise medicine journals throughout 2021. This analysis highlights that publication bias is frequently misused and misinterpreted, even in top tier journals. Because of conceptual and methodological problems when assessing and interpreting publication bias, preventive strategies (e.g., pre-registration, registered reports, disclosing protocol deviations, and reporting all study findings regardless of direction or magnitude) offer the best and most efficient solution to mitigate the misuse and misinterpretation of publication bias. Because true publication bias is very difficult to determine, we recommend that future publications use the term "risk of publication bias".

COVID-19 in people with neurofibromatosis 1, neurofibromatosis 2, or schwannomatosis

PURPOSE

People with pre-existing conditions may be more susceptible to severe COVID-19 when infected by SARS-CoV-2. The relative risk and severity of SARS-CoV-2 infection in people with rare diseases such as neurofibromatosis type 1 (NF1), neurofibromatosis type 2 (NF2), or schwannomatosis (SWN) is unknown.

METHODS

We investigated the proportions of people with NF1, NF2, or SWN in the National COVID Cohort Collaborative (N3C) electronic health record data set who had a positive test result for SARS-CoV-2 or COVID-19.

RESULTS

The cohort sizes in N3C were 2501 (NF1), 665 (NF2), and 762 (SWN). We compared these with N3C cohorts of patients with other rare diseases (98-9844 individuals) and the general non-NF population of 5.6 million. The site- and age-adjusted proportion of people with NF1, NF2, or SWN who had a positive test result for SARS-CoV-2 or COVID-19 (collectively termed positive cases) was not significantly higher than in individuals without NF or other selected rare diseases. There were no severe outcomes reported in the NF2 or SWN cohorts. The proportion of patients experiencing severe outcomes was no greater for people with NF1 than in cohorts with other rare diseases or the general population.

CONCLUSION

Having NF1, NF2, or SWN does not appear to increase the risk of being SARS-CoV-2 positive or of being a patient with COVID-19 or of developing severe complications from SARS-CoV-2.

Can authorship bias be detected in meta-analysis?

PURPOSE

Statistical approaches have been developed to detect bias in individual trials, but guidance on how to detect systematic differences at a meta-analytical level is lacking. In this paper, we elucidate whether author bias can be detected in a cohort of randomized trials included in a meta-analysis.

METHODS

We utilized mortality data from 35 trials (10,880 patients) included in our previously published meta-analysis. First, we linked each author with their trial (or trials). Then we calculated author-specific odds ratios using univariate cross table methods. Finally, we tested the effect of authorship by comparing each author's estimated odds ratio with all other pooled estimated odds ratios using meta-regression.

RESULTS

The median number of investigators named as authors on the primary trial reports was six (interquartile range: 5-8, range: 2-32). The results showed that the slope of author effect for mortality ranged from - 1.35 to 0.71. We identified only one author team showing a marginally significant effect (- 0.39; 95% confidence interval, - 0.78 to 0.00). This author team has a history of retractions due to data manipulations and ethical violations.

CONCLUSION

When combining trial-level data to produce a pooled effect estimate, investigators must consider sources of potential bias. Our results suggest that systematic errors can be detected using meta-regression, although further research is needed to examine the sensitivity of this model. Systematic reviewers will benefit from the availability of methods to guard against the dissemination of results with the potential to mislead decision-making.

Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies

The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light–especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or ‘spin’) in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government’s commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform’s champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended.