The certainty of the evidence in oral health has not improved according to GRADE: a meta-epidemiological study

OBJECTIVE

The aim of this meta-epidemiological study was to provide an update of the certainty of the evidence in oral health by using the GRADE rating reported in oral health Cochrane systematic reviews (CSR).

STUDY DESIGN AND SETTING

All interventional oral health CSR published between 2003-2021 were sourced. Study characteristics were extracted at the level of the CSR and the outcome/meta-analysis. One-hundred-five CSR were eligible and analysed.

RESULTS

Almost a third of CSR (n=67) were excluded as a GRADE rating was not available. The most prevalent type of primary studies included in the CSR were randomized studies (93.4%) and the most used measure of effect was the risk ratio (67.3%). Overall, the certainty of the evidence according to the GRADE rating for all examined outcomes was very low/low (88%). The two most common reasons for downgrading the confidence in the evidence were study limitations (Risk of bias) and imprecision. The odds of moderate/high vs. low/very low-GRADE rating are higher for the primary compared to the secondary outcomes after adjusting for year and number of trials (OR 2.49; 95% CI: 1.09, 5.65; P=0.02). Per year (2010-2021 period) the odds of moderate/high vs. low/very low-GRADE rating decrease (OR 0.73; 95% CI: 0.60, 0.90; P=0.01), and as the number of trials per comparison increase the odds of moderate/high vs. low/very low GRADE rating increase (OR 1.13; 95% CI: 1.01, 1.25; P=0.001).

CONCLUSIONS

The certainty of the evidence in oral health when assessed with the GRADE rating remains predominantly low or very low.

Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study

Background

In their research reports, scientists are expected to discuss limitations that their studies have. Previous research showed that often, such discussion is absent. Also, many journals emphasize the importance of avoiding overstatement of claims. We wanted to see to what extent editorial handling and peer review affects self-acknowledgment of limitations and hedging of claims.

Methods

Using software that automatically detects limitation-acknowledging sentences and calculates the level of hedging in sentences, we compared the submitted manuscripts and their ultimate publications of all randomized trials published in 2015 in 27 BioMed Central (BMC) journals and BMJ Open. We used mixed linear and logistic regression models, accounting for clustering of manuscript-publication pairs within journals, to quantify before-after changes in the mean numbers of limitation-acknowledging sentences, in the probability that a manuscript with zero self-acknowledged limitations ended up as a publication with at least one and in hedging scores.

Results

Four hundred forty-six manuscript-publication pairs were analyzed. The median number of manuscripts per journal was 10.5 (interquartile range 6–18). The average number of distinct limitation sentences increased by 1.39 (95% CI 1.09–1.76), from 2.48 in manuscripts to 3.87 in publications. Two hundred two manuscripts (45.3%) did not mention any limitations. Sixty-three (31%, 95% CI 25–38) of these mentioned at least one after peer review. Changes in mean hedging scores were negligible.

Conclusions

Our findings support the idea that editorial handling and peer review lead to more self-acknowledgment of study limitations, but not to changes in linguistic nuance.

A risk of bias instrument for non-randomized studies of exposures: A users' guide to its application in the context of GRADE

The objective of this paper is to explain how to apply, interpret, and present the results of a new instrument to assess the risk of bias (RoB) in non-randomized studies (NRS) dealing with effects of environmental exposures on health outcomes. This instrument is modeled on the Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I) instrument. The RoB instrument for NRS of exposures assesses RoB along a standardized comparison to a randomized target experiment, instead of the study-design directed RoB approach. We provide specific guidance for the integral steps of developing a research question and target experiment, distinguishing issues of indirectness from RoB, making individual-study judgments, and performing and interpreting sensitivity analyses for RoB judgments across a body of evidence. Also, we present an approach for integrating the RoB assessments within the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework to assess the certainty of the evidence in the systematic review. Finally, we guide the reader through an overall assessment to support the rating of all domains that determine the certainty of a body of evidence using the GRADE approach.

The spurious relationship between ecstasy use and neurocognitive deficits: A Bradford Hill review

Numerous studies have suggested that MDMA can cause neurocognitive deficits. However, the available data can only suggest an association - rather than a causal relationship - between MDMA use and neurocognitive deficits. The reliability and robustness of this association was evaluated using Bradford Hill's criteria for determining causation in epidemiology research. Several limitations in the literature were found. Studies have recruited people who abuse ecstasy - an illicit drug that does not always contain MDMA. There is inherent risk in consuming impure or falsely identified substances; and using this as a source as for scientific opinion may introduce biases in our understanding the actuals risks associated with MDMA. Importantly, given that ecstasy research is predominately retrospective, baseline functioning cannot be established; which may be influenced by a variety of preexisting factors. Many studies introduce statistical errors by inconsistently dichotomizing and comparing light and heavy ecstasy users, making dose-response relationships inconclusive. When interpreting the ecstasy literature effect sizes are a more meaningful indicator of neurocognitive functioning rather than relying on p-values alone. Most meta-analyses have failed to find clinically relevant differences between ecstasy users and controls. There is also consistent evidence of publication bias in this field of research, which indicates that the literature is both biased and incomplete. Finally, suggestions for improving the ecstasy literature are provided.

Using the contribution matrix to evaluate complex study limitations in a network meta-analysis: a case study of bipolar maintenance pharmacotherapy review

Background

Limitations in the primary studies constitute one important factor to be considered in the grading of recommendations assessment, development, and evaluation (GRADE) system of rating quality of evidence. However, in the network meta-analysis (NMA), such evaluation poses a special challenge because each network estimate receives different amounts of contributions from various studies via direct as well as indirect routes and because some biases have directions whose repercussion in the network can be complicated.

Findings

In this report we use the NMA of maintenance pharmacotherapy of bipolar disorder (17 interventions, 33 studies) and demonstrate how to quantitatively evaluate the impact of study limitations using netweight, a STATA command for NMA. For each network estimate, the percentage of contributions from direct comparisons at high, moderate or low risk of bias were quantified, respectively. This method has proven flexible enough to accommodate complex biases with direction, such as the one due to the enrichment design seen in some trials of bipolar maintenance pharmacotherapy.

Conclusions

Using netweight, therefore, we can evaluate in a transparent and quantitative manner how study limitations of individual studies in the NMA impact on the quality of evidence of each network estimate, even when such limitations have clear directions.