Bias in systematic reviews?

Systematic reviews in orthodontics: Impact of the PRISMA for Abstracts checklist on completeness of reporting

INTRODUCTION

This study evaluated and compared the completeness of reporting of abstracts of orthodontics systematic reviews before and after the publication of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for Abstracts Checklist (PRISMA-A).

METHODS

Abstracts of systematic reviews and meta-analyses in orthodontics published in PubMed, Latin American and Caribbean Health Sciences Literature, and the Cochrane Database of Systematic Reviews databases before March 23, 2018, that met the predefined inclusion and exclusion criteria, were evaluated using the 12 items of PRISMA-A, scoring each item from 0 to 2. Abstracts were classified into 2 groups: before and after publication of the PRISMA-A checklist. Three calibrated evaluators (intraclass correlation coefficient and kappa > 0.8) assessed the scores for compliance with the checklist. The number of authors, country of affiliation of the first author, performance of meta-analysis, and topic of the article were recorded. A regression analysis was performed to assess the associations between abstract characteristics and the PRISMA-A scores.

RESULTS

Of 1034 abstracts evaluated, 389 were included in the analysis. The mean PRISMA-A score was 53.39 (95% CI, 51.83-54.96). The overall score for studies published after the publication of the checklist was significantly higher than for studies published before (P ≤ 0.0001). The components returning significantly higher scores after publication of PRISMA-A were title (P = 0.024), information from databases (P = 0.026), risk of bias (P ≤ 0.0001), included studies (P ≤ 0.0001), synthesis of results (P ≤ 0.0001), interpretation of results (P = 0.035), financing and conflict of interest (P ≤ 0.0001), and registration (P ≤ 0.0001). These results showed the positive effect of PRISMA-A had on the quality of reporting of orthodontics systematic reviews. Nevertheless, the poor adherence revealed that there is still need for improvement in the quality of abstract reporting.

CONCLUSIONS

The quality of reporting of abstracts of orthodontic systematic reviews and meta-analyses increased after the introduction of PRISMA-A.

Heterogeneity in Cochrane and non-Cochrane meta-analyses in orthodontics

OBJECTIVES

Heterogeneity describes the percentage of variability across the study effects that can be attributed to between-study differences in a meta-analysis. The aim of this project was to explore the magnitude of heterogeneity in Cochrane and non-Cochrane meta-analyses in orthodontic research and to identify possible associations between heterogeneity (I²) and a number of study characteristics including number of studies, type of outcome and type of analysis.

METHODS

The contents of five major orthodontic journals and the Cochrane Database of Systematic Reviews were electronically searched from January 2000 to December 2017 to identify Systematic Reviews (SRs) with at least one meta-analysis. Included records were screened for reporting of I² classified into four categories: 0%, 1-29%, 30-59%, 60-100%. Associations between I² and review-level and synthesis-level characteristics were tested. Univariable and multivariable mixed effects ordinal logistic regression was used to identify significant predictors for statistical heterogeneity.

RESULTS

A total of 72 SRs comprising 391 meta-analyses were included with the majority based on non-Cochrane reviews (n = 54, 75%). Overall, 125 meta-analyses (32%) reported heterogeneity explained by chance (I² = 0%), whereas high values of I² (∼60-100%) were seen in 152 syntheses (39%). In the multivariable analysis, inclusion of each additional study within the synthesis presented 20% higher odds for substantial/considerable heterogeneity compared to lower heterogeneity categories (OR = 1.20; 95%CIs: 1.09, 1.31; p < 0.001). Use of fixed effect analysis (OR = 0.25; 95%CIs: 0.12, 0.55; p = 0.001) was associated with significantly lower odds. Cochrane versus non-Cochrane meta-analyses were not associated with higher odds for substantial/considerable heterogeneity (OR = 2.81; 95%CIs: 0.53, 14.91; p = 0.22).

CONCLUSIONS

Substantial statistical heterogeneity is present within a considerable number of orthodontic meta-analyses. Further efforts should be made to improve understanding of decisions to undertake meta-analyses and selection of studies eligible for inclusion.

CLINICAL SIGNIFICANCE

The consistency of meta-analyses could be improved with more careful consideration of individual study characteristics. Reduced heterogeneity in meta-analyses will ensue more solid evidence based decisions for clinical practice.

Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews

Objectives

To assess discrepancies in the analyzed outcomes between protocols and published reviews within Cochrane oral health systematic reviews (COHG) on the Cochrane Database of Systematic Reviews (CDSR).

Study Design and Setting

All COHG systematic reviews on the CDSR and the corresponding protocols were retrieved in November 2014 and information on the reported outcomes was recorded. Data was collected at the systematic review level by two reviewers independently.

Results

One hundred and fifty two reviews were included. In relation to primary outcomes, 11.2% were downgraded to secondary outcomes, 9.9% were omitted altogether in the final publication and new primary outcomes were identified in 18.4% of publications. For secondary outcomes, 2% were upgraded to primary, 12.5% were omitted and 30.9% were newly introduced in the publication. Overall, 45.4% of reviews had at least one discrepancy when compared to the protocol; these were reported in 14.5% reviews. The number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). The risk of reporting significant results was lower for both downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50] compared to outcomes with no discrepancies. The risk of reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57). None of the comparisons reached statistical significance.

Conclusion

While no evidence of selective outcome reporting was found in this study, based on the present analysis of SRs published within COHG systematic reviews, discrepancies between outcomes in pre-published protocols and final reviews continue to be common. Solutions such as the use of standardized outcomes to reduce the prevalence of this issue may need to be explored.

Quality in physical therapist clinical education: a systematic review

BACKGROUND

Many factors affect student learning throughout the clinical education (CE) component of professional (entry-level) physical therapist education curricula. Physical therapist education programs (PTEPs) manage CE, yet the material and human resources required to provide CE are generally overseen by community-based physical therapist practices.

PURPOSE

The purposes of this systematic review were: (1) to examine how the construct of quality is defined in CE literature and (2) to determine the methodological rigor of the available evidence on quality in physical therapist CE.

METHODS

This study was a systematic review of English-language journals using the American Physical Therapy Association's Open Door Portal to Evidence-Based Practice as the computer search engine. The search was categorized using terms for physical therapy and quality and for CE pedagogy and models or roles. Summary findings were characterized by 5 primary themes and 14 subthemes using a qualitative-directed content analysis.

RESULTS

Fifty-four articles were included in the study. The primary quality themes were: CE framework, CE sites, structure of CE, assessment in CE, and CE faculty. The methodological rigor of the studies was critically appraised using a binary system based on the McMaster appraisal tools. Scores ranged from 3 to 14.

LIMITATIONS

Publication bias and outcome reporting bias may be inherent limitations to the results.

CONCLUSION

The review found inconclusive evidence about what constitutes quality or best practice for physical therapist CE. Five key constructs of CE were identified that, when aggregated, could construe quality.

A descriptive analysis of oral health systematic reviews published 1991-2012: cross sectional study

Objectives

To identify all systematic reviews (SRs) published in the domain of oral health research and describe them in terms of their epidemiological and descriptive characteristics.

Design

Cross sectional, descriptive study.

Methods

An electronic search of seven databases was performed from inception through May 2012; bibliographies of relevant publications were also reviewed. Studies were considered for inclusion if they were oral health SRs defined as therapeutic or non-therapeutic investigations that studied a topic or an intervention related to dental, oral or craniofacial diseases/disorders. Data were extracted from all the SRs based on a number of epidemiological and descriptive characteristics. Data were analysed descriptively for all the SRs, within each of the nine dental specialities, and for Cochrane and non-Cochrane SRs separately.

Results

1,188 oral health (126 Cochrane and 1062 non-Cochrane) SRs published from 1991 through May 2012 were identified, encompassing the nine dental specialties. Over half (n = 676; 56.9%) of the SRs were published in specialty oral health journals, with almost all (n = 1,178; 99.2%) of the SRs published in English and almost none of the non-Cochrane SRs (n = 11; 0.9%) consisting of updates of previously published SRs. 75.3% of the SRs were categorized as therapeutic, with 64.5% examining non-drug interventions, while approximately half (n = 150/294; 51%) of the non-therapeutic SRs were classified as epidemiological SRs. The SRs included a median of 15 studies, with a meta-analysis conducted in 43.6%, in which a median of 9 studies/1 randomized trial were included in the largest meta-analysis conducted. Funding was received for 25.1% of the SRs, including nearly three-quarters (n = 96; 76.2%) of the Cochrane SRs.

Conclusion

Epidemiological and descriptive characteristics of the 1,188 oral health SRs varied across the nine dental specialties and by SR category (Cochrane vs. non-Cochrane). There is a clear need for more updates of SRs in all the dental specialties.