Evidence-based orthodontics: Appraisal of the methodologies of systematic reviews and meta-analyses in controversial areas of orthodontics

Methodological quality of meta-analyses of single-case experimental studies

BACKGROUND

Methodological rigor is a fundamental factor in the validity and credibility of the results of a meta-analysis.

AIM

Following an increasing interest in single-case experimental design (SCED) meta-analyses, the current study investigates the methodological quality of SCED meta-analyses.

METHODS AND PROCEDURES

We assessed the methodological quality of 178 SCED meta-analyses published between 1985 and 2015 through the modified Revised-Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist.

OUTCOMES AND RESULTS

The main finding of the current review is that the methodological quality of the SCED meta-analyses has increased over time, but is still low according to the R-AMSTAR checklist. A remarkable percentage of the studies (93.80% of the included SCED meta-analyses) did not even reach the midpoint score (22, on a scale of 0-44). The mean and median methodological quality scores were 15.57 and 16, respectively. Relatively high scores were observed for "providing the characteristics of the included studies" and "doing comprehensive literature search". The key areas of deficiency were "reporting an assessment of the likelihood of publication bias" and "using the methods appropriately to combine the findings of studies".

CONCLUSIONS AND IMPLICATIONS

Although the results of the current review reveal that the methodological quality of the SCED meta-analyses has increased over time, still more efforts are needed to improve their methodological quality.

Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties

OBJECTIVES

To summarize all available evidence on measurement properties in terms of reliability, validity, and feasibility of the Assessment of Multiple Systematic Reviews (AMSTAR) tool, including R(evised)-AMSTAR.

STUDY DESIGN AND SETTING

MEDLINE, EMBASE, Psycinfo, and CINAHL were searched for studies containing information on measurement properties of the tools in October 2013. We extracted data on study characteristics and measurement properties. These data were analyzed following measurement criteria.

RESULTS

We included 13 studies, four of them were labeled as validation studies. Nine articles dealt with AMSTAR, two articles dealt with R-AMSTAR, and one article dealt with both instruments. In terms of interrater reliability, most items showed a substantial agreement (>0.6). The median intraclass correlation coefficient (ICC) for the overall score of AMSTAR was 0.83 (range 0.60-0.98), indicating a high agreement. In terms of validity, ICCs were very high with all but one ICC lower than 0.8 when the AMSTAR score was compared with scores from other tools. Scoring AMSTAR takes between 10 and 20 minutes.

CONCLUSION

AMSTAR seems to be reliable and valid. Further investigations for systematic reviews of other study designs than randomized controlled trials are needed. R-AMSTAR should be further investigated as evidence for its use is limited and its measurement properties have not been studied sufficiently. In general, test-retest reliability should be investigated in future studies.