Raising the bar for systematic reviews with Assessment of Multiple Systematic Reviews (AMSTAR)

A Critical Analysis of Reporting in Systematic Reviews and Meta-Analyses in the Peyronie's Disease Literature

BACKGROUND

Despite an increasing number of publications on Peyronie's disease (PD), evidence-based clinical decision-making remains challenging due to the small number of well-designed clinical trials.

AIM

To perform a critical analysis of reporting quality in PD systematic reviews (SR) and meta-analyses (MA).

METHODS

Study protocol registration was performed on the Open Science Framework platform. In January 2021, a systematic electronic search of the Medline/PubMed, Embase, Ovid, Scopus, Joanne Briggs Institute, and Cochrane databases was performed. Search terms included "Peyronie's disease" and "systematic review OR meta-analysis OR meta-analysis." Eligibility criteria were English-language, relevance to PD and specification of "systematic review" or "meta-analysis" in the title or abstract. Oxford Center for Evidence-Based Medicine levels of evidence were used to classify original studies reviewed within each publication. Risk of bias was assessed using the ROBIS tool. Data were tabulated and reported as means with standard deviation, median with interquartile range and t-testing as appropriate. Strength of association between variables was calculated using Pearson correlation coefficient. Statistical analyses were performed on RStudio (version 1.4.1106).

OUTCOMES

Outcomes included review type, level of evidence, authorship, journal, publication date, "A MeaSurement Tool to Assess systematic Reviews" (AMSTAR-2) score and "Preferred Reporting Items for Systematic Reviews and Meta-Analyses" (PRISMA) score.

RESULTS

From 1974 to 2021, 340 articles were identified. After review, 17 full length articles were included. Thirteen were SR, 2 MA and 2 was combined. Significant heterogeneity was seen in evidence level of included studies. There was median 54% adherence to AMSTAR-2 criteria and 74% adherence to PRISMA criteria. Overall AMSTAR-2 confidence rating was Critically Low in 11 of 17 studies. Correlation analysis revealed very high positive association between AMSTAR 2 and PRISMA adherence (+0.95). ROBIS revealed "High" concern regarding methods used to collect data and appraise 12/17 studies (71%), and "High" concern regarding synthesis and findings in 8 of 17 studies (47%).

CLINICAL IMPLICATIONS

Many SR include markedly heterogenous levels of evidence and fail to meet accepted methodological criteria for reporting.

STRENGTHS AND LIMITATIONS

Main strengths include extensive literature review and analysis of standardized study reporting. One limitation is that aggregate scoring of AMSTAR-2 and PRISMA is not intended as primary method of quality assessment; however effect was minimized by reporting critical domains, overall quality assessments and specifics on globally poorly reported domains.

CONCLUSION

More high quality randomized controlled PD trials are necessary; SR and MA should focus on these studies alone. Bole R, Gottlich HC, Ziegelmann MJ, et al. A Critical Analysis of Reporting in Systematic Reviews and Meta-Analyses in the Peyronie's Disease Literature. J Sex Med 2022;19:629-640.

The quality of systematic reviews and meta-analyses assessing the treatment and management of male infertility

BACKGROUND

Male factor infertility (MFI) is a common medical condition which requires high-quality research to guide clinical practice; however, systematic reviews (SRs) and meta-analyses (MAs) often vary in quality, raising concerns regarding the validity of their results. We sought to perform an objective analysis of SRs and MAs in MFI treatment and management and to report on the quality of published literature.

METHODS

A comprehensive search in PubMed/MEDLINE and Embase was used to identify relevant publications. Primary search terms were male infertility, male sterility, and male subfertility. Two authors independently performed searches, screened citations for eligibility, extracted data for analysis, and graded methodological quality using the validated AMSTAR (A Measurement Tool to Assess Systematic Reviews) instrument, a validated tool used in the critical appraisal of SRs/MAs.

RESULTS

Of 27 publications met inclusion criteria and were included in the analysis. Mean AMSTAR score (± SD) among all publications was 7.4 (1.9) out of 11, reflecting "fair to good" quality. Non-pharmacological medical treatment for MFI was the most commonly assessed intervention (n = 13, 48.1%). No publications met all AMSTAR criteria. While the number of SRs/MAs has increased over time (P = 0.037), the quality of publications has not significantly changed (P = 0.72). SRs/MAs of the Cochrane Library had higher AMSTAR score than non-Cochrane SRs/MAs (8.5 vs 6.3, P = 0.002).

CONCLUSIONS

The methodological quality of SRs/MAs should be assessed to ensure high-quality evidence for clinical practice guidelines in MFI treatment and management. This review highlights a need for increased effort to publish high-quality studies in MFI treatment and management.

Clinical performance of zirconia implants: A meta-review

STATEMENT OF PROBLEM

The clinical effectiveness of zirconia implants as an alternative to titanium implants is still controversial.

PURPOSE

The purpose of this analysis was to identify and evaluate systematic reviews reporting on the clinical outcomes of zirconia implants for oral rehabilitation.

MATERIAL AND METHODS

An electronic search was undertaken on MEDLINE, Embase, and the Cochrane Oral Health Reviews databases up to December 24, 2018, without language restriction. Eligible reviews were screened and assessed. The eligibility criteria were systematic reviews or meta-analyses, implant survival rate, implant success, marginal bone loss, peri-implant soft tissue status, and biologic and functional complications of zirconia implants. Two review authors independently evaluated the quality assessment of the secondary studies by applying the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool.

RESULTS

Nine reviews fulfilled the inclusion criteria and were evaluated. Seven reviews were classified as moderate and 2 as high quality. The overall AMSTAR's quality of these reports was moderate. In the primary studies contained in these reviews, zirconia implant clinical outcomes were found to be similar or inferior to those for titanium implants. The few primary clinical studies contained in these reviews were not homogeneous among each other, presented poor methodology, and only offered promising short-term outcomes due to the lack of long-term follow-ups.

CONCLUSIONS

Based on this meta-review, in spite of short-term promising results of zirconia implants, evidence with long term is lacking.

Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR

Background

Systematic reviews (SRs) in the field of neuropathic pain (NeuP) are increasingly important for decision-making. However, methodological flaws in SRs can reduce the validity of conclusions. Hence, it is important to assess the methodological quality of NeuP SRs critically. Additionally, it remains unclear which assessment tool should be used. We studied the methodological quality of SRs published in the field of NeuP and compared two assessment tools.

Methods

We systematically searched 5 electronic databases to identify SRs of randomized controlled trials of interventions for NeuP available up to March 2015. Two independent reviewers assessed the methodological quality of the studies using the Assessment of Multiple Systematic Reviews (AMSTAR) and the revised AMSTAR (R-AMSTAR) tools. The scores were converted to percentiles and ranked into 4 grades to allow comparison between the two checklists. Gwet’s AC1 coefficient was used for interrater reliability assessment.

Results

The 97 included SRs had a wide range of methodological quality scores (AMSTAR median (IQR): 6 (5–8) vs. R-AMSTAR median (IQR): 30 (26–35)). The overall agreement score between the 2 raters was 0.62 (95% CI 0.39–0.86) for AMSTAR and 0.62 (95% CI 0.53–0.70) for R-AMSTAR. The 31 Cochrane systematic reviews (CSRs) were consistently ranked higher than the 66 non-Cochrane systematic reviews (NCSRs). The analysis of individual domains showed the best compliance in a comprehensive literature search (item 3) on both checklists. The results for the domain that was the least compliant differed: conflict of interest (item 11) was the item most poorly reported on AMSTAR vs. publication bias assessment (item 10) on R-AMSTAR. A high positive correlation between the total AMSTAR and R-AMSTAR scores for all SRs, as well as for CSRs and NCSRs, was observed.

Conclusions

The methodological quality of analyzed SRs in the field of NeuP was not optimal, and CSRs had a higher quality than NCSRs. Both AMSTAR and R-AMSTAR tools produced comparable quality ratings. Our results point out to weaknesses in the methodology of existing SRs on interventions for the management NeuP and call for future improvement by better adherence to analyzed quality checklists, either AMSTAR or R-AMSTAR.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0493-y) contains supplementary material, which is available to authorized users.

PRISMA and AMSTAR show systematic reviews on health literacy and cancer screening are of good quality

OBJECTIVES

The objectives of this study were to evaluate the reporting and methodological quality of systematic reviews (SRs) on health literacy and cancer screening and to investigate factors that may influence overall quality.

STUDY DESIGN AND SETTING

We reviewed SRs published between 2009 and 2017. We calculated indices to represent the included SRs' adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) and A Measurement Tool to Assess Systematic Reviews (AMSTAR). To assess possible determinants of SRs' quality, we regressed the index scores on year and region of publication, journal impact factor, authors' reported use of PRISMA, and presence of funding statements.

RESULTS

We included 19 SRs, and median index scores were 0.86 for PRISMA (interquartile range [IQR] = 0.11; range = 0.32-1.00) and 0.67 for AMSTAR (IQR = 0.30; range = 0.22-1.00). Methodological and reporting problems pertained to protocol registration or publication, number of raters used, gray literature searches, excluded article lists, and unintegrated discussions of risk of bias and efficacy. Only journal impact factor was statistically significantly associated (positively) with PRISMA and AMSTAR index scores.

CONCLUSION

The quality of SRs on health literacy and cancer screening was generally good. Systematic reviewers should register or publish their protocols, include PRISMA and AMSTAR checklists when submitting SRs to journals, and self-evaluate their SRs before submission.

A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer

OBJECTIVE

To compare cancer-related systematic reviews (SRs) published in the Cochrane Database of SRs (CDSR) and high-impact journals, with respect to type, content, quality and citation rates.

DESIGN

Methodological SR with assessment and comparison of SRs and meta-analyses. Two authors independently assessed methodological quality using an Assessment of Multiple Systematic Reviews (AMSTAR)-based extraction form. Both authors independently screened search results, extracted content-relevant characteristics and retrieved citation numbers of the included reviews using the Clarivate Analytics Web of Science database.

DATA SOURCES

Cancer-related SRs were retrieved from the CDSR, as well as from the 10 journals which publish oncological SRs and had the highest impact factors, using a comprehensive search in both the CDSR and MEDLINE.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included all cancer-related SRs and meta-analyses published from January 2011 to May 2016. Methodological SRs were excluded.

RESULTS

We included 346 applicable Cochrane reviews and 215 SRs from high-impact journals. Cochrane reviews consistently met more individual AMSTAR criteria, notably with regard to an a priori design (risk ratio (RR) 3.89; 95% CI 3.10 to 4.88), inclusion of the grey literature and trial registries (RR 3.52; 95% CI 2.84 to 4.37) in their searches, and the reporting of excluded studies (RR 8.80; 95% CI 6.06 to 12.78). Cochrane reviews were less likely to address questions of prognosis (RR 0.04; 95% CI 0.02 to 0.09), use individual patient data (RR 0.03; 95% CI 0.01 to 0.09) or be based on non-randomised controlled trials (RR 0.04; 95% CI 0.02 to 0.09). Citation rates of Cochrane reviews were notably lower than those for high-impact journals (Cochrane reviews: mean number of citations 6.52 (range 0-143); high-impact journal SRs: 74.45 (0-652)).

CONCLUSIONS

When comparing cancer-related SRs published in the CDSR versus those published in high-impact medical journals, Cochrane reviews were consistently of higher methodological quality, but cited less frequently.

AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both

The number of published systematic reviews of studies of healthcare interventions has increased rapidly and these are used extensively for clinical and policy decisions. Systematic reviews are subject to a range of biases and increasingly include non-randomised studies of interventions. It is important that users can distinguish high quality reviews. Many instruments have been designed to evaluate different aspects of reviews, but there are few comprehensive critical appraisal instruments. AMSTAR was developed to evaluate systematic reviews of randomised trials. In this paper, we report on the updating of AMSTAR and its adaptation to enable more detailed assessment of systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. With moves to base more decisions on real world observational evidence we believe that AMSTAR 2 will assist decision makers in the identification of high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.