Quality Assessment of Published Systematic Reviews in High Impact Cardiology Journals: Revisiting the Evidence Pyramid

Methodological quality of umbrella reviews in endodontics: A cross-sectional study

INTRODUCTION

In endodontics, the number of umbrella reviews has increased significantly over the last few years, but there is no evidence that they were methodologically sound. The aim of the current study was to appraise the methodological quality of umbrella reviews in endodontics, and to identify possible predictive factors associated with methodological quality.

METHODS

Umbrella reviews published in the discipline of endodontics until December 2023 were included. The methodological quality of the reviews was evaluated using a checklist consisting of 11 items. Each item in the checklist was evaluated by two independent assessors who assigned a score of '1' if it was fully addressed, '0.5' if it was partially ddressed, and '0' if it was not addressed. Bootstrapped multiple linear regression analysis was used to examine the association between the total scores awarded and five predictor variables (a priori protocol registration, year of publication, number of authors, journal impact factor (IF) and continent of the corresponding author). The statistical significance level was set as 5%.

RESULTS

A total of 27 reviews were included. Ninety-six per cent of the reviews adequately reported: eligibility criteria for selecting the reviews, details of the reviews, techniques for assessing the risk of bias or methodological quality of the individual systematic reviews they included. Only 30% of the reviews adequately managed overlapping primary studies within individual systematic reviews. Among the five predictors analysed, a priori protocol registration and journals with IFs were associated with significantly greater total methodological quality scores.

DISCUSSION

Several methodological shortcomings in the umbrella reviews published within the field of endodontics were revealed. Umbrella reviews published in journals with IFs and those with protocols registered a priori had significantly superior methodological quality scores.

CONCLUSION

In endodontics, authors intending to publish umbrella reviews should consider the limitations revealed in this study and follow the appropriate rules to ensure their reviews comply with the highest standards and provide accurate and dependable information and conclusions.

Percutaneous coronary intervention versus optimal medical therapy for stable coronary artery disease: An umbrella review

Background

Invasive management of stable coronary artery disease is still a controversial topic. The purpose of this umbrella review was to synthesize systematic reviews (SRs) that evaluate the benefits and harms of percutaneous coronary intervention (PCI) versus optimal medical therapy (OMT) in patients with stable coronary artery disease.

Methods

We systematically searched PubMed/MEDLINE, Embase, and CENTRAL from 2018 to August 7, 2022. We included SRs with meta-analyses of randomized controlled trials (RCTs) that evaluated the question of interest. We assessed the methodological quality of the SRs with the AMSTAR-2 tool. We summarized the results of the outcomes for each SR. We calculated the degree of overlap of the RCTs included in the SRs using the corrected covered area (CCA).

Results

We found 10 SRs with meta-analyses. The SRs included 3 to 15 RCTs. The degree of overlap among the SRs was very high (CCA > 15%). No SR evaluated the certainty of the evidence using the GRADE system and 9 out of 10 had critically low methodological quality. The SRs reported heterogeneous results for the outcomes of all-cause mortality, myocardial infarction, revascularization, and angina. On the other hand, for the outcomes of cardiovascular mortality and stroke, all SRs agreed that there were no differences between PCI and OMT alone.

Conclusions

We found 10 SRs on the use of PCI compared to OMT alone for patients with stable coronary artery disease. However, none had high methodological quality, none evaluated the certainty of the evidence using the GRADE approach, and the results were inconsistent for several outcomes. This variability in evidence may result in divergent clinical decisions for the management of stable coronary artery disease among healthcare professionals. It is necessary to perform a high-quality SR using the GRADE approach to clarify the balance of benefits and harms of PCI.

An overview of systematic reviews on imaging tests for diagnosis of pulmonary embolism applying different network meta-analytic methods

PURPOSE

This study aimed to apply different methods of diagnostic test accuracy network meta-analysis (DTA-NMA) for studies reporting results of five imaging tests for the diagnosis of suspected pulmonary embolism (PE): pulmonary angiography (PA), computed tomography angiography (CTPA), magnetic resonance angiography (MRA), planar ventilation/perfusion (V/Q) scintigraphy and single-photon emission computed tomography ventilation/perfusion (SPECT V/Q).

METHODS

We searched four databases (MEDLINE [via PubMed], Cochrane CENTRAL, Scopus, and Epistemonikos) from inception until June 2, 2022 to identify systematic reviews (SRs) describing diagnostic accuracy of PA, CTPA, MRA, V/Q scan and SPECT V/Q for suspected PE. Study-level data were extracted and pooled using a hierarchical summary receiver operating characteristic (HSROC) meta-regression approach and two DTA-NMA models to compare accuracy estimates of different imaging tests. Risk of bias was assessed using the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies-2) tool and certainty of evidence using the GRADE (Grading of Recommendations Assessment, Development and Evaluation) framework.

RESULTS

We identified 13 SRs, synthesizing data from 33 primary studies and for four imaging tests (PA, CTPA, MRA and V/Q scan). The HSROC meta-regression model using PA as the reference standard showed that MRA had the best overall diagnostic performance with sensitivity of 0.93 (95% confidence interval [CI]: 0.76, 1.00) and specificity of 0.94 (95% CI: 0.84, 0.99). However, DTA-NMA models indicated that V/Q scan had the highest sensitivity, while CTPA was most specific.

CONCLUSION

Selecting a different DTA-NMA method to assess multiple diagnostic tests can affect estimates of diagnostic accuracy. There is no established method, but the choice depends on the data and familiarity with Bayesian statistics.

Exploring the relationship between the number of systematic reviews and quality of evidence: an orthognathic surgery-based study

OBJECTIVE

We analyzed the quality and quantity of systematic reviews (SRs) of orthognathic surgery, the most frequently published topic in maxillofacial surgery.

STUDY DESIGN

We searched the PubMed database for SRs of orthognathic surgery with no restriction on the language of publication date. We assessed the certainty of evidence presented according to the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol and the Leiden Manifesto using the Grading of Recommendations Assessment, Development, and Evaluation approach. We analyzed the data using descriptive statistics, Pearson´s correlation test, and linear regression.

RESULTS

Of the 171 SRs evaluated, approximately one fifth presented evidence with a high level of certainty. The number of orthognathic surgery SRs has been increasing, and many SRs were published after very similar topics had already been published. There is no relationship between the impact factor and the certainty of evidence.

CONCLUSIONS

An excessive number of SRs of orthognathic surgery are published, and many SRs are superfluous, simply reporting previous findings. Clinicians should not base treatment decisions solely on the evidence presented in SRs, and journal editors and reviewers should evaluate these SRs more critically, particularly when they address topics that have already been covered in the literature.

Methodological assessment and overall confidence in the results of systematic reviews with meta-analyses focusing on traumatic dental injuries: A cross-sectional study

BACKGROUND/AIMS

High methodological quality is required to interpret results of systematic reviews (SRs) in a reliable and accurate manner. The primary aim of this study was to appraise the methodologic quality of SRs with meta-analysis within the field of traumatic dental injuries using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 tool and assess overall confidence in their results. A secondary aim was to identify potential predictive factors associated with methodological quality.

MATERIALS AND METHODS

SRs with meta-analyses published in English in the field of traumatic dental injuries from inception to March 2023 were identified. The methodological quality of the included reviews was assessed using the AMSTAR 2 checklist. Two independent evaluators scored each AMSTAR 2 item as "yes" if it was adequately addressed, "partial yes" if it was partially addressed, and "no" if it was not addressed. The overall confidence in the results of each review was classified as "High," "Moderate," "Low," or "Critically low." Using multiple regression, the relationship between five predictor variables (journal impact factor, year of publication, number of authors, journal adherence to Preferred Reporting Items for Systematic reviews and Meta-analyses [PRISMA] guidelines and a priori protocol registration) and the total AMSTAR 2 scores was analyzed. The p-value was 5%.

RESULTS

Forty-one SRs were included. The overall confidence in the results of 13 reviews was categorized as "Critically low," 18 as "Low," 3 as "Moderate" and 7 as "High." Among the five predictor variables analyzed statistically, impact factor of the journal and year of publication significantly influenced the total AMSTAR 2 scores. The number of authors, adherence to PRISMA guidelines, and a priori protocol registration had no significant impact on AMSTAR 2 scores.

CONCLUSION

The overall confidence in the results of SRs with meta-analysis within the field of traumatic dental injuries was "Low" or "Critically Low" in the vast majority of studies (31 of 41). SRs with meta-analyses published in journals with higher impact factors and more recent publications had significantly higher methodological quality.

Overviews of reviews in the cardiovascular field underreported critical methodological and transparency characteristics: a methodological study based on the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement

OBJECTIVES

This study aimed to evaluate the epidemiology, reporting characteristics, and adherence to the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement of overviews of reviews (overviews) of interventions in the cardiovascular field.

STUDY DESIGN AND SETTING

MEDLINE, Scopus, and the Cochrane Database of Systematic Reviews were searched from January 1, 2000, to October 15, 2020. An updated search was performed in MEDLINE, Epistemonikos, and Google Scholar up to August 25, 2022. Overviews of interventions published in English and primarily considering populations, interventions, and outcomes pertinent to the cardiovascular field were eligible. Study selection, data extraction, and PRIOR adherence assessment were performed by two authors independently.

RESULTS

We analyzed 96 overviews. Almost half (43/96 [45%]) were published between 2020 and 2022 and contained a median of 15 systematic reviews (SRs) (interquartile range, 9-28). The commonest title terminology was "overview of (systematic) reviews" (38/96 [40%]). Methods for handling SR overlap were reported in 24/96 (25%), methods for assessing primary study overlap in 18/96 (19%), handling of discrepant data in 11/96 (11%), and methods for methodological quality or risk of bias assessment of the primary studies within SRs in 23/96 (24%). Authors included data sharing statements in 28/96 (29%), complete funding disclosure in 43/96 (45%), protocol registration in 43/96 (45%), and conflict of interest statement in 82/96 (85%) overviews.

CONCLUSION

Insufficient reporting was identified in methodological characteristics unique in overviews' conduct and most transparency markers. Adoption of PRIOR from the research community could ameliorate overviews' reporting.

Mapping the characteristics, methodological quality and standards of reporting of network meta-analyses on antithrombotic therapies: An overview

BACKGROUND

Although a large number of network meta-analyses (NMAs) in the field of cardiology are available, little is known about their methodological quality. We aimed to map the characteristics and critically appraised the standards of conduct and evidence reporting of NMAs assessing antithrombotic therapies for the treatment or prophylaxis of heart diseases and cardiac surgical procedures.

METHODS

We systematically searched PubMed and Scopus to identify NMAs comparing the clinical effects of antithrombotic therapies. Overall characteristics of the NMAs were extracted and their reporting quality and methodological quality were evaluated using the PRISMA-NMA checklist and AMSTAR-2, respectively.

RESULTS

We found 86 NMAs published between 2007 and 2022. Comparisons among direct-acting oral anticoagulants were available in 61 (71%) NMAs. Although around 75% of NMAs stated that they followed international guidelines for conduct and reporting, only one third provided a protocol/register. Complete search strategies and publication bias assessment were lacking in around 53% and 59% of studies, respectively. Most NMAs (n = 77, 90%) provided supplemental material; however, only 5 (6%) made the complete raw data available. Network diagrams were depicted in most studies (n = 67, 78%), yet network geometry was described in only 11 (12.8%) of them. Mean adherence to the PRISMA-NMA checklist was 65.1 ± 16.5%. AMSTAR-2 assessment showed 88% of the NMAs had critically low methodological quality.

CONCLUSION

Although there is a wide diffusion of NMA-type studies on antithrombotics for heart diseases, their methodological and reporting quality remains suboptimal. This may reflect fragile clinical practices due to misleading conclusions from critically low-quality NMAs.

Trial Registry Searches In Plastic Surgery Systematic Reviews: A Meta-epidemiological Study

INTRODUCTION

Clinical trial registry searches for unpublished clinical trial data are a means of mitigating publication bias within systematic reviews (SRs). The purpose of our study is to look at the rate of clinical trial registry searches conducted by SRs in the top five Plastic and Reconstructive Surgery journals.

METHODS

We identified the top five plastic and reconstructive surgery journals using the Google h-5 index. We then searched Pubmed for SRs published in these journals and compared them to plastic surgery SRs published in the Cochrane Collaboration for SRs over the last 5 y. We included all SRs that were published within these top five journals and Cochrane between December 6, 2016 and December 6, 2021. We then conducted a secondary analysis on clinicaltrials.gov looking for unpublished clinical trials for 100 randomized SRs that did not conduct a clinical trial registry search.

RESULTS

In SRs, 3.3% (17/512) from plastic surgery journals conducted trial registry searches. In comparison, 95.0% (38/40) of Cochrane Collaboration SRs conducted trial registry searches. Our secondary analysis found that 50% (50/100) of SRs could have included at least one unpublished clinical trial data set.

CONCLUSIONS

We found that plastic surgery SRs rarely include searches for unpublished clinical trial data in clinical trial registries. To improve the data completeness of SRs in plastic surgery journals, we recommend journals alter their author guidelines to require a clinical trial registry search for unpublished literature.

Chinese Medicine as an Adjunctive Treatment for Gastric Cancer: Methodological Investigation of meta-Analyses and Evidence Map

Background: Many meta-analyses (MAs) on Chinese medicine (CM) as an adjunctive treatment for gastric cancer have been published in recent years. However, the pooled evidence reported in MAs and their methodological quality remain unknown. Therefore, we designed a study to comprehensively evaluate and summarize the current evidence of CMs for gastric cancer in published MAs.

Methods: A systematic search on MAs published in English from inception to 1st September 2021 was conducted in PubMed and Embase. The AMSTAR-2 tool was used to evaluate the methodological quality of the included MAs, and the results of the quality assessment were visualized using the evidence mapping method. Stata 17/SE was used for statistical analysis (Registration number: INPLASY202190005).

Results: A total of 20 MAs (16 pairwise and 4 network MAs) were included from 118 records. These MAs were published in 14 journals from 2013 to 2021, with the number of patients and trials ranging from 688 to 6,857, and from 10 to 85, respectively. A large number of CMs (e.g., AiDi, FuFangKuShen, and HuaChanSu) in combination with chemotherapy for gastric cancer were identified among the included MAs. According to the pooled results reported in MAs, when compared to chemotherapy alone, CMs in combination with chemotherapy not only improve various outcomes on efficacy (e.g., objective response rate, quality of life) but also reduce various adverse reactions (e.g., leucopenia, nausea and vomiting). Only 2 MAs were low in terms of the overall methodological quality, while the other 18 MAs were all critically low. The methodology was required to be advanced significantly, mainly involving: study protocol and registration, explanation for the inclusion of study design, list of excluded studies with justifications, adequate details of included studies, reporting on funding sources of primary studies, and evaluation of the potential impact of risk of bias. In addition, MAs that received funds support (β = 2.68; 95%CI: 0.40 to 4.96; p = 0.024) or were published in journals with higher impact factor (β = 2.81; 95%CI: 0.69 to 4.92; p = 0.012) had a higher score on the overall methodological quality in the univariate analysis, but the results were not statistically significant according to the multivariate analysis.

Conclusion: Combining CMs with chemotherapy can potentially improve clinical outcomes and reduce the relevant adverse effects in patients with gastric cancer. However, the methodological quality of relevant MAs requires significant improvement, and the current evidence needs to be validated through multinational trials that are well-designed and have a large sample size.