Methodological Quality of Systematic Reviews and Meta-Analysis on Asthma Treatments. A Cross-Sectional Study

Methodological quality of systematic reviews on atopic dermatitis treatments: a cross-sectional study

BACKGROUND

Systematic reviews (SRs) could offer the best evidence supporting interventions, but methodological flaws limit their trustworthiness in decision-making. This cross-sectional study appraised the methodological quality of SRs on atopic dermatitis (AD) treatments.

METHODS

We searched MEDLINE, EMBASE, PsycINFO, and Cochrane Database for SRs on AD treatments published in 2019-2022. We extracted SRs' bibliographical data and appraised SRs' methodological quality with AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2. We explored associations between methodological quality and bibliographical characteristics.

RESULTS

Among the 52 appraised SRs, only one (1.9%) had high methodological quality, while 45 (86.5%) critically low. For critical domains, only five (9.6%) employed comprehensive search strategy, seven (13.5%) provided list of excluded studies, 17 (32.7%) considered risk of bias in primary studies, 21 (40.4%) contained registered protocol, and 24 (46.2%) investigated publication bias. Cochrane reviews, SR updates, SRs with European corresponding authors, and SRs funded by European institutions had better overall quality. Impact factor and author number positively associated with overall quality.

CONCLUSIONS

Methodological quality of SRs on AD treatments is unsatisfactory. Future reviewers should improve the above critical methodological aspects. Resources should be devolved into upscaling evidence synthesis infrastructure and improving critical appraisal skills of evidence users.

Methodological quality of systematic reviews on sepsis treatments: A cross-sectional study

OBJECTIVE

Systematic reviews (SRs) offer updated evidence to support decision-making on sepsis treatments. However, the rigour of SRs may vary, and methodological flaws may limit their validity in guiding clinical practice. This cross-sectional study appraised the methodological quality of SRs on sepsis treatments.

METHODS

We searched MEDLINE, EMBASE, and Cochrane Database for eligible SRs on randomised controlled trials on sepsis treatments with at least one meta-analysis published between 2018 and 2023. We extracted SRs' bibliographical characteristics with a pre-designed form and appraised their methodological quality using AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2. We applied logistic regressions to explore associations between bibliographical characteristics and methodological quality ratings.

RESULTS

Among the 102 SRs, two (2.0%) had high overall quality, while respectively four (3.9%), seven (6.9%) and 89 (87.3%) were of moderate, low, and critically low quality. Performance in several critical methodological domains was poor, with only 32 (31.4%) considering the risk of bias in primary studies in result interpretation, 22 (21.6%) explaining excluded primary studies, and 16 (15.7%) applying comprehensive searching strategies. SRs published in higher impact factor journals (adjusted odds ratio: 1.19; 95% confidence interval: 1.05 to 1.36) was associated with higher methodological quality.

CONCLUSIONS

The methodological quality of recent SRs on sepsis treatments is unsatisfactory. Future reviewers should address the above critical methodological aspects. More resources should also be allocated to support continuous training in critical appraisal among healthcare professionals and other evidence users.

Methodological assessment and overall confidence in the results of systematic reviews with meta-analyses focusing on traumatic dental injuries: A cross-sectional study

BACKGROUND/AIMS

High methodological quality is required to interpret results of systematic reviews (SRs) in a reliable and accurate manner. The primary aim of this study was to appraise the methodologic quality of SRs with meta-analysis within the field of traumatic dental injuries using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 tool and assess overall confidence in their results. A secondary aim was to identify potential predictive factors associated with methodological quality.

MATERIALS AND METHODS

SRs with meta-analyses published in English in the field of traumatic dental injuries from inception to March 2023 were identified. The methodological quality of the included reviews was assessed using the AMSTAR 2 checklist. Two independent evaluators scored each AMSTAR 2 item as "yes" if it was adequately addressed, "partial yes" if it was partially addressed, and "no" if it was not addressed. The overall confidence in the results of each review was classified as "High," "Moderate," "Low," or "Critically low." Using multiple regression, the relationship between five predictor variables (journal impact factor, year of publication, number of authors, journal adherence to Preferred Reporting Items for Systematic reviews and Meta-analyses [PRISMA] guidelines and a priori protocol registration) and the total AMSTAR 2 scores was analyzed. The p-value was 5%.

RESULTS

Forty-one SRs were included. The overall confidence in the results of 13 reviews was categorized as "Critically low," 18 as "Low," 3 as "Moderate" and 7 as "High." Among the five predictor variables analyzed statistically, impact factor of the journal and year of publication significantly influenced the total AMSTAR 2 scores. The number of authors, adherence to PRISMA guidelines, and a priori protocol registration had no significant impact on AMSTAR 2 scores.

CONCLUSION

The overall confidence in the results of SRs with meta-analysis within the field of traumatic dental injuries was "Low" or "Critically Low" in the vast majority of studies (31 of 41). SRs with meta-analyses published in journals with higher impact factors and more recent publications had significantly higher methodological quality.

Methodological quality of systematic reviews on treatments for Alzheimer's disease: a cross-sectional study

BACKGROUND

Carefully conducted systematic reviews (SRs) can provide reliable evidence on the effectiveness of treatment strategies for Alzheimer's disease (AD). Nevertheless, the reliability of SR results can be limited by methodological flaws. This cross-sectional study aimed to examine the methodological quality of SRs on AD treatments, along with potentially relevant factors.

METHODS

To identify eligible SRs on AD treatments, four databases including the Cochrane Database of Systematic Reviews, MEDLINE, EMBASE, and PsycINFO were searched. The Assessing the Methodological Quality of Systematic Reviews 2 instrument was used for quality appraisal of SRs. Multivariable regression analyses were used to examine factors related to methodological quality.

RESULTS

A total of 102 SRs were appraised. Four (3.90%) SRs were considered as high quality; 14 (13.7%), 48 (47.1%), and 36 (35.3%) were as moderate, low, and critically low quality, respectively. The following significant methodological limitations were identified: only 22.5% of SRs registered protocols a priori, 6.9% discussed the rationales of chosen study designs, 21.6% gave a list of excluded studies with reasons, and 23.5% documented funding sources of primary studies. Cochrane SRs (adjusted odds ratio (AOR): 31.9, 95% confidence interval (CI): 3.81-266.9) and SRs of pharmacological treatments (AOR: 3.96, 95%CI: 1.27-12.3) were related to the higher overall methodological quality of SRs.

CONCLUSION

Methodological quality of SRs on AD treatments is unsatisfactory, especially among non-Cochrane SRs and SRs of non-pharmacological interventions. Improvement in the following methodological domains requires particular attention due to poor performance: registering and publishing protocols a priori, justifying study design selection, providing a list of excluded studies, and reporting funding sources of primary studies.

Low methodological quality of systematic reviews on acupuncture: a cross-sectional study

BACKGROUND

While well-conducted systematic reviews (SRs) can provide the best evidence on the potential effectiveness of acupuncture, limitations on the methodological rigour of SRs may impact the trustworthiness of their conclusions. This cross-sectional study aimed to evaluate the methodological quality of a representative sample of SRs on acupuncture effectiveness.

METHODS

Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE were searched for SRs focusing on the treatment effect of manual acupuncture or electro-acupuncture published during January 2018 and March 2020. Eligible SRs must contain at least one meta-analysis and be published in English language. Two independent reviewers extracted the bibliographical characteristics of the included SRs with a pre-designed questionnaire and appraised the methodological quality of the studies with the validated AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2). The associations between bibliographical characteristics and methodological quality ratings were explored using Kruskal-Wallis rank tests and Spearman's rank correlation coefficients.

RESULTS

A total of 106 SRs were appraised. Only one (0.9%) SR was of high overall methodological quality, zero (0%) was of moderate-quality, six (5.7%) and 99 (93.4%) were of low-quality and critically low-quality respectively. Among appraised SRs, only ten (9.4%) provided an a priori protocol, four (3.8%) conducted a comprehensive literature search, five (4.7%) provided a list of excluded studies, and six (5.7%) performed meta-analysis appropriately. Cochrane SRs, updated SRs, and SRs that did not search non-English databases had relatively higher overall quality.

CONCLUSIONS

Methodological quality of SRs on acupuncture is unsatisfactory. Future reviewers should improve critical methodological aspects of publishing protocols, performing comprehensive search, providing a list of excluded studies with justifications for exclusion, and conducting appropriate meta-analyses. These recommendations can be implemented via enhancing the technical competency of reviewers in SR methodology through established education approaches as well as quality gatekeeping by journal editors and reviewers. Finally, for evidence users, skills in SR critical appraisal remain to be essential as relevant evidence may not be available in pre-appraised formats.

The methodological quality of individual participant data meta-analysis on intervention effects: systematic review

OBJECTIVE

To assess the methodological quality of individual participant data (IPD) meta-analysis and to identify areas for improvement.

DESIGN

Systematic review.

DATA SOURCES

Medline, Embase, and Cochrane Database of Systematic Reviews.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Systematic reviews with IPD meta-analyses of randomised controlled trials on intervention effects published in English.

RESULTS

323 IPD meta-analyses covering 21 clinical areas and published between 1991 and 2019 were included: 270 (84%) were non-Cochrane reviews and 269 (84%) were published in journals with a high impact factor (top quarter). The IPD meta-analyses showed low compliance in using a satisfactory technique to assess the risk of bias of the included randomised controlled trials (43%, 95% confidence interval 38% to 48%), accounting for risk of bias when interpreting results (40%, 34% to 45%), providing a list of excluded studies with justifications (32%, 27% to 37%), establishing an a priori protocol (31%, 26% to 36%), prespecifying methods for assessing both the overall effects (44%, 39% to 50%) and the participant-intervention interactions (31%, 26% to 36%), assessing and considering the potential of publication bias (31%, 26% to 36%), and conducting a comprehensive literature search (19%, 15% to 23%). Up to 126 (39%) IPD meta-analyses failed to obtain IPD from 90% or more of eligible participants or trials, among which only 60 (48%) provided reasons and 21 (17%) undertook certain strategies to account for the unavailable IPD.

CONCLUSIONS

The methodological quality of IPD meta-analyses is unsatisfactory. Future IPD meta-analyses need to establish an a priori protocol with prespecified data syntheses plan, comprehensively search the literature, critically appraise included randomised controlled trials with appropriate technique, account for risk of bias during data analyses and interpretation, and account for unavailable IPD.