Low methodological quality of systematic reviews on acupuncture: a cross-sectional study

Methodological quality of systematic reviews on atopic dermatitis treatments: a cross-sectional study

BACKGROUND

Systematic reviews (SRs) could offer the best evidence supporting interventions, but methodological flaws limit their trustworthiness in decision-making. This cross-sectional study appraised the methodological quality of SRs on atopic dermatitis (AD) treatments.

METHODS

We searched MEDLINE, EMBASE, PsycINFO, and Cochrane Database for SRs on AD treatments published in 2019-2022. We extracted SRs' bibliographical data and appraised SRs' methodological quality with AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2. We explored associations between methodological quality and bibliographical characteristics.

RESULTS

Among the 52 appraised SRs, only one (1.9%) had high methodological quality, while 45 (86.5%) critically low. For critical domains, only five (9.6%) employed comprehensive search strategy, seven (13.5%) provided list of excluded studies, 17 (32.7%) considered risk of bias in primary studies, 21 (40.4%) contained registered protocol, and 24 (46.2%) investigated publication bias. Cochrane reviews, SR updates, SRs with European corresponding authors, and SRs funded by European institutions had better overall quality. Impact factor and author number positively associated with overall quality.

CONCLUSIONS

Methodological quality of SRs on AD treatments is unsatisfactory. Future reviewers should improve the above critical methodological aspects. Resources should be devolved into upscaling evidence synthesis infrastructure and improving critical appraisal skills of evidence users.

Time points of outcome are often neglected in acupuncture meta-analyses: a methodological survey

OBJECTIVES

To systematically understand the transparency of outcome measurement time point reporting in meta-analyses of acupuncture.

STUDY DESIGN AND SETTING

We searched for meta-analyses of acupuncture published between 2013 and 2022 in PubMed, Embase, and Cochrane Library. A team of method-trained investigators screened studies for eligibility and collected data using pilot-tested standardized questionnaires. We documented in detail the reporting of outcome measurement time points in acupuncture meta-analyses.

RESULTS

A total of 224 acupuncture meta-analyses were included. Of these, 98 (43.8%) studies did not specify the time points of primary outcome. Among 126 (56.3%) meta-analyses which reported the time points of primary outcome, only 22 (17.5%) meta-analyses specified time points in corresponding protocol. Among 48 (38.1%) meta-analyses that estimated treatment effects of multiple time points, 11 (22.9%) meta-analyses used inappropriate meta-analysis method (subgroup analysis) to pool effect size, and none of the meta-analyses used advanced methods for pooling effect sizes at different time points.

CONCLUSION

Transparency in reporting outcome time points for acupuncture meta-analyses and appropriate methods to pool the effect size of multiple time points were lacking. For future systematic reviews, the transparency of outcome measurement time points should be emphasized in the protocols and final reports. Furthermore, advanced methods should be considered for pooling effect sizes at multiple time points.

Methodological quality of systematic reviews on sepsis treatments: A cross-sectional study

OBJECTIVE

Systematic reviews (SRs) offer updated evidence to support decision-making on sepsis treatments. However, the rigour of SRs may vary, and methodological flaws may limit their validity in guiding clinical practice. This cross-sectional study appraised the methodological quality of SRs on sepsis treatments.

METHODS

We searched MEDLINE, EMBASE, and Cochrane Database for eligible SRs on randomised controlled trials on sepsis treatments with at least one meta-analysis published between 2018 and 2023. We extracted SRs' bibliographical characteristics with a pre-designed form and appraised their methodological quality using AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2. We applied logistic regressions to explore associations between bibliographical characteristics and methodological quality ratings.

RESULTS

Among the 102 SRs, two (2.0%) had high overall quality, while respectively four (3.9%), seven (6.9%) and 89 (87.3%) were of moderate, low, and critically low quality. Performance in several critical methodological domains was poor, with only 32 (31.4%) considering the risk of bias in primary studies in result interpretation, 22 (21.6%) explaining excluded primary studies, and 16 (15.7%) applying comprehensive searching strategies. SRs published in higher impact factor journals (adjusted odds ratio: 1.19; 95% confidence interval: 1.05 to 1.36) was associated with higher methodological quality.

CONCLUSIONS

The methodological quality of recent SRs on sepsis treatments is unsatisfactory. Future reviewers should address the above critical methodological aspects. More resources should also be allocated to support continuous training in critical appraisal among healthcare professionals and other evidence users.

Implementation science in traditional, complementary and integrative medicine: An overview of experiences from China and the United States

BACKGROUND

The introduction of traditional, complementary and integrative medicine (TCIM) services into health systems has been advocated by the World Health Organization, but there is a paucity of reviews synthesising the experiences of (i) implementing TCIM services in conventional healthcare settings and (ii) introducing evidence-based practice in TCIM. Knowledge of the first issue will assist policymakers to innovate implementation interventions in their own health system contexts. Addressing the second issue will facilitate the closure of the evidence-practice gap in TCIM and improve the translation of research evidence into health outcome benefits.

PURPOSE

The aim of this study was to identify, describe and analyse publications on these two key TCIM policy issues via an overview from an implementation science perspective.

METHODS

Publications describing international experiences of implementing TCIM services or evidence for TCIM practices were identified by searching MEDLINE, EMBASE and Global Health databases in November 2021. The findings were summarised using a narrative synthesis approach.

RESULTS

Sixty-three relevant publications were included in the analysis. Current experiences in China and the United Sates (US) reflect varying policy priorities at different stages of implementing TCIM services. In the US, where TCIM have yet to be introduced into mainstream healthcare settings, implementation interventions were designed to facilitate the provision of specific, evidence-based TCIM modalities via referrals from conventional clinicians. The application of these strategies at the health system, regulatory, financial, community, provider and patient levels provided a comprehensive picture of how TCIM implementation may be facilitated via multi-level interventions. In China, the major form of TCIM is traditional Chinese medicine (TCM), for which service provision has already been adopted at all levels of healthcare. With the high volume of clinical research that has been generated in the past several decades, a key policy question at this stage is how to translate TCM-related clinical evidence into practice. The development of clinical practice guidelines (CPGs) is the main implementation intervention, but adherence by TCM clinicians has been poor, due to the conflict between classical individualised practice and CPG standardisation. While tailoring interventions to facilitate CPG uptake is indicated, concurrent innovations in TCM clinical research methods would improve the compatibility between classical and CPG-based practice.

CONCLUSION

Policymakers managing different stages of TCIM implementation will benefit from the experiences of practitioners in the US and China. Multi-level implementation interventions launched in the US provide ideas for the initial introduction of TCIM into a conventional medicine-dominated health system. As TCIM service provision and related clinical research become more common, China's experience will inform how clinical evidence related to TCIM may be disseminated and implemented to improve service quality.

Acupuncture for Crohn's disease: protocol for a systematic review and meta-analysis of randomized clinical trials

BACKGROUND

Crohn's disease (CD) is a chronic recurrent gastrointestinal disorder with a high incidence of inflammation due to intestinal inflammation. Acupuncture is commonly used as an alternative therapy for patients with CD. The aim of this study was to design a systematic review and meta-analysis protocol, to provide guidance for the establishment of systematic evaluation and meta-analysis on the efficacy and safety of acupuncture on CD.

METHODS

We will search PubMed, the Cochrane Library, Embase, Web of Science, and 4 Chinese databases: China National Knowledge Infrastructure (CNKI), Wanfang database, VIP database, and Chinese Biomedical Database to obtain randomized controlled trials of CD treated with acupuncture from inception to November 5, 2022. Primary outcome include CD symptoms severity and clinical efficacy, secondary outcome indicators include laboratory indicators or inflammatory markers, severity of endoscopic lesions, quality of life, and safety outcomes. We will analyze the data using RevMan V.5.4 software. Two reviewers will assess the risk of bias and study quality by the Cochrane Collaboration Risk of Bias Tool and GRADE methods, respectively.

RESULTS

This systematic review and meta-analysis protocol will analyze the efficacy, symptom improvement, quality of life, and safety of acupuncture therapy for CD.

CONCLUSION

This protocol outlines the planned scope and methodology of a forthcoming systematic review and meta-analysis to provide guidelines for a rigorous assessment of the efficacy and safety of acupuncture for the treatment of CD.

Methodological quality of systematic reviews on treatments for Alzheimer's disease: a cross-sectional study

BACKGROUND

Carefully conducted systematic reviews (SRs) can provide reliable evidence on the effectiveness of treatment strategies for Alzheimer's disease (AD). Nevertheless, the reliability of SR results can be limited by methodological flaws. This cross-sectional study aimed to examine the methodological quality of SRs on AD treatments, along with potentially relevant factors.

METHODS

To identify eligible SRs on AD treatments, four databases including the Cochrane Database of Systematic Reviews, MEDLINE, EMBASE, and PsycINFO were searched. The Assessing the Methodological Quality of Systematic Reviews 2 instrument was used for quality appraisal of SRs. Multivariable regression analyses were used to examine factors related to methodological quality.

RESULTS

A total of 102 SRs were appraised. Four (3.90%) SRs were considered as high quality; 14 (13.7%), 48 (47.1%), and 36 (35.3%) were as moderate, low, and critically low quality, respectively. The following significant methodological limitations were identified: only 22.5% of SRs registered protocols a priori, 6.9% discussed the rationales of chosen study designs, 21.6% gave a list of excluded studies with reasons, and 23.5% documented funding sources of primary studies. Cochrane SRs (adjusted odds ratio (AOR): 31.9, 95% confidence interval (CI): 3.81-266.9) and SRs of pharmacological treatments (AOR: 3.96, 95%CI: 1.27-12.3) were related to the higher overall methodological quality of SRs.

CONCLUSION

Methodological quality of SRs on AD treatments is unsatisfactory, especially among non-Cochrane SRs and SRs of non-pharmacological interventions. Improvement in the following methodological domains requires particular attention due to poor performance: registering and publishing protocols a priori, justifying study design selection, providing a list of excluded studies, and reporting funding sources of primary studies.

Methodological quality of systematic reviews on Chinese herbal medicine: a methodological survey

BACKGROUND

Systematic reviews (SRs) synthesise the best evidence of effectiveness and safety on Chinese herbal medicine (CHM). Decision-making should be supported by the high-quality evidence of prudently conducted SRs, but the trustworthiness of conclusions may be limited by poor methodological rigour.

METHODS

This survey aimed to examine the methodological quality of a representative sample of SRs on CHM published during January 2018 to March 2020. We conducted literature search in Cochrane Database of Systematic Reviews, MEDLINE via Ovid, and EMBASE via Ovid. Eligible SRs must be in Chinese or English with at least one meta-analysis on the treatment effect of any CHM documented in the 2015 Chinese Pharmacopoeia. Two reviewers extracted the bibliographical characteristics of SRs and appraised their methodological quality using AMSTAR 2 (Assessing the Methodological Quality of Systematic Reviews 2). The associations between bibliographical characteristics and methodological quality were investigated using Kruskal-Wallis tests and Spearman's rank correlation coefficients.

RESULTS

We sampled and appraised one hundred forty-eight SRs. Overall, one (0.7%) was of high methodological quality; zero (0%), four (2.7%), and one-hundred forty-three (96.6%) SRs were of moderate, low, and critically-low quality. Only thirteen SRs (8.8%) provided a pre-defined protocol; none (0%) provided justifications for including particular primary study designs; six (4.1%) conducted a comprehensive literature search; two (1.4%) provided a list of excluded studies; nine (6.1%) undertook meta-analysis with appropriate methods; and seven (4.7%) reported funding sources of included primary studies. Cochrane reviews had higher overall quality than non-Cochrane reviews (P < 0.001). SRs with European funding support were less likely to have critically-low quality when compared with their counterparts (P = 0.020). SRs conducted by more authors (r_s = 0.23; P = 0.006) and published in higher impact factor journals (r_s = 0.20; P = 0.044) were associated with higher methodological quality.

CONCLUSIONS

Our results indicated that the methodological quality of SRs on CHM is low. Future authors should enhance the methodological quality through registering a priori protocols, justifying selection of study designs, conducting comprehensive literature search, providing a list of excluded studies with rationales, using appropriate method for meta-analyses, and reporting funding sources among primary studies.

Methodological quality assessment criteria for the evaluation of laboratory-based studies included in systematic reviews within the specialty of Endodontology: a development protocol

High quality systematic reviews in the field of Dentistry provide the most definitive overarching evidence for clinicians, guideline developers and healthcare policy makers to judge the foreseeable risks, anticipated benefits, and potential harms of dental treatment. In the process of carrying out a systematic review, it is essential that authors appraise the methodological quality of the primary studies they include, because studies which follow poor methodology will have a potentially serious negative impact on the overall strength of the evidence and the recommendations that can be drawn. In Endodontology, systematic reviews of laboratory studies have used quality assessment criteria developed subjectively by the individual authors as there are no comprehensive, well-structured, and universally accepted criteria that can be used objectively and universally to individual studies included in reviews. Unfortunately, these subjective criteria are likely to be inaccurately-defined, unreliably-applied, inadequately-analysed, unreasonably-biased, defective, and non-repeatable. The aim of the present paper is to outline the process to be followed in the development of comprehensive methodological quality assessment criteria to be used when evaluating laboratory studies that should be included in systematic reviews within Endodontology. The development of new methodological quality assessment criteria for appraising the laboratory-based studies included in systematic reviews within Endodontology will follow a three-stage process. First, a steering committee will be formed by the project leaders to develop a preliminary list of assessment criteria by modifying and adapting those already available, but with the addition of several new items relevant for Endodontology. The initial draft assessment criteria will be reviewed and refined by a Delphi Group (n=40) for their relevance and inclusion using a nine-point Likert scale. Second, the agreed items will then be discussed in an online or face-to-face meeting by a group of experts (n=10) to further refine the assessment criteria. Third, based on the feedback received from the online/face-to-face meeting, the steering committee will revise the quality assessment criteria and subsequently a group of authors will be selected to pilot the new system. Based on the feedback collected, the criteria may be revised further before being approved by the steering committee. The assessment criteria will be published in relevant journals, presented at national and international congresses/meetings, and will be freely available on a dedicated website. The steering committee will update the assessment criteria periodically based on feedback received from end-users.