Methodological quality of systematic reviews on influenza vaccination

The impact of funding on the quality and interpretation of systematic reviews of mechanical thrombectomy in stroke patients

BACKGROUND

Funding may impact the quality and findings of systematic reviews (SRs). We aimed to compare the methodological quality of funded and non-funded SRs that investigated the outcomes in ischemic stroke patients undergoing mechanical thrombectomy.

METHODS

We conducted a comprehensive search strategy in different databases, including Ovid Cochrane Central Register of Controlled Trials, Ovid Embase, Ovid Medline (including epub ahead of print, in-process & other non-indexed citations), PubMed, Scopus and Web of Science Core Collection to retrieve all relevant SRs. Random sequence generation matched each funded SR with a non-funded one. A Measurement Tool to Assess Systematic Reviews (AMSTAR)-2 tool was used to assess the bias and quality of the included SRs. We also used uni- and multivariate analysis to perform our analysis, and results were expressed in odds ratio (OR) and 95% confidence interval (CI).

RESULTS

We retrieved 150 articles, which were randomized and matched into 100 SRs, including 50 funded and 50 non-funded studies. By multivariate analysis, we found that including randomized clinical trials (RCTs) (OR: 5.7; 95% CI: 1.8-17.8; p = 0.003) and reporting conflict of interests (OR: 5.2; 95 CI: 1.1-24; p = 0.036) were the only significant differences between funded and non-funded SRs. No significant differences were found regarding the overall confidence for low-quality (OR: 0.54; 95% CI: 0.09-3.2; p = 0.49) and moderate/high-quality SRs (OR: 0.17; 95% CI: 0.02-1.87; p = 0.14).

CONCLUSION

Funded studies tend to include RCTs more often and report conflict of interests with no significant impact on overall confidence.

An overview of the characteristics and methodological standards across systematic reviews with Meta-analysis of efficacy/effectiveness of influenza antiviral drugs

BACKGROUND

Influenza antiviral drugs remain controversial and it is not clear if conclusions on their efficacy/effectiveness are based on high quality systematic reviews (SRs). We systematically identified, critically appraised, and summarized the characteristics and adherence to methodological standards in SRs with meta-analysis of efficacy/effectiveness of influenza antiviral drugs for prevention and/or treatment of influenza.

METHODS

We searched MEDLINE, Embase, Scopus, CINAHL, Global Health, and CDSR for English-language SR publications up to July 2020. We summarized the characteristics, adherence to methodological standards and SR quality (AMSTAR 2).

RESULTS

From a total 3,898 citations after removal of duplicates from all identified citations, we included 24 SRs. Seventy-five percent (n = 18) were of a critically low quality, 8% (n = 2) of a low quality, 17% (n = 4) of a moderate quality, and none were of a high quality. Seventeen percent (n = 4) were industry-funded, 4% (n = 1) coauthored by industry employee(s), and 33% (n = 8) commissioned by an organization or authority. Only 33% percent (n = 8) reported protocol registration, 4% (n = 1) reported collaboration with a knowledge synthesis librarian/information specialist, and 17% (n = 4) utilized a systematic review reporting checklist.

CONCLUSIONS

The evidence suggests that SRs of efficacy/effectiveness of influenza antiviral drugs are mostly of critically low quality and do not follow current best SR practices. These findings are significant in view of the important role of SRs in decision-making and the controversies that surround the use of the influenza antiviral drugs. However, the findings should not be interpreted to mean curtailment/cessation of use of antiviral drugs for influenza.

Characteristics and methodological standards across systematic reviews with Meta-analysis of efficacy and/or effectiveness of influenza vaccines: an overview of reviews

BACKGROUND

While systematic reviews (SR) generally suggest that vaccination is an effective way to prevent influenza infection, it is not clear if these conclusions are based on high quality SR methods. As such, we systematically identified, critically appraised, and summarised the characteristics and adherence to methodological standards in SRs with meta-analysis of efficacy/effectiveness of influenza vaccines.

METHODS

We searched MEDLINE, Embase, Scopus, CINAHL, Global Health, and CDSR for English-language SR publications up to July 11, 2022. We summarised the characteristics, adherence to methodological standards and SR quality (AMSTAR 2).

RESULTS

From 11,193 retrieved citations, we included 48 publications (47 SRs). Seventy-five percent were of a critically low quality, 19% of a low quality, 2% of a moderate quality, and 4% of a high quality. Thirteen percent were industry-funded, about 13% co-authored by industry employee(s), and 4% commissioned by an organisation or authority. Only 45% percent reported protocol registration, 6% reported collaboration with a knowledge synthesis librarian/information specialist, and 60% utilised a reporting checklist (e.g. PRISMA).

CONCLUSIONS AND RELEVANCE

SRs with meta-analysis of efficacy/effectiveness of influenza vaccines are mostly of critically low quality and even the more recent reviews did not follow current best SR practices. These findings are significant in view of the controversies that surround influenza vaccines, and the use of SRs in informed decision-making. However, the findings do not justify curtailment or cessation of influenza vaccine use as vaccines continue to offer substantial net public health benefit.HighlightsWe systematically identified, critically appraised, and summarised the characteristics and adherence to methodological standards in 47 systematic reviews with meta-analysis of efficacy/effectiveness of influenza vaccines.13% of the reviews were industry-funded.About 13% of the reviews were co-authored by industry employee(s).4% of the reviews were commissioned by an organisation/authority.45% of the reviews reported protocol registration.6% of the reviews reported collaborating with a knowledge synthesis librarian/information specialist to prepare the search strategy.60% of the reviews reported using the PRISMA (or similar) checklist.75% of the reviews were judged to be of critically low quality; 19% of low quality; 2% of moderate quality; 4% of high quality.

Impact of industry sponsorship on the quality of systematic reviews of vaccines: a cross-sectional analysis of studies published from 2016 to 2019

BACKGROUND

Systematic reviews (SRs) provide the highest level of evidence and inform evidence-based decision making in health care. Earlier studies found association with industry to be negatively associated with methodological quality of SRs. However, this has not been investigated in SRs on vaccines.

METHODS

We performed a systematic literature search using MEDLINE and EMBASE in March 2020. The results were restricted to those published between 2016 and 2019 with no language restrictions. Study characteristics were extracted by one person and checked by an experienced reviewer. The methodological quality of the SRs was assessed with the AMSTAR 2 tool by multiple reviewers after a calibration exercise was performed. A summary score for each SR was calculated. The Mann-Whitney U test and Fisher's exact test were performed to compare both groups.

RESULTS

Out of 185 SRs that met all inclusion criteria, 27 SRs were industry funded. Those were matched with 30 non-industry funded SRs resulting in a total sample size of 57. The mean AMSTAR 2 summary score across all SRs was 0.49. Overall, the median AMSTAR 2 summary score was higher for the non-industry funded SRs than for the industry-funded SRs (0.62 vs. 0.36; p < .00001). Lower ratings for industry funded SRs were consistent across all but one AMSTAR 2 item, though significantly lower only for three specific items.

CONCLUSION

The methodological quality of SRs in vaccination is comparable to SRs in other fields, while it is still suboptimal. We are not able to provide a satisfactory explanation why industry funded SRs had a lower methodological quality than non-industry funded SRs over recent years. Industry funding is an important indicator of methodological quality for vaccine SRs and should be carefully considered when appraising SR quality.

Predictors of Higher Quality of Systematic Reviews Addressing Nutrition and Cancer Prevention

Systematic reviews/meta-analyses (SR/MAs) are considered a reliable source of information in healthcare. We aimed to explore the association of several characteristics of SR/MAs addressing nutrition in cancer prevention and their quality/risk of bias (using assessments from AMSTAR-2 and ROBIS tools). The analysis included 101 SR/MAs identified in a systematic survey. Associations of each specified characteristic (e.g., information about the protocol, publication year, reported use of GRADE, or other methods for assessing overall certainty of evidence) with the number of AMSTAR-2 not met (‘No’ responses) and the number of ROBIS items met (‘Probably Yes’ or “Yes’ responses) were examined. Poisson regression was used to identify predictors of the number of ‘No’ answers (indicating lower quality) for all AMSTAR-2 items and the number of ‘Yes’ or ‘Probably Yes’ answers (indicating higher quality/lower concern for bias) for all ROBIS items. Logistic regression was used to identify variables associated with at least one domain assessed as ‘low concern for bias’ in the ROBIS tool. In multivariable analysis, SR/MAs not reporting use of any quality/risk of bias assessment instrument for primary studies were associated with a higher number of ‘No’ answers for all AMSTAR-2 items (incidence rate ratio (IRR) 1.26, 95% confidence interval (CI) 1.09–1.45), and a lower number of ‘Yes’ or ‘Probably Yes’ answers for all ROBIS items (IRR 0.76, 95% CI 0.66–0.87). Providing information about the protocol and search for unpublished studies was associated with a lower number of ‘No’ answers (IRR 0.73, 95% CI 0.56–0.97 and IRR 0.75, 95% CI 0.59–0.95, respectively) and a higher number of ‘Yes’ or ‘Probably Yes’ answers (IRR 1.43, 95% CI 1.17–1.74 and IRR 1.28, 95% CI 1.07–1.52, respectively). Not using at least one quality/risk of bias assessment tool for primary studies within an SR/MA was associated with lower odds that a study would be assessed as ‘low concern for bias’ in at least one ROBIS domain (odds ratio 0.061, 95% CI 0.007–0.527). Adherence to methodological standards in the development of SR/MAs was associated with a higher overall quality of SR/MAs addressing nutrition for cancer prevention.

Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study

Background

Healthcare providers need reliable evidence for supporting the adoption of new interventions, of which the source of evidence often originates from systematic reviews (SRs). However, little assessment on the rigor of SRs related to osteoarthritis interventions has been conducted. This cross-sectional study aimed to evaluate the methodological quality and predictors among SRs on osteoarthritis interventions.

Methods

Four electronic databases (Cochrane Database of Systematic Reviews, MEDLINE, Embase, and PsycINFO) were searched, from 1 January 2008 to 10 October 2019. An SR was eligible if it focused on osteoarthritis interventions, and we performed at least one meta-analysis. Methodological quality was assessed using the validated AMSTAR 2 instrument. Multivariate regression analyses were conducted to assess predictors of methodological quality.

Results

In total, 167 SRs were included. The most SRs were non-Cochrane reviews (88.6%), and 54.5% investigated non-pharmacological interventions. Only seven (4.2%) had high methodological quality. Respectively, eight (4.8%), 25 (15.0%), and 127 (76.0%) SRs had moderate, low, and critically low quality. Main methodological weaknesses were as follows: only 16.8% registered protocol a priori, 4.2% searched literature comprehensively, 25.7% included lists of excluded studies with justifications, and 30.5% assessed risk of bias appropriately by considering allocation concealment, blinding of patients and assessors, random sequence generation and selective reported outcomes. Cochrane reviews [adjusted odds ratio (AOR) 251.5, 95% confidence interval (CI) 35.5-1782.6], being updates of previous SRs (AOR 3.9, 95% CI 1.1-13.7), and SRs published after 2017 (AOR 7.7, 95% CI 2.8-21.5) were positively related to higher methodological quality.

Conclusion

Despite signs of improvement in recent years, most of the SRs on osteoarthritis interventions have critically low methodological quality, especially among non-Cochrane reviews. Future SRs should be improved by conducting comprehensive literature search, justifying excluded studies, publishing a protocol, and assessing the risk of bias of included studies appropriately.

Current Knowledge of Vaccinations in Chronic Kidney Disease Patients

Chronic kidney disease (CKD) patients are at high risk for infectious complications. This is partly due to their dysfunctional immune system, especially in advanced CKD stages. Vaccination represents an important prevention strategy in these patients, as several studies have reported lower infection rates and significantly reduced morbidity and mortality in hospitals adopting vaccination protocols. However, vaccination rates are particularly low in these patients, and the diminished immune responsiveness remains the main issue of vaccination in CKD patients. Besides, there are various immunization protocols across the world in the absence of optimal vaccination policies. This paper aims to discuss the current knowledge of vaccination in this immunocompromised group of patients based on recent evidence and recommendations.

Hypothyroidism and related diseases: a methodological quality assessment of meta-analysis

OBJECTIVES

There is an increasing prevalence of hypothyroidism and there is a growing body of meta-analyses (MAs) on the association between hypothyroidism and other diseases. However, the methodological quality of the MAs significantly varies. Thus, this study aimed to evaluate and summarise data on the methodological quality of MAs on the associations between hypothyroidism and other diseases using the Assessment of Multiple Systematic Reviews (AMSTAR) scale, providing suggestions for clinical decision-making processes.

DESIGN

To assess the methodological quality of MAs using the AMSTAR scale.

DATA SOURCES

A systematic literature search was performed in PubMed, EMBASE, the Cochrane Library, web of science and Chinese Biomedicine Literature Database.

ELIGIBILITY CRITERIA

We included MAs that had assessed the association between hypothyroidism and other diseases in humans and that had full texts regardless of the publication status. No restriction applied on language or date.

DATA EXTRACTION AND SYNTHESIS

Two reviewers independently screened the titles and abstracts of all searched literature to acquire potentially eligible publications. The full texts of possible eligible publications were downloaded and assessed. Inconsistent comments were resolved through discussions with a third reviewer.

RESULTS

52 studies were included. The average AMSTAR score of the included articles was 8.6 (range: 5-10), and those of English and Chinese MAs were 8.8 and 7.0, respectively. A total of 52 MAs were evaluated, and 19 (36.5%) and 33 (63.5%) of these MAs were of moderate and high quality, respectively. None of the MAs were of low quality. Only two MAs had an a priori design. Items 3, 5 and 9 had the highest compliance (50/52, 96.2%), and aside from item 1, items 7 and 8 had the lowest compliance (33/52,63.5%). According to the results of these MAs, hypothyroidism was significantly associated with cardiovascular diseases, metabolic diseases, neuropsychiatric disorders, breast cancer and pregnancy outcome.

CONCLUSIONS

The methodological quality of the included MAs on the association between hypothyroidism and other diseases was moderate to high. MAs with high qualities confirmed that hypothyroidism was significantly associated with cardiovascular diseases, metabolic syndrome, preterm birth and neonatal outcomes. Consideration of scientific quality when formulating conclusions should be made explicit and more attention should be paid to improving the methodological quality of MAs, and increasing their applicability for clinical decision-making.

Methodological quality of systematic reviews on treatments for depression: a cross-sectional study

AIMS

Depression is one of the most common mental disorders and identifying effective treatment strategies is crucial for the control of depression. Well-conducted systematic reviews (SRs) and meta-analyses can provide the best evidence for supporting treatment decision-making. Nevertheless, the trustworthiness of conclusions can be limited by lack of methodological rigour. This study aims to assess the methodological quality of a representative sample of SRs on depression treatments.

METHODS

A cross-sectional study on the bibliographical and methodological characteristics of SRs published on depression treatments trials was conducted. Two electronic databases (the Cochrane Database of Systematic Reviews and the Database of Abstracts of Reviews of Effects) were searched for potential SRs. SRs with at least one meta-analysis on the effects of depression treatments were considered eligible. The methodological quality of included SRs was assessed using the validated AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using logistic regression analysis.

RESULTS

A total of 358 SRs were included and appraised. Over half of included SRs (n = 195) focused on non-pharmacological treatments and harms were reported in 45.5% (n = 163) of all studies. Studies varied in methods and reporting practices: only 112 (31.3%) took the risk of bias among primary studies into account when formulating conclusions; 245 (68.4%) did not fully declare conflict of interests; 93 (26.0%) reported an 'a priori' design and 104 (29.1%) provided lists of both included and excluded studies. Results from regression analyses showed: more recent publications were more likely to report 'a priori' designs [adjusted odds ratio (AOR) 1.31, 95% confidence interval (CI) 1.09-1.57], to describe study characteristics fully (AOR 1.16, 95% CI 1.06-1.28), and to assess presence of publication bias (AOR 1.13, 95% CI 1.06-1.19), but were less likely to list both included and excluded studies (AOR 0.86, 95% CI 0.81-0.92). SRs published in journals with higher impact factor (AOR 1.14, 95% CI 1.04-1.25), completed by more review authors (AOR 1.12, 95% CI 1.01-1.24) and SRs on non-pharmacological treatments (AOR 1.62, 95% CI 1.01-2.59) were associated with better performance in publication bias assessment.

CONCLUSION

The methodological quality of included SRs is disappointing. Future SRs should strive to improve rigour by considering of risk of bias when formulating conclusions, reporting conflict of interests and authors should explicitly describe harms. SR authors should also use appropriate methods to combine the results, prevent language and publication biases, and ensure timely updates.

Methodological quality of meta-analyses of single-case experimental studies

BACKGROUND

Methodological rigor is a fundamental factor in the validity and credibility of the results of a meta-analysis.

AIM

Following an increasing interest in single-case experimental design (SCED) meta-analyses, the current study investigates the methodological quality of SCED meta-analyses.

METHODS AND PROCEDURES

We assessed the methodological quality of 178 SCED meta-analyses published between 1985 and 2015 through the modified Revised-Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist.

OUTCOMES AND RESULTS

The main finding of the current review is that the methodological quality of the SCED meta-analyses has increased over time, but is still low according to the R-AMSTAR checklist. A remarkable percentage of the studies (93.80% of the included SCED meta-analyses) did not even reach the midpoint score (22, on a scale of 0-44). The mean and median methodological quality scores were 15.57 and 16, respectively. Relatively high scores were observed for "providing the characteristics of the included studies" and "doing comprehensive literature search". The key areas of deficiency were "reporting an assessment of the likelihood of publication bias" and "using the methods appropriately to combine the findings of studies".

CONCLUSIONS AND IMPLICATIONS

Although the results of the current review reveal that the methodological quality of the SCED meta-analyses has increased over time, still more efforts are needed to improve their methodological quality.

Vaccination practices in dialysis patients: A narrative review

In a period of turmoil concerning vaccination practices, there is a serious conflict between scientifically reasonable, evidence-based guidelines and the far-fetched rumors or misconceptions concerning the vaccination practices in the general population. When a significant portion of the medical and paramedical personnel may be deliberately unvaccinated against common biological agents, achieving effective vaccination rates in the dialysis population may be complicated. Vaccination rates are unacceptably low in dialysis patients and seroconversion rates are even lower; further, serological follow-up is generally poor. The particularly anergic immune system of the advanced chronic kidney disease patients is partly a cause of both high rates of infection and low rates of seroconversions. This narrative review is an effort to summarize current knowledge concerning the vaccination practices in dialysis patients with some specific recommendations based on these facts. Of particular interest is a new vaccine, the Zoster Recombinant, Adjuvanted Vaccine (Shingrix), which we will include in our discussion.

Quality ratings of reviews in overviews: a comparison of reviews with and without dual (co-)authorship

BACKGROUND

Previous research shows that many authors of Cochrane overviews were also involved in some of the included systematic reviews (SRs). This type of dual (co-)authorship (DCA) may be a conflict of interest and a potential source of bias. Our objectives were to (1) additionally investigate DCA in non-Cochrane overviews; (2) investigate whether there is an association between DCA and quality assessments of SRs in Cochrane and non-Cochrane overviews.

METHODS

We selected a sample of Cochrane (n = 20) and non-Cochrane (n = 78) overviews for analysis. We extracted data on the number of reviews affected by DCA and whether quality assessment of included reviews was conducted independently. Differences in mean quality scores between SRs with and without DCA were calculated in each overview. These differences were standardized (using the standardized mean difference (SMD)) and meta-analyzed using a random effects model.

RESULTS

Forty out of 78 non-Cochrane overviews (51%) and 18 out of 20 Cochrane overviews (90%) had included at least one SR with DCA. For Cochrane overviews, a median of 5 [interquartile range (IQR) 2.5 to 7] SRs were affected by DCA (median of included reviews 10). For non-Cochrane overviews a median of 1 [IQR 0 to 2] of the included SRs were affected (median of included reviews 14). The meta-analysis showed a SMD of 0.58 (95% confidence interval (CI) 0.27 to 0.90) indicating higher quality scores in reviews with overlapping authors. The test for subgroup differences shows no evidence of a difference between Cochrane (SMD 0.44; 95% CI 0.07 to 0.81) and non-Cochrane overviews (SMD 0.62; 95% CI 0.06 to 1.17).

CONCLUSIONS

Many authors of overviews also often have an authorship on one or more of the underlying reviews. Our analysis shows that, on average, authors of overviews give higher quality ratings to SRs in which they were involved themselves than to other SRs. Conflict of interest is one explanation, but there are several others such as reviewer expertise. Independent and blinded reassessments of the reviews would provide more robust evidence on potential bias arising from DCA.

Impact of heterogeneity and effect size on the estimation of the optimal information size: analysis of recently published meta-analyses

OBJECTIVE

To estimate the proportion of systematic reviews that meet the optimal information size (OIS) and assess the impact heterogeneity and effect size have on the OIS estimate by type of outcome (eg, mortality, semiobjective or subjective).

METHODS

We carried out searches of Medline and Cochrane to retrieve meta-analyses published in systematic reviews from 2010 to 2012. We estimated the OIS using Trial Sequential Analysis software (TSA V.0.9) and based on several heterogeneity and effect size scenarios, stratifying by type of outcome (mortality/semiobjective/subjective) and by Cochrane/non-Cochrane reviews.

RESULTS

We included 137 meta-analyses out of 218 (63%) potential systematic reviews (one meta-analysis from each systematic review). Of these reviews, 83 (61%) were Cochrane and 54 (39%) non-Cochrane. The Cochrane reviews included a mean of 6.5 (SD 6.1) studies and the non-Cochrane included a mean of 13.2 (SD 10.2) studies. The mean number of patients was 2619.1 (SD 6245.8 or median 586.0) for the Cochrane and 19 888.5 (SD 32 925.7 or median 6566.5) patients for the non-Cochrane reviews. The percentage of systematic reviews that achieved the OIS for all-cause mortality outcome were 0% Cochrane and 25% for non-Cochrane reviews; for semiobjective outcome 17% for Cochrane and 46% for non-Cochrane reviews and for subjective outcome 45% for Cochrane and 72% for non-Cochrane reviews.

CONCLUSIONS

The number of systematic reviews that meet an optimal information size is low and varies depending on the type of outcome and the type of publication. Less than half of primary outcomes synthesised in systematic reviews achieve the OIS, and therefore the conclusions are subject to substantial uncertainty.

"Assessing the methodological quality of systematic reviews in radiation oncology: A systematic review"

OBJECTIVE

The objective of our study was to evaluate the methodological quality of systematic reviews and meta-analyses in Radiation Oncology.

METHODS

A systematic literature search was conducted for all eligible systematic reviews and meta-analyses in Radiation Oncology from 1966 to 2015. Methodological characteristics were abstracted from all works that satisfied the inclusion criteria and quality was assessed using the critical appraisal tool, AMSTAR. Regression analyses were performed to determine factors associated with a higher score of quality.

RESULTS

Following exclusion based on a priori criteria, 410 studies (157 systematic reviews and 253 meta-analyses) satisfied the inclusion criteria. Meta-analyses were found to be of fair to good quality while systematic reviews were found to be of less than fair quality. Factors associated with higher scores of quality in the multivariable analysis were including primary studies consisting of randomized control trials, performing a meta-analysis, and applying a recommended guideline related to establishing a systematic review protocol and/or reporting.

CONCLUSIONS

Systematic reviews and meta-analyses may introduce a high risk of bias if applied to inform decision-making based on AMSTAR. We recommend that decision-makers in Radiation Oncology scrutinize the methodological quality of systematic reviews and meta-analyses prior to assessing their utility to inform evidence-based medicine and researchers adhere to methodological standards outlined in validated guidelines when embarking on a systematic review.

Preterm birth: the role of knowledge transfer and exchange

BACKGROUND

Preterm birth (PTB) is the leading cause of death in children under age five. Healthcare policy and other decision-making relevant to PTB may rely on obsolete, incomplete or inapplicable research evidence, leading to worsened outcomes. Appropriate knowledge transfer and exchange (KTE) strategies are an important component of efforts to reduce the global PTB burden. We sought to develop a 'landscape' analysis of KTE strategies currently used in PTB and related contexts, and to make recommendations for optimising programmatic implementation and for future research.

METHODS

In the University of California, San Francisco's Preterm Birth Initiative, we convened a multidisciplinary working group and examined KTE frameworks. After selecting a widely-used, adaptable, theoretically-strong framework we reviewed the literature to identify evidence-based KTE strategies. We analysed KTE approaches focusing on key PTB stakeholders (individuals, families and communities, healthcare providers and policymakers). Guided by the framework, we articulated KTE approaches that would likely improve PTB outcomes. We further applied the KTE framework in developing recommendations.

RESULTS

We selected the Linking Research to Action framework. Searches identified 19 systematic reviews, including two 'reviews of reviews'. Twelve reviews provided evidence for KTE strategies in the context of maternal, neonatal and child health, though not PTB specifically; seven reviews provided 'cross-cutting' evidence that could likely be generalised to PTB contexts. For individuals, families and communities, potentially effective KTE strategies include community-based approaches, 'decision aids', regular discussions with providers and other strategies. For providers, KTE outcomes may be improved through local opinion leaders, electronic reminders, multifaceted strategies and other approaches. Policy decisions relevant to PTB may best be informed through the use of evidence briefs, deliberative dialogues, the SUPPORT tools for evidence-informed policymaking and other strategies. Our recommendations for research addressed knowledge gaps in regard to partner engagement, applicability and context, implementation strategy research, monitoring and evaluation, and infrastructure for sustainable KTE efforts.

CONCLUSIONS

Evidence-based KTE, using strategies appropriate to each stakeholder group, is essential to any effort to improve health at the population level. PTB stakeholders should be fully engaged in KTE and programme planning from its earliest stages, and ideally before planning begins.

Systematic review adherence to methodological or reporting quality

BACKGROUND

Guidelines for assessing methodological and reporting quality of systematic reviews (SRs) were developed to contribute to implementing evidence-based health care and the reduction of research waste. As SRs assessing a cohort of SRs is becoming more prevalent in the literature and with the increased uptake of SR evidence for decision-making, methodological quality and standard of reporting of SRs is of interest. The objective of this study is to evaluate SR adherence to the Quality of Reporting of Meta-analyses (QUOROM) and PRISMA reporting guidelines and the A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Overview Quality Assessment Questionnaire (OQAQ) quality assessment tools as evaluated in methodological overviews.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® databases were searched from January 1990 to October 2014. Title and abstract screening and full-text screening were conducted independently by two reviewers. Reports assessing the quality or reporting of a cohort of SRs of interventions using PRISMA, QUOROM, OQAQ, or AMSTAR were included. All results are reported as frequencies and percentages of reports and SRs respectively.

RESULTS

Of the 20,765 independent records retrieved from electronic searching, 1189 reports were reviewed for eligibility at full text, of which 56 reports (5371 SRs in total) evaluating the PRISMA, QUOROM, AMSTAR, and/or OQAQ tools were included. Notable items include the following: of the SRs using PRISMA, over 85% (1532/1741) provided a rationale for the review and less than 6% (102/1741) provided protocol information. For reports using QUOROM, only 9% (40/449) of SRs provided a trial flow diagram. However, 90% (402/449) described the explicit clinical problem and review rationale in the introduction section. Of reports using AMSTAR, 30% (534/1794) used duplicate study selection and data extraction. Conversely, 80% (1439/1794) of SRs provided study characteristics of included studies. In terms of OQAQ, 37% (499/1367) of the SRs assessed risk of bias (validity) in the included studies, while 80% (1112/1387) reported the criteria for study selection.

CONCLUSIONS

Although reporting guidelines and quality assessment tools exist, reporting and methodological quality of SRs are inconsistent. Mechanisms to improve adherence to established reporting guidelines and methodological assessment tools are needed to improve the quality of SRs.

Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study

BACKGROUND

The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to 'quality' assessment over time.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports.

RESULTS

Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ.

CONCLUSIONS

The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting.

Assessing the State of Knowledge Regarding the Effectiveness of Interventions to Contain Pandemic Influenza Transmission: A Systematic Review and Narrative Synthesis

BACKGROUND

Influenza pandemics occur when a novel influenza strain, to which humans are immunologically naïve, emerges to cause infection and illness on a global scale. Differences in the viral properties of pandemic strains, relative to seasonal ones, can alter the effectiveness of interventions typically implemented to control seasonal influenza burden. As a result, annual control activities may not be sufficient to contain an influenza pandemic.

PURPOSE

This study seeks to inform pandemic policy and planning initiatives by reviewing the effectiveness of previous interventions to reduce pandemic influenza transmission and infection. Results will inform the planning and design of more focused in-depth systematic reviews for specific types of interventions, thus providing the most comprehensive and current understanding of the potential for alternative interventions to mitigate the burden of pandemic influenza.

METHODS

A systematic review and narrative synthesis of existing systematic reviews and meta-analyses examining intervention effectiveness in containing pandemic influenza transmission was conducted using information collected from five databases (PubMed, Medline, Cochrane, Embase, and Cinahl/EBSCO). Two independent reviewers conducted study screening and quality assessment, extracting data related to intervention impact and effectiveness.

RESULTS AND DISCUSSION

Most included reviews were of moderate to high quality. Although the degree of statistical heterogeneity precluded meta-analysis, the present systematic review examines the wide variety of interventions that can impact influenza transmission in different ways. While it appears that pandemic influenza vaccination provides significant protection against infection, there was insufficient evidence to conclude that antiviral prophylaxis, seasonal influenza cross-protection, or a range of non-pharmaceutical strategies would provide appreciable protection when implemented in isolation. It is likely that an optimal intervention strategy will employ a combination of interventions in a layered approach, though more research is needed to substantiate this proposition.

TRIAL REGISTRATION

PROSPERO 42016039803.

A critical appraisal of the methodology and quality of evidence of systematic reviews and meta-analyses of traditional Chinese medical nursing interventions: a systematic review of reviews

OBJECTIVE

To assess the methodology and quality of evidence of systematic reviews and meta-analyses of traditional Chinese medical nursing (TCMN) interventions in Chinese journals. These interventions include acupressure, massage, Tai Chi, Qi Gong, electroacupuncture and use of Chinese herbal medicines-for example, in enemas, foot massage and compressing the umbilicus.

DESIGN

A systematic literature search for systematic reviews and meta-analyses of TCMN interventions was performed. Review characteristics were extracted. The methodological quality and the quality of the evidence were evaluated using the Assessment of Multiple Systematic Reviews (AMSTAR) and Grading of Recommendations Assessment, Development and Evaluation (GRADE) approaches.

RESULT

We included 20 systematic reviews and meta-analyses, and a total of 11 TCMN interventions were assessed in the 20 reviews. The compliance with AMSTAR checklist items ranged from 4.5 to 8 and systematic reviews/meta-analyses were, on average, of medium methodological quality. The quality of the evidence we assessed ranged from very low to moderate; no high-quality evidence was found. The top two causes for downrating confidence in effect estimates among the 31 bodies of evidence assessed were the risk of bias and inconsistency.

CONCLUSIONS

There is room for improvement in the methodological quality of systematic reviews/meta-analyses of TCMN interventions published in Chinese journals. Greater efforts should be devoted to ensuring a more comprehensive search strategy, clearer specification of the interventions of interest in the eligibility criteria and identification of meaningful outcomes for clinicians and patients (consumers). The overall quality of evidence among reviews remains suboptimal, which raise concerns about their roles in influencing clinical practice. Thus, the conclusions in reviews we assessed must be treated with caution and their roles in influencing clinical practice should be limited. A critical appraisal of systematic reviews/meta-analyses of TCMN interventions is particularly important to provide sound guidance for TCMN.

Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement

A Measurement Tool to Assess Systematic Reviews (AMSTAR) is a commonly used tool to assess the quality of systematic reviews; however, modifications are needed to improve its usability, reliability, and validity. In this commentary, we summarize our experience and the experiences of others who have used AMSTAR and provide suggestions for its improvement. We propose that AMSTAR should modify a number of individual items and their instructions and responses to make them more congruent with an assessment of the methodologic quality of systematic reviews. We recommend adding new items and modifying existing items to assess the quality of the body of evidence and to address subgroup and sensitivity analyses. More detailed instructions are needed for scoring individual items across multiple reviewers, and we recommend that a total score should not be calculated. These suggestions need to be empirically tested prior to implementation.

Systematic Reviews in Sports Medicine

BACKGROUND

The number of systematic reviews published in the orthopaedic literature has increased, and these reviews can help guide clinical decision making. However, the quality of these reviews can affect the reader's ability to use the data to arrive at accurate conclusions and make clinical decisions.

PURPOSE

To evaluate the methodological and reporting quality of systematic reviews and meta-analyses in the sports medicine literature to determine whether such reviews should be used to guide treatment decisions. The hypothesis was that many systematic reviews in the orthopaedic sports medicine literature may not follow the appropriate reporting guidelines or methodological criteria recommended for systematic reviews.

STUDY DESIGN

Systematic review.

METHODS

All clinical sports medicine systematic reviews and meta-analyses from 2009 to 2013 published in The American Journal of Sports Medicine (AJSM), The Journal of Bone and Joint Surgery (JBJS), Arthroscopy, Sports Health, and Knee Surgery, Sports Traumatology, Arthroscopy (KSSTA) were reviewed and evaluated for level of evidence according to the guidelines from the Oxford Centre for Evidence-Based Medicine, for reporting quality according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, and for methodological quality according to the Assessment of Multiple Systematic Reviews (AMSTAR) tool. Analysis was performed by year and journal of publication, and the levels of evidence included in the systematic reviews were also analyzed.

RESULTS

A total of 200 systematic reviews and meta-analyses were identified over the study period. Of these, 53% included evidence levels 4 and 5 in their analyses, with just 32% including evidence levels 1 and 2 only. There were significant differences in the proportion of articles with high levels of evidence (P < .001) and low levels of evidence (P = .005) by journal. The average PRISMA score was 87% and the average AMSTAR score was 73% among all journals. The average AMSTAR and PRISMA scores were significantly different by journal (P = .002 and .001, respectively) and by year (P = .046 and .019, respectively). Arthroscopy, AJSM, and JBJS all scored higher than Sports Health and KSSTA on the PRISMA and AMSTAR. The average PRISMA score by year varied from 85% to 89%, and the average AMSTAR score varied from 70% to 76%.

CONCLUSION

Systematic reviews and meta-analyses in orthopaedics sports medicine literature relied on evidence levels 4 and 5 in 53% of studies over the 5-year study period. Overall, PRISMA and AMSTAR scores are high and may be better than those in other disciplines. Readers need to be conscious of potential shortcomings when reading systematic reviews and using them in practice.

Methodological quality assessment of paper-based systematic reviews published in oral health

OBJECTIVES

This study aimed to conduct a methodological assessment of paper-based systematic reviews (SR) published in oral health using a validated checklist. A secondary objective was to explore temporal trends on methodological quality.

MATERIAL AND METHODS

Two electronic databases (OVID Medline and OVID EMBASE) were searched for paper-based SR of interventions published in oral health from inception to October 2014. Manual searches of the reference lists of paper-based SR were also conducted. Methodological quality of included paper-based SR was assessed using an 11-item questionnaire, Assessment of Multiple Systematic Reviews (AMSTAR) checklist. Methodological quality was summarized using the median and inter-quartile range (IQR) of the AMSTAR score over different categories and time periods.

RESULTS

A total of 643 paper-based SR were included. The overall median AMSTAR score was 4 (IQR 2-6). The highest median score (5) was found in the pain dentistry and periodontology fields, while the lowest median score (3) was found in implant dentistry, restorative dentistry, oral medicine, and prosthodontics. The number of paper-based SR per year and the median AMSTAR score increased over time (median score in 1990s was 2 (IQR 2-3), 2000s was 4 (IQR 2-5), and 2010 onwards was 5 (IQR 3-6)).

CONCLUSION

Although the methodological quality of paper-based SR published in oral health has improved in the last few years, there is still scope for improving quality in most evaluated dental specialties.

CLINICAL RELEVANCE

Large-scale assessment of methodological quality of dental SR highlights areas of methodological strengths and weaknesses that can be targeted in future publications to encourage better quality review methodology.

Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews

Objectives

To assess discrepancies in the analyzed outcomes between protocols and published reviews within Cochrane oral health systematic reviews (COHG) on the Cochrane Database of Systematic Reviews (CDSR).

Study Design and Setting

All COHG systematic reviews on the CDSR and the corresponding protocols were retrieved in November 2014 and information on the reported outcomes was recorded. Data was collected at the systematic review level by two reviewers independently.

Results

One hundred and fifty two reviews were included. In relation to primary outcomes, 11.2% were downgraded to secondary outcomes, 9.9% were omitted altogether in the final publication and new primary outcomes were identified in 18.4% of publications. For secondary outcomes, 2% were upgraded to primary, 12.5% were omitted and 30.9% were newly introduced in the publication. Overall, 45.4% of reviews had at least one discrepancy when compared to the protocol; these were reported in 14.5% reviews. The number of review updates appears to be associated with discrepancies between final review and protocol (OR: 3.18, 95% CI: 1.77, 5.74, p<0.001). The risk of reporting significant results was lower for both downgraded outcomes [RR: 0.52, 95% CI: 0.17, 1.58, p = 0.24] and upgraded or newly introduced outcomes [RR: 0.77, 95% CI: 0.36, 1.64, p = 0.50] compared to outcomes with no discrepancies. The risk of reporting significant results was higher for upgraded or newly introduced outcomes compared to downgraded outcomes (RR = 1.19, 95% CI: 0.65, 2.16, p = 0.57). None of the comparisons reached statistical significance.

Conclusion

While no evidence of selective outcome reporting was found in this study, based on the present analysis of SRs published within COHG systematic reviews, discrepancies between outcomes in pre-published protocols and final reviews continue to be common. Solutions such as the use of standardized outcomes to reduce the prevalence of this issue may need to be explored.

Review of the efficacy and safety of over-the-counter medicine

<p>Over-the-counter medicines are available without prescription because of their safety and effectiveness, to treat minor ailments and symptoms. The objective of the study was to analyze the availability and quality of systematic reviews published about nonprescription medicines, identifying the groups for which there are gaps in evidence. We identified published articles through the Cochrane Database of Systematic Review and MEDLINE, from the start of the database until May 2012, using the search terms "nonprescription drugs," "over the counter," and "OTC." We searched for articles that describe systematic reviews addressing the efficacy and safety of drugs dispensed without a prescription, according to the lists published by the Association of the European Self-Medication Industry and in Brazil, in the clinical conditions listed in Groups and Specified Therapeutic Indications. We included 49 articles, 18 articles were of moderate quality and 31 of high quality. Of the studies, 74.5% demonstrated efficacy in favor of the use of drugs evaluated. Of the 24 studies that evaluated safety, 21% showed evidence unfavorable to the drug. Overall, the evidence found in the studies included in the overview is favorable to the use of the drugs evaluated. However, there are gaps in evidence for some therapy groups.</p>

Influenza vaccination in patients with end-stage renal disease: systematic review and assessment of quality of evidence related to vaccine efficacy, effectiveness, and safety

BACKGROUND

Vaccination against influenza is recommended in patients with end-stage renal disease (ESRD). However, so far, no systematic review has summarized the available evidence on the effectiveness and safety of influenza vaccination in this patient group.

METHODS

We conducted a systematic review and meta-analysis and assessed the quality of evidence using the GRADE methodology. We searched MEDLINE, EMBASE, Cochrane Library databases, ClinicalTrials.gov, and reference lists for studies on efficacy, effectiveness, and/or safety of seasonal influenza vaccination in patients with ESRD receiving dialysis. All reported clinical outcomes were considered, including all-cause mortality, cardiac death, infectious death, all-cause hospitalization, hospitalization due to influenza or pneumonia, hospitalization due to bacteremia, viremia, or septicemia, hospitalization due to respiratory infection, ICU admission, and influenza-like illness.

RESULTS

Five observational studies and no randomized-controlled trial were identified. In four studies, risk of bias was high regarding all reported outcomes. Strong residual confounding was likely to be present in one study reporting on three outcomes, as indicated by significant protective effects of vaccination outside influenza seasons. Therefore, the statistically significant protective effects on all-cause mortality (vaccine effectiveness (VE), 32%; 95% CI, 24-39%), cardiac death (VE, 16%; 95% CI, 1-29%), hospitalization due to influenza or pneumonia (VE, 14%; 95% CI, 7-20%), ICU admission (VE, 81%; 95% CI, 63-86%), and influenza-like illness (VE, 12%; 95% CI, 10-14%) have to be taken with caution. According to GRADE, the quality of the body of evidence was considered very low for all outcomes. No study reported on laboratory-confirmed influenza virus infections or on safety endpoints.

CONCLUSIONS

Evidence on the protective effects of influenza vaccination in patients with ESRD is limited and of very low quality. Since VE estimates in the available literature are prone to unmeasured confounding, studies using randomization or quasi-experimental designs are needed to determine the extent by which vaccination prevents influenza and related clinical outcomes in this at-risk population. However, given the high rates of health-endangering events in these patients, even a low VE can be considered as sufficient to recommend annual influenza vaccination.

Influenza vaccination: a summary of Cochrane Reviews

The Cochrane Library was systematically searched for meta-analyses regarding influenza vaccination of various populations, both healthy and sick. An effect in reducing the number of cases of influenza, influenza-like illness or complications to influenza was found in some studies, but, generally, the quality of the studies was low, and several studies lacked hard clinical endpoints. Data on adverse effects were scarce. More randomised controlled trials investigating the effects of influenza vaccination are warranted.