A systematic review evaluating the potential for bias and the methodological quality of meta-analyses in vaccinology

The impact of funding on the quality and interpretation of systematic reviews of mechanical thrombectomy in stroke patients

BACKGROUND

Funding may impact the quality and findings of systematic reviews (SRs). We aimed to compare the methodological quality of funded and non-funded SRs that investigated the outcomes in ischemic stroke patients undergoing mechanical thrombectomy.

METHODS

We conducted a comprehensive search strategy in different databases, including Ovid Cochrane Central Register of Controlled Trials, Ovid Embase, Ovid Medline (including epub ahead of print, in-process & other non-indexed citations), PubMed, Scopus and Web of Science Core Collection to retrieve all relevant SRs. Random sequence generation matched each funded SR with a non-funded one. A Measurement Tool to Assess Systematic Reviews (AMSTAR)-2 tool was used to assess the bias and quality of the included SRs. We also used uni- and multivariate analysis to perform our analysis, and results were expressed in odds ratio (OR) and 95% confidence interval (CI).

RESULTS

We retrieved 150 articles, which were randomized and matched into 100 SRs, including 50 funded and 50 non-funded studies. By multivariate analysis, we found that including randomized clinical trials (RCTs) (OR: 5.7; 95% CI: 1.8-17.8; p = 0.003) and reporting conflict of interests (OR: 5.2; 95 CI: 1.1-24; p = 0.036) were the only significant differences between funded and non-funded SRs. No significant differences were found regarding the overall confidence for low-quality (OR: 0.54; 95% CI: 0.09-3.2; p = 0.49) and moderate/high-quality SRs (OR: 0.17; 95% CI: 0.02-1.87; p = 0.14).

CONCLUSION

Funded studies tend to include RCTs more often and report conflict of interests with no significant impact on overall confidence.

An overview of the characteristics and methodological standards across systematic reviews with Meta-analysis of efficacy/effectiveness of influenza antiviral drugs

BACKGROUND

Influenza antiviral drugs remain controversial and it is not clear if conclusions on their efficacy/effectiveness are based on high quality systematic reviews (SRs). We systematically identified, critically appraised, and summarized the characteristics and adherence to methodological standards in SRs with meta-analysis of efficacy/effectiveness of influenza antiviral drugs for prevention and/or treatment of influenza.

METHODS

We searched MEDLINE, Embase, Scopus, CINAHL, Global Health, and CDSR for English-language SR publications up to July 2020. We summarized the characteristics, adherence to methodological standards and SR quality (AMSTAR 2).

RESULTS

From a total 3,898 citations after removal of duplicates from all identified citations, we included 24 SRs. Seventy-five percent (n = 18) were of a critically low quality, 8% (n = 2) of a low quality, 17% (n = 4) of a moderate quality, and none were of a high quality. Seventeen percent (n = 4) were industry-funded, 4% (n = 1) coauthored by industry employee(s), and 33% (n = 8) commissioned by an organization or authority. Only 33% percent (n = 8) reported protocol registration, 4% (n = 1) reported collaboration with a knowledge synthesis librarian/information specialist, and 17% (n = 4) utilized a systematic review reporting checklist.

CONCLUSIONS

The evidence suggests that SRs of efficacy/effectiveness of influenza antiviral drugs are mostly of critically low quality and do not follow current best SR practices. These findings are significant in view of the important role of SRs in decision-making and the controversies that surround the use of the influenza antiviral drugs. However, the findings should not be interpreted to mean curtailment/cessation of use of antiviral drugs for influenza.

Characteristics and methodological standards across systematic reviews with Meta-analysis of efficacy and/or effectiveness of influenza vaccines: an overview of reviews

BACKGROUND

While systematic reviews (SR) generally suggest that vaccination is an effective way to prevent influenza infection, it is not clear if these conclusions are based on high quality SR methods. As such, we systematically identified, critically appraised, and summarised the characteristics and adherence to methodological standards in SRs with meta-analysis of efficacy/effectiveness of influenza vaccines.

METHODS

We searched MEDLINE, Embase, Scopus, CINAHL, Global Health, and CDSR for English-language SR publications up to July 11, 2022. We summarised the characteristics, adherence to methodological standards and SR quality (AMSTAR 2).

RESULTS

From 11,193 retrieved citations, we included 48 publications (47 SRs). Seventy-five percent were of a critically low quality, 19% of a low quality, 2% of a moderate quality, and 4% of a high quality. Thirteen percent were industry-funded, about 13% co-authored by industry employee(s), and 4% commissioned by an organisation or authority. Only 45% percent reported protocol registration, 6% reported collaboration with a knowledge synthesis librarian/information specialist, and 60% utilised a reporting checklist (e.g. PRISMA).

CONCLUSIONS AND RELEVANCE

SRs with meta-analysis of efficacy/effectiveness of influenza vaccines are mostly of critically low quality and even the more recent reviews did not follow current best SR practices. These findings are significant in view of the controversies that surround influenza vaccines, and the use of SRs in informed decision-making. However, the findings do not justify curtailment or cessation of influenza vaccine use as vaccines continue to offer substantial net public health benefit.HighlightsWe systematically identified, critically appraised, and summarised the characteristics and adherence to methodological standards in 47 systematic reviews with meta-analysis of efficacy/effectiveness of influenza vaccines.13% of the reviews were industry-funded.About 13% of the reviews were co-authored by industry employee(s).4% of the reviews were commissioned by an organisation/authority.45% of the reviews reported protocol registration.6% of the reviews reported collaborating with a knowledge synthesis librarian/information specialist to prepare the search strategy.60% of the reviews reported using the PRISMA (or similar) checklist.75% of the reviews were judged to be of critically low quality; 19% of low quality; 2% of moderate quality; 4% of high quality.

Impact of industry sponsorship on the quality of systematic reviews of vaccines: a cross-sectional analysis of studies published from 2016 to 2019

BACKGROUND

Systematic reviews (SRs) provide the highest level of evidence and inform evidence-based decision making in health care. Earlier studies found association with industry to be negatively associated with methodological quality of SRs. However, this has not been investigated in SRs on vaccines.

METHODS

We performed a systematic literature search using MEDLINE and EMBASE in March 2020. The results were restricted to those published between 2016 and 2019 with no language restrictions. Study characteristics were extracted by one person and checked by an experienced reviewer. The methodological quality of the SRs was assessed with the AMSTAR 2 tool by multiple reviewers after a calibration exercise was performed. A summary score for each SR was calculated. The Mann-Whitney U test and Fisher's exact test were performed to compare both groups.

RESULTS

Out of 185 SRs that met all inclusion criteria, 27 SRs were industry funded. Those were matched with 30 non-industry funded SRs resulting in a total sample size of 57. The mean AMSTAR 2 summary score across all SRs was 0.49. Overall, the median AMSTAR 2 summary score was higher for the non-industry funded SRs than for the industry-funded SRs (0.62 vs. 0.36; p < .00001). Lower ratings for industry funded SRs were consistent across all but one AMSTAR 2 item, though significantly lower only for three specific items.

CONCLUSION

The methodological quality of SRs in vaccination is comparable to SRs in other fields, while it is still suboptimal. We are not able to provide a satisfactory explanation why industry funded SRs had a lower methodological quality than non-industry funded SRs over recent years. Industry funding is an important indicator of methodological quality for vaccine SRs and should be carefully considered when appraising SR quality.

A Methodological Quality Assessment of Meta-Analysis Studies in Dance Therapy Using AMSTAR and AMSTAR 2

Although earlier meta-analysis studies have provided evidence-based information useful for decision-making, debate regarding their quality continues. This study aimed to evaluate the quality of meta-analysis studies in the field of dance therapy (DT) using the Assessment of Multiple Systematic Reviews (AMSTAR) and AMSTAR 2 assessment tools. Meta-analysis studies on DT were collected from various databases. Seven meta-analysis studies were selected for this study. Our findings showed that the quality level of the meta-analysis studies related to DT was “High” on the AMSTAR evaluation, but their quality decreased to “Low” on the AMSTAR 2 evaluation. Moreover, using AMSTAR 2, 71.43% of the studies fell within the category of “Moderate” or below. There was no statistically significant difference in the quality scores of the characteristics of these studies. Our results suggest that (1) education on meta-analysis guidelines is required to improve the quality of DT-related meta-analysis studies, and (2) methodological caution is warranted, since different outcomes in evaluation scores for each tool may be obtained when using AMSTAR and AMSTAR 2. Based on this study, it is expected that common and specific guidelines for meta-analysis in DT can be established.

Factors affecting the vaccination choices of pregnant women for their children: a systematic review of the literature

In recent years, an increase in vaccine hesitancy has led to a decrease in vaccination coverage in several countries. We conducted a systematic review of studies that assessed knowledge of and attitudes toward pediatric vaccinations, and the vaccination choices and their determinants among pregnant women. A total of 6,277 records were retrieved, and 16 full texts were included in the narrative synthesis. The published literature on the topic shows that, overall, pregnant women believe that vaccines are important for the protection of their children and the community, but various concerns and misunderstandings persist around vaccine safety and efficacy, which reduce the trust of expectant mothers in immunization. Nevertheless, such attitudes and choices vary depending on the vaccine being considered and the corresponding determinants should therefore be studied in the context of each specific vaccination. Further research on this topic is needed, particularly in non-western countries.

Knowledge and attitudes on pediatric vaccinations and intention to vaccinate in a sample of pregnant women from the City of Rome

BACKGROUND

In recent years, pediatric immunization rates in Italy have decreased well below the recommended thresholds, largely due to an increase in scepticism about the efficacy and safety of vaccines. We aimed to identify the degree of such scepticism, and the factors driving it, among a sample of pregnant women in the City of Rome.

METHODS

We conducted a cross-sectional survey on a sample of pregnant women attending antenatal classes (CANs) in Rome through distribution of a self-administered questionnaire. Multiple logistic regression models were built to analyze the determinants of knowledge, attitudes and intention to vaccinate in this population.

RESULTS

A total of 458 pregnant women attending CANs in 36 family health centers and two hospitals in Rome answered the survey. Mean age was 32.9 (±5.0) years, and over 90% of women were in their first pregnancy. More than 26% of respondents showed a good level of knowledge of the safety and efficacy of vaccines, but there were high rates of uncertainty or agreement with some of the most common anti-vaccination sentiments. Only 75% of women were sure about vaccinating their children with the hexavalent vaccine, and 64.3% with MMR. A good level of knowledge was the strongest predictor of positive attitudes towards vaccination (OR 11.61, 95% CI 6.43-20.96), which, in turn, influenced the intention to vaccinate for most vaccines with the perception of the benefit of immunization for protection against disease.

CONCLUSIONS

Scepticism about the safety, efficacy and importance of vaccines is associated to pregnant women's hesitancy to vaccinate their children, suggesting the need to develop strategies to increase vaccine acceptance in the antenatal period. The capacity of health care professionals, particularly midwives, to correctly deliver information to future parents should be strengthened in order to reduce the spread of misinformation and fear of vaccine safety.

Methodological quality of meta-analyses of single-case experimental studies

BACKGROUND

Methodological rigor is a fundamental factor in the validity and credibility of the results of a meta-analysis.

AIM

Following an increasing interest in single-case experimental design (SCED) meta-analyses, the current study investigates the methodological quality of SCED meta-analyses.

METHODS AND PROCEDURES

We assessed the methodological quality of 178 SCED meta-analyses published between 1985 and 2015 through the modified Revised-Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist.

OUTCOMES AND RESULTS

The main finding of the current review is that the methodological quality of the SCED meta-analyses has increased over time, but is still low according to the R-AMSTAR checklist. A remarkable percentage of the studies (93.80% of the included SCED meta-analyses) did not even reach the midpoint score (22, on a scale of 0-44). The mean and median methodological quality scores were 15.57 and 16, respectively. Relatively high scores were observed for "providing the characteristics of the included studies" and "doing comprehensive literature search". The key areas of deficiency were "reporting an assessment of the likelihood of publication bias" and "using the methods appropriately to combine the findings of studies".

CONCLUSIONS AND IMPLICATIONS

Although the results of the current review reveal that the methodological quality of the SCED meta-analyses has increased over time, still more efforts are needed to improve their methodological quality.

Smoking Status and Gastric Cancer Risk: An Updated Meta-Analysis of Case-Control Studies Published in the past Ten Years

Background

A meta-analysis of published studies was performed in order to clarify the risk of gastric cancer associated with cigarette smoking status.

Methods

Eligible studies were all the case-control studies investigating an association between smoking status and gastric cancer published from January 1, 1997, until June 30, 2006. In order to evaluate the quality of the published data, a qualitative scoring of papers was applied. The principal outcome measure was the odds ratio for the risk of gastric cancer associated with the smoking status using a random effects model. Cigarette smoking status was assessed in two ways: ever (current and ex) versus never smokers; current versus never smokers.

Results

We found a statistically significant result for the association between ever smoking status and gastric cancer risk (OR = 1.48; 95% CI, 1.28–1.71), considering 14,442 cases and 73,918 controls. Moreover, we found an odds ratio of 1.69 for current smoker status in comparison to never smokers (95% CI, 1.35–2.11). Considering only high quality studies, the odds ratio increased by 43% for gastric cancer risk in ever smokers (OR = 1.43; 95% CI, 1.24–1.66; Q = 378.60, P <0.00001; I2 = 90%) and by 57% in current smokers (OR = 1.57; 95% CI, 1.24–2.01). We also considered separately Caucasians and Asian studies, finding for ever smokers an odds ratio of 1.46 (95% CI, 1.25–1.70; Q = 125.68, P <0.00001; I2 = 82.5%) and of 1.47 (95% CI, 1.13–1.91; Q = 366.77, P <0.00001; I2 = 94%), respectively.

Conclusions

From the results of this quantitative meta-analysis, it appears that cigarette smoking has to be considered an important risk factor. The use of qualitative scoring decreases the magnitude of the relationship both for ever and current smoker exposure by 5–12%. Future studies on this topic need to clarify the biological interaction between environmental factors (such as cigarette smoking) and different polymorphisms on gastric cancer.

[Review of meta-analysis research on exercise in South Korea]

PURPOSE

The purpose of this study was to evaluate the quality of meta-analysis regarding exercise using Assessment of Multiple Systematic Reviews (AMSTAR) as well as to compare effect size according to outcomes.

METHODS

Electronic databases including the Korean Studies Information Service System (KISS), the National Assembly Library and the DBpia, HAKJISA and RISS4U for the dates 1990 to January 2014 were searched for 'meta-analysis' and 'exercise' in the fields of medical, nursing, physical therapy and physical exercise in Korea. AMSTAR was scored for quality assessment of the 33 articles included in the study. Data were analyzed using descriptive statistics, t-test, ANOVA and χ²-test.

RESULTS

The mean score for AMSTAR evaluations was 4.18 (SD=1.78) and about 67% were classified at the low-quality level and 30% at the moderate-quality level. The scores of quality were statistically different by field of research, number of participants, number of databases, financial support and approval by IRB. The effect size that presented in individual studies were different by type of exercise in the applied intervention.

CONCLUSION

This critical appraisal of meta-analysis published in various field that focused on exercise indicates that a guideline such as the PRISMA checklist should be strongly recommended for optimum reporting of meta-analysis across research fields.

Meta-epidemiology

The concept of meta-epidemiology has been introduced with considering the methodological limitations of systematic review for intervention trials. The paradigm of meta-epidemiology has shifted from a statistical method into a new methodology to close gaps between evidence and practice. Main interest of meta-epidemiology is to control potential biases in previous quantitative systematic reviews and draw appropriate evidences for establishing evidence-base guidelines. Nowadays, the network meta-epidemiology was suggested in order to overcome some limitations of meta-epidemiology. To activate meta-epidemiologic studies, implementation of tools for risk of bias and reporting guidelines such as the Consolidated Standards for Reporting Trials (CONSORT) should be done.

Methodological quality of systematic reviews on influenza vaccination

BACKGROUND

There is a growing body of evidence on the risks and benefits of influenza vaccination in various target groups. Systematic reviews are of particular importance for policy decisions. However, their methodological quality can vary considerably.

OBJECTIVES

To investigate the methodological quality of systematic reviews on influenza vaccination (efficacy, effectiveness, safety) and to identify influencing factors.

METHODS

A systematic literature search on systematic reviews on influenza vaccination was performed, using MEDLINE, EMBASE and three additional databases (1990-2013). Review characteristics were extracted and the methodological quality of the reviews was evaluated using the assessment of multiple systematic reviews (AMSTAR) tool. U-test, Kruskal-Wallis test, chi-square test, and multivariable linear regression analysis were used to assess the influence of review characteristics on AMSTAR-score.

RESULTS

Fourty-six systematic reviews fulfilled the inclusion criteria. Average methodological quality was high (median AMSTAR-score: 8), but variability was large (AMSTAR range: 0-11). Quality did not differ significantly according to vaccination target group. Cochrane reviews had higher methodological quality than non-Cochrane reviews (p=0.001). Detailed analysis showed that this was due to better study selection and data extraction, inclusion of unpublished studies, and better reporting of study characteristics (all p<0.05). In the adjusted analysis, no other factor, including industry sponsorship or journal impact factor had an influence on AMSTAR score.

CONCLUSIONS

Systematic reviews on influenza vaccination showed large differences regarding their methodological quality. Reviews conducted by the Cochrane collaboration were of higher quality than others. When using systematic reviews to guide the development of vaccination recommendations, the methodological quality of a review in addition to its content should be considered.

Polymorphic repeat length in the AIB1 gene and breast cancer risk in BRCA1 and BRCA2 mutation carriers: a meta-analysis of observational studies

OBJECTIVES

We carried out a meta-analysis focusing on the relationship between length of AIB1 gene poly-Q repeat domain as a modifier of breast cancer (BC) susceptibility in patients with BRCA1 and BRCA2 mutation carriers.

DATA SOURCES

We searched MEDLINE and EMBASE for all medical literature published until February, 2012.

STUDY ELIGIBILITY CRITERIA

Studies were included in the meta-analysis if they met all the predetermined criteria, such as: (a) case-control or cohort studies; (b) the primary outcome was clearly defined as BC; (c) the exposure of interest measured was AIB1 polyglutamine repeat length genotype; (d) provided relative risk (RR) or odds ratio (OR) estimates and their 95% confidence intervals (CIs). SYNTHESIS METHODS: Two of the authors independently evaluated the quality of the studies included and extracted the data. Meta-analyses were performed for case-control and cohort studies separately. Heterogeneity was examined and the publication bias was assessed with a funnel plot for asymmetry.

RESULT

7 studies met our predetermined inclusion criteria and were included in the meta-analysis. Overall quality ratings of the studies varied from 0.36 to 0.77, with a median of 0.5. The overall RR estimates of 29/29 poly-Q repeats on risk of BC in BRCA1/2, BRCA1, and BRCA2, were always greater than 1.00; however, this effect was not statistically significant. In the meta-analysis of studies reporting the effect of 28/28 poly-Q repeats on risk of BC in BRCA1/2, BRCA1, and BRCA2, the overall RR decreased below 1.00; however, this effect was not statistically significant. Similar estimates were shown for at least 1 allele of ≤26 repeats.

CONCLUSIONS

Genotypes of AIB1 polyglutamine polymorphism analyzed do not appear to be associated to a modified risk of BC development in BRCA1 and BRCA2 mutation carriers. Future research on length of poly-Q repeat domain and BC susceptibility should be discouraged and more promising potential sources of penetrance variation among BRCA1 and BRCA2 mutation carriers should be investigated.

Effectiveness and harms of seasonal and pandemic influenza vaccines in children, adults and elderly: a critical review and re-analysis of 15 meta-analyses

Fifteen meta-analyses have been published between 1995 and 2011 to evaluate the efficacy/effectiveness and harms of diverse influenza vaccines—seasonal, H5N1 and 2009(H1N1) —in various age-classes (healthy children, adults or elderly). These meta-analyses have often adopted different analyses and study selection criteria. Because it is difficult to have a clear picture of vaccine benefits and harms examining single systematic reviews, we compiled the main findings and evaluated which could be the most reasonable explanations for some differences in findings (or their interpretation) across previously published meta-analyses. For each age group, we performed analyses that included all trials that had been included in at least one relevant meta-analysis, also exploring whether effect sizes changed over time. Although we identified several discrepancies among the meta-analyses on seasonal vaccines for children and elderly, overall most seasonal influenza vaccines showed statistically significant efficacy/effectiveness, which was acceptable or high for laboratory-confirmed cases and of modest magnitude for clinically-confirmed cases. The available evidence on parenteral inactivated vaccines for children aged < 2 y remains scarce. Pre-pandemic “avian” H5N1 and pandemic 2009 (H1N1) vaccines can achieve satisfactory immunogenicity, but no meta-analysis has addressed H1N1 vaccination impact on clinical outcomes. Data on harms are overall reassuring, but their value is diminished by inconsistent reporting.

The quality of meta-analyses of genetic association studies: a review with recommendations

Although there has been a rapid rise in the publication of meta-analyses of genetic association studies, little is known about their methodological quality. The authors reviewed the quality of 120 randomly selected genetic meta-analyses published between 2005 and 2007. Data extracted included issues of general relevance and other issues specific to genetic epidemiology. Quality was markedly poorer in the 26% of the meta-analyses that accompanied a report on a primary study. Such meta-analyses were predominantly published in specialist journals, and their quality was positively associated with the impact factor of the journal. Among the meta-analyses that did not accompany a primary study, Human Genome Epidemiology reviews tended to score better than the others, although the comparison was limited by relatively small numbers. Comparison of the overall quality with that of genetic meta-analyses published before 2000 showed improvement in both conduct and reporting. However, the quality of the handling of specific genetic issues remains disappointingly low. For a few key general quality issues, the authors compared their findings with findings in other fields of medicine and found that general quality was similar. On the basis of this review, the authors provide practical recommendations for the conduct and reporting of genetic meta-analyses.