Evaluating the quality of systematic reviews in the emergency medicine literature

Quality of systematic reviews in African emergency medicine: a cross-sectional methodological study

Introduction

Reliable systematic reviews are essential to inform clinical practice guidelines, policies and further research priorities in Africa. For systematic review findings to be trustworthy, they need to be conducted with methodological rigour and reported transparently. We assessed the methodological quality of systematic reviews published in African emergency medicine journals, comparing them to those published in international emergency medicine journals. Additionally, we describe the types of review literature published in the African journals.

Methods

We performed a cross-sectional methodological study of systematic reviews published in selected African and international emergency medicine journals from 2012 to 2021. Studies were eligible if they were i) a systematic review on an emergency medicine topic, ii) published in one of the top five emergency medicine journals in the African region or internationally and iii) published between January 2012 and December 2021 in English or French. We searched PubMed, Web of Science and Scopus databases and hand-searched selected journals. Two authors screened titles, abstracts and full texts independently and in duplicate. Data extraction was performed by one reviewer, using a standardised form, after completing a calibration exercise. We described the characteristics of systematic reviews and assessed methodological quality using AMSTAR II.

Results

We identified 34 (37%) African and 511 (54%) international systematic reviews from 92 and 948 review articles respectively across 10 journals. We included all 34 African and a random sample of 100 international systematic reviews. Methodological quality was low or critically low for all the African systematic reviews (n=34, 100%) and all but three international systematic reviews (n=97, 97%). The median number of critical domain weaknesses was 4 (IQR 4;5) and 2 (IQR 2;4) for African and international systematic reviews respectively. The most common weaknesses across both African and international systematic reviews were i) not establishing a priori review protocols, ii) unclear selection of study designs iii) not providing a list of excluded studies and iv) unclear reporting on funding sources for included studies.

Conclusion

Emergency medicine systematic reviews published in African and international journals are lacking in methodological quality. Reporting an a priori protocol, developing a comprehensive search strategy, appropriate evidence synthesis and adequate assessment of risk of bias, heterogeneity and evidence certainty may improve the quality of systematic reviews.

Use of micronucleus assays for the prediction and detection of cervical cancer: a meta-analysis

Cervical cancer (CC) is the fourth most common cancer in women; the survival rates depend strongly on its early detection. The Pap test is the most frequently used diagnostic tool, but due to its limited sensitivity/specificity, additional screening tests are needed. Therefore, we evaluated the use of micronucleus (MN) assays with cervical cells for the prediction and diagnosis of CC. MN reflects structural and numerical chromosomal aberrations. A search was performed in Pubmed, Scopus, Thomson ISI and Google Scholar. Subsequently, meta-analyses were performed for different grades of abnormal findings in smears and biopsies from patients which were diagnosed with CC. Results of 21 studies in which findings of MN experiments were compared with data from Pap tests show that higher MN frequencies were found in women with abnormal cells that are indicative for increased cancer risks. MN frequency ratios increased in the order inflammation (2.1) < ASC-US and ASC-H (3.3) < LGSIL (4.4) < HGSIL (8.4). Furthermore, results are available from 17 investigations in which MN were scored in smears from patients with neoplasia. MN rates increased with the degree of neoplasia [CIN 1 (4.6) < CIN 2 (6.5) and CIN 3 (10.8)] and were significantly higher (8.8) in CC patients. Our meta-analysis indicates that the MN assay, which is easy to perform in combination with Pap tests, may be useful for the detection/prediction of CC. However, standardization (including definition of the optimal cell numbers and stains) and further validation is necessary before the MN test can be implemented in routine screening.

An International Educational Training Course for Conducting Systematic Reviews in Health Care: The Joanna Briggs Institute's Comprehensive Systematic Review Training Program

BACKGROUND

The cornerstone of evidence-based health care is the systematic review of international evidence. Systematic reviews follow a rigorous, standardized approach in their conduct and reporting, and as such, education and training are essential prior to commencement.

AIMS

This study reports on the evolution of the Joanna Briggs Institute Comprehensive Systematic Review Training Program (JBICSRTP) as an exemplar approach for teaching systematic review methods.

RESULTS

The Joanna Briggs Institute (JBI) is an international research and development center at the University of Adelaide, South Australia. Its mission is to promote and facilitate evidence-based best practice globally, largely through the provision of education and training. JBI was one of the first to consider all forms of evidence in systematic reviews, and as such, implementation of standardized training was essential. Since 1999, JBI has offered a systematic review training program. The JBICSRTP is now delivered face to face over 5 days, with an optional online component; the content aligns to that proposed in the Sicily statement. Over the last 3 years, JBI and its Collaboration have trained over 3,300 people from over 30 countries. A "train-the-trainer" (TtT) style program was established to cope with demand, and to date, hundreds of trainers have been licensed across the globe to deliver the JBICSRTP.

LINKING EVIDENCE TO ACTION

Providing standardized training materials, ensuring open and ongoing communication, and adopting a TtT style program while still allowing for local adaptability are strategies that have led to the establishment of a highly skilled global training network and ensured the success and longevity of the JBICSRTP.

Appraising the Quality of Systematic Reviews for Age-Related Macular Degeneration Interventions: A Systematic Review

Importance

Age-related macular degeneration (AMD) is a leading cause of vision impairment. It is imperative that AMD care is timely, appropriate, and evidence-based. It is thus essential that AMD systematic reviews are robust; however, little is known about the quality of this literature.

Objectives

To investigate the methodological quality of systematic reviews of AMD intervention studies, and to evaluate their use for guiding evidence-based care.

Evidence Review

This systematic review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. All studies that self-identified as a systematic review in their title or abstract or were categorized as a systematic review from a medical subject heading and investigated the safety, efficacy and/or effectiveness of an AMD intervention were included. Comprehensive electronic searches were performed in Ovid MEDLINE, Embase, and the Cochrane Library from inception to March 2017. Two reviewers independently assessed titles and abstracts, then full-texts for eligibility. Quality was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool. Study characteristics (publication year, type of intervention, journal, citation rate, and funding source) were extracted.

Findings

Of 983 citations retrieved, 71 studies (7.6%) were deemed eligible. The first systematic review relating to an AMD intervention was published in 2003. More than half were published since 2014. Methodological quality was highly variable. The mean (SD) AMSTAR score was 5.8 (3.2) of 11.0, with no significant improvement over time (r = -0.03; 95% CI, -0.26 to 0.21; P = .83). Cochrane systematic reviews were overall of higher quality than reviews in other journals (mean [SD] AMSTAR score, 9.9 [1.2], n = 15 vs 4.7 [2.2], n = 56; P < .001). Overall, there was poor adherence to referring to an a priori design (22 articles [31%]) and reporting conflicts of interest in both the review and included studies (16 articles [23%]). Reviews funded by government grants and/or institutions were generally of higher quality than industry-sponsored reviews or where the funding source was not reported.

Conclusions and Relevance

There are gaps in the conduct of systematic reviews in the field of AMD. Enhanced endorsement of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement by refereed journals may improve review quality and improve the dissemination of reliable evidence relating to AMD interventions to clinicians.

Systematic Reviews in Burns Care: Poor Quality and Getting Worse

The objective of this article is to investigate adherence to reporting standards and methodological quality in systematic reviews on burns care published in peer-reviewed journals to determine their utility for guiding evidence-based burns care. PubMed, Embase, Database of Abstracts of Reviews of Effects, Cochrane Database of Systematic Reviews, and the Joanna Briggs Institute (JBI) Database of Systematic Reviews and Implementation Reports were searched from 2009. Any systematic review on any question on therapeutic interventions in burns care was eligible for inclusion. Critical appraisal and data extraction were performed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) checklist by two independent reviewers. The overall quality of the 44 included burns care systematic reviews was low, with an average methodological quality of 55% and an average compliance with reporting guidelines of 70%. Correlation analysis showed that adherence to reporting guidelines has been relatively stable, but methodological quality has deteriorated (r = -.32, P < .05). Cochrane reviews had lower citation rates than reviews published in other journals, whereas reviews that included meta-analyses had more citations. Quality did not have a significant effect on citation rate. Health professionals working in burns should be able to expect that systematic reviews published in their field are of a high standard. Unfortunately, this is not the case. To address this problem, established guidelines on the conduct and reporting of systematic reviews should be adhered to by researchers and editors.

Systematic review adherence to methodological or reporting quality

BACKGROUND

Guidelines for assessing methodological and reporting quality of systematic reviews (SRs) were developed to contribute to implementing evidence-based health care and the reduction of research waste. As SRs assessing a cohort of SRs is becoming more prevalent in the literature and with the increased uptake of SR evidence for decision-making, methodological quality and standard of reporting of SRs is of interest. The objective of this study is to evaluate SR adherence to the Quality of Reporting of Meta-analyses (QUOROM) and PRISMA reporting guidelines and the A Measurement Tool to Assess Systematic Reviews (AMSTAR) and Overview Quality Assessment Questionnaire (OQAQ) quality assessment tools as evaluated in methodological overviews.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® databases were searched from January 1990 to October 2014. Title and abstract screening and full-text screening were conducted independently by two reviewers. Reports assessing the quality or reporting of a cohort of SRs of interventions using PRISMA, QUOROM, OQAQ, or AMSTAR were included. All results are reported as frequencies and percentages of reports and SRs respectively.

RESULTS

Of the 20,765 independent records retrieved from electronic searching, 1189 reports were reviewed for eligibility at full text, of which 56 reports (5371 SRs in total) evaluating the PRISMA, QUOROM, AMSTAR, and/or OQAQ tools were included. Notable items include the following: of the SRs using PRISMA, over 85% (1532/1741) provided a rationale for the review and less than 6% (102/1741) provided protocol information. For reports using QUOROM, only 9% (40/449) of SRs provided a trial flow diagram. However, 90% (402/449) described the explicit clinical problem and review rationale in the introduction section. Of reports using AMSTAR, 30% (534/1794) used duplicate study selection and data extraction. Conversely, 80% (1439/1794) of SRs provided study characteristics of included studies. In terms of OQAQ, 37% (499/1367) of the SRs assessed risk of bias (validity) in the included studies, while 80% (1112/1387) reported the criteria for study selection.

CONCLUSIONS

Although reporting guidelines and quality assessment tools exist, reporting and methodological quality of SRs are inconsistent. Mechanisms to improve adherence to established reporting guidelines and methodological assessment tools are needed to improve the quality of SRs.

Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study

BACKGROUND

The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to 'quality' assessment over time.

METHODS

The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports.

RESULTS

Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ.

CONCLUSIONS

The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting.

The writer's guide to education scholarship in emergency medicine: Systematic reviews and the scholarship of integration (part 4)

OBJECTIVE

Reviews help scholars consolidate evidence and guide their educational practice. However, few papers describe how to effectively publish review papers. We completed a scoping review to develop a set of quality indicators that will assist junior authors to publish reviews and integrative scholarship.

METHODS

MEDLINE, Embase, ERIC, and Google Scholar were searched for English language articles published between 2012 and January 2016 using the terms review, medical education, how to publish, and emergency medicine. Titles and abstracts were reviewed by two authors and included if they focused on how to publish a review or outlined reporting guidelines of reviews. The articles were reviewed in parallel for calibration, and disagreements were resolved through a consensus.

RESULTS

A full text review of the 25 articles was conducted, and 196 recommendations were extracted from 13 articles. A hand search of the included articles' reference lists and expert recommendation found an additional eight articles. These recommendations were thematically analysed into a list of seven themes and 32 items. Additionally, seven evaluation tools and reporting guidelines were found to guide researchers in optimizing their reviews for publication.

CONCLUSION

In emergency medicine education, review articles can help synthesize educational research so that educators can engage in evidence-based scholarly teaching. We hope that this work will act as an introduction to those interested in engaging in integrative scholarship by providing them with a guide to key quality markers and important checklists for improving their research.

Impact of librarians on reporting of the literature searching component of pediatric systematic reviews

OBJECTIVE

A critical element in conducting a systematic review is the identification of studies. To date, very little empirical evidence has been reported on whether the presence of a librarian or information professional can contribute to the quality of the final product. The goal of this study was to compare the reporting rigor of the literature searching component of systematic reviews with and without the help of a librarian.

METHOD

Systematic reviews published from 2002 to 2011 in the twenty highest impact factor pediatrics journals were collected from MEDLINE. Corresponding authors were contacted via an email survey to determine if a librarian was involved, the role that the librarian played, and functions that the librarian performed. The reviews were scored independently by two reviewers using a fifteen-item checklist.

RESULTS

There were 186 reviews that met the inclusion criteria, and 44% of the authors indicated the involvement of a librarian in conducting the systematic review. With the presence of a librarian as coauthor or team member, the mean checklist score was 8.40, compared to 6.61 (p<0.001) for reviews without a librarian.

CONCLUSIONS

Findings indicate that having a librarian as a coauthor or team member correlates with a higher score in the literature searching component of systematic reviews.

Systematic reviews in emergency medicine: Part II. Critical appraisal of review quality, data synthesis and result interpretation

Reviews of the medical literature have always been an important resource for physicians. Increasingly, qualitative and quantitative systematic reviews (SRs) have replaced the traditional “narrative review” as a means of capturing and summarizing current evidence on a topic or, when possible, answering a specific clinical question. This paper is Part II of a 2-part series designed to provide emergency physicians with the background necessary to locate, critically evaluate and interpret SRs. The paper expands on the critical appraisal principles discussed in Part I by focusing on quality assessment, data synthesis and interpretation of results. To illustrate key points and facilitate readability, examples from the emergency medicine literature have been included and technical details have been kept to a minimum. The references, however, are comprehensive and provide a resource for readers seeking further information.

Is there any benefit in associating neuraxial anesthesia to general anesthesia for coronary artery bypass graft surgery?

BACKGROUND AND OBJECTIVES

The use of neuraxial anesthesia in cardiac surgery is recent, but the hemodynamic effects of local anesthetics and anticoagulation can result in risk to patients.

OBJECTIVE

To review the benefits of neuraxial anesthesia in cardiac surgery for CABG through a systematic review of systematic reviews.

CONTENT

The search was performed in Pubmed (January 1966 to December 2012), Embase (1974 to December 2012), The Cochrane Library (volume 10, 2012) and Lilacs (1982 to December 2012) databases, in search of articles of systematic reviews. The following variables: mortality, myocardial infarction, stroke, in-hospital length of stay, arrhythmias and epidural hematoma were analyzed.

CONCLUSIONS

The use of neuraxial anesthesia in cardiac surgery remains controversial. The greatest benefit found by this review was the possibility of reducing postoperative arrhythmias, but this result was contradictory among the identified findings. The results of findings regarding mortality, myocardial infarction, stroke and in-hospital length of stay did not show greater efficacy of neuraxial anesthesia.

Reporting quality of systematic reviews/meta-analyses of acupuncture

BACKGROUND

The QUOROM and PRISMA statements were published in 1999 and 2009, respectively, to improve the consistency of reporting systematic reviews (SRs)/meta-analyses (MAs) of clinical trials. However, not all SRs/MAs adhere completely to these important standards. In particular, it is not clear how well SRs/MAs of acupuncture studies adhere to reporting standards and which reporting criteria are generally ignored in these analyses.

OBJECTIVES

To evaluate reporting quality in SRs/MAs of acupuncture studies.

METHODS

We performed a literature search for studies published prior to 2014 using the following public archives: PubMed, EMBASE, Web of Science, the Cochrane Database of Systematic Reviews (CDSR), the Chinese Biomedical Literature Database (CBM), the Traditional Chinese Medicine (TCM) database, the Chinese Journal Full-text Database (CJFD), the Chinese Scientific Journal Full-text Database (CSJD), and the Wanfang database. Data were extracted into pre-prepared Excel data-extraction forms. Reporting quality was assessed based on the PRISMA checklist (27 items).

RESULTS

Of 476 appropriate SRs/MAs identified in our search, 203, 227, and 46 were published in Chinese journals, international journals, and the Cochrane Database, respectively. In 476 SRs/MAs, only 3 reported the information completely. By contrast, approximately 4.93% (1/203), 8.81% (2/227) and 0.00% (0/46) SRs/Mas reported less than 10 items in Chinese journals, international journals and CDSR, respectively. In general, the least frequently reported items (reported≤50%) in SRs/MAs were "protocol and registration", "risk of bias across studies", and "additional analyses" in both methods and results sections.

CONCLUSIONS

SRs/MAs of acupuncture studies have not comprehensively reported information recommended in the PRISMA statement. Our study underscores that, in addition to focusing on careful study design and performance, attention should be paid to comprehensive reporting standards in SRs/MAs on acupuncture studies.

Factors underlying the success of behavioral HIV-prevention interventions for adolescents: a meta-review

The purpose of this meta-review was to identify characteristics of successful HIV prevention interventions for adolescents based on quantitative (i.e., meta-analyses) and qualitative reviews published to date, and to inform intervention utilization and future development. To that end, we were guided by principles of triangulation. Searches of seven electronic bibliographic databases yielded five meta-analyses and six qualitative reviews that satisfied the selection criteria. Reviews were subjected to careful content analysis. All reviews reported that behavioral interventions had positive outcomes on at least one of the following outcomes: HIV-related knowledge, subjective cognitions and beliefs enabling safer sex, abstinence, delaying next sexual intercourse, decreasing number of sexual partners, and actual condom use. Four categories, suggesting factors more prominently linked to intervention success, emerged: behavior change techniques (e.g., cognitive-behavior and motivation enhancement skills training); recipient characteristics (e.g., age, vulnerability to contracting STIs/HIV); prominent design features (e.g., use of theory, formative research); and socio-ecological features (e.g., supportive school environment). Future interventions would benefit from conducting preliminary formative research in order to enable optimal implementation of all these factors.

Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions

BACKGROUND

Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta-analysis (e.g. from multiple measurement scales and time points) and the choice of effect estimate to include in the meta-analysis is based on the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting occurs when the reporting of a subset of outcomes and analyses in the systematic review is based on the results (e.g. a protocol-defined outcome is omitted from the published systematic review).

OBJECTIVES

To summarise the characteristics and synthesise the results of empirical studies that have investigated the prevalence of selective inclusion or reporting in systematic reviews of randomised controlled trials (RCTs), investigated the factors (e.g. statistical significance or direction of effect) associated with the prevalence and quantified the bias.

SEARCH METHODS

We searched the Cochrane Methodology Register (to July 2012), Ovid MEDLINE, Ovid EMBASE, Ovid PsycINFO and ISI Web of Science (each up to May 2013), and the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (to June 2013). We also searched the abstract books of the 2011 and 2012 Cochrane Colloquia and the article alerts for methodological work in research synthesis published from 2009 to 2011 and compiled in Research Synthesis Methods.

SELECTION CRITERIA

We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. We included empirical studies assessing any type of selective inclusion or reporting, such as investigations of how frequently RCT outcome data is selectively included in systematic reviews based on the results, outcomes and analyses are discrepant between protocol and published review or non-significant outcomes are partially reported in the full text or summary within systematic reviews.

DATA COLLECTION AND ANALYSIS

Two review authors independently selected empirical studies for inclusion, extracted the data and performed a risk of bias assessment. A third review author resolved any disagreements about inclusion or exclusion of empirical studies, data extraction and risk of bias. We contacted authors of included studies for additional unpublished data. Primary outcomes included overall prevalence of selective inclusion or reporting, association between selective inclusion or reporting and the statistical significance of the effect estimate, and association between selective inclusion or reporting and the direction of the effect estimate. We combined prevalence estimates and risk ratios (RRs) using a random-effects meta-analysis model.

MAIN RESULTS

Seven studies met the inclusion criteria. No studies had investigated selective inclusion of results in systematic reviews, or discrepancies in outcomes and analyses between systematic review registry entries and published systematic reviews. Based on a meta-analysis of four studies (including 485 Cochrane Reviews), 38% (95% confidence interval (CI) 23% to 54%) of systematic reviews added, omitted, upgraded or downgraded at least one outcome between the protocol and published systematic review. The association between statistical significance and discrepant outcome reporting between protocol and published systematic review was uncertain. The meta-analytic estimate suggested an increased risk of adding or upgrading (i.e. changing a secondary outcome to primary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.43, 95% CI 0.71 to 2.85; two studies, n = 552 meta-analyses). Also, the meta-analytic estimate suggested an increased risk of downgrading (i.e. changing a primary outcome to secondary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.26, 95% CI 0.60 to 2.62; two studies, n = 484 meta-analyses). None of the included studies had investigated whether the association between statistical significance and adding, upgrading or downgrading of outcomes was modified by the type of comparison, direction of effect or type of outcome; or whether there is an association between direction of the effect estimate and discrepant outcome reporting.Several secondary outcomes were reported in the included studies. Two studies found that reasons for discrepant outcome reporting were infrequently reported in published systematic reviews (6% in one study and 22% in the other). One study (including 62 Cochrane Reviews) found that 32% (95% CI 21% to 45%) of systematic reviews did not report all primary outcomes in the abstract. Another study (including 64 Cochrane and 118 non-Cochrane reviews) found that statistically significant primary outcomes were more likely to be completely reported in the systematic review abstract than non-significant primary outcomes (RR 2.66, 95% CI 1.81 to 3.90). None of the studies included systematic reviews published after 2009 when reporting standards for systematic reviews (Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement, and Methodological Expectations of Cochrane Intervention Reviews (MECIR)) were disseminated, so the results might not be generalisable to more recent systematic reviews.

AUTHORS' CONCLUSIONS

Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review abstracts is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post-hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.

Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews

BACKGROUND

Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews.

OBJECTIVES

To compare search methods in systematic reviews of adverse effects with other reviews.

METHODS

The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews.

RESULTS

Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings.

CONCLUSIONS

The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.

Methodology and reporting of meta-analyses in the neurosurgical literature

OBJECT

Neurosurgeons are inundated with vast amounts of new clinical research on a daily basis, making it difficult and time-consuming to keep up with the latest literature. Meta-analysis is an extension of a systematic review that employs statistical techniques to pool the data from the literature in order to calculate a cumulative effect size. This is done to answer a clearly defined a priori question. Despite their increasing popularity in the neurosurgery literature, meta-analyses have not been scrutinized in terms of reporting and methodology.

METHODS

The authors performed a literature search using PubMed/MEDLINE to locate all meta-analyses that have been published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Accepted checklists for reporting (PRISMA) and methodology (AMSTAR) were applied to each meta-analysis, and the number of items within each checklist that were satisfactorily fulfilled was recorded. The authors sought to answer 4 specific questions: Are meta-analyses improving 1) with time; 2) when the study met their definition of a meta-analysis; 3) when clinicians collaborated with a potential expert in meta-analysis; and 4) when the meta-analysis was the only focus of the paper?

RESULTS

Seventy-two meta-analyses were published in the JNS Publishing Group journals and Neurosurgery between 1990 and 2012. The number of published meta-analyses has increased dramatically in the last several years. The most common topics were vascular, and most were based on observational studies. Only 11 papers were prepared using an established checklist. The average AMSTAR and PRISMA scores (proportion of items satisfactorily fulfilled divided by the total number of eligible items in the respective instrument) were 31% and 55%, respectively. Major deficiencies were identified, including the lack of a comprehensive search strategy, study selection and data extraction, assessment of heterogeneity, publication bias, and study quality. Almost one-third of the papers did not meet our basic definition of a meta-analysis. The quality of reporting and methodology was better 1) when the study met our definition of a meta-analysis; 2) when one or more of the authors had experience or expertise in conducting a meta-analysis; 3) when the meta-analysis was not conducted alongside an evaluation of the authors' own data; and 4) in more recent studies.

CONCLUSIONS

Reporting and methodology of meta-analyses in the neurosurgery literature is excessively variable and overall poor. As these papers are being published with increasing frequency, neurosurgical journals need to adopt a clear definition of a meta-analysis and insist that they be created using checklists for both reporting and methodology. Standardization will ensure high-quality publications.

Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality

BACKGROUND

Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases have increased over the past decade and become increasingly important to a wide range of clinicians, policy makers, and other health care stakeholders. While a few criticisms about their methodological rigor and synthesis approaches have recently appeared, no formal appraisal of their quality has been conducted yet.

OBJECTIVE

The primary aim of this critical review was to evaluate the methodology, quality, and reporting characteristics of prior reviews that have investigated the effects of home telemonitoring interventions in the context of chronic diseases.

METHODS

Ovid MEDLINE, the Database of Abstract of Reviews of Effects (DARE), and Health Technology Assessment Database (HTA) of the Cochrane Library were electronically searched to find relevant systematic reviews, published between January 1966 and December 2012. Potential reviews were screened and assessed for inclusion independently by three reviewers. Data pertaining to the methods used were extracted from each included review and examined for accuracy by two reviewers. A validated quality assessment instrument, R-AMSTAR, was used as a framework to guide the assessment process.

RESULTS

Twenty-four reviews, nine of which were meta-analyses, were identified from more than 200 citations. The bibliographic search revealed that the number of published reviews has increased substantially over the years in this area and although most reviews focus on studying the effects of home telemonitoring on patients with congestive heart failure, researcher interest has extended to other chronic diseases as well, such as diabetes, hypertension, chronic obstructive pulmonary disease, and asthma. Nevertheless, an important number of these reviews appear to lack optimal scientific rigor due to intrinsic methodological issues. Also, the overall quality of reviews does not appear to have improved over time. While several criteria were met satisfactorily by either all or nearly all reviews, such as the establishment of an a priori design with inclusion and exclusion criteria, use of electronic searches on multiple databases, and reporting of studies characteristics, there were other important areas that needed improvement. Duplicate data extraction, manual searches of highly relevant journals, inclusion of gray and non-English literature, assessment of the methodological quality of included studies and quality of evidence were key methodological procedures that were performed infrequently. Furthermore, certain methodological limitations identified in the synthesis of study results have affected the results and conclusions of some reviews.

CONCLUSIONS

Despite the availability of methodological guidelines that can be utilized to guide the proper conduct of systematic reviews and meta-analyses and eliminate potential risks of bias, this knowledge has not yet been fully integrated in the area of home telemonitoring. Further efforts should be made to improve the design, conduct, reporting, and publication of systematic reviews and meta-analyses in this area.

Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011

OBJECTIVE

An increasing amount of research and guidelines has been published on search methodology and the reporting of search strategies in systematic reviews. This research assessed whether this has lead to any improvements in the reporting and quality of searching in systematic reviews of adverse effects.

STUDY DESIGN AND SETTING

All records within Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects were scanned for systematic reviews of adverse effects. Data were then extracted on the methods used for information retrieval in these reviews and a descriptive analysis conducted by publication year.

RESULTS

A total of 849 reviews published from 1994 to 2011 met the inclusion criteria. There has been a significant increase (P<0.001) in the number of adverse effects reviews per year from 1994 (n=5) to 2010 (n=104). Some improvements were apparent, such as an increase in the number of databases searched and fewer date and language restrictions applied. However, there has been an increase in reviews limited to data from randomized controlled trials, whereas the reporting of search strategies could still be improved further, with only 9% (74/849) of the reviews reporting reproducible searches.

CONCLUSION

Some improvements in searching systematic reviews of adverse effects are apparent; however, poor reporting of search strategies remains a great obstacle to readers.

Neuraxial anesthesia compared to general anesthesia for procedures on the lower half of the body: systematic review of systematic reviews

BACKGROUND AND OBJECTIVES

Systematic reviews organize literature data by combining results from published studies in order to resolve conflicts in the area of medical knowledge describing the interventions. The inadequate reporting of systematic reviews can damage the credibility and interfere in the results' quality. The objective of this study was to determine the frequency of good quality systematic reviews comparing neuraxial anesthesia with general anesthesia for procedures on the lower half of the body.

METHODS

Systematic review of systematic reviews. Primary variable: The frequency of good quality systematic reviews. The information was analyzed from the following databases: LILACS (January 1982 to December 2010); PubMed (January 1950 to December 2010); The Cochrane Database of Systematic Review and Database of Abstracts of Reviews of Effects (volume 10, 2010); and SciELO (December 2010). The quality of systematic reviews was determined by the Overview Quality Assessment Questionnaire. The sample size calculation showed that it was necessary to analyze eight systematic reviews, taking into account that the frequency of good quality systematic reviews was 5%, an absolute precision of 15%, and a significance level of 5%.

RESULTS

Were identified 1,995 articles. The selection process eliminated 1,968 articles. Twenty-seven articles of systematic reviews were read in full, 9 were excluded due to incompatibility with the inclusion criteria, and 8 were duplicate publications. Ten systematic reviews were assessed for their quality. The frequency of good quality systematic reviews was 40% (4/10; 95% CI 9.6 to 70.4%).

CONCLUSION

The frequency of good quality systematic reviews was 40%.

The quality of meta-analyses of genetic association studies: a review with recommendations

Although there has been a rapid rise in the publication of meta-analyses of genetic association studies, little is known about their methodological quality. The authors reviewed the quality of 120 randomly selected genetic meta-analyses published between 2005 and 2007. Data extracted included issues of general relevance and other issues specific to genetic epidemiology. Quality was markedly poorer in the 26% of the meta-analyses that accompanied a report on a primary study. Such meta-analyses were predominantly published in specialist journals, and their quality was positively associated with the impact factor of the journal. Among the meta-analyses that did not accompany a primary study, Human Genome Epidemiology reviews tended to score better than the others, although the comparison was limited by relatively small numbers. Comparison of the overall quality with that of genetic meta-analyses published before 2000 showed improvement in both conduct and reporting. However, the quality of the handling of specific genetic issues remains disappointingly low. For a few key general quality issues, the authors compared their findings with findings in other fields of medicine and found that general quality was similar. On the basis of this review, the authors provide practical recommendations for the conduct and reporting of genetic meta-analyses.

Reprint--preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

Editor's Note: PTJ's Editorial Board has adopted PRISMA to help PTJ better communicate research to physical therapists. For more, read Chris Maher's editorial starting on page 870.

Membership of the PRISMA Group is provided in the Acknowledgments.

This article has been reprinted with permission from the Annals of Internal Medicine from Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Ann Intern Med. Available at: http://www.annals.org/cgi/content/full/151/4/264. The authors jointly hold copyright of this article. This article has also been published in PLoS Medicine, BMJ, Journal of Clinical Epidemiology, and Open Medicine.

Copyright © 2009 Moher et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

A systematic review of the quality and impact of anxiety disorder meta-analyses

Meta-analyses are seen as representing the pinnacle of a hierarchy of evidence used to inform clinical practice. Therefore, the potential importance of differences in the rigor with which they are conducted and reported warrants consideration. In this review, we use standardized instruments to describe the scientific and reporting quality of meta-analyses of randomized controlled trials of the treatment of anxiety disorders. We also use traditional and novel metrics of article impact to assess the influence of meta-analyses across a range of research fields in the anxiety disorders. Overall, although the meta-analyses that we examined had some flaws, their quality of reporting was generally acceptable. Neither the scientific nor reporting quality of the meta-analyses was predicted by any of the impact metrics. The finding that treatment meta-analyses were cited less frequently than quantitative reviews of studies in current "hot spots" of research (ie, genetics, imaging) points to the multifactorial nature of citation patterns. A list of the meta-analyses included in this review is available on an evidence-based website of anxiety and trauma-related disorders.

Building PROMIS item banks: librarians as co-investigators

PURPOSE

There is growing interest in the use of item response theory (IRT) for creation of measures of health-related quality of life (HRQOL). A first step in IRT modeling is development of item banks. Our aim is to describe the value of including librarians and to describe processes used by librarians, in the creation of such banks.

METHOD

Working collaboratively with PROMIS researchers at the University of Pittsburgh, a team of librarians designed and implemented comprehensive literature searches in a selected set of information resources, for the purpose of identifying existing measures of patient-reported emotional distress.

RESULTS

A step-by-step search protocol developed by librarians produced a set of 525 keywords and controlled vocabulary terms for use in search statements in 3 bibliographic databases. These searches produced 6,169 literature citations, allowing investigators to add 444 measurement scales to their item banks.

CONCLUSION

Inclusion of librarians on the Pittsburgh PROMIS research team allowed investigators to create large initial item banks, increasing the likelihood that the banks would attain high measurement precision during subsequent psychometric analyses. In addition, a comprehensive literature search protocol was developed that can now serve as a guide for other investigators in the creation of IRT item banks.

Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy

OBJECTIVE

To describe the quality and methods of systematic reviews of physiotherapy interventions, compare Cochrane and non-Cochrane reviews, and establish the interrater reliability of the Overview Quality Assessment Questionnaire (OQAQ) quality assessment tool.

STUDY DESIGN AND SETTING

A survey of 200 published systematic reviews was done. Two independent raters assessed the search strategy, assessment of trial quality, outcomes, pooling, conclusions, and overall quality (OQAQ). The study was carried out in the University research center.

RESULTS

In these reviews, the five most common databases searched were MEDLINE, EMBASE, Cochrane Library, CINAHL, and Cochrane Review Group Registers. The Cochrane allocation concealment system and Jadad Scale were most frequently used to assess trial quality. Cochrane reviews searched more databases and were more likely to assess trial quality, report dichotomous outcomes for individual trials, and conduct a meta-analysis than non-Cochrane reviews. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews were of higher quality than non-Cochrane reviews. There has been an increase in the quality of systematic reviews over time. The OQAQ has fair to good interrater reliability.

CONCLUSION

The quality of systematic reviews in physiotherapy is improving, and the use of Cochrane Collaboration procedures appears to improve the methods and quality.

Need for quality improvement in renal systematic reviews

BACKGROUND AND OBJECTIVES

Systematic reviews of clinical studies aim to compile best available evidence for various diagnosis and treatment options. This study assessed the methodologic quality of all systematic reviews relevant to the practice of nephrology published in 2005.

DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS

We searched electronic databases (Medline, Embase, American College of Physicians Journal Club, Cochrane) and hand searched Cochrane renal group records. Clinical practice guidelines, case reports, narrative reviews, and pooled individual patient data meta-analyses were excluded. Methodologic quality was measured using a validated questionnaire (Overview Quality Assessment Questionnaire). For reviews of randomized trials, we also evaluated adherence to recommended reporting guidelines (Quality of Reporting of Meta-Analyses).

RESULTS

Ninety renal systematic reviews were published in year 2005, 60 of which focused on therapy. Many systematic reviews (54%) had major methodologic flaws. The most common review flaws were failure to assess the methodologic quality of included primary studies and failure to minimize bias in study inclusion. Only 2% of reviews of randomized trials fully adhered to reporting guidelines. A minority of journals (four of 48) endorsed adherence to consensus guidelines for review reporting, and these journals published systematic reviews of higher methodologic quality (P < 0.001).

CONCLUSIONS

The majority of systematic reviews had major methodologic flaws. The majority of journals do not endorse consensus guidelines for review reporting in their instructions to authors; however, journals that recommended such adherence published systemic reviews of higher methodologic quality.

Primer on systematic review and meta-analysis

A systematic review uses an explicitly defined process to comprehensively identify all studies pertaining to a specific focused question, appraise the methods of the studies, summarize the results, identify reasons for different findings across studies, and cite limitations of current knowledge. Meta-analyses usually combine aggregate-level data reported in each primary study, which may provide a more precise estimate of the "true effect" than any individual study. However, the conclusions may be limited by between-trial heterogeneity, publication bias, or deficits in the conduct or reporting of individual primary studies.

State of the evidence on acute asthma management in children: a critical appraisal of systematic reviews

OBJECTIVE

Our goal was to evaluate clinical, methodologic, and reporting aspects of systematic reviews on the management of acute asthma in children.

METHODS

We undertook a systematic review of systematic reviews on acute asthma management in children. We identified eligible reviews by searching the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, Medline, and Embase 1990 to March 2006. Data were extracted on clinical issues, methodologic characteristics, and results of the reviews. Methodologic quality was assessed with the Overview Quality Assessment Questionnaire and with additional questions on heterogeneity. Separate reporting on children in mixed adult-pediatric population reviews was assessed. Methodologic quality of systematic reviews published in peer-reviewed journals was compared with Cochrane reviews.

RESULTS

A total of 23 systematic reviews were included: 14 were published in the Cochrane Library, and 9 were published in peer-reviewed journals. Eight reviews included children only, and 15 were mixed-population reviews. The majority of reviews defined the study population as having "acute asthma" without a more precise definition, and 16 different health outcomes were reported. The overall quality according to the Overview Quality Assessment Questionnaire was good, with Cochrane reviews showing minimal flaws and journal reviews showing minor flaws (median scores: 7 vs 5). Results on children were reported separately in 8 of 15 mixed-population reviews. Clinical heterogeneity was explored in only 2 of 23 reviews, and the methods used to identify and address heterogeneity were diverse.

CONCLUSIONS

The methodologic quality of both the Cochrane and journal reviews on the management of acute asthma in children seems good, with Cochrane reviews being more rigorous. However, their usefulness for clinical practice is hampered by a lack of clear definitions of included populations, clinically important health outcomes, and separate reporting on children in mixed reviews. A major threat to these reviews' validity is the insufficient identification and handling of heterogeneity.

A systematic review evaluating the potential for bias and the methodological quality of meta-analyses in vaccinology

A systematic review was undertaken to produce an annotated bibliography of meta-analyses in vaccinology and to evaluate their methodological quality. Based on our evaluation using the Oxman and Guyatt index, the methodological quality of the 121 meta-analyses included in this study is not satisfactory. The most frequent limitations include non-comprehensive bibliographic research; bias in the selection of the studies; lack of quality assessment of individual studies; absence of evaluation of heterogeneity among studies and publication bias. The methodological quality significantly increases with the year of publication and with declared financial support, without differences between profit and non-profit support. Meta-analyses with a higher Oxman and Guyatt quality score are more likely to include only randomized trials and to explore appropriately potential sources of heterogeneity. Most of the methodological deficiencies of meta-analyses in vaccinology could be corrected easily, and meta-analysts should improve the methodological quality of their work to maintain their impact on policy decisions.

Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects

OBJECTIVE

Systematic reviews incorporating adverse effects are assuming increasing importance as questions raised extend beyond clinical effectiveness to all effects (beneficial and harmful). The aim of this study was to survey the methods used to identify relevant studies for systematic reviews of adverse effects.

STUDY DESIGN AND SETTING

All records within the Database of Abstracts of Reviews of Effects and the Cochrane Database of Systematic Reviews were scanned for systematic reviews in which the primary outcomes were adverse effects. Two information professionals independently assessed the methods used to identify relevant research as reported in the 277 reviews that met the inclusion criteria.

RESULTS

A major weakness of the reviews was inadequate reporting of the search strategies used. In addition, of the reviews that did report a search strategy, few used the sensitive search strategies recommended for systematic reviews. The majority of reviews did not search more than one or two databases, and few other methods of identifying information were used.

CONCLUSION

This investigation shows the variation in the searching element of systematic reviews of adverse effects and demonstrates that the reporting of the methods used to identify research in such reviews could be vastly improved.

Bibliographic study showed improving methodology of meta-analyses published in leading journals 1993–2002

OBJECTIVE

To assess the methodology of meta-analyses published in leading general and specialist medical journals over a 10-year period.

STUDY DESIGN AND SETTING

Volumes 1993-2002 of four general medicine journals and four specialist journals were searched by hand for meta-analyses including at least five controlled trials. Characteristics were assessed using a standardized questionnaire.

RESULTS

A total of 272 meta-analyses, which included a median of 11 trials (range 5-195), were assessed. Most (81%) were published in general medicine journals. The median (range) number of databases searched increased from 1 (1-9) in 1993/1994 to 3.5 (1-21) in 2001/2002, P<0.0001. The proportion of meta-analyses including searches by hand (10% in 1993/1994, 25% in 2001/2002, P=0.005), searches of the grey literature (29%, 51%, P=0.010 by chi-square test), and of trial registers (10%, 32%, P=0.025) also increased. Assessments of the quality of trials also became more common (45%, 70%, P=0.008), including whether allocation of patients to treatment groups had been concealed (24%, 60%, P=0.001). The methodological and reporting quality was consistently higher in general medicine compared to specialist journals.

CONCLUSION

Many meta-analyses published in leading journals have important methodological limitations. The situation has improved in recent years but considerable room for further improvements remains.

Cochrane Skin Group systematic reviews are more methodologically rigorous than other systematic reviews in dermatology

BACKGROUND

The Cochrane collaboration aims to produce high-quality systematic reviews. It is not known whether the methods used in producing Cochrane Skin Group (CSG) reviews result in higher quality reviews than other systematic reviews in dermatology.

OBJECTIVES

To determine how the methodological quality of dermatological CSG reviews published in The Cochrane Library and in peer-reviewed journals compare with non-Cochrane systematic reviews.

METHODS

Two blinded investigators independently assessed review quality using the 10-item Oxman and Guyatt scale.

RESULTS

Thirty-eight systematic reviews (17 Cochrane reviews published in The Cochrane Library, 11 Cochrane reviews published in peer-reviewed journals and 10 non-Cochrane reviews published in peer-reviewed journals) were examined. The Cochrane Library reviews included quality of life (11/17 vs. 1/10, P = 0.014) and adverse outcomes (14/17 vs. 2/10, P = 0.003) more often than non-Cochrane reviews published in peer-reviewed journals. Cochrane reviews published in both peer-reviewed journals and The Cochrane Library were more likely to include comprehensive search strategies (11/11 and 17/17 vs. 6/10, P-values = 0.04 and 0.01), take steps to minimize selection bias (11/11 and 16/17 vs. 3/10, P-values = 0.003 and 0.001) and appropriately assess the validity of all included trials (10/11 and 16/17 vs. 4/10, P-values = 0.04 and 0.007) than non-Cochrane reviews. Overall, Cochrane reviews published both in peer-reviewed journals and in The Cochrane Library were assigned higher quality scores by reviewers than non-Cochrane reviews (median = 6.0 and 6.5 vs. 4.5, P-values = 0.01 and 0.002).

CONCLUSIONS

The Cochrane Library systematic review methodology leads to higher quality reviews on dermatological topics.

The physical environment and physical activity: a critical appraisal of review articles

BACKGROUND

Over the last few years an increasing number of studies investigating the association between the physical environment and physical activity have been published. Many reviews have also summarized this emerging body of research, and such review papers are frequently used by public health policymakers and researchers themselves to inform decision making.

METHODS

This paper systematically appraises methodologic aspects of literature reviews examining the relationship between physical activity and the physical environment published in peer-reviewed journals between 2000 and 2005. Eleven reviews and their antecedent source papers were examined.

RESULTS

The majority of these reviews omitted between one third and two thirds of the studies that could have been eligible for inclusion at the time they conducted the review. Methodologic information on how the review was conducted was not always provided. Furthermore, in some cases results of a study were reported incorrectly, or physical environmental aspects were conflated with social environmental or cognitive factors. Moreover, when results were reported incorrectly, physical environmental variables were almost always reported as significantly associated with physical activity, when these associations were nonsignificant, or were not assessed as part of the primary study.

CONCLUSIONS

Users of reviews in this field should be aware that there are significant methodologic variations among them, and that some reviews may include only a sample of the relevant primary studies. However, this is difficult to determine given the frequent incompleteness of review method reporting. Greater standardization in the reporting of review methods may assist with future efforts to summarize studies of the relationship between physical environments and physical activity.

The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal

OBJECTIVE

To independently appraise the methodological quality of a sample of reports of meta-analyses that address critical care topics in the Cochrane Database of Systematic Reviews compared with the quality of reports published in regular journals, using a validated assessment instrument, the Overview Quality Assessment Questionnaire (OQAQ).

DATA SOURCE

Studies were selected from a search of MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews from 1994 to 2003, using multiple search terms for critical care and sensitive filters to identify meta-analyses.

STUDY SELECTION

Two authors independently selected meta-analyses that addressed topics pertinent to critical care medicine.

DATA EXTRACTION

Two authors independently extracted the data. The proportion of reports that met each component of the OQAQ was determined, as was the overall quality score. Meta-analyses published in the Cochrane Database of Systematic Reviews were compared with those published in regular journals.

DATA SYNTHESIS

There were 36 reports of meta-analyses in the Cochrane Database of Systematic Reviews and 103 reports of meta-analyses published in regular journals; 11 of these were reports of Cochrane reviews. The meta-analyses published in the Cochrane Database of Systematic Reviews were more likely to fulfill most components of the OQAQ. The median overall OQAQ scores indicated significant methodological problems in the reports regardless of the source of publication, although the reports in the Cochrane database scored higher than those in regular journals (five compared with two, p<.001). Major methodological flaws, notably failure to appropriately refer to the validity of included studies, were found in meta-analyses in both the Cochrane Database of Systematic Reviews and regular journals (44.4% and 79.3%, respectively).

CONCLUSIONS

Although the quality of reports of meta-analyses published in the Cochrane Database of Systematic Reviews is superior to the quality of reports of meta-analyses published in regular journals, there is significant room for improvement. Clinicians should critically appraise all reports of meta-analyses before considering the results, regardless of the source of publication.

Knowledge Translation: Closing the Evidence-to-Practice Gap

Knowledge translation describes any activity or process that facilitates the transfer of high-quality evidence from research into effective changes in health policy, clinical practice, or products. This increasingly important discipline attempts to conceptually combine elements of research, education, quality improvement, and electronic systems development to create a seamless linkage between interventions that improve patient care and their routine implementation in daily clinical practice. We outline the gap between research and practice and present a case study of an emergency medicine example of validated evidence that has failed to achieve widespread implementation. The authors describe a model of organization of evidence and its relationship with the process that links research from the scientific endeavor to changes in practice that affect patient outcomes. Obstacles to evidence uptake are explored, as well as the limitations of current educational strategies. Innovative strategies in realms such as computerized decision support systems designed to enhance evidence uptake are also described. The potential interface between knowledge translation and continuous quality improvement, as well as the role for bedside tools, is also presented. Research in knowledge translation includes studies that attempt to quantify and understand the discrepancies between what is known and what is done, as well as those that examine the impact and acceptability of interventions designed to narrow or close these gaps. Sentinel examples in this line of research conducted in the emergency department setting are described.

Limited evidence for effects of diet for type 2 diabetes from systematic reviews

OBJECTIVE

Systematic reviews are an appraised method to summarize research in a concise and transparent way, and may enable to draw conclusions beyond the sum of results of individual studies. We assessed the results, quality and external validity of systematic reviews on diet in patients with type 2 diabetes.

DESIGN, SETTING, SUBJECTS

We systematically searched for systematic reviews on nutritional interventions in patients with type 2 diabetes that used a reproducible search strategy in at least one major database that applied some form of quality assessment. We assessed quality and the external validity of the retrieved systematic reviews. Outcomes were defined as statistical meta-analyses or narrative results using a predefined and reproducible method.

RESULTS

Six systematic reviews met the inclusion criteria, investigating dietary interventions in general (n=3), chromium supplementation (n=1), fish-oil (n=1) or herbs and nutrition supplements (n=1). Quality assessment showed minimal/minor flaws in four cases and major/extensive flaws in two cases. All reviews had insufficient data needed to judge external validity. In reviews with minimal/minor flaws, we found beneficial effects of very-low-calorie diets and fish-oil supplements. However, the external validity of these results could not be assessed sufficiently.

CONCLUSIONS

Systematic reviews largely failed to produce knowledge beyond the sum of the original studies. Furthermore, judgment of external validity was hampered in most cases owing to missing data. To improve the quality and usefulness of systematic reviews of dietary interventions, we recommend the application of more focused research questions, but with broader inclusion criteria, for example, the use of observational studies.

SPONSORSHIP

Internal funding Radboud University Nijmegen Medical Centre.