Identification of meta-analyses. The need for standard terminology

Search strategies (filters) to identify systematic reviews in MEDLINE and Embase

BACKGROUND

Bibliographic databases provide access to an international body of scientific literature in health and medical sciences. Systematic reviews are an important source of evidence for clinicians, researchers, consumers, and policymakers as they address a specific health-related question and use explicit methods to identify, appraise and synthesize evidence from which conclusions can be drawn and decisions made. Methodological search filters help database end-users search the literature effectively with different levels of sensitivity and specificity. These filters have been developed for various study designs and have been found to be particularly useful for intervention studies. Other filters have been developed for finding systematic reviews. Considering the variety and number of available search filters for systematic reviews, there is a need for a review of them in order to provide evidence about their retrieval properties at the time they were developed.

OBJECTIVES

To review systematically empirical studies that report the development, evaluation, or comparison of search filters to retrieve reports of systematic reviews in MEDLINE and Embase.

SEARCH METHODS

We searched the following databases from inception to January 2023: MEDLINE, Embase, PsycINFO; Library, Information Science & Technology Abstracts (LISTA) and Science Citation Index (Web of Science).

SELECTION CRITERIA

We included studies if one of their primary objectives is the development, evaluation, or comparison of a search filter that could be used to retrieve systematic reviews on MEDLINE, Embase, or both.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data using a pre-specified and piloted data extraction form using InterTASC Information Specialist Subgroup (ISSG) Search Filter Evaluation Checklist.

MAIN RESULTS

We identified eight studies that developed filters for MEDLINE and three studies that developed filters for Embase. Most studies are very old and some were limited to systematic reviews in specific clinical areas. Six included studies reported the sensitivity of their developed filter. Seven studies reported precision and six studies reported specificity. Only one study reported the number needed to read and positive predictive value. None of the filters were designed to differentiate systematic reviews on the basis of their methodological quality. For MEDLINE, all filters showed similar sensitivity and precision, and one filter showed higher levels of specificity. For Embase, filters showed variable sensitivity and precision, with limited study reports that may affect accuracy assessments. The report of these studies had some limitations, and the assessments of their accuracy may suffer from indirectness, considering that they were mostly developed before the release of the PRISMA 2009 statement or due to their limited scope in the selection of systematic review topics. Search filters for MEDLINE Three studies produced filters with sensitivity > 90% with variable degrees of precision, and only one of them was developed and validated in a gold-standard database, which allowed the calculation of specificity. The other two search filters had lower levels of sensitivity. One of these produced a filter with higher levels of specificity (> 90%). All filters showed similar sensitivity and precision in the external validation, except for one which was not externally validated and another one which was conceptually derived and only externally validated. Search filters for Embase We identified three studies that developed filters for this database. One of these studies developed filters with variable sensitivity and precision, including highly sensitive strategies (> 90%); however, it was not externally validated. The other study produced a filter with a lower sensitivity (72.7%) but high specificity (99.1%) with a similar performance in the external validation.

AUTHORS' CONCLUSIONS

Studies reporting the development, evaluation, or comparison of search filters to retrieve reports of systematic reviews in MEDLINE showed similar sensitivity and precision, with one filter showing higher levels of specificity. For Embase, filters showed variable sensitivity and precision, with limited information about how the filter was produced, which leaves us uncertain about their performance assessments. Newer filters had limitations in their methods or scope, including very focused subject topics for their gold standards, limiting their applicability across other topics. Our findings highlight that consensus guidance on the conduct of search filters and standardized reporting of search filters are needed, as we found highly heterogeneous development methods, accuracy assessments and outcome selection. New strategies adaptable across interfaces could enhance their usability. Moreover, the performance of existing filters needs to be evaluated in light of the impact of reporting guidelines, including the PRISMA 2009, on how systematic reviews are reported. Finally, future filter developments should also consider comparing the filters against a common reference set to establish comparative performance and assess the quality of systematic reviews retrieved by strategies.

Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design

Background: systematic reviews are of increasing importance to health care professionals seeking to provide evidence-based health care, because they provide systematically prepared summaries of the current state of research knowledge on the effectiveness of health care interventions. To be able to make use of them, both researchers preparing systematic reviews and health care professionals need to be able to identify systematic reviews easily. In the past, systematic reviews have been difficult to identify easily among the mass of literature labelled ‘reviews’. Search strategies have been published which were intended to provide easier access to systematic reviews and meta-analyses. These strategies, mostly devised for the MEDLINE database, have been constructed using the authors’ knowledge of the subject area, the databases and past experience.

Objective: the objective of this study was to design search strategies based on a more objective approach to strategy construction.

Methods: word frequencies from the titles, abstracts and subject keywords of a collection of systematic reviews of the effective health care interventions were analysed to derive a highly sensitive search strategy.

Results: the proposed strategy offers 98% sensitivity in retrieving systematic reviews, while retaining a low but acceptable level of precision (20%). Other strategies with other levels of sensitivity and precision are also presented.

Discussion: this study shows that a frequency analysis approach can be used to generate highly sensitive strategies which retain adequate levels of precision when retrieving systematic reviews.

Systematic Review Checklist: A Standardized Technique for Assessing and Reporting Reviews of Life Cycle Assessment Data

Systematic review, including meta-analysis, is increasingly utilized in life cycle assessment (LCA). There are currently no widely recognized guidelines for designing, conducting, or reporting systematic reviews in LCA. Other disciplines such as medicine, ecology, and software engineering have both recognized the utility of systematic reviews and created standardized protocols for conducting and reporting systematic reviews. Based largely on the 2009 Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, which updated the preferred format for reporting of such reviews in biomedical research, we provide an introduction to the topic and a checklist to guide the reporting of future LCA reviews in a standardized format. The standardized technique for assessing and reporting reviews of LCA (STARR-LCA) checklist is a starting point for improving the utility of systematic reviews in LCA.

The risk of oral contraceptives in the etiology of inflammatory bowel disease: a meta-analysis

OBJECTIVES

Several environmental and genetic factors have been implicated to date in the development of Crohn's disease (CD) and ulcerative colitis (UC). The aim of this study was to provide a quantification of the risk of oral contraceptive pill (OCP) use in the etiology of inflammatory bowel disease.

METHODS

A literature search was performed to identify comparative studies reporting on the association of oral contraceptive use in the etiology of UC and CD between 1983 and 2007. A random-effect meta-analysis was used to compare the incidence of UC or CD between the patients exposed to the OCP and nonexposed patients. The results were adjusted for smoking.

RESULTS

A total of 75,815 patients were reported on by 14 studies, with 36,797 exposed to OCP and 39,018 nonexposed women. The pooled relative risk (RR) for CD for women currently taking the OCP was 1.51 (95% confidence interval [CI] 1.17-1.96, P= 0.002), and 1.46 (95% CI 1.26-1.70, P < 0.001), adjusted for smoking. The RR for UC in women currently taking the OCP was 1.53 (95% CI 1.21-1.94, P= 0.001), and 1.28 (95% CI 1.06-1.54, P= 0.011), adjusted for smoking. The RR for CD increased with the length of exposure to OCP. Moreover, although the RR did not reduce once the OCP was stopped, it was no longer significant once the OCP was stopped (CI contains 1), both for CD and for UC.

CONCLUSIONS

This study provides evidence of an association between the use of oral contraceptive agents and development of IBD, in particular CD. The study also suggests that the risk for patients who stop using the OCP reverts to that of the nonexposed population.

Optimal search strategies for retrieving systematic reviews from Medline: analytical survey

OBJECTIVE

To develop optimal search strategies in Medline for retrieving systematic reviews.

DESIGN

Analytical survey.

DATA SOURCES

161 journals published in 2000 indexed in Medline.

MAIN OUTCOME MEASURES

The sensitivity, specificity, and precision of retrieval of systematic reviews of 4862 unique terms in 782,485 combinations of one to five terms were determined by comparison with a hand search of all articles (the criterion standard) in 161 journals published during 2000 (49,028 articles).

RESULTS

Only 753 (1.5%) of the 49,028 articles were systematic reviews. The most sensitive strategy included five terms and had a sensitivity of 99.9% (95% confidence interval 99.6% to 100%) and a specificity of 52% (51.6% to 52.5%). The strategy that best minimised the difference between sensitivity and specificity had a sensitivity of 98% (97% to 99%) and specificity of 90.8% (90.5% to 91.1%). Highest precision for multiterm strategies, 57% (54% to 60%), was achieved at a sensitivity of 71% (68% to 74%). The term "cochrane database of systematic reviews.jn." was the most precise single term search strategy (sensitivity of 56% (52% to 60%) and precision of 96% (94% to 98%)). These strategies are available through the "limit" screen of Ovid's search interface for Medline.

CONCLUSIONS

Systematic reviews can be retrieved from Medline with close to perfect sensitivity or specificity, or with high precision, by using empirical search strategies.

Systematic reviews: a cross-sectional study of location and citation counts

BACKGROUND

Systematic reviews summarize all pertinent evidence on a defined health question. They help clinical scientists to direct their research and clinicians to keep updated. Our objective was to determine the extent to which systematic reviews are clustered in a large collection of clinical journals and whether review type (narrative or systematic) affects citation counts.

METHODS

We used hand searches of 170 clinical journals in the fields of general internal medicine, primary medical care, nursing, and mental health to identify review articles (year 2000). We defined 'review' as any full text article that was bannered as a review, overview, or meta-analysis in the title or in a section heading, or that indicated in the text that the intention of the authors was to review or summarize the literature on a particular topic. We obtained citation counts for review articles in the five journals that published the most systematic reviews.

RESULTS

11% of the journals concentrated 80% of all systematic reviews. Impact factors were weakly correlated with the publication of systematic reviews (R2 = 0.075, P = 0.0035). There were more citations for systematic reviews (median 26.5, IQR 12 - 56.5) than for narrative reviews (8, 20, P <.0001 for the difference). Systematic reviews had twice as many citations as narrative reviews published in the same journal (95% confidence interval 1.5 - 2.7).

CONCLUSIONS

A few clinical journals published most systematic reviews. Authors cited systematic reviews more often than narrative reviews, an indirect endorsement of the 'hierarchy of evidence'.

Meta-analysis of MFT interventions

This article briefly reviews 20 meta-analyses of marital and family interventions. These meta-analyses support the efficacy of both MFT for distressed couples, and martial and family enrichment. Those effects are slightly reduced at follow-up, but still significant. Differences among kinds of marital and family interventions tend to be small. MFT produce clinically significant results in 40-50% of those treated, but the effects of MFT in clinically representative settings have not been much studied. The article also introduces the concept of meta-analytically supported treatments (MASTs), which are treatments that meet certain criteria for efficacy in meta-analysis, and which remedy certain problems in the empirically supported treatment (EST) literature. The article concludes with recommendations for doing better meta-analyses.

The growth of health-related meta-analyses published from 1980 to 2000

A MEDLINE search was conducted to estimate trends in the growth of health-related meta-analyses published during the past two decades. Employing a more specific than sensitive search strategy, and not supplementing these results with known sources of published meta-analyses or manual search strategies, 3,025 probable meta-analyses were selected from the 5,128 citations identified. The data showed a definitive upward (and generally linear) trend across time with no evidence for this genre of research either leveling off or decreasing.

Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. QUOROM Group

BACKGROUND

The Quality of Reporting of Meta-analyses (QUOROM) conference was convened to address standards for improving the quality of reporting of meta-analyses of clinical randomised controlled trials (RCTs).

METHODS

The QUOROM group consisted of 30 clinical epidemiologists, clinicians, statisticians, editors, and researchers. In conference, the group was asked to identify items they thought should be included in a checklist of standards. Whenever possible, checklist items were guided by research evidence suggesting that failure to adhere to the item proposed could lead to biased results. A modified Delphi technique was used in assessing candidate items.

FINDINGS

The conference resulted in the QUOROM statement, a checklist, and a flow diagram. The checklist describes our preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. It is organised into 21 headings and subheadings regarding searches, selection, validity assessment, data abstraction, study characteristics, and quantitative data synthesis, and in the results with 'trial flow', study characteristics, and quantitative data synthesis; research documentation was identified for eight of the 18 items. The flow diagram provides information about both the numbers of RCTs identified, included, and excluded and the reasons for exclusion of trials.

INTERPRETATION

We hope this report will generate further thought about ways to improve the quality of reports of meta-analyses of RCTs and that interested readers, reviewers, researchers, and editors will use the QUOROM statement and generate ideas for its improvement.

What contributions do languages other than English make on the results of meta-analyses?

Including only a portion of all available evidence may introduce systematic errors into the meta-analytic process and threaten its validity. We set out to examine whether language restricted meta-analyses, compared to language inclusive meta-analyses, provide different estimates of the effectiveness of interventions evaluated in randomized trials. We identified and retrieved all 79 meta-analyses from several disease areas in which explicit eligibility criteria regarding trial selection were reported. General characteristics and quality of reporting of the meta-analyses were assessed using a validated instrument. We explored the effects of language of publication of the randomized trials on the quantitative results using logistic regression analyses. Language restricted meta-analyses, compared to language inclusive meta-analyses, did not differ with respect to the estimate of benefit of the effectiveness of an intervention (ROR = 0.98; 95% CI: 0.81-1.17). These results were also robust after a series of sensitivity analyses. This study provides no evidence that language restricted meta-analyses lead to biased estimates of intervention effectiveness. We encourage others to replicate this study using different sampling frames, clinical topics and interventions.

Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses

BACKGROUND

The Quality of Reporting of Meta-analyses (QUOROM) conference was convened to address standards for improving the quality of reporting of meta-analyses of clinical randomised controlled trials (RCTs).

METHODS

The QUOROM group consisted of 30 clinical epidemiologists, clinicians, statisticians, editors, and researchers. In conference, the group was asked to identify items they thought should be included in a checklist of standards. Whenever possible, checklist items were guided by research evidence suggesting that failure to adhere to the item proposed could lead to biased results. A modified Delphi technique was used in assessing candidate items.

FINDINGS

The conference resulted in the QUOROM statement, a checklist, and a flow diagram. The checklist describes our preferred way to present the abstract, introduction, methods, results, and discussion sections of a report of a meta-analysis. It is organised into 21 headings and subheadings regarding searches, selection, validity assessment, data abstraction, study characteristics, and quantitative data synthesis, and in the results with "trial flow", study characteristics, and quantitative data synthesis; research documentation was identified for eight of the 18 items. The flow diagram provides information about both the numbers of RCTs identified, included, and excluded and the reasons for exclusion of trials.

INTERPRETATION

We hope this report will generate further thought about ways to improve the quality of reports of meta-analyses of RCTs and that interested readers, reviewers, researchers, and editors will use the QUOROM statement and generate ideas for its improvement.

Standardization of terminology in meta-analysis: a proposal for working definitions

In this paper we present a brief overview of the growing concern to standardize definitions and terminology in meta-analysis. This tool has become inescapable in both drug research and therapeutic evaluation. The performed and published meta-analyses are increasing, as well as the variation in the meaning of the terms used in meta-analysis. In the second part of the paper we propose glossary of the most common terms used in reports of meta-analyses. The glossary has been written by only one group of scientists, the definitions are therefore proposed to the scientific community as working definitions, to be subject to reactions from leaders in meta-analysis.

Meta-analysis and prospective meta-analysis in childhood leukemia clinical research

In this paper, we consider the role of meta-analysis and 'prospective meta-analysis' studies in childhood acute lymphocytic leukemia (ALL). In this issue, Valsecchi and Masera [1] give a thoughtful discourse, generally favorable to this approach. This article presents the opposite point of view. The aims of our article are to present the implications in clinical, rather than biostatistical terms, and to provide an extensive literature review of the subject of meta-analysis. We conclude that treatment assessments, resulting from meta-analysis of closed studies (retrospective) should be met with healthy skepticism. Trials requiring international resources should be true intergroup trials with a single coordinating center, rather than prospective meta-analysis, unless it is a question grafted onto each group's own research agenda. For example, each group might ask its own systemic control question, but a CNS protection question is asked collectively.

Lithium compared to valproic acid and carbamazepine in the treatment of mania: a statistical meta-analysis

The hypothesis that lithium is the most appropriate and efficacious pharmacological treatment in the therapy of manic-depressive illness was examined in a meta-analysis. The efficacy of lithium was compared with that of the antiepileptics, carbamazepine and valproic acid. Data sources include the Medline database and relevant references from articles. Only randomised controlled clinical trials with a double-blind assessment of outcomes were included. Lithium levels were required to be within the therapeutic range of 0.4-1.5 mmol/l. The effect sizes were measured by the odds ratio using the Mantel-Haenszel method. No significant difference was observed between the treatment effect of the 3 drugs compared. While the efficacy of the 3 treatments was identical, there is a general tendency in favour of the antiepileptics in the treatment of manic depression with regard to adverse events and treatment tolerance. The impact and importance of this study seriously question the relevance of today's practice of prescribing lithium in the acute management of manic-depressive illness.

Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology

A systematic search of the literature was performed to identify the maximum possible number of meta-analyses that evaluated analgesic interventions. Seventy-four reports were identified and retrieved and the scientific quality of 80 separate meta-analyses was assessed under blind conditions by 2 judges using Oxman and Guyatt's index. Most of the meta-analyses evaluated pharmacological interventions for chronic pain conditions and two-thirds were published since 1990. Ninety percent of the meta-analyses had methodological flaws that could limit their validity. The main deficiencies were lack of information on methods to retrieve and to assess the validity of primary studies and lack of data on the design of the primary studies. Meta-analyses of low quality produced significantly more positive conclusions. For several topics, different meta-analyses evaluating the same intervention produced conflicting results. The need to resolve these contradictions is highlighted.

The growth of meta-analytic literature from 1980 to 1993

A search of the literature was undertaken to estimate the number of meta-analytic studies that have been published in the social and health sciences to date. Altogether, 1,874 published quantitative reviews were located by the end of 1994: 892 in the health sciences, 982 in the social sciences. Trends from 1980 to 1993 indicate that, while the social sciences embraced the technique earlier, health researchers are now publishing almost twice as many meta-analyses as their social science colleagues.

A review of therapeutic studies of idiopathic membranous glomerulopathy

The treatment of idiopathic membranous glomerulopathy remains an enigma. We have reviewed many of the important clinical trials concerning membranous glomerulopathy using a meta-analysis and a secondary pooled analysis to test the effects of corticosteroid or alkylating, therapy compared with no treatment on renal survival and complete remission of the nephrotic syndrome. A search was performed using MEDLINE (1968 through 1993) for articles on idiopathic membranous glomerulopathy and glomerulonephritis. Bibliographies of articles were reviewed for completeness. Sixty-nine articles were reviewed. Meta-analysis was performed for four trials that evaluated corticosteroids compared with no treatment and for three trials that evaluated alkylating therapy compared with no treatment. Pooled analysis was performed on randomized and prospective studies (10 studies) and then with 22 case series added. All studies evaluated renal biopsy-proven disease. Meta-analysis was performed on the relative chance of being in complete remission for each study. Renal survival could be evaluated by pooled analysis only. For pooled analyses, Cox's proportional hazard and logistic regression models were used to test the effect of therapy on renal survival and the nephrotic syndrome, respectively. Data concerning gender, nephrotic syndrome, and geographic region were used in all statistical models. Evaluation of renal survival revealed no differences by treatment group (P > 0.1). By meta-analysis, the relative chance of complete remission was not improved for corticosteroid-treated patients (1.55; 95% confidence interval, 0.99 to 2.44; P > 0.1), but was improved for patients treated with alkylating agents (4.8; 95% confidence interval, 1.44 to 15.96; P < 0.05) when compared with no treatment. Pooled analysis of randomized and prospective studies, as well pooled analysis with all studies, supported the findings of the meta-analysis. Corticosteroids or alkylating therapy did not improve renal survival in idiopathic membranous glomerulopathy. Complete remission of the nephrotic syndrome was observed more frequently with the use of alkylating agents.

Reporting, updating, and correcting systematic reviews of the effects of health care

The recent growth in the numbers of published systematic reviews reflects growing recognition of their importance for improving knowledge about the effects of health care. In Britain the NHS R&D Programme has established two centres to prepare systematic reviews of existing information, and the Cochrane Collaboration--an international network of individuals and institutions--evolved to produce systematic, periodically updated reviews of randomised controlled trials. The large amount of existing evidence that needs to be considered creates a problem for the reporting of systematic reviews: the need to ensure that methods and results of systematic reviews are adequately described has to be reconciled with the limited space available in printed journals. A possible solution is the use of electronic publications: reviews could be published simultaneously in a short, printed form and in a more detailed electronic form. Electronic publications also have the advantage of the ease with which reviews may be updated as new evidence becomes available or mistakes are identified.

The statistical basis of meta-analysis

Two models for study-to-study variation in a meta-analysis are presented, critiqued and illustrated. One, the fixed effects model, takes the studies being analysed as the universe of interest; the other, the random effects model, takes these studies as representing a sample from a larger population of possible studies. With emphasis on clinical trials, this paper illustrates in some detail the application of both models to three summary measures of the effect of an experimental intervention versus a control: the standardized difference for comparing two means, and the relative risk and odds ratio for comparing two proportions.

[Meta-analysis in epidemiology (1): the general characteristics]

In the present work the general characteristics, aims and applications of meta-analysis in public health are described. The general rules and the relevant steps for the development of this type of studies are considered. These points are illustrated with a number of real examples and the advantages and limitations of the different methodological options are commented.

Validation of an index of the quality of review articles

The objective of this study was to assess the validity of an index of the scientific quality of research overviews, the Overview Quality Assessment Questionnaire (OQAQ). Thirty-six published review articles were assessed by 9 judges using the OQAQ. Authors reports of what they had done were compared to OQAQ ratings. The sensibility of the OQAQ was assessed using a 13 item questionnaire. Seven a priori hypotheses were used to assess construct validity. The review articles were drawn from three sampling frames: articles highly rated by criteria external to the study, meta-analyses, and a broad spectrum of medical journals. Three categories of judges were used to assess the articles: research assistants, clinicians with research training and experts in research methodology, with 3 judges in each category. The sensibility of the index was assessed by 15 randomly selected faculty members of the Department of Clinical Epidemiology and Biostatistics at McMaster. Authors' reports of their methods related closely to ratings from corresponding OQAQ items: for each criterion, the mean score was significantly higher for articles for which the authors responses indicated that they had used more rigorous methods. For 10 of the 13 questions used to assess sensibility the mean rating was 5 or greater, indicating general satisfaction with the instrument. The primary shortcoming noted was the need for judgement in applying the index. Six of the 7 hypotheses used to test construct validity held true. The OQAQ is a valid measure of the quality of research overviews.