No consensus exists on search reporting methods for systematic reviews

Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study

OBJECTIVES

To determine the reproducibility of biomedical systematic review search strategies.

STUDY DESIGN AND SETTING

A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records.

RESULTS

The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible.

CONCLUSION

Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.

Natural Language Processing Applications in the Clinical Neurosciences: A Machine Learning Augmented Systematic Review

Natural language processing (NLP), a domain of artificial intelligence (AI) that models human language, has been used in medicine to automate diagnostics, detect adverse events, support decision making and predict clinical outcomes. However, applications to the clinical neurosciences appear to be limited. NLP has matured with the implementation of deep transformer models (e.g., XLNet, BERT, T5, and RoBERTa) and transfer learning. The objectives of this study were to (1) systematically review NLP applications in the clinical neurosciences, and (2) explore NLP analysis to facilitate literature synthesis, providing clear examples to demonstrate the potential capabilities of these technologies for a clinical audience. Our NLP analysis consisted of keyword identification, text summarization and document classification. A total of 48 articles met inclusion criteria. NLP has been applied in the clinical neurosciences to facilitate literature synthesis, data extraction, patient identification, automated clinical reporting and outcome prediction. The number of publications applying NLP has increased rapidly over the past five years. Document classifiers trained to differentiate included and excluded articles demonstrated moderate performance (XLNet AUC = 0.66, BERT AUC = 0.59, RoBERTa AUC = 0.62). The T5 transformer model generated acceptable abstract summaries. The application of NLP has the potential to enhance research and practice in the clinical neurosciences.

A new method for testing reproducibility in systematic reviews was developed, but needs more testing

BACKGROUND

To develop and test an approach to test reproducibility of SRs.

METHODS

Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review.

RESULTS

We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory.

CONCLUSION

Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.

Quality of Literature Searches Published in Leading Neurosurgical Journals: A Review of Reviews

BACKGROUND

There is mounting evidence that the search strategies upon which systematic reviews (SRs) are based frequently contain errors are incompletely reported or insensitive.

OBJECTIVE

To appraise the quality of search strategies in the 10 leading specialty neurosurgical journals and identify factors associated with superior searches.

METHODS

This research-on-research study systematically surveyed SRs published in the 10 leading neurosurgical journals between 01/10/2017 and 31/10/2019. All SRs were eligible for assessment using a predefined coding manual that was adapted from the preferred reporting items for systematic reviews and meta-analyses (PRISMA), a measurement tool to assess systematic reviews (AMSTAR), and Cochrane Collaboration guidelines. The PubMed interface was used to search the MEDLINE database, which was supplemented by individual journal searches. Descriptive statistics were utilized to identify factors associated with improved search strategies.

RESULTS

A total of 633 articles were included and contained a median of 19.00 (2.00-1654.00) studies. Less than half (45.97%) of included search strategies were considered to be reproducible. Aggregated reporting score was positively associated with in-text reference to reporting guideline adherence (τb = 0.156, P < .01). The number of articles retrieved by a search (τb = 0.11, P < .01) was also associated with the reporting of a reproducible search strategy.

CONCLUSION

This study demonstrates that the search strategies used in neurosurgical SRs require improvement. In addition to increasing awareness of reporting standards, we propose that this be achieved by the incorporation of PRISMA and other guidelines into article submission and peer-review processes. This may lead to the conduct of more informative SRs, which may result in improved clinician decision-making and patient outcomes.

Clinical outcomes associated with robotic and computer-navigated total knee arthroplasty: a machine learning-augmented systematic review

BACKGROUND

Robotic (RTKA) and computer-navigated total knee arthroplasty (CNTKA) are increasingly replacing manual techniques in orthopaedic surgery. This systematic review compared clinical outcomes associated with RTKA and CNTKA and investigated the utility of natural language processing (NLP) for the literature synthesis.

METHODS

A comprehensive search strategy was implemented. Results of included studies were combined and analysed. A transfer learning approach was applied to train deep NLP classifiers (BERT, RoBERTa and XLNet), with cross-validation, to partially automate the systematic review process.

RESULTS

52 studies were included, comprising 5,067 RTKA and 2,108 CNTKA. Complication rates were 0-22% and 0-16% and surgical time was 70-116 and 77-102 min for RTKA and CNTKA, respectively. Technical failures were more commonly associated with RTKA (8%) than CNTKA (2-4%). Patient satisfaction was equivalent (94%). RTKA was associated with a higher likelihood of achieving target alignment, less femoral notching, shorter operative time and shorter length of stay. NLP models demonstrated moderate performance (AUC = 0.65-0.68).

CONCLUSIONS

RTKA and CNTKA appear to be associated with similarly positive clinical outcomes. Further work is required to determine whether the two techniques differ significantly with regard to specific outcome measures. NLP shows promise for facilitating the systematic review process.

Machine learning applications to neuroimaging for glioma detection and classification: An artificial intelligence augmented systematic review

Glioma is the most common primary intraparenchymal tumor of the brain and the 5-year survival rate of high-grade glioma is poor. Magnetic resonance imaging (MRI) is essential for detecting, characterizing and monitoring brain tumors but definitive diagnosis still relies on surgical pathology. Machine learning has been applied to the analysis of MRI data in glioma research and has the potential to change clinical practice and improve patient outcomes. This systematic review synthesizes and analyzes the current state of machine learning applications to glioma MRI data and explores the use of machine learning for systematic review automation. Various datapoints were extracted from the 153 studies that met inclusion criteria and analyzed. Natural language processing (NLP) analysis involved keyword extraction, topic modeling and document classification. Machine learning has been applied to tumor grading and diagnosis, tumor segmentation, non-invasive genomic biomarker identification, detection of progression and patient survival prediction. Model performance was generally strong (AUC = 0.87 ± 0.09; sensitivity = 0.87 ± 0.10; specificity = 0.0.86 ± 0.10; precision = 0.88 ± 0.11). Convolutional neural network, support vector machine and random forest algorithms were top performers. Deep learning document classifiers yielded acceptable performance (mean 5-fold cross-validation AUC = 0.71). Machine learning tools and data resources were synthesized and summarized to facilitate future research. Machine learning has been widely applied to the processing of MRI data in glioma research and has demonstrated substantial utility. NLP and transfer learning resources enabled the successful development of a replicable method for automating the systematic review article screening process, which has potential for shortening the time from discovery to clinical application in medicine.

PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. J Med Libr Assoc 2021;109:174-200. [PMID: 34285662 PMCID: PMC8270366 DOI: 10.5195/jmla.2021.962]

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a three-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes sixteen reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and, therefore, reproducible.

PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse, and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a 3-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes 16 reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and therefore reproducible.

Machine learning applications to clinical decision support in neurosurgery: an artificial intelligence augmented systematic review

Machine learning (ML) involves algorithms learning patterns in large, complex datasets to predict and classify. Algorithms include neural networks (NN), logistic regression (LR), and support vector machines (SVM). ML may generate substantial improvements in neurosurgery. This systematic review assessed the current state of neurosurgical ML applications and the performance of algorithms applied. Our systematic search strategy yielded 6866 results, 70 of which met inclusion criteria. Performance statistics analyzed included area under the receiver operating characteristics curve (AUC), accuracy, sensitivity, and specificity. Natural language processing (NLP) was used to model topics across the corpus and to identify keywords within surgical subspecialties. ML applications were heterogeneous. The densest cluster of studies focused on preoperative evaluation, planning, and outcome prediction in spine surgery. The main algorithms applied were NN, LR, and SVM. Input and output features varied widely and were listed to facilitate future research. The accuracy (F_(2,19) = 6.56, p < 0.01) and specificity (F_(2,16) = 5.57, p < 0.01) of NN, LR, and SVM differed significantly. NN algorithms demonstrated significantly higher accuracy than LR. SVM demonstrated significantly higher specificity than LR. We found no significant difference between NN, LR, and SVM AUC and sensitivity. NLP topic modeling reached maximum coherence at seven topics, which were defined by modeling approach, surgery type, and pathology themes. Keywords captured research foci within surgical domains. ML technology accurately predicts outcomes and facilitates clinical decision-making in neurosurgery. NNs frequently outperformed other algorithms on supervised learning tasks. This study identified gaps in the literature and opportunities for future neurosurgical ML research.

lilacs search strategy for systematic reviews of diagnostic test accuracy studies

BACKGROUND

There are few publications on search strategies to identify diagnostic test accuracy (DTA) studies in lilacs.

OBJECTIVE

To translate and customise medline search strategies for use in lilacs and assess their retrieval of studies in Cochrane DTA systematic reviews.

METHOD

We developed a six-step process to translate and customise medline search strategies for use in lilacs (iAHx interface). We identified medline search strategies of published Cochrane DTA reviews, translated/customised them for use in lilacs, ran searches in lilacs and compared the retrieval results of our translated search strategy versus the one used in the published reviews.

RESULTS

Our lilacs search strategies translated/customised from the medline strategies retrieved studies in 70 Cochrane DTA reviews. Only 29 of these reviews stated that they had searched the lilacs database and 21 published their lilacs search strategies. Few had used the lilacs database search tools, none exploded the subject headings, and 86% used only English terms.

CONCLUSION

Translating and tailoring a medline search strategy for the lilacs database resulted in the retrieval of DTA studies that would have been missed otherwise.

Errors in search strategies used in systematic reviews and their effects on information retrieval

OBJECTIVES

Errors in search strategies negatively affect the quality and validity of systematic reviews. The primary objective of this study was to evaluate searches performed in MEDLINE/PubMed to identify errors and determine their effects on information retrieval.

METHODS

A PubMed search was conducted using the systematic review filter to identify articles that were published in January of 2018. Systematic reviews or meta-analyses were selected from a systematic search for literature containing reproducible and explicit search strategies in MEDLINE/PubMed. Data were extracted from these studies related to ten types of errors and to the terms and phrases search modes.

RESULTS

The study included 137 systematic reviews in which the number of search strategies containing some type of error was very high (92.7%). Errors that affected recall were the most frequent (78.1%), and the most common search errors involved missing terms in both natural language and controlled language and those related to Medical Subject Headings (MeSH) search terms and the non-retrieval of their more specific terms.

CONCLUSIONS

To improve the quality of searches and avoid errors, it is essential to plan the search strategy carefully, which includes consulting the MeSH database to identify the concepts and choose all appropriate terms, both descriptors and synonyms, and combining search techniques in the free-text and controlled-language fields, truncating the terms appropriately to retrieve all their variants.

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

BACKGROUND

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence. Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

METHOD

A literature review. Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through 'pearl growing', citation chasing, a search of PubMed using the systematic review methods filter, and the authors' topic knowledge. The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of 'key stages' in the process of literature searching.

RESULTS

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

CONCLUSIONS

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

The methodological quality of systematic reviews indexed in the MEDLINE database

Purpose

The purpose of this paper is to describe and analyse the methodology quality of the literature search protocols of systematic reviews and to assess the relevance of the search filter that applies PubMed for retrieving this type of publication of the MEDLINE database.

Design/methodology/approach

For the selection of the document type, a literature search about nutritional and metabolic diseases was carried out in MEDLINE and the PubMed filter was used for retrieving “Systematic Reviews”, selecting “Nutritional and Metabolic Diseases” from the Medical Subject Headings (MeSH) database as Major Topic to determine the area of knowledge. Data analysis was carried out using “External Logistic Biplot”, a novel multivariate statistical technique in the field of medical documentation.

Findings

The results highlight the large variability of the methodology used in the literature search protocols of the systematic reviews analysed and confirm the low precision of the filter used by PubMed for the recovery of systematic reviews.

Originality/value

The Logistic Biplot used in this research allows an optimal categorization of the different documentary typologies and classifies the documents by their methodological quality, demonstrating its usefulness for the future development of the bibliometric analysis.

Acute and Chronic Effects of Endurance Running on Inflammatory Markers: A Systematic Review

In order to understand the effect of endurance running on inflammation, it is necessary to quantify the extent to which acute and chronic running affects inflammatory mediators. The aim of this study was to summarize the literature on the effects of endurance running on inflammation mediators. Electronic searches were conducted on PubMED and Science Direct with no limits of date and language of publication. Randomized controlled trials (RCTs) and non-randomized controlled trials (NRCTs) investigating the acute and chronic effects of running on inflammation markers in runners were reviewed by two researchers for eligibility. The modified Downs and Black checklist for the assesssments of the methodological quality of studies was subsequently used. Fifty-one studies were finally included. There were no studies with elite athletes. Only two studies were chronic interventions. Results revealed that acute and chronic endurance running may affect anti- and pro-inflammatory markers but methodological differences between studies do not allow comparisons or generalization of the results. The information provided in this systematic review would help practitioners for better designing further studies while providing reference values for a better understanding of inflammatory responses after different running events. Further longitudinal studies are needed to identify the influence of training load parameters on inflammatory markers in runners of different levels and training background.

Compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) literature search reporting guidelines

Objective

Complete, accurate reporting of systematic reviews facilitates assessment of how well reviews have been conducted. The primary objective of this study was to examine compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines for literature search reporting and to examine the completeness, bias, and reproducibility of the searches in these reviews from what was reported. The second objective was to examine reporting of the credentials and contributions of those involved in the search process.

Methods

A sample of systematic reviews or meta-analyses published in veterinary journals between 2011 and 2015 was obtained by searching PubMed. Reporting in the full text of each review was checked against certain PRISMA checklist items.

Results

Over one-third of reviews (37%) did not search the CAB Abstracts database, and 9% of reviews searched only 1 database. Over two-thirds of reviews (65%) did not report any search for grey literature or stated that they excluded grey literature. The majority of reviews (95%) did not report a reproducible search strategy.

Conclusions

Most reviews had significant deficiencies in reporting the search process that raise questions about how these searches were conducted and ultimately cast serious doubts on the validity and reliability of reviews based on a potentially biased and incomplete body of literature. These deficiencies also highlight the need for veterinary journal editors and publishers to be more rigorous in requiring adherence to PRISMA guidelines and to encourage veterinary researchers to include librarians or information specialists on systematic review teams to improve the quality and reporting of searches.

VP200 Untangling What Information Specialists Should Document and Report

INTRODUCTION:

Thorough documentation and clear reporting are essential when conducting a comprehensive literature search for a health technology assessment (HTA) or systematic review. The ultimate goal of this process is transparency and reproducibility with the added benefit of increasing the reader's confidence in the research. Thorough documentation of the search also allows for critical appraisal of the methodology used and facilitates future updating of a review (1,2).

It has been found that large numbers of systematic review searches are inadequately documented and there is little consensus on best practices for reporting standards (3).

As part of the SuRe Info Project, we conducted a review of all current reporting standards relevant to HTAs and systematic reviews in addition to looking at the published literature on this topic in order to synthesize the evidence in this area and create a standard set of agreed upon recommendations.

METHODS:

We conducted a comprehensive search of Medline, Embase, and LISA (Library & Info Studies Abstracts) databases. We also examined the Equator Network (http://www.equator-network.org/) website. Reference lists of included studies and reporting guidelines were also consulted. Eleven reporting guidelines and eight studies were included in the review by two independent reviewers. Anything published before 2006, that was not a research article (other than the guidelines), and/or that did not provide new recommendations (that is, a review of another set of recommendations) was excluded.

RESULTS:

After collecting data on the suggested reporting elements described in the literature, we pooled our results to create an overarching list of the most commonly recommended elements to describe and the most commonly recommended methods to use when documenting a comprehensive search. Not only did these elements pertain to documenting the search strategy for the final report, but they also pertained to the protocol and the abstract of a review.

CONCLUSIONS:

It is hoped that this overview of the literature and compilation of the evidence will clarify some of the confusion that seems to exist when documenting and reporting searches and perhaps it will even help to reduce the existence of poorly described strategies in the research literature.

Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study

Background

A high-quality search strategy is considered an essential component of systematic reviews but many do not contain reproducible search strategies. It is unclear if low reproducibility spans medical disciplines, is affected by librarian/search specialist involvement or has improved with increased awareness of reporting guidelines.

Objectives

To examine the reporting of search strategies in systematic reviews published in Pediatrics, Surgery or Cardiology journals in 2012 and determine rates and predictors of including a reproducible search strategy.

Methods

We identified all systematic reviews published in 2012 in the ten highest impact factor journals in Pediatrics, Surgery and Cardiology. Each search strategy was coded to indicate what elements were reported and whether the overall search was reproducible. Reporting and reproducibility rates were compared across disciplines and we measured the influence of librarian/search specialist involvement, discipline or endorsement of a reporting guideline on search reproducibility.

Results

272 articles from 25 journals were included. Reporting of search elements ranged widely from 91% of articles naming search terms to 33% providing a full search strategy and 22% indicating the date the search was executed. Only 22% of articles provided at least one reproducible search strategy and 13% provided a reproducible strategy for all databases searched in the article. Librarians or search specialists were reported as involved in 17% of articles. There were strong disciplinary differences on the reporting of search elements. In the multivariable analysis, only discipline (Pediatrics) was a significant predictor of the inclusion of a reproducible search strategy.

Conclusions

Despite recommendations to report full, reproducible search strategies, many articles still do not. In addition, authors often report a single strategy as covering all databases searched, further decreasing reproducibility. Further research is needed to determine how disciplinary culture may encourage reproducibility and the role that journal editors and peer reviewers could play.

PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement

OBJECTIVE

To develop an evidence-based guideline for Peer Review of Electronic Search Strategies (PRESS) for systematic reviews (SRs), health technology assessments, and other evidence syntheses.

STUDY DESIGN AND SETTING

An SR, Web-based survey of experts, and consensus development forum were undertaken to identify checklists that evaluated or validated electronic literature search strategies and to determine which of their elements related to search quality or errors.

RESULTS

Systematic review: No new search elements were identified for addition to the existing (2008-2010) PRESS 2015 Evidence-Based Checklist, and there was no evidence refuting any of its elements. Results suggested that structured PRESS could identify search errors and improve the selection of search terms. Web-based survey of experts: Most respondents felt that peer review should be undertaken after the MEDLINE search had been prepared but before it had been translated to other databases. Consensus development forum: Of the seven original PRESS elements, six were retained: translation of the research question; Boolean and proximity operators; subject headings; text word search; spelling, syntax and line numbers; and limits and filters. The seventh (skilled translation of the search strategy to additional databases) was removed, as there was consensus that this should be left to the discretion of searchers. An updated PRESS 2015 Guideline Statement was developed, which includes the following four documents: PRESS 2015 Evidence-Based Checklist, PRESS 2015 Recommendations for Librarian Practice, PRESS 2015 Implementation Strategies, and PRESS 2015 Guideline Assessment Form.

CONCLUSION

The PRESS 2015 Guideline Statement should help to guide and improve the peer review of electronic literature search strategies.

Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery

OBJECTIVE

Systematic reviews require comprehensive literature search strategies to avoid publication bias. This study aimed to assess and evaluate the reporting quality of search strategies within systematic reviews published in the field of stereotactic radiosurgery (SRS).

METHODS

Three electronic databases (Ovid MEDLINE(®), Ovid EMBASE(®) and the Cochrane Library) were searched to identify systematic reviews addressing SRS interventions, with the last search performed in October 2014. Manual searches of the reference lists of included systematic reviews were conducted. The search strategies of the included systematic reviews were assessed using a standardized nine-question form based on the Cochrane Collaboration guidelines and Assessment of Multiple Systematic Reviews checklist. Multiple linear regression analyses were performed to identify the important predictors of search quality.

RESULTS

A total of 85 systematic reviews were included. The median quality score of search strategies was 2 (interquartile range = 2). Whilst 89% of systematic reviews reported the use of search terms, only 14% of systematic reviews reported searching the grey literature. Multiple linear regression analyses identified publication year (continuous variable), meta-analysis performance and journal impact factor (continuous variable) as predictors of higher mean quality scores.

CONCLUSION

This study identified the urgent need to improve the quality of search strategies within systematic reviews published in the field of SRS.

ADVANCES IN KNOWLEDGE

This study is the first to address how authors performed searches to select clinical studies for inclusion in their systematic reviews. Comprehensive and well-implemented search strategies are pivotal to reduce the chance of publication bias and consequently generate more reliable systematic review findings.

Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors

Background

Previous research looking at published systematic reviews has shown that their search strategies are often suboptimal and that librarian involvement, though recommended, is low. Confidence in the results, however, is limited due to poor reporting of search strategies the published articles.

Objectives

To more accurately measure the use of recommended search methods in systematic reviews, the levels of librarian involvement, and whether librarian involvement predicts the use of recommended methods.

Methods

A survey was sent to all authors of English-language systematic reviews indexed in the Database of Abstracts of Reviews of Effects (DARE) from January 2012 through January 2014. The survey asked about their use of search methods recommended by the Institute of Medicine, Cochrane Collaboration, and the Agency for Healthcare Research and Quality and if and how a librarian was involved in the systematic review. Rates of use of recommended methods and librarian involvement were summarized. The impact of librarian involvement on use of recommended methods was examined using a multivariate logistic regression.

Results

1560 authors completed the survey. Use of recommended search methods ranged widely from 98% for use of keywords to 9% for registration in PROSPERO and were generally higher than in previous studies. 51% of studies involved a librarian, but only 64% acknowledge their assistance. Librarian involvement was significantly associated with the use of 65% of recommended search methods after controlling for other potential predictors. Odds ratios ranged from 1.36 (95% CI 1.06 to 1.75) for including multiple languages to 3.07 (95% CI 2.06 to 4.58) for using controlled vocabulary.

Conclusions

Use of recommended search strategies is higher than previously reported, but many methods are still under-utilized. Librarian involvement predicts the use of most methods, but their involvement is under-reported within the published article.

Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews

OBJECTIVES

To determine whether librarian and information specialist authorship was associated with better reported systematic review (SR) search quality.

STUDY DESIGN AND SETTING

SRs from high-impact general internal medicine journals were reviewed for search quality characteristics and reporting quality by independent reviewers using three instruments, including a checklist of Institute of Medicine Recommended Standards for the Search Process and a scored modification of the Peer Review of Electronic Search Strategies instrument.

RESULTS

The level of librarian and information specialist participation was significantly associated with search reproducibility from reported search strategies (Χ(2) = 23.5; P < 0.0001). Librarian co-authored SRs had significantly higher odds of meeting 8 of 13 analyzed search standards than those with no librarian participation and six more than those with mentioned librarian participation. One-way ANOVA showed that differences in total search quality scores between all three groups were statistically significant (F2,267 = 10.1233; P < 0.0001).

CONCLUSION

Problems remain with SR search quality and reporting. SRs with librarian or information specialist co-authors are correlated with significantly higher quality reported search strategies. To minimize bias in SRs, authors and editors could encourage librarian engagement in SRs including authorship as a potential way to help improve documentation of the search strategy.

Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews

BACKGROUND

Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews.

OBJECTIVES

To compare search methods in systematic reviews of adverse effects with other reviews.

METHODS

The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews.

RESULTS

Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings.

CONCLUSIONS

The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.

Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000-2010): are the searches clearly explained, systematic and reproducible?

Systematic reviews are an essential tool for researchers, prevention providers and policy makers who want to remain current with the evidence in the field. Systematic review must adhere to strict standards, as the results can provide a more objective appraisal of evidence for making scientific decisions than traditional narrative reviews. An integral component of a systematic review is the development and execution of a comprehensive systematic search to collect available and relevant information. A number of reporting guidelines have been developed to ensure quality publications of systematic reviews. These guidelines provide the essential elements to include in the review process and report in the final publication for complete transparency. We identified the common elements of reporting guidelines and examined the reporting quality of search methods in HIV behavioral intervention literature. Consistent with the findings from previous evaluations of reporting search methods of systematic reviews in other fields, our review shows a lack of full and transparent reporting within systematic reviews even though a plethora of guidelines exist. This review underscores the need for promoting the completeness of and adherence to transparent systematic search reporting within systematic reviews.

Methods for documenting systematic review searches: a discussion of common issues

INTRODUCTION

As standardized reporting requirements for systematic reviews are being adopted more widely, review authors are under greater pressure to accurately record their search process. With careful planning, documentation to fulfill the Preferred Reporting Items for Systematic Reviews and Meta-Analyses requirements can become a valuable tool for organizing a systematic review literature search and planning updates.

METHODS

A working group of information specialists convened to discuss current practice and were informed by a Web-based survey of over 260 systematic review authors, trials search coordinators, librarians, and other information specialists conducted in February/March 2011.

DISCUSSION

Survey responses provided insight into current practices and difficulties of reporting searches. These included a lack of time, tools, clear understanding of the requirements, and uncertainty about responsibility for documenting these elements. This paper will present some of the practical aspects of documenting the systematic literature search. Section 1 provides background information and rationale for this paper. Section 2 discusses issues and recommendations arising from survey results. Section 3 outlines specific elements to be recorded. Section 4 guides the reader through the information management process. Section 5 concludes with implications for future research and practice. These principles are applicable to any large literature search for systematic reviews, health technology assessments, and guideline development.

Methodological developments in searching for studies for systematic reviews: past, present and future?

The Cochrane Collaboration was established in 1993, following the opening of the UK Cochrane Centre in 1992, at a time when searching for studies for inclusion in systematic reviews was not well-developed. Review authors largely conducted their own searches or depended on medical librarians, who often possessed limited awareness and experience of systematic reviews. Guidance on the conduct and reporting of searches was limited. When work began to identify reports of randomized controlled trials (RCTs) for inclusion in Cochrane Reviews in 1992, there were only approximately 20,000 reports indexed as RCTs in MEDLINE and none indexed as RCTs in Embase. No search filters had been developed with the aim of identifying all RCTs in MEDLINE or other major databases. This presented The Cochrane Collaboration with a considerable challenge in identifying relevant studies.Over time, the number of studies indexed as RCTs in the major databases has grown considerably and the Cochrane Central Register of Controlled Trials (CENTRAL) has become the best single source of published controlled trials, with approximately 700,000 records, including records identified by the Collaboration from Embase and MEDLINE. Search filters for various study types, including systematic reviews and the Cochrane Highly Sensitive Search Strategies for RCTs, have been developed. There have been considerable advances in the evidence base for methodological aspects of information retrieval. The Cochrane Handbook for Systematic Reviews of Interventions now provides detailed guidance on the conduct and reporting of searches. Initiatives across The Cochrane Collaboration to improve the quality inter alia of information retrieval include: the recently introduced Methodological Expectations for Cochrane Intervention Reviews (MECIR) programme, which stipulates 'mandatory' and 'highly desirable' standards for various aspects of review conduct and reporting including searching, the development of Standard Training Materials for Cochrane Reviews and work on peer review of electronic search strategies. Almost all Cochrane Review Groups and some Cochrane Centres and Fields now have a Trials Search Co-ordinator responsible for study identification and medical librarians and other information specialists are increasingly experienced in searching for studies for systematic reviews.Prospective registration of clinical trials is increasing and searching trials registers is now mandatory for Cochrane Reviews, where relevant. Portals such as the WHO International Clinical Trials Registry Platform (ICTRP) are likely to become increasingly attractive, given concerns about the number of trials which may not be registered and/or published. The importance of access to information from regulatory and reimbursement agencies is likely to increase. Cross-database searching, gateways or portals and improved access to full-text databases will impact on how searches are conducted and reported, as will services such as Google Scholar, Scopus and Web of Science. Technologies such as textual analysis, semantic analysis, text mining and data linkage will have a major impact on the search process but efficient and effective updating of reviews may remain a challenge.In twenty years' time, we envisage that the impact of universal social networking, as well as national and international legislation, will mean that all trials involving humans will be registered at inception and detailed trial results will be routinely available to all. Challenges will remain, however, to ensure the discoverability of relevant information in diverse and often complex sources and the availability of metadata to provide the most efficient access to information. We envisage an ongoing role for information professionals as experts in identifying new resources, researching efficient ways to link or mine them for relevant data and managing their content for the efficient production of systematic reviews.

Are systematic reviews up-to-date at the time of publication?

BACKGROUND

Systematic reviews provide a synthesis of evidence for practitioners, for clinical practice guideline developers, and for those designing and justifying primary research. Having an up-to-date and comprehensive review is therefore important. Our main objective was to determine the recency of systematic reviews at the time of their publication, as measured by the time from last search date to publication. We also wanted to study the time from search date to acceptance, and from acceptance to publication, and measure the proportion of systematic reviews with recorded information on search dates and information sources in the abstract and full text of the review.

METHODS

A descriptive analysis of published systematic reviews indexed in Medline in 2009, 2010 and 2011 by three reviewers, independently extracting data.

RESULTS

Of the 300 systematic reviews included, 271 (90%) provided the date of search in the full-text article, but only 141 (47%) stated this in the abstract. The median (standard error; minimum to maximum) survival time from last search to acceptance was 5.1 (0.58; 0 to 43.8) months (95% confidence interval = 3.9 to 6.2) and from last search to first publication time was 8.0 (0.35; 0 to 46.7) months (95% confidence interval = 7.3 to 8.7), respectively. Of the 300 reviews, 295 (98%) stated which databases had been searched, but only 181 (60%) stated the databases in the abstract. Most researchers searched three (35%) or four (21%) databases. The top-three most used databases were MEDLINE (79%), Cochrane library (76%), and EMBASE (64%).

CONCLUSIONS

Being able to identify comprehensive, up-to-date reviews is important to clinicians, guideline groups, and those designing clinical trials. This study demonstrates that some reviews have a considerable delay between search and publication, but only 47% of systematic review abstracts stated the last search date and 60% stated the databases that had been searched. Improvements in the quality of abstracts of systematic reviews and ways to shorten the review and revision processes to make review publication more rapid are needed.

Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011

OBJECTIVE

An increasing amount of research and guidelines has been published on search methodology and the reporting of search strategies in systematic reviews. This research assessed whether this has lead to any improvements in the reporting and quality of searching in systematic reviews of adverse effects.

STUDY DESIGN AND SETTING

All records within Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects were scanned for systematic reviews of adverse effects. Data were then extracted on the methods used for information retrieval in these reviews and a descriptive analysis conducted by publication year.

RESULTS

A total of 849 reviews published from 1994 to 2011 met the inclusion criteria. There has been a significant increase (P<0.001) in the number of adverse effects reviews per year from 1994 (n=5) to 2010 (n=104). Some improvements were apparent, such as an increase in the number of databases searched and fewer date and language restrictions applied. However, there has been an increase in reviews limited to data from randomized controlled trials, whereas the reporting of search strategies could still be improved further, with only 9% (74/849) of the reviews reporting reproducible searches.

CONCLUSION

Some improvements in searching systematic reviews of adverse effects are apparent; however, poor reporting of search strategies remains a great obstacle to readers.

Expert searching in health librarianship: a literature review to identify international issues and Australian concerns

BACKGROUND

The traditional role of health librarians as expert searchers is under challenge.

OBJECTIVES

The purpose of this review is to establish health librarians' views, practices and educational processes on expert searching.

METHODS

The search strategy was developed in LISTA and then customised for ten other databases: ALISA, PubMed, Embase, Scopus, Web of Science, CINAHL, ERIC, PsycINFO, Cochrane Library and Google Scholar. The search terms were (expert search* OR expert retriev* OR mediated search* OR information retriev*) AND librar*. The searches, completed in December 2010 and repeated in May 2011, were limited to English language publications from 2000 to 2011 (unless seminal works).

RESULTS

Expert searching remains a key role for health librarians, especially for those supporting systematic reviews or employed as clinical librarians answering clinical questions.

CONCLUSIONS

Although clients tend to be satisfied with searches carried out for them, improvements are required to effectively position the profession. Evidence-based guidelines, adherence to transparent standards, review of entry-level education requirements and a commitment to accredited, rigorous, ongoing professional development will ensure best practice.

Complementary medicine and safety: a systematic investigation of design and reporting of systematic reviews

OBJECTIVES

The objective of this study was to examine the methods used in systematic reviews of safety across a range of complementary therapies to assess the variation in approach and the potential for developing guidance on conduct and reporting.

DESIGN

Systematic reviews focusing on safety were retrieved from NHS Evidence and searches of major databases. A pre-prepared template was used for data extraction. Information extracted included details of search strategies, sources, participants, interventions, reported adverse event/effect(s) and causality assessment. Data extraction was carried out by one researcher and a check for accuracy by a second researcher. Methods were assessed against criteria based on guidance provided by the Cochrane Adverse Effects Methods Group.

RESULTS

A total of 2563 citations were screened and 88 systematic reviews were selected for inclusion. The majority focused on the safety of herbs and nutritional supplements. Approximately half the reviews covered all aspects of safety; other reviews addressed specific adverse effects or interactions. Types of data included in the reviews did not always reflect the focus of the review. Search strategies, sources used, quality assessment and assessment of causality also varied.

CONCLUSIONS

Detailed examination and comparison of the methods has highlighted several areas in which there is potential for development of guidelines and consensus on standards. These include search strategies, sources of information, data extraction and assessment of causality. The value of systematic reviews in relation to large outcome studies requires further consideration.

Sublingual immunotherapy in children with allergic rhinitis: quality of systematic reviews

Systematic reviews have gained popularity as a way to combine the increasing amount of research information. This study assessed the quality of systematic reviews and meta-analyses of sublingual immunotherapy (SLIT) for allergic rhinitis in children, published since 2000. Eligible reviews were identified by searching Medline/Pubmed, Embase, and the Cochrane Library, from 2000 through 2008. Methodological quality was assessed using the assessment of multiple systematic reviews instrument. Ten systematic reviews were included, one of which was published in the Cochrane Library. Eight reviews gave some details about the search strategy. None of the reviews included measures to avoid selection bias. In 60% of the reviews, the methodological quality of the included studies was (partly) assessed. Four reviews pooled the results of individual studies, neglecting clinical heterogeneity. Three of the 10 reviews provided information about sources of funding or grants from industry. Of the 10 reviews, the six reviews with the highest overall score scored 5-8 points, indicating moderate quality. Systematic reviews are useful to evaluate the efficacy of SLIT in children. Although more reviews have become available, the methodological quality could be improved. SLIT for children could be promising, but methodological flaws in the reviews and individual studies are too serious to draw definite conclusions.

Checking reference lists to find additional studies for systematic reviews

BACKGROUND

Checking reference lists to identify relevant studies for systematic reviews is frequently recommended by systematic review manuals and is often undertaken by review authors. To date, no systematic review has explicitly examined the effectiveness of checking reference lists as a method to supplement electronic searching.

OBJECTIVES

To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.

SEARCH STRATEGY

We searched the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science abstracts (LISA) (1969 to July 2008) and MEDLINE (1966 to July 2008). We contacted experts in systematic review methods and examined reference lists of articles.

SELECTION CRITERIA

Studies of any design which examined checking reference lists as a search method for systematic reviews in any area. The primary outcome was the additional yield of relevant studies (i.e. studies not found through any other search methodologies); other outcomes were publication types identified and data pertaining to the costs (e.g. cost-effectiveness, cost-efficiency) of checking reference lists.

DATA COLLECTION AND ANALYSIS

We summarized data descriptively.

MAIN RESULTS

We included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias. The additional yield (calculated by dividing the additional 'unique' yield identified by checking reference lists by the total number of studies found to be eligible within the study) of relevant studies identified through checking reference lists ranged from 2.5% to 42.7%. Only two studies reported yield information by publication type (dissertations and systematic reviews). No cost data were reported although one study commented that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper, and for that particular study costs were not specifically estimated.

AUTHORS' CONCLUSIONS

There is some evidence to support the use of checking reference lists for locating studies in systematic reviews. However, this evidence is derived from weak study designs. In situations where the identification of all relevant studies through handsearching and database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. The challenge, therefore, is for review authors to recognize those situations.

Finding evidence for comparing medical interventions: AHRQ and the Effective Health Care Program

OBJECTIVE

This article discusses search methodology to identify evidence for comparative effectiveness reviews (CERs) as practiced by the Effective Health Care program.

STUDY DESIGN AND SETTING

Review.

RESULTS

Search methods described attempt to overcome the bias inherent in the publication and distribution of clinical evidence. Bibliographic databases and search strategies are discussed with special emphasis on searching for observational studies and harms data. Other techniques described include the use of key articles, citation tracking, hand searching, and personal communications. Strategies for locating gray literature, such as clinical trial protocols and regulatory information, are described. Search reporting and other practical matters are also discussed.

CONCLUSION

Better reporting and further research on search strategies is needed to develop additional evidence-based recommendations.

Risk factors for neck and upper extremity disorders among computers users and the effect of interventions: an overview of systematic reviews

BACKGROUND

To summarize systematic reviews that 1) assessed the evidence for causal relationships between computer work and the occurrence of carpal tunnel syndrome (CTS) or upper extremity musculoskeletal disorders (UEMSDs), or 2) reported on intervention studies among computer users/or office workers.

METHODOLOGY/PRINCIPAL FINDINGS

PubMed, Embase, CINAHL and Web of Science were searched for reviews published between 1999 and 2010. Additional publications were provided by content area experts. The primary author extracted all data using a purpose-built form, while two of the authors evaluated the quality of the reviews using recommended standard criteria from AMSTAR; disagreements were resolved by discussion. The quality of evidence syntheses in the included reviews was assessed qualitatively for each outcome and for the interventions. Altogether, 1,349 review titles were identified, 47 reviews were retrieved for full text relevance assessment, and 17 reviews were finally included as being relevant and of sufficient quality. The degrees of focus and rigorousness of these 17 reviews were highly variable. Three reviews on risk factors for carpal tunnel syndrome were rated moderate to high quality, 8 reviews on risk factors for UEMSDs ranged from low to moderate/high quality, and 6 reviews on intervention studies were of moderate to high quality. The quality of the evidence for computer use as a risk factor for CTS was insufficient, while the evidence for computer use and UEMSDs was moderate regarding pain complaints and limited for specific musculoskeletal disorders. From the reviews on intervention studies no strong evidence based recommendations could be given.

CONCLUSIONS/SIGNIFICANCE

Computer use is associated with pain complaints, but it is still not very clear if this association is causal. The evidence for specific disorders or diseases is limited. No effective interventions have yet been documented.

Where there is no evidence: use of expert consensus methods to fill the evidence gap in low-income countries and cultural minorities

BACKGROUND

In both developing countries and in relation to cultural minorities there have been calls to scale up mental health services and for evidence-informed policy and practice.

EVIDENCE BASED MEDICINE

The evidence based medicine movement has had a major influence in improving practice. However, implementation of this approach has some major difficulties. One that has been neglected is the situation where there is no relevant evidence. This situation is more likely to occur for healthcare decisions in developing countries or for cultural minorities within developed countries, because resources do not exist for expensive research studies.

CONSENSUS METHODS

Consensus methods, such as the Delphi process, can be useful in providing an evidence base in situations where there is insufficient evidence. They provide a way of systematically tapping the expertise of people working in the area and give evidence that is readily applicable for a particular country and culture. Although consensus methods are often thought of as low in the hierarchy of evidence, consensus is central to the scientific process. We present four examples where the Delphi method was used to assess expert consensus in situations where no other evidence existed: estimating the prevalence of dementia in developing countries, developing mental health first aid guidelines in Asian countries, mental health first aid guidelines for Australian Aboriginal people, and modification of the concept of 'recovery' for Australian immigrant communities.

CONCLUSION

Consensus methods can provide a basis for decision-making and considered action when there is no evidence or when there are doubts about the applicability of evidence that has been generated from other populations or health system settings.

Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment

BACKGROUND

Information retrieval (IR) in health technology assessment (HTA) calls for transparency and reproducibility, but common practice in the documentation and presentation of this process is inadequate in fulfilling this demand.

OBJECTIVES

Our objective is to promote good IR practice by presenting the conceptualization of retrieval and transcription readable to non-information specialists, and reporting of effectively processed search strategies.

METHODS

We performed a comprehensive database search (04/2010) to synthesize the current state-of-the-art. We then developed graphical and tabular presentation methods and tested their feasibility on existing research questions and defined recommendations.

RESULTS

No generally accepted standard of reporting of IR in HTA exists. We, therefore, developed templates for presenting the retrieval conceptualization, database selection, and additional hand-searching as well as for presenting search histories of complex and lengthy search strategies. No single template fits all conceptualizations, but some can be applied to most processes. Database interface providers report queries as entered, not as they are actually processed. In PubMed, the huge difference between entered and processed query is shown in "Details." Quality control and evaluation of search strategies using a validated tool such as the PRESS checklist is suboptimal when only entry-query based search histories are applied.

CONCLUSIONS

Moving toward an internationally accepted IR reporting standard calls for advances in common reporting practices. Comprehensive, process-based reporting and presentation would make IR more understandable to others than information specialists and facilitate quality control.

Systematic reviews on tobacco control from Cochrane and the Community Guide: different methods, similar findings

OBJECTIVES

To compare the methods and findings of systematic reviews (SRs) on common tobacco control interventions from two organizations: the Cochrane Collaboration ("Cochrane") and the US Task Force for Community Preventive Services ("the Guide").

STUDY DESIGN AND SETTING

Literature review. We retrieved all reviews pertaining to tobacco control produced by the Cochrane and the Guide. We identified seven common topics and compared methods and findings of the retrieved reviews.

RESULTS

There was considerable variability in the designs of included studies and methods of data synthesis. On average, Cochrane identified more studies than did the Guide (Mean 43.7 vs. 19.0), with only limited overlap between sets of included studies. Most Cochrane reviews (71.4%) were synthesized narratively, whereas most Guide reviews (85.7%) were synthesized using a median of effect size. Despite these differences, findings of the reviews yielded substantial agreement.

CONCLUSION

Cochrane and the Guide conduct SRs on similar tobacco control-related topics differently. The SRs of the two organizations include overlapping, but nonidentical sets, of studies. Still, they usually reach similar conclusions. Identification of all pertinent original studies seems to be a weak point in the SR process. Policy makers should use reviews from both organizations in formulating tobacco control policy.

Analysis of the reporting of search strategies in Cochrane systematic reviews

BACKGROUND

The Cochrane Handbook for Systematic Reviews of Interventions provides instructions for documenting a systematic review's electronic database search strategy, listing elements that should be in the description. Complete documentation of the search strategy allows readers to evaluate the search when critically appraising a review's quality.

OBJECTIVE

The research analyzed recently published Cochrane reviews to determine whether instructions for describing electronic database search strategies were being followed.

METHODS

Eighty-three new reviews added to the Cochrane Database of Systematic Reviews in the first quarter of 2006 were selected for analysis. Eighteen were subsequently excluded because their searches were conducted only in the specialized registers of Cochrane review groups. The remaining sixty-five reviews were analyzed for the seven elements of an electronic database search strategy description listed in the Cochrane Handbook, using dual review with consensus.

RESULTS

Of the 65 reviews analyzed, none included all 7 recommended elements. Four reviews (6%) included 6 elements. Thirty-two percent (21/65) included 5 or more elements, with 68% (44/65) including 4 or fewer. Three included only 2 elements. The 65 reviews represented 41 different Cochrane review groups.

CONCLUSION

The instructions from the Cochrane Handbook for reporting search strategies are not being consistently employed by groups producing Cochrane reviews.

An evidence-based practice guideline for the peer review of electronic search strategies

OBJECTIVE

Complex and highly sensitive electronic literature search strategies are required for systematic reviews; however, no guidelines exist for their peer review. Poor searches may fail to identify existing evidence because of inadequate recall (sensitivity) or increase the resource requirements of reviews as a result of inadequate precision. Our objective was to create an annotated checklist for electronic search strategy peer review.

STUDY DESIGN

A systematic review of the library and information retrieval literature for important elements in electronic search strategies was conducted, along with a survey of individuals experienced in systematic review searching.

RESULTS

Six elements with a strong consensus as to their importance in peer review were accurate translation of the research question into search concepts, correct choice of Boolean operators and of line numbers, adequate translation of the search strategy for each database, inclusion of relevant subject headings, and absence of spelling errors. Seven additional elements had partial support and are included in this guideline.

CONCLUSION

This evidence-based guideline facilitates the improvement of search quality through peer review, and thus the improvement in quality of systematic reviews. It is relevant for librarians/information specialists, journal editors, developers of knowledge translation tools, research organizations, and funding bodies.