Performance of search strategies to retrieve systematic reviews of diagnostic test accuracy from the Cochrane Library

Search strategies (filters) to identify systematic reviews in MEDLINE and Embase

BACKGROUND

Bibliographic databases provide access to an international body of scientific literature in health and medical sciences. Systematic reviews are an important source of evidence for clinicians, researchers, consumers, and policymakers as they address a specific health-related question and use explicit methods to identify, appraise and synthesize evidence from which conclusions can be drawn and decisions made. Methodological search filters help database end-users search the literature effectively with different levels of sensitivity and specificity. These filters have been developed for various study designs and have been found to be particularly useful for intervention studies. Other filters have been developed for finding systematic reviews. Considering the variety and number of available search filters for systematic reviews, there is a need for a review of them in order to provide evidence about their retrieval properties at the time they were developed.

OBJECTIVES

To review systematically empirical studies that report the development, evaluation, or comparison of search filters to retrieve reports of systematic reviews in MEDLINE and Embase.

SEARCH METHODS

We searched the following databases from inception to January 2023: MEDLINE, Embase, PsycINFO; Library, Information Science & Technology Abstracts (LISTA) and Science Citation Index (Web of Science).

SELECTION CRITERIA

We included studies if one of their primary objectives is the development, evaluation, or comparison of a search filter that could be used to retrieve systematic reviews on MEDLINE, Embase, or both.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data using a pre-specified and piloted data extraction form using InterTASC Information Specialist Subgroup (ISSG) Search Filter Evaluation Checklist.

MAIN RESULTS

We identified eight studies that developed filters for MEDLINE and three studies that developed filters for Embase. Most studies are very old and some were limited to systematic reviews in specific clinical areas. Six included studies reported the sensitivity of their developed filter. Seven studies reported precision and six studies reported specificity. Only one study reported the number needed to read and positive predictive value. None of the filters were designed to differentiate systematic reviews on the basis of their methodological quality. For MEDLINE, all filters showed similar sensitivity and precision, and one filter showed higher levels of specificity. For Embase, filters showed variable sensitivity and precision, with limited study reports that may affect accuracy assessments. The report of these studies had some limitations, and the assessments of their accuracy may suffer from indirectness, considering that they were mostly developed before the release of the PRISMA 2009 statement or due to their limited scope in the selection of systematic review topics. Search filters for MEDLINE Three studies produced filters with sensitivity > 90% with variable degrees of precision, and only one of them was developed and validated in a gold-standard database, which allowed the calculation of specificity. The other two search filters had lower levels of sensitivity. One of these produced a filter with higher levels of specificity (> 90%). All filters showed similar sensitivity and precision in the external validation, except for one which was not externally validated and another one which was conceptually derived and only externally validated. Search filters for Embase We identified three studies that developed filters for this database. One of these studies developed filters with variable sensitivity and precision, including highly sensitive strategies (> 90%); however, it was not externally validated. The other study produced a filter with a lower sensitivity (72.7%) but high specificity (99.1%) with a similar performance in the external validation.

AUTHORS' CONCLUSIONS

Studies reporting the development, evaluation, or comparison of search filters to retrieve reports of systematic reviews in MEDLINE showed similar sensitivity and precision, with one filter showing higher levels of specificity. For Embase, filters showed variable sensitivity and precision, with limited information about how the filter was produced, which leaves us uncertain about their performance assessments. Newer filters had limitations in their methods or scope, including very focused subject topics for their gold standards, limiting their applicability across other topics. Our findings highlight that consensus guidance on the conduct of search filters and standardized reporting of search filters are needed, as we found highly heterogeneous development methods, accuracy assessments and outcome selection. New strategies adaptable across interfaces could enhance their usability. Moreover, the performance of existing filters needs to be evaluated in light of the impact of reporting guidelines, including the PRISMA 2009, on how systematic reviews are reported. Finally, future filter developments should also consider comparing the filters against a common reference set to establish comparative performance and assess the quality of systematic reviews retrieved by strategies.

Electronic search strategies fail to identify randomized controlled trials (RCTs) in neurosurgery

Randomized controlled trials (RCTs) are the gold standard studies to evaluate the efficacy of therapeutic interventions. Although they are frequently identified through open searches in electronic databases, no studies have evaluated how easy it is to identify RCTs in neurosurgery using electronic search strategies. The present study evaluated the sensitivity and specificity of different search strategies applied to commonly used databases to identify RCTs in neurosurgery. The total number of RCTs in neurosurgery published between 1960 and 2013 was determined through a detailed search involving open keyword searches in PubMed, Cochrane Library and Center for Reviews and Dissemination (CRD) databases, a PubMed search based on clinical entity-related keywords and hand-searches on the reference list of identified articles. The sensitivity and specificity were calculated for the open keyword searches on PubMed, the Cochrane Library and the CRD database and for the Cochrane's HSSS, based on the total number of the identified RCTs. Compared to the total of 1102 RCTs identified, PubMed open search yielded 4660 articles, among which 365 were RCTs (sensitivity: 33.1%; specificity: 7.8%). Cochrane open search yielded 621 among which 36 were RCTs (sensitivity: 3.2%; specificity: 5.8%) and CRD open search returned 78 articles, among which 4 were RCTs (sensitivity: 0.4% sensitivity; specificity: 5.1%). The Cochrane HSSS retrieved 10702 results, among which 340 were RCTs (sensitivity: 30.9%; specificity: 3.2%). Most RCTs in neurosurgery cannot be identified by commonly used search strategies, which emphasizes the need to improve their indexing.

Optimal search strategies for identifying moderators and predictors of treatment effects in PubMed

BACKGROUND

Treatment effects differ across patients. To guide selection of treatments for patients, it is essential to acknowledge these differences and identify moderators or predictors. Our aim was to generate optimal search strategies (commonly known as filters) for PubMed to retrieve papers identifying moderators and predictors of treatment effects.

METHODS

Six journals were hand-searched for articles on moderators or predictors. Selected articles were randomly allocated to a development and validation set. Search terms were extracted from the development set and tested for their performance. Search filters were created from combinations of these terms and tested in the validation set.

RESULTS

Of 4407 articles, 198 were considered to be relevant. The most sensitive filter in the development set '("Epidemiologic Methods" [MeSH] OR assign* OR control*[tiab] OR trial*[tiab]) AND therapy*[sh]' yielded in the validation set a sensitivity of 89% [88%-90%] and a specificity of 80% [79%-82%].

CONCLUSIONS

The search filters created in this study can help to efficiently retrieve evidence on moderators and predictors of treatment effect. Testing of the filters in multiple domains should reveal robustness across disciplines. These filters can facilitate the retrieval of evidence on moderators and predictors of treatment effects, helping the implementation of stratified or personalised health care.