Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

Development and validation of search filters to retrieve medication discontinuation articles in Medline and Embase

BACKGROUND

Medication discontinuation studies explore the outcomes of stopping a medication compared to continuing it. Comprehensively identifying medication discontinuation articles in bibliographic databases remains challenging due to variability in terminology.

OBJECTIVES

To develop and validate search filters to retrieve medication discontinuation articles in Medline and Embase.

METHODS

We identified medication discontinuation articles in a convenience sample of systematic reviews. We used primary articles to create two reference sets for Medline and Embase, respectively. The reference sets were equally divided by randomization in development sets and validation sets. Terms relevant for discontinuation were identified by term frequency analysis in development sets and combined to develop two search filters that maximized relative recalls. The filters were validated against validation sets. Relative recalls were calculated with their 95% confidences intervals (95% CI).

RESULTS

We included 316 articles for Medline and 407 articles for Embase, from 15 systematic reviews. The Medline optimized search filter combined 7 terms. The Embase optimized search filter combined 8 terms. The relative recalls were respectively 92% (95% CI: 87-96) and 91% (95% CI: 86-94).

CONCLUSIONS

We developed two search filters for retrieving medication discontinuation articles in Medline and Embase. Further research is needed to estimate precision and specificity of the filters.

Use of deprescribing search filters in systematic review search strategies: A case study

Two deprescribing search filters for MEDLINE and one deprescribing search filter for Embase have been recently developed, including objectively developed search filters. The objective of this case study was to implement these three deprescribing search filters in systematic review (SR) search strategies and to assess their effect on performances. SR that independently developed original search strategies (OSS) were selected. The deprescribing filters were implemented in each OSS, generating two implemented search strategies (ISS1 and ISS2) in MEDLINE and one ISS (ISS3) in Embase. OSS were re-run on the same date as ISS. The performances of ISS and OSS were calculated and compared. Two SR were included (SR1 and SR2). For MEDLINE, SR1 included 12 articles. The sensitivity was 50% for OSS, 58% for ISS1 and 42% for ISS2. SR2 included four articles. The sensitivity of OSS, ISS 1 and 2 was 25%. For Embase, SR1 included 12 articles. The sensitivity was 33% for OSS and 58% for ISS3. SR2 included four articles. None of the four included articles were retrieved with OSS or ISS3. While sensitivity of OSS was moderate, the objectively developed deprescribing filters maintained or slightly improved this sensitivity when implementing.

Search strategies (filters) to identify systematic reviews in MEDLINE and Embase

BACKGROUND

Bibliographic databases provide access to an international body of scientific literature in health and medical sciences. Systematic reviews are an important source of evidence for clinicians, researchers, consumers, and policymakers as they address a specific health-related question and use explicit methods to identify, appraise and synthesize evidence from which conclusions can be drawn and decisions made. Methodological search filters help database end-users search the literature effectively with different levels of sensitivity and specificity. These filters have been developed for various study designs and have been found to be particularly useful for intervention studies. Other filters have been developed for finding systematic reviews. Considering the variety and number of available search filters for systematic reviews, there is a need for a review of them in order to provide evidence about their retrieval properties at the time they were developed.

OBJECTIVES

To review systematically empirical studies that report the development, evaluation, or comparison of search filters to retrieve reports of systematic reviews in MEDLINE and Embase.

SEARCH METHODS

We searched the following databases from inception to January 2023: MEDLINE, Embase, PsycINFO; Library, Information Science & Technology Abstracts (LISTA) and Science Citation Index (Web of Science).

SELECTION CRITERIA

We included studies if one of their primary objectives is the development, evaluation, or comparison of a search filter that could be used to retrieve systematic reviews on MEDLINE, Embase, or both.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data using a pre-specified and piloted data extraction form using InterTASC Information Specialist Subgroup (ISSG) Search Filter Evaluation Checklist.

MAIN RESULTS

We identified eight studies that developed filters for MEDLINE and three studies that developed filters for Embase. Most studies are very old and some were limited to systematic reviews in specific clinical areas. Six included studies reported the sensitivity of their developed filter. Seven studies reported precision and six studies reported specificity. Only one study reported the number needed to read and positive predictive value. None of the filters were designed to differentiate systematic reviews on the basis of their methodological quality. For MEDLINE, all filters showed similar sensitivity and precision, and one filter showed higher levels of specificity. For Embase, filters showed variable sensitivity and precision, with limited study reports that may affect accuracy assessments. The report of these studies had some limitations, and the assessments of their accuracy may suffer from indirectness, considering that they were mostly developed before the release of the PRISMA 2009 statement or due to their limited scope in the selection of systematic review topics. Search filters for MEDLINE Three studies produced filters with sensitivity > 90% with variable degrees of precision, and only one of them was developed and validated in a gold-standard database, which allowed the calculation of specificity. The other two search filters had lower levels of sensitivity. One of these produced a filter with higher levels of specificity (> 90%). All filters showed similar sensitivity and precision in the external validation, except for one which was not externally validated and another one which was conceptually derived and only externally validated. Search filters for Embase We identified three studies that developed filters for this database. One of these studies developed filters with variable sensitivity and precision, including highly sensitive strategies (> 90%); however, it was not externally validated. The other study produced a filter with a lower sensitivity (72.7%) but high specificity (99.1%) with a similar performance in the external validation.

AUTHORS' CONCLUSIONS

Studies reporting the development, evaluation, or comparison of search filters to retrieve reports of systematic reviews in MEDLINE showed similar sensitivity and precision, with one filter showing higher levels of specificity. For Embase, filters showed variable sensitivity and precision, with limited information about how the filter was produced, which leaves us uncertain about their performance assessments. Newer filters had limitations in their methods or scope, including very focused subject topics for their gold standards, limiting their applicability across other topics. Our findings highlight that consensus guidance on the conduct of search filters and standardized reporting of search filters are needed, as we found highly heterogeneous development methods, accuracy assessments and outcome selection. New strategies adaptable across interfaces could enhance their usability. Moreover, the performance of existing filters needs to be evaluated in light of the impact of reporting guidelines, including the PRISMA 2009, on how systematic reviews are reported. Finally, future filter developments should also consider comparing the filters against a common reference set to establish comparative performance and assess the quality of systematic reviews retrieved by strategies.

Calcium and vitamin D for increasing bone mineral density in premenopausal women

BACKGROUND

Osteoporosis is a condition where bones become fragile due to low bone density and impaired bone quality. This results in fractures that lead to higher morbidity and reduced quality of life. Osteoporosis is considered a major public health concern worldwide. For this reason, preventive measurements need to be addressed throughout the life course. Exercise and a healthy diet are among the lifestyle factors that can help prevent the disease, the latter including intake of key micronutrients for bone, such as calcium and vitamin D. The evidence on whether supplementation with calcium and vitamin D improves bone mineral density (BMD) in premenopausal women is still inconclusive. In this age group, bone accrual is considered to be the goal of supplementation, so BMD is relevant for the future stages of life.

OBJECTIVES

To evaluate the benefits and harms of calcium and vitamin D supplementation, alone or in combination, to increase the BMD, reduce fractures, and report the potential adverse events in healthy premenopausal women compared to placebo.

SEARCH METHODS

We used standard, extensive Cochrane search methods. The latest search was 12 April 2022.

SELECTION CRITERIA

We included randomised controlled trials in healthy premenopausal women (with or without calcium or vitamin D deficiency) comparing supplementation of calcium or vitamin D (or both) at any dose and by any route of administration versus placebo for at least three months. Vitamin D could have been administered as cholecalciferol (vitamin D3) or ergocalciferol (vitamin D2).

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methods. Outcomes included total hip bone mineral density (BMD), lumbar spine BMD, quality of life, new symptomatic vertebral fractures, new symptomatic non-vertebral fractures, withdrawals due to adverse events, serious adverse events, all reported adverse events and additional withdrawals for any reason.

MAIN RESULTS

We included seven RCTs with 941 participants, of whom 138 were randomised to calcium supplementation, 110 to vitamin D supplementation, 271 to vitamin D plus calcium supplementation, and 422 to placebo. Mean age ranged from 18.1 to 42.1 years. Studies reported results for total hip or lumbar spine BMD (or both) and withdrawals for various reasons, but none reported fractures or withdrawals for adverse events or serious adverse events. Results for the reported outcomes are presented for the three comparisons: calcium versus placebo, vitamin D versus placebo, and calcium plus vitamin D versus placebo. In all comparisons, there was no clinical difference in outcomes, and the certainty of the evidence was moderate to low. Most studies were at risk of selection, performance, detection, and reporting biases. Calcium versus placebo Four studies compared calcium versus placebo (138 participants in the calcium group and 123 in the placebo group) with mean ages from 18.0 to 47.3 years. Calcium supplementation may have little to no effect on total hip or lumbar spine BMD after 12 months in three studies and after six months in one study (total hip BMD: mean difference (MD) -0.04 g/cm2, 95% confidence interval (CI) -0.11 to 0.03; I2 = 71%; 3 studies, 174 participants; low-certainty evidence; lumbar spine BMD: MD 0 g/cm2, 95% CI -0.06 to 0.06; I2 = 71%; 4 studies, 202 participants; low-certainty evidence). Calcium alone supplementation does not reduce or increase the withdrawals in the trials (risk ratio (RR) 0.78, 95% CI 0.52 to 1.16; I2 = 0%; 4 studies, 261 participants: moderate-certainty evidence). Vitamin D versus placebo Two studies compared vitamin D versus placebo (110 participants in the vitamin D group and 79 in the placebo group), with mean ages from 18.0 to 32.7 years. These studies reported lumbar spine BMD as a mixture of MDs and percent of change and we were unable to pool the results. In the original studies, there were no differences in lumbar BMD between groups. Vitamin D alone supplementation does not reduce or increase withdrawals for any reason between groups (RR 0.74, 95% CI 0.46 to 1.19; moderate-certainty evidence). Calcium plus vitamin D versus placebo Two studies compared calcium plus vitamin D versus placebo (271 participants in the calcium plus vitamin D group and 270 in the placebo group; 220 participants from Woo 2007 and 50 participants from Islam 2010). The mean age range was 18.0 to 36 years. These studies measured different anatomic areas, one study reported total hip BMD and the other study reported lumbar spine BMD; therefore, data were not pooled for this outcome. The individual studies found no difference between groups in percent of change on total hip BMD (-0.03, 95% CI -0.06 to 0; moderate-certainty evidence), and lumbar spine BMD (MD 0.01, 95% CI -0.01 to 0.03; moderate-certainty evidence). Calcium plus vitamin D supplementation may not reduce or increase withdrawals for any reason (RR 0.82, 95% CI 0.29 to 2.35; I2 = 72%; 2 studies, 541 participants; low-certainty evidence).

AUTHORS' CONCLUSIONS

Our results do not support the isolated or combined use of calcium and vitamin D supplementation in healthy premenopausal women as a public health intervention to improve BMD in the total hip or lumbar spine, and therefore it is unlikely to have a benefit for the prevention of fractures (vertebral and non-vertebral). The evidence found suggests that there is no need for future studies in the general population of premenopausal women; however, studies focused on populations with a predisposition to diseases related to bone metabolism, or with low bone mass or osteoporosis diagnosed BMD would be useful.

Development and validation of search filters to identify articles on deprescribing in Medline and Embase

Background

Deprescribing literature has been increasing rapidly. Our aim was to develop and validate search filters to identify articles on deprescribing in Medline via PubMed and in Embase via Embase.com.

Methods

Articles published from 2011 to 2020 in a core set of eight journals (covering fields of interest for deprescribing, such as geriatrics, pharmacology and primary care) formed a reference set. Each article was screened independently in duplicate and classified as relevant or non-relevant to deprescribing. Relevant terms were identified by term frequency analysis in a 70% subset of the reference set. Selected title and abstract terms, MeSH terms and Emtree terms were combined to develop two highly sensitive filters for Medline via Pubmed and Embase via Embase.com. The filters were validated against the remaining 30% of the reference set. Sensitivity, specificity and precision were calculated with their 95% confidence intervals (95% CI).

Results

A total of 23,741 articles were aggregated in the reference set, and 224 were classified as relevant to deprescribing. A total of 34 terms and 4 MeSH terms were identified to develop the Medline search filter. A total of 27 terms and 6 Emtree terms were identified to develop the Embase search filter. The sensitivity was 92% (95% CI: 83–97%) in Medline via Pubmed and 91% (95% CI: 82–96%) in Embase via Embase.com.

Conclusions

These are the first deprescribing search filters that have been developed objectively and validated. These filters can be used in search strategies for future deprescribing reviews. Further prospective studies are needed to assess their effectiveness and efficiency when used in systematic reviews.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01515-x.

Development of an efficient search filter to retrieve systematic reviews from PubMed

Objective:

Locating systematic reviews is essential for clinicians and researchers when creating or updating reviews and for decision-making in health care. This study aimed to develop a search filter for retrieving systematic reviews that improves upon the performance of the PubMed systematic review search filter.

Methods:

Search terms were identified from abstracts of reviews published in Cochrane Database of Systematic Reviews and the titles of articles indexed as systematic reviews in PubMed. Both the precision of the candidate terms and the number of systematic reviews retrieved from PubMed were evaluated after excluding the subset of articles retrieved by the PubMed systematic review filter. Terms that achieved a precision greater than 70% and relevant publication types indexed with MeSH terms were included in the filter search strategy.

Results:

The search strategy used in our filter added specific terms not included in PubMed's systematic review filter and achieved a 61.3% increase in the number of retrieved articles that are potential systematic reviews. Moreover, it achieved an average precision that is likely greater than 80%.

Conclusions:

The developed search filter will enable users to identify more systematic reviews from PubMed than the PubMed systematic review filter with high precision.

Translation and validation of PubMed and Embase search filters for identification of systematic reviews, intervention studies, and observational studies in the field of first aid

Objective

The aim of this project was to validate search filters for systematic reviews, intervention studies, and observational studies translated from Ovid MEDLINE and Embase syntax and used for searches in PubMed and Embase.com during the development of evidence summaries supporting first aid guidelines. We aimed to achieve a balance among recall, specificity, precision, and number needed to read (NNR).

Methods

Reference gold standards were constructed per study type derived from existing evidence summaries. Search filter performance was assessed through retrospective searches and measurement of relative recall, specificity, precision, and NNR when using the translated search filters. Where necessary, search filters were optimized. Adapted filters were validated in separate validation gold standards.

Results

Search filters for systematic reviews and observational studies reached recall of ≥85% in both PubMed and Embase. Corresponding specificities for systematic review filters were ≥96% in both databases, with a precision of 9.7% (NNR 10) in PubMed and 5.4% (NNR 19) in Embase. For observational study filters, specificity, precision, and NNR were 68%, 2%, and 51 in PubMed and 47%, 0.8%, and 123 in Embase, respectively. These filters were considered sufficiently effective. Search filters for intervention studies reached a recall of 85% and 83% in PubMed and Embase, respectively. Optimization led to recall of ≥95% with specificity, precision, and NNR of 49%, 1.3%, and 79 in PubMed and 56%, 0.74%, and 136 in Embase, respectively.

Conclusions

We report validated filters to search for systematic reviews, observational studies, and intervention studies in guideline projects in PubMed and Embase.com.

Validation of a search strategy for randomized clinical trials related to periodontitis

Background

Systematic reviews, considered the gold standard for the assessment of scientific evidence, may present conflicting findings for the same clinical issue, and such dissent may be justified by the forms of elaboration of the electronic search strategy. This paper aims to validate a search strategy to identify randomized clinical trials related to periodontitis. A gold standard reference set was developed to validate the identified clinical trials using the relative recall method. The choice of periodontitis is due to the fact that this disease has a high prevalence among chronic non-communicable diseases, is considered the second most common oral disease in the world, is associated with several health problems, such as cardiovascular diseases and diabetes, and, principally, has not been investigated sufficiently to prevent possible damages resulting from it.

Methods

A validation study was developed in MEDLINE/PubMed. In Stage 1, a methodological filter recommended by the Cochrane Collaboration to identify randomized clinical trials was applied. Stage 2 identified articles related only to periodontitis (gold standard reference set) from among the articles retrieved using the eligibility criteria. In Stage 3, a search statement for the retrieval of periodontitis-related articles was elaborated by experts. Stage 4 defined the proposed search strategy comprising of the combination of the search statement developed with the aforementioned methodological filter and subsequent application in MEDLINE/PubMed. The obtained data were analyzed using the set of articles identified in Stage 2, as the gold standard reference set. The following performance values were calculated - sensitivity, specificity, accuracy, and number needed to read - with their respective 95% confidence interval (95%CI).

Results

The search strategy under evaluation compared to the gold-standard showed a sensitivity of 93.2% (_95%CI, 83.8–97.3), specificity of 99.9% (_95%CI 99.8–99.9), and a precision of 77.5% (_95%CI, 66.48–85.63). In addition, the number needed to read was 1.3.

Conclusion

According to the proposed methodological approach, the search strategy under evaluation performed well in the identification of randomized clinical trials related to periodontitis.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13690-021-00560-0.

PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. J Med Libr Assoc 2021;109:174-200. [PMID: 34285662 PMCID: PMC8270366 DOI: 10.5195/jmla.2021.962]

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a three-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes sixteen reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and, therefore, reproducible.

PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse, and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a 3-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes 16 reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and therefore reproducible.

The NICE MEDLINE and Embase (Ovid) health apps search filters: development of validated filters to retrieve evidence about health apps

OBJECTIVES

Health apps are software programs that are designed to prevent, diagnose, monitor, or manage conditions. Inconsistent terminology for apps is used in research literature and bibliographic database subject headings. It can therefore be challenging to retrieve evidence about them in literature searches. Information specialists at the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed novel validated search filters to retrieve evidence about apps from MEDLINE and Embase (Ovid).

METHODS

A selection of medical informatics journals was hand searched to identify a "gold standard" (GS) set of references about apps. The GS set was divided into a development and validation set. The filters' search terms were derived from and tested against the development set. An external development set containing app references from published NICE products was also used to inform the development of the filters. The filters were then validated using the validation set. Target recall was >90 percent. The filters' overall recall, specificity, and precision were calculated using all the references identified from the hand search.

RESULTS

Both filters achieved 98.6 percent recall against their validation sets. Overall, the MEDLINE filter had 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter had 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision.

CONCLUSIONS

The NICE health apps search filters retrieve evidence about apps from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about the interventions by information professionals, researchers, and clinicians.

Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE?

OBJECTIVE

The Cochrane Handbook of Systematic Reviews contains search filters to find randomized controlled trials (RCTs) in Ovid MEDLINE: one maximizing sensitivity and another balancing sensitivity and precision. These filters were originally published in 1994 and were adapted and updated in 2008. To determine the performance of these filters, the authors tested them and thirty-six other MEDLINE filters against a large new gold standard set of relevant records.

METHODS

We identified a gold standard set of RCT reports published in 2016 from the Cochrane CENTRAL database of controlled clinical trials. We retrieved the records in Ovid MEDLINE and combined these with each RCT filter. We calculated their sensitivity, relative precision, and f-scores.

RESULTS

The gold standard comprised 27,617 records. MEDLINE searches were run on July 16, 2019. The most sensitive RCT filter was Duggan et al. (sensitivity=0.99). The Cochrane sensitivity-maximizing RCT filter had a sensitivity of 0.96 but was more precise than Duggan et al. (0.14 compared to 0.04 for Duggan). The most precise RCT filters had 0.97 relative precision and 0.83 sensitivity.

CONCLUSIONS

The Cochrane Ovid MEDLINE sensitivity-maximizing RCT filter can continue to be used by Cochrane reviewers and to populate CENTRAL, as it has very high sensitivity and a slightly better precision relative to more sensitive filters. The results of this study, which used a very large gold standard to compare the performance of all known RCT filters, allows searchers to make better informed decisions about which filters to use for their work.

A validation study revealed differences in design and performance of search filters for qualitative research in PsycINFO and CINAHL

OBJECTIVES

Search filters can support qualitative evidence of information retrieval. Various search filters are available for the bibliographic databases PsycINFO and CINAHL. To date, no comparative overview of validation results of search filters verified with an independent gold standard exists.

STUDY DESIGN AND SETTING

Identified search filters for PsycINFO and CINAHL were tested for plausibility. Gold standards were generated according to the relative recall approach using references included in an overview of systematic reviews of qualitative studies. All included references were collected and checked for indexing in PsycINFO and CINAHL. Validation tests for each search filter were conducted in both databases to determine whether the references of the gold standards could be retrieved or not.

RESULTS

Twelve search filters for PsycINFO and fifteen for CINAHL were validated. The complexity and design of these search filters vary, as well as the validation results for the databases. When locating primary studies of qualitative research, the best sensitivity and precision ratio (among filters with a sensitivity of >80%) was achieved with a filter by McKibbon et al. for PsycINFO and a filter by Wilczynski et al. for CINAHL.

CONCLUSION

Project-specific requirements and resources influence the choice of a specific search filter for PsycINFO and CINAHL.

A validated search filter for the identification of endocrine disruptors based on the ECHA/EFSA guidance recommendations

A guidance document for the identification of endocrine disruptors (EDs) in the regulatory assessment of plant protection products (PPP) and biocidal products (BP) has been published by the European Chemical Agency (ECHA) and the European Food Safety Authority (EFSA). The ECHA/EFSA guidance, mainly addressing EATS (estrogen, androgen, thyroid, steroidogenesis) modalities, is intended to guide applicants and assessors of the competent regulatory authorities on the implementation of the scientific criteria for the determination of ED properties pursuant to the recently implemented PPP (EU 2018/605) and BP (EU 2017/2100) EU Regulations. In this study, a search filter for targeted literature search in context of assessing if a substance can be identified as an ED relevant for human health was developed and validated. Development of the search filter was based on the search strategy presented in the ECHA/EFSA guidance and using the estrogenic chemical Bisphenol AF (BPAF) as a model substance. Information specialists from two independent institutions developed refined search filters based on the suggested original search strategy published (ECHA/EFSA guidance - Appendix F). Articles identified by a systematic literature search for BPAF were screened for relevance with inclusion and exclusion criteria by two independent reviewers obtaining positive (relevant) and negative (irrelevant) controls. The developed search filter was quantitatively evaluated in terms of sensitivity, specificity and precision based on the positive and negative controls. The developed filter was then validated for T modality by its application to the known thyroid-disruptor perchlorate. The result is a sensitive search filter with sufficient specificity, which can be applied for all chemicals where a targeted literature search is needed to assess and identify ED properties of chemicals with relevance for humans. Future application of the filter to a broader range of chemicals may identify further points of improvement.

"It has no meaning to me." How do researchers understand the effectiveness of literature searches? A qualitative analysis and preliminary typology of understandings

This study aimed to address the question: what does "effectiveness" mean to researchers in the context of literature searching for systematic reviews? We conducted a thematic analysis of responses to an e-mail survey. Eighty-nine study authors, whose studies met inclusion in a recent review (2018), were contacted via e-mail and asked three questions; one directly asking the question: in literature searching, what does effective (or effectiveness in) literature searching mean to you? Thirty-eight (46%) responses were received from diverse professional groups, including: literature searchers, systematic reviewers, clinicians and researchers. A shared understanding of what effectiveness means was not identified. Instead, five themes were developed from data: (a) effectiveness is described as a metric; (b) effectiveness is a balance between metrics; (c) effectiveness can be categorized by search purpose; (d) effectiveness is an outcome; and, (e) effectiveness is an experimental concept. We propose that these themes constitute a preliminary typology of understandings. No single definition of effectiveness was identified. The proposed typology suggests that different researchers have differing understandings of effectiveness. This could lead to uncertainty as to the aim and the purpose of literature searches and confusion about the outcomes. The typology offers a potential route for further exploration.

Development and validation of study filters for identifying controlled non-randomized studies in PubMed and Ovid MEDLINE

A retrospective analysis published by the German Institute for Quality and Efficiency in Health Care (IQWiG) in 2018 concluded that no filter for non-randomized studies (NRS) achieved sufficient sensitivity (≥92%), a precondition for comprehensive information retrieval. New NRS filters are therefore required, taking into account the challenges related to this study type. Our evaluation focused on the development of study filters for NRS with a control group ("controlled NRS"), as this study type allows the calculation of an effect size. In addition, we assumed that due to the more explicit search syntax, controlled NRS are easier to identify than non-controlled ones, potentially resulting in better performance measures of study filters for controlled NRS. Our aim was to develop study filters for identifying controlled NRS in PubMed and Ovid MEDLINE. We developed two new search filters that can assist clinicians and researchers in identifying controlled NRS in PubMed and Ovid MEDLINE. The reference set was based on 2110 publications in Medline extracted from 271 Cochrane reviews and on 4333 irrelevant references. The first filter maximizes sensitivity (92.42%; specificity 79.67%, precision 68.49%) and should be used when a comprehensive search is needed. The second filter maximizes specificity (92.06%; precision 82.98%, sensitivity 80.94%) and should be used when a more focused search is sufficient.

A PRISMA assessment of the reporting quality of systematic reviews of nursing published in the Cochrane Library and paper-based journals

OBJECTIVE

The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) was released as a standard of reporting systematic reviewers (SRs). However, not all SRs adhere completely to this standard. This study aimed to evaluate the reporting quality of SRs published in the Cochrane Library and paper-based journals.

METHODS

The SRs which evaluate the effectiveness of nursing interventions in 2016 were identified via PubMed. The reporting quality of selected articles was evaluated using the PRISMA checklist. For comparison, we divided these articles into Cochrane review (CR) and non-Cochrane review (NCR). Based on the satisfaction of the applicable criteria, each article is assigned an accumulated score and a total percentage score.

RESULTS

Overall, 41.7% articles were concentrated in 19.0 to 22.5 points which represent the moderate quality, 22% articles were high quality. There were still 36.5% articles with low quality. The mean PRISMA score was 20.54 ± 2.367 for CRs, and 18.81 ± 2.536 for NCRs. Although no significant difference was exit between overall CR and NCR scores, there were differences between items 1, 5, 8, 16, 23. Analysis indicated that CR was significantly associated with the overall PRISMA score.

CONCLUSION

Compliance of CR and NCR with PRISMA checklist exhibited different strengths and weaknesses. Our study underscores that nursing researchers should pay more attention to comprehensive reporting of SRs in nursing to follow the PRISMA statement.

IMPLICATIONS FOR NURSING AND/OR HEALTH POLICY

Nursing researchers who participate in SRs should follow the latest Cochrane Handbook to prepare such study. Meanwhile, the PRISMA statement should be followed strictly to report SRs, so as to improve the quality of SRs.

A Productive Proposed Search Syntax for Health Disaster Preparedness Research

Objective:

To find a proper search strategy to do a systematic review related to preparedness for disasters.

Methods:

MeSH and Emtree terms were searched to detect synonyms for two main search terms “disaster” and “preparedness”. Expert opinion on the synonyms was examined applying a Google form. The adopted syntax was searched in PubMed and results were sifted. Hand searching in two top key journals was done and sensitivity was calculated.

Results:

Out of 1120 articles, 122 were included. In PDM journal, 10 articles were included by hand searching, out of which 5 were not spotted in PubMed search with the proposed syntax. In DMPHP journal, 13 publications were included, with 5 not found in PubMed search. Because of human error in hand searching 2 articles were added.

Conclusion:

The proposed syntax in this study achieves a sensitivity of search of 0.6 in PubMed which could be quite applicable for researchers. Moreover, in case only MeSH or Emtree terms were applied in search strategy or where hand searching was not performed, there were a number of articles missed.

A Productive Proposed Search Syntax for Health Disaster Preparedness Research

Objective

To find a proper search strategy to do a systematic review related to preparedness for disasters.

Methods

MeSH and Emtree terms were searched to detect synonyms for two main search terms "disaster" and "preparedness". Expert opinion on the synonyms was examined applying a Google form. The adopted syntax was searched in PubMed and results were sifted. Hand searching in two top key journals was done and sensitivity was calculated.

Results

Out of 1120 articles, 122 were included. In PDM journal, 10 articles were included by hand searching, out of which 5 were not spotted in PubMed search with the proposed syntax. In DMPHP journal, 13 publications were included, with 5 not found in PubMed search. Because of human error in hand searching 2 articles were added.

Conclusion

The proposed syntax in this study achieves a sensitivity of search of 0.6 in PubMed which could be quite applicable for researchers. Moreover, in case only MeSH or Emtree terms were applied in search strategy or where hand searching was not performed, there were a number of articles missed.

Search strategies to identify observational studies in MEDLINE and Embase

BACKGROUND

Systematic reviews are essential for decision-making. Systematic reviews on observational studies help answer research questions on aetiology, risk, prognosis, and frequency of rare outcomes or complications. However, identifying observational studies as part of systematic reviews efficiently is challenging due to poor and inconsistent indexing in literature databases. Search strategies that include a methodological filter focusing on study design of observational studies might be useful for improving the precision of the search performance.

OBJECTIVES

To assess the sensitivity and precision of a search strategy with a methodological filter to identify observational studies in MEDLINE and Embase.

SEARCH METHODS

We searched MEDLINE (1946 to April 2018), Embase (1974 to April 2018), CINAHL (1937 to April 2018), the Cochrane Library (1992 to April 2018), Google Scholar and Open Grey in April 2018, and scanned reference lists of articles.

SELECTION CRITERIA

Studies using a relative recall approach, i.e. comparing sensitivity or precision of a search strategy containing a methodological filter to identify observational studies in MEDLINE and Embase against a reference standard, or studies that compared two or more methodological filters.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened articles, extracted relevant information and assessed the quality of the search strategies using the InterTASC Information Specialists' Sub-Group (ISSG) Search Filter Appraisal Checklist.

MAIN RESULTS

We identified two eligible studies reporting 18 methodological filters. All methodological filters in these two studies were developed using terms from the reference standard records.The first study evaluated six filters for retrieving observational studies of surgical interventions. The study reported on six filters: one Precision Terms Filter (comprising terms with higher precision while maximum sensitivity was maintained) and one Specificity Terms Filter (comprising terms with higher specificity while maximum sensitivity was maintained), both of which were adapted for MEDLINE, for Embase, and for combined MEDLINE/Embase searches. The study reported one reference standard consisting of 217 articles from one systematic review of which 83.9% of the included studies were case seriesThe second study reported on 12 filters for retrieving comparative non-randomised studies (cNRSs) including cohort, case-control, and cross-sectional studies. This study reported on 12 filters using four different approaches: Fixed method A (comprising of a fixed set of controlled vocabulary (CV) words), Fixed method B (comprising a fixed set of CV words and text words (TW)), Progressive method (CV) (a random choice of study design-related CV terms), and Progressive method (CV or TW) (a random choice of study design-related CV terms, and title and abstracts-based TWs). The study reported four reference standards consisting of 89 cNRSs from four systematic reviews.The six methodological filters developed from the first study reported sensitivity of 99.5% to 100% and precision of 16.7% to 21.1%. The Specificity Terms Filter for combined MEDLINE/Embase was preferred because it had higher precision and equal sensitivity to the Precision Terms Filter. The 12 filters from the second study reported lower sensitivity (48% to 100%) and much lower precision (0.09% to 4.47%). The Progressive method (CV or TW) had the highest sensitivity.There were methodological limitations in both included studies. The first study used one surgical intervention-focused systematic review thus limiting the generalizability of findings. The second study used four systematic reviews but with less than 100 studies. The external validation was performed only on Specificity Terms Filter from the first study Both studies were published 10 years ago and labelling and indexing of observational studies has changed since then.

AUTHORS' CONCLUSIONS

We found 18 methodological filters across two eligible studies. Search strategies from the first study had higher sensitivity and precision, underwent external validation and targeted observational studies. Search strategies from the second study had lower sensitivity and precision, focused on cNRSs, and were not validated externally. Given this limited and heterogeneous evidence, and its methodological limitations, further research and better indexation are needed.

Topic search filters: a systematic scoping review

BACKGROUND

Searching for topics within large biomedical databases can be challenging, especially when topics are complex, diffuse, emerging or lack definitional clarity. Experimentally derived topic search filters offer a reliable solution to effective retrieval; however, their number and range of subject foci remain unknown.

OBJECTIVES

This systematic scoping review aims to identify and describe available experimentally developed topic search filters.

METHODS

Reports on topic search filter development (1990-) were sought using grey literature sources and 15 databases. Reports describing the conception and prospective development of a database-specific topic search and including an objectively measured estimate of its performance ('sensitivity') were included.

RESULTS

Fifty-four reports met inclusion criteria. Data were extracted and thematically synthesised to describe the characteristics of 58 topic search filters.

DISCUSSION

Topic search filters are proliferating and cover a wide range of subjects. Filter reports, however, often lack clear definitions of concepts and topic scope to guide users. Without standardised terminology, filters are challenging to find. Information specialists may benefit from a centralised topic filter repository and appraisal checklists to facilitate quality assessment.

CONCLUSION

Findings will help information specialists identify existing topic search filters and assist filter developers to build on current knowledge in the field.

Point-of-care ultrasonography for diagnosing thoracoabdominal injuries in patients with blunt trauma

BACKGROUND

Point-of-care sonography (POCS) has emerged as the screening modality of choice for suspected body trauma in many emergency departments worldwide. Its best known application is FAST (focused abdominal sonography for trauma). The technology is almost ubiquitously available, can be performed during resuscitation, and does not expose patients or staff to radiation. While many authors have stressed the high specificity of POCS, its sensitivity varied markedly across studies. This review aimed to compile the current best evidence about the diagnostic accuracy of POCS imaging protocols in the setting of blunt thoracoabdominal trauma.

OBJECTIVES

To determine the diagnostic accuracy of POCS for detecting and excluding free fluid, organ injuries, vascular lesions, and other injuries (e.g. pneumothorax) compared to a diagnostic reference standard (i.e. computed tomography (CT), magnetic resonance imaging (MRI), thoracoscopy or thoracotomy, laparoscopy or laparotomy, autopsy, or any combination of these) in patients with blunt trauma.

SEARCH METHODS

We searched Ovid MEDLINE (1946 to July 2017) and Ovid Embase (1974 to July 2017), as well as PubMed (1947 to July 2017), employing a prospectively defined literature and data retrieval strategy. We also screened the Cochrane Library, Google Scholar, and BIOSIS for potentially relevant citations, and scanned the reference lists of full-text papers for articles missed by the electronic search. We performed a top-up search on 6 December 2018, and identified eight new studies which may be incorporated into the first update of this review.

SELECTION CRITERIA

We assessed studies for eligibility using predefined inclusion and exclusion criteria. We included either prospective or retrospective diagnostic cohort studies that enrolled patients of any age and gender who sustained any type of blunt injury in a civilian scenario. Eligible studies had to provide sufficient information to construct a 2 x 2 table of diagnostic accuracy to allow for calculating sensitivity, specificity, and other indices of diagnostic test accuracy.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened titles, abstracts, and full texts of reports using a prespecified data extraction form. Methodological quality of individual studies was rated by the QUADAS-2 instrument (the revised and updated version of the original Quality Assessment of Diagnostic Accuracy Studies list of items). We calculated sensitivity and specificity with 95% confidence intervals (CI), tabulated the pairs of sensitivity and specificity with CI, and depicted these estimates by coupled forest plots using Review Manager 5 (RevMan 5). For pooling summary estimates of sensitivity and specificity, and investigating heterogeneity across studies, we fitted a bivariate model using Stata 14.0.

MAIN RESULTS

We included 34 studies with 8635 participants in this review. Summary estimates of sensitivity and specificity were 0.74 (95% CI 0.65 to 0.81) and 0.96 (95% CI 0.94 to 0.98). Pooled positive and negative likelihood ratios were estimated at 18.5 (95% CI 10.8 to 40.5) and 0.27 (95% CI 0.19 to 0.37), respectively. There was substantial heterogeneity across studies, and the reported accuracy of POCS strongly depended on the population and affected body area. In children, pooled sensitivity of POCS was 0.63 (95% CI 0.46 to 0.77), as compared to 0.78 (95% CI 0.69 to 0.84) in an adult or mixed population. Associated specificity in children was 0.91 (95% CI 0.81 to 0.96) and in an adult or mixed population 0.97 (95% CI 0.96 to 0.99). For abdominal trauma, POCS had a sensitivity of 0.68 (95% CI 0.59 to 0.75) and a specificity of 0.95 (95% CI 0.92 to 0.97). For chest injuries, sensitivity and specificity were calculated at 0.96 (95% CI 0.88 to 0.99) and 0.99 (95% CI 0.97 to 1.00). If we consider the results of all 34 included studies in a virtual population of 1000 patients, based on the observed median prevalence (pretest probability) of thoracoabdominal trauma of 28%, POCS would miss 73 patients with injuries and falsely suggest the presence of injuries in another 29 patients. Furthermore, in a virtual population of 1000 children, based on the observed median prevalence (pretest probability) of thoracoabdominal trauma of 31%, POCS would miss 118 children with injuries and falsely suggest the presence of injuries in another 62 children.

AUTHORS' CONCLUSIONS

In patients with suspected blunt thoracoabdominal trauma, positive POCS findings are helpful for guiding treatment decisions. However, with regard to abdominal trauma, a negative POCS exam does not rule out injuries and must be verified by a reference test such as CT. This is of particular importance in paediatric trauma, where the sensitivity of POCS is poor. Based on a small number of studies in a mixed population, POCS may have a higher sensitivity in chest injuries. This warrants larger, confirmatory trials to affirm the accuracy of POCS for diagnosing thoracic trauma.

Development and validation of a search filter to identify equity-focused studies: reducing the number needed to screen

Background

Health inequalities, worse health associated with social and economic disadvantage, are reported by a minority of research articles. Locating these studies when conducting an equity-focused systematic review is challenging due to a deficit in standardised terminology, indexing, and lack of validated search filters. Current reporting guidelines recommend not applying filters, meaning that increased resources are needed at the screening stage.

Methods

We aimed to design and test search filters to locate studies that reported outcomes by a social determinant of health. We developed and expanded a ‘specific terms strategy’ using keywords and subject headings compiled from recent systematic reviews that applied an equity filter. A ‘non-specific strategy’ was compiled from phrases used to describe equity analyses that were reported in titles and abstracts, and related subject headings. Gold standard evaluation and validation sets were compiled. The filters were developed in MEDLINE, adapted for Embase and tested in both. We set a target of 0.90 sensitivity (95% CI; 0.84, 0.94) in retrieving 150 gold standard validation papers. We noted the reduction in the number needed to screen in a proposed equity-focused systematic review and the proportion of equity-focused reviews we assessed in the project that applied an equity filter to their search strategy.

Results

The specific terms strategy filtered out 93-95% of all records, and retrieved a validation set of articles with a sensitivity of 0.84 in MEDLINE (0.77, 0.89), and 0.87 (0.81, 0.92) in Embase. When combined (Boolean ‘OR’) with the non-specific strategy sensitivity was 0.92 (0.86, 0.96) in MEDLINE (Embase 0.94; 0.89, 0.97). The number needed to screen was reduced by 77% by applying the specific terms strategy, and by 59.7% (MEDLINE) and 63.5% (Embase) by applying the combined strategy. Eighty-one per cent of systematic reviews filtered studies by equity.

Conclusions

A combined approach of using specific and non-specific terms is recommended if systematic reviewers wish to filter studies for reporting outcomes by social determinants. Future research should concentrate on the indexing standardisation for equity studies and further development and testing of both specific and non-specific terms for accurate study retrieval.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0567-x) contains supplementary material, which is available to authorized users.