Impact of PubMed search filters on the retrieval of evidence by physicians

Efficacy improvement in searching MEDLINE database using a novel PubMed visual analytic system: EEEvis

PubMed is the most extensively used database and search engine in the biomedical and healthcare fields. However, users could experience several difficulties in acquiring their target papers facing massive numbers of search results, especially in their unfamiliar fields. Therefore, we developed a novel user interface for PubMed and conducted three steps of study: step A, a preliminary user survey with 76 medical experts regarding the current usability for the biomedical literature search task at PubMed; step B is implementing EEEvis, a novel interactive visual analytic system for the search task; step C, a randomized user study comparing PubMed and EEEvis. First, we conducted a Google survey of 76 medical experts regarding the unmet needs of PubMed and the user requirements for a novel search interface. According to the data of preliminary Google survey, we implemented a novel interactive visual analytic system for biomedical literature search. This EEEvis provides enhanced literature data analysis functions including (1) an overview of the bibliographic features including publication date, citation count, and impact factors, (2) an overview of the co-authorship network, and (3) interactive sorting, filtering, and highlighting. In the randomized user study of 24 medical experts, the search speed of EEEvis was not inferior to PubMed in the time to reach the first article (median difference 3 sec, 95% CI -2.1 to 8.5, P = 0.535) nor in the search completion time (median difference 8 sec, 95% CI -4.7 to 19.1, P = 0.771). However, 22 participants (91.7%) responded that they are willing to use EEEvis as their first choice for a biomedical literature search task, and 21 participants (87.5%) answered the bibliographic sorting and filtering functionalities of EEEvis as a major advantage. EEEvis could be a supplementary interface for PubMed that can enhance the user experience in the search for biomedical literature.

Development and evaluation of a search filter to identify prognostic factor studies in Ovid MEDLINE

Background

Systematic reviews (SRs) are valuable resources as they address specific clinical questions by summarizing all existing relevant studies. However, finding all information to include in systematic reviews can be challenging. Methodological search filters have been developed to find articles related to specific clinical questions. To our knowledge, no filter exists for finding studies on the role of prognostic factor (PF). We aimed to develop and evaluate a search filter to identify PF studies in Ovid MEDLINE that has maximum sensitivity.

Methods

We followed current recommendations for the development of a search filter by first identifying a reference set of PF studies included in relevant systematic reviews on the topic, and by selecting search terms using a word frequency analysis complemented with an expert panel discussion. We evaluated filter performance using the relative recall methodology.

Results

We constructed a reference set of 73 studies included in six systematic reviews from a larger sample. After completing a word frequency analysis using the reference set studies, we compiled a list of 80 of the frequent methodological terms. This list of terms was evaluated by the Delphi panel for inclusion in the filter, resulting in a final set of 8 appropriate terms. The consecutive connection of these terms with the Boolean operator OR produced the filter. We then evaluated the filter using the relative recall method against the reference set, comparing the references included in the SRs with our new search using the filter. The overall sensitivity of the filter was calculated to be 95%, while the overall specificity was 41%. The precision of the filter varied considerably, ranging from 0.36 to 17%. The NNR (number needed to read) value varied largely from 6 to 278. The time saved by using the filter ranged from 13–70%.

Conclusions

We developed a search filter for OVID-Medline with acceptable performance that could be used in systematic reviews of PF studies. Using this filter could save as much as 40% of the title and abstract screening task. The specificity of the filter could be improved by defining additional terms to be included, although it is important to evaluate any modification to guarantee the filter is still highly sensitive.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01595-9.

Development of an efficient search filter to retrieve systematic reviews from PubMed

Objective:

Locating systematic reviews is essential for clinicians and researchers when creating or updating reviews and for decision-making in health care. This study aimed to develop a search filter for retrieving systematic reviews that improves upon the performance of the PubMed systematic review search filter.

Methods:

Search terms were identified from abstracts of reviews published in Cochrane Database of Systematic Reviews and the titles of articles indexed as systematic reviews in PubMed. Both the precision of the candidate terms and the number of systematic reviews retrieved from PubMed were evaluated after excluding the subset of articles retrieved by the PubMed systematic review filter. Terms that achieved a precision greater than 70% and relevant publication types indexed with MeSH terms were included in the filter search strategy.

Results:

The search strategy used in our filter added specific terms not included in PubMed's systematic review filter and achieved a 61.3% increase in the number of retrieved articles that are potential systematic reviews. Moreover, it achieved an average precision that is likely greater than 80%.

Conclusions:

The developed search filter will enable users to identify more systematic reviews from PubMed than the PubMed systematic review filter with high precision.

Validation of a search strategy for randomized clinical trials related to periodontitis

Background

Systematic reviews, considered the gold standard for the assessment of scientific evidence, may present conflicting findings for the same clinical issue, and such dissent may be justified by the forms of elaboration of the electronic search strategy. This paper aims to validate a search strategy to identify randomized clinical trials related to periodontitis. A gold standard reference set was developed to validate the identified clinical trials using the relative recall method. The choice of periodontitis is due to the fact that this disease has a high prevalence among chronic non-communicable diseases, is considered the second most common oral disease in the world, is associated with several health problems, such as cardiovascular diseases and diabetes, and, principally, has not been investigated sufficiently to prevent possible damages resulting from it.

Methods

A validation study was developed in MEDLINE/PubMed. In Stage 1, a methodological filter recommended by the Cochrane Collaboration to identify randomized clinical trials was applied. Stage 2 identified articles related only to periodontitis (gold standard reference set) from among the articles retrieved using the eligibility criteria. In Stage 3, a search statement for the retrieval of periodontitis-related articles was elaborated by experts. Stage 4 defined the proposed search strategy comprising of the combination of the search statement developed with the aforementioned methodological filter and subsequent application in MEDLINE/PubMed. The obtained data were analyzed using the set of articles identified in Stage 2, as the gold standard reference set. The following performance values were calculated - sensitivity, specificity, accuracy, and number needed to read - with their respective 95% confidence interval (95%CI).

Results

The search strategy under evaluation compared to the gold-standard showed a sensitivity of 93.2% (_95%CI, 83.8–97.3), specificity of 99.9% (_95%CI 99.8–99.9), and a precision of 77.5% (_95%CI, 66.48–85.63). In addition, the number needed to read was 1.3.

Conclusion

According to the proposed methodological approach, the search strategy under evaluation performed well in the identification of randomized clinical trials related to periodontitis.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13690-021-00560-0.

The yield and usefulness of PAIN+ and PubMed databases for accessing research evidence on pain management: a randomized crossover trial

Introduction

PAIN⁺ and PubMed are two electronic databases with two different mechanisms of evidence retrieval. PubMed is used to “Pull” evidence where clinicians can enter search terms to find answers while PAIN⁺ is a newly developed evidence repository where along with “Pull” service there is a “Push” service that alerts users about new research and the associated quality ratings, based on the individual preferences for content and altering criteria.

Purpose

The primary purpose of the study was to compare yield and usefulness of PubMed and PAIN⁺ in retrieving evidence to address clinical research questions on pain management. The secondary purpose of the study was to identify what search terms and methods were used by clinicians to target pain research.

Study design

Two-phase double blinded randomized crossover trial.

Methods

Clinicians (n = 76) who were exposed to PAIN⁺ for at least 1 year took part in this study. Participants were required to search for evidence 2 clinical question scenarios independently. The first clinical question was provided to all participants and thus, was multi-disciplinary. Participants were randomly assigned to search for evidence on their clinical question using either PAIN⁺ or PubMed through the electronic interface. Upon completion of the search with one search engine, they were crossed over to the other search engine. A similar process was done for a second scenario that was discipline-specific. The yield was calculated using number of retrieved articles presented to participants and usefulness was evaluated using a series of Likert scale questions embedded in the testing.

Results

Multidisciplinary scenario: Overall, the participants had an overall one-page yield of 715 articles for PAIN⁺ and 1135 articles for PubMed. The topmost article retrieved by PAIN⁺ was rated as more useful (p = 0.001). While, the topmost article retrieved by PubMed was rated as consistent with current clinical practice (p = 0.02). PubMed (48%) was preferred over PAIN⁺ (39%) to perform multidisciplinary search (p = 0.02).

Discipline specific scenario: The participants had an overall one-page yield of 1046 articles for PAIN⁺ and 1398 articles for PubMed. The topmost article retrieved by PAIN⁺ was rated as more useful (p = 0.001) and consistent with current clinical practice (p = 0.02) than the articles retrieved by PubMed. PAIN⁺ (52%) was preferred over PubMed (29%) to perform discipline specific search.

Conclusion

Clinicians from different disciplines find both PAIN⁺ and PubMed useful for retrieving research studies to address clinical questions about pain management. Greater preferences and perceived usefulness of the top 3 retrieved papers was observed for PAIN⁺, but other dimensions of usefulness did not consistently favor either search engine.

Trial registration

Registered with ClinicalTrials.gov Identifier: NCT01348802, Date: May 5, 2011.

A relevance and quality-based ranking algorithm applied to evidence-based medicine

BACKGROUND

The amount of information available about millions of different subjects is growing every day. This has led to the birth of new search tools specialized in different domains, because classical information retrieval models have trouble dealing with the special characteristics of some of these domains. Evidence-based Medicine is a case of a complex domain where classical information retrieval models can help search engines retrieve documents by considering the presence or absence of terms, but these must be complemented with other specific strategies which allow retrieving and ranking documents including the best current evidence and methodological quality.

OBJECTIVE

The goal is to present a ranking algorithm able to select the best documents for clinicians considering aspects related to the relevance and the quality of said documents.

METHODS

In order to assess the effectiveness of this proposal, an experimental methodology has been followed by using Medline as a data set and the Cochrane Library as a gold standard.

RESULTS

Applying the evaluation methodology proposed, and after submitting 40 queries on the platform developed, the MAP (Mean Average Precision) obtained was 20.26%.

CONCLUSIONS

Successful results have been achieved with the experiments, improving on other studies, but under different and even more complex circumstances.

Clinical information seeking behavior of physicians: A systematic review

INTRODUCTION

Physicians encounter a large number of questions during patient care. Therefore, finding high-quality evidence provides a good opportunity to improve patient care and continue learning.

OBJECTIVES

The present study aims to obtain a comprehensive understanding of the physicians' clinical information-seeking behavior.

METHODS

A systematic review was conducted according to the PRISMA guidelines. The Web of Science, PubMed, Scopus, ProQuest, Emerald, Wiley Online Library, Science Direct, Cochrane Library, and Embase databases were systematically searched based on the defined criteria. The inclusion criteria were the original articles in the context of patient care and full-text articles published in English from 2002 to 2017. The articles were reviewed, selected, and analyzed in group discussions.

RESULTS

Seventy-three articles met the study criteria. Therapy, diagnosis, and epidemiology ranked the highest ones among physicians' information needs. The mean frequency of questions raised during patient care was varied from 0.18 to 1.5 per patient, and the percentage of questions answered ranged between 22.8 and 93 %. The time taken to find the answers to the questions was between 2 to 32 min. Consultation with colleagues, alongside reviewing journal articles, Internet websites, textbooks, and MEDLINE/PubMed were the most frequently used sources reported in the retrieved articles. Further, common search strategies used by physicians included keywords, Boolean operators, similar medical terms, and advanced search. In addition, lack of time, and information searching skills and the unawareness of accessible sources were the most frequent barriers while easily searching and finding information and summaries and synthesized evidence-based materials mainly facilitated information seeking.

CONCLUSIONS

Physicians increasingly encounter clinical point-of-care questions. However, the facilitation of accessing online information sources has not enabled the physicians to find answers to many of their questions. Considering the fact that they often refer to colleagues, Internet websites, and databases to find answers, the quality of information should be improved by evaluating the physicians' information-seeking behavior in the first stage, and developing information technology in a point-of-care environment, integrating EHR systems to communicate with colleagues, and accessing databases in the second stage.

Optimal search strategies for identifying moderators and predictors of treatment effects in PubMed

BACKGROUND

Treatment effects differ across patients. To guide selection of treatments for patients, it is essential to acknowledge these differences and identify moderators or predictors. Our aim was to generate optimal search strategies (commonly known as filters) for PubMed to retrieve papers identifying moderators and predictors of treatment effects.

METHODS

Six journals were hand-searched for articles on moderators or predictors. Selected articles were randomly allocated to a development and validation set. Search terms were extracted from the development set and tested for their performance. Search filters were created from combinations of these terms and tested in the validation set.

RESULTS

Of 4407 articles, 198 were considered to be relevant. The most sensitive filter in the development set '("Epidemiologic Methods" [MeSH] OR assign* OR control*[tiab] OR trial*[tiab]) AND therapy*[sh]' yielded in the validation set a sensitivity of 89% [88%-90%] and a specificity of 80% [79%-82%].

CONCLUSIONS

The search filters created in this study can help to efficiently retrieve evidence on moderators and predictors of treatment effect. Testing of the filters in multiple domains should reveal robustness across disciplines. These filters can facilitate the retrieval of evidence on moderators and predictors of treatment effects, helping the implementation of stratified or personalised health care.

Physicians' perception of alternative displays of clinical research evidence for clinical decision support - A study with case vignettes

OBJECTIVE

To design alternate information displays that present summaries of clinical trial results to clinicians to support decision-making; and to compare the displays according to efficacy and acceptability.

METHODS

A 6-between (information display presentation order) by 3-within (display type) factorial design. Two alternate displays were designed based on Information Foraging theory: a narrative summary that reduces the content to a few sentences; and a table format that structures the display according to the PICO (Population, Intervention, Comparison, Outcome) framework. The designs were compared with the summary display format available in PubMed. Physicians were asked to review five clinical studies retrieved for a case vignette; and were presented with the three display formats. Participants were asked to rate their experience with each of the information displays according to a Likert scale questionnaire.

RESULTS

Twenty physicians completed the study. Overall, participants rated the table display more highly than either the text summary or PubMed's summary format (5.9vs. 5.4vs. 3.9 on a scale between 1 [strongly disagree] and 7 [strongly agree]). Usefulness ratings of seven pieces of information, i.e. patient population, patient age range, sample size, study arm, primary outcome, results of primary outcome, and conclusion, were high (average across all items=4.71 on a 1 to 5 scale, with 1=not at all useful and 5=very useful). Study arm, primary outcome, and conclusion scored the highest (4.9, 4.85, and 4.85 respectively). Participants suggested additional details such as rate of adverse effects.

CONCLUSION

The table format reduced physicians' perceived cognitive effort when quickly reviewing clinical trial information and was more favorably received by physicians than the narrative summary or PubMed's summary format display.

An automated system for retrieving herb-drug interaction related articles from MEDLINE

An automated, user-friendly and accurate system for retrieving herb-drug interaction (HDIs) related articles in MEDLINE can increase the safety of patients, as well as improve the physicians' article retrieving ability regarding speed and experience. Previous studies show that MeSH based queries associated with negative effects of drugs can be customized, resulting in good performance in retrieving relevant information, but no study has focused on the area of herb-drug interactions (HDI). This paper adapted the characteristics of HDI related papers and created a multilayer HDI article searching system. It achieved a sensitivity of 92% at a precision of 93% in a preliminary evaluation. Instead of requiring physicians to conduct PubMed searches directly, this system applies a more user-friendly approach by employing a customized system that enhances PubMed queries, shielding users from having to write queries, dealing with PubMed, or reading many irrelevant articles. The system provides automated processes and outputs target articles based on the input.

Factors affecting scientific productivity of German oral-maxillofacial surgery training centers: a retrospective cohort study

PURPOSE

To identify factors associated with scientific productivity (SP) of German oral-maxillofacial surgery (OMFS) training centers.

MATERIALS AND METHODS

This retrospective cohort study was composed of a set of data from German OMFS training centers. A total of eight predictor variables were grouped into demographic, structural, and personal categories. The outcome variables were average publications in 2013 per senior staff, and percentage of OMFS trainees with >1 publications. Descriptive and univariate statistics were computed using P < 0.05.

RESULTS

The sample included outputs from 62 OMFS departments (34 [54.8 %] university-based; 46 [74.2 %] in large cities). Average publications were 2.4 ± 3 per senior staff (range, 0-27), and 160 trainees (31.7 %) published >1 papers. The number of publications and productive trainees was not linked to department name and number of female senior staffs, but publication count was significantly increased when the hospital was in a metropolis (P = 0.018) or university-based (P < 0.0001), the OMFS' chairperson and >3 staffs within the department had a postdoctoral degree (German "Habilitation") (P = 0.013 and <0.0001), and the chairperson had h-index >10 or the first/last authorship in 2013 (P < 0.0001). Female senior surgeons were less scientifically productive than the male ones (P = 0.01).

CONCLUSION

SP of German OMFS training centers is greatly different across the country and influenced by city size, university base, educational backgrounds, and research activities of chairpersons and senior staffs. This helps students, trainees, and young surgeons to reach the career choice that is personally appropriate. The involved organizations may need to encourage research output of less productive surgeons/centers. Increasing postdoctoral-qualified staffs will increase SP of the department.

Analysis of PubMed User Sessions Using a Full-Day PubMed Query Log: A Comparison of Experienced and Nonexperienced PubMed Users

Background

PubMed is the largest biomedical bibliographic information source on the Internet. PubMed has been considered one of the most important and reliable sources of up-to-date health care evidence. Previous studies examined the effects of domain expertise/knowledge on search performance using PubMed. However, very little is known about PubMed users’ knowledge of information retrieval (IR) functions and their usage in query formulation.

Objective

The purpose of this study was to shed light on how experienced/nonexperienced PubMed users perform their search queries by analyzing a full-day query log. Our hypotheses were that (1) experienced PubMed users who use system functions quickly retrieve relevant documents and (2) nonexperienced PubMed users who do not use them have longer search sessions than experienced users.

Methods

To test these hypotheses, we analyzed PubMed query log data containing nearly 3 million queries. User sessions were divided into two categories: experienced and nonexperienced. We compared experienced and nonexperienced users per number of sessions, and experienced and nonexperienced user sessions per session length, with a focus on how fast they completed their sessions.

Results

To test our hypotheses, we measured how successful information retrieval was (at retrieving relevant documents), represented as the decrease rates of experienced and nonexperienced users from a session length of 1 to 2, 3, 4, and 5. The decrease rate (from a session length of 1 to 2) of the experienced users was significantly larger than that of the nonexperienced groups.

Conclusions

Experienced PubMed users retrieve relevant documents more quickly than nonexperienced PubMed users in terms of session length.

A search strategy to identify studies on the prognosis of work disability: a diagnostic test framework

OBJECTIVE

Searching the medical literature for evidence on prognosis is an important aspect of evidence-based disability evaluation. To facilitate this, we aimed to develop and evaluate a comprehensive and efficient search strategy in PubMed, to be used by either researchers or practitioners and that will identify articles on the prognosis of work disability.

METHODS

We used a diagnostic test analytic framework. First, we created a reference set of 225 articles on the prognosis of work disability by screening a total of 65,692 titles and abstracts from10 journals in the period 2000-2009. Included studies had a minimum follow-up of 6 months, participants in the age of 18-64 with a minimum sick leave of 4 weeks or longer or having serious activity limitations in 50% of the cases and outcome measures that reflect impairments, activity limitations or participation restrictions. Using text mining methods, we extracted search terms from the reference set and, according to sensitivity and relative frequency, we combined these into search strings.

RESULTS

Both the research and the practice search filter outperformed existing filters in occupational health, all combined with the Yale-prognostic filter. The Work Disability Prognosis filter for Research showed a comprehensiveness of 90% (95% CI 86 to 94) and efficiency expressed more user-friendly as Number Needed to Read=20 (95% CI 17 to 34).

CONCLUSIONS

The Work Disability Prognosis filter will help practitioners and researchers who want to find prognostic evidence in the area of work disability evaluation. However, further refining of this filter is possible and needed, especially for the practitioner for whom efficiency is especially important.

Automated Determination of Publications Related to Adverse Drug Reactions in PubMed

Timely dissemination of up-to-date information concerning adverse drug reactions (ADRs) at the point of care can significantly improve medication safety and prevent ADRs. Automated methods for finding relevant articles in MEDLINE which discuss ADRs for specific medications can facilitate decision making at the point of care. Previous work has focused on other types of clinical queries and on retrieval for specific ADRs or drug-ADR pairs, but little work has been published on finding ADR articles for a specific medication. We have developed a method to generate a PubMED query based on MESH, supplementary concepts, and textual terms for a particular medication. Evaluation was performed on a limited sample, resulting in a sensitivity of 90% and precision of 93%. Results demonstrated that this method is highly effective. Future work will integrate this method within an interface aimed at facilitating access to ADR information for specified drugs at the point of care.

Using PubMed search strings for efficient retrieval of manual therapy research literature

OBJECTIVE

The aim of this study was to construct PubMed search strings that could efficiently retrieve studies on manual therapy (MT), especially for time-constrained clinicians.

METHODS

Our experts chose 11 Medical Subject Heading terms describing MT along with 84 additional potential terms. For each term that was able to retrieve more than 100 abstracts, we systematically extracted a sample of abstracts from which we estimated the proportion of studies potentially relevant to MT. We then constructed 2 search strings: 1 narrow (threshold of pertinent articles ≥40%) and 1 expanded (including all terms for which a proportion had been calculated). We tested these search strings against articles on 2 conditions relevant to MT (thoracic and temporomandibular pain). We calculated the number of abstracts needed to read (NNR) to identify 1 potentially pertinent article in the context of these conditions. Finally, we evaluated the efficiency of the proposed PubMed search strings to identify relevant articles included in a systematic review on spinal manipulative therapy for chronic low back pain.

RESULTS

Fifty-five search terms were able to extract more than 100 citations. The NNR to find 1 potentially pertinent article using the narrow string was 1.2 for thoracic pain and 1.3 for temporomandibular pain, and the NNR for the expanded string was 1.9 and 1.6, respectively. The narrow search strategy retrieved all the randomized controlled trials included in the systematic review selected for comparison.

CONCLUSION

The proposed PubMed search strings may help health care professionals locate potentially pertinent articles and review a large number of MT studies efficiently to better implement evidence-based practice.

All Information Is Not Equal: Using the Literature Databases PubMed and The Cochrane Library for Identifying the Evidence on Granulocyte Transfusion Therapy

To be able to take well-informed decisions or carry out sound research, clinicians and researchers alike require specific information seeking skills matching their respective information needs. Biomedical information is traditionally available via different literature databases. This article gives an introduction to two diverging sources, PubMed (23 million references) and The Cochrane Library (800,000 references), both of which offer sophisticated instruments for searching an increasing amount of medical publications of varied quality and ambition. Whereas PubMed as an unfiltered source of primary literature comprises all different kinds of publication types occurring in academic journals, The Cochrane Library is a pre-filtered source which offers access to either synthesized publication types or critically appraised and carefully selected references. A search approach has to be carried out deliberately and requires a good knowledge on the scope and features of the databases as well as on the ability to build a search strategy in a structured way. We present a specific and a sensitive search approach, making use of both databases within two application case scenarios in order to identify the evidence on granulocyte transfusions for infections in adult patients with neutropenia.

High-performance information search filters for acute kidney injury content in PubMed, Ovid Medline and Embase

BACKGROUND

We frequently fail to identify articles relevant to the subject of acute kidney injury (AKI) when searching the large bibliographic databases such as PubMed, Ovid Medline or Embase. To address this issue, we used computer automation to create information search filters to better identify articles relevant to AKI in these databases.

METHODS

We first manually reviewed a sample of 22 992 full-text articles and used prespecified criteria to determine whether each article contained AKI content or not. In the development phase (two-thirds of the sample), we developed and tested the performance of >1.3-million unique filters. Filters with high sensitivity and high specificity for the identification of AKI articles were then retested in the validation phase (remaining third of the sample).

RESULTS

We succeeded in developing and validating high-performance AKI search filters for each bibliographic database with sensitivities and specificities in excess of 90%. Filters optimized for sensitivity reached at least 97.2% sensitivity, and filters optimized for specificity reached at least 99.5% specificity. The filters were complex; for example one PubMed filter included >140 terms used in combination, including 'acute kidney injury', 'tubular necrosis', 'azotemia' and 'ischemic injury'. In proof-of-concept searches, physicians found more articles relevant to topics in AKI with the use of the filters.

CONCLUSIONS

PubMed, Ovid Medline and Embase can be filtered for articles relevant to AKI in a reliable manner. These high-performance information filters are now available online and can be used to better identify AKI content in large bibliographic databases.

Retrieving clinical evidence: a comparison of PubMed and Google Scholar for quick clinical searches

BACKGROUND

Physicians frequently search PubMed for information to guide patient care. More recently, Google Scholar has gained popularity as another freely accessible bibliographic database.

OBJECTIVE

To compare the performance of searches in PubMed and Google Scholar.

METHODS

We surveyed nephrologists (kidney specialists) and provided each with a unique clinical question derived from 100 renal therapy systematic reviews. Each physician provided the search terms they would type into a bibliographic database to locate evidence to answer the clinical question. We executed each of these searches in PubMed and Google Scholar and compared results for the first 40 records retrieved (equivalent to 2 default search pages in PubMed). We evaluated the recall (proportion of relevant articles found) and precision (ratio of relevant to nonrelevant articles) of the searches performed in PubMed and Google Scholar. Primary studies included in the systematic reviews served as the reference standard for relevant articles. We further documented whether relevant articles were available as free full-texts.

RESULTS

Compared with PubMed, the average search in Google Scholar retrieved twice as many relevant articles (PubMed: 11%; Google Scholar: 22%; P<.001). Precision was similar in both databases (PubMed: 6%; Google Scholar: 8%; P=.07). Google Scholar provided significantly greater access to free full-text publications (PubMed: 5%; Google Scholar: 14%; P<.001).

CONCLUSIONS

For quick clinical searches, Google Scholar returns twice as many relevant articles as PubMed and provides greater access to free full-text articles.

Developing topic-specific search filters for PubMed with click-through data

OBJECTIVES

Search filters have been developed and demonstrated for better information access to the immense and ever-growing body of publications in the biomedical domain. However, to date the number of filters remains quite limited because the current filter development methods require significant human efforts in manual document review and filter term selection. In this regard, we aim to investigate automatic methods for generating search filters.

METHODS

We present an automated method to develop topic-specific filters on the basis of users' search logs in PubMed. Specifically, for a given topic, we first detect its relevant user queries and then include their corresponding clicked articles to serve as the topic-relevant document set accordingly. Next, we statistically identify informative terms that best represent the topic-relevant document set using a background set composed of topic irrelevant articles. Lastly, the selected representative terms are combined with Boolean operators and evaluated on benchmark datasets to derive the final filter with the best performance.

RESULTS

We applied our method to develop filters for four clinical topics: nephrology, diabetes, pregnancy, and depression. For the nephrology filter, our method obtained performance comparable to the state of the art (sensitivity of 91.3%, specificity of 98.7%, precision of 94.6%, and accuracy of 97.2%). Similarly, high-performing results (over 90% in all measures) were obtained for the other three search filters.

CONCLUSION

Based on PubMed click-through data, we successfully developed a high-performance method for generating topic-specific search filters that is significantly more efficient than existing manual methods. All data sets (topic-relevant and irrelevant document sets) used in this study and a demonstration system are publicly available at http://www.ncbi.nlm.nih.gov/CBBresearch/Lu/downloads/CQ_filter/

Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures

BACKGROUND

Two of the current methodological barriers to implementation science efforts are the lack of agreement regarding constructs hypothesized to affect implementation success and identifiable measures of these constructs. In order to address these gaps, the main goals of this paper were to identify a multi-level framework that captures the predominant factors that impact implementation outcomes, conduct a systematic review of available measures assessing constructs subsumed within these primary factors, and determine the criterion validity of these measures in the search articles.

METHOD

We conducted a systematic literature review to identify articles reporting the use or development of measures designed to assess constructs that predict the implementation of evidence-based health innovations. Articles published through 12 August 2012 were identified through MEDLINE, CINAHL, PsycINFO and the journal Implementation Science. We then utilized a modified five-factor framework in order to code whether each measure contained items that assess constructs representing structural, organizational, provider, patient, and innovation level factors. Further, we coded the criterion validity of each measure within the search articles obtained.

RESULTS

Our review identified 62 measures. Results indicate that organization, provider, and innovation-level constructs have the greatest number of measures available for use, whereas structural and patient-level constructs have the least. Additionally, relatively few measures demonstrated criterion validity, or reliable association with an implementation outcome (e.g., fidelity).

DISCUSSION

In light of these findings, our discussion centers on strategies that researchers can utilize in order to identify, adapt, and improve extant measures for use in their own implementation research. In total, our literature review and resulting measures compendium increases the capacity of researchers to conceptualize and measure implementation-related constructs in their ongoing and future research.

MEDLINE clinical queries are robust when searching in recent publishing years

OBJECTIVE

To determine if the PubMed and Ovid MEDLINE clinical queries (which were developed in the publishing year 2000, for the purpose categories therapy, diagnosis, prognosis, etiology, and clinical prediction guides) perform as well when searching in current publishing years.

METHODS

A gold standard database of recently published research literature was created using the McMaster health knowledge refinery (http://hiru.mcmaster.ca/hiru/HIRU_McMaster_HKR.aspx) and its continuously updated database, McMaster PLUS (http://hiru.mcmaster.ca/hiru/HIRU_McMaster_PLUS_projects.aspx). This database contains articles from over 120 clinical journals that are tagged for meeting or not meeting criteria for scientific merit and clinical relevance. The clinical queries sensitive ('broad') and specific ('narrow') search filters were tested in this gold standard database, and sensitivity and specificity were calculated and compared with those originally reported for the clinical queries.

RESULTS

In all cases, the sensitivity of the highly sensitive search filters and the specificity of the highly specific search filters did not differ substantively when comparing results derived in 2000 with those derived in a more current database. In addition, in all cases, the specificities for the highly sensitive search filters and the sensitivities for the highly specific search filters remained above 50% when testing them in the current database.

DISCUSSION

These results are reassuring for modern-day searchers. The clinical queries that were derived in the year 2000 perform equally well a decade later.

CONCLUSION

The PubMed and Ovid MEDLINE clinical queries have been revalidated and remain a useful public resource for searching the world's medical literature for research that is most relevant to clinical care.

Glomerular disease search filters for Pubmed, Ovid Medline, and Embase: a development and validation study

BACKGROUND

Tools to enhance physician searches of Medline and other bibliographic databases have potential to improve the application of new knowledge in patient care. This is particularly true for articles about glomerular disease, which are published across multiple disciplines and are often difficult to track down. Our objective was to develop and test search filters for PubMed, Ovid Medline, and Embase that allow physicians to search within a subset of the database to retrieve articles relevant to glomerular disease.

METHODS

We used a diagnostic test assessment framework with development and validation phases. We read a total of 22,992 full text articles for relevance and assigned them to the development or validation set to define the reference standard. We then used combinations of search terms to develop 997,298 unique glomerular disease filters. Outcome measures for each filter included sensitivity, specificity, precision, and accuracy. We selected optimal sensitive and specific search filters for each database and applied them to the validation set to test performance.

RESULTS

High performance filters achieved at least 93.8% sensitivity and specificity in the development set. Filters optimized for sensitivity reached at least 96.7% sensitivity and filters optimized for specificity reached at least 98.4% specificity. Performance of these filters was consistent in the validation set and similar among all three databases.

CONCLUSIONS

PubMed, Ovid Medline, and Embase can be filtered for articles relevant to glomerular disease in a reliable manner. These filters can now be used to facilitate physician searching.