Analysis of queries sent to PubMed at the point of care: observation of search behaviour in a medical teaching hospital

Representation of Social Determinants of Health terminology in medical subject headings: impact of added terms

OBJECTIVES

To enhance and evaluate the quality of PubMed search results for Social Determinants of Health (SDoH) through the addition of new SDoH terms to Medical Subject Headings (MeSH).

MATERIALS AND METHODS

High priority SDoH terms and definitions were collated from authoritative sources, curated based on publication frequencies, and refined by subject matter experts. Descriptive analyses were used to investigate how PubMed search details and best match results were affected by the addition of SDoH concepts to MeSH. Three information retrieval metrics (Precision, Recall, and F measure) were used to quantitatively assess the accuracy of PubMed search results. Pre- and post-update documents were clustered into topic areas using a Natural Language Processing pipeline, and SDoH relevancy assessed.

RESULTS

Addition of 35 SDoH terms to MeSH resulted in more accurate algorithmic translations of search terms and more reliable best match results. The Precision, Recall, and F measures of post-update results were significantly higher than those of pre-update results. The percentage of retrieved publications belonging to SDoH clusters was significantly greater in the post- than pre-update searches.

DISCUSSION

This evaluation confirms that inclusion of new SDoH terms in MeSH can lead to qualitative and quantitative enhancements in PubMed search retrievals. It demonstrates the methodology for and impact of suggesting new terms for MeSH indexing. It provides a foundation for future efforts across behavioral and social science research (BSSR) domains.

CONCLUSION

Improving the representation of BSSR terminology in MeSH can improve PubMed search results, thereby enhancing the ability of investigators and clinicians to build and utilize a cumulative BSSR knowledge base.

Online Searching as a Practice for Evidence-Based Medicine in the Neonatal Intensive Care Unit, University of Malaya Medical Center, Malaysia: Cross-sectional Study

Background

The use of the internet for research is essential in the practice of evidence-based medicine. The online search habits of medical practitioners in clinical settings, particularly from direct observation, have received little attention.

Objective

The goal of the research is to explore online searching for information as an evidence-based practice among medical practitioners.

Methods

A cross-sectional study was conducted to evaluate the clinical teams’ use of evidence-based practice when making clinical decisions for their patients' care. Data were collected through online searches from 2015 to 2018. Participants were medical practitioners and medical students in a Malaysian public teaching hospital’s neonatal intensive care unit who performed online searches to find answers to clinical questions that arose during ward rounds.

Results

In search sessions conducted by the participants, 311 queries were observed from 2015 to 2018. Most participants (34/47, 72%) were house officers and medical students. Most of the searches were conducted by house officers (51/99, 52%) and medical students (32/99, 32%). Most searches (70/99, 71%) were directed rather than self-initiated, and 90% (89/99) were completed individually rather than collaboratively. Participants entered an average of 4 terms in each query; three-quarters of the queries yielded relevant evidence, with two-thirds yielding more than one relevant source of evidence.

Conclusions

Our findings suggest that junior doctors and medical students need more training in evidence-based medicine skills such as clinical question formulation and online search techniques for performing independent online searches effectively. However, because the findings were based on intermittent opportunistic observations in a specific clinical setting, they may not be generalizable.

Identification of the Best Semantic Expansion to Query PubMed Through Automatic Performance Assessment of Four Search Strategies on All Medical Subject Heading Descriptors: Comparative Study

Background

With the continuous expansion of available biomedical data, efficient and effective information retrieval has become of utmost importance. Semantic expansion of queries using synonyms may improve information retrieval.

Objective

The aim of this study was to automatically construct and evaluate expanded PubMed queries of the form “preferred term”[MH] OR “preferred term”[TIAB] OR “synonym 1”[TIAB] OR “synonym 2”[TIAB] OR …, for each of the 28,313 Medical Subject Heading (MeSH) descriptors, by using different semantic expansion strategies. We sought to propose an innovative method that could automatically evaluate these strategies, based on the three main metrics used in information science (precision, recall, and F-measure).

Methods

Three semantic expansion strategies were assessed. They differed by the synonyms used to build the queries as follows: MeSH synonyms, Unified Medical Language System (UMLS) mappings, and custom mappings (Catalogue et Index des Sites Médicaux de langue Française [CISMeF]). The precision, recall, and F-measure metrics were automatically computed for the three strategies and for the standard automatic term mapping (ATM) of PubMed. The method to automatically compute the metrics involved computing the number of all relevant citations (A), using National Library of Medicine indexing as the gold standard (“preferred term”[MH]), the number of citations retrieved by the added terms (”synonym 1“[TIAB] OR ”synonym 2“[TIAB] OR …) (B), and the number of relevant citations retrieved by the added terms (combining the previous two queries with an “AND” operator) (C). It was possible to programmatically compute the metrics for each strategy using each of the 28,313 MeSH descriptors as a “preferred term,” corresponding to 239,724 different queries built and sent to the PubMed application program interface. The four search strategies were ranked and compared for each metric.

Results

ATM had the worst performance for all three metrics among the four strategies. The MeSH strategy had the best mean precision (51%, SD 23%). The UMLS strategy had the best recall and F-measure (41%, SD 31% and 36%, SD 24%, respectively). CISMeF had the second best recall and F-measure (40%, SD 31% and 35%, SD 24%, respectively). However, considering a cutoff of 5%, CISMeF had better precision than UMLS for 1180 descriptors, better recall for 793 descriptors, and better F-measure for 678 descriptors.

Conclusions

This study highlights the importance of using semantic expansion strategies to improve information retrieval. However, the performances of a given strategy, relatively to another, varied greatly depending on the MeSH descriptor. These results confirm there is no ideal search strategy for all descriptors. Different semantic expansions should be used depending on the descriptor and the user’s objectives. Thus, we developed an interface that allows users to input a descriptor and then proposes the best semantic expansion to maximize the three main metrics (precision, recall, and F-measure).

Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features

OBJECTIVES

Finding recent clinical studies that warrant changes in clinical practice ("high impact" clinical studies) in a timely manner is very challenging. We investigated a machine learning approach to find recent studies with high clinical impact to support clinical decision making and literature surveillance.

METHODS

To identify recent studies, we developed our classification model using time-agnostic features that are available as soon as an article is indexed in PubMed®, such as journal impact factor, author count, and study sample size. Using a gold standard of 541 high impact treatment studies referenced in 11 disease management guidelines, we tested the following null hypotheses: (1) the high impact classifier with time-agnostic features (HI-TA) performs equivalently to PubMed's Best Match sort and a MeSH-based Naïve Bayes classifier; and (2) HI-TA performs equivalently to the high impact classifier with both time-agnostic and time-sensitive features (HI-TS) enabled in a previous study. The primary outcome for both hypotheses was mean top 20 precision.

RESULTS

The differences in mean top 20 precision between HI-TA and three baselines (PubMed's Best Match, a MeSH-based Naïve Bayes classifier, and HI-TS) were not statistically significant (12% vs. 3%, p = 0.101; 12% vs. 11%, p = 0.720; 12% vs. 25%, p = 0.094, respectively). Recall of HI-TA was low (7%).

CONCLUSION

HI-TA had equivalent performance to state-of-the-art approaches that depend on time-sensitive features. With the advantage of relying only on time-agnostic features, the proposed approach can be used as an adjunct to help clinicians identify recent high impact clinical studies to support clinical decision-making. However, low recall limits the use of HI-TA for literature surveillance.

PEDro searching has improved over time: A comparison of search commands from two six-month periods three years apart

BACKGROUND

In 2014-2015, the Physiotherapy Evidence Database (PEDro) was searched poorly by users; few search commands used sophisticated features and ∼20% contained errors. To improve the quality of PEDro searches, users now receive error messages when using incorrect search commands and have access to video tutorials.

OBJECTIVES

To determine whether search quality has improved since error messages and tutorials were implemented; and evaluate the content of PEDro searches.

METHODS

Google Analytics was used to access all search commands on PEDro (between 1 August 2017 and 31 January 2018) and extract the following data: total number of search commands; 25 most common simple and advanced search commands; and frequency of search errors (e.g. Boolean operators) or use of sophisticated features (e.g. truncation/wildcards). Two researchers independently coded the subdiscipline (e.g. musculoskeletal, neurology) and PICO elements (Population; Intervention; Comparison; Outcome) from a random sample of 200 simple and 200 advanced search commands. Data were compared to an identical analysis performed in 2014-2015 to determine whether the content or quality of search commands had changed.

RESULTS

There has been a very small increase in the use of truncation/wildcards since 2014-2015 (1.4% increase in simple and 1.9% in advanced search commands; p < 0.001) and small reductions in search errors (Boolean operators: 3.7% reduction in simple and 3.2% in advanced; brackets: 0.9% and 0.4%; non-ASCII characters: 3.1% and 1.6%; p < 0.001 for all analyses). Overall, only 6% of simple and 9% of advanced search commands used sophisticated features, while 16% of simple and 12% of advanced search commands contained errors. The content of PEDro search commands was largely similar to searches from 2014 to 2015.

CONCLUSION

There has been a small reduction in the number of search commands containing errors, and only a very small increase in the use of sophisticated features. These improvements may be explained by video tutorials on how to optimise searching and warnings that appear when users enter search commands containing errors. However, with 16% of simple and 12% of advanced search commands still containing errors, additional strategies to further improve the quality of searches are needed.

Automatic identification of high impact articles in PubMed to support clinical decision making

OBJECTIVES

The practice of evidence-based medicine involves integrating the latest best available evidence into patient care decisions. Yet, critical barriers exist for clinicians' retrieval of evidence that is relevant for a particular patient from primary sources such as randomized controlled trials and meta-analyses. To help address those barriers, we investigated machine learning algorithms that find clinical studies with high clinical impact from PubMed®.

METHODS

Our machine learning algorithms use a variety of features including bibliometric features (e.g., citation count), social media attention, journal impact factors, and citation metadata. The algorithms were developed and evaluated with a gold standard composed of 502 high impact clinical studies that are referenced in 11 clinical evidence-based guidelines on the treatment of various diseases. We tested the following hypotheses: (1) our high impact classifier outperforms a state-of-the-art classifier based on citation metadata and citation terms, and PubMed's® relevance sort algorithm; and (2) the performance of our high impact classifier does not decrease significantly after removing proprietary features such as citation count.

RESULTS

The mean top 20 precision of our high impact classifier was 34% versus 11% for the state-of-the-art classifier and 4% for PubMed's® relevance sort (p=0.009); and the performance of our high impact classifier did not decrease significantly after removing proprietary features (mean top 20 precision=34% vs. 36%; p=0.085).

CONCLUSION

The high impact classifier, using features such as bibliometrics, social media attention and MEDLINE® metadata, outperformed previous approaches and is a promising alternative to identifying high impact studies for clinical decision support.

Searching for rare diseases in PubMed: a blind comparison of Orphanet expert query and query based on terminological knowledge

BACKGROUND

Despite international initiatives like Orphanet, it remains difficult to find up-to-date information about rare diseases. The aim of this study is to propose an exhaustive set of queries for PubMed based on terminological knowledge and to evaluate it versus the queries based on expertise provided by the most frequently used resource in Europe: Orphanet.

METHODS

Four rare disease terminologies (MeSH, OMIM, HPO and HRDO) were manually mapped to each other permitting the automatic creation of expended terminological queries for rare diseases. For 30 rare diseases, 30 citations retrieved by Orphanet expert query and/or query based on terminological knowledge were assessed for relevance by two independent reviewers unaware of the query's origin. An adjudication procedure was used to resolve any discrepancy. Precision, relative recall and F-measure were all computed.

RESULTS

For each Orphanet rare disease (n = 8982), there was a corresponding terminological query, in contrast with only 2284 queries provided by Orphanet. Only 553 citations were evaluated due to queries with 0 or only a few hits. There were no significant differences between the Orpha query and terminological query in terms of precision, respectively 0.61 vs 0.52 (p = 0.13). Nevertheless, terminological queries retrieved more citations more often than Orpha queries (0.57 vs. 0.33; p = 0.01). Interestingly, Orpha queries seemed to retrieve older citations than terminological queries (p < 0.0001).

CONCLUSION

The terminological queries proposed in this study are now currently available for all rare diseases. They may be a useful tool for both precision or recall oriented literature search.

An automated system for retrieving herb-drug interaction related articles from MEDLINE

An automated, user-friendly and accurate system for retrieving herb-drug interaction (HDIs) related articles in MEDLINE can increase the safety of patients, as well as improve the physicians' article retrieving ability regarding speed and experience. Previous studies show that MeSH based queries associated with negative effects of drugs can be customized, resulting in good performance in retrieving relevant information, but no study has focused on the area of herb-drug interactions (HDI). This paper adapted the characteristics of HDI related papers and created a multilayer HDI article searching system. It achieved a sensitivity of 92% at a precision of 93% in a preliminary evaluation. Instead of requiring physicians to conduct PubMed searches directly, this system applies a more user-friendly approach by employing a customized system that enhances PubMed queries, shielding users from having to write queries, dealing with PubMed, or reading many irrelevant articles. The system provides automated processes and outputs target articles based on the input.

What Searches Do Users Run on PEDro? An Analysis of 893,971 Search Commands Over a 6-Month Period

BACKGROUND

Clinicians must be able to search effectively for relevant research if they are to provide evidence-based healthcare. It is therefore relevant to consider how users search databases of evidence in healthcare, including what information users look for and what search strategies they employ. To date such analyses have been restricted to the PubMed database. Although the Physiotherapy Evidence Database (PEDro) is searched millions of times each year, no studies have investigated how users search PEDro.

OBJECTIVES

To assess the content and quality of searches conducted on PEDro.

METHODS

Searches conducted on the PEDro website over 6 months were downloaded and the 'get' commands and page-views extracted. The following data were tabulated: the 25 most common searches; the number of search terms used; the frequency of use of simple and advanced searches, including the use of each advanced search field; and the frequency of use of various search strategies.

RESULTS

Between August 2014 and January 2015, 893,971 search commands were entered on PEDro. Fewer than 18 % of these searches used the advanced search features of PEDro. 'Musculoskeletal' was the most common subdiscipline searched, while 'low back pain' was the most common individual search. Around 20 % of all searches contained errors.

CONCLUSIONS

PEDro is a commonly used evidence resource, but searching appears to be sub-optimal in many cases. The effectiveness of searches conducted by users needs to improve, which could be facilitated by methods such as targeted training and amending the search interface.

Classification of clinically useful sentences in clinical evidence resources

Most patient care questions raised by clinicians can be answered by online clinical knowledge resources. However, important barriers still challenge the use of these resources at the point of care.

OBJECTIVE

To design and assess a method for extracting clinically useful sentences from synthesized online clinical resources that represent the most clinically useful information for directly answering clinicians' information needs.

MATERIALS AND METHODS

We developed a Kernel-based Bayesian Network classification model based on different domain-specific feature types extracted from sentences in a gold standard composed of 18 UpToDate documents. These features included UMLS concepts and their semantic groups, semantic predications extracted by SemRep, patient population identified by a pattern-based natural language processing (NLP) algorithm, and cue words extracted by a feature selection technique. Algorithm performance was measured in terms of precision, recall, and F-measure.

RESULTS

The feature-rich approach yielded an F-measure of 74% versus 37% for a feature co-occurrence method (p<0.001). Excluding predication, population, semantic concept or text-based features reduced the F-measure to 62%, 66%, 58% and 69% respectively (p<0.01). The classifier applied to Medline sentences reached an F-measure of 73%, which is equivalent to the performance of the classifier on UpToDate sentences (p=0.62).

CONCLUSIONS

The feature-rich approach significantly outperformed general baseline methods. This approach significantly outperformed classifiers based on a single type of feature. Different types of semantic features provided a unique contribution to overall classification performance. The classifier's model and features used for UpToDate generalized well to Medline abstracts.

Advances in website information resources to aid in clinical practice

The World Wide Web, which has been widely implemented for roughly two decades, is humankind's most impressive effort to aggregate and organize knowledge to date. The medical community was slower to embrace the Internet than others, but the majority of clinicians now use it as part of their everyday practice. For the practicing oncologist, there is a daunting quantity of information to master. For example, a new article relating to cancer is added to the MEDLINE database approximately every 3 minutes. Fortunately, Internet resources can help organize the deluge of information into useful knowledge. This manuscript provides an overview of resources related to general medicine, oncology, and social media that will be of practical use to the practicing oncologist. It is clear from the vast size of the Internet that we are all life-long learners, and the challenge is to acquire "just-in-time" information so that we can provide the best possible care to our patients. The resources that we have presented in this article should help the practicing oncologist continue along the path of transforming information to knowledge to wisdom.

Classification of Clinically Useful Sentences in MEDLINE

OBJECTIVE

In a previous study, we investigated a sentence classification model that uses semantic features to extract clinically useful sentences from UpToDate, a synthesized clinical evidence resource. In the present study, we assess the generalizability of the sentence classifier to Medline abstracts.

METHODS

We applied the classification model to an independent gold standard of high quality clinical studies from Medline. Then, the classifier trained on UpToDate sentences was optimized by re-retraining the classifier with Medline abstracts and adding a sentence location feature.

RESULTS

The previous classifier yielded an F-measure of 58% on Medline versus 67% on UpToDate. Re-training the classifier on Medline improved F-measure to 68%; and to 76% (p<0.01) after adding the sentence location feature.

CONCLUSIONS

The classifier's model and input features generalized to Medline abstracts, but the classifier needed to be retrained on Medline to achieve equivalent performance. Sentence location provided additional contribution to the overall classification performance.

Automated Determination of Publications Related to Adverse Drug Reactions in PubMed

Timely dissemination of up-to-date information concerning adverse drug reactions (ADRs) at the point of care can significantly improve medication safety and prevent ADRs. Automated methods for finding relevant articles in MEDLINE which discuss ADRs for specific medications can facilitate decision making at the point of care. Previous work has focused on other types of clinical queries and on retrieval for specific ADRs or drug-ADR pairs, but little work has been published on finding ADR articles for a specific medication. We have developed a method to generate a PubMED query based on MESH, supplementary concepts, and textual terms for a particular medication. Evaluation was performed on a limited sample, resulting in a sensitivity of 90% and precision of 93%. Results demonstrated that this method is highly effective. Future work will integrate this method within an interface aimed at facilitating access to ADR information for specified drugs at the point of care.

LIt.search: fast tracking access to Aboriginal and Torres Strait Islander health literature

OBJECTIVE

To develop and validate a PubMed search filter, LIt.search, that automatically retrieves Aboriginal and Torres Strait Islander health literature and to make it publicly accessible through the Lowitja Institute website.

METHODS

Search filter development phases included: (1) scoping of the publication characteristics of Aboriginal and Torres Start Islander literature; (2) advisory group input and review; (3) systematic identification and testing of MeSH and text word terms; (4) relevance assessment of the search filter's retrieved items; and (5) translation for use in PubMed through the web.

RESULTS

Scoping study analyses demonstrated complexity in the nature and use of possible search terms and publication characteristics. The search filter achieved a recall rate of 84.8% in the full gold standard test set. To determine real-world performance, post-hoc assessment of items retrieved by the search filter in PubMed was undertaken with 87.2% of articles deemed as relevant. The search filter was constructed as a series of URL hyperlinks to enable one-click searching.

CONCLUSION

LIt.search is a search tool that facilitates research into practice for improving outcomes in Aboriginal and Torres Strait Islander health and is publicly available on the Lowitja Institute website. WHAT IS KNOWN ABOUT THIS TOPIC?: Health professionals, researchers and decision makers can find it difficult to retrieve published literature on Aboriginal and Torres Strait Islander health easily, effectively and in a timely way. WHAT DOES THIS PAPER ADD?: This paper describes a new web-based searching tool, LIt.search, which facilitates access to the relevant literature. WHAT ARE THE IMPLICATIONS FOR PRACTICE?: Ready access to published literature on Aboriginal and Torres Strait Islander health reduces a barrier to the use of this evidence in practice. LIt.search encourages the use of this evidence to inform clinical judgement and policy and service decision-making as well as reducing the burdens associated with searching for community practitioners, academics and policy makers.

Increasing the quantity and quality of searching for current best evidence to answer clinical questions: protocol and intervention design of the MacPLUS FS Factorial Randomized Controlled Trials

BACKGROUND & AIMS

Finding current best evidence for clinical decisions remains challenging. With 3,000 new studies published every day, no single evidence-based resource provides all answers or is sufficiently updated. McMaster Premium LiteratUre Service--Federated Search (MacPLUS FS) addresses this issue by looking in multiple high quality resources simultaneously and displaying results in a one-page pyramid with the most clinically useful at the top. Yet, additional logistical and educational barriers need to be addressed to enhance point-of-care evidence retrieval. This trial seeks to test three innovative interventions, among clinicians registered to MacPLUS FS, to increase the quantity and quality of searching for current best evidence to answer clinical questions.

METHODS & DESIGN

In a user-centered approach, we designed three interventions embedded in MacPLUS FS: (A) a web-based Clinical Question Recorder; (B) an Evidence Retrieval Coach composed of eight short educational videos; (C) an Audit, Feedback and Gamification approach to evidence retrieval, based on the allocation of 'badges' and 'reputation scores.' We will conduct a randomized factorial controlled trial among all the 904 eligible medical doctors currently registered to MacPLUS FS at the hospitals affiliated with McMaster University, Canada. Postgraduate trainees (n=429) and clinical faculty/staff (n=475) will be randomized to each of the three following interventions in a factorial design (AxBxC). Utilization will be continuously recorded through clinicians’ accounts that track logins and usage, down to the level of individual keystrokes. The primary outcome is the rate of searches per month per user during the six months of follow-up. Secondary outcomes, measured through the validated Impact Assessment Method questionnaire, include: utility of answers found (meeting clinicians’ information needs), use (application in practice), and perceived usefulness on patient outcomes.

DISCUSSION

Built on effective models for the point-of-care teaching, these interventions approach evidence retrieval as a clinical skill. If effective, they may offer the opportunity to enhance it for a large audience, at low cost, providing better access to relevant evidence across many top EBM resources in parallel.

TRIAL REGISTRATION

ClinicalTrials.Gov NCT02038439.

How to improve your PubMed/MEDLINE searches: 1. background and basic searching

PubMed provides free access via the Internet to more than 23 million records, of which over 19 million are from the MEDLINE database of journal articles. PubMed also provides access to other databases, such as the NCBI Bookshelf. To perform a basic search, you can simply enter the search terms or the concept that you are looking for in the search box. However, taking care to clarify your key concepts may save much time later on, because a non-specific search is likely to produce an overwhelming number of result hits. One way to make your search more specific is to specify which field you want to search using field tags. By default, the results of a search are sorted by the date added to PubMed and displayed in summary format with 20 result hits (records) on each page. In summary format, the title of the article, list of authors, source of information (e.g., journal name followed by date of publication, volume, issue, pages) and the unique PubMed record number called the PubMed identifier (PMID) are shown. Although information is stored about the articles, PubMed/MEDLINE does not store the full text of the papers themselves. However, PubMedCentral (PMC) stores more than 2.8 million articles (roughly 10% of the articles in PubMed) and provides access to them for free to the users.

Comparing data accuracy between structured abstracts and full-text journal articles: implications in their use for informing clinical decisions

BACKGROUND

The abstract is the most frequently read section of a research article. The use of 'Consensus Abstracts', a clinician-oriented web application formatted for mobile devices to search MEDLINE/PubMed, for informing clinical decisions was proposed recently; however, inaccuracies between abstracts and the full-text article have been shown. Efforts have been made to improve quality.

METHODS

We compared data in 60 recent-structured abstracts and full-text articles from six highly read medical journals.

RESULTS

Data inaccuracies were identified and then classified as either clinically significant or not significant. Data inaccuracies were observed in 53.33% of articles ranging from 3.33% to 45% based on the IMRAD format sections. The Results section showed the highest discrepancies (45%) although these were deemed to be mostly not significant clinically except in one. The two most common discrepancies were mismatched numbers or percentages (11.67%) and numerical data or calculations found in structured abstracts but not mentioned in the full text (40%). There was no significant relationship between journals and the presence of discrepancies (Fisher's exact p value =0.3405). Although we found a high percentage of inaccuracy between structured abstracts and full-text articles, these were not significant clinically.

CONCLUSIONS

The inaccuracies do not seem to affect the conclusion and interpretation overall. Structured abstracts appear to be informative and may be useful to practitioners as a resource for guiding clinical decisions.

A study on PubMed search tag usage pattern: association rule mining of a full-day PubMed query log

Background

The practice of evidence-based medicine requires efficient biomedical literature search such as PubMed/MEDLINE. Retrieval performance relies highly on the efficient use of search field tags. The purpose of this study was to analyze PubMed log data in order to understand the usage pattern of search tags by the end user in PubMed/MEDLINE search.

Methods

A PubMed query log file was obtained from the National Library of Medicine containing anonymous user identification, timestamp, and query text. Inconsistent records were removed from the dataset and the search tags were extracted from the query texts. A total of 2,917,159 queries were selected for this study issued by a total of 613,061 users. The analysis of frequent co-occurrences and usage patterns of the search tags was conducted using an association mining algorithm.

Results

The percentage of search tag usage was low (11.38% of the total queries) and only 2.95% of queries contained two or more tags. Three out of four users used no search tag and about two-third of them issued less than four queries. Among the queries containing at least one tagged search term, the average number of search tags was almost half of the number of total search terms. Navigational search tags are more frequently used than informational search tags. While no strong association was observed between informational and navigational tags, six (out of 19) informational tags and six (out of 29) navigational tags showed strong associations in PubMed searches.

Conclusions

The low percentage of search tag usage implies that PubMed/MEDLINE users do not utilize the features of PubMed/MEDLINE widely or they are not aware of such features or solely depend on the high recall focused query translation by the PubMed’s Automatic Term Mapping. The users need further education and interactive search application for effective use of the search tags in order to fulfill their biomedical information needs from PubMed/MEDLINE.

Comparing patient characteristics, type of intervention, control, and outcome (PICO) queries with unguided searching: a randomized controlled crossover trial

BACKGROUND

Translating a question into a query using patient characteristics, type of intervention, control, and outcome (PICO) should help answer therapeutic questions in PubMed searches. The authors performed a randomized crossover trial to determine whether the PICO format was useful for quick searches of PubMed.

METHODS

Twenty-two residents and specialists working at the Radboud University Nijmegen Medical Centre were trained in formulating PICO queries and then presented with a randomized set of questions derived from Cochrane reviews. They were asked to use the best query possible in a five-minute search, using standard and PICO queries. Recall and precision were calculated for both standard and PICO queries.

RESULTS

Twenty-two physicians created 434 queries using both techniques. Average precision was 4.02% for standard queries and 3.44% for PICO queries (difference nonsignificant, t(21) = -0.56, P = 0.58). Average recall was 12.27% for standard queries and 13.62% for PICO queries (difference nonsignificant, t(21) = -0.76, P = 0.46).

CONCLUSIONS

PICO queries do not result in better recall or precision in time-limited searches. Standard queries containing enough detail are sufficient for quick searches.

Medical literature searches: a comparison of PubMed and Google Scholar

BACKGROUND

Medical literature searches provide critical information for clinicians. However, the best strategy for identifying relevant high-quality literature is unknown.

OBJECTIVES

We compared search results using PubMed and Google Scholar on four clinical questions and analysed these results with respect to article relevance and quality.

METHODS

Abstracts from the first 20 citations for each search were classified into three relevance categories. We used the weighted kappa statistic to analyse reviewer agreement and nonparametric rank tests to compare the number of citations for each article and the corresponding journals' impact factors.

RESULTS

Reviewers ranked 67.6% of PubMed articles and 80% of Google Scholar articles as at least possibly relevant (P = 0.116) with high agreement (all kappa P-values < 0.01). Google Scholar articles had a higher median number of citations (34 vs. 1.5, P < 0.0001) and came from higher impact factor journals (5.17 vs. 3.55, P = 0.036).

CONCLUSIONS

PubMed searches and Google Scholar searches often identify different articles. In this study, Google Scholar articles were more likely to be classified as relevant, had higher numbers of citations and were published in higher impact factor journals. The identification of frequently cited articles using Google Scholar for searches probably has value for initial literature searches.

Sensitivity and predictive value of 15 PubMed search strategies to answer clinical questions rated against full systematic reviews

Background

Clinicians perform searches in PubMed daily, but retrieving relevant studies is challenging due to the rapid expansion of medical knowledge. Little is known about the performance of search strategies when they are applied to answer specific clinical questions.

Objective

To compare the performance of 15 PubMed search strategies in retrieving relevant clinical trials on therapeutic interventions.

Methods

We used Cochrane systematic reviews to identify relevant trials for 30 clinical questions. Search terms were extracted from the abstract using a predefined procedure based on the population, interventions, comparison, outcomes (PICO) framework and combined into queries. We tested 15 search strategies that varied in their query (PIC or PICO), use of PubMed’s Clinical Queries therapeutic filters (broad or narrow), search limits, and PubMed links to related articles. We assessed sensitivity (recall) and positive predictive value (precision) of each strategy on the first 2 PubMed pages (40 articles) and on the complete search output.

Results

The performance of the search strategies varied widely according to the clinical question. Unfiltered searches and those using the broad filter of Clinical Queries produced large outputs and retrieved few relevant articles within the first 2 pages, resulting in a median sensitivity of only 10%–25%. In contrast, all searches using the narrow filter performed significantly better, with a median sensitivity of about 50% (all P < .001 compared with unfiltered queries) and positive predictive values of 20%–30% (P < .001 compared with unfiltered queries). This benefit was consistent for most clinical questions. Searches based on related articles retrieved about a third of the relevant studies.

Conclusions

The Clinical Queries narrow filter, along with well-formulated queries based on the PICO framework, provided the greatest aid in retrieving relevant clinical trials within the 2 first PubMed pages. These results can help clinicians apply effective strategies to answer their questions at the point of care.

Text summarization as a decision support aid

Background

PubMed data potentially can provide decision support information, but PubMed was not exclusively designed to be a point-of-care tool. Natural language processing applications that summarize PubMed citations hold promise for extracting decision support information. The objective of this study was to evaluate the efficiency of a text summarization application called Semantic MEDLINE, enhanced with a novel dynamic summarization method, in identifying decision support data.

Methods

We downloaded PubMed citations addressing the prevention and drug treatment of four disease topics. We then processed the citations with Semantic MEDLINE, enhanced with the dynamic summarization method. We also processed the citations with a conventional summarization method, as well as with a baseline procedure. We evaluated the results using clinician-vetted reference standards built from recommendations in a commercial decision support product, DynaMed.

Results

For the drug treatment data, Semantic MEDLINE enhanced with dynamic summarization achieved average recall and precision scores of 0.848 and 0.377, while conventional summarization produced 0.583 average recall and 0.712 average precision, and the baseline method yielded average recall and precision values of 0.252 and 0.277. For the prevention data, Semantic MEDLINE enhanced with dynamic summarization achieved average recall and precision scores of 0.655 and 0.329. The baseline technique resulted in recall and precision scores of 0.269 and 0.247. No conventional Semantic MEDLINE method accommodating summarization for prevention exists.

Conclusion

Semantic MEDLINE with dynamic summarization outperformed conventional summarization in terms of recall, and outperformed the baseline method in both recall and precision. This new approach to text summarization demonstrates potential in identifying decision support data for multiple needs.

Rethinking information delivery: using a natural language processing application for point-of-care data discovery

OBJECTIVE

This paper examines the use of Semantic MEDLINE, a natural language processing application enhanced with a statistical algorithm known as Combo, as a potential decision support tool for clinicians. Semantic MEDLINE summarizes text in PubMed citations, transforming it into compact declarations that are filtered according to a user's information need that can be displayed in a graphic interface. Integration of the Combo algorithm enables Semantic MEDLINE to deliver information salient to many diverse needs.

METHODS

The authors selected three disease topics and crafted PubMed search queries to retrieve citations addressing the prevention of these diseases. They then processed the citations with Semantic MEDLINE, with the Combo algorithm enhancement. To evaluate the results, they constructed a reference standard for each disease topic consisting of preventive interventions recommended by a commercial decision support tool.

RESULTS

Semantic MEDLINE with Combo produced an average recall of 79% in primary and secondary analyses, an average precision of 45%, and a final average F-score of 0.57.

CONCLUSION

This new approach to point-of-care information delivery holds promise as a decision support tool for clinicians. Health sciences libraries could implement such technologies to deliver tailored information to their users.

Performance evaluation of Unified Medical Language System®'s synonyms expansion to query PubMed

BACKGROUND

PubMed is the main access to medical literature on the Internet. In order to enhance the performance of its information retrieval tools, primarily non-indexed citations, the authors propose a method: expanding users' queries using Unified Medical Language System' (UMLS) synonyms i.e. all the terms gathered under one unique Concept Unique Identifier.

METHODS

This method was evaluated using queries constructed to emphasize the differences between this new method and the current PubMed automatic term mapping. Four experts assessed citation relevance.

RESULTS

Using UMLS, we were able to retrieve new citations in 45.5% of queries, which implies a small increase in recall. The new strategy led to a heterogeneous 23.7% mean increase in non-indexed citation retrieved. Of these, 82% have been published less than 4 months earlier. The overall mean precision was 48.4% but differed according to the evaluators, ranging from 36.7% to 88.1% (Inter rater agreement was poor: kappa = 0.34).

CONCLUSIONS

This study highlights the need for specific search tools for each type of user and use-cases. The proposed strategy may be useful to retrieve recent scientific advancement.

Public access and use of health research: an exploratory study of the National Institutes of Health (NIH) Public Access Policy using interviews and surveys of health personnel

BACKGROUND

In 2008, the National Institutes of Health (NIH) Public Access Policy mandated open access for publications resulting from NIH funding (following a 12-month embargo). The large increase in access to research that will take place in the years to come has potential implications for evidence-based practice (EBP) and lifelong learning for health personnel.

OBJECTIVE

This study assesses health personnel's current use of research to establish whether grounds exist for expecting, preparing for, and further measuring the impact of the NIH Public Access Policy on health care quality and outcomes in light of time constraints and existing information resources.

METHODS

In all, 14 interviews and 90 surveys of health personnel were conducted at a community-based clinic and an independent teaching hospital in 2010. Health personnel were asked about the research sources they consulted and the frequency with which they consulted these sources, as well as motivation and search strategies used to locate articles, perceived level of access to research, and knowledge of the NIH Public Access Policy.

RESULTS

In terms of current access to health information, 65% (57/88) of the health personnel reported being satisfied, while 32% (28/88) reported feeling underserved. Among the sources health personnel reported that they relied upon and consulted weekly, 83% (73/88) reported turning to colleagues, 77% (67/87) reported using synthesized information resources (eg, UpToDate and Cochrane Systematic Reviews), while 32% (28/88) reported that they consulted primary research literature. The dominant resources health personnel consulted when actively searching for health information were Google and Wikipedia, while 27% (24/89) reported using PubMed weekly. The most prevalent reason given for accessing research on a weekly basis, reported by 35% (31/88) of survey respondents, was to help a specific patient, while 31% (26/84) were motivated by general interest in research.

CONCLUSIONS

The results provide grounds for expecting the NIH Public Access Policy to have a positive impact on EBP and health care more generally given that between a quarter and a third of participants in this study (1) frequently accessed research literature, (2) expressed an interest in having greater access, and (3) were aware of the policy and expect it to have an impact on their accessing research literature in the future. Results also indicate the value of promoting a greater awareness of the NIH policy, providing training and education in the location and use of the literature, and continuing improvements in the organization of biomedical research for health personnel use.

MeSHy: Mining unanticipated PubMed information using frequencies of occurrences and concurrences of MeSH terms

MOTIVATION

PubMed is the most widely used database of biomedical literature. To the detriment of the user though, the ranking of the documents retrieved for a query is not content-based, and important semantic information in the form of assigned Medical Subject Headings (MeSH) terms is not readily presented or productively utilized. The motivation behind this work was the discovery of unanticipated information through the appropriate ranking of MeSH term pairs and, indirectly, documents. Such information can be useful in guiding novel research and following promising trends.

METHODS

A web-based tool, called MeSHy, was developed implementing a mainly statistical algorithm. The algorithm takes into account the frequencies of occurrences, concurrences, and the semantic similarities of MeSH terms in retrieved PubMed documents to create MeSH term pairs. These are then scored and ranked, focusing on their unexpectedly frequent or infrequent occurrences.

RESULTS

MeSHy presents results through an online interactive interface facilitating further manipulation through filtering and sorting. The results themselves include the MeSH term pairs, along with MeSH categories, the score, and document IDs, all of which are hyperlinked for convenience. To highlight the applicability of the tool, we report the findings of an expert in the pharmacology field on querying the molecularly-targeted drug imatinib and nutrition-related flavonoids. To the best of our knowledge, MeSHy is the first publicly available tool able to directly provide such a different perspective on the complex nature of published work.

IMPLEMENTATION AND AVAILABILITY

Implemented in Perl and served by Apache2 at http://bat.ina.certh.gr/tools/meshy/ with all major browsers supported.

Risk factors for bladder cancer: challenges of conducting a literature search using PubMed

The objective of this study was to assess the risk factors for bladder cancer using PubMed articles from January 2000 to December 2009. The study also aimed to describe the challenges encountered in the methodology of a literature search for bladder cancer risk factors using PubMed. Twenty-six categories of risk factors for bladder cancer were identified using the National Cancer Institute Web site and the Medical Subject Headings (MeSH) Web site. A total of 1,338 PubMed searches were run using the term "urinary bladder cancer" and a risk factor term (e.g., "cigarette smoking") and were screened to identify 260 articles for final analysis. The search strategy had an overall precision of 3.42 percent, relative recall of 12.64 percent, and an F-measure of 5.39 percent. Although search terms derived from MeSH had the highest overall precision and recall, the differences did not reach significance, which indicates that for generalized, free-text searches of the PubMed database, the searchers' own terms are generally as effective as MeSH terms.

Clinician search behaviors may be influenced by search engine design

Background

Searching the Web for documents using information retrieval systems plays an important part in clinicians’ practice of evidence-based medicine. While much research focuses on the design of methods to retrieve documents, there has been little examination of the way different search engine capabilities influence clinician search behaviors.

Objectives

Previous studies have shown that use of task-based search engines allows for faster searches with no loss of decision accuracy compared with resource-based engines. We hypothesized that changes in search behaviors may explain these differences.

Methods

In all, 75 clinicians (44 doctors and 31 clinical nurse consultants) were randomized to use either a resource-based or a task-based version of a clinical information retrieval system to answer questions about 8 clinical scenarios in a controlled setting in a university computer laboratory. Clinicians using the resource-based system could select 1 of 6 resources, such as PubMed; clinicians using the task-based system could select 1 of 6 clinical tasks, such as diagnosis. Clinicians in both systems could reformulate search queries. System logs unobtrusively capturing clinicians’ interactions with the systems were coded and analyzed for clinicians’ search actions and query reformulation strategies.

Results

The most frequent search action of clinicians using the resource-based system was to explore a new resource with the same query, that is, these clinicians exhibited a “breadth-first” search behaviour. Of 1398 search actions, clinicians using the resource-based system conducted 401 (28.7%, 95% confidence interval [CI] 26.37-31.11) in this way. In contrast, the majority of clinicians using the task-based system exhibited a “depth-first” search behavior in which they reformulated query keywords while keeping to the same task profiles. Of 585 search actions conducted by clinicians using the task-based system, 379 (64.8%, 95% CI 60.83-68.55) were conducted in this way.

Conclusions

This study provides evidence that different search engine designs are associated with different user search behaviors.

Open Biomedical Ontology-based Medline exploration

BACKGROUND

Effective Medline database exploration is critical for the understanding of high throughput experimental results and the development of novel hypotheses about the mechanisms underlying the targeted biological processes. While existing solutions enhance Medline exploration through different approaches such as document clustering, network presentations of underlying conceptual relationships and the mapping of search results to MeSH and Gene Ontology trees, we believe the use of multiple ontologies from the Open Biomedical Ontology can greatly help researchers to explore literature from different perspectives as well as to quickly locate the most relevant Medline records for further investigation.

RESULTS

We developed an ontology-based interactive Medline exploration solution called PubOnto to enable the interactive exploration and filtering of search results through the use of multiple ontologies from the OBO foundry. The PubOnto program is a rich internet application based on the FLEX platform. It contains a number of interactive tools, visualization capabilities, an open service architecture, and a customizable user interface. It is freely accessible at: http://brainarray.mbni.med.umich.edu/brainarray/prototype/pubonto.