Study filters for non-randomized studies of interventions consistently lacked sensitivity upon external validation

BACKGROUND

Little evidence is available on searches for non-randomized studies (NRS) in bibliographic databases within the framework of systematic reviews. For instance, it is currently unclear whether, when searching for NRS, effective restriction of the search strategy to certain study types is possible. The following challenges need to be considered: 1) For non-randomized controlled trials (NRCTs): whether they can be identified by established filters for randomized controlled trials (RCTs). 2) For other NRS types (such as cohort studies): whether study filters exist for each study type and, if so, which performance measures they have. The aims of the present analysis were to identify and validate existing NRS filters in MEDLINE as well as to evaluate established RCT filters using a set of MEDLINE citations.

METHODS

Our analysis is a retrospective analysis of study filters based on MEDLINE citations of NRS from Cochrane reviews. In a first step we identified existing NRS filters. For the generation of the reference set, we screened Cochrane reviews evaluating NRS, which covered a broad range of study types. The citations of the studies included in the Cochrane reviews were identified via the reviews' bibliographies and the corresponding PubMed identification numbers (PMIDs) were extracted from PubMed. Random samples comprising up to 200 citations (i.e. 200 PMIDs) each were created for each study type to generate the test sets.

RESULTS

A total of 271 Cochrane reviews from 41 different Cochrane groups were eligible for data extraction. We identified 14 NRS filters published since 2001. The study filters generated between 660,000 and 9.5 million hits in MEDLINE. Most filters covered several study types. The reference set included 2890 publications classified as NRS for the generation of the test sets. Twelve test sets were generated (one for each study type), of which 8 included 200 citations each. None of the study filters achieved sufficient sensitivity (≥ 92%) for all of the study types targeted.

CONCLUSIONS

The performance of current NRS filters is insufficient for effective use in daily practice. It is therefore necessary to develop new strategies (e.g. new NRS filters in combination with other search techniques). The challenges related to NRS should be taken into account.

How equity is addressed in clinical practice guidelines: a content analysis

OBJECTIVES

Considering equity into guidelines presents methodological challenges. This study aims to qualitatively synthesise the methods for incorporating equity in clinical practice guidelines (CPGs).

SETTING

Content analysis of methodological publications.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Methodological publications were included if they provided checklists/frameworks on when, how and to what extent equity should be incorporated in CPGs.

DATA SOURCES

We electronically searched MEDLINE, retrieved references, and browsed guideline development organisation websites from inception to January 2013. After study selection by two authors, general characteristics and checklists items/framework components from included studies were extracted. Based on the questions or items from checklists/frameworks (unit of analysis), content analysis was conducted to identify themes and questions/items were grouped into these themes.

PRIMARY OUTCOMES

The primary outcomes were methodological themes and processes on how to address equity issues in guideline development.

RESULTS

8 studies with 10 publications were included from 3405 citations. In total, a list of 87 questions/items was generated from 17 checklists/frameworks. After content analysis, questions were grouped into eight themes ('scoping questions', 'searching relevant evidence', 'appraising evidence and recommendations', 'formulating recommendations', 'monitoring implementation', 'providing a flow chart to include equity in CPGs', and 'others: reporting of guidelines and comments from stakeholders' for CPG developers and 'assessing the quality of CPGs' for CPG users). Four included studies covered more than five of these themes. We also summarised the process of guideline development based on the themes mentioned above.

CONCLUSIONS

For disadvantaged population-specific CPGs, eight important methodological issues identified in this review should be considered when including equity in CPGs under the guidance of a scientific guideline development manual.

Failure or success of search strategies to identify adverse effects of medical devices: a feasibility study using a systematic review

BACKGROUND

Research has indicated that adverse effects terms are increasingly prevalent in the title, abstract or indexing terms of articles that contain adverse drug effects data in MEDLINE and Embase. However, it is unknown whether adverse effects terms are present in the database records of articles that contain adverse effects data of medical devices, and thus, to what extent the development of an adverse effects search filter for medical devices may be feasible.

METHODS

A case study systematic review of a medical device was selected. The included studies from a systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion were used in the analysis. For each included study, the corresponding database record on MEDLINE and Embase was assessed to measure the presence or absence of adverse effects terms in the title, abstract or indexing. The performance of each potential adverse effects search term was also measured and compared.

RESULTS

There were 82 publications (49 studies) included in the systematic review with 51 of these indexed on MEDLINE and 55 on Embase. Ninety-four percent (48/51) of the records on MEDLINE and 95% (52/55) of the records on Embase contained at least one adverse effects related search term. The wide variety of adverse effects terms included in the title, abstract or indexing of bibliographic records, and the lack of any individual high-performing search terms suggests that a combination of terms in different fields is required to identify adverse effects of medical devices. In addition, the most successful search terms differed from the most successful terms for identifying adverse drug effects.

CONCLUSIONS

The search filters currently available for adverse drug effects are not necessarily useful for searching adverse effects data of medical devices. The presence of adverse effects terms in the bibliographic records of articles on medical devices, however, indicates that combinations of adverse effects search terms may be useful in search strategies in MEDLINE and Embase. The results, therefore, suggest that not only a search filter for the adverse effects of medical devices is feasible, but also that it should be a research priority.

Filtering Medline for a clinical discipline: diagnostic test assessment framework

OBJECTIVE

To develop and test a Medline filter that allows clinicians to search for articles within a clinical discipline, rather than searching the entire Medline database.

DESIGN

Diagnostic test assessment framework with development and validation phases.

SETTING

Sample of 4657 articles published in 2006 from 40 journals. Reviews Each article was manually reviewed, and 19.8% contained information relevant to the discipline of nephrology. The performance of 1 155 087 unique renal filters was compared with the manual review.

MAIN OUTCOME MEASURES

Sensitivity, specificity, precision, and accuracy of each filter.

RESULTS

The best renal filters combined two to 14 terms or phrases and included the terms "kidney" with multiple endings (that is, truncation), "renal replacement therapy", "renal dialysis", "kidney function tests", "renal", "nephr" truncated, "glomerul" truncated, and "proteinuria". These filters achieved peak sensitivities of 97.8% and specificities of 98.5%. Performance of filters remained excellent in the validation phase.

CONCLUSIONS

Medline can be filtered for the discipline of nephrology in a reliable manner. Storing these high performance renal filters in PubMed could help clinicians with their everyday searching. Filters can also be developed for other clinical disciplines by using similar methods.

[Hand searching for controlled clinical trials in German surgical journals. A contribution to evidence-based surgery]

BACKGROUND

The identification of all available and relevant study reports is mandatory for a comprehensive assessment of randomised (RCTs) and controlled clinical trials (CCTs) in systematic reviews. Incomplete compilation of health care journals in electronic databases and incorrect indexing of the studies impair the result of a systematic literature search. An additional search in medical journals that are not listed in electronic databases can obtain higher search precision.

METHODS

In the hand searching project of the Cochrane Collaboration, in Germany 14 surgical journals were searched manually for RCTs and CCTs. The identified study reports were compared with Medline records, and the publication frequency was analysed. The study reports were published in the Cochrane Library and are henceforth available for inclusion in systematic reviews.

RESULTS

Four hundred (77%) of the 519 published volumes in the 14 surgical journals were searched for RCTs and CCTs. Of the 1152 controlled trials (670 RCTs and 482 CCTs) identified, 674 (58%) were not included in Medline.

CONCLUSIONS

The gap between the number of hand search results and the number of Medline indexed RCTs and CCTs is also reflected in other special medical fields. To ensure completeness of the literature compilation, the hand searching project should be continued.

Web-based citation management compared to EndNote: options for medical sciences

The authors of this article analyzed the differences in output when searching MEDLINE direct and MEDLINE via citation management software, EndNote X1, EndNote Web, and RefWorks. Several searches were performed on Ovid MEDLINE and PubMed directly. These searches were compared against the same searches conducted in Ovid MEDLINE and PubMed using the search features in EndNote X1, EndNote Web, and RefWorks. Findings indicated that for in-depth research users, should search the databases directly rather than through the citation management software interface. The search results indicated it would be appropriate to search databases via citation management software for citation verification tasks and for cursory keyword searching.

Relemed: sentence-level search engine with relevance score for the MEDLINE database of biomedical articles

BACKGROUND

Receiving extraneous articles in response to a query submitted to MEDLINE/PubMed is common. When submitting a multi-word query (which is the majority of queries submitted), the presence of all query words within each article may be a necessary condition for retrieving relevant articles, but not sufficient. Ideally a relationship between the query words in the article is also required. We propose that if two words occur within an article, the probability that a relation between them is explained is higher when the words occur within adjacent sentences versus remote sentences. Therefore, sentence-level concurrence can be used as a surrogate for existence of the relationship between the words. In order to avoid the irrelevant articles, one solution would be to increase the search specificity. Another solution is to estimate a relevance score to sort the retrieved articles. However among the >30 retrieval services available for MEDLINE, only a few estimate a relevance score, and none detects and incorporates the relation between the query words as part of the relevance score.

RESULTS

We have developed "Relemed", a search engine for MEDLINE. Relemed increases specificity and precision of retrieval by searching for query words within sentences rather than the whole article. It uses sentence-level concurrence as a statistical surrogate for the existence of relationship between the words. It also estimates a relevance score and sorts the results on this basis, thus shifting irrelevant articles lower down the list. In two case studies, we demonstrate that the most relevant articles appear at the top of the Relemed results, while this is not necessarily the case with a PubMed search. We have also shown that a Relemed search includes not only all the articles retrieved by PubMed, but potentially additional relevant articles, due to the extended 'automatic term mapping' and text-word searching features implemented in Relemed.

CONCLUSION

By using sentence-level matching, Relemed can deliver higher specificity, thus eliminating more false-positive articles. By introducing an appropriate relevance metric, the most relevant articles on which the user wishes to focus are listed first. Relemed also shrinks the displayed text, and hence the time spent scanning the articles.

Description of controlled trials published in Methods and Findings, 1979-2002

It is thought that the controlled trial (CT) is the most adequate research method to assess a therapeutic intervention in terms of efficacy, and it also constitutes the basis for the development of systematic reviews on health interventions. To identify and obtain the majority of published CTs is not an easy task, mainly because of limitations concerning the currently available electronic sources. The aim of the present work was to identify, describe, and assess the quality of CTs published in the journal Methods and Findings in Experimental and Clinical Pharmacology (M&F). Additionally, to assess the retrievability of both methods, a search was performed in Medline (PubMed access) through the use of an optimal search strategy for CTs. A total of 189 original studies out of a total of 2796 reviewed articles met the CT criteria according to the Jadad scale score, we could hold that only 58% of the CTs were of good quality. The present work confirms, once again, the limitations of a CT search performed exclusively through Medline (sensitivity 64% and specificity 98%). In conclusion, we suggest that the journal M&F explicitly joins the International CONSORT Statement.

Identifying observational studies of surgical interventions in MEDLINE and EMBASE

BACKGROUND

Health technology assessments of surgical interventions frequently require the inclusion of non-randomised evidence. Literature search strategies employed to identify this evidence often exclude a methodological component because of uncertainty surrounding the use of appropriate search terms. This can result in the retrieval of a large number of irrelevant records. Methodological filters would help to minimise this, making literature searching more efficient.

METHODS

An objective approach was employed to develop MEDLINE and EMBASE filters, using a reference standard derived from screening the results of an electronic literature search that contained only subject-related terms. Candidate terms for MEDLINE (N = 37) and EMBASE (N = 35) were derived from examination of the records of the reference standard. The filters were validated on two sets of studies that had been included in previous health technology assessments.

RESULTS

The final filters were highly sensitive (MEDLINE 99.5%, EMBASE 100%, MEDLINE/EMBASE combined 100%) with precision ranging between 16.7%-21.1%, specificity 35.3%-43.5%, and a reduction in retrievals of over 30%. Against the validation standards, the individual filters retrieved 85.2%-100% of records. In combination, however, the MEDLINE and EMBASE filters retrieved 100% against both validation standards with a reduction in retrieved records of 28.4% and 30.1%

CONCLUSION

The MEDLINE and EMBASE filters were highly sensitive and substantially reduced the number of records retrieved, indicating that they are useful tools for efficient literature searching.

Today's Medical Knowledge. Evolution of Data Exchange between Tradition and Globalization

The authors historically review bibliographic research concepts and define globalization in time. Moreover, recent free online data exchange is important in terms of medical progress, scientific updating and patient-care improving. In the author's opinion, data globalization is favoring medical knowledge flow even more. The concept of a traditional library has radically changed over time, gradually missing their pivotal role in research activity. To date, the birth of the Internet and its sudden development has given a great boost to the spread of worldwide information, quickly and cheaply. Nevertheless, besides the advantages, the Internet also hides misleading risks. In this paper, the professional Medline source is compared to common Internet sources. The authors state that Internet sources have a great importance in the spread of medical knowledge. They conclude, however, that the risk of too much available information could lead to a decrease in quality.

Can electronic search engines optimize screening of search results in systematic reviews: an empirical study

BACKGROUND

Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG and Ovid. Our objective is to test the ability of an Ultraseek search engine to rank MEDLINE records of the included studies of Cochrane reviews within the top half of all the records retrieved by the Boolean MEDLINE search used by the reviewers.

METHODS

Collections were created using the MEDLINE bibliographic records of included and excluded studies listed in the review and all records retrieved by the MEDLINE search. Records were converted to individual HTML files. Collections of records were indexed and searched through a statistical search engine, Ultraseek, using review-specific search terms. Our data sources, systematic reviews published in the Cochrane library, were included if they reported using at least one phase of the Cochrane Highly Sensitive Search Strategy (HSSS), provided citations for both included and excluded studies and conducted a meta-analysis using a binary outcome measure. Reviews were selected if they yielded between 1000-6000 records when the MEDLINE search strategy was replicated.

RESULTS

Nine Cochrane reviews were included. Included studies within the Cochrane reviews were found within the first 500 retrieved studies more often than would be expected by chance. Across all reviews, recall of included studies into the top 500 was 0.70. There was no statistically significant difference in ranking when comparing included studies with just the subset of excluded studies listed as excluded in the published review.

CONCLUSION

The relevance ranking provided by the search engine was better than expected by chance and shows promise for the preliminary evaluation of large results from Boolean searches. A statistical search engine does not appear to be able to make fine discriminations concerning the relevance of bibliographic records that have been pre-screened by systematic reviewers.

Handsearching the EMHJ for reports of randomized controlled trials by U.K. Cochrane Centre (Bahrain)

This study used handsearching to find reports of randomized controlled trials in the Eastern Mediterranean Health Journal (EMHJ). EMBASE and MEDLINE were also searched electronically to identify if the reports found by the handsearch were already included in either of these databases. Nine reports were identified: 7 randomized controlled trials and 2 controlled clinical trials. The added value of the handsearch over EMBASE was 6 additional reports and over MEDLINE was 4. Reports identified were sent to the UK Cochrane Centre for verification and publication in The Cochrane Central Register of Controlled Trials (CENTRAL).

Death, dying and informatics: misrepresenting religion on MedLine

Background

The globalization of medical science carries for doctors worldwide a correlative duty to deepen their understanding of patients' cultural contexts and religious backgrounds, in order to satisfy each as a unique individual. To become better informed, practitioners may turn to MedLine, but it is unclear whether the information found there is an accurate representation of culture and religion. To test MedLine's representation of this field, we chose the topic of death and dying in the three major monotheistic religions.

Methods

We searched MedLine using PubMed in order to retrieve and thematically analyze full-length scholarly journal papers or case reports dealing with religious traditions and end-of-life care. Our search consisted of a string of words that included the most common denominations of the three religions, the standard heading terms used by the National Reference Center for Bioethics Literature (NRCBL), and the Medical Subject Headings (MeSH) used by the National Library of Medicine. Eligible articles were limited to English-language papers with an abstract.

Results

We found that while a bibliographic search in MedLine on this topic produced instant results and some valuable literature, the aggregate reflected a selection bias. American writers were over-represented given the global prevalence of these religious traditions. Denominationally affiliated authors predominated in representing the Christian traditions. The Islamic tradition was under-represented.

Conclusion

MedLine's capability to identify the most current, reliable and accurate information about purely scientific topics should not be assumed to be the same case when considering the interface of religion, culture and end-of-life care.

EMBASE versus MEDLINE for family medicine searches: can MEDLINE searches find the forest or a tree?

OBJECTIVE

Many physicians access electronic databases to obtain up-to-date and reliable medical information. In North America, physicians typically use MEDLINE as their sole electronic database whereas in Europe, physicians typically use EMBASE. While MEDLINE and EMBASE are similar, their coverage of the published literature differs. Searching a single literature database (eg, MEDLINE or EMBASE) has been shown not to yield all available citations, and using two or more databases yields a greater percentage of these available citations. This difference has been demonstrated in a variety of disciplines and in family medicine using the term "family medicine," but differences have not been shown using specific diagnostic terms common in family medicine. We sought to determine whether searching EMBASE with terms for common family medicine diagnoses yields additional references beyond those found by using MEDLINE alone.

DESIGN

Literature search comparison.

SETTING

An academic medical centre in the United States.

INTERVENTIONS

Fifteen family medicine topics were selected based on common diagnoses in US primary care health visits as described in a National Health Care Survey on Ambulatory Care Visits. To promote relevance to family medicine physicians and researchers, the qualifiers "family medicine" and "therapy/therapeutics" were added. These topics were searched in EMBASE and MEDLINE. Searches were executed using Ovid search engine and were limited to the years 1992 to 2003, the English language, and human subjects. Total, duplicated, and unique (ie, nonduplicated) citations were recorded for each search in each database.

MAIN OUTCOME MEASURES

Number of citations for the 15 topics.

RESULTS

EMBASE yielded 2246 (65%) of 3445 total citations, whereas MEDLINE yielded 1199 citations. Of the total citations, only 177 articles were cited in both databases. EMBASE had 2092 unique citations to MEDLINE's 999 unique citations. EMBASE consistently found more unique citations in 14 of the 15 searches (P = .0005).

CONCLUSION

Overall, EMBASE provides twice as many citations per search as MEDLINE and provides greater coverage of total retrieved citations. More citations do not necessarily mean higher-quality citations. In a comprehensive search specific to family medicine, combined EMBASE and MEDLINE searches could yield more articles than MEDLINE could alone.

Rédaction des adresses sur les publications

OBJECTIVE

To assess, through the example of the université Claude Bernard Lyon 1, whether the poor international visibility of French universities may be due in part to a lack of care by researchers in listing their affiliations and address on their scientific publications.

METHOD

Querying the heading "address" of Medline, Embase.com and Pascal Biomed for 2003 in searches for publications from Université Claude Bernard Lyon 1.

RESULTS

In 300 bibliographic entries studied, we found 44 different forms of the university's name. Only 15% of the addresses were correct, and 44% did not mention the author's affiliation with the university.

DISCUSSION

Authors' lack of care in listing their affiliations and address and the absence of a standardized official name for the university may explain the poor visibility of French universities. Recommendations about this topic might help to promote the reputation and work of French researchers internationally.

askMEDLINE: a free-text, natural language query tool for MEDLINE/PubMed

Background

Plain language search tools for MEDLINE/PubMed are few. We wanted to develop a search tool that would allow anyone using a free-text, natural language query and without knowing specialized vocabularies that an expert searcher might use, to find relevant citations in MEDLINE/PubMed. This tool would translate a question into an efficient search.

Results

The accuracy and relevance of retrieved citations were compared to references cited in BMJ POEMs and CATs (critically appraised topics) questions from the University of Michigan Department of Pediatrics. askMEDLINE correctly matched the cited references 75.8% in POEMs and 89.2 % in CATs questions on first pass. When articles that were deemed to be relevant to the clinical questions were included, the overall efficiency in retrieving journal articles was 96.8% (POEMs) and 96.3% (CATs.)

Conclusion

askMEDLINE might be a useful search tool for clinicians, researchers, and other information seekers interested in finding current evidence in MEDLINE/PubMed. The text-only format could be convenient for users with wireless handheld devices and those with low-bandwidth connections in remote locations.

Optimal search strategies for retrieving systematic reviews from Medline: analytical survey

OBJECTIVE

To develop optimal search strategies in Medline for retrieving systematic reviews.

DESIGN

Analytical survey.

DATA SOURCES

161 journals published in 2000 indexed in Medline.

MAIN OUTCOME MEASURES

The sensitivity, specificity, and precision of retrieval of systematic reviews of 4862 unique terms in 782,485 combinations of one to five terms were determined by comparison with a hand search of all articles (the criterion standard) in 161 journals published during 2000 (49,028 articles).

RESULTS

Only 753 (1.5%) of the 49,028 articles were systematic reviews. The most sensitive strategy included five terms and had a sensitivity of 99.9% (95% confidence interval 99.6% to 100%) and a specificity of 52% (51.6% to 52.5%). The strategy that best minimised the difference between sensitivity and specificity had a sensitivity of 98% (97% to 99%) and specificity of 90.8% (90.5% to 91.1%). Highest precision for multiterm strategies, 57% (54% to 60%), was achieved at a sensitivity of 71% (68% to 74%). The term "cochrane database of systematic reviews.jn." was the most precise single term search strategy (sensitivity of 56% (52% to 60%) and precision of 96% (94% to 98%)). These strategies are available through the "limit" screen of Ovid's search interface for Medline.

CONCLUSIONS

Systematic reviews can be retrieved from Medline with close to perfect sensitivity or specificity, or with high precision, by using empirical search strategies.

An evolution of experts: MEDLINE in the library school

QUESTION

What is the real value that medical librarians bring to the health care environment? How have library science educators, frequently former practitioners themselves, responded to the challenge of expert searching?

METHODS

In this paper, I give an historical introduction to the landscape of medical information retrieval through the development of MEDLINE. I then look at the evolution of training in online searching and its place in the context of library school education and particularly the effect of generalist education on future specialists. Finally, I acknowledge the new role of the informationist as another assertion of our professional expertise.

CONCLUSIONS

The three interdependent subsystems of our professional machine-our schools, our association, and our professional peers- must all respond to this challenge by asserting our expertise in our curricula, in our continuing education, and in our dialogues with each other. Only by acknowledging the interaction of these subsystems will real and positive changes be effected to benefit our profession and our constituencies.

Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly

Using EndNote version 7.0, the authors tested the search capabilities of the EndNote search engine for retrieving citations from MEDLINE for importation into EndNote, a citation management software package. Ovid MEDLINE and PubMed were selected for the comparison. Several searches were performed on Ovid MEDLINE and PubMed using EndNote as the search engine, and the same searches were run on both Ovid and PubMed directly. Findings indicate that it is preferable to search MEDLINE directly rather than using EndNote. The publishers of EndNote do warn its users about the limitations of their product as a search engine when searching external databases. In this article, the limitations of EndNote as a search engine for searching MEDLINE were explored as related to MeSH, non-MeSH, citation verification, and author searching.

Comparison of Medical Subject Headings and text-word searches in MEDLINE to retrieve studies on sleep in healthy individuals

OBJECTIVE

The objective was to investigate the performance of two search strategies in the retrieval of primary research papers containing descriptive information on the sleep of healthy people from MEDLINE.

METHODOLOGY

Two search strategies-one based on the use of only Medical Subject Headings (MeSH), the second based on text-word searching-were evaluated as to their specificity and sensitivity in retrieving a set of relevant research papers published in the journal Sleep from 1996 to 2001 that were preselected by a hand search.

RESULTS

The subject search provided higher specificity than the text-word search (66% and 47%, respectively) but lower sensitivity (78% for the subject search versus 88% for the text-word search). Each search strategy gave some unique relevant hits.

CONCLUSIONS

The two search strategies complemented each other and should be used together for maximal retrieval. No combination of MeSH terms could provide comprehensive yet reasonably precise retrieval of relevant articles. The text-word searching had sensitivity and specificity comparable to the subject search. In addition, use of text words "normal," "healthy," and "control" in the title or abstract fields to limit the final sets provided an efficient way to increase the specificity of both search strategies.

TXTGate: profiling gene groups with text-based information

We implemented a framework called TXTGate that combines literature indices of selected public biological resources in a flexible text-mining system designed towards the analysis of groups of genes. By means of tailored vocabularies, term- as well as gene-centric views are offered on selected textual fields and MEDLINE abstracts used in LocusLink and the Saccharomyces Genome Database. Subclustering and links to external resources allow for in-depth analysis of the resulting term profiles.

Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey

OBJECTIVE

To develop optimal search strategies in Medline for retrieving sound clinical studies on the diagnosis of health disorders.

DESIGN

Analytical survey.

SETTING

Medline, 2000.

PARTICIPANTS

170 journals for 2000 of which 161 were indexed in Medline.

MAIN OUTCOME MEASURES

The sensitivity, specificity, precision ("positive predictive value"), and accuracy of 4862 unique terms in 17,287 combinations were determined by comparison with a hand search of all articles (the "gold standard") in 161 journals published during 2000 (49,028 articles).

RESULTS

Only 147 (18.9%) of 778 articles about diagnostic tests met basic criteria for scientific merit. Combinations of search terms reached peak sensitivities of 98.6% at a specificity of 74.3%. Compared with best single terms, best multiple terms increased sensitivity for sound studies by 6.8% (absolute increase), while also increasing specificity (absolute increase 6.0%) when sensitivity was maximised. When terms were combined to maximise specificity, the single term, specificity.tw. (98.4%), outperformed combinations of terms. The strategies newly reported in this paper outperformed other validated search strategies except for one strategy that had slightly higher sensitivity (99.3% v 98.6%) but lower specificity (54.7% v 74.3%).

CONCLUSION

New empirical search strategies in Medline can optimise retrieval of articles reporting high quality clinical studies of diagnosis.

Analysis of current nuclear cardiology literature in MEDLINE database: a study of gated SPECT imaging using PubMed

BACKGROUND

Easy access to relevant clinical information is necessary for physicians to make the best decisions for patient management. The increasing amount of information available has made it difficult for physicians to retrieve up-to-date information efficiently. We sought to determine the accessibility and accuracy of indexing in the nuclear cardiology literature by conducting sample searches in the MEDLINE database on the topic of gated single photon emission computed tomography (SPECT) imaging.

METHODS AND RESULTS

The MEDLINE database was initially searched by use of both a primary and a comprehensive search strategy on PubMed for publications in English from 1994 to 2000. A total of 260 papers were retrieved from the primary search and 306 additional papers from the comprehensive search. Only 204 of the 566 citations from the combined electronic searches were truly relevant to gated SPECT. The resulting specificity index (precision) was 36%. A hand search was conducted in 11 top journals from 1994 to 2000. It yielded 81 additional citations that were missed by the PubMed search. The sensitivity index (recall) was calculated for all 11 journals. The Journal of Nuclear Cardiology had the highest rate of publication but the lowest rate of recall (44%). The clinical nuclear cardiology terminology and classifications were compared with the available Medical Subject Headings (MeSH) and MeSH Trees used for indexing in MEDLINE.

CONCLUSIONS

There are 6 nuclear cardiology techniques, including gated SPECT and myocardial perfusion imaging, that are not specifically indexed in the current MEDLINE database. The lack of specific MeSH headings and indexing structure results in low recall and precision for retrieval of nuclear cardiology literature. We recommend 2 additions to the MeSH Tree Structure and 6 new MeSH headings.

Beyond MEDLINE for literature searches

PURPOSE

To describe strategies for a comprehensive literature search.

ORGANIZING CONSTRUCT

MEDLINE searches result in limited numbers of studies that are often biased toward statistically significant findings. Diversified search strategies are needed.

METHODS

Empirical evidence about the recall and precision of diverse search strategies is presented. Challenges and strengths of each search strategy are identified.

FINDINGS

Search strategies vary in recall and precision. Often sensitivity and specificity are inversely related. Valuable search strategies include examination of multiple diverse computerized databases, ancestry searches, citation index searches, examination of research registries, journal hand searching, contact with the "invisible college," examination of abstracts, Internet searches, and contact with sources of synthesized information.

CONCLUSIONS

Extending searches beyond MEDLINE enables researchers to conduct more systematic comprehensive searches.

Searching biomedical databases on complementary medicine: the use of controlled vocabulary among authors, indexers and investigators

BACKGROUND

The optimal retrieval of a literature search in biomedicine depends on the appropriate use of Medical Subject Headings (MeSH), descriptors and keywords among authors and indexers. We hypothesized that authors, investigators and indexers in four biomedical databases are not consistent in their use of terminology in Complementary and Alternative Medicine (CAM).

METHODS

Based on a research question addressing the validity of spinal palpation for the diagnosis of neuromuscular dysfunction, we developed four search concepts with their respective controlled vocabulary and key terms. We calculated the frequency of MeSH, descriptors, and keywords used by authors in titles and abstracts in comparison to standard practices in semantic and analytic indexing in MEDLINE, MANTIS, CINAHL, and Web of Science.

RESULTS

Multiple searches resulted in the final selection of 38 relevant studies that were indexed at least in one of the four selected databases. Of the four search concepts, validity showed the greatest inconsistency in terminology among authors, indexers and investigators. The use of spinal terms showed the greatest consistency. Of the 22 neuromuscular dysfunction terms provided by the investigators, 11 were not contained in the controlled vocabulary and six were never used by authors or indexers. Most authors did not seem familiar with the controlled vocabulary for validity in the area of neuromuscular dysfunction. Recently, standard glossaries have been developed to assist in the research development of manual medicine.

CONCLUSIONS

Searching biomedical databases for CAM is challenging due to inconsistent use of controlled vocabulary and indexing procedures in different databases. A standard terminology should be used by investigators in conducting their search strategies and authors when writing titles, abstracts and submitting keywords for publications.

[Systematic reviews in practice. X. Searching, selecting and the methodological assessment of diagnostic evaluation research]

Literature concerning the value of diagnostic tests is poorly indexed in electronic bibliographical databases. An extensive, sensitive and specific search strategy for evaluation of a diagnostic test in Medline is a combination of key words or text words concerning the diagnostic test with: 'sensitivity and specificity' (exploded), 'mass screening' (exploded), 'reference values', 'false positive reactions', 'false negative reactions', 'specificit$.tw', 'screening.tw', 'false positive$.tw', 'false negative$.tw', 'accuracy.tw', 'predictive value$.tw', 'reference value$.tw', 'roc$.tw' or 'likelihood ratio$.tw'. Inclusion and exclusion criteria select publications which meet certain minimum requirements, such as the presence of an accepted reference standard, a clear definition of 'diseased', and the presence of a suitable outcome measure. The methodological assessment list should contain a number of validity criteria as well as a number of criteria by which to measure the extent to which the study results can be generalised. Important assessment criteria that may affect the diagnostic value of the test are: study design, blinding, use of different reference standards, the absence of details about the diagnostic criteria for the diagnostic test and insufficient information about the study population.

Head and neck oncology: the UK experience. Who is publishing what?

Using the MEDLINE database (OVID), a retrospective review of the UK literature on head and neck oncology was performed for the period 1994-2000, each publication being categorised by department and first author. Tumours almost exclusively managed by general surgeons, cardiothoracic surgeons and neurosurgeons were excluded. In the years 1994 to 2000, there were a total of 120 UK publications, 72% of which came from non-academic NHS units; 23% of the publications were from ENT units, 23% from oral and maxillofacial (OMF) surgery units and 18% from plastic surgery units. The majority of plastic surgery publications described reconstructive techniques, whereas a wider range of topics was observed in the publications by ENT and OMP surgeons. Several irregularities in the MEDLINE database were discovered and are discussed. The findings of this study may be relevant to the future planning of head and neck oncology services.

Searching literature databases for health care economic evaluations: how systematic can we afford to be?

BACKGROUND

As the health care economic-evaluation literature continues to grow, a need for sound methods to conduct systematic reviews of the existing evidence is emerging. So far, reviews of economic evaluations have relied upon noncomprehensive sources and have adopted simplistic search methods, both likely to lead to biased results.

OBJECTIVE

To provide evidence of the performance of alternative approaches for identifying published health care economic evaluations and to illustrate what forms of bias may affect systematic reviews of such studies.

METHODS

The sensitivity and specificity of alternative search strategies were tested for the period January to March 1997, using seven major medical and social science literature databases, one economic evaluation database and a published bibliography. Studies were selected blindly by pairs of reviewers (agreement 94.1%-96.5%), using a two-stage procedure.

RESULTS

By limiting the scope of a review to Medline and by using appropriate search strategies, researchers may significantly reduce the number of nonrelevant references retrieved by their electronic searches (sensitivity 72%, specificity 75%, compared with more extensive strategies), which require exclusion by manual screening. The yield of searches based on specialized bibliographies and databases may be different because of variations in selection criteria, coverage and time lag for inclusion of references.

CONCLUSIONS

Medline is the key source for reviews of economic evaluations. Researchers may select from the search strategies proposed in this paper the one that offers an optimal balance between sensitivity and specificity in relation to the aims of their review. Manual searches and searches of databases other than Medline have a limited incremental yield. The sensitivity of all search strategies increases when tighter methodological standards are set, but more research is needed on methods for identifying methodologically sound studies.

Variations in Medical Subject Headings (MeSH) mapping: from the natural language of patron terms to the controlled vocabulary of mapped lists

OBJECTIVES

This study compared the mapping of natural language patron terms to the Medical Subject Headings (MeSH) across six MeSH interfaces for the MEDLINE database.

METHODS

Test data were obtained from search requests submitted by patrons to the Library of the Health Sciences, University of Illinois at Chicago, over a nine-month period. Search request statements were parsed into separate terms or phrases. Using print sources from the National Library of Medicine, Each parsed patron term was assigned corresponding MeSH terms. Each patron term was entered into each of the selected interfaces to determine how effectively they mapped to MeSH. Data were collected for mapping success, accessibility of MeSH term within mapped list, and total number of MeSH choices within each list.

RESULTS

The selected MEDLINE interfaces do not map the same patron term in the same way, nor do they consistently lead to what is considered the appropriate MeSH term.

CONCLUSIONS

If searchers utilize the MEDLINE database to its fullest potential by mapping to MeSH, the results of the mapping will vary between interfaces. This variance may ultimately impact the search results. These differences should be considered when choosing a MEDLINE interface and when instructing end users.

Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials

BACKGROUND

We would expect information on adverse drug reactions in randomised clinical trials to be easily retrievable from specific searches of electronic databases. However, complete retrieval of such information may not be straightforward, for two reasons. First, not all clinical drug trials provide data on the frequency of adverse effects. Secondly, not all electronic records of trials include terms in the abstract or indexing fields that enable us to select those with adverse effects data. We have determined how often automated search methods, using indexing terms and/or textwords in the title or abstract, would fail to retrieve trials with adverse effects data.

METHODS

We used a sample set of 107 trials known to report frequencies of adverse drug effects, and measured the proportion that (i) were not assigned the appropriate adverse effects indexing terms in the electronic databases, and (ii) did not contain identifiable adverse effects textwords in the title or abstract.

RESULTS

Of the 81 trials with records on both MEDLINE and EMBASE, 25 were not indexed for adverse effects in either database. Twenty-six trials were indexed in one database but not the other. Only 66 of the 107 trials reporting adverse effects data mentioned this in the abstract or title of the paper. Simultaneous use of textword and indexing terms retrieved only 82/107 (77%) papers.

CONCLUSIONS

Specific search strategies based on adverse effects textwords and indexing terms will fail to identify nearly a quarter of trials that report on the rate of drug adverse effects.

A comparison of the coverage of clinical medicine provided by PASCAL BIOMED and MEDLINE

The bibliographic databases PASCAL BIOMED and MEDLINE, available both at SilverPlatter, were compared for their coverage of clinical medicine. The main objective of the study was to identify the information in PASCAL BIOMED that is supplementary to MEDLINE. In PASCAL BIOMED and MEDLINE 10 searches were performed, all limited to publication year 1996. Search profiles in PASCAL BIOMED were free text only, while in MEDLINE they were mainly composed of MeSH headings, with additional free text. Search results were analysed for numbers of (relevant) references retrieved, unique and duplicate references, precision and uniqueness ratio. Also the presence of abstracts in both databases, and the abstract language and the presence of English descriptors and identifiers in PASCAL BIOMED were studied. MEDLINE contained more relevant information than PASCAL BIOMED on nearly all topics studied. The precision of the PASCAL BIOMED searches varied from 29 to 87% (median = 68%) and of the MEDLINE searches from 20 to 91% (median = 66%). The uniqueness ratio in PASCAL BIOMED varied from 0 to 33% (median = 20%) and in MEDLINE from 28 to 83% (median 37%). Considering the numbers of relevant and unique references retrieved, MEDLINE performed better than PASCAL BIOMED. Yet a sizeable proportion of the relevant references were unique to PASCAL BIOMED. Probably PASCAL BIOMED would have performed better if lists of descriptors had been accessible in the database, preferably with support to trace them.

The accuracy of MEDLINE and Journal contents pages for papers published in Clinical Otolaryngology

MEDLINE is widely used as a source for identifying and reviewing medical journal literature. Its accuracy is generally taken for granted, as is that of the contents pages published by the journals themselves. In this study of citation accuracy we examined the articles published in Clinical Otolaryngology and Allied Sciences from 1976 to 1998. The entries in MEDLINE were compared with the entries in the Journal's contents pages, and with the actual articles. Of 1651 articles published in the journal, one was omitted from MEDLINE and 25 (1.5%) were incorrectly cited, while 88 (5.3%) were incorrectly cited in the contents pages. Twenty-one (84%) of the errors in MEDLINE involved names of authors. Apart from incomplete retrieval of information for practice and research, errors could result in an author not getting credit for publications.

[Medical-bioscientific databanks and the Impact Factor]

BACKGROUND

Citation frequencies of medical journals are measured by the Journal Impact Factors (IF) published annually in the Journal Citation Report (JCR). Conclusions drawn from IF concerning the distribution of single journal articles are used worldwide for academic evaluation purposes. Because of this importance, IF are widely and controversially discussed, also regarding their derivation from a limited pool of databases (Science Citation Index, SCI).

OBJECTIVE

To determine the comprehensiveness of IF's data basis by testing (i) SCI's sources, (ii) SCI's output. Are the IF sufficient for an objective evaluation of medical journals?

METHODS

Comparative searches in 38 databases and their combinations (SCI; MEDLINE [ME]; EMBASE [EM]; BIOSIS Previews [BA] and other relevant systems).

RESULTS

(i) Journals with higher IF (> or = 1) are almost completely retrievable in SCI (98%), but only approx. 60% in ME, EM, BA. (ii) Reverse: three samples of mainly German-language journals frequently indexed in SCI were represented in JCR by 90%; but only 23.5-57% of sample periodicals had an IF when indexed in ME, EM and BA, but not in SCI. (iii) Compiled average search results in the most productive databases in 18 biomedical queries, when titles were searched: SCI = 34%, ME = 27%, EM = 33%, BA = 25%; and, when combined: SCI + ME = 44%, SCI + ME + EM = 55%, SCI + ME + EM + BA = 65%, compared to the results in a 38-databases cluster. Costs increase in the order ME < EM < SCI < BA < Derwent, CAS.

CONCLUSIONS

(i) The citation analyses presented in JCRs appear limited especially regarding German-language biomedical journals. Evaluation of publications based on IF therefore should be complemented by corrective measures. (ii) Single-database searches, including SCI, at best render orientating results; database combinations are recommended when higher completeness is required.