Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

BACKGROUND

Effective study identification is essential for conducting health research, developing clinical guidance and health policy and supporting health-care decision-making. Methodological search filters (combinations of search terms to capture a specific study design) can assist in searching to achieve this.

OBJECTIVES

This project investigated the methods used to assess the performance of methodological search filters, the information that searchers require when choosing search filters and how that information could be better provided.

METHODS

Five literature reviews were undertaken in 2010/11: search filter development and testing; comparison of search filters; decision-making in choosing search filters; diagnostic test accuracy (DTA) study methods; and decision-making in choosing diagnostic tests. We conducted interviews and a questionnaire with experienced searchers to learn what information assists in the choice of search filters and how filters are used. These investigations informed the development of various approaches to gathering and reporting search filter performance data. We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator.

RESULTS

The development of filters most frequently involved using a reference standard derived from hand-searching journals. Most filters were validated internally only. Reporting of methods was generally poor. Sensitivity, precision and specificity were the most commonly reported performance measures and were presented in tables. Aspects of DTA study methods are applicable to search filters, particularly in the development of the reference standard. There is limited evidence on how clinicians choose between diagnostic tests. No published literature was found on how searchers select filters. Interviewing and questioning searchers via a questionnaire found that filters were not appropriate for all tasks but were predominantly used to reduce large numbers of retrieved records and to introduce focus. The Inter Technology Appraisal Support Collaboration (InterTASC) Information Specialists' Sub-Group (ISSG) Search Filters Resource was most frequently mentioned by both groups as the resource consulted to select a filter. Randomised controlled trial (RCT) and systematic review filters, in particular the Cochrane RCT and the McMaster Hedges filters, were most frequently mentioned. The majority indicated that they used different filters depending on the requirement for sensitivity or precision. Over half of the respondents used the filters available in databases. Interviewees used various approaches when using and adapting search filters. Respondents suggested that the main factors that would make choosing a filter easier were the availability of critical appraisals and more detailed performance information. Provenance and having the filter available in a central storage location were also important.

LIMITATIONS

The questionnaire could have been shorter and could have included more multiple choice questions, and the reviews of filter performance focused on only four study designs.

CONCLUSIONS

Search filter studies should use a representative reference standard and explicitly report methods and results. Performance measures should be presented systematically and clearly. Searchers find filters useful in certain circumstances but expressed a need for more user-friendly performance information to aid filter choice. We suggest approaches to use, adapt and report search filter performance. Future work could include research around search filters and performance measures for study designs not addressed here, exploration of alternative methods of displaying performance results and numerical synthesis of performance comparison results.

FUNDING

The National Institute for Health Research (NIHR) Health Technology Assessment programme and Medical Research Council-NIHR Methodology Research Programme (grant number G0901496).

Applications of justification and optimization in medical imaging: examples of clinical guidance for computed tomography use in emergency medicine

Availability, reliability, and technical improvements have led to continued expansion of computed tomography (CT) imaging. During a CT scan, there is substantially more exposure to ionizing radiation than with conventional radiography. This has led to questions and critical conclusions about whether the continuous growth of CT scans should be subjected to review and potentially restraints or, at a minimum, closer investigation. This is particularly pertinent to populations in emergency departments, such as children and patients who receive repeated CT scans for benign diagnoses. During the last several decades, among national medical specialty organizations, the American College of Emergency Physicians and the American College of Radiology have each formed membership working groups to consider value, access, and expedience and to promote broad acceptance of CT protocols and procedures within their disciplines. Those efforts have had positive effects on the use criteria for CT by other physician groups, health insurance carriers, regulators, and legislators.

Applications of justification and optimization in medical imaging: examples of clinical guidance for computed tomography use in emergency medicine

Availability, reliability, and technical improvements have led to continued expansion of computed tomography (CT) imaging. During a CT scan, there is substantially more exposure to ionizing radiation than with conventional radiography. This has led to questions and critical conclusions about whether the continuous growth of CT scans should be subjected to review and potentially restraints or, at a minimum, closer investigation. This is particularly pertinent to populations in emergency departments, such as children and patients who receive repeated CT scans for benign diagnoses. During the last several decades, among national medical specialty organizations, the American College of Emergency Physicians and the American College of Radiology have each formed membership working groups to consider value, access, and expedience and to promote broad acceptance of CT protocols and procedures within their disciplines. Those efforts have had positive effects on the use criteria for CT by other physician groups, health insurance carriers, regulators, and legislators.

U.S. trends in computed tomography use and diagnoses in emergency department visits by patients with symptoms suggestive of pulmonary embolism, 2001-2009

OBJECTIVES

Using computed tomography (CT) to evaluate patients with chest symptoms is common in emergency departments (EDs). This article describes recent trends of CT use in U.S. EDs for patients presenting with symptoms common to acute pulmonary embolism (PE).

METHODS

The 2001-2009 National Hospital Ambulatory Medical Care Survey (NHAMCS), a nationally representative survey of U.S. ED encounters, was used for data collection. Patients with at least one of three complaints (chest pain, dyspnea, or hemoptysis) were categorized into the chest symptom study (CSS) group. The yearly increases in CT use for the complaints were tabulated first, then linear regression analysis was used to calculate average rates of increases in CT use between 2001 and 2007, the years where CT use increased, for the overall population and among specific subgroups. The ratios of the number of visits when CT was ordered and there was a target diagnosis relative to the total number of visits with CT in the CSS group (diagnosis/CT ratio) were calculated for PE and pneumonia.

RESULTS

Annual CT rates for the CSS group increased from 2.6% in 2001 to 13.2% in 2007, subsequently leveling off at approximately 12.5% in 2008 and 2009. The average growth rate of CT use for the CSS group was 28.1% (95% confidence interval [CI] = 20.9% to 35.7%) per year between 2001 and 2007. Testing rates for all subgroups increased. The lowest growth rate was among Hispanic patients, whose CT rates grew 14.2% (95% CI = 5.7% to 23.5%) per year. The highest growth rate was in nonurban hospitals, at 43.1% (95% CI = 15.2% to 77.8%) per year. Patients triaged as nonurgent received the fewest CTs, compared to those who should be seen in 2 hours or less. With regard to sources of payment, the self-pay subgroup experienced the highest rate of increase at 35.1% (95% CI = 18.6% to 53.9%). The PE diagnosis/CT ratio from 2002 to 2009 was 2.7% for the CSS group. The pneumonia diagnosis/CT ratio grew from 5.8% in 2002 to 2005 to 7.8% in 2006 to 2009.

CONCLUSIONS

Computed tomography use in ED visits by patients with chest symptoms increased dramatically from 2001 to 2007 and seems to have leveled off in subsequent years. The low PE diagnosis-to-CT ratio suggests that EDs may need to promote evidence-based use of CT.

The influence of education on appropriateness rates for CT pulmonary angiography in emergency department patients

RATIONALE AND OBJECTIVES

To evaluate appropriate utilization rates for computed tomography (CT) pulmonary angiography (CTPA) in a tertiary center emergency department (ED), before and after a health care provider educational intervention.

MATERIALS AND METHODS

Institutional Review Board-approved retrospective study. Records for 100 consecutive CTPA studies ordered by the ED were retrieved from a radiology database. Appropriateness rates for the studies were determined using information from existing literature (clinical decision rules and society guidelines). Where pretest probability was not performed, it was calculated by the authors. After ED health care provider education regarding appropriateness guidelines through a dedicated lecture and question-and-answer session, appropriateness rates for another 100 consecutive CTPA ordered by the ED were calculated.

RESULTS

In the preeducational intervention, 1% of patients had Wells scores performed, 65% were women, and 29% were age <40 years. Before CTPA, 40% patients had d-dimer testing, 15% of patients had a "negative" d-dimer, 17% had alternative explanations for chest pain, and 76% had low or intermediate pretest probability. Appropriateness rates for CTPA was 7%, and 8% of studies were positive. Postintervention, no Wells scores were performed, 59% were women, and 34% <40 years. Before CTPA, 32% of patients had d-dimer, 16% had a "negative" d-dimer, 22% had alternative explanations for chest pain, and 84% had low or intermediate pretest probability. The appropriateness rate for CTPA was 6% and 10% of studies were positive.

CONCLUSION

A single educational intervention had no effect on appropriate utilization rates for CTPA. Repeated and sustained educational interventions may help improve imaging ordering pathways through the ED and other departments.

Associations between presence of relevant information in referrals to radiology and prevalence rates in patients with suspected pulmonary embolism

RATIONALE AND OBJECTIVES

The purpose of this study was to assess if the presence of information including the pretest probability (Wells score), other known risk factors, and symptoms given on referrals for computed tomography (CT) pulmonary angiography correlated with prevalence rates for pulmonary embolism (PE). Also, to evaluate for differences between a university and a regional hospital setting regarding patient characteristics, amount of relevant information provided on referrals, and prevalence rates for pulmonary embolism.

MATERIALS AND METHODS

Retrospective review of all consecutive referrals (emergency room, inpatient, and outpatient) for CT performed on children and adults for suspected PE from two sites: a tertiary (university) hospital (site 1) and a secondary (regional) hospital (site 2) over a 5-year period.

RESULTS

The overall prevalence rate was 510/3641 or 14% of all referrals. Significantly higher number of males had a positive CT compared to women (18% versus 12%, P < .001). Although no statistically significant relationship between a greater amount of relevant information on the referral and the probability for positive finding existed, a slight trend was noted (P = .09). In two categories, "hypoxia" and "signs of deep vein thrombosis," the presence of this information conferred a higher probability for pulmonary embolism, P < .001. In the categories, "chest pain," "malaise," and "smoker/chronic obstructive pulmonary disease", the absence of information conferred a higher probability for pulmonary embolism.

CONCLUSIONS

The amount of relevant clinical information on the request did not correlate with prevalence rates, which may reflect a lack of documentation on the part of emergency physicians who may use a "gestalt" approach. Request forms likely did not capture all relevant patient risks and many factors may interact with each other, both positively and negatively. Pretest probability estimations were rarely performed, despite their inclusion in major society guidelines.

Controversies in Diagnosis of Pulmonary Embolism

The approach to the diagnosis of acute pulmonary embolism (PE) is under constant revision with advances in technology, noninvasive approaches, and increasing awareness of the risks of ionizing radiation. Optimal approaches in some categories of patients are controversial. Data are insufficient for evidence-based recommendations. Therefore, this survey of investigators in the field was undertaken. Even among experts there were marked differences of opinion regarding the approach to the diagnosis of acute PE. Although CT pulmonary angiography was usually the imaging test of choice, the respondents were keenly aware of the dangers of ionizing radiation. In view of advances in scintigraphic diagnosis since the Prospective Investigation of Pulmonary Embolism Diagnosis (PIOPED) trial, ventilation/perfusion (V/Q) lung scans or perfusion scans alone and single photon emission computed tomography (SPECT) V/Q lung scans are often recommended. The choice depends on the patient’s age, gender, and complexity of the findings on the plain chest radiograph.