Validated geographic search filters for bibliographic databases: a scoping review protocol

OBJECTIVE

The purpose of this scoping review is to identify validated geographic search filters and report their methodology and performance measures.

INTRODUCTION

Data on specific geographic areas can be required for evidence syntheses topics, such as the investigation of regional inequalities in health care or to answer context-specific epidemiological questions. Search filters are useful tools for reviewers aiming to identify publications with common characteristics in bibliographic databases. Geographic search filters limit the literature search results to a specific geographic feature (eg, a country or region).

INCLUSION CRITERIA

We will include reports on validated geographic search filters that aim to identify research evidence about a defined geographic area (eg, a country/region or a group of countries/regions).

METHODS

This review will be conducted in accordance with JBI methodology for scoping reviews. The literature search will be conducted in PubMed and Embase. The InterTASC Information Specialists' Sub-Group Search Filter resource and Google Scholar will also be searched. Reports published in any language, from database inception to the present, will be considered for inclusion. Two researchers will independently screen the title, abstract, and full text of the search results. A third reviewer will be consulted in the event of any disagreements. The data extraction will include study characteristics, basic characteristics of the geographical search filter (eg, country/region), and the methods used to develop and validate the search filter. The extracted data will be summarized narratively and presented in a table.

REVIEW REGISTRATION

Open Science Framework https://osf.io/5czhs.

Economic evaluations of artificial intelligence-based healthcare interventions: a systematic literature review of best practices in their conduct and reporting

Objectives: Health economic evaluations (HEEs) help healthcare decision makers understand the value of new technologies. Artificial intelligence (AI) is increasingly being used in healthcare interventions. We sought to review the conduct and reporting of published HEEs for AI-based health interventions. Methods: We conducted a systematic literature review with a 15-month search window (April 2021 to June 2022) on 17th June 2022 to identify HEEs of AI health interventions and update a previous review. Records were identified from 3 databases (Medline, Embase, and Cochrane Central). Two reviewers screened papers against predefined study selection criteria. Data were extracted from included studies using prespecified data extraction tables. Included studies were quality assessed using the National Institute for Health and Care Excellence (NICE) checklist. Results were synthesized narratively. Results: A total of 21 studies were included. The most common type of AI intervention was automated image analysis (9/21, 43%) mainly used for screening or diagnosis in general medicine and oncology. Nearly all were cost-utility (10/21, 48%) or cost-effectiveness analyses (8/21, 38%) that took a healthcare system or payer perspective. Decision-analytic models were used in 16/21 (76%) studies, mostly Markov models and decision trees. Three (3/16, 19%) used a short-term decision tree followed by a longer-term Markov component. Thirteen studies (13/21, 62%) reported the AI intervention to be cost effective or dominant. Limitations tended to result from the input data, authorship conflicts of interest, and a lack of transparent reporting, especially regarding the AI nature of the intervention. Conclusion: Published HEEs of AI-based health interventions are rapidly increasing in number. Despite the potentially innovative nature of AI, most have used traditional methods like Markov models or decision trees. Most attempted to assess the impact on quality of life to present the cost per QALY gained. However, studies have not been comprehensively reported. Specific reporting standards for the economic evaluation of AI interventions would help improve transparency and promote their usefulness for decision making. This is fundamental for reimbursement decisions, which in turn will generate the necessary data to develop flexible models better suited to capturing the potentially dynamic nature of AI interventions.

The NICE OECD countries' geographic search filters: Part 2-validation of the MEDLINE and Embase (Ovid) filters

Objective:

We previously developed draft MEDLINE and Embase (Ovid) geographic search filters for Organisation for Economic Co-operation and Development (OECD) countries to assess their feasibility for finding evidence about the countries. Here, we describe the validation of these search filters.

Methods:

We identified OECD country references from thirty National Institute for Health and Care Excellence (NICE) guidelines to generate gold standard sets for MEDLINE (n=2,065) and Embase (n=2,023). We validated the filters by calculating their recall against these sets. We then applied the filters to existing search strategies for three OECD-focused NICE guideline reviews (NG103 on flu vaccination, NG140 on abortion care, and NG146 on workplace health) to calculate the filters' impact on the number needed to read (NNR) of the searches.

Results:

The filters both achieved 99.95% recall against the gold standard sets. Both filters achieved 100% recall for the three NICE guideline reviews. The MEDLINE filter reduced NNR from 256 to 232 for the NG103 review, from 38 to 27 for the NG140 review, and from 631 to 591 for the NG146 review. The Embase filter reduced NNR from 373 to 341 for the NG103 review, from 101 to 76 for the NG140 review, and from 989 to 925 for the NG146 review.

Conclusion:

The NICE OECD countries' search filters are the first validated filters for the countries. They can save time for research topics about OECD countries by finding the majority of evidence about OECD countries while reducing search result volumes in comparison to no filter use.

The NICE OECD countries' geographic search filters: Part 1-methodology for developing the draft MEDLINE and Embase (Ovid) filters

Objective:

There are no existing validated search filters for the group of 37 Organisation for Economic Co-operation and Development (OECD) countries. This study describes how information specialists from the United Kingdom's National Institute for Health and Care Excellence (NICE) developed and evaluated novel OECD countries’ geographic search filters for MEDLINE and Embase (Ovid) to improve literature search effectiveness for evidence about OECD countries.

Methods:

We created the draft filters using an alternative approach to standard filter construction. They are composed entirely of geographic subject headings and are designed to retain OECD country evidence by excluding non-OECD country evidence using the NOT Boolean operator. To evaluate the draft filters’ effectiveness, we used MEDLINE and Embase literature searches for three NICE guidelines that retrieved >5,000 search results. A 10% sample of the excluded references was screened to check that OECD country evidence was not inadvertently excluded.

Results:

The draft MEDLINE filter reduced results for each NICE guideline by 9.5% to 12.9%. In Embase, search results were reduced by 10.7% to 14%. Of the sample references, 7 of 910 (0.8%) were excluded inadvertently. These references were from a guideline about looked-after minors that concerns both OECD and non-OECD countries.

Conclusion:

The draft filters look promising—they reduced search result volumes while retaining most OECD country evidence from MEDLINE and Embase. However, we advise caution when using them in topics about both non-OECD and OECD countries. We have created final versions of the search filters and will validate them in a future study.

The NICE Evidence Standards Framework for digital health and care technologies - Developing and maintaining an innovative evidence framework with global impact

Objective

In 2018, the UK National Institute for Health and Care Excellence (NICE), in partnership with Public Health England, NHS England, NHS Improvement and others, developed an evidence standards framework (ESF) for digital health and care technologies (DHTs). The ESF was designed to provide a standardised approach to guide developers and commissioners on the levels of evidence needed for the clinical and economic evaluation of DHTs by health and care systems.

Methods

The framework was developed using an agile and iterative methodology that included a literature review of existing initiatives and comparison of these against the requirements set by NHS England; iterative consultation with stakeholders through an expert working group and workshops; and questionnaire-based stakeholder input on a publicly available draft document.

Results

The evidence standards framework has been well-received and to date the ESF has been viewed online over 55,000 times and downloaded over 19,000 times.

Conclusions

In April 2021 we published an update to the ESF. Here, we summarise the process through which the ESF was developed, reflect on its global impact to date, and describe NICE’s ongoing work to maintain and improve the framework in the context for a fast moving, innovative field.

The NICE UK geographic search filters for MEDLINE and Embase (Ovid): Post-development study to further evaluate precision and number-needed-to-read when retrieving UK evidence

BACKGROUND

The National Institute for Health and Care Excellence's (NICE) United Kingdom (UK) geographic search filters for MEDLINE and Embase (OVID) retrieve evidence in literature searches for UK-focused research topics with high recall. Their precision and number-needed-to-read (NNR) was examined previously in case studies using a single review. This paper details a larger post-development study that was conducted to test the NICE UK filters' precision and NNR more extensively.

METHODS

The filters' recall of included UK references from 100 reviews was calculated. As reproducible search strategies were not available for every review, the MEDLINE filter's precision and NNR were calculated using strategies from 25 reviews. Strategies from nine reviews were used for the Embase filter.

RESULTS

The MEDLINE filter achieved an average of 96.4% recall for the included UK references from the 100 reviews and the Embase filter achieved an average of 97.4% recall. Compared to not using a filter, the MEDLINE filter achieved an average of 98.9% recall for the 25 reviews. Precision was increased by an average of 7.8 times, reducing the NNR from 357 to 46. The Embase filter achieved an average of 97.1% recall for the nine reviews. Precision was increased by an average of 5.1 times, reducing the NNR from 746 to 146.

CONCLUSION

There is more evidence to demonstrate that the NICE UK filters retrieve the majority of UK evidence from MEDLINE and Embase while increasing precision and reducing NNR. The filters can save time spent on selecting evidence for UK-focused research topics.

How to Develop a Validated Geographic Search Filter: Five Key Steps

No abstract.

The Embase UK filter: validation of a geographic search filter to retrieve research about the UK from OVID Embase

BACKGROUND

The authors developed a validated geographic search filter to retrieve research about the United Kingdom (UK) from OVID Embase. It was created to be used alongside their previously published OVID MEDLINE UK filter in systematic literature searches for context-sensitive topics.

OBJECTIVES

To develop a validated geographic search filter to retrieve research about the UK from OVID Embase.

METHODS

The Embase UK filter was translated from the MEDLINE UK filter. A gold standard set of references was generated using the relative recall method. The set contained references to publications about the UK that had informed National Institute for Health and Care Excellence (NICE) guidance and it was used to validate the filter. Recall, precision and number-needed-to-read (NNR) were calculated using a case study.

RESULTS

The validated Embase UK filter demonstrated 99.8% recall against the references with UK identifiers in the gold standard set. In the case study, the Embase UK filter demonstrated 98.5% recall, 7.6% precision and a NNR of 13.

CONCLUSION

The Embase UK filter can be used alongside the MEDLINE UK filter. The filters have the potential to save time and associated resource costs when they are used for context-sensitive topics that require research about UK settings.

The medline UK filter: development and validation of a geographic search filter to retrieve research about the UK from OVID medline

BACKGROUND

A validated geographic search filter for the retrieval of research about the United Kingdom (UK) from bibliographic databases had not previously been published.

OBJECTIVES

To develop and validate a geographic search filter to retrieve research about the UK from OVID medline with high recall and precision.

METHODS

Three gold standard sets of references were generated using the relative recall method. The sets contained references to studies about the UK which had informed National Institute for Health and Care Excellence (NICE) guidance. The first and second sets were used to develop and refine the medline UK filter. The third set was used to validate the filter. Recall, precision and number-needed-to-read (NNR) were calculated using a case study.

RESULTS

The validated medline UK filter demonstrated 87.6% relative recall against the third gold standard set. In the case study, the medline UK filter demonstrated 100% recall, 11.4% precision and a NNR of nine.

CONCLUSION

A validated geographic search filter to retrieve research about the UK with high recall and precision has been developed. The medline UK filter can be applied to systematic literature searches in OVID medline for topics with a UK focus.

Systematic review of recent evidence for the safety and efficacy of elective endovascular repair in the management of infrarenal abdominal aortic aneurysm

Background

Conventional management of abdominal aortic aneurysm (AAA) is by open repair and is associated with a mortality rate of 2–6 per cent. Endovascular aneurysm repair (EVAR) is an alternative technique first introduced in 1991. A systematic review was undertaken of the evidence for the safety and efficacy of elective EVAR in the management of asymptomatic infrarenal AAA.

Methods

Thirteen electronic bibliographical databases were searched, covering biomedical, health-related, science and social science literature. Outcomes were assessed with respect to efficacy (successful deployment, technical success, conversion rates and secondary intervention rates) and safety (30-day mortality rate, procedure morbidity rates and technical issues—endoleaks, graft thrombosis, stenosis and migration).

Results

Of 606 reports identified, 61 met the inclusion criteria (three randomized and 15 non-randomized controlled trials, and 43 uncontrolled studies). There were 29 059 participants in total; 19 804 underwent EVAR. Deployment was successful in 97·6 per cent of cases. Technical success (complete aneurysm exclusion) was 81·9 per cent at discharge and 88·8 per cent at 30 days. Secondary intervention to treat endoleak or maintain graft patency was required in 16·2 per cent of patients. Mean stay in the intensive care unit and mean hospital stay were significantly shorter following EVAR. The 30-day mortality rate for EVAR was 1·6 per cent (randomized controlled trials) and 2·0 per cent in nonrandomized trials and case series. Technical complications comprised stent migration (4·0 per cent), graft limb thrombosis (3·9 per cent), endoleak (type I, 6·8 per cent; type II, 10·3 per cent; type III, 4·2 per cent) and access artery injury (4·8 per cent).

Discussion

EVAR is technically effective and safe, with lower short-term morbidity and mortality rates than open surgery. However, there is a need for extended follow-up as the long-term success of EVAR in preventing aneurysm-related deaths is not yet known.