Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews

An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review

BACKGROUND

Systematic reviews (SRs) are time-consuming and labor-intensive to perform. With the growing number of scientific publications, the SR development process becomes even more laborious. This is problematic because timely SR evidence is essential for decision-making in evidence-based healthcare and policymaking. Numerous methods and tools that accelerate SR development have recently emerged. To date, no scoping review has been conducted to provide a comprehensive summary of methods and ready-to-use tools to improve efficiency in SR production.

OBJECTIVE

To present an overview of primary studies that evaluated the use of ready-to-use applications of tools or review methods to improve efficiency in the review process.

METHODS

We conducted a scoping review. An information specialist performed a systematic literature search in four databases, supplemented with citation-based and grey literature searching. We included studies reporting the performance of methods and ready-to-use tools for improving efficiency when producing or updating a SR in the health field. We performed dual, independent title and abstract screening, full-text selection, and data extraction. The results were analyzed descriptively and presented narratively.

RESULTS

We included 103 studies: 51 studies reported on methods, 54 studies on tools, and 2 studies reported on both methods and tools to make SR production more efficient. A total of 72 studies evaluated the validity (n = 69) or usability (n = 3) of one method (n = 33) or tool (n = 39), and 31 studies performed comparative analyses of different methods (n = 15) or tools (n = 16). 20 studies conducted prospective evaluations in real-time workflows. Most studies evaluated methods or tools that aimed at screening titles and abstracts (n = 42) and literature searching (n = 24), while for other steps of the SR process, only a few studies were found. Regarding the outcomes included, most studies reported on validity outcomes (n = 84), while outcomes such as impact on results (n = 23), time-saving (n = 24), usability (n = 13), and cost-saving (n = 3) were less often evaluated.

CONCLUSION

For title and abstract screening and literature searching, various evaluated methods and tools are available that aim at improving the efficiency of SR production. However, only few studies have addressed the influence of these methods and tools in real-world workflows. Few studies exist that evaluate methods or tools supporting the remaining tasks. Additionally, while validity outcomes are frequently reported, there is a lack of evaluation regarding other outcomes.

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

BACKGROUND

The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice.

METHODS

In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies.

RESULTS

From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs.

CONCLUSIONS

Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.

Comparing dietary strategies to manage cardiovascular risk in primary care: a narrative review of systematic reviews

BACKGROUND

Nutrition care in general practice is crucial for cardiovascular disease (CVD) prevention and management, although comparison between dietary strategies is lacking.

AIM

To compare the best available (most recent, relevant, and high-quality) evidence for six dietary strategies that are effective for primary prevention/absolute risk reduction of CVD.

DESIGN AND SETTING

A pragmatic narrative review of systematic reviews of randomised trials focused on primary prevention of cardiovascular events.

METHOD

Studies about: 1) adults without a history of cardiovascular events; 2) target dietary strategies postulated to reduce CVD risk; and 3) direct cardiovascular or all-cause mortality outcomes were included. Six dietary strategies were examined: energy deficit, Mediterranean-like diet, sodium reduction (salt reduction and substitution), the Dietary Approaches to Stop Hypertension (DASH) diet, alcohol reduction, and fish/fish oil consumption. Reviews were selected based on quality, recency, and relevance. Quality and certainty of evidence was assessed using GRADE.

RESULTS

Twenty-five reviews met inclusion criteria; eight were selected as the highest quality, recent, and relevant. Three dietary strategies showed modest, significant reductions in cardiovascular events: energy deficit (relative risk reduction [RRR] 30%, 95% confidence interval [CI] = 13 to 43), Mediterranean-like diet (RRR 40%, 95% CI = 20 to 55), and salt substitution (RRR 30%, 95% CI = 7 to 48). Still, some caveats remain on the effectiveness of these dietary strategies. Salt reduction, DASH diet, and alcohol reduction showed small, significant reductions in blood pressure, but no reduction in cardiovascular events. Fish/fish oil consumption showed little or no effect; supplementation of fish oil alone showed small reductions in CVD events.

CONCLUSION

For primary prevention, energy deficit, Mediterranean-like diets, and sodium substitution have modest evidence for risk reduction of CVD events. Strategies incorporated into clinical nutrition care should ensure guidance is person centred and tailored to clinical circumstances.

Assessment of search strategies in Medline to identify studies on the impact of long COVID on workability

Objectives

Studies on the impact of long COVID on work capacity are increasing but are difficult to locate in bibliographic databases, due to the heterogeneity of the terms used to describe this new condition and its consequences. This study aims to report on the effectiveness of different search strategies to find studies on the impact of long COVID on work participation in PubMed and to create validated search strings.

Methods

We searched PubMed for articles published on Long COVID and including information about work. Relevant articles were identified and their reference lists were screened. Occupational health journals were manually scanned to identify articles that could have been missed. A total of 885 articles potentially relevant were collected and 120 were finally included in a gold standard database. Recall, Precision, and Number Needed to Read (NNR) of various keywords or combinations of keywords were assessed.

Results

Overall, 123 search-words alone or in combination were tested. The highest Recalls with a single MeSH term or textword were 23 and 90%, respectively. Two different search strings were developed, one optimizing Recall while keeping Precision acceptable (Recall 98.3%, Precision 15.9%, NNR 6.3) and one optimizing Precision while keeping Recall acceptable (Recall 90.8%, Precision 26.1%, NNR 3.8).

Conclusions

No single MeSH term allows to find all relevant studies on the impact of long COVID on work ability in PubMed. The use of various MeSH and non-MeSH terms in combination is required to recover such studies without being overwhelmed by irrelevant articles.

Precision and Recall of Search Strategies for Identifying Studies on Work-Related Psychosocial Risk Factors in PubMed

PURPOSE

This study aims to report on the effectiveness of various search strategies and keywords to find studies on work-related psychosocial risk factors (PRF) in the PubMed bibliographic database.

METHODS

We first selected by hand-searching 191articles published on PRF and indexed in PubMed. We extracted 30 relevant MeSH terms and 38 additional textwords. We then searched PubMed combining these 68 keywords and 27 general keywords on work-related factors. Among the 2953 articles published in January 2020, we identified 446 articles concerning exposure to PRF, which were gathered in a Gold Standard database. We then computed the Recall, Precision, and Number Needed to Read of each keyword or combination of keywords.

RESULTS

Overall, 189 search-words alone or in combination were tested. The highest Recall with a single MeSH term or textword was 43% and 35%, respectively. Subsequently, we developed two different search strings, one optimizing Recall while keeping Precision acceptable (Recall 98.2%, Precision 5.9%, NNR 16.9) and one optimizing Precision while keeping Recall acceptable (Recall 73.1%, Precision 25.5%, NNR 9.7).

CONCLUSIONS

No single MeSH term is available to identify relevant studies on PRF in PubMed. Locating these types of studies requires the use of various MeSH and non-MeSH terms in combination to obtain a satisfactory Recall. Nevertheless, enhancing the Recall of search strategies may lead to lower Precision, and higher NNR, although with a non-linear trend. This factor must be taken into consideration when searching PubMed.

Rapid reviews methods series: Guidance on literature search

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RR) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. In this paper, we address considerations for RR searches. We cover the main areas relevant to the search process: preparation and planning, information sources and search methods, search strategy development, quality assurance, reporting, and record management. Two options exist for abbreviating the search process: (1) reducing time spent on conducting searches and (2) reducing the size of the search result. Because screening search results is usually more resource-intensive than conducting the search, we suggest investing time upfront in planning and optimising the search to save time by reducing the literature screening workload. To achieve this goal, RR teams should work with an information specialist. They should select a small number of relevant information sources (eg, databases) and use search methods that are highly likely to identify relevant literature for their topic. Database search strategies should aim to optimise both precision and sensitivity, and quality assurance measures (peer review and validation of search strategies) should be applied to minimise errors.

Complement-activating donor-specific anti-HLA antibodies in solid organ transplantation: systematic review, meta-analysis, and critical appraisal

Introduction

Several studies have investigated the impact of circulating complement-activating anti-human leukocyte antigen donor-specific antibodies (anti-HLA DSAs) on organ transplant outcomes. However, a critical appraisal of these studies and a demonstration of the prognostic value of complement-activating status over anti-HLA DSA mean fluorescence intensity (MFI) level are lacking.

Methods

We conducted a systematic review, meta-analysis and critical appraisal evaluating the role of complement-activating anti-HLA DSAs on allograft outcomes in different solid organ transplants. We included studies through Medline, Cochrane, Scopus, and Embase since inception of databases till May 05, 2023. We evaluated allograft loss as the primary outcome, and allograft rejection as the secondary outcome. We used the Newcastle-Ottawa Scale and funnel plots to assess risk of bias and used bias adjustment methods when appropriate. We performed multiple subgroup analyses to account for sources of heterogeneity and studied the added value of complement assays over anti-HLA DSA MFI level.

Results

In total, 52 studies were included in the final meta-analysis (11,035 patients). Complement-activating anti-HLA DSAs were associated with an increased risk of allograft loss (HR 2.77; 95% CI 2.33-3.29, p<0.001; I²=46.2%), and allograft rejection (HR 4.98; 95% CI 2.96-8.36, p<0.01; I²=70.9%). These results remained significant after adjustment for potential sources of bias and across multiple subgroup analyses. After adjusting on pan-IgG anti-HLA DSA defined by the MFI levels, complement-activating anti-HLA DSAs were significantly and independently associated with an increased risk of allograft loss.

Discussion

We demonstrated in this systematic review, meta-analysis and critical appraisal the significant deleterious impact and the independent prognostic value of circulating complement-activating anti-HLA DSAs on solid organ transplant risk of allograft loss and rejection.

Considerations for conducting systematic reviews: evaluating the performance of different methods for de-duplicating references

BACKGROUND

Systematic reviews involve searching multiple bibliographic databases to identify eligible studies. As this type of evidence synthesis is increasingly pursued, the use of various electronic platforms can help researchers improve the efficiency and quality of their research. We examined the accuracy and efficiency of commonly used electronic methods for flagging and removing duplicate references during this process.

METHODS

A heterogeneous sample of references was obtained by conducting a similar topical search in MEDLINE, Embase, Cochrane Central Register of Controlled Trials, and PsycINFO databases. References were de-duplicated via manual abstraction to create a benchmark set. The default settings were then used in Ovid multifile search, EndNote desktop, Mendeley, Zotero, Covidence, and Rayyan to de-duplicate the sample of references independently. Using the benchmark set as reference, the number of false-negative and false-positive duplicate references for each method was identified, and accuracy, sensitivity, and specificity were determined.

RESULTS

We found that the most accurate methods for identifying duplicate references were Ovid, Covidence, and Rayyan. Ovid and Covidence possessed the highest specificity for identifying duplicate references, while Rayyan demonstrated the highest sensitivity.

CONCLUSION

This study reveals the strengths and weaknesses of commonly used de-duplication methods and provides strategies for improving their performance to avoid unintentionally removing eligible studies and introducing bias into systematic reviews. Along with availability, ease-of-use, functionality, and capability, these findings are important to consider when researchers are selecting database platforms and supporting software programs for conducting systematic reviews.

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

INTRODUCTION

Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies.

METHODS

Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions.

RESULTS

Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases.

DISCUSSION

Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective.

CONCLUSIONS

The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

The risk of conclusion change in systematic review updates can be estimated by learning from a database of published examples

OBJECTIVES

To determine which systematic review characteristics are needed to estimate the risk of conclusion change in systematic review updates.

STUDY DESIGN AND SETTING

We applied classification trees (a machine learning method) to model the risk of conclusion change in systematic review updates, using pairs of systematic reviews and their updates as samples. The classifiers were constructed using a set of features extracted from systematic reviews and the relevant trials added in published updates. Model performance was measured by recall, precision, and area under the receiver operating characteristic curve (AUC).

RESULTS

We identified 63 pairs of systematic reviews and updates, of which 20 (32%) exhibited a change in conclusion in their updates. A classifier using information about new trials exhibited the highest performance (AUC: 0.71; recall: 0.75; precision: 0.43) compared to a classifier that used fewer features (AUC: 0.65; recall: 0.75; precision: 0.39).

CONCLUSION

When estimating the risk of conclusion change in systematic review updates, information about the sizes of trials that will be added in an update are most useful. Future tools aimed at signaling conclusion change risks would benefit from complementary tools that automate screening of relevant trials.

Study-based registers reduce waste in systematic reviewing: discussion and case report

BACKGROUND

Maintained study-based registers (SBRs) have, at their core, study records linked to, potentially, multiple other records such as references, data sets, standard texts and full-text reports. Such registers can minimise and refine searching, de-duplicating, screening and acquisition of full texts. SBRs can facilitate new review titles/updates and, within seconds, inform the team about the potential workload of each task.

METHODS

We discuss the advantages/disadvantages of SBRs and report a case of how such a register was used to develop a successful grant application and deliver results-reducing considerable redundancy of effort.

RESULTS

SBRs saved time in question-setting and scoping and made rapid production of nine Cochrane systematic reviews possible.

CONCLUSION

Whilst helping prioritise and conduct systematic reviews, SBRs improve quality. Those funding information specialists for literature reviewing could reasonably stipulate the resulting SBR to be delivered for dissemination and use beyond the life of the project.

Design and implementation of a tool for conversion of search strategies between PubMed and Ovid MEDLINE

BACKGROUND

Both PubMed and Ovid MEDLINE contain records from the MEDLINE database. However, there are subtle differences in content, functionality, and search syntax between the two. There are many instances in which researchers may wish to search both interfaces, such as when conducting supplementary searching for a systematic review to retrieve a unique content from PubMed or when using a previously published search strategy from a different interface, but little guidance on how to best conduct these searches. The aim of this project is to describe differences in search functionality between Ovid MEDLINE and PubMed, provide guidance for converting search strategies between the two, and develop an easy-to-use, freely available web-based tool to automate search syntax translations.

CASE PRESENTATION

In this paper, we present a custom-built freely available online tool, Medline Transpose, to streamline the process of converting search strategies between Ovid MEDLINE and PubMed. With this tool, users can paste a strategy formatted for one interface into the search box and immediately retrieve an output formatted for use in the other interface, with recommendations for changes that users can make to the strategy where an exact translation does not exist.

CONCLUSION

This novel approach has the potential to reduce time and errors that database users spend translating search strategies.