Handsearching had best recall but poor efficiency when exporting to a bibliographic tool: case study

Searching and reporting in Campbell Collaboration systematic reviews: A systematic assessment of current methods

The search methods used in systematic reviews provide the foundation for establishing the body of literature from which conclusions are drawn and recommendations made. Searches should aim to be comprehensive and reporting of search methods should be transparent and reproducible. Campbell Collaboration systematic reviews strive to adhere to the best methodological guidance available for this type of searching. The current work aims to provide an assessment of the conduct and reporting of searches in Campbell Collaboration systematic reviews. Our objectives were to examine how searches are currently conducted in Campbell systematic reviews, how search strategies, search methods and search reporting adhere to the Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR) and PRISMA standards, and identify emerging or novel methods used in searching in Campbell systematic reviews. We also investigated the role of information specialists in Campbell systematic reviews. We handsearched the Campbell Systematic Reviews journal tables of contents from January 2017 to March 2024. We included all systematic reviews published since 2017. We excluded other types of evidence synthesis (e.g., evidence and gap maps), updates to systematic reviews when search methods were not changed from the original pre-2017 review, and systematic reviews that did not conduct their own original searches. We developed a data extraction form in part based on the conduct and reporting items in MECCIR and PRISMA. In addition, we extracted information about the general quality of searches based on the use of Boolean operators, keywords, database syntax and subject headings. Data extraction included information about reporting of sources searched, some aspects of search quality, the use and reporting of supplementary search methods, reporting of the search strategy, the involvement of information specialists, date of the most recent search, and citation of the Campbell search methods guidance. Items were rated as fully, partially or not conducted or reported. We cross-walked our data extraction items to the 2019 MECCIR standards and 2020 PRISMA guidelines and provide descriptive analyses of the conduct and reporting of searches in Campbell systematic reviews, indicating level of adherence to standards where applicable. We included 111 Campbell systematic reviews across all coordinating groups published since 2017 up to the search date. Almost all (98%) included reviews searched at least two relevant databases and all reported the databases searched. All reviews searched grey literature and most (82%) provided a full list of grey literature sources. Detailed information about databases such as platform and date range coverage was lacking in 16% and 77% of the reviews, respectively. In terms of search strategies, most used Boolean operators, search syntax and phrase searching correctly, but subject headings in databases with controlled vocabulary were used in only about half of the reviews. Most reviews reported at least one full database search strategy (90%), with 63% providing full search strategies for all databases. Most reviews conducted some supplementary searching, most commonly searching the references of included studies, whereas handsearching of journals and forward citation searching were less commonly reported (51% and 62%, respectively). Twenty-nine percent of reviews involved an information specialist co-author and about 45% did not mention the involvement of any information specialist. When information specialists were co-authors, there was a concomitant increase in adherence to many reporting and conduct standards and guidelines, including reporting website URLs, reporting methods for forward citation searching, using database syntax correctly and using subject headings. No longitudinal trends in adherence to conducting and reporting standards were found and the Campbell search methods guidance published in 2017 was cited in only twelve reviews. We also found a median time lag of 20 months between the most recent search and the publication date. In general, the included Campbell systematic reviews searched a wide range of bibliographic databases and grey literature, and conducted at least some supplementary searching such as searching references of included studies or contacting experts. Reporting of mandatory standards was variable with some frequently unreported (e.g., website URLs and database date ranges) and others well reported in most reviews. For example, database search strategies were reported in detail in most reviews. For grey literature, source names were well reported but search strategies were less so. The findings will be used to identify opportunities for advancing current practices in Campbell reviews through updated guidance, peer review processes and author training and support.

Supplementary strategies identified additional eligible studies in qualitative systematic reviews

BACKGROUND AND OBJECTIVES

There is an increasing number of qualitative systematic reviews being produced. Searching for qualitative literature to include in these systematic reviews is, however, more challenging and may result in less than favorable recall. Database searches relying solely on key elements of the research question may not retrieve all relevant qualitative studies for synthesis, and supplementary searches may be pertinent to complement the searches. This study aimed to determine, if a) supplementary search strategies (citation searches and alternative search strategies) were able to identify relevant publications for qualitative systematic reviews that were nonretrievable, when conducting traditional database searches based on key elements; and b) to investigate the total number of identified publications when combining traditional database searches with these supplementary search strategies.

METHODS

From a previous study, a gold standard of 12 qualitative reviews including 101 PubMed-indexed publications was used. One of the reviews had only one included publication and in one review, the two included studies were identifiable in PubMed. In the remaining 10 reviews, 61 publications were retrievable through traditional database searches, and 37 were nonidentifiable. The 61 publications were used as basis for possible identification of the 37 publications by using the supplementary search strategies: citations searches (review of reference lists, PubMed "Cited by" function; Scopus "Cited by" function, Citationchaser, CoCites plugin for PubMed) and alternative search strategies (PubMed "similar articles" function; Scopus "Related documents based on references").

RESULTS

Traditional database searches retrieved 62.4% of the 101 publications. Citations searches in Scopus, Citationchaser and CoCites identified 21 (56.8%) of the 37 remaining publications. The PubMed "Cited by" function did not identify any of the 37 publications. The alternative search strategies, the PubMed "Similar articles" together with Scopus "Related documents based on references"-function) identified 15 (40.5%) of the 37 publications. Together, these supplementary search strategies identified 25 (67.6%) of the 37 publications, resulting in an overall retrieval of 87.1%, when combining traditional database searches and supplementary search strategies.

CONCLUSION

The results of this study indicate that supplementary search strategies (citation searches and alternative search strategies) increase the retrieval potential, when searching for qualitative publications and should be included, when identifying literature for qualitative reviews.

Reporting of the methodological quality of search strategies in orthodontic quantitative systematic reviews

BACKGROUND

This study aimed to assess the reporting of the methodological quality of search strategies undertaken in orthodontic quantitative systematic reviews (SRs) and hence their reproducibility.

MATERIALS AND METHODS

A search of a single electronic database (Medline via PubMed) was undertaken to identify interventional orthodontic SRs with meta-analysis published within a 10-year period. The Cochrane Library of Systematic Reviews was also sourced. Full articles were reviewed by two assessors against the eligibility criteria. The reporting quality of each search strategy was assessed using a previously validated checklist with a score of 1 or 2 given for each of the eight items. Cumulative totals were calculated. Guided by previous research, the authors agreed the following cut-offs to categorize the overall level of quality: 8-10 (poor), 10-12 (fair), and greater than 13 (good).

RESULTS

A total of 127 SRs were analysed. The overall median quality score for the reporting of the search strategy was 14 [interquartile range (IQR): 13-15]. Cochrane SRs and those originating in Europe received higher aggregate scores, whereas no difference was evident based on Prospero registration. The continent of the corresponding author predicated the overall score. Non-Cochrane reviews achieved lower overall scores compared to Cochrane reviews (-1.0, 95% confidence interval: -1.65, -0.34, P = 0.003). The most frequently searched database was EMBASE (N = 93) and the median number of authors was 5 (IQR 4-6). Authors of 26.8% of SRs searched the grey literature. Language restrictions were applied to the search strategies of 88 (69.3%) SRs.

CONCLUSIONS

The reporting quality of search strategies undertaken in orthodontic SRs is at a good level but differences between Cochrane and non-Cochrane reviews currently exist. The reporting of searching of the grey literature and application of no language restrictions can be improved.

Searching for medical devices - practical guidance

Systematic reviews of medical devices have generally adopted the same methodology for the conduct of their reviews as reviews of other clinical interventions, in particular, medicines. It has been suggested that these methods may need to be developed to account for the challenges of reviewing the evidence for medical devices when compared to reviews of medicines. The purpose of this research note is to focus on the methods of searching for medical devices. Our aim is to set out guidance on 'how to search' for medical devices. This includes: Defining what you are searching for; How to design a search strategy; Searching bibliographic databases; Searching beyond bibliographic databases; and Search reporting. The research note is written by three experienced searchers/researchers with experience of critically appraising Medical Technologies Evaluation Programme (MTEP) submissions, undertaking systematic searches and/or reviews of medical devices, or developing guidance for searching for studies for systematic reviews of the effectiveness of interventions, including medical devices. The research note includes a worked example of a search for a fictitious medical device. This article is protected by copyright. All rights reserved.

A technical review of three clinical trials register resources indicates where improvements to the search interfaces are needed

Clinical trials registers form an important part of the search for studies in systematic reviews of intervention effectiveness but the search interfaces and functionality of registers can be challenging to search systematically and resource intensive to search well. We report a technical review of the search interfaces of three leading trials register resources: ClinicalTrials.gov, the EU Clinical Trials Register and the WHO International Clinical Trials Registers Platform. The technical review used a validated checklist to identify areas where the search interfaces of these trials register resources performed well, where performance was adequate, where performance was poor, and to identify differences between search interfaces. The review found low overall scores for each of the interfaces (ClinicalTrials.gov 55/165, the EU Clinical Trials Register 25/165, the WHO International Clinical Trials Registers Platform 32/165). This finding suggests a need for joined-up dialogue between the producers of the registers and researchers who search them via these interfaces. We also set out a series of four proposed changes which might improve the search interfaces. Trials registers are an invaluable resource in systematic reviews of intervention effectiveness. With the continued growth in systematic reviews, and initiatives such as 'AllTrials', there is an anticipated need for these resources. We conclude that small changes to the search interfaces, and improved dialogue with providers, might improve the future search functionality of these valuable resources.

"It has no meaning to me." How do researchers understand the effectiveness of literature searches? A qualitative analysis and preliminary typology of understandings

This study aimed to address the question: what does "effectiveness" mean to researchers in the context of literature searching for systematic reviews? We conducted a thematic analysis of responses to an e-mail survey. Eighty-nine study authors, whose studies met inclusion in a recent review (2018), were contacted via e-mail and asked three questions; one directly asking the question: in literature searching, what does effective (or effectiveness in) literature searching mean to you? Thirty-eight (46%) responses were received from diverse professional groups, including: literature searchers, systematic reviewers, clinicians and researchers. A shared understanding of what effectiveness means was not identified. Instead, five themes were developed from data: (a) effectiveness is described as a metric; (b) effectiveness is a balance between metrics; (c) effectiveness can be categorized by search purpose; (d) effectiveness is an outcome; and, (e) effectiveness is an experimental concept. We propose that these themes constitute a preliminary typology of understandings. No single definition of effectiveness was identified. The proposed typology suggests that different researchers have differing understandings of effectiveness. This could lead to uncertainty as to the aim and the purpose of literature searches and confusion about the outcomes. The typology offers a potential route for further exploration.