Methods for documenting systematic review searches: a discussion of common issues

Expert searchers identified time, team, technology and tension as challenges when carrying out supplementary searches for systematic reviews: A thematic network analysis

BACKGROUND

Systematic reviews require detailed planning of complex processes which can present logistical challenges. Understanding these logistical challenges can help with planning and execution of tasks OBJECTIVES: To describe the perspectives of expert searchers on the main logistical challenges when carrying out supplementary searches for systematic reviews, in particular, forward citation searching and web searching.

METHODS

Qualitative interviews were undertaken with 15 experts on searching for studies for systematic reviews (e.g. information specialists) working in health and social care research settings. Interviews were undertaken by video-call between September 2020 and June 2021. Data analysis used thematic network analysis.

RESULTS

We identified three logistical challenges of using forward citation searching and web searching which were organised under the global theme of 'tension': time, team and technology. Several subthemes were identified which supported the organising themes, including allocating time, justifying time and keeping to time; reviewer expectations and contact with review teams; and access to resources and reference management.

CONCLUSION

Forward citation searching and web searching are logistically challenging search methods for a systematic review. An understanding of these challenges should encourage expert searchers and review teams to maintain open channels of communication, which should also facilitate improved working relationships.

Systematic Review and Meta Analysis of the Relative Effect on Plaque Index among Pediatric Patients Using Powered (Electric) versus Manual Toothbrushes

Although many randomized controlled trials (RCT) have evaluated the efficacy of powered or electric toothbrushes compared with manual or traditional toothbrushes to remove biofilm and plaque, only one systematic review has been published for pediatric patients. The primary objective of this study was to perform a systematic review and meta analysis for this population. Using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) protocol, N = 321 studies were initially identified. Three independent, blinded abstract reviews were completed resulting in a total of n = 38/322 or 11.8% for the final analysis (n = 27 non-orthodontic, n = 11 orthodontic studies). Meta analysis of these outcome data have revealed a strong reduction in plaque index scores among pediatric patients using electric toothbrushes of approximately 17.2% for non-orthodontic patients and 13.9% for orthodontic patients. These results provide strong clinical evidence for recommending electric toothbrushing to pediatric patients, as well as those patients undergoing orthodontic therapy and treatment.

A suggested data structure for transparent and repeatable reporting of bibliographic searching

Academic searching is integral to research activities: (1) searching to retrieve specific information, (2) to expand our knowledge iteratively, (3) and to collate a representative and unbiased selection of the literature. Rigorous searching methods are vital for reliable, repeatable and unbiased searches needed for these second and third forms of searches (exploratory and systematic searching, respectively) that form a core part of evidence syntheses. Despite the broad awareness of the importance of transparency in reporting search activities in evidence syntheses, the importance of searching has been highlighted only recently and has been the explicit focus of reporting guidance (PRISMA-S). Ensuring bibliographic searches are reported in a way that is transparent enough to allow for full repeatability or evaluation is challenging for a number of reasons. Here, we detail these reasons and provide for the first time a standardised data structure for transparent and comprehensive reporting of search histories. This data structure was produced by a group of international experts in informatics and library sciences. We explain how the data structure was produced and describe its components in detail. We also demonstrate its practical applicability in tools designed to support literature review authors and explain how it can help to improve interoperability across tools used to manage literature reviews. We call on the research community and developers of reference and review management tools to embrace the data structure to facilitate adequate reporting of academic searching in an effort to raise the standard of evidence syntheses globally.

A Tailored Approach: A model for literature searching in complex systematic reviews

Our previous work identified that nine leading guidance documents for seven different types of systematic review advocated the same process of literature searching. We defined and illustrated this process and we named it ‘the Conventional Approach’. The Conventional Approach appears to meet the needs of researchers undertaking literature searches for systematic reviews of clinical interventions. In this article, we report a new and alternate process model of literature searching called ‘A Tailored Approach’. A Tailored Approach is indicated as a search process for complex reviews which do not focus on the evaluation of clinical interventions. The aims of this article are to (1) explain the rationale for, and the theories behind, the design of A Tailored Approach; (2) report the current conceptual illustration of A Tailored Approach and to describe a user’s interaction with the process model; and (3) situate the elements novel to A Tailored Approach (when compared with the Conventional Approach) in the relevant literature. A Tailored Approach suggests investing time at the start of a review, to develop the information needs from the research objectives, and to tailor the search approach to studies or data. Tailored Approaches should be led by the information specialist (librarian) but developed by the research team. The aim is not necessarily to focus on comprehensive retrieval. Further research is indicated to evaluate the use of supplementary search methods, methods of team-working to define search approaches, and to evaluate the use of conceptual models of information retrieval for testing and evaluation.

PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. J Med Libr Assoc 2021;109:174-200. [PMID: 34285662 PMCID: PMC8270366 DOI: 10.5195/jmla.2021.962]

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a three-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes sixteen reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and, therefore, reproducible.

PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews

BACKGROUND

Literature searches underlie the foundations of systematic reviews and related review types. Yet, the literature searching component of systematic reviews and related review types is often poorly reported. Guidance for literature search reporting has been diverse, and, in many cases, does not offer enough detail to authors who need more specific information about reporting search methods and information sources in a clear, reproducible way. This document presents the PRISMA-S (Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension) checklist, and explanation and elaboration.

METHODS

The checklist was developed using a 3-stage Delphi survey process, followed by a consensus conference and public review process.

RESULTS

The final checklist includes 16 reporting items, each of which is detailed with exemplar reporting and rationale.

CONCLUSIONS

The intent of PRISMA-S is to complement the PRISMA Statement and its extensions by providing a checklist that could be used by interdisciplinary authors, editors, and peer reviewers to verify that each component of a search is completely reported and therefore reproducible.

Use of a search summary table to improve systematic review search methods, results, and efficiency

BACKGROUND

Systematic reviews are comprehensive, robust, inclusive, transparent, and reproducible when bringing together the evidence to answer a research question. Various guidelines provide recommendations on the expertise required to conduct a systematic review, where and how to search for literature, and what should be reported in the published review. However, the finer details of the search results are not typically reported to allow the search methods or search efficiency to be evaluated.

CASE PRESENTATION

This case study presents a search summary table, containing the details of which databases were searched, which supplementary search methods were used, and where the included articles were found. It was developed and published alongside a recent systematic review. This simple format can be used in future systematic reviews to improve search results reporting.

CONCLUSIONS

Publishing a search summary table in all systematic reviews would add to the growing evidence base about information retrieval, which would help in determining which databases to search for which type of review (in terms of either topic or scope), what supplementary search methods are most effective, what type of literature is being included, and where it is found. It would also provide evidence for future searching and search methods research.

"It has no meaning to me." How do researchers understand the effectiveness of literature searches? A qualitative analysis and preliminary typology of understandings

This study aimed to address the question: what does "effectiveness" mean to researchers in the context of literature searching for systematic reviews? We conducted a thematic analysis of responses to an e-mail survey. Eighty-nine study authors, whose studies met inclusion in a recent review (2018), were contacted via e-mail and asked three questions; one directly asking the question: in literature searching, what does effective (or effectiveness in) literature searching mean to you? Thirty-eight (46%) responses were received from diverse professional groups, including: literature searchers, systematic reviewers, clinicians and researchers. A shared understanding of what effectiveness means was not identified. Instead, five themes were developed from data: (a) effectiveness is described as a metric; (b) effectiveness is a balance between metrics; (c) effectiveness can be categorized by search purpose; (d) effectiveness is an outcome; and, (e) effectiveness is an experimental concept. We propose that these themes constitute a preliminary typology of understandings. No single definition of effectiveness was identified. The proposed typology suggests that different researchers have differing understandings of effectiveness. This could lead to uncertainty as to the aim and the purpose of literature searches and confusion about the outcomes. The typology offers a potential route for further exploration.

Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Rigorous evidence identification is essential for systematic reviews and meta-analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query-based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

A systematic approach to searching: an efficient and complete method to develop literature searches

Creating search strategies for systematic reviews, finding the best balance between sensitivity and specificity, and translating search strategies between databases is challenging. Several methods describe standards for systematic search strategies, but a consistent approach for creating an exhaustive search strategy has not yet been fully described in enough detail to be fully replicable. The authors have established a method that describes step by step the process of developing a systematic search strategy as needed in the systematic review. This method describes how single-line search strategies can be prepared in a text document by typing search syntax (such as field codes, parentheses, and Boolean operators) before copying and pasting search terms (keywords and free-text synonyms) that are found in the thesaurus. To help ensure term completeness, we developed a novel optimization technique that is mainly based on comparing the results retrieved by thesaurus terms with those retrieved by the free-text search words to identify potentially relevant candidate search terms. Macros in Microsoft Word have been developed to convert syntaxes between databases and interfaces almost automatically. This method helps information specialists in developing librarian-mediated searches for systematic reviews as well as medical and health care practitioners who are searching for evidence to answer clinical questions. The described method can be used to create complex and comprehensive search strategies for different databases and interfaces, such as those that are needed when searching for relevant references for systematic reviews, and will assist both information specialists and practitioners when they are searching the biomedical literature.

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

BACKGROUND

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence. Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

METHOD

A literature review. Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through 'pearl growing', citation chasing, a search of PubMed using the systematic review methods filter, and the authors' topic knowledge. The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of 'key stages' in the process of literature searching.

RESULTS

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

CONCLUSIONS

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review

BACKGROUND

The PRISMA Statement is a reporting guideline designed to improve transparency of systematic reviews (SRs) and meta-analyses. Seven extensions to the PRISMA Statement have been published to address the reporting of different types or aspects of SRs, and another eight are in development. We performed a scoping review to map the research that has been conducted to evaluate the uptake and impact of the PRISMA Statement and extensions. We also synthesised studies evaluating how well SRs published after the PRISMA Statement was disseminated adhere to its recommendations.

METHODS

We searched for meta-research studies indexed in MEDLINE® from inception to 31 July 2017, which investigated some component of the PRISMA Statement or extensions (e.g. SR adherence to PRISMA, journal endorsement of PRISMA). One author screened all records and classified the types of evidence available in the studies. We pooled data on SR adherence to individual PRISMA items across all SRs in the included studies and across SRs published after 2009 (the year PRISMA was disseminated).

RESULTS

We included 100 meta-research studies. The most common type of evidence available was data on SR adherence to the PRISMA Statement, which has been evaluated in 57 studies that have assessed 6487 SRs. The pooled results of these studies suggest that reporting of many items in the PRISMA Statement is suboptimal, even in the 2382 SRs published after 2009 (where nine items were adhered to by fewer than 67% of SRs). Few meta-research studies have evaluated the adherence of SRs to the PRISMA extensions or strategies to increase adherence to the PRISMA Statement and extensions.

CONCLUSIONS

Many studies have evaluated how well SRs adhere to the PRISMA Statement, and the pooled result of these suggest that reporting of many items is suboptimal. An update of the PRISMA Statement, along with a toolkit of strategies to help journals endorse and implement the updated guideline, may improve the transparency of SRs.

Receptive and expressive English language assessments used for young children: a scoping review protocol

BACKGROUND

The majority of a child's language development occurs in the first 5 years of life when brain development is most rapid. There are significant long-term benefits to supporting all children's language and literacy development such as maximizing their developmental potential (i.e., cognitive, linguistic, social-emotional), when children are experiencing a critical period of development (i.e., early childhood to 9 years of age). A variety of people play a significant role in supporting children's language development, including parents, guardians, family members, educators, and/or speech-language pathologists. Speech-language pathologists and educators are the professionals who predominantly support children's language development in order for them to become effective communicators and lay the foundation for later developing literacy skills (i.e., reading and writing skills). Therefore, these professionals need formal and informal assessments that provide them information on a child's understanding and/or use of the increasingly complex aspects of language in order to identify and support the receptive and expressive language learning needs of diverse children during their early learning experiences (i.e., aged 1.5 to 9 years). However, evidence on what methods and tools are being used is lacking.

METHODS

The authors will carry out a scoping review of the literature to identify studies and map the receptive and expressive English language assessment methods and tools that have been published and used since 1980. Arksey and O'Malley's (2005) six-stage approach to conducting a scoping review was drawn upon to design the protocol for this investigation: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consultation.

DISCUSSION

This information will help these professionals identify and select appropriate assessment methods or tools that can be used to support development and/or identify areas of delay or difficulty and plan, implement, and monitor the progress of interventions supporting the development of receptive and expressive language skills in individuals with diverse language needs (e.g., typically developing children, children with language delays and disorders, children learning English as a second or additional language, Indigenous children who may be speaking dialects of English). Researchers plan to evaluate the effectiveness of the assessment methods or tools identified in the scoping review as an extension of this study.

VP200 Untangling What Information Specialists Should Document and Report

INTRODUCTION:

Thorough documentation and clear reporting are essential when conducting a comprehensive literature search for a health technology assessment (HTA) or systematic review. The ultimate goal of this process is transparency and reproducibility with the added benefit of increasing the reader's confidence in the research. Thorough documentation of the search also allows for critical appraisal of the methodology used and facilitates future updating of a review (1,2).

It has been found that large numbers of systematic review searches are inadequately documented and there is little consensus on best practices for reporting standards (3).

As part of the SuRe Info Project, we conducted a review of all current reporting standards relevant to HTAs and systematic reviews in addition to looking at the published literature on this topic in order to synthesize the evidence in this area and create a standard set of agreed upon recommendations.

METHODS:

We conducted a comprehensive search of Medline, Embase, and LISA (Library & Info Studies Abstracts) databases. We also examined the Equator Network (http://www.equator-network.org/) website. Reference lists of included studies and reporting guidelines were also consulted. Eleven reporting guidelines and eight studies were included in the review by two independent reviewers. Anything published before 2006, that was not a research article (other than the guidelines), and/or that did not provide new recommendations (that is, a review of another set of recommendations) was excluded.

RESULTS:

After collecting data on the suggested reporting elements described in the literature, we pooled our results to create an overarching list of the most commonly recommended elements to describe and the most commonly recommended methods to use when documenting a comprehensive search. Not only did these elements pertain to documenting the search strategy for the final report, but they also pertained to the protocol and the abstract of a review.

CONCLUSIONS:

It is hoped that this overview of the literature and compilation of the evidence will clarify some of the confusion that seems to exist when documenting and reporting searches and perhaps it will even help to reduce the existence of poorly described strategies in the research literature.

Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic?

Websites and online resources outside academic bibliographic databases can be significant sources for identifying literature, though there are challenges in searching and managing the results. These are pertinent to systematic reviews that are underpinned by principles of transparency, accountability and reproducibility. We consider how the conduct of searching these resources can be compatible with the principles of a systematic search. We present an approach to address some of the challenges. This is particularly relevant when websites are relied upon to identify important literature for a review. We recommend considering the process as three stages and having a considered rationale and sufficient recordkeeping at each stage that balances transparency with practicality of purpose. Advances in technology and recommendations for website providers are briefly discussed.

Effectiveness of interventions for adult peripheral intravenous catheterization: A systematic review and meta-analysis of randomized controlled trials

BACKGROUND

Peripheral intravenous catheterization (PIVC) is commonly performed on emergency departments and inpatient units. Unsuccessful PIVC first attempts increase pain, and lead to treatment and diagnostic delays.

OBJECTIVE

To determine strategies associated with PIVC first attempt success in adult emergency department patients and inpatients.

METHODS

We searched MEDLINE, EMBASE, CINAHL, TRIP, Cochrane Central Register of Controlled Trials (OVID), and grey literatures databases such as Proquest Dissertation and Theses Global, and Open Grey databases between November and December, 2014. The search was updated on January 28, 2016. We included full text reports of randomized controlled trials testing PIVC interventions versus standard of care. Risk of bias was assessed using the Cochrane Collaboration's tool.

RESULTS

We included 14 randomized controlled trials involving 3201 participants. Interventions included the AccuVein™, AccuCath™ catheter system, ultrasound, safety catheters, and topical anesthetics. Three studies compared AutoGuard and Insyte catheters and were suitable for meta-analysis. There was no difference in first attempt success with a relative risk of 0.0 (95% CI, -0.04, 0.04). There was limited evidence to support the use of ultrasound to increase first attempt success.

CONCLUSIONS

Well-designed and reported randomized controlled trials examining the effectiveness of ultrasound compared to standard of care are warranted.

REGISTRATION

PROSPERO registration: CRD42014015428.

Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar

AIM

To examine how effectively forwards citation searching with Web of Science (WOS) or Google Scholar (GS) identified evidence to support public health guidance published by the National Institute for Health and Care Excellence.

METHOD

Forwards citation searching was performed using GS on a base set of 46 publications and replicated using WOS.

OUTCOMES

WOS and GS were compared in terms of recall; precision; number needed to read (NNR); administrative time and costs; and screening time and costs. Outcomes for all publications were compared with those for a subset of highly important publications.

RESULTS

The searches identified 43 relevant publications. The WOS process had 86.05% recall and 1.58% precision. The GS process had 90.7% recall and 1.62% precision. The NNR to identify one relevant publication was 63.3 with WOS and 61.72 with GS. There were nine highly important publications. WOS had 100% recall, 0.38% precision and NNR of 260.22. GS had 88.89% recall, 0.33% precision and NNR of 300.88. Administering the WOS results took 4 h and cost £88-£136, compared with 75 h and £1650-£2550 with GS.

CONCLUSION

WOS is recommended over GS, as citation searching was more effective, while the administrative and screening times and costs were lower.