Implications of applying methodological shortcuts to expedite systematic reviews: three case studies using systematic reviews from agri-food public health

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Effectiveness of Online Food-Safety Educational Programs: A Systematic Review, Random-Effects Meta-Analysis, and Thematic Synthesis

Online food-safety educational programs are increasingly important to educate different populations as technology and culture shift to using more technology. However, the broad effectiveness of these programs has yet to be examined. A systematic review, random-effects meta-analysis, and thematic synthesis are conducted to identify the effect size of online food-safety educational programs on knowledge, attitudes, and practices of consumers, food workers, and students and their respective barriers and recommendations. Online food-safety education was found to be of moderate and low effectiveness, with attitudes being the lowest in all populations. Consumers struggled with staying focused, and it was found that messaging should focus on risk communication. Students struggled with social isolation and a lack of time, and it was recommended that videos be used. Food workers struggled with a lack of time for training and difficulty understanding the material, and future programs are recommended to implement shorter but more frequent trainings with simple language. Future online food-safety educational programs should focus on incorporating social elements, as they can remain a huge barrier to learning. They should also focus on changing the participant's attitude to risk perception and beliefs in the importance of food safety.

What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies

Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.

Rapid reviews methods series: Guidance on team considerations, study selection, data extraction and risk of bias assessment

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group (RRMG). Rapid reviews (RRs) use modified systematic review (SR) methods to accelerate the review process while maintaining systematic, transparent and reproducible methods to ensure integrity. This paper addresses considerations around the acceleration of study selection, data extraction and risk of bias (RoB) assessment in RRs. If a RR is being undertaken, review teams should consider using one or more of the following methodological shortcuts: screen a proportion (eg, 20%) of records dually at the title/abstract level until sufficient reviewer agreement is achieved, then proceed with single-reviewer screening; use the same approach for full-text screening; conduct single-data extraction only on the most relevant data points and conduct single-RoB assessment on the most important outcomes, with a second person verifying the data extraction and RoB assessment for completeness and correctness. Where available, extract data and RoB assessments from an existing SR that meets the eligibility criteria.

Increasing the efficiency of study selection for systematic reviews using prioritization tools and a single-screening approach

BACKGROUND

Systematic literature screening is a key component in systematic reviews. However, this approach is resource intensive as generally two persons independently of each other (double screening) screen a vast number of search results. To develop approaches for increasing efficiency, we tested the use of text mining to prioritize search results as well as the involvement of only one person (single screening) in the study selection process.

METHOD

Our study is based on health technology assessments (HTAs) of drug and non-drug interventions. Using a sample size calculation, we consecutively included 11 searches resulting in 33 study selection processes. Of the three screeners for each search, two used screening tools with prioritization (Rayyan, EPPI Reviewer) and one a tool without prioritization. For each prioritization tool, we investigated the proportion of citations classified as relevant at three cut-offs or STOP criteria (after screening 25%, 50% and 75% of the citation set). For each STOP criterion, we measured sensitivity (number of correctly identified relevant studies divided by the total number of relevant studies in the study pool). In addition, we determined the number of relevant studies identified per single screening round and investigated whether missed studies were relevant to the HTA conclusion.

RESULTS

Overall, EPPI Reviewer performed better than Rayyan and identified the vast majority (88%, Rayyan 66%) of relevant citations after screening half of the citation set. As long as additional information sources were screened, it was sufficient to apply a single-screening approach to identify all studies relevant to the HTA conclusion. Although many relevant publications (n = 63) and studies (n = 29) were incorrectly excluded, ultimately only 5 studies could not be identified at all in 2 of the 11 searches (1x 1 study, 1x 4 studies). However, their omission did not change the overall conclusion in any HTA.

CONCLUSIONS

EPPI Reviewer helped to identify relevant citations earlier in the screening process than Rayyan. Single screening would have been sufficient to identify all studies relevant to the HTA conclusion. However, this requires screening of further information sources. It also needs to be considered that the credibility of an HTA may be questioned if studies are missing, even if they are not relevant to the HTA conclusion.

Efficient searching for NICE public health guidelines: Would using fewer sources still find the evidence?

Systematic searches are integral to identifying the evidence that is used in National Institute for Health and Care Excellence (NICE) public health guidelines (PHGs). This study analyses the sources, including bibliographic databases and other techniques, required for PHGs. The aims were to analyse the sources used to identify the publications included in NICE PHGs; and to assess whether fewer sources could have been searched to retrieve these publications. Data showing how the included publications had been identified was collated using search summary tables. Three scenarios were created to test various combinations of sources to determine whether fewer sources could have been used. The sample included 29 evidence reviews, compiled using 13 searches, to support 10 PHG topics. Across the PHGs, 23 databases and six other techniques retrieved included publications. A mean reduction in total results of 6.5% could have been made if the minimum set of sources plus Cochrane Library, Embase, and MEDLINE were searched. On average, Cochrane Library, Embase, and MEDLINE contributed 76.8% of the included publications, with other databases adding 11% and other techniques 12.2%. None of the searches had a minimum set that was comprised entirely of databases. There was not a core set of sources for PHGs. A range of databases and techniques, covering a multi-disciplinary evidence base, was required to identify all included publications. It would be possible to reduce the number of sources searched and make some gains in productivity. It is important to create a tailored set of sources to do an efficient search.

Comparison of Weightlifting, Traditional Resistance Training and Plyometrics on Strength, Power and Speed: A Systematic Review with Meta-Analysis

Background

Weightlifting training (WLT) is commonly used to improve strength, power and speed in athletes. However, to date, WLT studies have either not compared training effects against those of other training methods, or been limited by small sample sizes, which are issues that can be resolved by pooling studies in a meta-analysis. Therefore, the objective of this systematic review with meta-analysis was to evaluate the effects of WLT compared with traditional resistance training (TRT), plyometric training (PLYO) and/or control (CON) on strength, power and speed.

Methods

The systematic review included peer-reviewed articles that employed a WLT intervention, a comparison group (i.e. TRT, PLYO, CON), and a measure of strength, power and/or speed. Means and standard deviations of outcomes were converted to Hedges’ g effect sizes using an inverse variance random-effects model to generate a weighted mean effect size (ES).

Results

Sixteen studies were included in the analysis, comprising 427 participants. Data indicated that when compared with TRT, WLT resulted in greater improvements in weightlifting load lifted (4 studies, p = 0.02, g = 1.35; 95% CI 0.20–2.51) and countermovement jump (CMJ) height (9 studies, p = 0.00, g = 0.95; 95% CI 0.04–1.87). There was also a large effect in terms of linear sprint speed (4 studies, p = 0.13, g = 1.04; 95% CI − 0.03 to 2.39) and change of direction speed (CODS) (2 studies, p = 0.36, g = 1.21; 95% CI − 1.41 to 3.83); however, this was not significant. Interpretation of these findings should acknowledge the high heterogeneity across the included studies and potential risk of bias. WLT and PLYO resulted in similar improvements in speed, power and strength as demonstrated by negligible to moderate, non-significant effects in favour of WLT for improvements in linear sprint speed (4 studies, p = 0.35, g = 0.20; 95% CI − 0.23 to 0.63), CODS (3 studies, p = 0.52, g = 0.17; 95% CI − 0.35 to 0.68), CMJ (6 studies, p = 0.09, g = 0.31; 95% CI − 0.05 to 0.67), squat jump performance (5 studies, p = 0.08, g = 0.34; 95% CI − 0.04 to 0.73) and strength (4 studies, p = 0.20, g = 0.69; 95% CI − 0.37 to 1.75).

Conclusion

Overall, these findings support the notion that if the training goal is to improve strength, power and speed, supplementary weightlifting training may be advantageous for athletic development. Whilst WLT and PLYO may result in similar improvements, WLT can elicit additional benefits above that of TRT, resulting in greater improvements in weightlifting and jumping performance.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40279-021-01627-2.

Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses

BACKGROUND

Systematic reviews are the cornerstone of evidence-based medicine. However, systematic reviews are time consuming and there is growing demand to produce evidence more quickly, while maintaining robust methods. In recent years, artificial intelligence and active-machine learning (AML) have been implemented into several SR software applications. As some of the barriers to adoption of new technologies are the challenges in set-up and how best to use these technologies, we have provided different situations and considerations for knowledge synthesis teams to consider when using artificial intelligence and AML for title and abstract screening.

METHODS

We retrospectively evaluated the implementation and performance of AML across a set of ten historically completed systematic reviews. Based upon the findings from this work and in consideration of the barriers we have encountered and navigated during the past 24 months in using these tools prospectively in our research, we discussed and developed a series of practical recommendations for research teams to consider in seeking to implement AML tools for citation screening into their workflow.

RESULTS

We developed a seven-step framework and provide guidance for when and how to integrate artificial intelligence and AML into the title and abstract screening process. Steps include: (1) Consulting with Knowledge user/Expert Panel; (2) Developing the search strategy; (3) Preparing your review team; (4) Preparing your database; (5) Building the initial training set; (6) Ongoing screening; and (7) Truncating screening. During Step 6 and/or 7, you may also choose to optimize your team, by shifting some members to other review stages (e.g., full-text screening, data extraction).

CONCLUSION

Artificial intelligence and, more specifically, AML are well-developed tools for title and abstract screening and can be integrated into the screening process in several ways. Regardless of the method chosen, transparent reporting of these methods is critical for future studies evaluating artificial intelligence and AML.

Use of Race, Ethnicity, and National Origin in Studies Assessing Cardiovascular Risk in Women With a History of Hypertensive Disorders of Pregnancy

Women with a history of hyperBtensive disorders of pregnancy (HDP) are at particularly high risk for cardiovascular disease (CVD) and CVD-related death, and certain racial and ethnic subpopulations are disproportionately affected by these conditions. We examined the use of race, ethnicity, and national origin in observational studies assessing CVD morbidity and mortality in women with a history of HDP. A total of 124 studies, published between 1976 and 2021, were reviewed. We found that white women were heavily overrepresented, encompassing 53% of all participants with HDP. There was limited and heterogeneous reporting of race and ethnicity information across studies and only 27 studies reported including race and/or ethnicity variables in at least 1 statistical analysis. Only 2 studies mentioned the use of these variables as a strength; several others (k = 18) reported a lack of diversity among participants as a study limitation. Just over half of included articles (k = 68) reported at least 1 sociodemographic variable other than race and ethnicity (eg, marital status and income); however, none investigated how they might have worked synergistically or antagonistically with race and/or ethnicity to influence participants' risk of CVD. These findings highlight significant areas for improvement in cardiovascular obstetrics research, including the need for more robust and standardized methods for collecting, reporting, and using sociodemographic information. Future studies of CVD risk in women with a history of HDP should explicitly examine racial and ethnic differences and use an intersectional approach.

Literature searching methods or guidance and their application to public health topics: A narrative review

Background

Information specialists conducting searches for systematic reviews need to consider key questions around which and how many sources to search. This is particularly important for public health topics where evidence may be found in diverse sources.

Objectives

The objective of this review is to give an overview of recent studies on information retrieval guidance and methods that could be applied to public health evidence and used to guide future searches.

Methods

A literature search was performed in core databases and supplemented by browsing health information journals and citation searching. Results were sifted and reviewed.

Results

Seventy‐two papers were found and grouped into themes covering sources and search techniques. Public health topics were poorly covered in this literature.

Discussion

Many researchers follow the recommendations to search multiple databases. The review topic influences decisions about sources. Additional sources covering grey literature eliminate bias but are time‐consuming and difficult to search systematically. Public health searching is complex, often requiring searches in multidisciplinary sources and using additional methods.

Conclusions

Search planning is advisable to enable decisions about which and how many sources to search. This could improve with more work on modelling search scenarios, particularly in public health topics, to examine where publications were found and guide future research.

The quality of systematic reviews and other synthesis in the time of COVID-19

COVID-19 research has been produced at an unprecedented rate and managing what is currently known is in part being accomplished through synthesis research. Here we evaluated how the need to rapidly produce syntheses has impacted the quality of the synthesis research. Thus, we sought to identify, evaluate and map the synthesis research on COVID-19 published up to 10 July 2020. A COVID-19 literature database was created using pre-specified COVID-19 search algorithms carried out in eight databases. We identified 863 citations considered to be synthesis research for evaluation in this project. Four-hundred and thirty-nine reviews were fully assessed with A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2) and rated as very low-quality (n = 145), low-quality (n = 80), medium-quality (n = 208) and high-quality (n = 151). The quality of these reviews fell short of what is expected for synthesis research with key domains being left out of the typical methodology. The increase in risk of bias due to non-adherence to systematic review methodology is unknown and prevents the reader from assessing the validity of the review. The responsibility to assure the quality is held by both producers and publishers of synthesis research and our findings indicate there is a need to equip readers with the expertise to evaluate the review conduct before using it for decision-making purposes.

Evaluating the relationship between citation set size, team size and screening methods used in systematic reviews: a cross-sectional study

Background

Standard practice for conducting systematic reviews (SRs) is time consuming and involves the study team screening hundreds or thousands of citations. As the volume of medical literature grows, the citation set sizes and corresponding screening efforts increase. While larger team size and alternate screening methods have the potential to reduce workload and decrease SR completion times, it is unknown whether investigators adapt team size or methods in response to citation set sizes. Using a cross-sectional design, we sought to understand how citation set size impacts (1) the total number of authors or individuals contributing to screening and (2) screening methods.

Methods

MEDLINE was searched in April 2019 for SRs on any health topic. A total of 1880 unique publications were identified and sorted into five citation set size categories (after deduplication): < 1,000, 1,001–2,500, 2,501–5,000, 5,001–10,000, and > 10,000. A random sample of 259 SRs were selected (~ 50 per category) for data extraction and analysis.

Results

With the exception of the pairwise t test comparing the under 1000 and over 10,000 categories (median 5 vs. 6, p = 0.049) no statistically significant relationship was evident between author number and citation set size. While visual inspection was suggestive, statistical testing did not consistently identify a relationship between citation set size and number of screeners (title-abstract, full text) or data extractors. However, logistic regression identified investigators were significantly more likely to deviate from gold-standard screening methods (i.e. independent duplicate screening) with larger citation sets. For every doubling of citation size, the odds of using gold-standard screening decreased by 15 and 20% at title-abstract and full text review, respectively. Finally, few SRs reported using crowdsourcing (n = 2) or computer-assisted screening (n = 1).

Conclusions

Large citation set sizes present a challenge to SR teams, especially when faced with time-sensitive health policy questions. Our study suggests that with increasing citation set size, authors are less likely to adhere to gold-standard screening methods. It is possible that adjunct screening methods, such as crowdsourcing (large team) and computer-assisted technologies, may provide a viable solution for authors to complete their SRs in a timely manner.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01335-5.

An evaluation of DistillerSR's machine learning-based prioritization tool for title/abstract screening - impact on reviewer-relevant outcomes

BACKGROUND

Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings.

METHODS

Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented.

RESULTS

Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%).

CONCLUSIONS

The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

Combining abbreviated literature searches with single-reviewer screening: three case studies of rapid reviews

BACKGROUND

Decision-makers increasingly request rapid answers to clinical or public health questions. To save time, personnel, and financial resources, rapid reviews streamline the methodological steps of the systematic review process. We aimed to explore the validity of a rapid review approach that combines a substantially abbreviated literature search with a single-reviewer screening of abstracts and full texts using three case studies.

METHODS

We used a convenience sample of three ongoing Cochrane reviews as reference standards. Two reviews addressed oncological topics and one addressed a public health topic. For each of the three topics, three reviewers screened the literature independently. Our primary outcome was the change in conclusions between the rapid reviews and the respective Cochrane reviews. In case the rapid approach missed studies, we recalculated the meta-analyses for the main outcomes and asked Cochrane review authors if the new body of evidence would change their original conclusion compared with the reference standards. Additionally, we assessed the sensitivity of the rapid review approach compared with the results of the original Cochrane reviews.

RESULTS

For the two oncological topics (case studies 1 and 2), the three rapid reviews each yielded the same conclusions as the Cochrane reviews. However, the authors would have had less certainty about their conclusion in case study 2. For case study 3, the public health topic, only one of the three rapid reviews led to the same conclusion as the Cochrane review. The other two rapid reviews provided insufficient information for the authors to draw conclusions. Using the rapid review approach, the sensitivity was 100% (3 of 3) for case study 1. For case study 2, the three rapid reviews identified 40% (4 of 10), 50% (5 of 10), and 60% (6 of 10) of the included studies, respectively; for case study 3, the respective numbers were 38% (8 of 21), 43% (9 of 21), and 48% (10 of 21).

CONCLUSIONS

Within the limitations of these case studies, a rapid review approach that combines abbreviated literature searches with single-reviewer screening may be feasible for focused clinical questions. For complex public health topics, sensitivity seems to be insufficient.

"It has no meaning to me." How do researchers understand the effectiveness of literature searches? A qualitative analysis and preliminary typology of understandings

This study aimed to address the question: what does "effectiveness" mean to researchers in the context of literature searching for systematic reviews? We conducted a thematic analysis of responses to an e-mail survey. Eighty-nine study authors, whose studies met inclusion in a recent review (2018), were contacted via e-mail and asked three questions; one directly asking the question: in literature searching, what does effective (or effectiveness in) literature searching mean to you? Thirty-eight (46%) responses were received from diverse professional groups, including: literature searchers, systematic reviewers, clinicians and researchers. A shared understanding of what effectiveness means was not identified. Instead, five themes were developed from data: (a) effectiveness is described as a metric; (b) effectiveness is a balance between metrics; (c) effectiveness can be categorized by search purpose; (d) effectiveness is an outcome; and, (e) effectiveness is an experimental concept. We propose that these themes constitute a preliminary typology of understandings. No single definition of effectiveness was identified. The proposed typology suggests that different researchers have differing understandings of effectiveness. This could lead to uncertainty as to the aim and the purpose of literature searches and confusion about the outcomes. The typology offers a potential route for further exploration.

Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

OBJECTIVES

To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews.

STUDY DESIGN AND SETTING

We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single- and dual-reviewer screening using two published systematic reviews as reference standards.

RESULTS

Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%-91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%-98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%-80.9%) and 68.7% (95% CI, 66.4%-71.0%), respectively.

CONCLUSIONS

Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.

Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review

BACKGROUND

Stringent requirements exist regarding the transparency of the study selection process and the reliability of results. A 2-step selection process is generally recommended; this is conducted by 2 reviewers independently of each other (conventional double-screening). However, the approach is resource intensive, which can be a problem, as systematic reviews generally need to be completed within a defined period with a limited budget. The aim of the following methodological systematic review was to analyse the evidence available on whether single screening is equivalent to double screening in the screening process conducted in systematic reviews.

METHODS

We searched Medline, PubMed and the Cochrane Methodology Register (last search 10/2018). We also used supplementary search techniques and sources ("similar articles" function in PubMed, conference abstracts and reference lists). We included all evaluations comparing single with double screening. Data were summarized in a structured, narrative way.

RESULTS

The 4 evaluations included investigated a total of 23 single screenings (12 sets for screening involving 9 reviewers). The median proportion of missed studies was 5% (range 0 to 58%). The median proportion of missed studies was 3% for the 6 experienced reviewers (range: 0 to 21%) and 13% for the 3 reviewers with less experience (range: 0 to 58%). The impact of missing studies on the findings of meta-analyses had been reported in 2 evaluations for 7 single screenings including a total of 18,148 references. In 3 of these 7 single screenings - all conducted by the same reviewer (with less experience) - the findings would have changed substantially. The remaining 4 of these 7 screenings were conducted by experienced reviewers and the missing studies had no impact or a negligible on the findings of the meta-analyses.

CONCLUSIONS

Single screening of the titles and abstracts of studies retrieved in bibliographic searches is not equivalent to double screening, as substantially more studies are missed. However, in our opinion such an approach could still represent an appropriate methodological shortcut in rapid reviews, as long as it is conducted by an experienced reviewer. Further research on single screening is required, for instance, regarding factors influencing the number of studies missed.

Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews

BACKGROUND

Citation screening for scoping searches and rapid review is time-consuming and inefficient, often requiring days or sometimes months to complete. We examined the reliability of PICo-based title-only screening using keyword searches based on the PICo elements-Participants, Interventions, and Comparators, but not the Outcomes.

METHODS

A convenience sample of 10 datasets, derived from the literature searches of completed systematic reviews, was used to test PICo-based title-only screening. Search terms for screening were generated from the inclusion criteria of each review, specifically the PICo elements-Participants, Interventions and Comparators. Synonyms for the PICo terms were sought, including alternatives for clinical conditions, trade names of generic drugs and abbreviations for clinical conditions, interventions and comparators. The MeSH database, Wikipedia, Google searches and online thesauri were used to assist generating terms. Title-only screening was performed by five reviewers independently in Endnote X7 reference management software using OR Boolean operator. Outcome measures were recall of included studies and the reduction in screening effort. Recall is the proportion of included studies retrieved using PICo title-only screening out of the total number of included studies in the original reviews. The percentage reduction in screening effort is the proportion of records not needing screening because the method eliminates them from the screen set.

RESULTS

Across the 10 reviews, the reduction in screening effort ranged from 11 to 78% with a median reduction of 53%. In nine systematic reviews, the recall of included studies was 100%. In one review (oxygen therapy), four of five reviewers missed the same included study (median recall 67%). A post hoc analysis was performed on the dataset with the lowest reduction in screening effort (11%), and it was rescreened using only the intervention and comparator keywords and omitting keywords for participants. The reduction in screening effort increased to 57%, and the recall of included studies was maintained (100%).

CONCLUSIONS

In this sample of datasets, PICo-based title-only screening was able to expedite citation screening for scoping searches and rapid reviews by reducing the number of citations needed to screen but requires a thorough workup of the potential synonyms and alternative terms. Further research which evaluates the feasibility of this technique with heterogeneous datasets in different fields would be useful to inform the generalisability of this technique.

Expediting evidence synthesis for healthcare decision-making: exploring attitudes and perceptions towards rapid reviews using Q methodology

Background

Rapid reviews expedite the knowledge synthesis process with the goal of providing timely information to healthcare decision-makers who want to use evidence-informed policy and practice approaches. A range of opinions and viewpoints on rapid reviews is thought to exist; however, no research to date has formally captured these views. This paper aims to explore evidence producer and knowledge user attitudes and perceptions towards rapid reviews.

Methods

A Q methodology study was conducted to identify central viewpoints about rapid reviews based on a broad topic discourse. Participants rank-ordered 50 text statements and explained their Q-sort in free-text comments. Individual Q-sorts were analysed using Q-Assessor (statistical method: factor analysis with varimax rotation). Factors, or salient viewpoints on rapid reviews, were identified, interpreted and described.

Results

Analysis of the 11 individual Q sorts identified three prominent viewpoints: Factor A cautions against the use of study design labels to make judgements. Factor B maintains that rapid reviews should be the exception and not the rule. Factor C focuses on the practical needs of the end-user over the review process.

Conclusion

Results show that there are opposing viewpoints on rapid reviews, yet some unity exists. The three factors described offer insight into how and why various stakeholders act as they do and what issues may need to be resolved before increase uptake of the evidence from rapid reviews can be realized in healthcare decision-making environments.

Implications of applying methodological shortcuts to expedite systematic reviews: three case studies using systematic reviews from agri-food public health

Background

The rapid review is an approach to synthesizing research evidence when a shorter timeframe is required. The implications of what is lost in terms of rigour, increased bias and accuracy when conducting a rapid review have not yet been elucidated.

Methods

We assessed the potential implications of methodological shortcuts on the outcomes of three completed systematic reviews addressing agri‐food public health topics. For each review, shortcuts were applied individually to assess the impact on the number of relevant studies included and whether omitted studies affected the direction, magnitude or precision of summary estimates from meta‐analyses.

Results

In most instances, the shortcuts resulted in at least one relevant study being omitted from the review. The omission of studies affected 39 of 143 possible meta‐analyses, of which 14 were no longer possible because of insufficient studies (<2). When meta‐analysis was possible, the omission of studies generally resulted in less precise pooled estimates (i.e. wider confidence intervals) that did not differ in direction from the original estimate.

Conclusions

The three case studies demonstrated the risk of missing relevant literature and its impact on summary estimates when methodological shortcuts are applied in rapid reviews. © 2016 The Authors. Research Synthesis Methods Published by John Wiley & Sons Ltd.