Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Game-based learning in early childhood education: a systematic review and meta-analysis

Game-based learning has gained popularity in recent years as a tool for enhancing learning outcomes in children. This approach uses games to teach various subjects and skills, promoting engagement, motivation, and fun. In early childhood education, game-based learning has the potential to promote cognitive, social, and emotional development. This systematic review and meta-analysis aim to summarize the existing literature on the effectiveness of game-based learning in early childhood education This systematic review and meta-analysis examine the effectiveness of game-based learning in early childhood education. The results show that game-based learning has a moderate to large effect on cognitive, social, emotional, motivation, and engagement outcomes. The findings suggest that game-based learning can be a promising tool for early childhood educators to promote children's learning and development. However, further research is needed to address the remaining gaps in the literature. The study's findings have implications for educators, policymakers, and game developers who aim to promote positive child development and enhance learning outcomes in early childhood education.

What are the best methods for rapid reviews of the research evidence? A systematic review of reviews and primary studies

Rapid review methodology aims to facilitate faster conduct of systematic reviews to meet the needs of the decision-maker, while also maintaining quality and credibility. This systematic review aimed to determine the impact of different methodological shortcuts for undertaking rapid reviews on the risk of bias (RoB) of the results of the review. Review stages for which reviews and primary studies were sought included the preparation of a protocol, question formulation, inclusion criteria, searching, selection, data extraction, RoB assessment, synthesis, and reporting. We searched 11 electronic databases in April 2022, and conducted some supplementary searching. Reviewers worked in pairs to screen, select, extract data, and assess the RoB of included reviews and studies. We included 15 systematic reviews, 7 scoping reviews, and 65 primary studies. We found that several commonly used shortcuts in rapid reviews are likely to increase the RoB in the results. These include restrictions based on publication date, use of a single electronic database as a source of studies, and use of a single reviewer for screening titles and abstracts, selecting studies based on the full-text, and for extracting data. Authors of rapid reviews should be transparent in reporting their use of these shortcuts and acknowledge the possibility of them causing bias in the results. This review also highlights shortcuts that can save time without increasing the risk of bias. Further research is needed for both systematic and rapid reviews on faster methods for accurate data extraction and RoB assessment, and on development of more precise search strategies.

A Systematic Approach to Evidence-Based Design for Measurable Health and Wellness Outcomes: Curating and Translating Evidence to Practice

OBJECTIVES

Evaluating evidence from peer-review literature for use in evidence-based design is often challenging for the design disciplines, requiring access to the peer-reviewed literature, expertise in evaluating methods and findings, and translating the results into actionable design and operational recommendations.

PURPOSE

The purpose of this methods paper is to elucidate the process for systematic evaluation of research to translate evidence into practical application to improve design for occupant health and wellness.

BACKGROUND

Researchers have found strong connections in environmental design influence on health and wellness that have proven to be substantiative in the effort to improve health and well-being. Design has the capacity to encourage healthy choices and decisions within the built environment. Translation of evidence into applied design solutions may improve public health.

METHODS

A protocol is presented that culminates in the translation of evidence into design recommendations focused on improving occupant health. The protocol includes preparation for the literature search and review, search strategy, study selection, data analysis, and development of the literature review.

RESULTS

After evaluation of the evidence is completed, there were several positive findings in the example that stakeholders could utilize to improve the health of building occupants with programs and design to support nutrition, physical activity, and circadian entrainment.

CONCLUSIONS

There are a variety of software tools and processes to utilize in the curation of evidence to improve the built environment with relevant design recommendations and operational considerations affecting the personal, social, and economic health of our society.

Rapid reviews methods series: Guidance on team considerations, study selection, data extraction and risk of bias assessment

This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group (RRMG). Rapid reviews (RRs) use modified systematic review (SR) methods to accelerate the review process while maintaining systematic, transparent and reproducible methods to ensure integrity. This paper addresses considerations around the acceleration of study selection, data extraction and risk of bias (RoB) assessment in RRs. If a RR is being undertaken, review teams should consider using one or more of the following methodological shortcuts: screen a proportion (eg, 20%) of records dually at the title/abstract level until sufficient reviewer agreement is achieved, then proceed with single-reviewer screening; use the same approach for full-text screening; conduct single-data extraction only on the most relevant data points and conduct single-RoB assessment on the most important outcomes, with a second person verifying the data extraction and RoB assessment for completeness and correctness. Where available, extract data and RoB assessments from an existing SR that meets the eligibility criteria.

Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey

OBJECTIVE

Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs.

STUDY DESIGN AND SETTING

We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis.

RESULTS

162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%).

CONCLUSION

Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.

Recommendations for the extraction, analysis, and presentation of results in scoping reviews

Scoping reviewers often face challenges in the extraction, analysis, and presentation of scoping review results. Using best-practice examples and drawing on the expertise of the JBI Scoping Review Methodology Group and an editor of a journal that publishes scoping reviews, this paper expands on existing JBI scoping review guidance. The aim of this article is to clarify the process of extracting data from different sources of evidence; discuss what data should be extracted (and what should not); outline how to analyze extracted data, including an explanation of basic qualitative content analysis; and offer suggestions for the presentation of results in scoping reviews.

Paper 2: Performing rapid reviews

BACKGROUND

Health policy-makers must often make decisions in compressed time frames and with limited resources. Hence, rapid reviews have become a pragmatic alternative to comprehensive systematic reviews. However, it is important that rapid review methods remain rigorous to support good policy development and decisions. There is currently little evidence about which streamlined steps in a rapid review are less likely to introduce unacceptable levels of uncertainty while still producing a product that remains useful to policy-makers.

METHODS

This paper summarizes current research describing commonly used methods and practices that are used to conduct rapid reviews and presents key considerations and options to guide methodological choices for a rapid review.

RESULTS

The most important step for a rapid review is for an experienced research team to have early and ongoing engagement with the people who have requested the review. A clear research protocol, derived from a needs assessment conducted with the requester, serves to focus the review, defines the scope of the rapid review, and guides all subsequent steps. Common recommendations for rapid review methods include tailoring the literature search in terms of databases, dates, and languages. Researchers can consider using a staged search to locate high-quality systematic reviews and then subsequently published primary studies. The approaches used for study screening and selection, data extraction, and risk-of-bias assessment should be tailored to the topic, researcher experience, and available resources. Many rapid reviews use a single reviewer for study selection, risk-of-bias assessment, or data abstraction, sometimes with partial or full verification by a second reviewer. Rapid reviews usually use a descriptive synthesis method rather than quantitative meta-analysis. Use of brief report templates and standardized production methods helps to speed final report publication.

CONCLUSIONS

Researchers conducting rapid reviews need to make transparent methodological choices, informed by stakeholder input, to ensure that rapid reviews meet their intended purpose. Transparency is critical because it is unclear how or how much streamlined methods can bias the conclusions of reviews. There are not yet internationally accepted standards for conducting or reporting rapid reviews. Thus, this article proposes interim guidance for researchers who are increasingly employing these methods.

Methodological guidance for rapid reviews in healthcare: a scoping review

OBJECTIVE

The aim of the present work was to identify published methodological guidance for rapid reviews (RRs) and to analyze the recommendations with regard to time-saving measures.

STUDY DESIGN AND SETTING

A literature search was performed in PubMed and EMBASE in November 2020. In addition, a search based on Google Scholar and websites of governmental and non-governmental organizations was conducted. Literature screening was carried out by two researchers independently.

RESULTS

A total of 34 publications were included. These describe 38 distinct RR types. The timeframe to complete the identified RR types ranges from 24 hours to six months (mean time 2.2 months). For most RR types a specific research question (n=21) and a prioritizing search (n=25; preference for e.g. systematic reviews, meta-analyses) is employed. Different approaches such as reduced personnel in literature screening (n=21) and data extraction (n=21) are recommended. The majority of RR types include a bias assessment (n=28) and suggest a narrative report focusing on safety and efficacy.

CONCLUSION

The included RR types are heterogeneous in terms of completion time, considered domains and strategies to alter the standard systematic review methods. A rationale for the recommended shortcuts is rarely presented. This article is protected by copyright. All rights reserved.

An overview of methodological approaches in systematic reviews

AIM

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

METHODS

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full-text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

RESULTS

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non-English, gray, and unpublished literature, and (c) use of text-mining approaches during title and abstract screening.

CONCLUSION

The overview identified limited SR-level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

Anti-vascular endothelial growth factor therapy for age-related macular degeneration: a systematic review and network meta-analysis

BACKGROUND

The comparative safety and efficacy between anti-vascular endothelial growth factor agents (anti-VEGFs) and between combined therapies for patients with neovascular age-related macular degeneration (nAMD) is unclear. We conducted a systematic review to examine the comparative safety and efficacy anti-VEGFs for adults with nAMD.

METHODS

Studies were identified through MEDLINE, EMBASE, and Cochrane CENTRAL (inception to June 3, 2019), grey literature, and scanning reference lists. Two reviewers independently screened citations and full-text articles to identify randomized controlled trials (RCTs), extracted data, and appraised risk of bias. Pairwise random-effects meta-analysis and Bayesian network meta-analysis (NMA) were conducted. The primary outcomes were the proportion of patients experiencing moderate vision gain (≥ 15 letters on the Early Treatment Diabetic Retinopathy Study chart) and the proportion of patients experiencing moderate vision loss (≤ 15 letters).

RESULTS

After screening 3647 citations and 485 potentially relevant full-text articles, 92 RCTs with 24,717 patients were included. NMA (34 RCTs, 8809 patients, 12 treatments) showed small differences among anti-VEGFs in improving the proportion of patients with moderate vision gain, with the largest for conbercept versus broluczumab (OR 0.15, 95% CrI: 0.05-0.56), conbercept versus ranibizumab (OR 0.17, 95% CrI: 0.05-0.59), conbercept versus aflibercept (OR 0.19, 95% CrI: 0.06-0.65), and conbercept versus bevacizumab (OR 0.2, 95% CrI: 0.06-0.69). In NMA (36 RCTs, 9081 patients, 13 treatments) for the proportion of patients with moderate vision loss, small differences were observed among anti-VEGFs, with the largest being for conbercept versus aflibercept (OR 0.24, 95% CrI: 0-4.29), conbercept versus brolucizumab (OR 0.24, 95% CrI: 0-4.71), conbercept versus bevacizumab (OR 0.26, 95% CrI: 0-4.65), and conbercept versus ranibizumab (OR 0.27, 95% CrI: 0-4.67).

CONCLUSION

The only observed differences were that ranibizumab, bevacizumab, aflibercept, and brolucizumab were statistically superior to conbercept in terms of the proportion of patients with nAMD who experienced moderate vision gain. However, this finding is based on indirect evidence through one small trial comparing conbercept with placebo. This does not account for drug-specific differences when assessing anatomic and functional treatment efficacy in variable dosing regimens.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO registration number CRD42015022041.

Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses

BACKGROUND

Systematic reviews are the cornerstone of evidence-based medicine. However, systematic reviews are time consuming and there is growing demand to produce evidence more quickly, while maintaining robust methods. In recent years, artificial intelligence and active-machine learning (AML) have been implemented into several SR software applications. As some of the barriers to adoption of new technologies are the challenges in set-up and how best to use these technologies, we have provided different situations and considerations for knowledge synthesis teams to consider when using artificial intelligence and AML for title and abstract screening.

METHODS

We retrospectively evaluated the implementation and performance of AML across a set of ten historically completed systematic reviews. Based upon the findings from this work and in consideration of the barriers we have encountered and navigated during the past 24 months in using these tools prospectively in our research, we discussed and developed a series of practical recommendations for research teams to consider in seeking to implement AML tools for citation screening into their workflow.

RESULTS

We developed a seven-step framework and provide guidance for when and how to integrate artificial intelligence and AML into the title and abstract screening process. Steps include: (1) Consulting with Knowledge user/Expert Panel; (2) Developing the search strategy; (3) Preparing your review team; (4) Preparing your database; (5) Building the initial training set; (6) Ongoing screening; and (7) Truncating screening. During Step 6 and/or 7, you may also choose to optimize your team, by shifting some members to other review stages (e.g., full-text screening, data extraction).

CONCLUSION

Artificial intelligence and, more specifically, AML are well-developed tools for title and abstract screening and can be integrated into the screening process in several ways. Regardless of the method chosen, transparent reporting of these methods is critical for future studies evaluating artificial intelligence and AML.

The impact of conducting preclinical systematic reviews on researchers and their research: A mixed method case study

BACKGROUND

Systematic reviews (SRs) are cornerstones of evidence-based medicine and have contributed significantly to breakthroughs since the 1980's. However, preclinical SRs remain relatively rare despite their many advantages. Since 2011 the Dutch health funding organisation (ZonMw) has run a grant scheme dedicated to promoting the training, coaching and conduct of preclinical SRs. Our study focuses on this funding scheme to investigate the relevance, effects and benefits of conducting preclinical SRs on researchers and their research.

METHODS

We recruited researchers who attended funded preclinical SR workshops and who conducted, are still conducting, or prematurely stopped a SR with funded coaching. We gathered data using online questionnaires followed by semi-structured interviews. Both aimed to explore the impact of conducting a SR on researchers' subsequent work, attitudes, and views about their research field. Data-analysis was performed using Excel and ATLAS.ti.

RESULTS

Conducting preclinical SRs had two distinct types of impact. First, the researchers acquired new skills and insights, leading to a change in mindset regarding the quality of animal research. This was mainly seen in the way participants planned, conducted and reported their subsequent animal studies, which were more transparent and of a higher quality than their previous work. Second, participants were eager to share their newly acquired knowledge within their laboratories and to advocate for change within their research teams and fields of interest. In particular, they emphasised the need for preclinical SRs and improved experimental design within preclinical research, promoting these through education and published opinion papers.

CONCLUSION

Being trained and coached in the conduct of preclinical SRs appears to be a contributing factor to many beneficial changes which will impact the quality of preclinical research in the long-term. Our findings suggest that this ZonMw funding scheme is helpful in improving the quality and transparency of preclinical research. Similar funding schemes should be encouraged, preferably by a broader group of funders or financers, in the future.

Reporting of methods to prepare, pilot and perform data extraction in systematic reviews: analysis of a sample of 152 Cochrane and non-Cochrane reviews

BACKGROUND

Previous research on data extraction methods in systematic reviews has focused on single aspects of the process. We aimed to provide a deeper insight into these methods by analysing a current sample of reviews.

METHODS

We included systematic reviews of health interventions in humans published in English. We analysed 75 Cochrane reviews from May and June 2020 and a random sample of non-Cochrane reviews published in the same period and retrieved from Medline. We linked reviews with protocols and study registrations. We collected information on preparing, piloting, and performing data extraction and on use of software to assist review conduct (automation tools). Data were extracted by one author, with 20% extracted in duplicate. Data were analysed descriptively.

RESULTS

Of the 152 included reviews, 77 reported use of a standardized extraction form (51%); 42 provided information on the type of form used (28%); 24 on piloting (16%); 58 on what data was collected (38%); 133 on the extraction method (88%); 107 on resolving disagreements (70%); 103 on methods to obtain additional data or information (68%); 52 on procedures to avoid data errors (34%); and 47 on methods to deal with multiple study reports (31%). Items were more frequently reported in Cochrane than non-Cochrane reviews. The data extraction form used was published in 10 reviews (7%). Use of software was rarely reported except for statistical analysis software and use of RevMan and GRADEpro GDT in Cochrane reviews. Covidence was the most frequent automation tool used: 18 reviews used it for study selection (12%) and 9 for data extraction (6%).

CONCLUSIONS

Reporting of data extraction methods in systematic reviews is limited, especially in non-Cochrane reviews. This includes core items of data extraction such as methods used to manage disagreements. Few reviews currently use software to assist data extraction and review conduct. Our results can serve as a baseline to assess the uptake of such tools in future analyses.

Quality appraisal of systematic reviews of interventions for children with cerebral palsy reveals critically low confidence

AIM

To evaluate the methodological quality of recent systematic reviews of interventions for children with cerebral palsy in order to determine the level of confidence in the reviews' conclusions.

METHOD

A comprehensive search of 22 databases identified eligible systematic reviews with and without meta-analysis published worldwide from 2015 to 2019. We independently extracted data and used A Measurement Tool to Assess Systematic Reviews-2 (AMSTAR-2) to appraise methodological quality.

RESULTS

Eighty-three systematic reviews met strict eligibility criteria. Most were from Europe and Latin America and reported on rehabilitative interventions. AMSTAR-2 appraisal found critically low confidence in 88% (n=73) because of multiple and varied deficiencies. Only 7% (n=6) had no AMSTAR-2 critical domain deficiency. The number of systematic reviews increased fivefold from 2015 to 2019; however, quality did not improve over time.

INTERPRETATION

Most of these systematic reviews are considered unreliable according to AMSTAR-2. Current recommendations for treating children with CP based on these flawed systematic reviews need re-evaluation. Findings are comparable to reports from other areas of medicine, despite the general perception that systematic reviews are high-level evidence. The required use of current widely accepted guidance for conducting and reporting systematic reviews by authors, peer reviewers, and editors is critical to ensure reliable, unbiased, and transparent systematic reviews. What this paper adds Confidence was critically low in the conclusions of 88% of systematic reviews about interventions for children with cerebral palsy (CP). Quality issues in the sample were not limited to systematic reviews of non-randomized trials, or to those about certain populations of CP or interventions. The inclusion of meta-analysis did not improve the level of confidence in these systematic reviews. Numbers of systematic reviews on this topic increased over the 5 search years but their methodological quality did not improve.

Effects of Sling Exercise for Neck Pain: A Systematic Review and Meta-Analysis

OBJECTIVE

The purpose of this review was to evaluate the effects of sling exercise on pain intensity, disability, and health-related quality of life in adults with neck pain.

METHODS

The Cochrane Central Register of Controlled Trials, EMBASE, Physiotherapy Evidence Database (PEDro), and 6 other databases were searched from inception to July 2020. The reference lists of relevant articles to identify additional trials were also screened. Randomized controlled trials were included if they investigated the effects of sling suspension therapy in patients with neck pain, including mechanical neck disorders, cervicogenic headache, and neck disorders with radicular findings. Studies were required to be published in English or Chinese. The methodological quality and levels of evidence of studies were assessed using the PEDro scale and the Grading of Recommendations Assessment, Development and Evaluation approach, respectively. The random-effects model was used to perform meta-analyses.

RESULTS

Eleven randomized controlled trials were included (n = 595). The mean total PEDro score was 4.64 (SD = 1.21) of 10, which indicated a fair methodological quality. The intervention groups showed significant improvements in pain intensity (SMD = -1.23; 95% CI = -1.88 to -0.58) immediately postintervention compared with the control groups. No significant effects were found for disability, cervical range of motion, and health-related quality of life. However, sensitivity analyses revealed significant short-term improvements in pain intensity, disability, and cervical range of motion and sustained effects on disability at intermediate-term follow-up.

CONCLUSION

Sling exercise appears to be beneficial for improvements in pain intensity (moderate- to low-level evidence) among patients with neck pain. However, no definitive conclusion could be made regarding the effect of sling exercise for neck pain due to methodological limitations and high heterogeneity in the included studies.

IMPACT

This review provides overall moderate- to very low-level evidence for health care professionals who may consider including sling exercise in the intervention program for patients with neck pain.

A new method for testing reproducibility in systematic reviews was developed, but needs more testing

BACKGROUND

To develop and test an approach to test reproducibility of SRs.

METHODS

Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review.

RESULTS

We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory.

CONCLUSION

Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.

Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow

BACKGROUND

Current text mining tools supporting abstract screening in systematic reviews are not widely used, in part because they lack sensitivity and precision. We set out to develop an accessible, semi-automated "workflow" to conduct abstract screening for systematic reviews and other knowledge synthesis methods.

METHODS

We adopt widely recommended text-mining and machine-learning methods to (1) process title-abstracts into numerical training data; and (2) train a classification model to predict eligible abstracts. The predicted abstracts are screened by human reviewers for ("true") eligibility, and the newly eligible abstracts are used to identify similar abstracts, using near-neighbor methods, which are also screened. These abstracts, as well as their eligibility results, are used to update the classification model, and the above steps are iterated until no new eligible abstracts are identified. The workflow was implemented in R and evaluated using a systematic review of insulin formulations for type-1 diabetes (14,314 abstracts) and a scoping review of knowledge-synthesis methods (17,200 abstracts). Workflow performance was evaluated against the recommended practice of screening abstracts by 2 reviewers, independently. Standard measures were examined: sensitivity (inclusion of all truly eligible abstracts), specificity (exclusion of all truly ineligible abstracts), precision (inclusion of all truly eligible abstracts among all abstracts screened as eligible), F1-score (harmonic average of sensitivity and precision), and accuracy (correctly predicted eligible or ineligible abstracts). Workload reduction was measured as the hours the workflow saved, given only a subset of abstracts needed human screening.

RESULTS

With respect to the systematic and scoping reviews respectively, the workflow attained 88%/89% sensitivity, 99%/99% specificity, 71%/72% precision, an F1-score of 79%/79%, 98%/97% accuracy, 63%/55% workload reduction, with 12%/11% fewer abstracts for full-text retrieval and screening, and 0%/1.5% missed studies in the completed reviews.

CONCLUSION

The workflow was a sensitive, precise, and efficient alternative to the recommended practice of screening abstracts with 2 reviewers. All eligible studies were identified in the first case, while 6 studies (1.5%) were missed in the second that would likely not impact the review's conclusions. We have described the workflow in language accessible to reviewers with limited exposure to natural language processing and machine learning, and have made the code available to reviewers.

Successful incorporation of single reviewer assessments during systematic review screening: development and validation of sensitivity and work-saved of an algorithm that considers exclusion criteria and count

BACKGROUND

Accepted systematic review (SR) methodology requires citation screening by two reviewers to maximise retrieval of eligible studies. We hypothesized that records could be excluded by a single reviewer without loss of sensitivity in two conditions; the record was ineligible for multiple reasons, or the record was ineligible for one or more specific reasons that could be reliably assessed.

METHODS

Twenty-four SRs performed at CHEO, a pediatric health care and research centre in Ottawa, Canada, were divided into derivation and validation sets. Exclusion criteria during abstract screening were sorted into 11 specific categories, with loss in sensitivity determined by individual category and by number of exclusion criteria endorsed. Five single reviewer algorithms that combined individual categories and multiple exclusion criteria were then tested on the derivation and validation sets, with success defined a priori as less than 5% loss of sensitivity.

RESULTS

The 24 SRs included 930 eligible and 27390 ineligible citations. The reviews were mostly focused on pediatrics (70.8%, N=17/24), but covered various specialties. Using a single reviewer to exclude any citation led to an average loss of sensitivity of 8.6% (95%CI, 6.0-12.1%). Excluding citations with ≥2 exclusion criteria led to 1.2% average loss of sensitivity (95%CI, 0.5-3.1%). Five specific exclusion criteria performed with perfect sensitivity: conference abstract, ineligible age group, case report/series, not human research, and review article. In the derivation set, the five algorithms achieved a loss of sensitivity ranging from 0.0 to 1.9% and work-saved ranging from 14.8 to 39.1%. In the validation set, the loss of sensitivity for all 5 algorithms remained below 2.6%, with work-saved between 10.5% and 48.2%.

CONCLUSIONS

Findings suggest that targeted application of single-reviewer screening, considering both type and number of exclusion criteria, could retain sensitivity and significantly decrease workload. Further research is required to investigate the potential for combining this approach with crowdsourcing or machine learning methodologies.

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews

The methods and results of systematic reviews should be reported in sufficient detail to allow users to assess the trustworthiness and applicability of the review findings. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was developed to facilitate transparent and complete reporting of systematic reviews and has been updated (to PRISMA 2020) to reflect recent advances in systematic review methodology and terminology. Here, we present the explanation and elaboration paper for PRISMA 2020, where we explain why reporting of each item is recommended, present bullet points that detail the reporting recommendations, and present examples from published reviews. We hope that changes to the content and structure of PRISMA 2020 will facilitate uptake of the guideline and lead to more transparent, complete, and accurate reporting of systematic reviews.

Development, testing and use of data extraction forms in systematic reviews: a review of methodological guidance

Background

Data extraction forms link systematic reviews with primary research and provide the foundation for appraising, analysing, summarising and interpreting a body of evidence. This makes their development, pilot testing and use a crucial part of the systematic reviews process. Several studies have shown that data extraction errors are frequent in systematic reviews, especially regarding outcome data.

Methods

We reviewed guidance on the development and pilot testing of data extraction forms and the data extraction process. We reviewed four types of sources: 1) methodological handbooks of systematic review organisations (SRO); 2) textbooks on conducting systematic reviews; 3) method documents from health technology assessment (HTA) agencies and 4) journal articles. HTA documents were retrieved in February 2019 and database searches conducted in December 2019. One author extracted the recommendations and a second author checked them for accuracy. Results are presented descriptively.

Results

Our analysis includes recommendations from 25 documents: 4 SRO handbooks, 11 textbooks, 5 HTA method documents and 5 journal articles. Across these sources the most common recommendations on form development are to use customized or adapted standardised extraction forms (14/25); provide detailed instructions on their use (10/25); ensure clear and consistent coding and response options (9/25); plan in advance which data are needed (9/25); obtain additional data if required (8/25); and link multiple reports of the same study (8/25). The most frequent recommendations on piloting extractions forms are that forms should be piloted on a sample of studies (18/25); and that data extractors should be trained in the use of the forms (7/25). The most frequent recommendations on data extraction are that extraction should be conducted by at least two people (17/25); that independent parallel extraction should be used (11/25); and that procedures to resolve disagreements between data extractors should be in place (14/25).

Conclusions

Overall, our results suggest a lack of comprehensiveness of recommendations. This may be particularly problematic for less experienced reviewers. Limitations of our method are the scoping nature of the review and that we did not analyse internal documents of health technology agencies.

"One more time": why replicating some syntheses of evidence relevant to COVID-19 makes sense

•Given the urgent need for credible answers to high-priority questions about the health and social impacts of COVID-19, many systematic reviewers seek to contribute their skills and expertise. •Rather than embarking on unnecessary, duplicate reviews, we encourage the evidence synthesis community to prioritise purposeful replication of systematic reviews of evidence relevant to COVID-19. •We explain why replication of systematic reviews is important, how to carry out a replication, and when to consider replication of reviews.

Best-worst scaling identified adequate statistical methods and literature search as the most important items of AMSTAR2 (A measurement tool to assess systematic reviews)

OBJECTIVE

To assess the relative importance of A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR2) items.

STUDY DESIGN AND SETTING

A best-worst scaling object case was conducted among a sample of experts in the field of systematic reviews (SRs) and meta-analyses (MAs). Respondents were asked in a series of 15 choice tasks to choose the most and the least important item from a set of four items from the master list, which included the 16 AMSTAR2 items. Hierarchical Bayes analysis was used to generate the relative importance score for each item.

RESULTS

The most important items highlighted by our 242 experts to conduct overview of reviews and critically assess SRs/MAs were the appropriateness of statistical analyses and adequacy of the literature search, followed by items regarding the assessment of risk of bias, the research protocol, and the assessment of heterogeneity (relative importance score >6.5). Items related to funding sources and the assessment of study selection and data extraction in duplicate were rated as least important.

CONCLUSION

Although all AMSTAR2 items can be considered as important, our results highlighted the importance of keeping the two items (the appropriateness of statistical analyses and the adequacy of the literature search) among the critical items proposed by AMSTAR2 to critically appraise SRs/MAs.

Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

OBJECTIVES

Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.

DESIGN

A meta-epidemiological study.

PARTICIPANTS

We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.

PRIMARY AND SECONDARY OUTCOMES

Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.

RESULTS

According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.

CONCLUSION

The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.

Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

OBJECTIVES

Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.

DESIGN

A meta-epidemiological study.

PARTICIPANTS

We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.

PRIMARY AND SECONDARY OUTCOMES

Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.

RESULTS

According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.

CONCLUSION

The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.

Effectiveness of trauma centers verification: Protocol for a systematic review

BACKGROUND

The implementation of trauma systems in many high-income countries over the last 50 years has led to important reductions in injury mortality and disability in many healthcare jurisdictions. Injury organizations including the American College of Surgeons and the Trauma Association of Canada as well as the World Health Organization provide consensus-based recommendations on resources and processes for optimal injury care. Many hospitals treating trauma patients seek verification to demonstrate that they meet these recommendations. This process may be labeled differently across jurisdictions. In Canada for example, it is called accreditation, but it has the same objective and very similar modalities. The objective of the study described in this protocol is to systematically review evidence on the effectiveness of trauma center verification for improving clinical processes and patient outcomes in injury care.

METHODS

We will perform a systematic review of studies evaluating the association between trauma center verification and hospital mortality (primary outcome), as well as morbidity, resource utilization, and processes of care (secondary outcomes). We will search CINAHL, EMBASE, HealthStar, MEDLINE, and ProQuest databases, as well as key injury organization websites for gray literature. We will assess the methodological quality of studies using the Risk Of Bias In Non-randomized Studies - of Interventions (ROBINS-I) assessment tool. We are planning to conduct a meta-analysis if feasible based on the number of included studies and their heterogeneity. We will evaluate the quality of cumulative evidence and strength of recommendations using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group methodology.

DISCUSSION

This review will provide a synthesis of the body of evidence on trauma center verification effectiveness. Results could reinforce current verification modalities and may suggest ways to optimize them. Results will be published in a peer-reviewed journal and presented at an international clinical conference.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018107083.

Low-value clinical practices in adult traumatic brain injury: an umbrella review protocol

INTRODUCTION

Traumatic brain injury (TBI) leads to 50 000 deaths, 85 000 disabilities and costs $60 billion each year in the USA. Despite numerous interventions and treatment options, the outcomes of TBI have improved little over the last three decades. In a previous scoping review and expert consultation survey, we identified 13 potentially low-value clinical practices in acute TBI. The objective of this umbrella review is to synthesise the evidence on potentially low-value clinical practices in the care of acute TBI.

METHODS AND ANALYSIS

Using umbrella review methodology, we will search Cochrane Central Register of Controlled Trials, Embase, Epistemonikos, International Prospective Register of Systematic Reviews (PROSPERO) and PubMed to identify systematic reviews evaluating the effect of potential intrahospital low-value practices using tailored population, intervention, comparator, outcome and study design questions based on the results of a previous scoping review. We will present data on the methodological quality of these reviews (Assessing the Methodological Quality of Systematic Reviews-2), reported effect sizes and strength of evidence (Grading of Recommendations, Assessment, Development and Evaluation).

ETHICS AND DISSEMINATION

Ethics approval is not required as original data will not be collected. Knowledge users from five healthcare quality organisations and clinical associations are involved in the design and conduct of the study. Results will be disseminated in a peer-reviewed journal, at international scientific meetings and to clinical, healthcare quality and patient-partner associations. This work will support the development of metrics to measure the use of low-value practices, inform policy makers on potential targets for deimplementation and in the long term reduce the use of low-value clinical practices in acute TBI care.

PROSPERO REGISTRATION NUMBER

CRD42019132428.