Using GRADE to respond to health questions with different levels of urgency

Evidence Synthesis of Observational Studies in Environmental Health: Lessons Learned from a Systematic Review on Traffic-Related Air Pollution

BACKGROUND

There is a long tradition in environmental health of using frameworks for evidence synthesis, such as those of the U.S. Environmental Protection Agency for its Integrated Science Assessments and the International Agency for Research on Cancer Monographs. The framework, Grading of Recommendations Assessment, Development, and Evaluation (GRADE), was developed for evidence synthesis in clinical medicine. The U.S. Office of Health Assessment and Translation (OHAT) elaborated an approach for evidence synthesis in environmental health building on GRADE.

METHODS

We applied a modified OHAT approach and a broader "narrative" assessment to assess the level of confidence in a large systematic review on traffic-related air pollution and health outcomes.

DISCUSSION

We discuss several challenges with the OHAT approach and its implementation and suggest improvements for synthesizing evidence from observational studies in environmental health. We consider the determination of confidence using a formal rating scheme of up- and downgrading of certain factors, the treatment of every factor as equally important, and the lower initial confidence rating of observational studies to be fundamental issues in the OHAT approach. We argue that some observational studies can offer high-confidence evidence in environmental health. We note that heterogeneity in magnitude of effect estimates should generally not weaken the confidence in the evidence, and consistency of associations across study designs, populations, and exposure assessment methods may strengthen confidence in the evidence. We mention that publication bias should be explored beyond statistical methods and is likely limited when large and collaborative studies comprise most of the evidence and when accrued over several decades. We propose to identify possible key biases, their most likely direction, and their potential impacts on the results. We think that the OHAT approach and other GRADE-type frameworks require substantial modification to align better with features of environmental health questions and the studies that address them. We emphasize that a broader, "narrative" evidence assessment based on the systematic review may complement a formal GRADE-type evaluation. https://doi.org/10.1289/EHP11532.

Testing guidelines during times of crisis: challenges and limitations of developing rapid and living guidelines

BACKGROUND

The start of the COVID-19 pandemic presented a situation in which there was an urgent need for decision-making that relates to diagnosis, but the evidence was lacking, of low certainty or constantly changing. Rapid and living guideline development methods were needed and had to be applied to rigorous guideline approaches, such as the Grading of Recommendations Assessment, Development, and Evaluation approach.

OBJECTIVES

To describe the process of developing rapid diagnosis guidelines when there is limited and imperfect available data at the time of crisis.

SOURCES

Case example from four Infectious Disease Society of America COVID-19 diagnostic guidelines.

CONTENT

As the world was experiencing panic with COVID-19, there were serious doubts about the feasibility of following a rigorous process for guideline development when timeliness was of extreme value. The Infectious Disease Society of America guideline panels supported by several methodologists strongly believed that at times of crisis, it is more important than ever to follow a rigorous process. The panel adopted a rapid and living systematic review methodology and applied the Grading of Recommendations Assessment, Development and Evaluation approach to four diagnosis guidelines despite the challenges of scarce and dynamic evidence. We describe the methodological details of the rapid and living approach (data extraction, meta-analysis, Evidence to Decision framework, and recommendation development), the challenge of resources, the challenge of scarce evidence, the challenge of rapidly changing evidence, as well as 'wins' from the Infectious Disease Society of America experience.

IMPLICATIONS

Mitigation of pandemics relies on rapid and accurate diagnosis, which is challenged by many knowledge gaps. This necessitates emerging evidence is rapidly incorporated in a living fashion with several decisional and contextual factors to ensure the best public health strategies and care for patients. This process must be systematic and transparent for developing trustworthy guidelines and should be supported by all stakeholders, including researchers, editors, publishers, professional societies, and policymakers.

Rapid development of an evidence- and consensus-based guideline for controlling transmission of SARS-CoV-2 in schools during a public health emergency - A process evaluation

Introduction

Different measures to prevent and control the spread of SARS-CoV-2 have been implemented in German schools. Decisions regarding such measures should be informed by evidence regarding their effectiveness, and their unintended consequences for health and society. A multi-stakeholder panel was convened to develop an evidence- and consensus-based guideline for school measures, using the novel WHO-INTEGRATE framework. Developing a guideline to inform decision-making outside of the clinical realm during a public health emergency was unprecedented in Germany. This study aims to identify lessons learnt for similar endeavours by addressing the following research question: What were the strengths and weaknesses of the guideline development process as perceived by the different groups involved?

Methods

Fifteen semi-structured interviews were conducted virtually. We recruited participants aiming to include the perspectives of all groups contributing to the guideline development, including both panel members (scientists, practitioners, school family and observers) and the guideline secretariat. For analysis, we carried out deductive-inductive thematic qualitative text analysis according to Kuckartz, structuring findings using a category system.

Results

Due to time pressure, the guideline secretariat was heavily involved not only in synthesising the evidence but also in developing and drafting recommendations. Participants critically reflected on certain methods-related decisions, including the development of draft recommendations and application of the WHO-INTEGRATE framework by scientists only. The full potential of the framework might not have been harnessed. Participants' understanding of relevant and valid evidence varied, and the available evidence base was limited. Participants represented different types of expertise, notably expertise informed by scientific evidence and expertise grounded in lived experience, influencing their involvement in the guideline development process and discussions during meetings.

Conclusion

Developing an evidence- and consensus-based public health guideline in only three months was challenging, notably because of the involvement of a broad range of stakeholders and the use of a novel Evidence-to-Decision framework, both unprecedented in Germany. Learning from this process with a view to "institutionalising" the development of public health guidelines and refining methods can contribute to more evidence-informed public health decision-making in Germany and beyond, in general and during a public health emergency.

International alliance and AGREE-ment of 71 clinical practice guidelines on the management of critical care patients with COVID-19: a living systematic review

OBJECTIVE

We aimed to systematically identify and critically assess the clinical practice guidelines (CPGs) for the management of critically ill patients with COVID-19 with the AGREE II instrument.

STUDY DESIGN AND SETTING

We searched Medline, CINAHL, EMBASE, CNKI, CBM, WanFang, and grey literature from November 2019 - November 2020. We did not apply language restrictions. One reviewer independently screened the retrieved titles and abstracts, and a second reviewer confirmed the decisions. Full texts were assessed independently and in duplicate. Disagreements were resolved by consensus. We included any guideline that provided recommendations on the management of critically ill patients with COVID-19. Data extraction was performed independently and in duplicate by two reviewers. We descriptively summarized CPGs characteristics. We assessed the quality with the AGREE II instrument and we summarized relevant therapeutic interventions.

RESULTS

We retrieved 3,907 records and 71 CPGs were included. Means (Standard Deviations) of the scores for the 6 domains of the AGREE II instrument were 65%(SD19.56%), 39%(SD19.64%), 27%(SD19.48%), 70%(SD15.74%), 26%(SD18.49%), 42%(SD34.91) for the scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, editorial independence domains, respectively. Most of the CPGs showed a low overall quality (less than 40%).

CONCLUSION

Future CPGs for COVID-19 need to rely, for their development, on standard evidence-based methods and tools.

The effect of exposure to radiofrequency fields on cancer risk in the general and working population: A protocol for a systematic review of human observational studies

BACKGROUND

The World Health Organization (WHO) has an ongoing project to assess potential health effects of exposure to radiofrequency electromagnetic fields (RF-EMF) in the general and working population. Here we present the protocol for a systematic review of the scientific literature on cancer hazards from exposure to RF-EMF in humans, commissioned by the WHO as part of that project.

OBJECTIVE

To assess the quality and strength of the evidence provided by human observational studies for a causal association between exposure to RF-EMF and risk of neoplastic diseases.

ELIGIBILITY CRITERIA

We will include cohort and case-control studies investigating neoplasia risks in relation to three types of exposure to RF-EMF: near-field, head-localized, exposure from wireless phone use (SR-A); far-field, whole body, environmental exposure from fixed-site transmitters (SR-B); near/far-field occupational exposures from use of handheld transceivers or RF-emitting equipment in the workplace (SR-C). While no restriction on tumour type will be applied, we will focus on selected neoplasms of the central nervous system (brain, meninges, pituitary gland, acoustic nerve) and salivary gland tumours (SR-A); brain tumours and leukaemias (SR-B, SR-C).

INFORMATION SOURCES

Eligible studies will be identified through Medline, Embase, and EMF-Portal.

RISK-OF-BIAS ASSESSMENT

We will use a tailored version of the OHAT's tool to evaluate the study's internal validity.

DATA SYNTHESIS

We will consider separately studies on different tumours, neoplasm-specific risks from different exposure sources, and a given exposure-outcome pair in adults and children. When a quantitative synthesis of findings can be envisaged, the main aims of the meta-analysis will be to assess the strength of association and the shape of the exposure-response relationship; to quantify the degree of heterogeneity across studies; and explore the sources of inconsistency (if any). When a meta-analysis is judged inappropriate, we will perform a narrative synthesis, complemented by a structured tabulation of results and appropriate visual displays.

EVIDENCE ASSESSMENT

Confidence in evidence will be assessed in line with the GRADE approach.

FUNDING

This project is supported by the World Health Organization. Co-financing was provided by the New Zealand Ministry of Health; the Istituto Superiore di Sanità in its capacity as a WHO Collaborating Centre for Radiation and Health; ARPANSA as a WHO Collaborating Centre for Radiation Protection.

REGISTRATION

PROSPERO CRD42021236798.

[GRADE guidelines 20: Assessing the certainty of evidence in the importance of outcomes or values and preferences-inconsistency, imprecision, and other domains]

OBJECTIVE

To provide Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) guidance for assessing inconsistency, imprecision, and other domains for the certainty of evidence about the relative importance of outcomes.

STUDY DESIGN AND SETTING

We applied the GRADE domains to rate the certainty of evidence in the importance of outcomes to several systematic reviews, iteratively reviewed draft guidance, and consulted GRADE members and other stakeholders for feedback.

RESULTS

We describe the rationale for considering the remaining GRADE domains when rating the certainty in a body of evidence for the relative importance of outcomes. As meta-analyses are not common in this context, inconsistency and imprecision assessments are challenging. Furthermore, confusion exists about inconsistency, imprecision, and true variability in the relative importance of outcomes. To clarify this issue, we suggest that the true variability is neither equivalent to inconsistency nor imprecision. Specifically, inconsistency arises from population, intervention, comparison and outcome and methodological elements that should be explored and, if possible, explained. The width of the confidence interval and sample size inform judgments about imprecision. We also provide suggestions on how to detect publication bias and discuss the domains to rate up the certainty.

CONCLUSION

We provide guidance and examples for rating inconsistency, imprecision, and other domains for a body of evidence describing the relative importance of outcomes.

Use of Chest Imaging in the Diagnosis and Management of COVID-19: A WHO Rapid Advice Guide

The World Health Organization (WHO) undertook the development of a rapid guide on the use of chest imaging in the diagnosis and management of coronavirus disease 2019 (COVID-19). The rapid guide was developed over 2 months by using standard WHO processes, except for the use of "rapid reviews" and online meetings of the panel. The evidence review was supplemented by a survey of stakeholders regarding their views on the acceptability, feasibility, impact on equity, and resource use of the relevant chest imaging modalities (chest radiography, chest CT, and lung US). The guideline development group had broad expertise and country representation. The rapid guide includes three diagnosis recommendations and four management recommendations. The recommendations cover patients with confirmed or who are suspected of having COVID-19 with different levels of disease severity, throughout the care pathway from outpatient facility or hospital entry to home discharge. All recommendations are conditional and are based on low certainty evidence (n = 2), very low certainty evidence (n = 2), or expert opinion (n = 3). The remarks accompanying the recommendations suggest which patients are likely to benefit from chest imaging and what factors should be considered when choosing the specific imaging modality. The guidance offers considerations about implementation, monitoring, and evaluation, and also identifies research needs. Published under a CC BY 4.0 license. Online supplemental material is available for this article.

Challenges in applying the GRADE approach in public health guidelines and systematic reviews: a concept article from the GRADE Public Health Group

BACKGROUND AND OBJECTIVE

This article explores the need for conceptual advances and practical guidance in the application of the GRADE approach within public health contexts.

METHODS

We convened an expert workshop and conducted a scoping review to identify challenges experienced by GRADE users in public health contexts. We developed this concept article through thematic analysis and an iterative process of consultation and discussion conducted with members electronically and at three GRADE Working Group meetings.

RESULTS

Five priority issues can pose challenges for public health guideline developers and systematic reviewers when applying GRADE: (1) incorporating the perspectives of diverse stakeholders; (2) selecting and prioritizing health and "nonhealth" outcomes; (3) interpreting outcomes and identifying a threshold for decision-making; (4) assessing certainty of evidence from diverse sources, including nonrandomized studies; and (5) addressing implications for decision makers, including concerns about conditional recommendations. We illustrate these challenges with examples from public health guidelines and systematic reviews, identifying gaps where conceptual advances may facilitate the consistent application or further development of the methodology and provide solutions.

CONCLUSION

The GRADE Public Health Group will respond to these challenges with solutions that are coherent with existing guidance and can be consistently implemented across public health decision-making contexts.

Developing trustworthy recommendations as part of an urgent response (1-2 weeks): a GRADE concept paper

OBJECTIVES

The aim of this study is to propose an approach for developing trustworthy recommendations as part of urgent responses (1-2 week) in the clinical, public health, and health systems fields.

STUDY DESIGN AND SETTING

We conducted a review of the literature, outlined a draft approach, refined the concept through iterative discussions, a workshop by the Grading of Recommendations Assessment, Development and Evaluation Rapid Guidelines project group, and obtained feedback from the larger Grading of Recommendations Assessment, Development and Evaluation working group.

RESULTS

A request for developing recommendations within 2 week is the usual trigger for an urgent response. Although the approach builds on the general principles of trustworthy guideline development, we highlight the following steps: (1) assess the level of urgency; (2) assess feasibility; (3) set up the organizational logistics; (4) specify the question(s); (5) collect the information needed; (6) assess the adequacy of identified information; (7) develop the recommendations using one of the 4 potential approaches: adopt existing recommendations, adapt existing recommendations, develop new recommendations using existing adequate systematic review, or develop new recommendations using expert panel input; and (8) consider an updating plan.

CONCLUSION

An urgent response for developing recommendations requires building a cohesive, skilled, and highly motivated multidisciplinary team with the necessary clinical, scientific, and methodological expertise; adapting to shifting needs; complying with the principles of transparency; and properly managing conflicts of interest.

Using GRADE in situations of emergencies and urgencies: certainty in evidence and recommendations matters during the COVID-19 pandemic, now more than ever and no matter what

OBJECTIVES

The public, policy makers, and science communities are subject to many false, uninformed, overly optimistic, premature, or simply ridiculous health claims. The coronavirus disease 2019 (COVID-19) pandemic and its context is a paramount example for such claims. In this article, we describe why expressing the certainty in evidence to support a decision is critical and why the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach matters now, perhaps more than ever and no matter what the specific topic is in relation to COVID-19. We finally also offer suggestions for how it can be used appropriately to support decision-making at global, national, and local level when emergency, urgent or rapid responses are needed.

STUDY DESIGN AND SETTING

This is an invited commentary to address the objectives above building on examples from the recent COVID-19 pandemic. This includes an iterative discussion of examples and development of guidance.

RESULTS

The GRADE approach is a transparent and structured method for assessing the certainty of evidence and when developing recommendations that requires little additional time. We describe why emergency, urgent, or rapid responses do not justify omitting this critical assessment of the evidence. In situations of emergencies and urgencies, such as the COVID-19 pandemic, GRADE can similarly be used to express and convey certainty in intervention effects, test accuracy, risk and prognostic factors, consequences of public health measures, and qualitative bodies of evidence.

CONCLUSIONS

Assessing and communicating the certainty of evidence during the COVID-19 pandemic is critical. Those offering evidence synthesis or making recommendations should use transparent ratings of the body of evidence supporting a claim regardless of time that is available or needed to provide this response.

Guidance vs. guidelines: the role of evidence-based medicine in the COVID-19 pandemic

The COVID-19 pandemic has challenged the practice of high-quality evidence-based medicine because no uniformly accepted treatment protocols or rigorous guidelines are widely available. This article examines the role of practice management guidelines versus clinical guidance for safe and evidence-based clinical care of patients with COVID-19.

GRADE Guidelines 30: the GRADE approach to assessing the certainty of modeled evidence-An overview in the context of health decision-making

OBJECTIVES

The objective of the study is to present the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) conceptual approach to the assessment of certainty of evidence from modeling studies (i.e., certainty associated with model outputs).

STUDY DESIGN AND SETTING

Expert consultations and an international multidisciplinary workshop informed development of a conceptual approach to assessing the certainty of evidence from models within the context of systematic reviews, health technology assessments, and health care decisions. The discussions also clarified selected concepts and terminology used in the GRADE approach and by the modeling community. Feedback from experts in a broad range of modeling and health care disciplines addressed the content validity of the approach.

RESULTS

Workshop participants agreed that the domains determining the certainty of evidence previously identified in the GRADE approach (risk of bias, indirectness, inconsistency, imprecision, reporting bias, magnitude of an effect, dose-response relation, and the direction of residual confounding) also apply when assessing the certainty of evidence from models. The assessment depends on the nature of model inputs and the model itself and on whether one is evaluating evidence from a single model or multiple models. We propose a framework for selecting the best available evidence from models: 1) developing de novo, a model specific to the situation of interest, 2) identifying an existing model, the outputs of which provide the highest certainty evidence for the situation of interest, either "off-the-shelf" or after adaptation, and 3) using outputs from multiple models. We also present a summary of preferred terminology to facilitate communication among modeling and health care disciplines.

CONCLUSION

This conceptual GRADE approach provides a framework for using evidence from models in health decision-making and the assessment of certainty of evidence from a model or models. The GRADE Working Group and the modeling community are currently developing the detailed methods and related guidance for assessing specific domains determining the certainty of evidence from models across health care-related disciplines (e.g., therapeutic decision-making, toxicology, environmental health, and health economics).

Ventilation Techniques and Risk for Transmission of Coronavirus Disease, Including COVID-19: A Living Systematic Review of Multiple Streams of Evidence

BACKGROUND

Mechanical ventilation is used to treat respiratory failure in coronavirus disease 2019 (COVID-19).

PURPOSE

To review multiple streams of evidence regarding the benefits and harms of ventilation techniques for coronavirus infections, including that causing COVID-19.

DATA SOURCES

21 standard, World Health Organization-specific and COVID-19-specific databases, without language restrictions, until 1 May 2020.

STUDY SELECTION

Studies of any design and language comparing different oxygenation approaches in patients with coronavirus infections, including severe acute respiratory syndrome (SARS) or Middle East respiratory syndrome (MERS), or with hypoxemic respiratory failure. Animal, mechanistic, laboratory, and preclinical evidence was gathered regarding aerosol dispersion of coronavirus. Studies evaluating risk for virus transmission to health care workers from aerosol-generating procedures (AGPs) were included.

DATA EXTRACTION

Independent and duplicate screening, data abstraction, and risk-of-bias assessment (GRADE for certainty of evidence and AMSTAR 2 for included systematic reviews).

DATA SYNTHESIS

123 studies were eligible (45 on COVID-19, 70 on SARS, 8 on MERS), but only 5 studies (1 on COVID-19, 3 on SARS, 1 on MERS) adjusted for important confounders. A study in hospitalized patients with COVID-19 reported slightly higher mortality with noninvasive ventilation (NIV) than with invasive mechanical ventilation (IMV), but 2 opposing studies, 1 in patients with MERS and 1 in patients with SARS, suggest a reduction in mortality with NIV (very-low-certainty evidence). Two studies in patients with SARS report a reduction in mortality with NIV compared with no mechanical ventilation (low-certainty evidence). Two systematic reviews suggest a large reduction in mortality with NIV compared with conventional oxygen therapy. Other included studies suggest increased odds of transmission from AGPs.

LIMITATION

Direct studies in COVID-19 are limited and poorly reported.

CONCLUSION

Indirect and low-certainty evidence suggests that use of NIV, similar to IMV, probably reduces mortality but may increase the risk for transmission of COVID-19 to health care workers.

PRIMARY FUNDING SOURCE

World Health Organization. (PROSPERO: CRD42020178187).

The need of health policy perspective to protect Healthcare Workers during COVID-19 pandemic. A GRADE rapid review on the N95 respirators effectiveness

Protecting Health Care Workers (HCWs) during routine care of suspected or confirmed COVID-19 patients is of paramount importance to halt the SARS-CoV-2 (Severe Acute Respiratory Syndrome-Coronavirus-2) pandemic. The WHO, ECDC and CDC have issued conflicting guidelines on the use of respiratory filters (N95) by HCWs. We searched PubMed, Embase and The Cochrane Library from the inception to March 21, 2020 to identify randomized controlled trials (RCTs) comparing N95 respirators versus surgical masks for prevention of COVID-19 or any other respiratory infection among HCWs. The grading of recommendations, assessment, development, and evaluation (GRADE) was used to evaluate the quality of evidence. Four RCTs involving 8736 HCWs were included. We did not find any trial specifically on prevention of COVID-19. However, wearing N95 respirators can prevent 73 more (95% CI 46-91) clinical respiratory infections per 1000 HCWs compared to surgical masks (2 RCTs; 2594 patients; low quality of evidence). A protective effect of N95 respirators in laboratory-confirmed bacterial colonization (RR = 0.41; 95%CI 0.28-0.61) was also found. A trend in favour of N95 respirators was observed in preventing laboratory-confirmed respiratory viral infections, laboratory-confirmed respiratory infection, and influenza like illness. We found no direct high quality evidence on whether N95 respirators are better than surgical masks for HCWs protection from SARS-CoV-2. However, low quality evidence suggests that N95 respirators protect HCWs from clinical respiratory infections. This finding should be contemplated to decide the best strategy to support the resilience of healthcare systems facing the potentially catastrophic SARS-CoV-2 pandemic.

Safe management of bodies of deceased persons with suspected or confirmed COVID-19: a rapid systematic review

INTRODUCTION

Proper strategies to minimise the risk of infection in individuals handling the bodies of deceased persons infected with 2019 novel coronavirus (2019-nCoV) are urgently needed. The objective of this study was to systematically review the literature to scope and assess the effects of specific strategies for the management of the bodies.

METHODS

We searched five general, three Chinese and four coronavirus disease (COVID-19)-specific electronic databases. We searched registries of clinical trials, websites of governmental and other relevant organisations, reference lists of the included papers and relevant systematic reviews, and Epistemonikos for relevant systematic reviews. We included guidance documents providing practical advice on the handling of bodies of deceased persons with suspected or confirmed COVID-19. Then, we sought primary evidence of any study design reporting on the efficacy and safety of the identified strategies in coronaviruses. We included evidence relevant to contextual factors (ie, acceptability). A single reviewer extracted data using a pilot-tested form and graded the certainty of the evidence using the GRADE approach. A second reviewer verified the data and assessments.

RESULTS

We identified one study proposing an uncommon strategy for autopsies for patients with severe acute respiratory syndrome. The study provided very low-certainty evidence that it reduced the risk of transmission. We identified 23 guidance documents providing practical advice on the steps of handling the bodies: preparation, packing, and others and advice related to both the handling of the dead bodies and the use of personal protective equipment by individuals handling them. We did not identify COVID-19 evidence relevant to any of these steps.

CONCLUSION

While a substantive number of guidance documents propose specific strategies, we identified no study providing direct evidence for the effects of any of those strategies. While this review highlights major research gaps, it allows interested entities to build their own guidance.

GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy

OBJECTIVES

This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can assess the results and the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA).

STUDY DESIGN AND SETTING

We present an overview of the GRADE approach and guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. Part 1 of the two parts in this 21st guidance article about how to apply GRADE focuses on understanding study design issues in test accuracy, provide an overview of the domains, and describe risk of bias and indirectness specifically.

RESULTS

Supplemented by practical examples, we describe how raters of the evidence using GRADE can evaluate study designs focusing on tests and how they apply the GRADE domains risk of bias and indirectness to a body of evidence of TA studies.

CONCLUSION

Rating the certainty of a body of evidence using GRADE in Cochrane and other reviews and World Health Organization and other guidelines dealing with in TA studies helped refining our approach. The resulting guidance will help applying GRADE successfully for questions and recommendations focusing on tests.

GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence

OBJECTIVE

To provide guidance on how systematic review authors, guideline developers, and health technology assessment practitioners should approach the use of the risk of bias in nonrandomized studies of interventions (ROBINS-I) tool as a part of GRADE's certainty rating process.

STUDY DESIGN AND SETTING

The study design and setting comprised iterative discussions, testing in systematic reviews, and presentation at GRADE working group meetings with feedback from the GRADE working group.

RESULTS

We describe where to start the initial assessment of a body of evidence with the use of ROBINS-I and where one would anticipate the final rating would end up. The GRADE accounted for issues that mitigate concerns about confounding and selection bias by introducing the upgrading domains: large effects, dose-effect relations, and when plausible residual confounders or other biases increase certainty. They will need to be considered in an assessment of a body of evidence when using ROBINS-I.

CONCLUSIONS

The use of ROBINS-I in GRADE assessments may allow for a better comparison of evidence from randomized controlled trials (RCTs) and nonrandomized studies (NRSs) because they are placed on a common metric for risk of bias. Challenges remain, including appropriate presentation of evidence from RCTs and NRSs for decision-making and how to optimally integrate RCTs and NRSs in an evidence assessment.

Development of rapid guidelines: 2. A qualitative study with WHO guideline developers

BACKGROUND

Situations such as public health emergencies and outbreaks necessitate the development and publication of high-quality recommendations within a condensed timeframe. For example, WHO has produced examples of and guidance for the development of rapid guidelines (RGs). However, more information is needed to understand the experiences and perceptions of guideline developers. This is the second of a series of three articles addressing methodological issues around RGs. This study describes the perceptions and experiences of guideline developers at WHO about RGs.

METHODS

We conducted interviews consisting of open- and closed-ended questions with guideline developers at WHO. Our analysis described the definition and rationale of RGs, the differences from regular guidelines with regard to timelines from topic definition until publication, barriers to identifying the evidence and the lack of a standard methodology to develop RGs.

RESULTS

We interviewed 10 participants, the majority of whom were comfortable with the current WHO definition of RGs. Most stated that the rationale for developing RGs should be in response to new evidence about efficacy, cost-effectiveness or safety. Respondents differed with regards to the amount of time RGs should take. While the majority of participants agreed that guidelines should be based on a systematic review, this step in the process was considered the most time and resource intensive. Challenges for developing RGs included limited personnel and financial resources as well as the lack of evidence. Facilitators, in turn, that may improve RG development include additional financial and personnel resources as well as the use of virtual meetings.

CONCLUSIONS

While our study suggests a strong need and rationale for the development of RGs, standardisation of timelines and guidance on panel composition, peer-review process, conduct of meetings and sources of permissible evidence require further research.

Development of rapid guidelines: 1. Systematic survey of current practices and methods

BACKGROUND

Guidelines in the healthcare field generally should contain evidence-based recommendations to inform healthcare decisions. Guidelines often require 2 years or more to develop, but certain circumstances necessitate the development of rapid guidelines (RGs) in a short period of time. Upholding methodological rigor while meeting the reduced development timeframe presents a challenge for developing RGs. Our objective was to review current practices and standards for the development of RGs. This is the first of a series of three articles addressing methodological issues around RGs.

METHODS

We conducted a systematic survey of methods manuals and published RGs to identify reasons for the development of RGs. Data sources included existing guideline manuals, published RGs, Trip Medical Database, MEDLINE, EMBASE and communication with guideline developers until February 2018.

RESULTS

We identified 46 guidelines that used a shortened timeframe for their development. Nomenclature describing RGs varied across organisations, wherein the United States Centers for Disease Control and Prevention produced 'Interim Guidelines', the National Institute for Health and Care Excellence in the United Kingdom developed 'Short Clinical Guidelines', and WHO provided 'Rapid Advice'. The rationale for RGs included response to emergencies, rapid increases in cases of a condition or disease severity, or new evidence regarding treatment. In general, the methods to assess the quality of evidence, the consensus process and the management of the conflict of interest were not always clear. While we identified another 11 RGs from other institutions, there was no reference to timeframe and reasons for conducting a RG. The three organisations mentioned above provide guidance for the development of RGs.

CONCLUSIONS

There is a lack of standardised nomenclature and definitions regarding RGs and there is inconsistency in the methods described in manuals and in RG. It is therefore important that all RGs provide a detailed and transparent description of their methods in order for readers and end-users to be able to assess their quality and validate their findings.

Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations

BACKGROUND

Practice guidelines require a substantial investment of resources and time, often taking between 1 and 3 years from conceptualisation to publication. However, urgent situations require the development of recommendations in a shorter timeframe. In this third and final article in the series exploring challenges and solutions in developing rapid guidelines (RGs), we propose guiding principles for the development of RGs.

METHODS

We utilised the Guideline International Network-McMaster Guideline Development Checklist (GDC) as a starting point for elements to consider during RG development. We built on those elements using the findings from a systematic review of guideline manuals, a survey of international organisations conducting RGs, and interviews of guideline developers within WHO. We reviewed initial findings and developed an intermediate list of elements, as well as narrative guidance. We then invited experts to validate the intermediate list, reviewing for placement, brevity and redundancy. We used this iterative process and group consensus to determine the final elements for RG development guidance.

RESULTS

Our work identified 21 principles within the topics of the Guideline International Network-McMaster GDC to guide the planning and development of RGs. Principles fell within 15 of the 18 checklist topics, highlighting strategies to streamline and expedite the guideline development process.

CONCLUSIONS

We defined principles to guide the development of RGs, while maintaining a standardised, rigorous and transparent process. These principles will serve as guidance for guideline developers responding to urgent situations such as public health urgencies. Integration of these principles within currently disseminated guideline development standards will facilitate the use of those tools in situations necessitating RG recommendations.

Rapid response in health technology assessment: a Delphi study for a Brazilian guideline

Background

Rapid response in health technology assessment is a synthesis of the best available evidence prepared in a timely manner to meet specific demands. We build a consensus among Brazilian specialists in health technology assessment to propose guidelines for the development of rapid response.

Methods

Based on a systematic review that proposed eight methodological steps to conduct rapid response, we applied a modified Delphi technique (without open questions in the first round) to reach consensus among Brazilian experts in health technology assessment. Twenty participants were invited to judge the feasibility of each methodological step in a five-point Likert scale. Consensus was reached if the step had 70% positive approval or interquartile range ≤ 1.

Results

The achievement of consensus was reached in the second round. Between the first and the second round, we scrutinized all points reported by the experts. The Delphi panel reached consensus of eight steps: definition of the structured question of rapid response (with a restricted scope); definition of the eligibility criteria for study types (preferably systematic reviews); search strategy (language and data limits) and sources of information (minimum two); selection of studies (independently by two responders); critical appraisal of the included studies and the risk of bias for the outcomes of interest; data extraction from the included articles; summary of evidence; and preparation of the report.

Conclusions

The guidelines for rapid response in health technology assessment may help governments to make better decisions in a short period of time (35 days). The adoption of methodological processes should improve both the quality and consistency of health technology assessments of rapid decisions in the Brazilian setting.

Guidance on the use of the weight of evidence approach in scientific assessments

EFSA requested the Scientific Committee to develop a guidance document on the use of the weight of evidence approach in scientific assessments for use in all areas under EFSA's remit. The guidance document addresses the use of weight of evidence approaches in scientific assessments using both qualitative and quantitative approaches. Several case studies covering the various areas under EFSA's remit are annexed to the guidance document to illustrate the applicability of the proposed approach. Weight of evidence assessment is defined in this guidance as a process in which evidence is integrated to determine the relative support for possible answers to a question. This document considers the weight of evidence assessment as comprising three basic steps: (1) assembling the evidence into lines of evidence of similar type, (2) weighing the evidence, (3) integrating the evidence. The present document identifies reliability, relevance and consistency as three basic considerations for weighing evidence.

Rating the certainty in evidence in the absence of a single estimate of effect

When studies measure or report outcomes differently, it may not be feasible to pool data across studies to generate a single effect estimate (ie, perform meta-analysis). Instead, only a narrative summary of the effect across different studies might be available. Regardless of whether a single pooled effect estimate is generated or whether data are summarised narratively, decision makers need to know the certainty in the evidence in order to make informed decisions. In this guide, we illustrate how to apply the constructs of the GRADE (Grading of Recommendation, Assessment, Development and Evaluation) approach to assess the certainty in evidence when a meta-analysis has not been performed and data were summarised narratively.

Zika Virus Infection as a Cause of Congenital Brain Abnormalities and Guillain-Barré Syndrome: Systematic Review

BACKGROUND

The World Health Organization (WHO) stated in March 2016 that there was scientific consensus that the mosquito-borne Zika virus was a cause of the neurological disorder Guillain-Barré syndrome (GBS) and of microcephaly and other congenital brain abnormalities based on rapid evidence assessments. Decisions about causality require systematic assessment to guide public health actions. The objectives of this study were to update and reassess the evidence for causality through a rapid and systematic review about links between Zika virus infection and (a) congenital brain abnormalities, including microcephaly, in the foetuses and offspring of pregnant women and (b) GBS in any population, and to describe the process and outcomes of an expert assessment of the evidence about causality.

METHODS AND FINDINGS

The study had three linked components. First, in February 2016, we developed a causality framework that defined questions about the relationship between Zika virus infection and each of the two clinical outcomes in ten dimensions: temporality, biological plausibility, strength of association, alternative explanations, cessation, dose-response relationship, animal experiments, analogy, specificity, and consistency. Second, we did a systematic review (protocol number CRD42016036693). We searched multiple online sources up to May 30, 2016 to find studies that directly addressed either outcome and any causality dimension, used methods to expedite study selection, data extraction, and quality assessment, and summarised evidence descriptively. Third, WHO convened a multidisciplinary panel of experts who assessed the review findings and reached consensus statements to update the WHO position on causality. We found 1,091 unique items up to May 30, 2016. For congenital brain abnormalities, including microcephaly, we included 72 items; for eight of ten causality dimensions (all except dose-response relationship and specificity), we found that more than half the relevant studies supported a causal association with Zika virus infection. For GBS, we included 36 items, of which more than half the relevant studies supported a causal association in seven of ten dimensions (all except dose-response relationship, specificity, and animal experimental evidence). Articles identified nonsystematically from May 30 to July 29, 2016 strengthened the review findings. The expert panel concluded that (a) the most likely explanation of available evidence from outbreaks of Zika virus infection and clusters of microcephaly is that Zika virus infection during pregnancy is a cause of congenital brain abnormalities including microcephaly, and (b) the most likely explanation of available evidence from outbreaks of Zika virus infection and GBS is that Zika virus infection is a trigger of GBS. The expert panel recognised that Zika virus alone may not be sufficient to cause either congenital brain abnormalities or GBS but agreed that the evidence was sufficient to recommend increased public health measures. Weaknesses are the limited assessment of the role of dengue virus and other possible cofactors, the small number of comparative epidemiological studies, and the difficulty in keeping the review up to date with the pace of publication of new research.

CONCLUSIONS

Rapid and systematic reviews with frequent updating and open dissemination are now needed both for appraisal of the evidence about Zika virus infection and for the next public health threats that will emerge. This systematic review found sufficient evidence to say that Zika virus is a cause of congenital abnormalities and is a trigger of GBS.