Considerations of complexity in rating certainty of evidence in systematic reviews: a primer on using the GRADE approach in global health

Is movement variability altered in people with chronic non-specific low back pain? A systematic review

BACKGROUND

Variability in spine kinematics is a common motor adaptation to pain, which has been measured in various ways. However, it remains unclear whether low back pain (LBP) is typically characterised by increased, decreased or unchanged kinematic variability. Therefore, the aim of this review was to synthesise the evidence on whether the amount and structure of spine kinematic variability is altered in people with chronic non-specific LBP (CNSLBP).

METHODS

Electronic databases, grey literature, and key journals were searched from inception up to August 2022, following a published and registered protocol. Eligible studies must investigated kinematic variability in CNSLBP people (adults ≥18 years) while preforming repetitive functional tasks. Two reviewers conducted screening, data extraction, and quality assessment independently. Data synthesis was conducted per task type and individual results were presented quantitatively to provide a narrative synthesis. The overall strength of evidence was rated using the Grading of Recommendations, Assessment, Development and Evaluation guidelines.

FINDINGS

Fourteen observational studies were included in this review. To facilitate the interpretation of the results, the included studies were grouped into four categories according to the task preformed (i.e., repeated flexion and extension, lifting, gait, and sit to stand to sit task). The overall quality of evidence was rated as a very low, primarily due to the inclusion criteria that limited the review to observational studies. In addition, the use of heterogeneous metrics for analysis and varying effect sizes contributed to the downgrade of evidence to a very low level.

INTERPRETATION

Individuals with chronic non-specific LBP exhibited altered motor adaptability, as evidenced by differences in kinematic movement variability during the performance of various repetitive functional tasks. However, the direction of the changes in movement variability was not consistent across studies.

Characteristics of the sources, evaluation, and grading of the certainty of evidence in systematic reviews in public health: A methodological study

Objectives

To systematically explore how the sources of evidence, types of primary studies, and tools used to assess the quality of the primary studies vary across systematic reviews (SRs) in public health.

Methods

We conducted a methodological survey of SRs in public health by searching the of literature in selected journals from electronic bibliographic databases. We selected a 10% random sample of the SRs that met the explicit inclusion criteria. Two researchers independently extracted data for analysis.

Results

We selected 301 SRs for analysis: 94 (31.2%) of these were pre-registered, and 211 (70.1%) declared to have followed published reporting standard. All SRs searched for evidence in electronic bibliographic databases, and more than half (n = 180, 60.0%) searched also the references of the included studies. The common types of primary studies included in the SRs were primarily cross-sectional studies (n = 132, 43.8%), cohort studies (n = 126, 41.9%), randomized controlled trials (RCTs, n = 89, 29.6%), quasi-experimental studies (n = 83, 27.6%), case-control studies (n = 58, 19.3%) qualitative studies (n = 38, 12.6%) and mixed-methods studies (n = 32, 10.6%). The most frequently used quality assessment tools were the Newcastle-Ottawa Scale (used for 50.0% of cohort studies and 55.6% of case-control studies), Cochrane Collaboration's Risk of Bias tool (50.7% of RCTs) and Critical Appraisal Skills Program (38.5% of qualitative studies). Only 20 (6.6%) of the SRs assessed the certainty of the body of evidence, of which 19 (95.0%) used the GRADE approach. More than 65% of the evidence in the SRs using GRADE was of low or very low certainty.

Conclusions

SRs should always assess the quality both at the individual study level and the body of evidence for outcomes, which will benefit patients, health care practitioners, and policymakers.

The effects of dietary patterns and food groups on symptomatic osteoarthritis: A systematic review

AIM

To systematically review current literature to determine the association between symptomatic osteoarthritis and dietary patterns, diet quality and food groups in adults aged ≥45 years.

METHODS

The review was registered on PROSPERO (CRD42021270891). Cochrane Central Library, Cumulative Index of Nursing and Allied Health Literature, Embase, Medline and Web of Science databases were searched. A total of 3816 records were identified. Eligible articles involved populations aged ≥45 years with symptomatic osteoarthritis, assessing dietary patterns, diet quality or food groups, with pain in joints as outcomes. The Joanna Briggs Institute Critical Appraisal Checklists were used for quality assessment. Grading of Recommendations, Assessment, Development and Evaluation was used to assess the certainty of evidence.

RESULTS

Six cohort studies were included. The Prudent dietary pattern and the Mediterranean dietary pattern reduced the progression of osteoarthritis symptoms. The Western dietary pattern increased symptomatic osteoarthritis progression. Increased total fibre consumption reduced symptomatic osteoarthritis progression and pain worsening, but the effects of fibre from each food group were inconclusive. Diet with high inflammatory potential increased risk of new onset symptomatic osteoarthritis, but the effects of overall diet quality were inconclusive.

CONCLUSIONS

The Prudent dietary pattern showed the highest protection on symptomatic osteoarthritis in adults aged 45 years and over. The body of evidence is limited, suggesting that further research is needed to corroborate the estimated effect at a high certainty of evidence, and to incorporate previously unstudied dietary patterns and food groups. Identifying the most beneficial dietary pattern may inform future guidelines for reducing symptomatic osteoarthritis in middle aged and older adults.

A framework for synthesizing intervention evidence from multiple sources into a single certainty of evidence rating: Methodological developments from a US National Academies of Sciences, Engineering, and Medicine Committee

Despite research investment and a growing body of diverse evidence there has been no comprehensive review and grading of evidence for public health emergency preparedness and response practices comparable to those in medicine and other public health fields. The National Academies of Sciences, Engineering, and Medicine convened an ad hoc committee to develop and use methods for grading and synthesizing diverse types of evidence to create a single certainty of intervention-related evidence to support recommendations for Public Health Emergency Preparedness and Response Research. A 13-step consensus building method was used. Experts were first canvassed in public meetings, and a comprehensive review of existing methods was undertaken. Although aspects of existing review methodologies and evidence grading systems were relevant, none adequately covered all requirements for this specific context. Starting with a desire to synthesize diverse sources of evidence not usually included in systematic reviews and using GRADE for assessing certainty and confidence in quantitative and qualitative evidence as the foundation, we developed a mixed-methods synthesis review and grading methodology that drew on (and in some cases adapted) those elements of existing frameworks and methods that were most applicable. Four topics were selected as test cases. The process was operationalized with a suite of method-specific reviews of diverse evidence types for each topic. Further consensus building was undertaken through stakeholder engagement and feedback The NASEM committee's GRADE adaption for mixed-methods reviews will further evolve over time and has yet to be endorsed by the GRADE working group.

Using GRADE Evidence to Decision frameworks to support the process of health policy-making: an example application regarding taxation of sugar-sweetened beverages

BACKGROUND

Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Evidence to Decision (EtD) frameworks are well-known tools that enable guideline panels to structure the process of developing recommendations and making decisions in healthcare and public health. To date, they have not regularly been used for health policy-making. This article aims to illustrate the application of the GRADE EtD frameworks in the process of nutrition-related policy-making for a European country.

METHODS

Based on methodological guidance by the GRADE Working Group and the findings of our recently published scoping review, we illustrate the process of moving from evidence to recommendations, by applying the EtD frameworks to a fictitious example. Sugar-sweetened beverage (SSB) taxation based on energy density was chosen as an example application.

RESULTS

A fictitious guideline panel was convened by a national nutrition association to develop a population-level recommendation on SSB taxation aiming to reduce the burden of overweight and obesity. Exemplary evidence was summarized for each EtD criterion and conclusions were drawn based on all judgements made in relation to each criterion. As a result of the high priority to reduce the burden of obesity and because of the moderate desirable effects on health outcomes, but considering scarce or varying research evidence for other EtD criteria, the panel made a conditional recommendation for SSB taxation. Decision-makers may opt for conducting a pilot study prior to implementing the policy on a national level.

CONCLUSIONS

GRADE EtD frameworks can be used by guideline panels to make the process of developing recommendations in the field of health policy more systematic, transparent and comprehensible.

Research and implementation interactions in a social accountability study: utilizing guidance for conducting process evaluations of complex interventions

BACKGROUND

In recent years, researchers and evaluators have made efforts to identify and use appropriate and innovative research designs that account for the complexity in studying social accountability. The relationship between the researchers and those implementing the activities and how this impacts the study have received little attention. In this paper, we reflect on how we managed the relationship between researchers and implementers using the United Kingdom Medical Research Council (MRC) guidance on process evaluation of a complex intervention.

MAIN BODY

The MRC guidance focuses on three areas of interaction between researchers and stakeholders involved in developing and delivering the intervention: (i) working with program developers and implementers; (ii) communication of emerging findings between researchers/evaluators and implementers; and (iii) overlapping roles of the intervention and research/evaluation. We summarize how the recommendations for each of the three areas were operationalized in the Community and Provider driven Social Accountability Intervention (CaPSAI) Project and provide reflections based on experience. We co-developed various tools, including standard operating procedures, contact lists, and manuals. Activities such as training sessions, regular calls, and meetings were also conducted to enable a good working relationship between the different partners.

CONCLUSIONS

Studying social accountability requires the collaboration of multiple partners that need to be planned to ensure a good working relationship while safeguarding both the research and intervention implementation. The MRC guidance is a useful tool for making interaction issues explicit and establishing procedures. Planning procedures for dealing with research and implementers' interactions could be more comprehensive and better adapted to social accountability interventions if both researchers and implementers are involved. There is a need for social accountability research to include clear statements explaining the nature and types of relationships between researchers and implementers involved in the intervention.

Randomized controlled trials in de-implementation research: a systematic scoping review

BACKGROUND

Healthcare costs are rising, and a substantial proportion of medical care is of little value. De-implementation of low-value practices is important for improving overall health outcomes and reducing costs. We aimed to identify and synthesize randomized controlled trials (RCTs) on de-implementation interventions and to provide guidance to improve future research.

METHODS

MEDLINE and Scopus up to May 24, 2021, for individual and cluster RCTs comparing de-implementation interventions to usual care, another intervention, or placebo. We applied independent duplicate assessment of eligibility, study characteristics, outcomes, intervention categories, implementation theories, and risk of bias.

RESULTS

Of the 227 eligible trials, 145 (64%) were cluster randomized trials (median 24 clusters; median follow-up time 305 days), and 82 (36%) were individually randomized trials (median follow-up time 274 days). Of the trials, 118 (52%) were published after 2010, 149 (66%) were conducted in a primary care setting, 163 (72%) aimed to reduce the use of drug treatment, 194 (85%) measured the total volume of care, and 64 (28%) low-value care use as outcomes. Of the trials, 48 (21%) described a theoretical basis for the intervention, and 40 (18%) had the study tailored by context-specific factors. Of the de-implementation interventions, 193 (85%) were targeted at physicians, 115 (51%) tested educational sessions, and 152 (67%) multicomponent interventions. Missing data led to high risk of bias in 137 (60%) trials, followed by baseline imbalances in 99 (44%), and deficiencies in allocation concealment in 56 (25%).

CONCLUSIONS

De-implementation trials were mainly conducted in primary care and typically aimed to reduce low-value drug treatments. Limitations of current de-implementation research may have led to unreliable effect estimates and decreased clinical applicability of studied de-implementation strategies. We identified potential research gaps, including de-implementation in secondary and tertiary care settings, and interventions targeted at other than physicians. Future trials could be improved by favoring simpler intervention designs, better control of potential confounders, larger number of clusters in cluster trials, considering context-specific factors when planning the intervention (tailoring), and using a theoretical basis in intervention design.

REGISTRATION

OSF Open Science Framework hk4b2.

Guideline development in harm reduction: Considerations around the meaningful involvement of people who access services

Background

Harm reduction seeks to minimizes the negative effects of drug use while respecting the rights of people with lived and living experience of substance use (PWLLE). Guideline standards ("guidelines for guidelines") provide direction on developing healthcare guidelines. To identify essential considerations for guideline development within harm reduction, we examined whether guideline standards are consistent with a harm reduction approach in their recommendations on involving people who access services.

Methods

We searched the literature from 2011-2021 to identify guideline standards used in harm reduction and publications on involving PWLLE in developing harm reduction services. We used thematic analysis to compare their guidance on involving people who access services. Findings were validated with two organizations of PWLLE.

Results

Six guideline standards and 18 publications met inclusion criteria. We identified three themes related to involving people who access services: Reasons for Involvement, Methods of Involvement, and Factors in Success. Subthemes varied across the literature. We identified five essential considerations for guideline development in harm reduction: establishing a shared understanding of reasons for involving PWLLE; respecting their expertise; partnering with PWLLE to ensure appropriate engagement; incorporating perspectives of populations disproportionately affected by substance use; and securing resources.

Conclusion

Guideline standards and the harm reduction literature approach the involvement of people who access services from different perspectives. Thoughtful integration of the two paradigms can improve guidelines while empowering PWLLE. Our findings can support the development of high-quality guidelines that align with the fundamental principles of harm reduction in their involvement of PWLLE.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Effects of goal-oriented care for adults with multimorbidity: A systematic review and meta-analysis

OBJECTIVE

To systematically review the evidence from randomized controlled trials comparing the effects of goal-oriented care against standard care for multimorbid adults.

DATA SOURCES/STUDY SETTING

The literature presenting the results of randomized trials assessing the outcomes of goal-oriented care compared with usual care for adults with multimorbidity.

STUDY DESIGN

Systematic review and meta-analysis.

DATA COLLECTION/EXTRACTION METHODS

We searched the Cochrane Database of Systematic Reviews (CENTRAL), EMBASE, MEDLINE, CINHAL, trial registries such as ClinicalTrial.gov and World Health Organization International Clinical Trials Registry Platform (ICTRP), and the references of eligible trials and relevant reviews. Goal-oriented care was defined as an approach that engages patients, establishes personal goals, and sets targets for patients and clinicians to plan a course of action and measure outcome. We reviewed 228 trials, and 12 were included. We extracted outcome data on quality of life, hospital admission, patients' satisfaction, patient and caregiver burden. Risk of bias was assessed and certainty of evidence was evaluated using GRADE.

PRINCIPAL FINDINGS

No study was fully free of bias. No effect was found on quality of life (standardized mean difference [SMD]: 0.05; 95% CI: -0.05 to 0.16) and hospital admission (risk ratio [RR]: 0.87; 95% CI: 0.65 to 1.17). There was a very small effect for patients' satisfaction (SMD: 0.15; 95% CI: 0.00 to 0.29) and caregiver burden (SMD: -0.13; 95% CI: -0.26 to 0.00). Certainty of evidence was low for all outcomes.

CONCLUSIONS

No firm conclusions can be reached about the effects of goal-oriented care for multimorbid adults. Future research should overcome the shortcomings of trials assessed in this meta-analysis. Sound application of the indications for research of complex healthcare interventions is warranted.

Strengthening women's empowerment and gender equality in fragile contexts towards peaceful and inclusive societies: A systematic review and meta-analysis

BACKGROUND

Across the globe, gender disparities still exist with regard to equitable access to resources, participation in decision-making processes, and gender and sexual-based violence. This is particularly true in fragile and conflict-affected settings, where women and girls are affected by both fragility and conflict in unique ways. While women have been acknowledged as key actors in peace processes and post-conflict reconstruction (e.g., through the United Nations Security Council Resolution 1325 and the Women, Peace and Security Agenda) evidence on the effectiveness of gender-specific and gender-transformative interventions to improve women's empowerment in fragile and conflict-affected states and situations (FCAS) remains understudied.

OBJECTIVES

The purpose of this review was to synthesize the body of evidence around gender-specific and gender-transformative interventions aimed at improving women's empowerment in fragile and conflict-affected settings with high levels of gender inequality. We also aimed to identify barriers and facilitators that could affect the effectiveness of these interventions and to provide implications for policy, practice and research designs within the field of transitional aid.

METHODS

We searched for and screened over 100,000 experimental and quasi-experimental studies focused on FCAS at the individual and community levels. We used standard methodological procedures outlined by the Campbell Collaboration for the data collection and analysis, including quantitative and qualitative analyses, and completed the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) methodology to assess the certainty around each body of evidence.

RESULTS

We identified 104 impact evaluations (75% randomised controlled trials) assessing the effects of 14 different types of interventions in FCAS. About 28% of included studies were assessed as having a high risk of bias (45% among quasi-experimental designs). Interventions supporting women's empowerment and gender equality in FCAS produced positive effects on the outcomes related to the primary focus of the intervention. There are no significant negative effects of any included interventions. However, we observe smaller effects on behavioural outcomes further along the causal chain of empowerment. Qualitative syntheses indicated that gender norms and practices are potential barriers to intervention effectiveness, while working with local powers and institutions can facilitate the uptake and legitimacy of interventions.

CONCLUSIONS

We observe gaps of rigorous evidence in certain regions (notably MENA and Latin America) and in interventions specifically targeting women as actors of peacebuilding. Gender norms and practices are important elements to consider in programme design and implementation to maximise potential benefits: focusing on empowerment only might not be enough in the absence of targeting the restrictive gender norms and practices that may undermine intervention effectiveness. Lastly, programme designers and implementation should consider explicitly targeting specific empowerment outcomes, promoting social capital and exchange, and tailoring the intervention components to the desired empowerment-related outcomes.

The Effectiveness and Acceptability of Comprehensive and Multicomponent School Health Services: A Systematic Review

PURPOSE

Comprehensive school-based health services were reviewed as part of preparing World Health Organization guidelines.

METHODS

Six databases were searched for controlled trials with schoolchildren aged 5-19 years, involving school-linked health services incorporating 4 of 7 health areas.

RESULTS

Seventeen quasi-experimental studies conducted in high-income countries found school health services associated with reductions in suicide planning (male: 7.1% vs. 7.7%, p < .01), hospitalization (relative risks 3.403, 95% confidence interval [CI] 1.536 to 8.473, p < .05), emergency department visits (odds ratio .85, 95% CI .75 to .95, p = .006), school absence (odds ratio .78, 95% CI .69 to .87, p < .0001), carrying weapons (male: 16.1% vs. 25.1%, p < .01), fighting (male: 32.6% vs. 43.1%, p < .01), sexual activity (53.5% vs. 60.5%, p < .05), drinking alcohol (60.1% vs. 70.5%, p < .001), using drugs (28.0% vs. 38.3%, p < .001), and physical activity (female: 57.4% vs. 50.4%, p < .01). They can be cost-saving, were also associated with smoking and less contraceptive use, and had no effect on depression prevalence. Acceptability/satisfaction was good. The certainty of evidence was low to moderate. Heterogeneity was high for some outcomes.

CONCLUSIONS

Results suggest moderate desirable and trivial undesirable effects of school health services in high-income countries. This article compiles evidence on effectiveness, cost-effectiveness, and acceptability into one review, enabling decision-makers to consider all essential factors.

Weight loss interventions and nonalcoholic fatty liver disease: Optimizing liver outcomes

The growth in prevalence of obesity, type 2 diabetes and nonalcoholic fatty liver disease (NAFLD) has become one of the most important global health challenges. The three chronic diseases are closely linked in their epidemiology and pathophysiology. Currently, weight loss is the most effective treatment for NAFLD (even in the minority of patients with NAFLD who do not have obesity) and is recommended in all national and international guidelines. Accumulating evidence has shown that weight loss, whether achieved by diet and lifestyle interventions, bariatric surgery or pharmacotherapy, can improve biomarkers of NAFLD, as well as prevent progression and, in some cases, reverse fibrosis. There is a dose dependency of weight loss with NAFLD improvement. Pharmacotherapy with antiobesity medications, alone or in combination with intensive lifestyle interventions or other weight-loss drugs, is closing the efficacy gap between diet and exercise and weight-loss surgery in efficacy at reversing obesity. Given the importance of providing effective weight-loss treatment to patients with NAFLD, weight management services need to be made increasingly available and embedded within hepatology services. This narrative review addresses the evidence that weight loss optimizes liver outcomes in people with NAFLD.

Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

BACKGROUND

The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

METHODS

There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

RESULTS

Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

LIMITATIONS

The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

CONCLUSIONS

This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

FUTURE WORK

We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

FUNDING

This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

PROTOCOL: Strengthening women's empowerment and gender equality in fragile contexts towards peaceful and inclusive societies: A systematic review and meta-analysis

This review builds on 3ie's (international initiative for impact evaluation) evidence gap map (EGM) of the impact evaluation and systematic review (SR) evidence base of interventions aiming to promote peaceful and inclusive societies in fragile contexts. The EGM identified a cluster of studies evaluating gender equality-focused behaviour change communication programmes and raised interest in investigating the evidence base for understanding the role of women more broadly as agents of change in developing peaceful and inclusive societies. Building on the cluster of evidence identified in the EGM, our review will increase generalisability of findings from single studies and focus on interventions across a broad range of geographical locations, settings and populations, types of implementations and outcomes. We will also address (when possible) the identified gaps in literature regarding metaanalysis in conflict-affected contexts. As such, we propose the following objectives: (1) The primary objective of this review is to identify, assess and synthesise evidence on the effect of gender specific and gender transformative interventions within the context of the four pillars of United Nations Security Council Resolution (UNSCR) 1325 on women's empowerment and gender equality in Fragile and Conflict Affected States/Situations (FCAS). The SR will facilitate the use of evidence in informing policy and practice decisions within the field of transition aid, particularly as it relates to gender focused programming. (2) Our second objective is to assess how these interventions contribute to inclusive and sustainable peace in conflict affected situations. We will compare the effectiveness of these different types of interventions through the lenses of their ecological level, types of impact on women's empowerment, local context of gender inequality and conflict. To achieve these objectives we aim to answer the following questions: (1) What are the impacts of gender transformative and specific interventions on women's empowerment and gender equality in FCAS? (2) What are the effects of these interventions on sustainable peace? (3) To what extent do effects vary by population group, ecological level and types of interventions? (4) What are contextual barriers to and facilitators of intervention effectiveness?

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

A review of methods for addressing components of interventions in meta-analysis

Many healthcare interventions are complex, consisting of multiple, possibly interacting, components. Several methodological articles addressing complex interventions in the meta-analytical context have been published. We hereby provide an overview of methods used to evaluate the effects of complex interventions with meta-analytical models. We summarized the methodology, highlighted new developments, and described the benefits, drawbacks, and potential challenges of each identified method. We expect meta-analytical methods focusing on components of several multicomponent interventions to become increasingly popular due to recently developed, easy-to-use, software tools that can be used to conduct the relevant analyses. The different meta-analytical methods are illustrated through two examples comparing psychotherapies for panic disorder.

Challenges in applying the GRADE approach in public health guidelines and systematic reviews: a concept article from the GRADE Public Health Group

BACKGROUND AND OBJECTIVE

This article explores the need for conceptual advances and practical guidance in the application of the GRADE approach within public health contexts.

METHODS

We convened an expert workshop and conducted a scoping review to identify challenges experienced by GRADE users in public health contexts. We developed this concept article through thematic analysis and an iterative process of consultation and discussion conducted with members electronically and at three GRADE Working Group meetings.

RESULTS

Five priority issues can pose challenges for public health guideline developers and systematic reviewers when applying GRADE: (1) incorporating the perspectives of diverse stakeholders; (2) selecting and prioritizing health and "nonhealth" outcomes; (3) interpreting outcomes and identifying a threshold for decision-making; (4) assessing certainty of evidence from diverse sources, including nonrandomized studies; and (5) addressing implications for decision makers, including concerns about conditional recommendations. We illustrate these challenges with examples from public health guidelines and systematic reviews, identifying gaps where conceptual advances may facilitate the consistent application or further development of the methodology and provide solutions.

CONCLUSION

The GRADE Public Health Group will respond to these challenges with solutions that are coherent with existing guidance and can be consistently implemented across public health decision-making contexts.

When complexity matters: a step-by-step guide to incorporating a complexity perspective in guideline development for public health and health system interventions

BACKGROUND

Guidelines on public health and health system interventions often involve considerations beyond effectiveness and safety to account for the impact that these interventions have on the wider systems in which they are implemented. This paper describes how a complexity perspective may be adopted in guideline development to facilitate a more nuanced consideration of a range of factors pertinent to decisions regarding public health and health system interventions. These factors include acceptability and feasibility, and societal, economic, and equity and equality implications of interventions.

MAIN MESSAGE

A 5-step process describes how to incorporate a complexity perspective in guideline development with examples to illustrate each step. The steps include: (i) guideline scoping, (ii) formulating questions, (iii) retrieving and synthesising evidence, (iv) assessing the evidence, and (v) developing recommendations. Guideline scoping using stakeholder consultations, complexity features, evidence mapping, logic modelling, and explicit decision criteria is emphasised as a key step that informs all subsequent steps.

CONCLUSIONS

Through explicit consideration of a range of factors and enhanced understanding of the specific circumstances in which interventions work, a complexity perspective can yield guidelines with better informed recommendations and facilitate local adaptation and implementation. Further work will need to look into the methods of collecting and assessing different types of evidence beyond effectiveness and develop procedural guidance for prioritising across a range of decision criteria.

Virtual follow-up and care for patients with cardiac electronic implantable devices: protocol for a systematic review

BACKGROUND

Capacity to deliver outpatient care for patients with cardiac implantable electronic devices (CIEDs) may soon be outweighed by need. This systematic review aims to investigate the comparative effectiveness, safety, and cost for virtual or remote clinic interventions for patients with CIEDs and explores how outcomes may be influenced by patient or system factors in-depth.

METHODS

We will perform a systematic literature search in MEDLINE, Embase, PsycINFO, CINAHL, Proquest Dissertations & Theses, other EBM Reviews, and trial registry databases. Two authors will independently screen titles and abstracts for eligibility. We will include randomized and non-randomized controlled trials, quasi-randomized and experimental studies, cohort, and case-control studies. Study populations of interest are individuals with a CIED (pacemaker, ICD, CRT). Eligibility will be restricted to virtual or remote follow-up or care interventions compared to any other approach. The co-primary outcomes of interest are mortality and patient satisfaction. Secondary outcomes include clinical effectiveness (e.g., ICD shock, time-to-detection of medical event, hospitalizations), safety (e.g., serious or device-related adverse events), device efficacy (e.g., transmissions, malfunctions), costs, workflow (e.g., resources, process outcomes, time-saved), and patient reported (e.g., burden, quality of life). Data will be extracted by one author and checked by a second using a standardized template. We will use published frameworks to capture data relevant to intervention effects that may be influenced by intervention definition or complexity, context and setting, or in socially disadvantaged populations. Detailed descriptive results will be presented for all included studies and outcomes, and where feasible, synthesized using meta-analysis. Risk of bias will be assessed by two review authors independently using Cochrane Risk of Bias tools. Certainty of evidence will be assessed using the GRADE approach.

DISCUSSION

Increases in number of CIEDs implanted, combined with an aging population and finite health resource allocations at the system-level may lead to increased reliance on virtual follow-up or care models in the future. These models must prioritize consistent, equitable, and timely care as a priority. Results from this systematic review will provide important insight into the potential contextual factors which moderate or mediate the effectiveness, safety, and cost of virtual follow-up or care models for patients.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO registration number CRD42020145210.

Patient and healthcare provider reported barriers and enablers to virtual or remote-only follow-up models for cardiovascular implantable electronic devices: protocol for a qualitative framework synthesis

BACKGROUND

Virtual care models are used to follow-up patients with cardiovascular implantable electronic devices (CIED), including pacemakers, implantable cardioverter defibrillators, and cardiac resynchronization therapy. There is increasing interest in the expansion of virtual, or even remote-only, CIED care models to alleviate resource and economic burden to both patients and specialty device clinics and to maintain or improve equity and access to high-quality cardiovascular care. This qualitative framework synthesis aims to identify barriers and enablers to virtual care models from both the perspective of the patient and device clinics. How setting, context, equity factors or other aspects influence these factors, or satisfaction with care, will also be investigated.

METHODS

We will perform a systematic literature search in MEDLINE, Embase, PsycINFO, CINAHL, Proquest Dissertations & Theses, other EBM Reviews, and trial registry databases. Screening will be completed by two independent review authors. Original research articles having a qualitative component (i.e., qualitative, mixed-, or multi-method) are eligible. Study populations of interest are (a) individuals with a CIED or (b) healthcare providers involved in any aspect of virtual or remote follow-up of patients with CIEDs. Eligibility will be restricted to studies published after January 1, 2000 in English or French. Data will be captured using standardized templates based on the domains and constructs of the Theoretical Domains Framework and the Warwick Patient Experiences Framework. The Joanna Briggs Institute Critical Appraisal Checklist for Qualitative Research will be applied to all included studies. The GRADE-CERQual approach will be applied to assess and summarize confidence in key findings. Reporting will follow the enhancing transparency in reporting the synthesis of qualitative research (ENTREQ) statement. Detailed descriptive results will be presented, and summary of qualitative findings tables will be produced.

DISCUSSION

While a number of trials have captured the clinical effectiveness and safety of virtual follow-up for CIEDs, there has been less attention given to factors affecting use and implementation of remote care by patients and healthcare providers or satisfaction with care. Results from this qualitative framework synthesis will provide important lived experience data from both patients and healthcare providers which will be essential to incorporate in clinical guidelines.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42020160533.

Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies

BACKGROUND

Nutrition and physical activity policies have the potential to influence lifestyle patterns and reduce the burden of non-communicable diseases. In the world of health-related guidelines, GRADE (Grading of Recommendations, Assessment, Development and Evaluation) is the most widely used approach for assessing the certainty of evidence and determining the strength of recommendations. Thus, it is relevant to explore its usefulness also in the process of nutrition and physical activity policymaking and evaluation. The purpose of this scoping review was (i) to generate an exemplary overview of documents using the GRADE approach in the process of nutrition and physical activity policymaking and evaluation, (ii) to find out how the GRADE approach has been applied, and (iii) to explore which facilitators of and barriers to the use of GRADE have been described on the basis of the identified documents. The overarching aim of this work is to work towards improving the process of evidence-informed policymaking in the areas of dietary behavior, physical activity, and sedentary behavior.

METHODS

A scoping review was conducted according to current reporting standards. MEDLINE via Ovid, the Cochrane Library, and Web of Science were systematically searched up until 4 July 2019. Documents describing a body of evidence which was assessed for the development or evaluation of a policy, including documents labeled as "guidelines," or systematic reviews used to inform policymaking were included.

RESULTS

Thirty-six documents were included. Overall, 313 GRADE certainty of evidence ratings were identified in systematic reviews and guidelines; the strength of recommendations/policies was assessed in four documents, and six documents mentioned facilitators or barriers for the use of GRADE. The major reported barrier was the initial low starting level of a body of evidence from non-randomized studies when assessing the certainty of evidence.

CONCLUSION

This scoping review found that the GRADE approach has been used for policy evaluations, in the evaluation of the effectiveness of policy-relevant interventions (policymaking), as well as in the development of guidelines intended to guide policymaking. Several areas for future research were identified to explore the use of GRADE in health policymaking and evaluation.

How to Distinguish Best Evidence from Bias: A Basic Guide to Understanding a Systematic Review

A systematic review is a powerful tool to synthesize and show concise and robust evidence for clinical practice. Thus, the inclusion of biased, low-quality studies should be avoided, for otherwise, the resulting systematic review will not reflect the best medical evidence. Because the methodology of systematic review is relatively new, this review aims to present basic rules to guide the reader's interpretation of a systematic review.

Prioritising gender, equity, and human rights in a GRADE-based framework to inform future research on self care for sexual and reproductive health and rights

Introduction

In January 2019, the WHO reviewed evidence to develop global recommendations on self-care interventions for sexual and reproductive health and rights (SRHR). Identification of research gaps is part of the WHO guidelines development process, but reliable methods to do so are currently lacking with gender, equity and human rights (GER) infrequently prioritised.

Methods

We expanded a prior framework based on Grading of Evidence, Assessment, Development and Evaluation (GRADE) to include GER. The revised framework is applied systematically during the formulation of research questions and comprises: (1) assessment of the GRADE strength and quality rating of recommendations; (2) mandatory inclusion of research questions identified from a global stakeholder survey; and (3) selection of the GER standards and principles most relevant to the question through discussion and consensus. For each question, we articulated: (1) the most appropriate and robust study design; (2) an alternative pragmatic design if the ideal design was not feasible; and (3) the methodological challenges facing researchers through identifying potential biases.

Results

We identified 39 research questions, 7 overarching research approaches and 13 discrete feasible study designs. Availability and accessibility were most frequently identified as the GER standards and principles to consider when planning studies, followed by privacy and confidentiality. Selection and detection bias were the primary methodological challenges across mixed methods, quantitative and qualitative studies. A lack of generalisability potentially limits the use of study results with non-participation in research potentially highest in more vulnerable populations.

Conclusion

A framework based on GRADE that includes stakeholders’ values and identification of core GER standards and principles provides a practical, systematic approach to identifying research questions from a WHO guideline. Clear guidance for future studies will contribute to an anticipated ‘living guidelines’ approach within WHO. Foregrounding GER as a separate component of the framework is innovative but further elaboration to operationalise appropriate indicators for SRHR self-care interventions is required.

Interventions to reduce ambient air pollution and their effects on health: An abridged Cochrane systematic review

BACKGROUND

A broad range of interventions have been implemented to improve ambient air quality, and many of these have been evaluated. Yet to date no systematic review has been conducted to identify and synthesize these studies. In this systematic review, we assess the effectiveness of interventions in reducing ambient particulate matter air pollution and improving adverse health outcomes.

METHODS

We searched a range of electronic databases across multiple disciplines, as well as grey literature databases, trial registries, reference lists of included studies and the contents of relevant journals, through August 2016. Eligible for inclusion were randomized and cluster randomized controlled trials, as well as several non-randomized study designs often used for evaluating air quality interventions. We included studies that evaluated interventions targeting industrial, residential, vehicular and multiple sources, with respect to their effect on mortality, morbidity and the concentrations of particulate matter (PM - including PM₁₀, PM_2.5, coarse particulate matter and combustion-related PM), as well as several criteria pollutants, including ozone, carbon monoxide, nitrogen oxides, nitrogen dioxide, nitric oxide and sulphur dioxide. We did not restrict studies based on the population, setting or comparison. Two authors independently assessed studies for inclusion, extracted data and assessed risk of bias. We assessed risk of bias using the Graphic Appraisal Tool for Epidemiological studies (GATE) for correlation studies, as modified and employed by the UK National Institute for Health and Care Excellence. We synthesized evidence narratively, as well as graphically using harvest plots. We assessed the certainty of evidence using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system.

RESULTS

We included 42 studies assessing 38 unique interventions. These comprised a heterogeneous mix of interventions, including those aiming to address industrial sources (n = 5; e.g. the closure of a factory), residential sources (n = 7; e.g. coal ban), vehicular sources (n = 22; e.g. low emission zones), and multiple sources (n = 4; e.g. tailored measures that target both local traffic and industrial polluters). Evidence for effectiveness was mixed. Most included studies observed either no significant association or an association favoring the intervention, with little evidence that the assessed interventions might be harmful.

CONCLUSIONS

Given the heterogeneity across interventions, outcomes, and methods, it was difficult to derive overall conclusions regarding the effectiveness of interventions in terms of improved air quality or health. Some evidence suggests that interventions are associated with improvements in air quality and human health, with very little evidence suggesting interventions were harmful. The evidence base highlights the challenges related to establishing the effectiveness of specific air pollution interventions on outcomes. It also points to the need for improved study design and analysis methods, as well as more uniform evaluations. The prospective planning of evaluations and an evaluation component built into the design and implementation of interventions may also be particularly beneficial.

Interventions to reduce ambient particulate matter air pollution and their effect on health

BACKGROUND

Ambient air pollution is associated with a large burden of disease in both high-income countries (HICs) and low- and middle-income countries (LMICs). To date, no systematic review has assessed the effectiveness of interventions aiming to reduce ambient air pollution.

OBJECTIVES

To assess the effectiveness of interventions to reduce ambient particulate matter air pollution in reducing pollutant concentrations and improving associated health outcomes.

SEARCH METHODS

We searched a range of electronic databases with diverse focuses, including health and biomedical research (CENTRAL, Cochrane Public Health Group Specialised Register, MEDLINE, Embase, PsycINFO), multidisciplinary research (Scopus, Science Citation Index), social sciences (Social Science Citation Index), urban planning and environment (Greenfile), and LMICs (Global Health Library regional indexes, WHOLIS). Additionally, we searched grey literature databases, multiple online trial registries, references of included studies and the contents of relevant journals in an attempt to identify unpublished and ongoing studies, and studies not identified by our search strategy. The final search date for all databases was 31 August 2016.

SELECTION CRITERIA

Eligible for inclusion were randomized and cluster randomized controlled trials, as well as several non-randomized study designs, including controlled interrupted time-series studies (cITS-EPOC), interrupted time-series studies adhering to EPOC standards (ITS-EPOC), interrupted time-series studies not adhering to EPOC standards (ITS), controlled before-after studies adhering to EPOC standards (CBA-EPOC), and controlled before-after studies not adhering to EPOC standards (CBA); these were classified as main studies. Additionally, we included uncontrolled before-after studies (UBA) as supporting studies. We included studies that evaluated interventions to reduce ambient air pollution from industrial, residential, vehicular and multiple sources, with respect to their effect on mortality, morbidity and several air pollutant concentrations. We did not restrict studies based on the population, setting or comparison.

DATA COLLECTION AND ANALYSIS

After a calibration exercise among the author team, two authors independently assessed studies for inclusion, extracted data and assessed risk of bias. We conducted data extraction, risk of bias assessment and evidence synthesis only for main studies; we mapped supporting studies with regard to the types of intervention and setting. To assess risk of bias, we used the Graphic Appraisal Tool for Epidemiological studies (GATE) for correlation studies, as modified and employed by the Centre for Public Health Excellence at the UK National Institute for Health and Care Excellence (NICE). For each intervention category, i.e. those targeting industrial, residential, vehicular and multiple sources, we synthesized evidence narratively, as well as graphically using harvest plots.

MAIN RESULTS

We included 42 main studies assessing 38 unique interventions. These were heterogeneous with respect to setting; interventions were implemented in countries across the world, but most (79%) were implemented in HICs, with the remaining scattered across LMICs. Most interventions (76%) were implemented in urban or community settings.We identified a heterogeneous mix of interventions, including those aiming to address industrial (n = 5), residential (n = 7), vehicular (n = 22), and multiple sources (n = 4). Some specific interventions, such as low emission zones and stove exchanges, were assessed by several studies, whereas others, such as a wood burning ban, were only assessed by a single study.Most studies assessing health and air quality outcomes used routine monitoring data. Studies assessing health outcomes mostly investigated effects in the general population, while few studies assessed specific subgroups such as infants, children and the elderly. No identified studies assessed unintended or adverse effects.The judgements regarding the risk of bias of studies were mixed. Regarding health outcomes, we appraised eight studies (47%) as having no substantial risk of bias concerns, five studies (29%) as having some risk of bias concerns, and four studies (24%) as having serious risk of bias concerns. Regarding air quality outcomes, we judged 11 studies (31%) as having no substantial risk of bias concerns, 16 studies (46%) as having some risk of bias concerns, and eight studies (23%) as having serious risk of bias concerns.The evidence base, comprising non-randomized studies only, was of low or very low certainty for all intervention categories and primary outcomes. The narrative and graphical synthesis showed that evidence for effectiveness was mixed across the four intervention categories. For interventions targeting industrial, residential and multiple sources, a similar pattern emerged for both health and air quality outcomes, with essentially all studies observing either no clear association in either direction or a significant association favouring the intervention. The evidence base for interventions targeting vehicular sources was more heterogeneous, as a small number of studies did observe a significant association favouring the control. Overall, however, the evidence suggests that the assessed interventions do not worsen air quality or health.

AUTHORS' CONCLUSIONS

Given the heterogeneity across interventions, outcomes, and methods, it was difficult to derive overall conclusions regarding the effectiveness of interventions in terms of improved air quality or health. Most included studies observed either no significant association in either direction or an association favouring the intervention, with little evidence that the assessed interventions might be harmful. The evidence base highlights the challenges related to establishing a causal relationship between specific air pollution interventions and outcomes. In light of these challenges, the results on effectiveness should be interpreted with caution; it is important to emphasize that lack of evidence of an association is not equivalent to evidence of no association.We identified limited evidence for several world regions, notably Africa, the Middle East, Eastern Europe, Central Asia and Southeast Asia; decision-makers should prioritize the development and implementation of interventions in these settings. In the future, as new policies are introduced, decision-makers should consider a built-in evaluation component, which could facilitate more systematic and comprehensive evaluations. These could assess effectiveness, but also aspects of feasibility, fidelity and acceptability.The production of higher quality and more uniform evidence would be helpful in informing decisions. Researchers should strive to sufficiently account for confounding, assess the impact of methodological decisions through the conduct and communication of sensitivity analyses, and improve the reporting of methods, and other aspects of the study, most importantly the description of the intervention and the context in which it is implemented.

Implications of a complexity perspective for systematic reviews and guideline development in health decision making

There is growing interest in the potential for complex systems perspectives in evaluation. This reflects a move away from interest in linear chains of cause-and-effect, towards considering health as an outcome of interlinked elements within a connected whole. Although systems-based approaches have a long history, their concrete implications for health decisions are still being assessed. Similarly, the implications of systems perspectives for the conduct of systematic reviews require further consideration. Such reviews underpin decisions about the implementation of effective interventions, and are a crucial part of the development of guidelines. Although they are tried and tested as a means of synthesising evidence on the effectiveness of interventions, their applicability to the synthesis of evidence about complex interventions and complex systems requires further investigation. This paper, one of a series of papers commissioned by the WHO, sets out the concrete methodological implications of a complexity perspective for the conduct of systematic reviews. It focuses on how review questions can be framed within a complexity perspective, and on the implications for the evidence that is reviewed. It proposes criteria which can be used to determine whether or not a complexity perspective will add value to a review or an evidence-based guideline, and describes how to operationalise key aspects of complexity as concrete research questions. Finally, it shows how these questions map onto specific types of evidence, with a focus on the role of qualitative and quantitative evidence, and other types of information.

Synthesising quantitative evidence in systematic reviews of complex health interventions

Public health and health service interventions are typically complex: they are multifaceted, with impacts at multiple levels and on multiple stakeholders. Systematic reviews evaluating the effects of complex health interventions can be challenging to conduct. This paper is part of a special series of papers considering these challenges particularly in the context of WHO guideline development. We outline established and innovative methods for synthesising quantitative evidence within a systematic review of a complex intervention, including considerations of the complexity of the system into which the intervention is introduced. We describe methods in three broad areas: non-quantitative approaches, including tabulation, narrative and graphical approaches; standard meta-analysis methods, including meta-regression to investigate study-level moderators of effect; and advanced synthesis methods, in which models allow exploration of intervention components, investigation of both moderators and mediators, examination of mechanisms, and exploration of complexities of the system. We offer guidance on the choice of approach that might be taken by people collating evidence in support of guideline development, and emphasise that the appropriate methods will depend on the purpose of the synthesis, the similarity of the studies included in the review, the level of detail available from the studies, the nature of the results reported in the studies, the expertise of the synthesis team and the resources available.

The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective

INTRODUCTION

Evidence-to-decision (EtD) frameworks intend to ensure that all criteria of relevance to a health decision are systematically considered. This paper, part of a series commissioned by the WHO, reports on the development of an EtD framework that is rooted in WHO norms and values, reflective of the changing global health landscape, and suitable for a range of interventions and complexity features. We also sought to assess the value of this framework to decision-makers at global and national levels, and to facilitate uptake through suggestions on how to prioritise criteria and methods to collect evidence.

METHODS

In an iterative, principles-based approach, we developed the framework structure from WHO norms and values. Preliminary criteria were derived from key documents and supplemented with comprehensive subcriteria obtained through an overview of systematic reviews of criteria employed in health decision-making. We assessed to what extent the framework can accommodate features of complexity, and conducted key informant interviews among WHO guideline developers. Suggestions on methods were drawn from the literature and expert consultation.

RESULTS

The new WHO-INTEGRATE (INTEGRATe Evidence) framework comprises six substantive criteria-balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations-and the meta-criterion quality of evidence. It is intended to facilitate a structured process of reflection and discussion in a problem-specific and context-specific manner from the start of a guideline development or other health decision-making process. For each criterion, the framework offers a definition, subcriteria and example questions; it also suggests relevant primary research and evidence synthesis methods and approaches to assessing quality of evidence.

CONCLUSION

The framework is deliberately labelled version 1.0. We expect further modifications based on focus group discussions in four countries, example applications and input across concerned disciplines.

Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods

Guideline developers are increasingly dealing with more difficult decisions concerning whether to recommend complex interventions in complex and highly variable health systems. There is greater recognition that both quantitative and qualitative evidence can be combined in a mixed-method synthesis and that this can be helpful in understanding how complexity impacts on interventions in specific contexts. This paper aims to clarify the different purposes, review designs, questions, synthesis methods and opportunities to combine quantitative and qualitative evidence to explore the complexity of complex interventions and health systems. Three case studies of guidelines developed by WHO, which incorporated quantitative and qualitative evidence, are used to illustrate possible uses of mixed-method reviews and evidence. Additional examples of methods that can be used or may have potential for use in a guideline process are outlined. Consideration is given to the opportunities for potential integration of quantitative and qualitative evidence at different stages of the review and guideline process. Encouragement is given to guideline commissioners and developers and review authors to consider including quantitative and qualitative evidence. Recommendations are made concerning the future development of methods to better address questions in systematic reviews and guidelines that adopt a complexity perspective.