Leading by example: a culture change at WHO

Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization

This study explored experiences, perceptions and views among World Health Organization (WHO) staff about the changes, progress and challenges brought by the guideline development reforms initiated in 2007. Thirty-five semistructured interviews were conducted with senior WHO staff. Sixteen of the interviewees had in-depth experience with WHO's formal guideline development process. Thematic analysis was conducted to identify key themes in the qualitative data, and these were interpreted in the context of the existing literature on WHO's guideline development processes. First, the reforms were seen to have transformed and improved the quality of WHO's guidelines. Second, independent evaluation and feedback by the Guidelines Review Committee (GRC) was described to have strengthened the legitimacy of WHO's recommendations. Third, WHO guideline development processes are not yet designed to systematically make use of all types of research evidence needed to inform decisions about health systems and public health interventions. For example, several interviewees expressed dissatisfaction with the insufficient attention paid to qualitative evidence and evidence from programme experience, and how the Grading of Recommendations Assessment, Development and Evaluation (GRADE) process evaluates the quality of evidence from non-randomised study designs, while others believed that GRADE was just not properly understood or applied. Fourth, some staff advocated for a more centralised quality assurance process covering all outputs from WHO's departments and scientific advisory committees, especially to eliminate strategic efforts aimed at bypassing the GRC's requirements. Overall, the 'culture change' senior WHO staff called for over 10 years ago appears to have gradually spread throughout the organisation. However, at least two major challenges remain: (1) ensuring that all issued advice benefits from independent evaluation, monitoring and feedback for quality and (2) designing guideline development processes to better acquire, assess, adapt and apply the full range of evidence that can inform recommendations on health systems and public health interventions.

Scientific Advisory Committees at the World Health Organization: A Qualitative Study of How Their Design Affects Quality, Relevance, and Legitimacy

Governments and international organizations frequently convene scientific advisory committees (SACs) to support decision-making with scientific advice. In this study, thematic analysis of interviews with 35 senior WHO staff identified five main themes characterizing WHO's experience with designing SACs to ensure quality, relevance, and legitimacy of scientific advice. First, in addition to technical matters, SACs are established to serve broader strategic objectives, including consensus building to promote high-level political messages. Second, for SACs to be fully independent, they must have autonomy from the institutions convening or funding them, from the institutions from where SAC members are recruited, and from the institutions to whom the advice is directed. Third, since choices affecting quality, relevance, and legitimacy are closely linked, designing SACs often require trade-offs among these three attributes. Fourth, staff supporting SACs need to balance between safeguarding SACs from external influence and being receptive to the external political environment. Fifth, the design of SACs need to balance the involvement of stakeholders with the power to act on recommendations against the need to protect the independence and integrity of the scientific process. Overall, this study highlights key choices conveners of SACs must make when seeking to promote quality, relevance, and legitimacy of scientific advice.

A very low number of national adaptations of the World Health Organization guidelines for HIV and tuberculosis reported their processes

BACKGROUND

Low- and middle-income countries adapt World Health Organization (WHO) guidelines instead of de novo development for financial, epidemiologic, sociopolitical, cultural, organizational, and other reasons.

OBJECTIVE

To systematically evaluate reported processes used in the adaptation of WHO guidelines for human immunodeficiency virus (HIV) and tuberculosis (TB).

METHODS

We searched three online databases/repositories: United States Agency for International Development (USAID) AIDS Support and Technical Resources - Sector One program (AIDSTAR-One) National Treatment Database; the AIDSspace Guideline Repository, and WHO Database of national HIV and TB guidelines. We assessed the rigor and quality of reported adaptation methodology using the ADAPTE process as benchmark.

RESULTS

Of 170 eligible guidelines, only 32 (19%) reported documentation on the adaptation process. The median and interquartile range of the number of ADAPTE steps fulfilled by the eligible guidelines were 11.5 (10, 13.5) (out of 23 steps). The number of guidelines (out of 32 steps) fulfilling each ADAPTE step was 18 (interquartile range, 5-27). Seventeen of 32 guidelines (53%) met all steps relevant to the setup phase, whereas none met all steps relevant to the adaptation phase.

CONCLUSION

The number of well-documented adaptation methodologies in national HIV and/or TB guidelines is very low. There is a need for the use of standardized and systematic framework for guideline adaptation and improved reporting of processes used.

Quality Assessment of Clinical Practice Guidelines Developed by Professional Societies in Turkey

Background

Clinical practice guidelines (CPGs) are systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances. There is a limited number of studies on guidelines in Turkey. The quality of Ministry of Health guidelines have formerly been assessed whereas there is no information on the other guidelines developed in the country.

Aim

This study aims to assess the quality of CPGs that are developed by professional societies that work for the health sector in Turkey, and compare the findings with international guidelines.

Methodology

Professional societies that work for the health sector were determined by using the data obtained from the Ministry of Internal Affairs. Inclusion and exclusion criteria were defined for selecting the CPGs. Guidelines containing recommendations about disease management to the doctors, accessible online, developed within the past 5 years, citing references for recommendations, about the diseases over 1% prevalence according to the “Statistical Yearbook of Turkey 2012” were included in the study. The quality of CPGs were assessed with the AGREE II instrument, which is an internationally recognized tool for this purpose. Four independent reviewers, who did not participate in the development of the selected guidelines and were trained in CPG appraisal, used the AGREE instrument for assessment of the selected guidelines.

Findings

47 professional societies were defined which provided access to CPGs in their websites; 3 of them were only open to members so these could not be reached. 8 CPGs from 7 societies were selected from a total of 401 CPGs from 44 societies. The mean scores of the domains of the guidelines which were assessed by the AGREE II tool were; Scope and purpose: 64%, stakeholder involvement: 37.9%, rigour of development: 35.3%, clarity and presentation: 77.9%, applicability: 49.0% and editorial independence: 46.0%.

Conclusion

This is the first study in Turkey regarding quality appraisal of guidelines developed by the local professional societies. It adds to the limited amount of information in the literature that comes from Turkey as well as other developing countries.

The role of Cochrane reviews in informing international guidelines: a case study of using the Grading of Recommendations, Assessment, Development and Evaluation system to develop World Health Organization guidelines for the psychosocially assisted pharmacological treatment of opioid dependence

BACKGROUND AND AIMS

The World Health Organization (WHO), and a growing number of other organizations, have adopted the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system in order to both assess the quality of research evidence and develop clinical practice guidelines. In 2009 WHO published a guideline on psychosocially assisted pharmacological treatment of opioid dependence, based on the results of Cochrane Reviews summarized using the GRADE methodology. The main features of this system are an a priori definition of outcomes and their relevance, and distinction between the quality of evidence (also referred to as confidence in the estimate of intervention effect) and the strength of recommendations. We consider how successful this approach has been.

ANALYSIS AND EVIDENCE

We discuss the merits and limitations of using Cochrane Reviews and GRADE framework in developing guidelines in the field of drug addiction. In 2009 a panel of multi-disciplinary international experts identified 15 clinical questions and eight relevant outcomes. Cochrane reviews were available for each clinical question and four outcomes. The panel formulated 15 recommendations. Eight recommendations were classified as strong, two of which were based on high-quality evidence and three on very low-quality evidence. For example, the strong recommendation to use methadone in adequate doses in preference to buprenorphine was based on high-quality evidence, while the strong recommendation not to use the combination of opioid antagonists with heavy sedation in the management of opioid withdrawal was based on low-quality evidence.

CONCLUSIONS

An explicit stepwise process of moving from evaluation of the quality of evidence to the definition of the strength of recommendations is important in providing practical and clear clinical guidance for practitioners and policy-makers in addiction.

Appraisal of guidelines developed by the World Health Organization

OBJECTIVE

To appraise the quality of guidelines developed by the World Health Organization (WHO) that were approved by its Guidelines Review Committee (GRC) and identify strengths and weaknesses in the guideline development process.

STUDY DESIGN

Cross-sectional.

METHODS

Three individuals independently assessed GRC-approved WHO guidelines using the Appraisal of Guidelines for Research and Evaluation II instrument (AGREE II). Scores were standardized across domains and overall quality was determined through consensus.

RESULTS

124 guidelines met inclusion criteria and were assessed. 58 guidelines were recommended for use, 58 were recommended with modifications and eight were not recommended. The highest scoring domains across guidelines were scope and purpose, and clarity of presentation. The recommended guidelines had higher rigor of development and applicability domain scores in comparison to other guidelines. 77% of the guidelines referenced an underlying evidence review and 49% used GRADE to assess the body of evidence or the strength of the recommendation. The domains in need of improvement included stakeholder engagement, editorial independence, and applicability. Guidelines not recommended for use were generally insufficient in their rigor of development.

CONCLUSIONS

WHO guidelines need further improvement, most importantly in the rigor of their development (i.e., use of evidence reviews). Other areas for improvement include increased stakeholder engagement, a more explicit process for recommendation formulation and disclosure of interests, discussion of the facilitators, barriers, resource implications, and criteria for monitoring the outcomes of guideline implementation. WHO guidelines can improve through increased transparency, adherence to the WHO Handbook for Guideline Development, and better oversight by the GRC.

World Health Organization guideline development: an evaluation

BACKGROUND

Research in 2007 showed that World Health Organization (WHO) recommendations were largely based on expert opinion, rarely used systematic evidence-based methods, and did not follow the organization's own "Guidelines for Guidelines". In response, the WHO established a "Guidelines Review Committee" (GRC) to implement and oversee internationally recognized standards. We examined the impact of these changes on WHO guideline documents and explored senior staff's perceptions of the new procedures.

METHODS AND FINDINGS

We used the AGREE II guideline appraisal tool to appraise ten GRC-approved guidelines from nine WHO departments, and ten pre-GRC guidelines matched by department and topic. We interviewed 20 senior staff across 16 departments and analyzed the transcripts using the framework approach. Average AGREE II scores for GRC-approved guidelines were higher across all six AGREE domains compared with pre-GRC guidelines. The biggest changes were noted for "Rigour of Development" (up 37.6%, from 30.7% to 68.3%) and "Editorial Independence" (up 52.7%, from 20.9% to 73.6%). Four main themes emerged from the interviews: (1) high standards were widely recognized as essential for WHO credibility, particularly with regard to conflicts of interest; (2) views were mixed on whether WHO needed a single quality assurance mechanism, with some departments purposefully bypassing the procedures; (3) staff expressed some uncertainties in applying the GRADE approach, with departmental staff concentrating on technicalities while the GRC remained concerned the underlying principles were not fully institutionalized; (4) the capacity to implement the new standards varied widely, with many departments looking to an overstretched GRC for technical support.

CONCLUSIONS

Since 2007, WHO guideline development methods have become more systematic and transparent. However, some departments are bypassing the procedures, and as yet neither the GRC, nor the quality assurance standards they have set, are fully embedded within the organization.

Challenges in developing evidence-based recommendations using the GRADE approach: the case of mental, neurological, and substance use disorders

Corrado Barbui and colleagues describe their use and adaptation of the GRADE approach in developing the guidelines for the WHO mental health Gap Action Programme (mhGAP).

WHO policy development processes for a new vaccine: case study of malaria vaccines

Background

Recommendations from the World Health Organization (WHO) are crucial to inform developing country decisions to use, or not, a new intervention. This article analysed the WHO policy development process to predict its course for a malaria vaccine.

Methods

The decision-making processes for one malaria intervention and four vaccines were classified through (1) consultations with staff and expert advisors to WHO's Global Malaria Programme (GMP) and Immunization, Vaccines and Biologicals Department (IVB); (2) analysis of the procedures and recommendations of the major policy-making bodies of these groups; (3) interviews with staff of partnerships working toward new vaccine availability; and (4) review and analyses of evidence informing key policy decisions.

Case description

WHO policy formulation related to use of intermittent preventive treatment in infancy (IPTi) and the following vaccine interventions: Haemophilus influenzae type b conjugate vaccine (Hib), pneumococcal conjugate vaccine (PCV), rotavirus vaccine (RV), and human papillomavirus vaccine (HPV), five interventions which had relatively recently been through systematic WHO policy development processes as currently constituted, was analysed. Required information was categorized in three areas defined by a recent WHO publication on development of guidelines: safety and efficacy in relevant populations, implications for costs and population health, and localization of data to specific epidemiological situations.

Discussion and evaluation

Data needs for a malaria vaccine include safety; the demonstration of efficacy in a range of epidemiological settings in the context of other malaria prevention interventions; and information on potential rebound in which disease increases subsequent to the intervention. In addition, a malaria vaccine would require attention to additional factors, such as costs and cost-effectiveness, supply and demand, impact of use on other interventions, and distribution issues.

Conclusions

Although policy issues may be more complex for future vaccines, the lead-time between the date of product regulatory approval and a recommendation for its use in developing countries is decreasing. This study presents approaches to define in advance core data needs to support evidence-based decisions, to further decrease this lead-time, accelerating the availability of a malaria vaccine. Specific policy areas for which information should be collected are defined, including studying its use within the context of other malaria interventions.

Developing WHO guidelines with pragmatic, structured, evidence-based processes: A case study

Many guidelines, including those produced by the World Health Organisation (WHO), have failed to adhere to rigorous methodological standards. Operational examples of guideline development processes may provide important lessons learned to improve the rigour and quality of future guidelines. To this end, this paper describes the process of developing WHO guidelines on prevention and care interventions for adults and adolescents living with HIV. Using a pragmatic, structured, evidence-based approach, we created an organising committee, identified topics, conducted systematic reviews, identified experts and distributed evidence summaries. Subsequently, 55 global HIV experts drafted and anonymously submitted guideline statements at the beginning of a conference. During the conference, participants voted on statements using scales evaluating appropriateness of the statements, strength of recommendation and level of evidence. After review of voting results, open discussion, re-voting and refinement of statements, a draft version of the guidelines was completed. A post-conference writing team refined the guidelines based on pre-determined guideline writing principles and incorporated external comments into a final document. Successes and challenges of the guideline development process were identified and are used to highlight current issues and debates in developing guidelines with a focus on implications for future guideline development at WHO.

Evidence and rationale for the World Health Organization recommended standards for Japanese encephalitis surveillance

BACKGROUND

Japanese encephalitis (JE) is the most important form of viral encephalitis in Asia. Surveillance for the disease in many countries has been limited. To improve collection of accurate surveillance data in order to increase understanding of the full impact of JE and monitor control programs, World Health Organization (WHO) Recommended Standards for JE Surveillance have been developed. To aid acceptance of the Standards, we describe the process of development, provide the supporting evidence, and explain the rationale for the recommendations made in the document.

METHODS

A JE Core Working Group was formed in 2002 and worked on development of JE surveillance standards. A series of questions on specific topics was initially developed. A literature review was undertaken and the findings were discussed and documented. The group then prepared a draft document, with emphasis placed on the feasibility of implementation in Asian countries. A field test version of the Standards was published by WHO in January 2006. Feedback was then sought from countries that piloted the Standards and from public health professionals in forums and individual meetings to modify the Standards accordingly.

RESULTS

After revisions, a final version of the JE surveillance standards was published in August 2008. The supporting information is presented here together with explanations of the rationale and levels of evidence for specific recommendations.

CONCLUSION

Provision of the supporting evidence and rationale should help to facilitate successful implementation of the JE surveillance standards in JE-endemic countries which will in turn enable better understanding of disease burden and the impact of control programs.

SUPPORT Tools for evidence-informed health Policymaking (STP) 2: Improving how your organisation supports the use of research evidence to inform policymaking

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.

In this article, we address ways of organising efforts to support evidence-informed health policymaking. Efforts to link research to action may include a range of activities related to the production of research that is both highly relevant to – and appropriately synthesised for – policymakers. Such activities may include a mix of efforts used to link research to action, as well as the evaluation of such efforts. Little is known about how best to organise the range of activity options available and, until recently, there have been relatively few organisations responsible for supporting the use of research evidence in developing health policy. We suggest five questions that can help guide considerations of how to improve organisational arrangements to support the use of research evidence to inform health policy decision making. These are: 1. What is the capacity of your organisation to use research evidence to inform decision making? 2. What strategies should be used to ensure collaboration between policymakers, researchers and stakeholders? 3. What strategies should be used to ensure independence as well as the effective management of conflicts of interest? 4. What strategies should be used to ensure the use of systematic and transparent methods for accessing, appraising and using research evidence? 5. What strategies should be used to ensure adequate capacity to employ these methods?

Antimicrobial resistance: international control strategies, with a focus on limited-resource settings

Microorganisms resistant to multiple anti-infective agents have increased worldwide. These organisms threaten both optimal care of patients with infection as well as the viability of current healthcare systems. In addition, antimicrobials are valuable resources that enhance both prevention and treatment of infections. As resistance diminishes this resource, it is a societal goal to minimise resistance and therefore to reduce forces that produce resistance. This review considers strategies for minimising resistance that are needed at several different levels of responsibility, ranging from the patient care provider to international agencies. It then describes responses that might be appropriate according to the resources available for control, focusing on limited-resource settings. Antimicrobial resistance represents an international concern. Response to this problem demands concerted efforts from multiple sectors both in developed and developing countries, as well as the strengthening of multinational/international partnerships and regulations. Both medical and public health agencies should be in the forefront of these efforts.

Evidence-based tuberculosis diagnosis

Madhukar Pai and colleagues discuss how systematic reviews on tuberculosis diagnostics can influence research, policy, and clinical practice.

A cohort study to assess the new WHO Japanese encephalitis surveillance standards

OBJECTIVE

To assess the field-test version of the new WHO Japanese encephalitis (JE) surveillance standards.

METHODS

We applied the clinical case definition of acute encephalitis syndrome (AES), laboratory diagnostic criteria and case classifications to patients with suspected central nervous system (CNS) infections in southern Viet Nam.

FINDINGS

Of the 380 patients (149 children) recruited with suspected CNS infections, 296 (96 children) met the AES case definition. 54 children were infected with JE virus (JEV), of whom 35 (65%) had AES, giving a sensitivity of 65% (95% CI: 56-73) and specificity of 39% (95% CI: 30-48). Nine adults with JEV presented with AES. 19 JEV-infected children missed by surveillance included 10 with acute flaccid paralysis, two with flaccid hemiparesis and six with meningism only. Altering the case definition to include limb paralysis and meningism improved sensitivity to 89% (95% CI: 83-95), while reducing specificity to 23% (95% CI: 15-30). Six children that did not have AES on admission had reduced consciousness after admission. Cerebrospinal fluid (CSF) analysis diagnosed seven patients negative on serum analysis. Five patients with neurological manifestations of dengue infection had JEV antibodies in serum and would have been misdiagnosed had we not tested for dengue antibodies in parallel.

CONCLUSION

Children infected with JEV that presented with acute limb paralysis or neck stiffness only were missed by the surveillance standards, although some of them subsequently became encephalopathic. A footnote in the surveillance standards drawing attention to these presentations would be helpful. An acute CSF sample is more sensitive and specific than an acute serum sample.