Priority Setting in Guideline Development: Article 2 in Integrating and Coordinating Efforts in COPD Guideline Development. An Official ATS/ERS Workshop Report

Prioritization of clinical questions for the Australian Living Guideline for the Pharmacological Management of Inflammatory Arthritis

AIM

Living guidelines aim to reduce delays in translating new knowledge into practice by updating individual recommendations as soon as relevant new evidence emerges. We surveyed members of the Australian Rheumatology Association (ARA) to develop a list of priority questions for the Australian Living Guideline for the Pharmacological Management of Inflammatory Arthritis (ALG) and to explore clinicians' use of clinical practice guidelines.

METHODS

An electronic survey of ARA members was performed in two phases. The first survey contained questions about current guideline use and beliefs and invited participants to submit at least three questions relevant to the management of rheumatoid arthritis (RA). In the second round, participants selected 10 questions they considered to be the highest priority from the collated list and ranked them in priority order. The sum of ranks was used to generate a final priority list.

RESULTS

There were 115 (21%) and 78 (14%) responses to the first and second survey rounds respectively. 87% of respondents use existing rheumatology guidelines in their usual practice, primarily EULAR guidelines. Most respondents favored the development of Australian rheumatology guidelines. In total, 34 potential recommendation topics were identified and ranked in order of priority.

CONCLUSION

A list of 34 clinical questions about RA management, ranked in order of importance by clinicians, has informed the development of the ALG. Similar prioritization exercises in other contexts may permit guidelines to be tailored to the needs of guideline users in their specific context, which may facilitate international collaboration and promote efficient translation of evidence to practice.

Improving prioritization processes for clinical practice guidelines: new methods and an evaluation from the National Heart Foundation of Australia

BACKGROUND

Releasing timely and relevant clinical guidelines is challenging for organizations globally. Priority-setting is crucial, as guideline development is resource-intensive. Our aim, as a national organization responsible for developing cardiovascular clinical guidelines, was to develop a method for generating and prioritizing topics for future clinical guideline development in areas where guidance was most needed.

METHODS

Several novel processes were developed, adopted and evaluated, including (1) initial public consultation for health professionals and the general public to generate topics; (2) thematic and qualitative analysis, according to the International Classification of Diseases (ICD-11), to aggregate topics; (3) adapting a criteria-based matrix tool to prioritize topics; (4) achieving consensus through a modified-nominal group technique and voting on priorities; and (5) process evaluation via survey of end-users. The latter comprised the organization's Expert Committee of 12 members with expertise across cardiology and public health, including two citizen representatives.

RESULTS

Topics (n = 405; reduced to n = 278 when duplicates removed) were identified from public consultation responses (n = 107 respondents). Thematic analysis synthesized 127 topics that were then categorized into 37 themes using ICD-11 codes. Exclusion criteria were applied (n = 32 themes omitted), resulting in five short-listed topics: (1) congenital heart disease, (2) valvular heart disease, (3) hypercholesterolaemia, (4) hypertension and (5) ischaemic heart diseases and diseases of the coronary artery. The Expert Committee applied the prioritization matrix to all five short-listed topics during a consensus meeting and voted to prioritize topics. Unanimous consensus was reached for the topic voted the highest priority: ischaemic heart disease and diseases of the coronary arteries, resulting in the decision to update the organization's 2016 clinical guidelines for acute coronary syndromes. Evaluation indicated that initial public consultation was highly valued by the Expert Committee, and the matrix tool was easy to use and improved transparency in priority-setting.

CONCLUSION

Developing a multistage, systematic process, incorporating public consultation and an international classification system led to improved transparency in our clinical guideline priority-setting processes and that topics chosen would have the greatest impact on health outcomes. These methods are potentially applicable to other national and international organizations responsible for developing clinical guidelines.

Questions and health outcomes prioritization for the development of a COVID-19 dental clinical practice guideline: A case study

RATIONALE, AIMS AND OBJECTIVES

In the context of a pandemic, the rapid development of clinical practice guidelines (CPGs) is critical. The guideline development process includes prioritization of the guideline topic, questions and health outcomes. This case study describes the application of a new methodology to prioritize questions and rate the importance of health outcomes for a COVID-19 dental guideline.

METHODS

Panel members rated the topic and the questions' overall importance, using a 9-point scale (1 = least important; 9 = most important). In addition, they rated six criteria if multiple questions received the same overall importance rating: common in practice, uncertainty in practice, variation in practice, new evidence available, cost consequences, not previously addressed. Panellists also rated the importance of each outcome, defined with health outcome descriptors, using a 9-point scale and the utility of health outcomes on a visual analogue scale. The correlation between each criterion and overall question importance was tested by Spearman correlation coefficient.

RESULTS

Of seven topics, four were rated as high priority and three were rated as important, but not of high priority. Thirty-six percent of the questions (18/50) were rated as high priority to address in the guideline and 64% (32/50) were rated as an important question but not of high priority. Of the 11 outcomes, 72.7% were rated as critical for decision making. The mean utility rating was 0.57 (SD 0.32), with a minimum mean rating of 0.16 and a maximum of 0.76 (SD 0.23).

CONCLUSION

This case study demonstrated that this approach provides a rigorous and transparent methodology to conduct the prioritizations of guideline topics, questions and health outcomes.

Development and application of health outcome descriptors facilitated decision-making in the production of practice guidelines

OBJECTIVE

Stakeholders involved in developing recommendations need to have a common understanding of health outcomes and the perspective of affected individuals. In this paper we report on the development and application of health outcome descriptors (HODs) to inform decision-making by panels developing guideline recommendations.

STUDY DESIGN AND SETTING

Ten American Society of Hematology guideline panels addressing the management of venous thromboembolism developed HODs, rated their importance and health utility, applied them to prioritize outcomes, and to balance potential benefits and harms to formulate recommendations.

RESULTS

It was feasible to involve 18 panelists in developing 127 HODs. There was high agreement (82%) across the ten panels about outcomes perceived as critical or important for decision-making. Panelists' utility ratings of the outcomes were strongly correlated with panelists' outcome importance ratings (Pearson's r=-0.88). HODs were incorporated into Grading of Recommendations Assessment, Development and Evaluation (GRADE) evidence-to-decision (EtD) frameworks to support a shared understanding of health outcomes in panel deliberations.

CONCLUSION

HODs serve as a valuable tool to promote an explicit, common understanding of health outcomes during clinical guideline development and across different stakeholders. They are helpful across multiple steps of guideline development to facilitate panels' judgements, aiming to avoid variable implicit interpretations of health outcomes.

The implementation of prioritization exercises in the development and update of health practice guidelines: A scoping review

Background

The development of trustworthy guidelines requires substantial investment of resources and time. This highlights the need to prioritize topics for guideline development and update.

Objective

To systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines.

Methods

We searched Medline and CINAHL electronic databases from inception to July 2019, supplemented by hand-searching Google Scholar and the reference lists of relevant studies. We included studies describing prioritization exercises that have been conducted during the de novo development, update or adaptation of guidelines addressing clinical, public health or health systems topics. Two reviewers worked independently and in duplicate to complete study selection and data extraction. We consolidated findings in a semi-quantitative and narrative way.

Results

Out of 33,339 identified citations, twelve studies met the eligibility criteria. All included studies focused on prioritizing topics; none on questions or outcomes. While three exercises focused on updating guidelines, nine were on de novo development. All included studies addressed clinical topics. We adopted a framework that categorizes prioritization into 11 steps clustered in three phases (pre-prioritization, prioritization and post-prioritization). Four studies covered more than half of the 11 prioritization steps across the three phases. The most frequently reported steps for generating initial list of topics were stakeholders’ input (n = 8) and literature review (n = 7). The application of criteria to determine research priorities was used in eight studies. We used and updated a common framework of 22 prioritization criteria, clustered in 6 domains. The most frequently reported criteria related to the health burden of disease (n = 9) and potential impact of the intervention on health outcomes (n = 5). All the studies involved health care providers in the prioritization exercises. Only one study involved patients. There was a variation in the number and type of the prioritization exercises’ outputs.

Conclusions

This review included 12 prioritization exercises that addressed different aspects of priority setting for guideline development and update that can guide the work of researchers, funders, and other stakeholders seeking to prioritize guideline topics.

Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development

Patients and caregivers are increasingly recognized as key stakeholders in developing clinical practice guidelines (CPGs). Online engagement approaches offer the promise of a rigorous, scalable, and convenient engagement method. This paper illustrates how an online modified-Delphi approach could be used to engage patients, caregivers, and other stakeholder in CPG development. It provides practical guidance for conducting online modified-Delphi panels that covers (1) joint development of the engagement approach with relevant stakeholders, (2) adaptation of methods used by experts in guideline development, (3) pilot testing, (4) participant recruitment, (5) determining panel size and composition, (6) building participant capacity, (7) facilitation of two-way interaction, (8) continuous engagement and retention of participants, (9) rigorous data analysis, (10) evaluation of engagement activities, and (11) result dissemination. The paper is based on a recently completed study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining the patient-centeredness of DMD care guidelines.

Prioritization approaches in the development of health practice guidelines: a systematic review

Background

Given the considerable efforts and resources required to develop practice guidelines, developers need to prioritize what topics and questions to address. This study aims to identify and describe prioritization approaches in the development of clinical, public health, or health systems guidelines.

Methods

We searched Medline and CINAHL electronic databases in addition to Google Scholar. We included papers describing prioritization approaches in sufficient detail allowing for reproducibility. We synthesized findings in a semi-quantitative way. We followed an iterative process to develop a common framework of prioritization criteria that captures all of the criteria reported by each included study.

Results

Our search captured 33,339 unique citations out of which we identified 10 papers reporting prioritization approaches for guideline development. All of the identified approaches focused on prioritizing guideline topics but none on prioritizing recommendation questions or outcomes. The two most frequently reported steps of the development process for these approaches were reviewing the grey literature (9 out of 10, 90%) and engaging various stakeholders (9 out of 10, 90%). We derived a common framework of 20 prioritization criteria that can be used when prioritizing guideline topics. The most frequently reported criteria were the health burden of disease which was included in all of the approaches, practice variation (8 out of 10, 80%), and impact on health outcomes (7 out of 10, 70%). Two of the identified approaches stood out as being comprehensive and detailed.

Conclusions

We described 10 prioritization approaches in the development of health practice guidelines. There is a need to assess the effectiveness, efficiency and transparency of the identified approaches and to develop standardized and validated priority setting tools.

[Increasing the efficiency of guideline production: a narrative review]

BACKGROUND

The development of high-quality clinical practice guidelines is laborious and time-consuming. New methods have become available to streamline this process. However, the awareness of these methods should be improved.

METHODS

Selective literature search in PubMed/MEDLINE und Embase.

RESULTS

Simple tools such as surveys or voting systems can facilitate the organization, planning and communication. Adequate methods should be used to prioritize all potential questions that should be addressed in the guideline. Published or ongoing international guidelines and systematic reviews can be used meaningfully for the planned guideline. In the case of guideline updates, it should be determined whether all parts of the guideline require an update of the evidence. The need for an update should be investigated. The concept of living guidelines has the biggest potential to provide gains in efficiency. Living guidelines are continuously updated based on new evidence instead of being regularly updated at a predefined time.

CONCLUSIONS

New methods allowing for more efficient guideline production have been developed and, in part, already been introduced. Before starting with the production of a guideline (or its update), the potential advantages and disadvantages/risks of the corresponding methods should be balanced.

Practical Considerations for Using Online Methods to Engage Patients in Guideline Development

Clinical practice guidelines (CPGs) have been widely used in healthcare policy, practice, and for suggesting future research. As patients increasingly become involved in CPG development to produce patient-centered recommendations, more research is needed on methods to engage patients, particularly methods allowing for scalable engagement of large, diverse, and geographically distributed groups of patients. In this article, we discuss practical considerations for using online methods to engage patients in CPG development. To inform this discussion, we conducted a rapid, systematic review of literature on patient involvement in CPG development and used qualitative evidence synthesis techniques to make inferences about potential advantages and challenges of using online methods to engage patients in this context. We identified 79 articles containing information about involving patients in CPG development. Potential advantages include the ability of online methods to facilitate greater openness and honesty by patients, as well as to reflect the diversity of patient views, which in turn further improve the utility of CPGs. Potential challenges of using online methods may include the extra skill, time, and certain types of resources that may be needed for patient engagement, as well as the difficulty engaging specific patient populations. However, these challenges are mitigated by growing calls for patient engagement as normative for CPG development in addition to patients' increasing familiarity with online technologies. These practical considerations should be examined empirically as guideline development groups further explore the appropriateness of using online methods to engage patients across different stages of CPG development.

A systematic decision-making process on the need for updating clinical practice guidelines proved to be feasible in a pilot study

OBJECTIVES

The objective of this study was to test and evaluate a new decision-making process on the need for updating within the update of a German clinical practice guideline (CPG).

STUDY DESIGN AND SETTING

The pilot study comprised (1) limited searches in Pubmed to identify new potentially relevant evidence, (2) an online survey among the members of the CPG group to assess the need for update, and (3) a consensus conference for determination and prioritization of guideline sections with a high need for update. Subsequently, we conducted a second online survey to evaluate the procedure.

RESULTS

The searches resulted in 902 abstracts that were graded as new potentially relevant evidence. Twenty five of 39 members of the CPG group (64%) participated in the online survey. Seventy six percent of those took part in the second online survey. The evaluation study found on average a grade of support of the procedure regarding the determination of the need for update of 3.65 (standard deviation: 0.76) on a likert scale with 1 = "no support" to 5 = "very strong support."

CONCLUSION

The conducted procedure presents a systematic approach for assessing whether and to what extent a CPG requires updating and enables setting priorities for which particular guideline section to update within a CPG.

Identifying and prioritising systematic review topics with public health stakeholders: A protocol for a modified Delphi study in Switzerland to inform future research agendas

INTRODUCTION

The Cochrane Collaboration aims to produce relevant and top priority evidence that responds to existing evidence gaps. Hence, research priority setting (RPS) is important to identify which potential research gaps are deemed most important. Moreover, RPS supports future health research to conform both health and health evidence needs. However, studies that are prioritising systematic review topics in public health are surprisingly rare. Therefore, to inform the research agenda of Cochrane Public Health Europe (CPHE), we introduce the protocol of a priority setting study on systematic review topics in several European countries, which is conceptualised as pilot.

METHODS AND ANALYSIS

We will conduct a two-round modified Delphi study in Switzerland, incorporating an anonymous web-based questionnaire, to assess which topics should be prioritised for systematic reviews in public health. In the first Delphi round public health stakeholders will suggest relevant assessment criteria and potential priority topics. In the second Delphi round the participants indicate their (dis)agreement to the aggregated results of the first round and rate the potential review topics with the predetermined criteria on a four-point Likert scale. As we invite a wide variety of stakeholders we will compare the results between the different stakeholder groups.

ETHICS AND DISSEMINATION

We have received ethical approval from the ethical board of the University of Bremen, Germany (principal investigation is conducted at the University of Bremen) and a certificate of non-objection from the Canton of Zurich, Switzerland (fieldwork will be conducted in Switzerland). The results of this study will be further disseminated through peer reviewed publication and will support systematic review author groups (i.a. CPHE) to improve the relevance of the groups´ future review work. Finally, the proposed priority setting study can be used as a framework by other systematic review groups when conducting a priority setting study in a different context.

Methodological systematic review identifies major limitations in prioritization processes for updating

OBJECTIVES

The aim of the study was to identify and describe strategies to prioritize the updating of systematic reviews (SRs), health technology assessments (HTAs), or clinical guidelines (CGs).

STUDY DESIGN AND SETTING

We conducted an SR of studies describing one or more methods to prioritize SRs, HTAs, or CGs for updating. We searched MEDLINE (PubMed, from 1966 to August 2016) and The Cochrane Methodology Register (The Cochrane Library, Issue 8 2016). We hand searched abstract books, reviewed reference lists, and contacted experts. Two reviewers independently screened the references and extracted data.

RESULTS

We included 14 studies. Six studies were classified as descriptive (6 of 14, 42.9%) and eight as implementation studies (8 of 14, 57.1%). Six studies reported an updating strategy (6 of 14, 42.9%), six a prioritization process (6 of 14, 42.9%), and two a prioritization criterion (2 of 14, 14.2%). Eight studies focused on SRs (8 of 14, 57.1%), six studies focused on CGs (6 of 14, 42.9%), and none were about HTAs. We identified 76 prioritization criteria that can be applied when prioritizing documents for updating. The most frequently cited criteria were as follows: available evidence (19 of 76, 25.0%), clinical relevance (10 of 76; 13.2%), and users' interest (10 of 76; 13.2%).

CONCLUSION

There is wide variability and suboptimal reporting of the methods used to develop and implement processes to prioritize updating of SRs, HTAs, and CGs.

Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise

BACKGROUND

Although several tools to evaluate the credibility of health care guidelines exist, guidance on practical steps for developing guidelines is lacking. We systematically compiled a comprehensive checklist of items linked to relevant resources and tools that guideline developers could consider, without the expectation that every guideline would address each item.

METHODS

We searched data sources, including manuals of international guideline developers, literature on guidelines for guidelines (with a focus on methodology reports from international and national agencies, and professional societies) and recent articles providing systematic guidance. We reviewed these sources in duplicate, extracted items for the checklist using a sensitive approach and developed overarching topics relevant to guidelines. In an iterative process, we reviewed items for duplication and omissions and involved experts in guideline development for revisions and suggestions for items to be added.

RESULTS

We developed a checklist with 18 topics and 146 items and a webpage to facilitate its use by guideline developers. The topics and included items cover all stages of the guideline enterprise, from the planning and formulation of guidelines, to their implementation and evaluation. The final checklist includes links to training materials as well as resources with suggested methodology for applying the items.

INTERPRETATION

The checklist will serve as a resource for guideline developers. Consideration of items on the checklist will support the development, implementation and evaluation of guidelines. We will use crowdsourcing to revise the checklist and keep it up to date.

A guide to guidelines for pulmonary, sleep, and critical care medicine clinicians

Both the Institute of Medicine and the Guidelines International Network have recently published standards for trustworthy guidelines. The standards address multiple aspects of guideline development, including being transparent about funding and methodology, minimizing bias related to conflicts of interest, assembling writing committees with broad stakeholder representation, using rigorous and systematic methods to synthesize evidence and formulate recommendations, and periodically assessing guidelines for currency and updating them as required. In this article, we present the perspective of the Documents Development and Implementation Committee of the American Thoracic Society (ATS) on these and other guideline-related topics of relevance to ATS members. In addition, we summarize the many important take-home messages from a workshop that was jointly sponsored by the ATS and the European Respiratory Society, and attempt to place these messages in the context of a methodology that is rapidly evolving and a landscape in which clinical practice guidelines are subjected to ever-increasing scrutiny by clinicians, patients, and other third-party stakeholders.