Methods for living guidelines: early guidance based on practical experience. Article 5: decisions on methods for evidence synthesis and recommendation development for living guidelines

Reimagining How We Synthesize Information to Impact Clinical Care, Policy, and Research Priorities in Real Time: Examples and Lessons Learned from COVID-19

Real-time clinical care, policy, and research decisions need real-time evidence synthesis. However, as we found during the COVID-19 pandemic, it is challenging to rapidly address key clinical and policy questions through rigorous, relevant, and usable evidence. Our objective is to present three exemplar cases of rapid evidence synthesis products from the Veterans Healthcare Administration Evidence Synthesis Program (ESP) and, in the context of these examples, outline ESP products, challenges, and lessons learned. We faced challenges in (1) balancing scientific rigor with the speed in which evidence synthesis was needed, (2) sorting through rapidly evolving large bodies of evidence, and (3) assessing the impact of evidence synthesis products on clinical care, policy, and research. We found solutions in (1) engaging stakeholders early, (2) utilizing artificial intelligence capabilities, (3) building infrastructure to establish living reviews, and (4) planning for dissemination to maximize impact.

[Development of criteria for the prospective assessment of the need for updating guideline recommendations: The AGIL criteria]

BACKGROUND

Evidence-based guideline and vaccination recommendations should continuously be updated to appropriately support health care decisions. However, resources for updating guidelines are often limited. The aim of this project was to develop a list of criteria for the prospective assessment of the need for updating individual guideline or vaccination recommendations, which can be applied from the time a guideline or guideline update is finalised.

METHODS

In this article we describe the development of the AGIL criteria (Assessment of Guidelines for Updating Recommendations). The AGIL criteria were developed by experienced scientists and experts in the field of guideline development in a multi-step process. The five steps included: 1) development of an initial list of criteria by the project team; 2) online survey of guideline experts on the initial version of the criteria list; 3) revision of the criteria list based on the results of the online survey; 4) workshop on the criteria list at the EbM Congress 2023; 5) creation of version 1.0 of the AGIL criteria based on the workshop results.

RESULTS

The initial list included the following three criteria: 1) relevance of the question 2) availability of new relevant evidence, and 3) impact of potentially new evidence. The response rate of the online survey for fully completed questionnaires was 31.0% (N=195; 630 guideline experts were contacted by email). For 90.3% (n=176) of the respondents, the criteria list included all essential aspects for assessing the need for updating guideline recommendations. More than three quarters of respondents rated the importance of the three criteria as "very important" or "important" (criteria 1-3: 75.3%, 86.1%, 85.2%) and - with the exception of criterion 1 - comprehensibility as "very comprehensible" or "comprehensible" (criteria 1-3: 58.4%, 75.9%, 78.5%). The results of the online survey and the workshop generally confirmed the three criteria with their two sub-questions. The incorporation of all feedback resulted in the AGIL criteria (version 1.0), recapping: 1) relevance of the question regarding a) PICO components and b) other factors, e.g. epidemiological aspects; 2) availability of new evidence a) on health-related benefits and harms and b) on other decision factors, e.g. feasibility, acceptability; 3) impact of new evidence a) on the certainty of evidence on which the recommendation is based and b) on the present recommendation, e.g.

STRENGTH OF RECOMMENDATION

DISCUSSION

The moderate response rate of the online survey may have limited its representativeness. Nevertheless, we consider the response rate to be satisfactory in this research context. The inclusion of many experts in the online survey and the EbM Congress workshop is a strength of the project and supports the quality of the results.

CONCLUSIONS

The AGIL criteria provide a structured guidance for the prospective assessment of the need for updating individual guideline recommendations and other evidence-based recommendations. The implementation and evaluation of the AGIL criteria 1.0 in a field test is planned.

Optimizing process and methods for a living systematic review: 30 search updates and three review updates later

OBJECTIVE

To describe the living systematic review (LSR) process and to share experience of planning, searches, screening, extraction, publishing and dissemination to inform and assist authors planning their own LSR. Many LSR do not publish more than one update, we hope this paper helps to increase this.

STUDY DESIGN AND SETTING

A Cochrane LSR with an international author team that has been 'living' for two years, with monthly search updates and three full updates published in this time. LSRs are regularly updated systematic reviews that allow new evidence to be incorporated as it becomes available. LSR are ideally suited to policy-relevant topics where there is uncertainty and new evidence will likely impact the interpretation and/or certainty of outcomes.

RESULTS

The key features of the process that require consideration are: specifying the frequency of searches and triggers for full updates in the protocol; stakeholder input; publishing and disseminating monthly search findings. A strong team, incorporating methodological and topic expertise, with core members that meet regularly is essential. Regular search updates make it important to have a clear cyclical schedule of activity. To achieve timely updates this process should be streamlined, for example, using automated monthly searches, and systematic reviewing software for screening. LSR provide a unique opportunity to incorporate stakeholder feedback.

CONCLUSIONS

We recommend that LSRs should be: justified; carefully planned including the timing of search updates, triggers for publication and termination; published in a timely manner; have a clear dissemination plan; and a strong core team of authors.

Evidence surveillance for a living clinical guideline: Case study of the Australian stroke guidelines

BACKGROUND

Continual evidence surveillance is an integral feature of living guidelines. The Australian Stroke Guidelines include recommendations on 100 clinical topics and have been 'living' since 2018.

OBJECTIVES

To describe the approach for establishing and evaluating an evidence surveillance system for the living Australian Stroke Guidelines.

METHODS

We developed a pragmatic surveillance system based on an analysis of the searches for the 2017 Stroke Guidelines and evaluated its reliability by assessing the potential impact on guideline recommendations. Search retrieval and screening workload are monitored monthly, together with the frequency of changes to the guideline recommendations.

RESULTS

Evidence surveillance was guided by practical considerations of efficiency and sustainability. A single PubMed search covering all guideline topics, limited to systematic reviews and randomised trials, is run monthly. The search retrieves about 400 records a month of which a sixth are triaged to the guideline panels for further consideration. Evaluations with Epistemonikos and the Cochrane Stroke Trials Register demonstrated the robustness of adopting this more restrictive approach. Collaborating with the guideline team in designing, implementing and evaluating the surveillance is essential for optimising the approach.

CONCLUSION

Monthly evidence surveillance for a large living guideline is feasible and sustainable when applying a pragmatic approach.

What could health technology assessment learn from living clinical practice guidelines?

A "living" approach to clinical practice guidelines is when the identification, appraisal and synthesis of evidence is maintained and repeated at an agreed frequency, with a clear process for when and how new evidence is to be incorporated. The value of a living approach to guidelines was emphasised during the COVID-19 pandemic when health professionals and policymakers needed to make decisions regarding patient care in the context of a nascent but rapidly evolving evidence base. In this perspective, we draw on our recent experience developing Australian and international living guidelines and reflect on the feasibility of applying living guideline methods and processes to a lifecycle approach to health technology assessment (HTA). We believe the opportunities and challenges of adopting a living approach in HTA fall into five key themes: identification, appraisal and synthesis of evidence; optimising the frequency of updates; embedding ongoing multi-stakeholder engagement; linking the emergence of new evidence to reimbursement; and system capacity to support a living approach. We acknowledge that the suitability of specific living approaches to HTA will be heavily influenced by the type of health technology, its intended use in the health system, local reimbursement pathways, and other policy settings. But we believe that the methods and processes applied successfully to guideline development to manage evidentiary uncertainty could be applied in the context of HTA and reimbursement decision-making to help manage similar sources of uncertainty.

Methods for living guidelines: early guidance based on practical experience. Paper 3: selecting and prioritizing questions for living guidelines

OBJECTIVES

This article is part of a series on methods for living guidelines, consolidating practical experiences from developing living guidelines. It focuses on methods for identification, selection, and prioritization of clinical questions for a living approach to guideline development.

STUDY DESIGN AND SETTING

Members of the Australian Living Evidence Consortium, the National Institute of Health and Care Excellence and the US Grading of Recommendations, Assessment, Development and Evaluations Network, convened a working group. All members have expertize and practical experience in the development of living guidelines. We collated methods, documents on prioritization from each organization's living guidelines, conducted interviews and held working group discussions. We consolidated these to form best practice principles which were then edited and agreed on by the working group members.

RESULTS

We developed best practice principles for (1) identification, (2) selection, and (3) prioritization, of questions for a living approach to guideline development. Several different strategies for undertaking prioritizing questions are explored.

CONCLUSION

The article provides guidance for prioritizing questions in living guidelines. Subsequent articles in this series explore consumer involvement, search decisions, and methods decisions that are appropriate for questions with different priority levels.