Guidelines rarely used GRADE and applied methods inconsistently: A methodological study of Australian guidelines

Uncovering gaps in GRADE reporting of Eastern Association for the Surgery of Trauma (EAST) guidelines

OBJECTIVES

To formally evaluate the uptake and reporting of the Grading of Recommendation Assessments, Development and Evaluation (GRADE) approach in clinical practice guidelines (CPGs) developed by the Eastern Association for the Surgery of Trauma (EAST).

STUDY DESIGN AND SETTING

Based on an a priori, written protocol, we developed a dedicated data abstraction form that included the six suggested criteria for using and applying GRADE. By searching the EAST website, we identified all EAST guidelines that referenced the use of GRADE. All steps of the data abstraction process were completed independently and in duplicate by two members of the research team.

RESULTS

We identified a total of 48 CPGs that used GRADE. Trauma and violence prevention (n = 11; 23.9%) was the most common topic. The median number of patient/population, intervention, comparison, and outcomes (PICO) questions addressed was 3 (interquartile range: 2; 4) with a median of 2.5 (interquartile range: 1; 4) critical outcomes. A conditional/weak recommendation was provided for n = 79 (51.4%) PICOs, whereas a strong recommendation was provided for 33 PICOs (23.9%). For 22 PICOs (15.9%), no recommendation was made. Nearly all guideline documents provided search dates (n = 44; 95.7%) and a Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram (n = 44; 95.7%). Most described categories for rating down (n = 35; 76.1%). GRADE decision-making domains related to the ratio of benefits to harms, values and preferences, and resource utilization were referenced by 43.5% (n = 20), 43.5% (n = 20), and 30.4% (n = 14) of CPGs, respectively. For nearly half of PICO questions (n = 59; 44.2%) authors did not provide an evidence profile or summary of findings table. Comparing time periods from 2014-2018 to 2019-2022, the proportion of recommendations with an overall certainty of evidence increased (52.4% vs 83.9%; P < 0.001).

CONCLUSION

EAST has successfully adopted GRADE to develop many trauma-related guidelines, each addressing a finite number of focused clinical questions based on systematic reviews conducted in-house. Overall reporting improved over time. There is for improvement when it comes to consistent provision of an overall certainty of evidence, the reporting of the evidence to decision-making process, and the justification of strong recommendations based on low/very low certainty evidence.

Comprehensive consideration of multiple determinants from evidence to recommendations in guidelines for most traditional Chinese medicine was suboptimal: a systematic review

BACKGROUND

The overall comprehensive consideration of the factors influencing the recommendations in the traditional Chinese medicine (TCM) guidelines remains poorly studied. This study systematically evaluate the factors influencing recommendations formation in the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) clinical practice guidelines (CPGs) and TCM CPGs.

METHODS

This was a methodological review in which we searched six databases and multiple related websites. The GRADE CPGs were identified as the guidelines developed by the GRADE Working Group or the two Co-Chairs. For the TCM CPGs, we randomly selected guidelines that were published by the TCM or integrative medicine academic societies from China mainland (published by the TCM or integrative medicine academic societies of China mainland). Two reviewers independently screened and extracted data. We included CPGs published in 2018-2022. We extracted information on the influencing factors of evidence to recommendation and conducted the analyses using descriptive statistics and calculated the proportion of relevant items by IBM SPSS Statistics and Microsoft Excel to compare the differences between the GRADE CPGs and the TCM CPGs.

RESULTS

Forty-five GRADE CPGs (including 912 recommendations) and 88 TCM CPGs (including 2452 recommendations) were included. TCM recommendations mainly considered the four key determinants of desirable anticipated effects, undesirable anticipated effects, balance between desirable and undesirable effects, certainty of evidence, with less than 20% of other dimensions. And TCM CPGs presented more strong recommendations (for or against) and inappropriate discordant recommendations than GRADE CPGs. GRADE CPGs were more comprehensive considered about the factors affecting the recommendations, and considered more than 70% of all factors in the evidence to recommendation.

CONCLUSIONS

The TCM CPGs lack a comprehensive consideration of multiple influencing determinants from evidence to recommendations. In the future, the correct application of the GRADE approaches should be emphasized.

Advancing guideline quality through country-wide and regional quality assessment of CPGs using AGREE: a scoping review

BACKGROUND AND OBJECTIVE

Clinical practice guidelines (CPGs) are evaluated for quality with the Appraisal of Guidelines for Research and Evaluation (AGREE) tool, and this is increasingly done for different countries and regional groupings. This scoping review aimed to describe, map, and compare these geographical synthesis studies, that assessed CPG quality using the AGREE tool. This allowed a global interpretation of the current landscape of these country-wide or regional synthesis studies, and a closer look at its methodology and results.

STUDY DESIGN AND METHODS

A scoping review was conducted searching databases Medline, Embase, Epistemonikos, and grey literature on 5 October 2021 for synthesis studies using the later versions of AGREE (AGREE II, AGREE-REX and AGREE GRS) to evaluate country-wide or regional CPG quality. Country-wide or regional synthesis studies were the units of analysis, and simple descriptive statistics was used to conduct the analysis. AGREE scores were analysed across subgroups into one of the seven Sustainable Development Goal regions, to allow for meaningful interpretation.

RESULTS

Fifty-seven studies fulfilled our eligibility criteria, which had included a total of 2918 CPGs. Regions of the Global North, and Eastern and South-Eastern Asia were most represented. Studies were consistent in reporting and presenting their AGREE domain and overall results, but only 18% (n = 10) reported development methods, and 19% (n = 11) reported use of Grading of Recommendations Assessment, Development, and Evaluation (GRADE). Overall scores for domains Rigor of development and Editorial independence were low, notably in middle-income countries. Editorial Independence scores, especially, were low across all regions with a maximum domain score of 46%. There were no studies from low-income countries.

CONCLUSION

There is an increasing tendency to appraise country-wide and regionally grouped CPGs, using quality appraisal tools. The AGREE tool, evaluated in this scoping review, was used well and consistently across studies. Findings of low report rates of development of CPGs and of use of GRADE is concerning, as is low domain scores globally for Editorial Independence. Transparent reporting of funding and competing interests, as well as highlighting evidence-to-decision processes, should assist in further improving CPG quality as clinicians are in dire need of high-quality guidelines.

Reporting quality and risk of bias in JBI systematic reviews evaluating the effectiveness of interventions: a methodological review protocol

OBJECTIVE

The objective of this methodological review is to evaluate the adherence of systematic reviews of effectiveness published in JBI Evidence Synthesis to reporting guidelines and methodological quality.

INTRODUCTION

Systematic reviews of effectiveness are essential tools for health practitioners and policy-makers. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines and the Risk of Bias in Systematic Reviews (ROBIS) tool are used to ensure maintenance of high reporting standards and methodological quality, respectively. This review will utilize these tools to identify strengths and shortfalls in the reporting quality of JBI systematic reviews of effectiveness.

INCLUSION CRITERIA

This review will include the 20 most recent systematic reviews of effectiveness published in JBI Evidence Synthesis .

METHODS

This review will search MEDLINE (PubMed) for effectiveness reviews published in JBI Evidence Synthesis . Abstract and full-text screening will be performed by 2 independent reviewers, and the most recent 20 studies will be selected for inclusion. Data regarding adherence to PRISMA 2020 and ROBIS will be extracted by 2 independent reviewers. Data will be presented descriptively with tables and synthesized narratively.

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study

OBJECTIVE

To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this.

DATA SOURCES

For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020.

STUDY SELECTION AND DATA EXTRACTION

Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed.

DATA SYNTHESIS

The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs.

CONCLUSIONS

One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

Factors Associated with the Quality and Transparency of National Guidelines: A Mixed-Methods Study

We assessed the methodological quality and transparency of all the national clinical practice guidelines that were published in Croatia up until 2017 and explored the factors associated with their quality rating. An in-depth quantitative and qualitative analysis was performed using rigorous methodology. We evaluated the guidelines using a validated AGREE II instrument with four raters; we used multiple linear regressions to identify the predictors of quality; and two focus groups, including guideline developers, to further explore the guideline development process. The majority of the guidelines (N = 74) were developed by medical societies. The guidelines’ quality was rated low: the median standardized AGREE II score was low, 36% (IQR 28–42), and so were the overall-assessments. The aspects of the guidelines that were rated best were the “clarity of presentation” and the “scope and purpose” (median ≥ 59%); however, the other four domains received very low scores (15–33%). Overall, the guideline quality did not improve over time. The guidelines that were developed by medical societies scored significantly worse than those developed by governmental, or unofficial working groups (12–43% per domain). In focus group discussions, inadequate methodology, a lack of implementation systems in place, a lack of awareness about editorial independence, and broader expertise/perspectives in working groups were identified as factors behind the low scores. The factors identified as affecting the quality of the national guidelines may help stakeholders who are developing interventions and education programs aimed at improving guideline quality worldwide.

The Grading of Recommendations, assessment, development, and evaluation approach was rarely followed and inconsistently applied in pressure injury prevention guidelines development: A cross-sectional survey

INTRODUCTION

The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach was developed to assess the certainty (or quality) of evidence and strength of recommendations in guidelines and endorsed internationally as a standard. Some guidelines had been developed to promote pressure injury prevention.

AIMS

We explored whether and to what extent the development of pressure injury prevention guidelines had followed or been informed by the GRADE approach. If this approach was not used, we examined which other methods were used instead.

METHODS

A cross-sectional study of pressure injury prevention guidelines was conducted. PubMed, Embase, CINAHL, and Chinese databases as well as guideline repositories and websites of professional bodies were searched for guidelines from 1990 to 2020. The grading systems of the certainty (or quality) of evidence and strength of recommendations of included guidelines were extracted. For the GRADE approach guidelines, compliance was assessed with the GRADE application criteria.

RESULTS

Twenty guidelines were identified. Among them, four guidelines (20%) indicated the use of the GRADE approach. The compliance rate ranged from 33.3%-94.4%. Other approaches, such as the Scottish Intercollegiate Guidelines Network (SIGN) approach, were also used.

CONCLUSION

The GRADE approach is rarely followed and inconsistently applied in pressure injury prevention guidelines. Other systems, such as the SIGN approach, are being used despite being outdated or inconsistent. Strategies for further uptake and appropriate application of the GRADE approach among guideline developers are needed in the future.

AGREE-S: AGREE II extension for surgical interventions - United European Gastroenterology and European Association for Endoscopic Surgery methodological guide

BACKGROUND

The Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument has been developed to inform the methodology, reporting and appraisal of clinical practice guidelines. Evidence suggests that the quality of surgical guidelines can be improved, and the structure and content of AGREE II can be modified to help enhance the quality of guidelines of surgical interventions.

OBJECTIVE

To develop an extension of AGREE II specifically designed for guidelines of surgical interventions.

METHODS

In the tripartite Guideline Assessment Project (GAP) funded by United European Gastroenterology and the European Association for Endoscopic Surgery, (i) we assessed the quality of surgical guidelines and we identified factors associated with higher quality (GAP I); (ii) we applied correlation analysis, factor analysis and the item response theory to inform an adaption of AGREE II for the purposes of surgical guidelines (GAP II); and (iii) we developed an AGREE II extension for surgical interventions, informed by the results of GAP I, GAP II, and a Delphi process of stakeholders, including representation from interventional and surgical disciplines; the Guideline International Network (GIN); the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group; the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) initiative; and representation of surgical journal editors and patient/public.

RESULTS

We developed AGREE-S, an AGREE II extension for surgical interventions, which comprises 24 items organized in 6 domains; Scope and purpose, Stakeholders, Evidence synthesis, Development of recommendations, Editorial independence, and Implementation and update. The panel of stakeholders proposed 3 additional items: development of a guideline protocol, consideration of practice variability and surgical/interventional expertise in different settings, and specification of infrastructures required to implement the recommendations. Three of the existing items were amended, 7 items were rearranged among the domains, and one item was removed. The domain Rigour of Development was divided into domains on Evidence Synthesis and Development of Recommendations. The new domain Development of Recommendations incorporates items from the original AGREE II domain Clarity of Presentation.

CONCLUSION

AGREE-S is an evidence-based and stakeholder-informed extension of the AGREE II instrument, that can be used as a guide for the development and adaption of guidelines on surgical interventions.

Analysis of the Status and Trends of Chinese Clinical Practice Guideline Development Between 2010 and 2020: A Systematic Review

Objective: This study aimed to systematically review the status and trends of Chinese clinical practice guidelines (CPGs) during the time period 2010-2020 and explore their methodological characteristics. Then, based on the strengths and weaknesses in development, offer several recommendations for the quality improvement which will serve as a reference for the users and developers of CPG. Introduction: With the development of evidence-based medicine (EBM), the CPGs play an increasingly important role in healthcare decision-making both in China and worldwide. Inclusion criteria: The CPGs that have been used to help the health professionals in the healthcare decision-making were included. Methodology: The China National Knowledge Infrastructure (CNKI) and WanFang databases were searched from 2010 to 2020 for the studies describing the general and methodological characteristics of Chinese CPGs. Comparisons of the methodological characteristics between the groups were conducted using the chi-square test or Fisher's exact test. The M-K test was adopted to identify the monotonically increasing or decreasing trends of methodological characteristics over the timespan. Results: A total of 2,654 CPGs fulfilled the inclusion criteria. The quantity and quality of the guidelines developed in China have improved over the time span. From 2010 to 2020,the guidelines had differing characteristics and covered a wide range of subjects. In total, 2,318(87.34%) guidelines focused on Western Medicine. Eight (0.30%) had been developed for patient versions of guidelines, 10(0.38%) were tentative guidelines, and 16(0.60%) were rapid advice guidelines. Medical specialty societies (including their branches) (71.1%) were the main guideline makers. The most addressed diseases were neoplasms (14.43%). The target population is mainly adults (84.97%). The methodological quality of consensus-based (CB)-CPGs was obviously lower than evidence-based (EB)-CPGs. Except for the item, "recommendations were based on evidence of systematic reviews," there were statistical differences in all other methodological items between the EB-CPGS and CB-CPGS (P < 0.01). Higher methodological quality has been observed in EB-CPGs. All the data relating to the methodological characteristics indicated that higher methodological quality was present in the guidelines using GRADE (P < 0.01). Conclusion: The quantity and quality of the guidelines developed in China have improved between 2010 and 2020. CB-CPGs have also paid attention to the methodology quality, but obviously, this is lower than that in the EB-CPGs.