The quality of European dermatological guidelines: critical appraisal of the quality of EDF guidelines using the AGREE II instrument

The Reporting and Methodological Quality of Systematic Reviews Underpinning Clinical Practice Guidelines Focused on the Management of Cutaneous Melanoma: Cross-Sectional Analysis

BACKGROUND

Clinical practice guidelines (CPGs) inform evidence-based decision-making in the clinical setting; however, systematic reviews (SRs) that inform these CPGs may vary in terms of reporting and methodological quality, which affects confidence in summary effect estimates.

OBJECTIVE

Our objective was to appraise the methodological and reporting quality of the SRs used in CPGs for cutaneous melanoma and evaluate differences in these outcomes between Cochrane and non-Cochrane reviews.

METHODS

We conducted a cross-sectional analysis by searching PubMed for cutaneous melanoma guidelines published between January 1, 2015, and May 21, 2021. Next, we extracted SRs composing these guidelines and appraised their reporting and methodological rigor using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and AMSTAR (A Measurement Tool to Assess Systematic Reviews) checklists. Lastly, we compared these outcomes between Cochrane and non-Cochrane SRs. All screening and data extraction occurred in a masked, duplicate fashion.

RESULTS

Of the SRs appraised, the mean completion rate was 66.5% (SD 12.29%) for the PRISMA checklist and 44.5% (SD 21.05%) for AMSTAR. The majority of SRs (19/50, 53%) were of critically low methodological quality, with no SRs being appraised as high quality. There was a statistically significant association (P<.001) between AMSTAR and PRISMA checklists. Cochrane SRs had higher PRISMA mean completion rates and higher methodological quality than non-Cochrane SRs.

CONCLUSIONS

SRs supporting CPGs focused on the management of cutaneous melanoma vary in reporting and methodological quality, with the majority of SRs being of low quality. Increasing adherence to PRISMA and AMSTAR checklists will likely increase the quality of SRs, thereby increasing the level of evidence supporting cutaneous melanoma CPGs.

Advancing guideline quality through country-wide and regional quality assessment of CPGs using AGREE: a scoping review

BACKGROUND AND OBJECTIVE

Clinical practice guidelines (CPGs) are evaluated for quality with the Appraisal of Guidelines for Research and Evaluation (AGREE) tool, and this is increasingly done for different countries and regional groupings. This scoping review aimed to describe, map, and compare these geographical synthesis studies, that assessed CPG quality using the AGREE tool. This allowed a global interpretation of the current landscape of these country-wide or regional synthesis studies, and a closer look at its methodology and results.

STUDY DESIGN AND METHODS

A scoping review was conducted searching databases Medline, Embase, Epistemonikos, and grey literature on 5 October 2021 for synthesis studies using the later versions of AGREE (AGREE II, AGREE-REX and AGREE GRS) to evaluate country-wide or regional CPG quality. Country-wide or regional synthesis studies were the units of analysis, and simple descriptive statistics was used to conduct the analysis. AGREE scores were analysed across subgroups into one of the seven Sustainable Development Goal regions, to allow for meaningful interpretation.

RESULTS

Fifty-seven studies fulfilled our eligibility criteria, which had included a total of 2918 CPGs. Regions of the Global North, and Eastern and South-Eastern Asia were most represented. Studies were consistent in reporting and presenting their AGREE domain and overall results, but only 18% (n = 10) reported development methods, and 19% (n = 11) reported use of Grading of Recommendations Assessment, Development, and Evaluation (GRADE). Overall scores for domains Rigor of development and Editorial independence were low, notably in middle-income countries. Editorial Independence scores, especially, were low across all regions with a maximum domain score of 46%. There were no studies from low-income countries.

CONCLUSION

There is an increasing tendency to appraise country-wide and regionally grouped CPGs, using quality appraisal tools. The AGREE tool, evaluated in this scoping review, was used well and consistently across studies. Findings of low report rates of development of CPGs and of use of GRADE is concerning, as is low domain scores globally for Editorial Independence. Transparent reporting of funding and competing interests, as well as highlighting evidence-to-decision processes, should assist in further improving CPG quality as clinicians are in dire need of high-quality guidelines.

Editor's Choice - Quality Assessment of European Society for Vascular Surgery Clinical Practice Guidelines

OBJECTIVE

An increasing number of clinical practice guidelines (CPGs) have emerged over recent years. To have clinical utility, they need to be rigorously developed and scientifically robust. Instruments have been developed to assess the quality of clinical guideline development and reporting. The aim of this study was to evaluate CPGs from the European Society for Vascular Surgery (ESVS) using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) instrument.

METHODS

CPGs published by the ESVS during the period January 2011 to January 2023 were included. Two independent reviewers assessed the guidelines after receiving training in the use and application of the AGREE II instrument. Inter-reviewer reliability was assessed with the intraclass correlation coefficient. Maximum scaled scores were 100. Statistical analysis was performed using SPSS Statistics v.26.

RESULTS

Sixteen guidelines were included in the study. Good inter-reviewer score reliability was found on statistical analysis (> 0.9). The mean ± standard deviation domain scores were 68.1 ± 20.3% for scope and purpose, 57.1 ± 21.1% for stakeholder involvement, 67.8 ± 19.5% for rigour of development, 78.1 ± 20.6% for clarity of presentation, 50.3 ± 15.4% for applicability, 77.6 ± 17.6% for editorial independence, and 69.8 ± 20.1% for overall quality. Stakeholder involvement and applicability have improved in quality over time but are still the lowest scoring domains.

CONCLUSION

Most ESVS clinical guidelines are of high quality and reporting. There is scope for improvement, specifically by addressing the domains of stakeholder involvement and clinical applicability.

Quality assessment of Clinical Practice Guidelines (CPG) for the diagnosis and treatment of inflammatory bowel disease using the AGREE II instrument: a systematic review

BACKGROUND

The incidence and diagnosis of inflammatory bowel disease (IBD) has increased considerably in recent years. Many clinical practice guidelines (CPG) have been developed for the management of this disease across different clinical contexts, however, little evidence exists on their methodological quality. Therefore, we aimed to systematically evaluate the quality of CPGs for the diagnosis and treatment of IBD using the Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument.

METHODS

We identified CPGs by searching databases (MEDLINE - PubMed, EMBASE, CINAHL, LILACS) and other sources of gray literature on January 2022. We included guidelines with specific recommendations for the diagnosis and treatment of IBD and evaluated them with the AGREE II instrument to assess their methodological quality. Six independent reviewers assessed the quality of the guidelines and resolved conflicts by consensus. We assessed the degree of agreement using the intraclass correlation coefficient (ICC) and change in quality over time was appraised in two periods: from 2012 to 2017 and from 2018 to 2022.

RESULTS

We analyzed and evaluated 26 CPGs that met the inclusion criteria. The overall agreement among reviewers was moderate (ICC: 0.74; 95% CI 0.36 - 0.89). The mean scores of the AGREE II domains were: "Scope and purpose" 84.51%, "Stakeholder involvement" 60.90%, "Rigor of development" 69.95%, "Clarity of presentation" 85.58%, "Applicability" 26.60%, and "Editorial independence" 62.02%. No changes in quality were found over time.

CONCLUSIONS

The quality of the CPGs evaluated was generally good, with a large majority of the assessed guidelines being "recommended" and "recommended with modifications"; despite this, there is still room for improvement, especially in terms of stakeholder involvement and applicability. Efforts to develop high quality CPGs for IBD need to be further optimized.

Global Guidelines in Dermatology Mapping Project (GUIDEMAP): a scoping review of dermatology clinical practice guidelines

BACKGROUND

Clinical practice guidelines (CPGs) play a critical role in standardizing and improving treatment outcomes based on the available evidence. It is unclear how many CPGs are available globally to assist clinicians in the management of patients with skin disease.

OBJECTIVES

To search for and identify CPGs for dermatological conditions with the highest burden globally.

METHODS

We adapted a list of 12 dermatological conditions with the highest burden from the Global Burden of Disease (GBD) study 2019. A systematic literature search was done to identify CPGs published between October 2014 to October 2019. The scoping review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework.

RESULTS

A total of 226 CPGs were included. Melanoma had the greatest representation in the CPGs, followed by dermatitis and psoriasis. Skin cancers had a relatively high CPG representation but with lower GBD disease burden ranking. There was an uneven distribution by geographical region, with resource-poor settings being under-represented. The skin disease categories of the CPGs correlated weakly with the GBD disability-adjusted life-years metrics. Eighty-nine CPGs did not have funding disclosures and 34 CPGs were behind a paywall.

CONCLUSIONS

The global production of dermatology CPGs showed wide variation in geographical representation, article accessibility and reporting of funding. The number of skin disease CPGs were not commensurate with its disease burden. Future work will critically appraise the methodology and quality of dermatology CPGs and lead to the production of an accessible online resource summarizing these findings.

Integrating approaches for quality guideline development in LactaMap, an online lactation care support system

Background

Health professionals caring for women and infants experiencing difficulty with breastfeeding have reported deficiencies in evidence-based lactation knowledge. LactaMap is an online lactation care support system with more than 100 clinical practice guidelines to support breastfeeding care. Clinical practice guidelines support medical decision-making by summarising scientific evidence into systematically developed statements for specific clinical circumstances. Both common-sense and theory-based approaches have been used for guideline development and debate continues regarding which is superior. LactaMap clinical practice guidelines were created over the course of 5 years using a common-sense approach that was refined inductively. The aim of this study was to incorporate a theory-based framework approach into the methodology for ongoing update and review of LactaMap clinical practice guidelines.

Methods

The Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument was chosen as the framework-based approach to appraise LactaMap guideline quality. The study was conducted in two phases. The first phase appraised all 103 original LactaMap guidelines. The second phase appraised a subset of 15 updated LactaMap guidelines using improved methodology guided by phase 1, as well as 15 corresponding original (un-updated) guidelines.

Results

Mean Domain scores for 103 LactaMap original guidelines were above 75% in 3 of the 6 AGREE II quality Domains and no mean Domain score rated poorly. Update of guideline methodology was guided by phase 1 appraisals. Improved documentation of methods relating to questions in the Rigour of Development Domain resulted in improvement in mean Domain score from 39 to 72%.

Conclusions

This study showed that a theory-based approach to guideline development methodology can be readily integrated with a common-sense approach. Factors identified by AGREE II theory-based framework provided practical guidance for changes in methodology that were integrated prior to LactaMap website publication. Demonstration of high quality in LactaMap clinical practice guideline methodology ensures clinicians and the public can have trust that the content founded on them is robust, scientific and of highest possible quality.

A Critical Appraisal of Evidence- and Consensus-Based Guidelines for Actinic Keratosis

Actinic keratoses (AK) are common lesions of the skin that can be effectively treated with several lesion- and field-directed treatments. Clinical practice guidelines assist physicians in choosing the appropriate treatment options for their patients. Here, we aimed to systematically identify and evaluate the methodological quality of currently available guidelines for AK. Guidelines published within the last 5 years were identified in a systematic search of guideline databases, Medline and Embase. Then, six independent reviewers evaluated the methodological quality using the tools "Appraisal of Guidelines for Research and Evaluation" (AGREE II) and "Recommendation EXcellence" (AGREE-REX). The Kruskal-Wallis (H) test was used to explore differences among subgroups and Spearman's correlation to examine the relationship between individual domains. Three guidelines developed by consortia from Canada, Germany and the United Kingdom were eligible for the evaluation. The German guideline achieved the highest scores, fulfilling 65 to 92% of the criteria in AGREE II and 67 to 84% in AGREE-REX, whereas the Canadian guideline scored 31 to 71% of the criteria in AGREE II and 33 to 46% in AGREE-REX. The domains "stakeholder involvement" and "values and preferences" were identified as methodological weaknesses requiring particular attention and improvement in future guideline efforts.

Assessing the Quality of Clinical Practice Guidelines in the Middle East and North Africa (MENA) Region: A Systematic Review

AIM

Clinical practice guidelines (CPGs) have progressively become a popular tool for making optimal clinical decisions. The literature shows that the poor quality of CPGs can form a barrier against adhering to them, resulting in a suboptimal level of healthcare. The objective of this systematic review is to evaluate the quality of CPGs in the Middle East and North Africa (MENA) region using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) Instrument.

METHODS

The authors searched in the MEDLINE and EMBASE databases through the Ovid interface on May 25, 2019. Keywords relating to CPGs and MENA countries were combined using Boolean search operators. The search was not limited to specific diseases. The quality of guidelines was appraised by two reviewers independently using the AGREE II Instrument. Discrepancies within a group were resolved through the involvement of a principle investigator.

RESULTS

A total of 61 CPGs were appraised. These guidelines were mainly from Saudi Arabia, and the most covered disease topic was cancer. Among the six domains of the AGREE II Instrument, CPGs scored the highest on clarity of presentation (mean 82%), while the lowest score was granted to the rigor of development domain (mean 28%). This indicates substantial deficiencies in reporting the developmental processes of CPGs and the resources used for the synthesis of evidence.

CONCLUSION

From this review, it was found that the number of retrieved guidelines published in the MENA region is limited considering the large geographical area of the MENA region. The main domains that have higher quality scores were clarity of presentation and scope and purpose, whereas domains with the lowest scores were rigor of development and applicability. The authors' findings will help policymakers identify areas for improvement in CPGs, which can lead them to implement strategies such as the training of individuals and recruitment of international experts to ultimately develop high-quality CPGs.

The quality assessment of intraabdominal infection guidelines/consensuses in 2 decades - which are better and any changes?

BACKGROUND AND AIM

Intraabdominal infection (IAI) is a common and important disease worldwide. An increasing number of related guidelines/consensuses have been published in recent years, the quality evaluation for these guidelines/consensuses is necessary to identify lower-quality documents and explore the quality distribution in different time range and areas in this field.

METHODS

The Appraisal of Guidelines for Research & Evaluation Instrument tool was adopted to assess the quality of IAI guidelines/consensuses by 3 researchers independently. Intraclass correlation coefficients (ICCs) among the researchers were retrieved to reflect reliability. The quality differences of these guidelines/consensuses issued before and after May 2009, both international and non-international, were compared by a Mann-Whitney U test.

RESULTS

Fourteen IAI guidelines/consensuses published in English were obtained following a literature search. The ICCs among the researchers were all above 0.75, indicating satisfactory reliability. This outcome showed that the overall quality of these guidelines/consensuses was mediocre and considered acceptable in all items. A few guidelines/consensuses were better in their scientific and methodological characteristics than the others. Moreover, there were no significant differences in the scores between the guidelines/consensuses issued before and after May 2009 or between international vs regional guidelines/consensuses.

CONCLUSIONS

Overall, the quality of the IAI guidelines/consensuses was generally acceptable and applicable, with a few deficiencies. Therefore, continuous improvement is essential. The guideline assessment tools should be applied in guideline/consensus development both widely and strictly to improve the methodological quality.

A Methodological Assessment of Diabetic Foot Syndrome Clinical Practice Guidelines

OBJECTIVE

Diabetic foot syndrome (DFS) contributes to significant morbidity in diabetic patients. Diagnostic and therapeutic approaches to DFS may be summarised in clinical practice guidelines (CPGs) to aid clinical practice but may only benefit patients if the CPG is of high quality. This study determines the methodological quality of DFS CPGs using a validated assessment tool to identify CPGs adequate for use in clinical practice.

METHODS

Medline, EMBASE, and CPG databases were searched to 31 May 2019. Reference lists were also searched. Full text English evidence based DFS CPGs were included. CPGs based on expert consensus, guideline summaries, or those only available if purchased were excluded. Four reviewers independently assessed methodological quality using the Appraisal of Guidelines for Research and Evaluation II instrument. An overall guideline assessment scaled score of ≥80% was considered to be of adequate quality to recommend use.

RESULTS

Sixteen CPGs were identified. Good inter-reviewer reliability (ICC 0.985, 95% CI 0.980-0.989) was achieved. Poor scores were noted in domains 2 (stakeholder involvement), 5 (applicability), and 6 (editorial independence). Significant methodological heterogeneity was observed in all domains with the most noted in domain 6 (mean scaled score 43.2 ± 32.1%). Four CPGs achieved overall assessment scores of ≥80%.

CONCLUSION

Four CPGs were considered to be adequate for clinical practice based on methodological quality. However, elements of methodological quality were still lacking, and all CPGs had areas for improvement, potentially through increased multidisciplinary team involvement and trial application of recommendations. Methodological rigour may be improved using structured approaches with validated CPG creation tools in the future. Future work should also assess recommendation accuracy using available validated assessment tools.

AGREE II assessments of recent acne treatment guidelines: how well do they reveal trustworthiness as defined by the U.S. Institute of Medicine criteria?

BACKGROUND

Up-to-date, trustworthy guidelines are a widely relied upon means of promoting excellent patient care.

OBJECTIVES

To determine the quality of recently published acne treatment guidelines by utilizing the Appraisal of Guidelines for Research and Evaluation (AGREE) II Reporting Checklist, the U.S. Institute of Medicine's (IOM) criteria of trustworthiness, the red flags of Lenzer et al. and CheckUp.

METHODS

Systematic searches were conducted in bibliographic databases, guideline depositories and using Google to identify acne treatment guidelines published since 2013. Six assessors independently scored each guideline using the AGREE II Reporting Checklist. Guidelines were concomitantly assessed for trustworthiness using the IOM criteria and for the red flags of Lenzer et al., indicative of potential bias. Updates were screened using CheckUp.

RESULTS

Eight guidelines were identified, two of which were updates. Lowest scoring AGREE II domains across all guidelines were applicability (six poor, one fair, one average) and rigour (four poor, one fair, three average). Two of the three highest-scoring guidelines were developed using AGREE II. No guideline fully met each IOM criterion and all raised at least one red flag indicative of potential bias. One updated guideline did not address seven of 16 items on CheckUp and the other did not address four. Patient involvement in guideline development was minimal.

CONCLUSIONS

Use of the AGREE II instrument during guideline development did not have as great an effect on guideline quality as might be expected. There is considerable room for improvement in acne treatment guidelines in order to satisfy the IOM trustworthiness criteria and avoid bias.

Quality assessment of atopic dermatitis clinical practice guidelines in ≤ 18 years

Atopic dermatitis (AD) is a chronic inflammatory skin disease that affects the patients' quality of life greatly often from a very young age. Its worldwide incidence in children and adults varies, but it is usually among the first ten causes of dermatological consultation worldwide. There is a wide variety of treatment options for this condition including topical and systemic regimes. The decision to choose a treatment option in dermatological diseases is greatly influenced by the personal experience of each specialist, which increases variability in the selection of available therapies. Clinical practice guidelines (CPGs) not only offer recommendations supported on the available scientific evidence, but also are intended to assist in making appropriate decisions in clinical scenarios. To standardize the way in which CPGs should be developed, an instrument called AGREE II (Appraisal of Guidelines for Research and Evaluation) is used. In this study, ten clinical practice guidelines in ≤ 18 years were evaluated. Six domains (scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence) were assessed for each guideline, by three reviewers. Most of the domains obtained high scores except in the applicability domain. It is suggested that future atopic dermatitis CPGs should emphasize in the facilitating factors and barriers that may influence the application of guideline recommendations.