E-learning interventions are comparable to user's manual in a randomized trial of training strategies for the AGREE II

Reproducibility of the AGREE II Tool for Assessing the Methodological Quality of Clinical Practice Guidelines for the Management of Antithrombotic Agents in Patients Undergoing GI Endoscopy

BACKGROUND

Clinical practice guidelines (CPGs) exist for the management of antithrombotic agents in the periendoscopic period; however, their methodological qualities vary. The Appraisal of Guidelines for Research & Evaluation II (AGREE II) tool has been validated for the assessment of the methodological quality of CPGs; however, its reproducibility has not been assessed. The goal of this study was to assess the reproducibility of the AGREE II tool for CPGs published within the last 6 years for the management of antithrombotic agents in the periendoscopic period.

STUDY

A systematic search of PubMed and Embase databases was performed to identify eligible CPGs published between January 1, 2016 and April 14, 2022. The quality of the CPG was independently assessed by 6 reviewers using the AGREE II instrument. The reproducibility was summarized as weighted κ statistic and intraclass correlation coefficient using the SPSS statistical analysis package.

RESULTS

The search yielded 343 citations with 7 CPGs from Europe, Asia, and the United States included in the critical appraisal. The overall mean weighted κ score across all guidelines was 0.300 (range, 0.093 to 0.384) indicating a fair agreement. The overall intraclass correlation coefficient was 0.462 (range, 0.175 to 0.570) for single measures and 0.837 (range, 0.560 to 0.888) for average measures indicating moderate reliability.

CONCLUSIONS

Our study shows only a fair overall interobserver agreement in the methodological quality of the included CPGs. The results suggest the need for education and training of CPG raters to enhance the application of the AGREE II tool to improve its reproducibility.

The effectiveness of evidence-based healthcare educational interventions on healthcare professionals' knowledge, skills, attitudes, professional practice and healthcare outcomes: Systematic review and meta-analysis

OBJECTIVE

The primary aim of this systematic review is to assess the effectiveness of evidence-based healthcare (EBHC) educational interventions on healthcare professionals' knowledge, skills, attitudes, behaviour of EBHC, clinical process and care outcomes. A secondary aim of the review is to assess the effects of important pedagogical moderating factors for EBHC educational interventions.

METHOD

This systematic review used a forward and backward citation search strategy on the Web of Science platform (date of inception to 28 April 2023). Only randomised controlled trials (RCTs) and cluster RCTs which compared EBHC educational interventions for healthcare professionals were included. A random effects meta-analysis was undertaken for EBHC compared with an active and nonactive control for all outcomes.

RESULTS

Sixty-one RCTs were identified which included a total of 5208 healthcare professionals. There was a large effect for EBHC educational interventions compared with waiting list/no treatment/sham control on knowledge (SMD, 2.69; 95% CI, 1.26-4.14, GRADE Low), skills (SMD, 0.88; 95% CI, 0.25-1.73, Very Low Certainty), attitude (SMD, 0.81; 95% CI, 0.16-1.47, Very Low Certainty) and behaviour of EBHC (SMD, 0.82; 95% CI, 0.25-1.40, Very Low Certainty). Over time the effect of EBHC educational interventions substantially decreased with no evidence of effect at 6 months for any outcome except behaviour (SMD,1.72; 95% CI, 0.74-2.71, Low Certainty). There was some evidence that blended learning, active learning and consistency in the individual delivering the intervention may be important positive moderating factors.

CONCLUSION

These findings suggest that EBHC educational interventions may have a large short-term effect on improving healthcare professionals' knowledge, skills, attitude and behaviour of EBHC. These effects may be longer-lasting regarding EBHC behaviour. In terms of pedagogy, blended learning, active learning, and consistency of the individual delivering the intervention may be important positive moderating factors.

Methodological transparency of preoperative clinical practice guidelines for elective surgery. Systematic review

BACKGROUND

Clinical practice guidelines (CPG) are statements that provide recommendations regarding the approach to different diseases and aim to increase quality while decreasing the risk of complications in health care. Numerous guidelines in the field of perioperative care have been published in the previous decade but their methodological quality and transparency are relatively unknown.

OBJECTIVE

To critically evaluate the transparency and methodological quality of published CPG in the preoperative assessment and management of adult patients undergoing elective surgery.

DESIGN

Systematic review and methodological appraisal study.

DATA SOURCES

We searched for eligible CPG published in English or Spanish between January 1, 2010, and June 30, 2022, in Pubmed MEDLINE, TRIP Database, Embase, the Cochrane Library, as well as in representatives' medical societies of Anaesthesiology and developers of CPG.

ELIGIBILITY CRITERIA

CPG dedicated on preoperative fasting, cardiac assessment for non-cardiac surgery, and the use of routine preoperative tests were included. Methodological quality and transparency of CPG were assessed by 3 evaluators using the 6 domains of the AGREE-II tool.

RESULTS

We included 20 CPG of which 14 were classified as recommended guidelines. The domain of "applicability" scored the lowest (44%), while the domains "scope and objective" and "editorial interdependence" received the highest median scores of 93% and 97% respectively. The remaining domains received scores ranging from 44% to 84%. The top mean scored CPG in preoperative fasting was ASA 2017 (93%); among cardiac evaluation, CPG for non-cardiac surgery were CCS 2017 (91%), ESC-ESA 2014 (90%), and AHA-ACC 2014 (89%); in preoperative testing ICSI 2020 (97%).

CONCLUSIONS

In the last ten years, most published CPG in the preoperative assessment or management of adult patients undergoing elective surgery focused on preoperative fasting, cardiac assessment for non-cardiac surgery, and use of routine preoperative tests, present moderate to high methodological quality and can be recommended for their use or adaptation. Applicability and stakeholder involvement domains must be improved in the development of future guidelines.

Factors Associated with the Quality and Transparency of National Guidelines: A Mixed-Methods Study

We assessed the methodological quality and transparency of all the national clinical practice guidelines that were published in Croatia up until 2017 and explored the factors associated with their quality rating. An in-depth quantitative and qualitative analysis was performed using rigorous methodology. We evaluated the guidelines using a validated AGREE II instrument with four raters; we used multiple linear regressions to identify the predictors of quality; and two focus groups, including guideline developers, to further explore the guideline development process. The majority of the guidelines (N = 74) were developed by medical societies. The guidelines’ quality was rated low: the median standardized AGREE II score was low, 36% (IQR 28–42), and so were the overall-assessments. The aspects of the guidelines that were rated best were the “clarity of presentation” and the “scope and purpose” (median ≥ 59%); however, the other four domains received very low scores (15–33%). Overall, the guideline quality did not improve over time. The guidelines that were developed by medical societies scored significantly worse than those developed by governmental, or unofficial working groups (12–43% per domain). In focus group discussions, inadequate methodology, a lack of implementation systems in place, a lack of awareness about editorial independence, and broader expertise/perspectives in working groups were identified as factors behind the low scores. The factors identified as affecting the quality of the national guidelines may help stakeholders who are developing interventions and education programs aimed at improving guideline quality worldwide.

Systematic review of guidelines for managing physical health during treatment for substance use disorders: Implications for the alcohol and other drug workforce

ISSUES

Substance use disorders are associated with significant physical health comorbidities, necessitating an integrated treatment response. However, service fragmentation can preclude the management of physical health problems during addiction treatment. The aim of this systematic review was to synthesise the recommendations made by clinical practice guidelines for addressing the physical health of people attending alcohol and other drug (AOD) treatment.

APPROACH

An iterative search strategy of grey literature sources was conducted from September 2020 to February 2021 to identify clinical practice guidelines. Content pertaining to physical health care during AOD treatment was extracted. Quality of guidelines were appraised using the Appraisal of Guidelines Research and Evaluation II (AGREE-II) tool.

FINDINGS

Thirty-three guidelines were included for review. Fourteen guidelines were considered high quality based on AGREE-II scores. Neurological conditions (90.9%) and hepatitis (81.8%) were the most frequent health problems addressed. Most guidelines recommended establishing referral pathways to address physical health comorbidities (90.9%). Guidance on facilitating these referral pathways was less common (42.4%). Guidelines were inconsistent in their recommendations related to oral health, tobacco use, physical activity, nutrition and the use of standardised assessment tools.

IMPLICATIONS AND CONCLUSIONS

Greater consistency and specificity in the recommendations made for integrating physical health care within addiction treatment is needed. Ensuring that recommendations are applicable to the AOD workforce and to treatment services limited by funding and resource constraints should enhance implementation. Future guideline development groups should consider increased consultation with the AOD workforce and inclusion of clinical tools and decision aids to facilitate referral pathways.

Biopsychosocial Model Domains in Clinical Practice Guidelines for Return to Sport After ACL Injury: Systematic Review Using the AGREE II Checklist

CONTEXT

The current status of return-to-sport (RTS) criteria can be understood from the International Classification of Functioning, Disability, and Health (ICF), which emphasizes an individual-centered approach and inclusion of all domains of human functioning, and ensures the multifactorial and biopsychosocial nature of decision-making.

OBJECTIVE

To analyze the inclusion of biopsychosocial model domains in clinical practice guidelines (CPGs) for RTS after anterior cruciate ligament (ACL) injury, as well as the quality of these CPGs.

STUDY DESIGN

Systematic review of CPGs.

LEVEL OF EVIDENCE

Level 1.

SEARCH STRATEGY

Two independent reviewers developed the search strategy, and a third reviewer corrected and compiled the developed strategies used.

DATA SOURCES

Ovid/Medline, Embase, and PEDro without restriction dates.

STUDY SELECTION

CPGs for RTS after ACL injury at any age or sport level, and published in English.

DATA EXTRACTION

Two independent reviewers codified the RTS criteria recommended in the CPGs according to the ICF domains, and the Appraisal of Guidelines for Research and Evaluation II (AGREE II Checklist) was used for critical appraisal.

RESULTS

A total of 715 records were identified, and 7 CPGs were included. Frequency distribution of the biopsychosocial model domains was as follows: body functions (37.77%), activity and participation (20.00%), body structure (13.33%), environmental factors (11.11%), and personal factors (8.88%). In the AGREE II Checklist, the lowest mean domain scores were for rigor of development (37.86 ± 36.35) and applicability (49.29 ± 22.30), and 71.42% were of low or moderate quality.

CONCLUSION

The CPGs cannot address the biopsychosocial model domains satisfactorily and some do not address all the ICF conceptual model components, emphasizing body functions and activity and participation domains. Therefore, the functioning model advocated by the World Health Organization has not yet been adequately incorporated into the recommendations for RTS after ACL injury. Moreover, most CPGs are of limited quality.

Appraisal of physical health guidelines for severe mental illness

Objectives The aim of this study was to identify current physical health guidelines for severe mental illness (SMI) and appraise them using the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument. Methods Relevant research databases and grey literature were systematically searched to identify physical health guidelines for people with SMI. The grey literature was explored by web searches and targeted searches of the English websites of relevant peak bodies and government health agencies from Organisation for Economic Co-operation and Development countries. Included guidelines were independently appraised by two authors (NF and JH) using the AGREE II Instrument. Results Of the 5352 records screened, 33 were assessed for eligibility. Fifteen practice guidelines met the inclusion criteria. The median domain scores as rated by the AGREE II Instrument ranged from 17% to 69%. The World Health Organization guideline, which demonstrated a broad range of clinical recommendations and sound methodological rigour, was rated the best. Conclusions Most guidelines scored poorly when rated by the AGREE II Instrument. However, these guidelines may still be useful in assisting evidence-based clinical practice. The methodological rigour of future guidelines can be improved by ensuring the AGREE II domains are addressed during the development phase. What is known about the topic? Compared with the general population, people with SMI experience greater chronic disease morbidity and mortality. There is limited evidence from randomised controlled trials to guide physical health care monitoring for people with SMI. Current guidelines and practice are largely based on expert consensus, clinical experience and good intentions. What does this paper add? Using the AGREE II Instrument, this paper appraises the current physical health guidelines for people with SMI. The attributes of the guidelines examined included the evidence base, clarity of presentation, applicability in the real world, the involvement of stakeholders and conflicts of interest of various parties involved in guideline development. What are the implications for practitioners? This review highlights the scarcity of high-quality and evidence-based guidelines for clinicians and researchers to address the physical health of people with SMI.

Guidelines on the Role of Nursing in Patients with Chronic Inflammatory Arthritis: Do we "AGREE II"?

In the era of evidence-based practice, the role of the nurse in Rheumatology has broadened and became more complex, as a part of a multidisciplinary team. The European League Against Rheumatism (EULAR) has published recommendations for the role of the nurse in the management of chronic inflammatory arthritis in 2012, with an updated version in 2018. The objective of this study was to assess the methodological quality and reporting clarity of these recommendations using the AGREE II tool.

Enhancing capacity for clinical practice guidelines in South Africa

INTRODUCTION

Use of good quality, evidence-informed and up-to-date clinical practice guidelines (CPGs) has the potential to impact health outcomes. This paper describes the development, implementation and evaluation of a dedicated CPG training course to address the training needs of CPG stakeholders in South Africa.

METHODS

We first reviewed the content and teaching strategies of existing CPG courses. This review consisted of a systematic review of teaching and learning strategies for guideline teams and a document review of existing courses offered by international guideline groups, universities and professional groups. We then strengthened an existing CPG course and evaluated it.

RESULTS

We found no studies on teaching and learning strategies for guideline teams. We identified six CPG courses being offered as full courses (part of a postgraduate degree program) by universities or as independent training for continuing professional education by professional groups. Contents focused on new guideline development. One course included alternative methods of guideline approaches such as contextualization and adaptation. The format varied from face-to-face sessions, to online sessions, group exercises and discussions, seminar format and project based activities. The revised CPG four-month long course that we implemented was designed to be pragmatic, reflective and contextually relevant. It used local guideline examples, authentic tasks, and an online forum for discussions and resources. It covered de novo CPG development, alternative methods of development (adopting, contextualising, adapting), and implementing CPGs. Course evaluation identified strengths and areas for improvement.

CONCLUSION

Dedicated capacity development has potential to positively influence CPG development and implementation.

A Randomized Controlled Trial Investigating Online Training for Prelinguistic Communication

Purpose This study explored the utility of online training as a platform for teaching early intervention speech-language pathologists to recognize potentially communicative, prelinguistic behaviors in young children with physical disabilities and complex communication needs. Method Using a randomized controlled trial, 45 early intervention speech-language pathologists were randomly assigned to one of three conditions within an online training: practice with implicit problem-solving (identification condition), practice with explicit problem-solving (reflection condition), or no practice (control condition). Knowledge about early communication, skill at recognizing prelinguistic behaviors, time taken to complete the training, and perceptions of the training experience were examined. Results Participants in the no-practice control condition took significantly less time to complete the training, achieved the same positive outcomes on the knowledge and skill assessments, and rated the training as appealing as compared with participants assigned to the more time-intensive identification and reflection practice conditions. Conclusions Results suggest the importance of considering efficiency and appeal when designing successful trainings for moving evidence into practice.

Systematic Appraisal of the American College of Rheumatology Clinical Practice Guidelines

Objective

Because the American College of Rheumatology (ACR) practice guidelines affect the United States' and international treatment practice, we used the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument to characterize the quality of the guidelines and to identify potential areas for improvement.

Methods

Using the six quality domains in the AGREE II online tool, four reviewers assessed the practice guidelines available at the ACR website and the immediately previous version to summarize the domain scores for each guideline and examine trends over time.

Results

As of April 2016, the ACR website listed nine guidelines, four with immediate previous versions. Based on AGREE II, the minimum and maximum for each domain (with higher being better) of the current guidelines were 78-99 for Scope and Purpose, 57-99 for Stakeholder Involvement, 87- 96 for Rigor of the Methodology, 83-99 for Clarity of Presentation, 49-78 for Applicability, 69-85 for Editorial Independence, and 71-96 overall. Over time, although the average domain quality of the guidelines improved for all, the Applicability and Editorial Independence domains had the least amount of improvement. For the four guidelines with previous versions, the mean (SD) absolute improvements for each domain were 18 (±11) for Scope and Purpose, 13 (±8) for Stakeholder Involvement, 38 (±22) for Rigor of the Methodology, 25 (±15) for Clarity of Presentation, 22 (±12) for Applicability, 24 (±17) for Editorial Independence, and 31 (±5) overall.

Conclusion

Based on the AGREE II instrument, the ACR guidelines have achieved high quality over the past 16 years. The Applicability and Editorial Independence domains have the greatest potential for future improvement.

How We Evaluate Postgraduate Medical E-Learning: Systematic Review

BACKGROUND

Electronic learning (e-learning) in postgraduate medical education has seen a rapid evolution; however, we tend to evaluate it only on its primary outcome or learning aim, whereas its effectiveness also depends on its instructional design. We believe it is important to have an overview of all the methods currently used to evaluate e-learning design so that the preferred method may be identified and the next steps needed to continue to evaluate postgraduate medical e-learning may be outlined.

OBJECTIVE

This study aimed to identify and compare the outcomes and methods used to evaluate postgraduate medical e-learning.

METHODS

We performed a systematic literature review using the Web of Science, PubMed, Education Resources Information Center, and Cumulative Index of Nursing and Allied Health Literature databases. Studies that used postgraduates as participants and evaluated any form of e-learning were included. Studies without any evaluation outcome (eg, just a description of e-learning) were excluded.

RESULTS

The initial search identified 5973 articles, of which we used 418 for our analysis. The types of studies were trials, prospective cohorts, case reports, and reviews. The primary outcomes of the included studies were knowledge, skills, and attitude. A total of 12 instruments were used to evaluate a specific primary outcome, such as laparoscopic skills or stress related to training. The secondary outcomes mainly evaluated satisfaction, motivation, efficiency, and usefulness. We found 13 e-learning design methods across 19 studies (4% 19/418). The methods evaluated usability, motivational characteristics, and the use of learning styles or were based on instructional design theories, such as Gagne's instructional design, the Heidelberg inventory, Kern's curriculum development steps, and a scale based on the cognitive load theory. Finally, 2 instruments attempted to evaluate several aspects of a design, based on the experience of creating e-learning.

CONCLUSIONS

Evaluating the effect of e-learning design is complicated. Given the diversity of e-learning methods, there are many ways to carry out such an evaluation, and probably, many ways to do so correctly. However, the current literature shows us that we have yet to reach any form of consensus about which indicators to evaluate. There is a great need for an evaluation tool that is properly constructed, validated, and tested. This could be a more homogeneous way to compare the effects of e-learning and for the authors of e-learning to continue to improve their product.

Impact of a multifaceted education program on implementing a pediatric palliative care guideline: a pilot study

BACKGROUND

A national clinical practice guideline for pediatric palliative care was published in 2013. So far there are only few reports available on whether an educational program fosters compliance with such a guideline implementation. We aimed to test the effect of the education program on actual compliance as well as documentation of compliance to the guideline.

METHODS

We performed a prospective study with pre- and post-intervention evaluation on compliance to the guideline of the nurse specialists of a pediatric palliative care team for case management at a children's university hospital. Eleven quality indicators were selected from 192 recommendations from the pediatric palliative care guideline, based on frequency, measurability and relevance. The multifaceted education program included e-learning and an interactive educational meeting. Four e-learning modules addressed 19 patient cases on symptoms, diagnostics and treatment, and a chart-documentation exercise. During the interactive educational meeting patient cases were discussed on how to use the guideline. Documentation of compliance to the guideline in the web-based patient-charts as well as actual compliance to the guideline through weekly web-based parent reports was measured before and after completion of the e-learning.

RESULTS

Eleven quality indicators were selected. The educational program did not result in significant improvement in compliance for any of these indicators. The indicators "treatment of nausea", "pain medications two steps ahead" and "pain medication for 48 h present", measured through parent reports, scored a compliance beyond 80 % before and after e-learning. The remaining indicators measuring compliance, as well as six indicators measuring documentation by chart review, showed a compliance below 80 % before and after e-learning.

CONCLUSIONS

The multifaceted education program did not lead to improvement in documentation of compliance to the guideline. Parent reported outcome revealed better performance and might be the more adequate assessment tool for future studies.

The AGREE Enterprise: a decade of advancing clinical practice guidelines

BACKGROUND

The original AGREE (Appraisal of Guidelines for REsearch and Evaluation) Instrument was published in 2003, and its revision, the AGREE II, in 2009. Together, they filled an important gap in the guideline and quality of care fields. Ten years later, the AGREE Enterprise reflects on a trajectory of projects and international collaboration that have contributed to advancing the science and quality of practice guidelines and the uptake of AGREE/AGREE II.

FINDINGS

The AGREE Enterprise has undertaken activities to improve the tool and to develop resources to support its use. Since 2003, the uptake and adoption of AGREE by the international community has been swift and broad. A total of 33 language translations of the original AGREE Instrument and the current AGREE II are available and were initiated by the international community. A recent scan of the published literature identified over 600 articles that referenced the AGREE tools. The AGREE tools have been widely received and applied, with several organizations having incorporated the AGREE as part of their formal practice guideline programs. Since its redevelopment in 2010, the AGREE Enterprise website (www.agreetrust.org) continues to experience steady increases in visitors per month and currently has over 10,000 registered users.

CONCLUSIONS

The AGREE Enterprise has contributed to the advancements of guidelines through research activities and international participation by scientific and user communities. As we enter a new decade, we look forward to ongoing collaborations and contributing to further advancements to improve quality of care and health care systems.