Colonoscopy competence assessment tools: a systematic review of validity evidence

Computer-aided quality assessment of endoscopist competence during colonoscopy: a systematic review

BACKGROUND AND AIMS

Endoscopists' competence can vary widely, as shown in the variation in the adenoma detection rate (ADR). Computer-aided quality assessment (CAQ) can automatically assess performance during individual procedures. In this review we identified and described different CAQ systems for colonoscopy.

METHODS

A systematic review of the literature was done using MEDLINE, EMBASE, and Scopus based on 3 blocks of terms according to the inclusion criteria: colonoscopy, competence assessment, and automatic evaluation. Articles were systematically reviewed by 2 reviewers, first by abstract and then in full text. The methodological quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

Of 12,575 identified studies, 6831 remained after removal of duplicates and 6806 did not pass the eligibility criteria and were excluded, leaving 25 studies, of which 13 studies were included in the final analysis. Five categories of CAQ systems were identified: withdrawal speedometer (7 studies), endoscope movement analysis (3 studies), effective withdrawal time (1 study), fold examination quality (1 study), and visual gaze pattern (1 study). The withdrawal speedometer was the only CAQ system that tested its feedback by examining changes in ADR. Three studies observed an improvement in ADR, and 2 studies did not. The methodological quality of the studies was high (mean MERSQI, 15.2 points; maximum, 18 points).

CONCLUSIONS

Thirteen studies developed or tested CAQ systems, most frequently by correlating it to the ADR. Only 5 studies tested feedback by implementing the CAQ system. A meta-analysis was impossible because of the heterogeneous study designs, and more studies are warranted.

Supporting self-regulated-learning in colonoscopy training-A comparison cohort trial

Aim

This study aimed to demonstrate that using a self-regulated learning (SRL) approach can improve colonoscopy performance skills.

Background

Colonoscopy is the gold standard for detecting colorectal cancer and removing its precursors: polyps. Acquiring proficiency in colonoscopy is challenging, requiring completion of several hundred procedures. SRL seems to be beneficial to help trainees acquire competencies in regulating their future learning processes and enhance the outcomes of current learning situations. SRL is a learner-centred approach that refers to a trainee's ability to understand and control their learning environment, including cognitions, motivations and emotions. The key abilities include self- and situational awareness, task analysis, and strategic planning. This study is the first to use an SRL approach for workplace-based colonoscopy training.

Methods

In this comparison cohort trial, participants used two SRL supports: a self-review of videotaped performance, and an online learning platform with procedural and conceptual knowledge about colonoscopy. In the control cohort, participants performed patient-based colonoscopy as usual in their departments. Improvement was monitored via three video-based ratings (study start, end of the study period, and follow-up) using the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT). Outcomes were analysed using two-way analysis of variance with repeated measurements.

Results

This study recruited 21 participants (12, intervention cohort; nine, control cohort); 58 videos were recorded. The intraclass correlation coefficient was 0.88 (95% CI 0.61-0.98; p < 0.001). The global rating scale (GRS) and checklist (CL) in GiECAT were analysed separately. No statistically significant main effects of cohort (GRS: F(1,16) = 2.84, p = 0.11; CL: F(1,16) = 1.06, p = 0.32), test (GRS: F(2,32) = 2.56, p = 0.09; CL: F(2,32) = 0.76, p = 0.48), or interactions between cohort and test were observed (GRS: F(2,32) = 1.16, p = 0.33; CL: F(2,32) = 1.01, p = 0.37).

Conclusions

SRL in patient-based colonoscopy is feasible; however, no clear effect on performance scores was observed.

Exploring Endoscopic Competence in Gastroenterology Training: A Simulation-Based Comparative Analysis of GAGES, DOPS, and ACE Assessment Tools

Purpose

Accurate and convenient evaluation tools are essential to document endoscopic competence in Gastroenterology training programs. The Direct Observation of Procedural Skills (DOPS), Global Assessment of Gastrointestinal Endoscopic Skills (GAGES), and Assessment of Endoscopic Competency (ACE) are widely used validated competency assessment tools for gastrointestinal endoscopy. However, studies comparing these 3 tools are lacking, leading to lack of standardization in this assessment. Through simulation, this study seeks to determine the most reliable, comprehensive, and user-friendly tool for standardizing endoscopy competency assessment.

Methods

A mixed-methods quantitative-qualitative approach was utilized with sequential deductive design. All nine trainees in a gastroenterology training program were assessed on endoscopic procedural competence using the Simbionix Gi-bronch-mentor high-fidelity simulator, with 2 faculty raters independently completing the 3 assessment forms of DOPS, GAGES, and ACE. Psychometric analysis was used to evaluate the tools' reliability. Additionally, faculty trainers participated in a focused group discussion (FGD) to investigate their experience in using the tools.

Results

For upper GI endoscopy, Cronbach's alpha values for internal consistency were 0.53, 0.8, and 0.87 for ACE, DOPS, and GAGES, respectively. Inter-rater reliability (IRR) scores were 0.79 (0.43-0.92) for ACE, 0.75 (-0.13-0.82) for DOPS, and 0.59 (-0.90-0.84) for GAGES. For colonoscopy, Cronbach's alpha values for internal consistency were 0.53, 0.82, and 0.85 for ACE, DOPS, and GAGES, respectively. IRR scores were 0.72 (0.39-0.96) for ACE, 0.78 (-0.12-0.86) for DOPS, and 0.53 (-0.91-0.78) for GAGES. The FGD yielded three key themes: the ideal tool should be scientifically sound, comprehensive, and user-friendly.

Conclusion

The DOPS tool performed favourably in both the qualitative assessment and psychometric evaluation to be considered the most balanced amongst the three assessment tools. We propose that the DOPS tool be used for endoscopic skill assessment in gastroenterology training programs. However, gastroenterology training programs need to match their learning outcomes with the available assessment tools to determine the most appropriate one in their context.

Pediatric endoscopy: how can we improve patient outcomes and ensure best practices?

INTRODUCTION

Strategies to promote high-quality endoscopy in children require consensus around pediatric-specific quality standards and indicators. Using a rigorous guideline development process, the international Pediatric Endoscopy Quality Improvement Network (PEnQuIN) was developed to support continuous quality improvement efforts within and across pediatric endoscopy services.

AREAS COVERED

This review presents a framework, informed by the PEnQuIN guidelines, for assessing endoscopist competence, granting procedural privileges, audit and feedback, and for skill remediation, when required. As is critical for promoting quality, PEnQuIN indicators can be benchmarked at the individual endoscopist, endoscopy facility, and endoscopy community levels. Furthermore, efforts to incorporate technologies, including electronic medical records and artificial intelligence, into endoscopic quality improvement processes can aid in creation of large-scale networks to facilitate comparison and standardization of quality indicator reporting across sites.

EXPERT OPINION

PEnQuIN quality standards and indicators provide a framework for continuous quality improvement in pediatric endoscopy, benefiting individual endoscopists, endoscopy facilities, and the broader endoscopy community. Routine and reliable measurement of data, facilitated by technology, is required to identify and drive improvements in care. Engaging all stakeholders in endoscopy quality improvement processes is crucial to enhancing patient outcomes and establishing best practices for safe, efficient, and effective pediatric endoscopic care.

Validity evidence for observational ERCP competency assessment tools: a systematic review

BACKGROUND : Assessment of competence in endoscopic retrograde cholangiopancreatography (ERCP) is critical for supporting learning and documenting attainment of skill. Validity evidence supporting ERCP observational assessment tools has not been systematically evaluated. METHODS : We conducted a systematic search using electronic databases and hand-searching from inception until August 2021 for studies evaluating observational assessment tools of ERCP performance. We used a unified validity framework to characterize validity evidence from five sources: content, response process, internal structure, relations to other variables, and consequences. Each domain was assigned a score of 0-3 (maximum score 15). We assessed educational utility and methodological quality using the Accreditation Council for Graduate Medical Education framework and the Medical Education Research Quality Instrument, respectively. RESULTS : From 2769 records, we included 17 studies evaluating 7 assessment tools. Five tools were studied for clinical ERCP, one for simulated ERCP, and one for simulated and clinical ERCP. Validity evidence scores ranged from 2 to 12. The Bethesda ERCP Skills Assessment Tool (BESAT), ERCP Direct Observation of Procedural Skills Tool (ERCP DOPS), and The Endoscopic Ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) had the strongest validity evidence, with scores of 10, 12, and 11, respectively. Regarding educational utility, most tools were easy to use and interpret, and required minimal additional resources. Overall methodological quality (maximum score 13.5) was strong, with scores ranging from 10 to 12.5. CONCLUSIONS : The BESAT, ERCP DOPS, and TEESAT had strong validity evidence compared with other assessments. Integrating tools into training may help drive learners' development and support competency decision making.

Upskilling Pediatric Ileocolonoscopy

Upskilling in ileocolonoscopy is an important aspect of pediatric endoscopic practice as it enables endoscopists to learn additional skills through education and training to improve outcomes. With the advent of technologies, endoscopy is continuously evolving. Many devices can be applied to improve endoscopy quality and ergonomics. In addition, techniques such as dynamic position change can be employed to increase procedural efficiency and completeness. Key to upskilling is enhancing endoscopists' cognitive, technical and nontechnical skills and the concept of "training the trainer" to ensure trainers have the requisite skills to teach endoscopy effectively. This chapter details aspects of upskilling pediatric ileocolonoscopy.

Global trends in training and credentialing guidelines for gastrointestinal endoscopy: a systematic review

Background and study aims  Credentialing, the process through which an institution assesses and validates an endoscopist's qualifications to independently perform a procedure, can vary by region and country. Little is known about these inter-societal and geographic differences. We aimed to systematically characterize credentialing recommendations and requirements worldwide. Methods  We conducted a systematic review of credentialing practices among gastrointestinal and endoscopy societies worldwide. An electronic search as well as hand-search of World Endoscopy Organization members' websites was performed for credentialing documents. Abstracts were screened in duplicate and independently. Data were collected on procedures included in each document (e. g. colonoscopy, ERCP) and types of credentialing statements (procedural volume, key performance indicators (KPIs), and competency assessments). The primary objective was to qualitatively describe and compare the available credentialing recommendations and requirements from the included studies. Descriptive statistics were used to summarize data when appropriate. Results  We screened 653 records and included 20 credentialing documents from 12 societies. Guidelines most commonly included credentialing statements for colonoscopy, esophagogastroduodenoscopy (EGD), and ERCP. For colonoscopy, minimum procedural volumes ranged from 150 to 275 and adenoma detection rate (ADR) from 20 % to 30%. For EGD, minimum procedural volumes ranged from 130 to 1000, and duodenal intubation rate of 95 % to 100%. For ERCP, minimum procedural volumes ranged from 100 to 300 with selective duct cannulation success rate of 80 % to 90 %. Guidelines also reported on flexible sigmoidoscopy, capsule endoscopy, and endoscopic ultrasound. Conclusions  While some metrics such as ADR were relatively consistent among societies, there was substantial variation among societies with respect to procedural volume and KPI statements.

Accuracy of self-assessment in gastrointestinal endoscopy: a systematic review and meta-analysis

BACKGROUND

Assessment is necessary to ensure both attainment and maintenance of competency in gastrointestinal (GI) endoscopy, and this can be accomplished through self-assessment. We conducted a systematic review with meta-analysis to evaluate the accuracy of self-assessment among GI endoscopists.

METHODS

This was an individual participant data meta-analysis of studies that investigated self-assessment of endoscopic competency. We performed a systematic search of the following databases: Ovid MEDLINE, Ovid EMBASE, Wiley Cochrane CENTRAL, and ProQuest Education Resources Information Center. We included studies if they were primary investigations of self-assessment accuracy in GI endoscopy that used statistical analyses to determine accuracy. We conducted a meta-analysis of studies using a limits of agreement (LoA) approach to meta-analysis of Bland-Altman studies.

RESULTS

After removing duplicate entries, we screened 7138 records. After full-text review, we included 16 studies for qualitative analysis and three for meta-analysis. In the meta-analysis, we found that the LoA were wide (-41.0 % to 34.0 %) and beyond the clinically acceptable difference. Subgroup analyses found that both novice and intermediate endoscopists had wide LoA (-45.0 % to 35.1 % and -54.7 % to 46.5 %, respectively) and expert endoscopists had narrow LoA (-14.2 % to 21.4 %).

CONCLUSIONS

GI endoscopists are inaccurate in self-assessment of their endoscopic competency. Subgroup analyses demonstrated that novice and intermediate endoscopists were inaccurate, while expert endoscopists have accurate self-assessment. While we advise against the sole use of self-assessment among novice and intermediate endoscopists, expert endoscopists may wish to integrate it into their practice.

The Toronto Upper Gastrointestinal Cleaning Score: a prospective validation study

BACKGROUND : Assessment of mucosal visualization during esophagogastroduodenoscopy (EGD) can be improved with a standardized scoring system. To address this need, we created the Toronto Upper Gastrointestinal Cleaning Score (TUGCS). METHODS : We developed the TUGCS using Delphi methodology, whereby an international group of endoscopy experts iteratively rated their agreement with proposed TUGCS items and anchors on a 5-point Likert scale. After each Delphi round, we analyzed responses and refined the TUGCS using an 80 % agreement threshold for consensus. We used the intraclass correlation coefficient (ICC) to assess inter-rater and test-retest reliability. We assessed internal consistency with Cronbach's alpha and item-total and inter-item correlations with Pearson's correlation coefficient. We compared TUGCS ratings with an independent endoscopist's global rating of mucosal visualization using Spearman's ρ. RESULTS : We achieved consensus with 14 invited participants after three Delphi rounds. Inter-rater reliability was high at 0.79 (95 %CI 0.64-0.88). Test-retest reliability was excellent at 0.83 (95 %CI 0.77-0.87). Cronbach's α was 0.81, item-total correlation range was 0.52-0.70, and inter-item correlation range was 0.38-0.74. There was a positive correlation between TUGCS ratings and a global rating of visualization (r = 0.41, P = 0.002). TUGCS ratings for EGDs with global ratings of excellent were significantly higher than those for EGDs with global ratings of fair (P = 0.01). CONCLUSION : The TUGCS had strong evidence of validity in the clinical setting. The international group of assessors, broad variety of EGD indications, and minimal assessor training improves the potential for dissemination.

An Italian prospective multicenter study on colonoscopy practice and quality: What has changed in the last 10 years

BACKGROUND

A relevant number of adenomas can be missed during colonoscopy.

AIMS

Assess the current status of colonoscopy procedures in Italian centers.

METHODS

A prospective observational study involving 17 hospitals (34 endoscopists) included consecutive patients undergoing standard colonoscopy. In the first phase, endoscopists performed consecutive colonoscopies. In the second phase, retraining via an online learning platform was planned, while in the third phase data were collected analogously to phase 1.

RESULTS

A total of 3,504 patients were enrolled. Overall, a BBPS score ≥6 was obtained in 95.6% of cases (94.8% and 96.9% in the pre- and post-training phases, respectively). 88.4% of colonoscopies had a withdrawal time ≥6 min (88.2% and 88.7% in the pre- and post-training phases). Median adenoma detection rate (ADR) was 39.1%, with no significant differences between the pre- and post-training phases (40.1% vs 36.9%; P = 0.83). In total, 81% of endoscopists had a ADR performance above the 25% threshold.

CONCLUSION

High colonoscopy quality standards are achieved by the Italian hospitals involved. Quality improvement initiatives and repeated module-based colonoscopy-training have been promoted in Italy during the last decade, which appear to have had a significant impact on quality colonoscopy metrics together with the activation of colorectal cancer screening programs.

Endoscopy training in the Netherlands: a national survey among gastroenterology residents

Background and study aims  Training in endoscopy is a key objective of gastroenterology residency. There is currently no standardized or systematic training approach. This study evaluated and compared the current status of gastrointestinal endoscopy training programs in all teaching hospitals in the Netherlands from a resident perspective. Materials and methods  A national online survey with open and closed questions on gastrointestinal endoscopy training was administered to all gastroenterology residents (N = 180) in the eight educational regions in the Netherlands. Results  One hundred residents who had already started endoscopy training were included in the analyses. Sixty-five residents (65 %) were satisfied with their endoscopy training program. Participation in a preclinical endoscopy course was mandatory in seven of eight educational regions. Residents from the region without a mandatory endoscopy training course were significantly less likely to be satisfied with their endoscopy training program (32 %, P  = .011). Criteria used to determine the level of supervision differed greatly between teaching hospitals (e. g. assessed endoscopy competence, predefined period of time or number of procedures). Only 26 residents (26 %) reported uniformity in teaching methods and styles between different supervising gastroenterologists in their teaching hospital. Conclusions  Although most gastroenterology residents were satisfied with the endoscopy training program and endoscopy supervision in their teaching hospital, this study identified considerable local and regional variability. Future studies should be conducted to evaluate the trainers' perspective and trainers' behavior during endoscopy training sessions, which might eventually lead to the development of best practices regarding endoscopy training, including standardization of training programs and supervision methods.

Peer evaluation and feedback for invasive medical procedures: a systematic review

BACKGROUND

There is significant variability in the performance and outcomes of invasive medical procedures such as percutaneous coronary intervention, endoscopy, and bronchoscopy. Peer evaluation is a common mechanism for assessment of clinician performance and care quality, and may be ideally suited for the evaluation of medical procedures. We therefore sought to perform a systematic review to identify and characterize peer evaluation tools for practicing clinicians, assess evidence supporting the validity of peer evaluation, and describe best practices of peer evaluation programs across multiple invasive medical procedures.

METHODS

A systematic search of Medline and Embase (through September 7, 2021) was conducted to identify studies of peer evaluation and feedback relating to procedures in the field of internal medicine and related subspecialties. The methodological quality of the studies was assessed. Data were extracted on peer evaluation methods, feedback structures, and the validity and reproducibility of peer evaluations, including inter-observer agreement and associations with other quality measures when available.

RESULTS

Of 2,135 retrieved references, 32 studies met inclusion criteria. Of these, 21 were from the field of gastroenterology, 5 from cardiology, 3 from pulmonology, and 3 from interventional radiology. Overall, 22 studies described the development or testing of peer scoring systems and 18 reported inter-observer agreement, which was good or excellent in all but 2 studies. Only 4 studies, all from gastroenterology, tested the association of scoring systems with other quality measures, and no studies tested the impact of peer evaluation on patient outcomes. Best practices included standardized scoring systems, prospective criteria for case selection, and collaborative and non-judgmental review.

CONCLUSIONS

Peer evaluation of invasive medical procedures is feasible and generally demonstrates good or excellent inter-observer agreement when performed with structured tools. Our review identifies common elements of successful interventions across specialties. However, there is limited evidence that peer-evaluated performance is linked to other quality measures or that feedback to clinicians improves patient care or outcomes. Additional research is needed to develop and test peer evaluation and feedback interventions.

Colonoscopy Quality Assurance and Maintenance of Competency Among Pediatric Gastroenterology Staff Members: A Canadian Center Experience

Introduction

Colonoscopy quality indicators and maintenance of competency skills are relatively well established in the adult literature as compared to the pediatric gastroenterology. One of the suggested quality assurance measures is cecal intubation rate, which is suggested to be >90% in all colonoscopies as per American Society of Gastrointestinal Endoscopy (ASGE) guidelines. Terminal ileum (TI) intubations are essentially required for diagnostic reasons in pediatric colonoscopies as compared to the screening reasons in adults. Maintenance of competency in pediatric colonoscopies has been described in the literature but in smaller studies contrary to the adult ones. The aims of this study are to compare our center’s individual and group cecal intubation rates and compare it with the published literature, assess the group’s terminal ileal intubation rates in comparison with the published literature, assess the most common reasons for failure to intubate the cecum and/or terminal ileum, and to assess whether the presence of a trainee affects the intubation rates and the duration of the procedure.

Methods

A retrospective chart review was performed on all pediatric patients (0-18 years). Colonoscopies performed over a two-year period at our single center were included in the study. Patients scheduled for sigmoidoscopy and with altered anatomy of their colon were excluded from the study. The endoscopy and pathology reports were reviewed to ascertain whether the cecum and TI were reached. Quality of bowel preparation and any other stated reasons for incompletion were obtained. Clinical charts were reviewed to obtain indication for colonoscopy. Skin-to-skin time, which is the time from starting to the finishing of the procedure, was recorded for each procedure.

Results

A total of 391 colonoscopies were performed during the two-year study period by six gastroenterologists. The number of colonoscopies per staff ranged from 57 to 89 procedures. The overall cecal intubation rate was observed to be 98.5% (range: 95.9%-98.9%). TI intubation rate was lower at a rate of 83.1% (range: 63.3%-92.1%). The main stated reason for the inability to attain cecum/TI was technical difficulty and poor bowel prep. Daytime colonoscopies were shorter (39.5 minutes vs 50.3 minutes) compared to after-hours ones and had a higher TI intubation rate (84.5% vs 62.5%). No complications were encountered in the procedures.

Conclusion

Despite relatively low volumes, cecal intubation rates are very high, exceeding suggested standards. TI intubation rates were low, and there was noted to be a high degree of variability. However, multicentric collaborative evaluations are required over a longer period of time to establish relevant parameters for quality assurance and competency in pediatric endoscopy.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.