Simulation-based training for colonoscopy: establishing criteria for competency

Development and validation of a simulation-based assessment tool in colonoscopy

BACKGROUND

Colonoscopy is difficult to learn. Virtual reality simulation training is helpful, but how and when novices should progress to patient-based training has yet to be established. To date, there is no assessment tool for credentialing novice endoscopists prior to clinical practice. The aim of this study was to develop such an assessment tool based on metrics provided by the simulator. The metrics used for the assessment tool should be able to discriminate between novices, intermediates, and experts and include essential checklist items for patient safety.

METHODS

The validation process was conducted based on the Standards for Educational and Psychological Testing. An expert panel decided upon three essential checklist items for patient safety based on Lawshe's method: perforation, hazardous tension to the bowel wall, and cecal intubation. A power calculation was performed. In this study, the Simbionix GI Mentor II simulator was used. Metrics with discriminatory ability were identified with variance analysis and combined to form an aggregate score. Based on this score and the essential items, pass/fail standards were set and reliability was tested.

RESULTS

Twenty-four participants (eight novices, eight intermediates, and eight expert endoscopists) performed two simulated colonoscopies. Four metrics with discriminatory ability were identified. The aggregate score ranged from 4.2 to 51.2 points. Novices had a mean score of 10.00 (SD 5.13), intermediates 24.63 (SD 7.91), and experts 30.72 (SD 11.98). The difference in score between novices and the other two groups was statistically significant (p<0.01). Although expert endoscopists had a higher score, the difference was not statistically significant (p=0.40). Reliability was good (Cronbach's alpha=0.86). A pass/fail score was defined at 17.1 points with correct completion of three essential checklist items, resulting in three experts and three intermediates failing and one novice passing the assessment.

CONCLUSION

We established a valid and reliable assessment tool with a pass/fail standard on the simulator. We suggest using the assessment after simulation-based training before commencing work-based learning.

Assessment of esophagogastroduodenoscopy skills on simulators before real-life performance

Background and study aims  Operator competency is essential for esophagogastroduodenoscopy (EGD) quality, which makes appropriate training with a final test important. The aims of this study were to develop a test for assessing skills in performing EGD, gather validity evidence for the test, and establish a credible pass/fail score. Methods  An expert panel developed a practical test using the Simbionix GI Mentor II simulator (3 D Systems) and an EGD phantom (OGI 4, CLA Medical) with a diagnostic (DP) and a technical skills part (TSP) for a prospective validation study. During the test a supervisor measured: 1) total time; 2) degree of mucosal visualization; and 3) landmarks and pathology identification. The contrasting groups standard setting method was used to establish a pass/fail score. Results  We included 15 novices (N), 10 intermediates (I), and 10 experienced endoscopists (E). The internal structure was high with a Cronbach's alpha of 0.76 for TSP time consumption and 0.74 for the identification of landmarks. Mean total times, in minutes, for the DP were N 15.7, I 11.3, and E 7.0, and for TSP., they were N 7.9, I 8.9, and E 2.9. The total numbers of identified landmarks were N 26, I 41, and E 48. Mean visualization percentages were N 80, I 71, and E 71. A pass/fail standard was established requiring identification of all landmarks and performance of the TSP in < 5 minutes. All experienced endoscopists passed, while none of the endoscopists in the other categories did. Conclusions  We established a test that can distinguish between participants with different competencies. This enables an objective and evidence-based approach to assessment of competencies in EGD.

Defining medical simulators for simulation-based education in EUS: Theoretical approach and a narrative review

Choosing the right simulator for tasks in simulation-based education in medicine will affect the trainees’ skills. However, there is a shortage in the vocabularies used for describing medical simulators and the contextual usage of simulators. We propose methods for approaching the task of choosing and defining the simulators needed, regardless of it being an acquisition or development process. It is advocated that efforts are made in defining the simulator's requirements before making any choice in regards to development processes. Multiple advantages are attained by keeping the simulator simple, both educational and development wise. Issues on validating simulators are discussed and highlighted as actions where interprofessional communication is likely to fail. The following conventional terms in medical education are problematic in regard to establishing a clear communication: Virtual reality, fidelity, validation, and simulation. The text is finalized in a short discussion on applying the methods in an EUS/endobronchial ultrasound (EBUS) context. The work is the authors’ interpretation of an invitation having the title “Development of EUS and EBUS training models and simulators.”

Colonoscopy competence assessment tools: a systematic review of validity evidence

BACKGROUND

Assessment tools are essential for endoscopy training, being required to support feedback provision, optimize learner capabilities, and document competence. We aimed to evaluate the strength of validity evidence that supports the available colonoscopy direct observation assessment tools using the unified framework of validity.

METHODS

We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until 8 April 2020. We extracted data outlining validity evidence (content, response process, internal structure, relations to other variables, and consequences) from the five sources and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

From 10 841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 each assessed cognitive and integrative skills. Validity evidence scores ranged from 1-15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret, and required minimal resources. MERSQI scores ranged from 9.5-11.5 (maximum score 14.5).

CONCLUSIONS

The ACE, DOPS, and GiECAT have strong validity evidence compared with other assessments. Future studies should identify barriers to widespread implementation and report on the use of these tools in credentialing examinations.

Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations

Background

The contrasting groups’ standard setting method is commonly used for consequences analysis in validity studies for performance in medicine and surgery. The method identifies a pass/fail cut-off score, from which it is possible to determine false positives and false negatives based on observed numbers in each group. Since groups in validity studies are often small, e.g., due to a limited number of experts, these analyses are sensitive to outliers on the normal distribution curve.

Methods

We propose that these shortcomings can be addressed in a simple manner using the cumulative distribution function.

Results

We demonstrate considerable absolute differences between the observed false positives/negatives and the theoretical false positives/negatives. In addition, several important examples are given.

Conclusions

We propose that a better reporting strategy is to report theoretical false positives and false negatives together with the observed false positives and negatives, and we have developed an Excel sheet to facilitate such calculations.

Trial registration

Not relevant.

Electronic supplementary material

The online version of this article (10.1186/s41077-018-0064-7) contains supplementary material, which is available to authorized users.

Methods for certification in colonoscopy - a systematic review

INTRODUCTION

Reliable, valid, and feasible assessment tools are essential to ensure competence in colonoscopy. This study aims to provide an overview of the existing assessment methods and the validity evidence that supports them.

METHODS

A systematic search was conducted in October 2016. Pubmed, EMBASE, and PsycINFO were searched for studies evaluating assessment methods to ensure competency in colonoscopy. Outcome variables were described and evidence of validity was explored using a contemporary framework.

RESULTS

Twenty-five observational studies were included in the systematic review. Most studies were based on small sample sizes. The studies were categorized after outcome measures into five groups: Clinical process related outcome metrics (n = 2), direct observational colonoscopy assessment (n = 8), simulator based metrics (n = 11), automatic computerized metrics (n = 2), and self-assessment (n = 1). Validity score varied among the studies and only five studies presented sufficient evidence to recommend the tool for clinical assessment.

CONCLUSIONS

The objectives vary throughout the presented tools. Some tools are global tools where others focus on procedural technical skill assessment or even part-task skills. There is a tendency in the most recent studies towards more specific assessment of technical skills. The majority of assessment methods lack sufficient validity evidence.

An automatic measure of progression during colonoscopy correlates to patient experienced pain

BACKGROUND

Colonoscopy screening and surveillance programs depend on patient's tolerable experience, which is associated with competence of the endoscopist. The Colonoscopy Progression Score (CoPS) is an automated tool based on recording of the Magnetic Scope Imager (MEI) picture in order to track progression. CoPS deliver a numeric score and a graphic map. A high score expresses a rapid and smooth progression. Aims of study were to explore the correlation between CoPS and patient experienced pain and to identity locations associated with pain.

METHODS AND MATERIALS

Patients listed for colonoscopy were included and asked to reply to pain by pressing a rubber ball. The signal was recorded simultaneous to CoPS. Patients evaluated the experience on a Visual Analogue Scale (VAS). CoPS and recorded pain events were used to create a pain sensitive CoPS-map (S-CoPS map).

RESULTS

A total of 58 complete recordings were used for evaluation. We demonstrated a moderate correlation between CoPS and patient experienced pain, Pearson's r = -0.47 (p < .001). A low CoPS was associated with a painful colonoscopy and a high CoPS excluded severe pain. Sensitivity and specificity was 0.79 and 0.60 and AUC was 0.61 Passage of the sigmoid colon, right and left flexures were associated with pain for 51%, 33% and 25% of the patients, respectively.

CONCLUSION

A moderate correlation between CoPS and patient experienced pain suggest that CoPS measure inserting skills but might also be a measure of a gentle performance. The graphic S-CoPS-map can be used to point-out painful passages and aid planning of future colonoscopies.

Colonoscopy procedure simulation: virtual reality training based on a real time computational approach

BACKGROUND

Colonoscopy plays an important role in the clinical screening and management of colorectal cancer. The traditional 'see one, do one, teach one' training style for such invasive procedure is resource intensive and ineffective. Given that colonoscopy is difficult, and time-consuming to master, the use of virtual reality simulators to train gastroenterologists in colonoscopy operations offers a promising alternative.

METHODS

In this paper, a realistic and real-time interactive simulator for training colonoscopy procedure is presented, which can even include polypectomy simulation. Our approach models the colonoscopy as thick flexible elastic rods with different resolutions which are dynamically adaptive to the curvature of the colon. More material characteristics of this deformable material are integrated into our discrete model to realistically simulate the behavior of the colonoscope.

CONCLUSION

We present a simulator for training colonoscopy procedure. In addition, we propose a set of key aspects of our simulator that give fast, high fidelity feedback to trainees. We also conducted an initial validation of this colonoscopic simulator to determine its clinical utility and efficacy.

A System to Assess the Competency for Interpretation of Esophageal Manometry Identifies Variation in Learning Curves

BACKGROUND & AIMS

Quality esophageal high-resolution manometry (HRM) studies require competent interpretation of data. However, there is little understanding of learning curves, training requirements, or measures of competency for HRM. We aimed to develop and use a competency assessment system to examine learning curves for interpretation of HRM data.

METHODS

We conducted a prospective multicenter study of 20 gastroenterology trainees with no experience in HRM, from 8 centers, over an 8-month period (May through December 2015). We designed a web-based HRM training and competency assessment system. After reviewing the training module, participants interpreted 50 HRM studies and received answer keys at the fifth and then at every second interpretation. A cumulative sum procedure produced individual learning curves with preset acceptable failure rates of 10%; we classified competency status as competency not achieved, competency achieved, or competency likely achieved.

RESULTS

Five (25%) participants achieved competence, 4 (20%) likely achieved competence, and 11 (55%) failed to achieve competence. A minimum case volume to achieve competency was not identified. There was no significant agreement between diagnostic accuracy and accuracy for individual HRM skills.

CONCLUSIONS

We developed a competency assessment system for HRM interpretation; using this system, we found significant variation in learning curves for HRM diagnosis and individual skills. Our system effectively distinguished trainee competency levels for HRM interpretation and contrary to current recommendations, found that competency for HRM is not case-volume specific.

Accredited residents perform colonoscopy to the same high standards as consultants

BACKGROUND

Endoscopy remains a critical component of General Surgery and Gastroenterology training. Whilst residents need to gain experience, the quality of endoscopy which patients receive cannot be compromised. We conducted this study to compare quality indicators between consultants and residents with regards to colonoscopy.

METHODS

A review of colonoscopies from a prospectively collected database was performed from September 2011 to February 2016. Quality indicators such as caecum intubation rate, adenoma detection rate, adherence to a 6-min withdrawal rule, mean number of polyps detected per colonoscope, and complications were collected and compared between the two groups.

RESULTS

In total, out of 25,749 colonoscopies that were performed, 14,168 (55.0%) were performed by Consultants. Consultants achieved a better caecum intubation rate compared with residents (96.0% vs 94.9%, p < 0.001), and were more compliant to the 6-min withdrawal rule (74.7% vs 68.6%, p < 0.001). There were, however, no statistically significant differences in the adenoma detection rate (33.5% vs 34.5%, p = 0.098). Bleeding was a rare complication that was encountered more frequently in colonoscopies performed by consultants than for residents (0.002% vs 0.00008%, p < 0.001). There were only three (%) perforations in the entire series, and all were from colonoscopies performed by Consultants.

CONCLUSION

Given the proper training, residents are able to perform colonoscopy with the same level of competence as consultants. Whilst colonoscopic related complications are often tied to the difficulty of the procedures, the adherence to the 6-min withdrawal rule must be reinforced and continually educated to both residents and consultants.

Existing and Future Educational Needs in Graduate and Postgraduate Education

INTRODUCTION

This review was designed to make recommendations on future educational needs, principles of curricular development, and how the International Society for Sexual Medicine (ISSM) should address the need to enhance and promote human sexuality education around the world.

AIM

To explore the ways in which graduate and postgraduate medical education in human sexuality has evolved and is currently delivered.

METHODS

We reviewed existing literature concerning sexuality education, curriculum development, learning strategies, educational formats, evaluation of programs, evaluation of students, and faculty development. We reviewed literature relating to four main areas: (i) the current status of the international regulation of training in sexual medicine; (ii) the current delivery of education and training in sexual medicine; (iii) resident and postgraduate education in sexual medicine surgery; and (iv) education and training for allied health professionals.

RESULTS

The main findings in these four areas are as follows. Sexual medicine has grown considerably as a specialty during the past 20 years, with many drivers being identified. However, the regulatory aspects of training, assessment, and certification are currently in the early stages of development and are in many ways lagging behind the scientific and clinical knowledge in the field. However, there are examples of the development of curricula with accompanying assessments that have attempted to set standards of education and training that might underlie the delivery of high-quality care to patients in sexual medicine. The development of competence assessment has been applied to surgical training in sexual medicine, and there is increasing interest in simulation as a means of enhancing technical skills training. Although the focus of curriculum development has largely been the medical profession, there is early interest in the development of standards for training and education of allied health professionals.

CONCLUSION

Organizations of professionals in sexual health, such as the ISSM, have an opportunity, and indeed a responsibility, to provide and disseminate learning opportunities, curricula, and standards of training for doctors and allied health professionals in sexual medicine. Eardley I, Reisman Y, Goldstein S, et al. Existing and Future Educational Needs in Graduate and Postgraduate Education. J Sex Med 2017;14:475-485.

Combining different methods improves assessment of competence in colonoscopy

OBJECTIVES

To develop a reliable method of assessing competence in colonoscopy based on multiple sources.

MATERIALS AND METHODS

Physicians with varying degrees of experience in colonoscopy performed two colonoscopies each in a standardized simulated environment. Their performances were assessed under direct observation by an expert rater and by automatic computerized analysis of operator movements and scope movements, respectively. Reliability (Cronbach's alpha) for subjective assessment, time to cecum, analysis of operator movement and analysis of scope movements were calculated. Composite score calculations were used to explore different combinations of the measures.

RESULTS

Twenty physicians were included in the study. The reliability (Cronbach's alpha) were 0.92, 0.57, 0.87 and 0.55 for the subjective score assessed under direct observation, time to cecum, distance between operator's hands and colonoscopy progression score, respectively. Equal weight (=25%) to all four methods resulted in a reliability of 0.91 and optimal weighting of the methods (55%, 10%, 25% and 10%, respectively) resulted in a maximum reliability of 0.95.

CONCLUSION

Combining subjective expert ratings with automated objective assessments results in a less biased and more reliable assessment of competence in colonoscopy.

The efficacy of training insertion skill on a physical model colonoscopy simulator

Background and study aims: Prior research supports the validity of performance measures derived from the use of a physical model colonoscopy simulator - the Kyoto Kagaku Colonoscope Training Model (Kyoto Kagaku Co. Ltd., Kyoto, Japan) - for assessing insertion skill. However, its use as a training tool has received little research attention. We assessed the efficacy of a brief structured program to develop basic colonoscope insertion skill through unsupervised practice on the model. Participants and methods: This was a training study with pretesting and post-testing. Thirty-two colonoscopy novices completed an 11-hour training regime in which they practiced cases on the model in a colonoscopy simulation research laboratory. They also attempted a series of test cases before and after training. For each outcome measure (completion rates, time to cecum and peak force applied to the model), we compared trainees' post-test performance with the untrained novices and experienced colonoscopists from a previously-reported validation study. Results: Compared with untrained novices, trained novices had higher completion rates and shorter times to cecum overall (Ps < .001), but were out-performed by the experienced colono-scopists on these metrics (Ps < .001). Nevertheless, their performance was generally closer to that of the experienced group. Overall, trained novices did not differ from either experience-level comparison group in the peak forces they applied (P > .05). We also present the results broken down by case. Conclusions: The program can be used to teach trainees basic insertion skill in a more or less self-directed way. Individuals who have completed the program (or similar training on the model) are better prepared to progress to supervised live cases.

Competency based medical education in gastrointestinal motility

Traditional apprenticeship-based medical education methods focusing on subjective evaluations and case-volume requirements do not reliably produce clinicians that provide high-quality care in unsupervised practice. Consequently, training approaches are shifting towards competency based medical education, which incorporates robust assessment methods and credible standards of physician proficiency. However, current gastroenterology and hepatology training in the US continues to utilize procedural volume and global impressions without standardized criteria as markers of competence. In particular, efforts to optimize competency based training in gastrointestinal (GI) motility are not underway, even though GI motility disorders account for nearly half of outpatient gastroenterology visits. These deficiencies compromise the quality of patient care. Thus, there is a great need and opportunity to shift our focus in GI motility training towards a competency based approach. First, we need to clarify the variable rates of learning for individual diagnostic tests. We must develop integrated systems that standardize training and monitor physician competency for GI motility diagnostics. Finally, as a profession and society, we must create certification processes to credential competent physicians. These advances are critical to optimizing the quality of GI motility diagnostics in practice.

Systematic review to establish absolute standards for technical performance in surgery

BACKGROUND

Standard setting allows educators to create benchmarks that distinguish between those who pass and those who fail an assessment. It can also be used to create standards in clinical and simulated procedural skill. The objective of this review was to perform a systematic review of the literature using absolute standard-setting methodology to create benchmarks in technical performance.

METHODS

A systematic review was conducted by searching MEDLINE, Embase, PsycINFO and the Cochrane Database of Systematic Reviews. Abstracts of retrieved studies were reviewed and those meeting the inclusion criteria were selected for full-text review. The quality of evidence presented in the included studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI), where a score of 14 or more of 18 indicates high-quality evidence.

RESULTS

Of 1809 studies identified, 37 used standard-setting methodology for assessment of procedural skill. Of these, 24 used participant-centred and 13 employed item-centred methods. Thirty studies took place in a simulated environment, and seven in a clinical setting. The included studies assessed residents (26 of 37), fellows (6 of 37) and staff physicians (17 of 37). Seventeen articles achieved a MERSQI score of 14 or more of 18, whereas 20 did not meet this mark.

CONCLUSION

Absolute standard-setting methodologies can be used to establish cut-offs for procedural skill assessments.

Senior Surgical Resident Confidence in Performing Flexible Endoscopy: What Can We Do Differently?

INTRODUCTION

The American Board of Surgery endoscopy requirements for general surgery training are evolving. In 2006, the Residency Review Committee in Surgery increased the total number of endoscopy cases required before completion of general surgery residency training. This requirement is set to change further, given the new Flexible Endoscopic Curriculum that would be a requirement for applicants graduating surgical training during or after the 2017 to 2018 academic year. Given these changes, our goal was to evaluate the confidence of senior surgical residents performing flexible endoscopy.

METHODS

A survey was developed and sent to general surgery residents nationally, querying them regarding demographics and program-specific characteristics; additionally they were asked to rate their confidence level in performing flexible upper endoscopy and colonoscopy on a Likert scale of 1 to 5. We then compared those residents who indicated confidence (Likert scale 4-5) to those who did not (Likert scale 1-3). For the purpose of this study, only senior (postgraduate year 4 and 5) general surgery residents were assessed.

RESULTS

We received 1176 responses from senior surgical residents: 56% of these were postgraduate year 5 residents, 65% male, 68% from University Programs, and 56% from programs associated with a Veteran's Affairs Hospital; 33% were from programs in the Northeast, 29% in the South, 24% in the Midwest, and 14% in the West; 75% were going on to additional fellowship training after the completion of residency; 42% indicated that they would go into academic practice and 32% into private practice; 66.7% reported confidence performing upper endoscopy and 52.7% reported confidence performing colonoscopy. Male gender, overall operative volume, and graduating from a medium-sized program or program in the South were associated with increased confidence performing flexible endoscopy.

CONCLUSIONS

A large percentage of senior residents do not report confidence in performing flexible endoscopy. Although increasing the number of cases required for graduation has likely helped improve the training of residents in endoscopy, additional improvements in training are required. The Flexible Endoscopic Curriculum helps standardize the curriculum and demonstrate that the graduating resident has the fundamental knowledge and skills required in the performance of endoscopy. Simulation training and dedicated endoscopic rotations during the course of residency training could help improve endoscopy training and proficiency for future graduating residents.

A Review of Endoscopic Simulation: Current Evidence on Simulators and Curricula

Upper and lower endoscopy is an important tool that is being utilized more frequently by general surgeons. Training in therapeutic endoscopic techniques has become a mandatory requirement for general surgery residency programs in the United States. The Fundamentals of Endoscopic Surgery has been developed to train and assess competency in these advanced techniques. Simulation has been shown to increase the skill and learning curve of trainees in other surgical disciplines. Several types of endoscopy simulators are commercially available; mechanical trainers, animal based, and virtual reality or computer-based simulators all have their benefits and limitations. However they have all been shown to improve trainee's endoscopic skills. Endoscopic simulators will play a critical role as part of a comprehensive curriculum designed to train the next generation of surgeons. We reviewed recent literature related to the various types of endoscopic simulators and their use in an educational curriculum, and discuss the relevant findings.