Assessment of competence in local anaesthetic thoracoscopy: development and validity investigation of a new assessment tool

Background

The aims of the study were to develop an assessment tool in local anaesthetic thoracoscopy (LAT), investigate validity evidence, and establish a pass/fail standard.

Methods

Validity evidence for the assessment tool was gathered using the unified Messick framework. The tool was developed by five experts in respiratory medicine and medical education. Doctors with varying experience performed two consecutive procedures in a standardized, simulation-based setting using a newly developed thorax/lung silicone model. Performances were video-recorded and assessed by four expert raters using the new tool. Contrasting groups’ method was used to set a pass/fail standard.

Results

Nine novices and 8 experienced participants were included, generating 34 recorded performances and 136 expert assessments. The tool had a high internal consistency (Cronbach’s alpha =0.94) and high inter-rater reliability (Cronbach’s alpha =0.91). The total item score significantly correlated with the global score (r_s=0.86, P<0.001). Participants’ first performance correlated to second performance (test-retest reliability) with a Pearson’s r of 0.93, P<0.001. Generalisability (G) study showed a G-coefficient of 0.92 and decision (D) study estimated that one performance assessed by two raters or four performances assessed by one rater are needed to reach an acceptable reliability, i.e., G-coefficient >0.80. The tool was able to discriminate between the two groups in both performances: experienced mean score =30.8±4.2; novice mean score =15.8±2.3, P<0.001. Pass/fail standard was set at 22 points.

Conclusions

The newly developed assessment tool showed solid evidence of validity and can be used to ensure competence in LAT.

Structured and evidence-based training of technical skills in respiratory medicine and thoracic surgery

Within the last decade, the number of technical procedures in respiratory medicine and thoracic surgery has grown and created a higher demand for effective and evidence-based education. Today, trainees are often allowed to perform procedures unsupervised on patients after obtaining a course certificate from a theoretic course and having performed a fixed number of supervised procedures. However, these methods do not ensure adequate competence. Well-structured and effective educational programmes including validated tests are needed to reduce economic expenses, optimize time spend, and ensure patient safety. The aim of this article is to summarize current state of educational strategies for technical procedures within respiratory medicine and thoracic surgery. Thus, to discuss future recommendations for curriculum development and assessment of competences based on Kern’s framework. The approach by Kern consists of six topics, which needs to be considered and evaluated and in order to educate physicians and surgeons most effective and evidence-based. We present a practical guide contributing to future educators’ considerations on (I) problem identification and general needs assessment, (II) targeted needs assessment, (III) goals and objectives, (IV) educational strategies, (V) implementation, and finally (VI) evaluation and feedback.

Reliable test of clinicians' mastery in skin cancer diagnostics

Differentiating between benign and malignant skin lesions can be very difficult and should only be done by sufficiently trained and skilled clinicians. To our knowledge there are no validated tests for reliable assessments of clinicians' ability to perform skin cancer diagnostics. To develop and gather validity evidence for a test in skin cancer diagnostics, a multiple-choice questionnaire (MCQ) was developed based on informal interviews with seven content experts from five skin cancer centers in Denmark. Validity evidence for the test was gathered from May until July 2019 using Messick's validity framework (content, response process, internal structure, relationship to other variables and consequences). Item content was revised through a Delphi-like review process and then piloted on 36 medical students and 136 doctors using a standardized response process. Results enabled an analysis of the internal structure and relationship to other variables of the test. Finally, the contrasting groups method was used to investigate the test's consequences (pass-fail standard). The initial 90-item MCQ was reduced to 40 items during the Delphi-like review process. Item analysis revealed that 25 of the 40 selected items were level I-III quality items with a high internal consistency (Cronbach's α = 0.83) and highly significant (P ≤ 0.0001) differences in test scores between participants with different occupations or levels of experience. A pass-fail standard of 12 (48%) correct answers was established using the contrasting groups' method. The skin cancer diagnostics MCQ developed in this study can be used for reliable assessments of clinicians' competencies.

Development and validation of a multiple-choice questionnaire-based theoretical test in direct ophthalmoscopy

PURPOSE

Direct ophthalmoscopy can reveal systemic, neurologic and ophthalmic conditions, but is poorly mastered among young physicians. A theoretical test is needed to measure effect of educational interventions. We developed and gathered validity evidence for a multiple-choice questionnaire (MCQ)-based theoretical test in direct ophthalmoscopy.

METHODS

The MCQ was developed by interviewing experts. Then, validity evidence was evaluated using Messick's validity framework. Content was ensured by inviting the experts to contribute in a Delphi-like process. Response process was ensured by piloting and by streamlining all instructions. Then, the test was taken by ophthalmologists and by medical students without experience in direct ophthalmoscopy. Results were used to evaluate internal structure (item quality analysis and internal consistency), relations to other variables (correlation of test scores to experience level) and consequences (establishment of pass-fail score and the consequences of its use).

RESULTS

The first phase of the study yielded 100 MCQs. In second phase, we identified that 60 items fulfilled predefined relevance and item quality requirements. These items demonstrated very high internal consistency (Cronbach's alpha = 0.95), significantly discriminated medical students from specialists (p < 0.001, independent samples t-test) and the established pass-fail score of 50 (83%) correct answers resulted in no false positives (students passing) and no false negatives (specialists failing). A Decision study identified that sampling 15 items suffice for certification.

CONCLUSION

We developed and validated an MCQ-based theoretical test in direct ophthalmoscopy that enables an evidence-based approach to measuring, evaluating and certifying the theoretical knowledge necessary for direct ophthalmoscopy.

Development of and Gathering Validity Evidence for a Theoretical Test in Thoracic Ultrasound

BACKGROUND

Thoracic ultrasound (TUS) has a high diagnostic accuracy for many common pulmonary diseases, but theoretic knowledge in sonographic physics, thoracic anatomy and physiology, and sonopathologic patterns is required to develop competence.

OBJECTIVES

The aims of the study were to develop and gather validity evidence for a theoretical test in TUS and to establish a pass/fail standard.

METHODS

Content was provided based on expert interviews, leading to the creation of 113 initial multiple-choice question (MCQ) items. Consensus was reached on 92 proceeding items through a Delphi process, and items were presented to physicians with different knowledge and experience in TUS. Answers were used for item statistics in order to select the items with the most optimal item discrimination and difficulty (i.e., level I items) to be included in the final test. Mean scores of the novice, intermediate and experienced groups were compared, and a pass/fail score was established using the contrasting groups' standard setting method.

RESULTS

Item statistics revealed 38 level I items, of which 30 were selected to be included in the final test. The internal consistency was high (Cronbach's alpha = 0.88). Differences in mean scores were 8.6 points (p < 0.001), 6.3 points (p = 0.01), and 14.9 points (p < 0.001) between novices and intermediates, intermediates and experienced, and novices and experienced, respectively. A pass/fail standard of 20 points was established.

CONCLUSION

The established MCQ test can distinguish between physicians with different levels of competence in TUS and enables an objective, evidence-based approach for assessing the theoretical knowledge of trainees undergoing an educational programme in TUS.

Competence and training in interventional pulmonology

Interventional pulmonology (IP) is experiencing a rapid evolution of new technologies. There is a need to develop structured training programs, organized in high volume expert centers in order to improve trainee education, and including the development of validated metrics for their competency assessment. Concerning teaching methods, a gradual progression from theory to practice, using new teaching techniques, including live sessions and low and high-fidelity simulation, flipped classroom models and problem-based learning (PBL) exercises would provide a training setting more suitable for our current need to improve skills and update professionals. Training programs should be learner-centered and competence-oriented, as well as being based on a spiral-shaped approach in which the same subject is addressed many times, from new and different perspectives of knowledge, ability, behavior and attitude, until the trainee has demonstrated a high degree of skill and professionalism. Furthermore there is a need to standardize the training programs as guide for physicians wishing to undertake a gradual and voluntary improvement of their own competencies, and assist those planning and organizing training programs in IP. The article includes a general part on core curriculum contents, innovative training methods and simulation, and introduces the following articles on the skills that the Interventional Pulmonologist must master in order to perform the different procedures. This monography should be considered a starting point that will evolve over time and results in better training for practitioners and better care for our patients. The task of establishing a trainee's competence to practice independently as an Interventional Pulmonologist remains the responsibility of the IP fellowship program director and faculty, who validate logbooks and assess competence for each procedure. These standards need to be reviewed and approved by national and International Scientific Societies and Healthcare Institutions with the aim to improve, disseminate and incorporate them in healthcare programs.

EBUS-STAT Subscore Analysis to Predict the Efficacy and Assess the Validity of Virtual Reality Simulation for EBUS-TBNA Training Among Experienced Bronchoscopists

BACKGROUND

Linear endobronchial ultrasound transbronchial needle aspiration (EBUS-TBNA) represents a pivotal innovation in interventional pulmonology; determining the best approach to guarantee systematic and efficient training is expected to become a main issue in the forthcoming years. Virtual reality simulators have been proposed as potential EBUS-TBNA training instruments, to avoid unskilled beginners practicing directly in real-life settings. A validated and perfected simulation program could be used before allowing beginners to practice on patients. Our goal was to test the reliability of the EBUS-Skills and Task Assessment Tool (STAT) and its subscores for measuring the competence of experienced bronchoscopists approaching EBUS-guided TBNA, using only the virtual reality simulator as both a training and an assessment tool.

METHODS

Fifteen experienced bronchoscopists, with poor or no experience in EBUS-TBNA, participated in this study. They were all administered the Italian version of the EBUS-STAT evaluation tool, during a high-fidelity virtual reality simulation. This was followed by a single 7-hour theoretical and practical (on simulators) session on EBUS-TBNA, at the end of which their skills were reassessed by EBUS-STAT.

RESULTS

An overall, significant improvement in EBUS-TBNA skills was observed, thereby confirming that (a) virtual reality simulation can facilitate practical learning among practitioners, and (b) EBUS-STAT is capable of detecting these improvements. The test's overall ability to detect differences was negatively influenced by the minimal variation of the scores relating to items 1 and 2, was not influenced by the training, and improved significantly when the 2 items were not considered. Apart from these 2 items, all the remaining subscores were equally capable of revealing improvements in the learner. Lastly, we found that trainees with presimulation EBUS-STAT scores above 79 did not show any significant improvement after virtual reality training, suggesting that this score represents a cutoff value capable of predicting the likelihood that simulation can be beneficial.

CONCLUSIONS

Virtual reality simulation is capable of providing a practical learning tool for practitioners with previous experience in flexible bronchoscopy, and the EBUS-STAT questionnaire is capable of detecting these changes. A pretraining EBUS-STAT score below 79 is a good indicator of those candidates who will benefit from the simulation training. Further studies are needed to verify whether a modified version of the questionnaire would be capable of improving its performance among experienced bronchoscopists.

Training and proficiency in endobronchial ultrasound-guided transbronchial needle aspiration: A systematic review

Endobronchial ultrasound (EBUS)-guided transbronchial needle aspiration (TBNA) is currently the modality of choice for evaluation of mediastinal lymphadenopathy. However, there is still uncertainty regarding the training methodology and the number of procedures required to attain proficiency in EBUS. Herein, we performed a systematic review of studies selected from PubMed, EmBase and Scopus databases describing the training and assessment of proficiency during EBUS, specifically studies investigating various methods for training, its outcome and the number of procedures required to overcome the learning curve for EBUS. Twenty-seven (simulator-based learning (n = 8), tools for assessing competence in EBUS-TBNA (n = 5) and threshold numbers needed to attain proficiency in EBUS-TBNA (n = 16)) studies were identified. An EBUS simulator accurately stratified individuals based on the level of experience in performing EBUS. Training received on a simulator was comparable with traditional apprentice-based training. Importantly, skills acquired on a simulator could be transferred to real-world patients. The number needed to overcome the initial learning curve of EBUS varied from 10 to 100 in individual studies with a mean of 37-44 procedures. Tools such as EBUS-STAT (EBUS skill and task assessment tool) and EBUSAT (EBUS skill and assessment tool) were effective in evaluating the EBUS trainees. We conclude that an EBUS simulator or EBUS assessment tools can objectively assess the training of an EBUS trainee. Simulator-based training is a useful modality in EBUS training. The number of procedures needed to overcome the initial learning curve is about 40. Centres involved in EBUS training could incorporate simulator-based training in their curriculum before allowing operators to perform EBUS on patients.

Training and certification in endobronchial ultrasound-guided transbronchial needle aspiration

Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) plays a key role in the staging of lung cancer, which is crucial for allocation to surgical treatment. EBUS-TBNA is a complicated procedure and simulation-based training is helpful in the first part of the long learning curve prior to performing the procedure on actual patients. New trainees should follow a structured training programme consisting of training on simulators to proficiency as assessed with a validated test followed by supervised practice on patients. The simulation-based training is superior to the traditional apprenticeship model and is recommended in the newest guidelines. EBUS-TBNA and oesophageal ultrasound-guided fine needle aspiration (EUS-FNA or EUS-B-FNA) are complementary to each other and the combined techniques are superior to either technique alone. It is logical to learn and to perform the two techniques in combination, however, for lung cancer staging solely EBUS-TBNA simulators exist, but hopefully in the future simulation-based training in EUS will be possible.

Development and validation of a theoretical test in non-anaesthesiologist-administered propofol sedation for gastrointestinal endoscopy

OBJECTIVE

Safety with non-anaesthesiologist-administered propofol sedation (NAAP) during gastrointestinal (GI) endoscopy is related to theoretical knowledge. A summative testing of knowledge before attempting supervised nurse-administered propofol sedation (NAPS) in the clinic is advised. The aims of this study were to develop a theoretical test about propofol sedation, to gather validity evidence for the test and to measure the effect of a NAPS-specific training course.

MATERIAL AND METHODS

A three-phased psychometric study on multiple choice questionnaire (MCQ) test development, gathering of validity evidence and evaluation of the effect of a specific NAAP course on the test result. A MCQ containing 86 questions was developed and administered 113 times to 91 participants representing novices, intermediates and experienced.

RESULTS

Question difficulty analyses revealed 50 level I and II questions. The 50 MCQs showed mean (SD) intergroup differences (p = 0.001) between novices = 28.6 (4.82), intermediates = 36.8 (5.43) and experienced = 41.8 (4.65) and provided a pass score of 35.2. The course with pre-course test had significant effect on the knowledge of nurses (18% increase) and physicians (19% increase; p = 0.001 and 0.001, respectively).

CONCLUSIONS

Data supported the validity of the developed MCQ test. The NAPS-specific course with pre-course testing adds theoretical knowledge to already well-prepared participants.

Simulator training for endobronchial ultrasound: a randomised controlled trial

Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is very operator dependent and has a long learning curve. Simulation-based training might shorten the learning curve, and an assessment tool with solid validity evidence could ensure basic competency before unsupervised performance.A total of 16 respiratory physicians, without EBUS experience, were randomised to either virtual-reality simulator training or traditional apprenticeship training on patients, and then each physician performed EBUS-TBNA procedures on three patients. Three blinded, independent assessor assessed the video recordings of the procedures using a newly developed EBUS assessment tool (EBUSAT).The internal consistency was high (Cronbach's α=0.95); the generalisability coefficient was good (0.86), and the tool had discriminatory ability (p<0.001). Procedures performed by simulator-trained novices were rated higher than procedures performed by apprenticeship-trained novices: mean±sd are 24.2±7.9 points and 20.2±9.4 points, respectively; p=0.006. A pass/fail standard of 28.9 points was established using the contrasting groups method, resulting in 16 (67%) and 20 (83%) procedures performed by simulator-trained novices and apprenticeship-trained novices failing the test, respectively; p<0.001.The endobronchial ultrasound assessment tool could be used to provide reliable and valid assessment of competence in EBUS-TBNA, and act as an aid in certification. Virtual-reality simulator training was shown to be more effective than traditional apprenticeship training.

Assessment of competence in simulated flexible bronchoscopy using motion analysis

BACKGROUND

Flexible bronchoscopy should be performed with a correct posture and a straight scope to optimize bronchoscopy performance and at the same time minimize the risk of work-related injuries and endoscope damage.

OBJECTIVES

We aimed to test whether an automatic motion analysis system could be used to explore if there is a correlation in scope movements and the operator's level of experience. Our hypothesis was that experienced bronchoscopists move less and keep the flexible scope straighter than less-experienced bronchoscopists while performing procedures.

METHODS

Eleven novices, 9 intermediates and 9 experienced bronchoscopy operators performed 3 procedures each on a bronchoscopy simulator. The Microsoft Kinect system was used to automatically measure the total deviation of the scope from a perfectly straight, vertical line.

RESULTS

The low-cost motion analysis system could measure the accumulated deviation of the scope precisely during the procedure. The deviations were greatest for the novices and smallest for the most experienced operators for all 3 procedures (p = 0.01, p = 0.01 and p = 0.04, respectively). The total deviation from the straight scope correlated negatively with the performance on the simulator (virtual-reality simulator score; p < 0.001).

CONCLUSION

The motion analysis system could discriminate between different levels of experience. Automatic feedback on correct movements during self-directed training on simulators might help new bronchoscopists learn how to handle the bronchoscope like an expert.

Endobronchial ultrasound learning curve in interventional pulmonary fellows

BACKGROUND AND OBJECTIVE

Little published data exist regarding the learning curve for endobronchial ultrasound-transbronchial needle aspiration (EBUS-TBNA). We sought to assess the improvement in skill as trainees learned EBUS-TBNA in a clinical setting.

METHODS

This is a multicentre cohort study of EBUS-TBNA technical skill of interventional pulmonology (IP) fellows as assessed with EBUS-TBNA computer simulator testing every 25 clinical cases throughout IP fellowship training.

RESULTS

Nine fellows from three academic centres in the United States and Canada were enrolled in the study. Ongoing improvements were seen for EBUS-TBNA efficiency score and percentage of lymph nodes correctly identified on ultrasound exam, even after 200 clinical cases. Expert-level technical skill was obtained for EBUS efficiency score and for percentage of lymph nodes correctly identified on ultrasound exam at a median of 212 and 164 procedures, respectively; however, 33% of fellows did not achieve expert-level technical skill for either metric during their fellowship training. Significant variation in learning curves of the fellows was observed.

CONCLUSIONS

Significant variation is seen in the EBUS-TBNA learning curves of individual IP fellows and for individual procedure components, with ongoing improvement in EBUS-TBNA skill even after 200 clinical cases. These results highlight the need for validated, objective measures of individual competence, and can assist training programmes in ensuring adequate procedure volumes required for a majority of trainees to successfully complete these assessments.