Validation of educational assessments: a primer for simulation and beyond

Design and validation of a cross-specialty simulation-based training course in basic robotic surgical skills

BACKGROUND

The aim of this study was to design and validate a cross-specialty basic robotic surgical skills training program on the RobotiX Mentor virtual reality simulator.

METHODS

A Delphi panel reached consensus on six modules to include in the training program. Validity evidence was collected according to Messick's framework with three performances in each simulator module by 11 experienced robotic surgeons and 11 residents without robotic surgical experience.

RESULTS

For five of the six modules, a compound metrics-based score could significantly discriminate between the performances of novices and experienced robotic surgeons. Pass/fail levels were established, resulting in very few novices passing in their first attempt.

CONCLUSIONS

This validated course can be used for structured simulation-based basic robotic surgical skills training within a mastery learning framework where the individual trainee can practice each module until they achieve proficiency and can continue training on other modalities and more specific to their specialty.

Educational Perspectives: Educational Strategies to Improve Outcomes from Neonatal Resuscitation

Since 1987, the Neonatal Resuscitation Program (NRP) course has taught the cognitive, technical, and behavioral skills required to effectively resuscitate newborns. To remain relevant and effective, the NRP course needs to continually evolve and embrace evidence-based educational strategies proven to improve outcomes from resuscitation. In this Educational Perspectives article, 6 educational strategies that can be applied to neonatal resuscitation education are reviewed. These educational strategies include mastery learning and deliberate practice, spaced practice, contextual learning, feedback and debriefing, assessment, and innovative educational strategies Then knowledge translation and implementation of these educational strategies through passive and active knowledge translation, change theory, design thinking, performance measurement, deadoption strategies, continuous quality improvement, incentive and penalties, and psychological marketing are explored. Finally, ways to optimize faculty development of NRP instructors, including both initial instructor training and ongoing instructor development, are examined. The goal of this review is to help NRP program developers and instructors use evidence-based educational strategies to improve neonatal resuscitation outcomes.

A model to measure self-assessed proficiency in electronic medical records: Validation using maturity survey data from Canadian community-based physicians

OBJECTIVE

Adoption of electronic medical records (EMRs) does not necessarily translate to proficiency -referred to here as EMR maturity. To realize the full benefit of wide scale EMR adoption, the focus must shift from adoption to advancing mature use. This calls for validated assessment models so that researchers, health system planners and digital health developers can better understand what contributes to maturity among physicians. This research aims to validate a measurement model for self-assessed EMR maturity among community-based physicians.

METHODS

As part of an Ontario government-funded EMR adoption program, the EMR Maturity Model for community-based practices was adapted from a hospital-based EMR maturity model. A survey instrument was developed on the foundation of the new model and revised by experts and stakeholders. Content validity, face validity and user acceptance were established before survey administration. Internal consistency and construct validity of the model were tested after survey data were collected. Finally, physicians' comments collected via the survey were qualitatively analyzed to provide additional insights that can be applied to refinement of the model and survey.

RESULTS

As of August 1, 2019, 1588 physicians completed the survey. Ordinal alpha tests for reliability and content validity yielded an alpha value of 0.86 across all key measures specifically associated with maturity. Among most of these, there was a pattern of weak to moderate significant (p < .0001) positive Spearman inter-correlations. One factor was extracted for items measuring dimensions of maturity and all factor loadings of the key measures were greater than 0.40. The fit of the one-factor model was moderately adequate. This indicates the model is valid and reliable, with consistency across key measures for measuring one factor: maturity.

CONCLUSIONS

This is the first known validated model published in English that measures EMR maturity among community-based physicians. While the model is shown to be valid and reliable statistically and qualitative analysis supports this, there is room for improvement. Both the statistical analysis and portions of the qualitative analysis suggest areas of exploration to strengthen the model and survey. Future efforts will include refining the survey to improve user interface and accrue further data, as the sample to date is insufficient for generalizability.

Validation of Training and Acquisition of Surgical Skills in Veterinary Laparoscopic Surgery: A Review

At present, veterinary laparoscopic surgery training is lacking in experiences that provide a controlled and safe environment where surgeons can practice specific techniques while receiving experts' feedback. Surgical skills acquired using simulators must be certified and transferable to the operating room. Most models for practicing laparoscopic skills in veterinary minimally invasive surgery are general task trainers and consist of boxes (simulators) designed for training human surgery. These simulators exhibit several limitations, including anatomic species and procedural differences, as well as general psychomotor training rather than in vivo skill recreation. In this paper, we review the existing methods of training, evaluation, and validation of technical skills in veterinary laparoscopic surgery. Content includes global and specific scales, and the conditions a structured curriculum should meet for improving the performance of novice surgeons during and after training. A focus on trainee-specific assessment and tailored-technical instruction should influence training programs. We provide a comprehensive analysis of current theories and concepts related to the evaluation and validation of simulators for training laparoscopic surgery in small animal surgery. We also highlight the need to develop new training models and complementary evaluation scales for the validation of training and acquisition of basic and advanced skills in veterinary laparoscopic surgery.

Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training

INTRODUCTION/BACKGROUND

Owing to restrictions in operative experiences, urology residents can no longer solely rely on 'hands-on' operative time to master their surgical skills by the end of residency. Simulation training could help residents master basic surgical skills and steps of a procedure to maximize time in the operative room. However, simulators can be expensive or tedious to set up, limiting the availability to residents and training programs.

OBJECTIVE

The authors sought to develop and validate an inexpensive, high-fidelity training model for robotic pyeloplasty.

STUDY DESIGN

Pyeloplasty models were created using Dragon Skin® FX-Pro tissue-mimicking silicone cast over 3-dimensional molds. Urology faculty and trainees completed a demographic questionnaire. The participants viewed a brief instructional video and then independently performed robotic dismembered pyeloplasty on the model. Acceptability and content validity were evaluated via post-task evaluation of the model. Construct validity was evaluated by comparing procedure completion time, the Global Evaluative Assessment of Robotic Skills (GEARS) score, blinded subjective physical evaluation of repair quality (1-10 scale), and flow rate between experts and novices.

RESULTS

In total, 5 urology faculty, 6 fellows, and 14 residents participated. The median robotic console experience among faculty, fellows, and residents was 8 years (interquartile range [IQR] = 6-11), 3.5 years (IQR = 2-4 years), and 0 years (IQR = 0-0.5 years), respectively. The median procedure completion time was 29 min (IQR = 26-40 min), and the median flow rate was 1.11 mL/s (IQR = 0-1.34 mL/s). All faculty had flow rates >1.25 mL/s and procedure times <30 min compared with 2 of 6 fellows and none of the residents (P < 0.001). All faculty, half of the fellows, and none of the residents achieved a GEARS score ≥20, with a median resident score of 12.5 (IQR = 8-13) (P < 0.001). For repair quality, all faculty scored ≥9 (out of 10), all fellows scored ≥8, and the median score among residents was 6 (IQR = 2-6) (P < 0.001). The material cost was $1.32/model, and the average production time was 0.12 person-hours/model.

DISCUSSION AND CONCLUSION

This low-cost pyeloplasty model exhibits acceptability and content validity. Construct validity is supported by significant correlation between participant expertise and simulator performance across multiple assessment domains. The model has excellent potential to be used as a training tool in urology and allows for repetitive practice of pyeloplasty skills before live cases.

Provoking a Conversation Around Students' and Supervisors' Expectations Regarding Workplace Learning

Construct: This study presents a tool that can facilitate a conversation about students' and supervisors' expectations concerning responsibilities during workplace learning. Background: It is often unclear who is responsible for facilitating learning opportunities in the workplace. In order to increase learning opportunities, it is important that expectations are discussed and alignment is reached between the student's and supervisor's expectations. This study collected and interpreted validity evidence for a tool that aims to provoke such a conversation. Approach: Three types of validity evidence were collected: response process, content, and consequences evidence. Educational leaders, medical teachers, and students of four medical schools were involved. The data collection consisted of cognitive interviews, a modified Delphi approach (with three rounds of inquiry), completed tools, and narrative comments. Findings: This study showed that the expectations of most students and supervisors were not initially aligned. The conversation, for which the tool aims to be a catalyst, facilitated better alignment of expectations about responsibilities during workplace learning. Moreover, the students' perceived degree of consensus and satisfaction after the conversation were very high. Conclusions: This study underlined the relevance and usefulness of a tool that facilitates conversation about expectations regarding responsibilities, potentially enhancing learning opportunities at the workplace.

Assessing validity evidence for a serious game dedicated to patient clinical deterioration and communication

Background

A serious game (SG) is a useful tool for nurse training. The objectives of this study were to assess validity evidence of a new SG designed to improve nurses’ ability to detect patient clinical deterioration.

Methods

The SG (LabForGames Warning) was developed through interaction between clinical and pedagogical experts and one developer. For the game study, consenting nurses were divided into three groups: nursing students (pre-graduate) (group S), recently graduated nurses (graduated < 2 years before the study) (group R) and expert nurses (graduated > 4 years before the study and working in an ICU) (group E). Each volunteer played three cases of the game (haemorrhage, brain trauma and obstructed intestinal tract). The validity evidence was assessed following Messick’s framework: content, response process (questionnaire, observational analysis), internal structure, relations to other variables (by scoring each case and measuring playing time) and consequences (a posteriori analysis).

Results

The content validity was supported by the game design produced by clinical, pedagogical and interprofessional experts in accordance with the French nurse training curriculum, literature review and pilot testing. Seventy-one nurses participated in the study: S (n = 25), R (n = 25) and E (n = 21). The content validity in all three cases was highly valued by group E. The response process evidence was supported by good security control. There was no significant difference in the three groups’ high rating of the game’s realism, satisfaction and educational value. All participants stated that their knowledge of the different steps of the clinical reasoning process had improved. Regarding the internal structure, the factor analysis showed a common source of variance between the steps of the clinical reasoning process and communication or the situational awareness errors made predominantly by students. No statistical difference was observed between groups regarding scores and playing time. A posteriori analysis of the results of final examinations assessing study-related topics found no significant difference between group S participants and students who did not participate in the study.

Conclusion

While it appears that this SG cannot be used for summative assessment (score validity undemonstrated), it is positively valued as an educational tool.

Trial registration

ClinicalTrials.gov ID: NCT03092440

Construct Validity of a Serious Game for Laparoscopic Skills Training: Validation Study

Background

Surgical residents underutilize opportunities for traditional laparoscopic simulation training. Serious gaming may increase residents’ motivation to practice laparoscopic skills. However, little is known about the effectiveness of serious gaming for laparoscopic skills training.

Objective

The aim of this study was to establish construct validity for the laparoscopic serious game Underground.

Methods

All study participants completed 2 levels of Underground. Performance for 2 novel variables (time and error) was compared between novices (n=65, prior experience <10 laparoscopic procedures), intermediates (n=26, prior experience 10-100 laparoscopic procedures), and experts (n=20, prior experience >100 laparoscopic procedures) using analysis of covariance. We corrected for gender and video game experience.

Results

Controlling for gender and video game experience, the effects of prior laparoscopic experience on the time variable differed significantly (F_2,106=4.77, P=.01). Both experts and intermediates outperformed novices in terms of task completion speed; experts did not outperform intermediates. A similar trend was seen for the rate of gameplay errors. Both gender (F_1,106=14.42, P<.001 in favor of men) and prior video game experience (F_1,106=5.20, P=.03 in favor of experienced gamers) modulated the time variable.

Conclusions

We established construct validity for the laparoscopic serious game Underground. Serious gaming may aid laparoscopic skills development. Previous gaming experience and gender also influenced Underground performance. The in-game performance metrics were not suitable for statistical evaluation. To unlock the full potential of serious gaming for training, a more formal approach to performance metric development is needed.

Effect of monitor positioning on visual attention and situation awareness during neonatal resuscitation: a randomised simulation study

OBJECTIVES

To compare situation awareness (SA), visual attention (VA) and protocol adherence in simulated neonatal resuscitations using two different monitor positions.

DESIGN

Randomised controlled simulation study.

SETTINGS

Simulation lab at the Royal Alexandra Hospital, Edmonton, Canada.

PARTICIPANTS

Healthcare providers (HCPs) with Neonatal Resuscitation Program (NRP) certification within the last 2 years and trained in neonatal endotracheal intubations.

INTERVENTION

HCPs were randomised to either central (eye-level on the radiant warmer) or peripheral (above eye-level, wall-mounted) monitor positions. Each led a complex resuscitation with a high-fidelity mannequin and a standardised assistant. To measure SA, situation awareness global assessment tool (SAGAT) was used, where simulations were paused at three predetermined points, with five questions asked each pause. Videos were analysed for SAGAT and adherence to a NRP checklist. Eye-tracking glasses recorded participants' VA.

MAIN OUTCOME MEASURE

The main outcome was SA as measured by composite SAGAT score. Secondary outcomes included VA and adherence to NRP checklist.

RESULTS

Thirty simulations were performed; 29 were completed per protocol and analysed. Twenty-two eye-tracking recordings were of sufficient quality and analysed. Median composite SAGAT was 11.5/15 central versus 11/15 peripheral, p=0.56. Checklist scores 46/50 central versus 46/50 peripheral, p=0.75. Most VA was directed at the mannequin (30.6% central vs 34.1% peripheral, p=0.76), and the monitor (28.7% central vs 20.5% peripheral, p=0.06).

CONCLUSIONS

Simulation, SAGAT and eye-tracking can be used to evaluate human factors of neonatal resuscitation. During simulated neonatal resuscitation, monitor position did not affect SA, VA or protocol adherence.

The future is here! Pediatric surgery and the move to the royal college of physicians and surgeons of Canada's competence by design

This interactive session was held at the 51st Annual Meeting of the Canadian Association of Pediatric Surgeons (CAPS) in preparation for the transition of Pediatric Surgery training in Canada to Competency by Design (a CBME-based model of residency training developed by the Royal College of Physicians and Surgeons of Canada).

Validation of an evaluation instrument for responders in tactical casualty care simulations

OBJECTIVE

to construct and validate a tool for the evaluation of responders in tactical casualty care simulations.

METHOD

three rubrics for the application of a tourniquet, an emergency bandage and haemostatic agents recommended by the Hartford Consensus were developed and validated. Validity and reliability were studied. Validation was performed by 4 experts in the field and 36 nursing participants who were selected through convenience sampling. Three rubrics with 8 items were evaluated (except for the application of an emergency bandage, for which 7 items were evaluated). Each simulation was evaluated by 3 experts.

RESULTS

an excellent score was obtained for the correlation index for the 3 simulations and 2 levels that were evaluated (competent and expert). The mean score for the application of a tourniquet was 0.897, the mean score for the application of an emergency bandage was 0.982, and the mean score for the application of topical haemostats was 0.805.

CONCLUSION

this instrument for the evaluation of nurses in tactical casualty care simulations is considered useful, valid and reliable for training in a prehospital setting for both professionals who lack experience in tactical casualty care and those who are considered to be experts.

Roadmap for Developing Complex Virtual Reality Simulation Scenarios: Subpial Neurosurgical Tumor Resection Model

BACKGROUND

Advancement and evolution of current virtual reality (VR) surgical simulation technologies are integral to improve the available armamentarium of surgical skill education. This is especially important in high-risk surgical specialties. Such fields including neurosurgery are beginning to explore the utilization of virtual reality simulation in the assessment and training of psychomotor skills. An important issue facing the available VR simulation technologies is the lack of complexity of scenarios that fail to replicate the visual and haptic realities of complex neurosurgical procedures. Therefore there is a need to create more realistic and complex scenarios with the appropriate visual and haptic realities to maximize the potential of virtual reality technology.

METHODS

We outline a roadmap for creating complex virtual reality neurosurgical simulation scenarios using a step-wise description of our team's subpial tumor resection project as a model.

RESULTS

The creation of complex neurosurgical simulations involves integrating multiple modules into a scenario-building roadmap. The components of each module are described outlining the important stages in the process of complex VR simulation creation.

CONCLUSIONS

Our roadmap of a stepwise approach for the creation of complex VR-simulated neurosurgical procedures may also serve as a guide to aid the development of other VR scenarios in a variety of surgical fields. The generation of new VR complex simulated neurosurgical procedures, by surgeons for surgeons, with the help of computer scientists and engineers may improve the assessment and training of residents and ultimately improve patient care.

The Variable Journey in Learning to Interpret Pediatric Point-of-care Ultrasound Images: A Multicenter Prospective Cohort Study

OBJECTIVES

To complement bedside learning of point-of-care ultrasound (POCUS), we developed an online learning assessment platform for the visual interpretation component of this skill. This study examined the amount and rate of skill acquisition in POCUS image interpretation in a cohort of pediatric emergency medicine (PEM) physician learners.

METHODS

This was a multicenter prospective cohort study. PEM physicians learned POCUS using a computer-based image repository and learning assessment system that allowed participants to deliberately practice image interpretation of 400 images from four pediatric POCUS applications (soft tissue, lung, cardiac, and focused assessment sonography for trauma [FAST]). Participants completed at least one application (100 cases) over a 4-week period.

RESULTS

We enrolled 172 PEM physicians (114 attendings, 65 fellows). The increase in accuracy from the initial to final 25 cases was 11.6%, 9.8%, 7.4%, and 8.6% for soft tissue, lung, cardiac, and FAST, respectively. For all applications, the average learners (50th percentile) required 0 to 45, 25 to 97, 66 to 175, and 141 to 290 cases to reach 80, 85, 90, and 95% accuracy, respectively. The least efficient (95th percentile) learners required 60 to 288, 109 to 456, 160 to 666, and 243 to 1040 cases to reach these same accuracy benchmarks. Generally, the soft tissue application required participants to complete the least number of cases to reach a given proficiency level, while the cardiac application required the most.

CONCLUSIONS

Deliberate practice of pediatric POCUS image cases using an online learning and assessment platform may lead to skill improvement in POCUS image interpretation. Importantly, there was a highly variable rate of achievement across learners and applications. These data inform our understanding of POCUS image interpretation skill development and could complement bedside learning and performance assessments.

Characterizing the Development of Colonoscopy Competence Using Assessment Data

Assessment of endoscopist competence is an increasingly important component of colonoscopy quality assurance. In this study from the Joint Advisory Group on Gastrointestinal Endoscopy, validity evidence is provided for the use of the Direct Observation of Procedural Skills assessment tool in the formative setting during training. In this national UK dataset, overall colonoscopy competence was typically achieved after 200-249 procedures, although certain complex procedural skills ("proactive problem solving" and "loop management") had not reached the threshold for competence even after 300 procedures. These data will help inform the development and/or refinement of certification policies and practices in jurisdictions around the world.

Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training

INTRODUCTION

Formative colonoscopy direct observation of procedural skills (DOPS) assessments were updated in 2016 and incorporated into UK training but lack validity evidence. We aimed to appraise the validity of DOPS assessments, benchmark performance, and evaluate competency development during training in diagnostic colonoscopy.

METHODS

This prospective national study identified colonoscopy DOPS submitted over an 18-month period to the UK training e-portfolio. Generalizability analyses were conducted to evaluate internal structure validity and reliability. Benchmarking was performed using receiver operator characteristic analyses. Learning curves for DOPS items and domains were studied, and multivariable analyses were performed to identify predictors of DOPS competency.

RESULTS

Across 279 training units, 10,749 DOPS submitted for 1,199 trainees were analyzed. The acceptable reliability threshold (G > 0.70) was achieved with 3 assessors performing 2 DOPS each. DOPS competency rates correlated with the unassisted caecal intubation rate (rho 0.404, P < 0.001). Demonstrating competency in 90% of assessed items provided optimal sensitivity (90.2%) and specificity (87.2%) for benchmarking overall DOPS competence. This threshold was attained in the following order: "preprocedure" (50-99 procedures), "endoscopic nontechnical skills" and "postprocedure" (150-199), "management" (200-249), and "procedure" (250-299) domain. At item level, competency in "proactive problem solving" (rho 0.787) and "loop management" (rho 0.780) correlated strongest with the overall DOPS rating (P < 0.001) and was the last to develop. Lifetime procedure count, DOPS count, trainer specialty, easier case difficulty, and higher cecal intubation rate were significant multivariable predictors of DOPS competence.

DISCUSSION

This study establishes milestones for competency acquisition during colonoscopy training and provides novel validity and reliability evidence to support colonoscopy DOPS as a competency assessment tool.

Validation of a cost-effective appendicectomy model for surgical training

BACKGROUND

Appendicitis is a commonly occurring condition worldwide. The gold standard treatment is appendicectomy. Although training models are commercially available for this procedure, they are often associated with high cost. Here we present a cost-effective model.

AIM

To establish construct validity of a cost-effective laparoscopic appendicectomy simulation model.

METHODS

Three groups of surgeons were recruited; novices (n = 31), of intermediate expertise (n = 13) and experts (n = 5) and asked to perform a simulated laparoscopic appendicectomy using the new model. Their performance was assessed by a faculty member and compared between the three groups using a validated scoring system (Global Operative Assessment of Laparoscopic Skills [GOALS] score).

RESULTS

One-way ANOVA test showed a significant difference in task performance between groups (p < 0.0001). Post-hoc comparisons after the application of Bonferroni correction (statistically significant p value <0.017) demonstrate a significant difference in performance between all groups for all GOALS categories as well as the total score. Effect size calculations showed that experience level had moderate (Eta-squared >0.5 and <0.8) and significant (>0.8) impact on the performance of the simulated procedure.

CONCLUSION

The model described in this study is cost-effective, valid and can adequately simulate appendicectomy. The authors recommend inclusion of this model to postgraduate surgical training.

What Is New in Validation and Simulation Training?: Best Articles From the Past Year

This month we focus on current research in validation and simulation training. Dr. Buek discusses four recent publications, which are concluded with a "bottom line" that is the take-home message. A complete reference for each can be found on on this page along with direct links to the abstracts.

Resuscitation Education Science: Educational Strategies to Improve Outcomes From Cardiac Arrest: A Scientific Statement From the American Heart Association

The formula for survival in resuscitation describes educational efficiency and local implementation as key determinants in survival after cardiac arrest. Current educational offerings in the form of standardized online and face-to-face courses are falling short, with providers demonstrating a decay of skills over time. This translates to suboptimal clinical care and poor survival outcomes from cardiac arrest. In many institutions, guidelines taught in courses are not thoughtfully implemented in the clinical environment. A current synthesis of the evidence supporting best educational and knowledge translation strategies in resuscitation is lacking. In this American Heart Association scientific statement, we provide a review of the literature describing key elements of educational efficiency and local implementation, including mastery learning and deliberate practice, spaced practice, contextual learning, feedback and debriefing, assessment, innovative educational strategies, faculty development, and knowledge translation and implementation. For each topic, we provide suggestions for improving provider performance that may ultimately optimize patient outcomes from cardiac arrest.

Paediatric Colonoscopy Direct Observation of Procedural Skills: Evidence of Validity and Competency Development

INTRODUCTION

The paediatric series of direct observation of procedural skills (DOPS) were introduced into the UK national endoscopy training curriculum in 2016, but lack validity evidence. We aimed to present validity evidence for paediatric colonoscopy DOPS and study competency development in a national trainee cohort.

METHODS

This prospective UK-wide study analysed formative paediatric colonoscopy DOPS which were submitted to the e-Portfolio between 2016 and 2018. Item, domain, and average DOPS scores were correlated with the overall DOPS rating to evidence internal structure validity. Overall DOPS ratings were compared over lifetime procedure count to demonstrate learning curves (discriminant validity). Consequential validity was founded on receiver operating characteristic curve analyses.

RESULTS

A total of 203 DOPS assessments were completed for 29 trainees from 11 UK training centres. Internal structure validity was provided through item-total correlation analyses. DOPS scores positively correlated with trainee seniority (P < 0.001) and lifetime procedure count (P < 0.001). Competency acquisition followed the order of: "preprocedure," "postprocedure," "endoscopic nontechnical skills," "management," "procedure" domains, followed by overall DOPS competency, which was achieved in 81% of the cohort after 125 to 149 procedures. Mean DOPS scores could be used to predict overall procedure competence (area under receiver operating characteristic curve 0.969, P < 0.001), with a mean score of 3.9 demonstrating optimal sensitivity (93.5%) and specificity (87.6%).

CONCLUSIONS

This study provides validity evidence supporting the use of paediatric colonoscopy DOPS as an in-training competence assessment tool. DOPS may also be used to measure competency development and benchmark performance during training, which may be of value to trainees, trainers, and training programmes.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

Background

Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.

Methods

This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.

Results

In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.

Conclusion

This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

Electronic supplementary material

The online version of this article (10.1007/s00464-019-06737-7) contains supplementary material, which is available to authorised users.

Assessing noncognitive domains of respiratory therapy applicants: Messick's framework appraisal of the multiple mini-interview

Educators who assess incoming applicants into a health professional training program are looking for a wide array of cognitive and noncognitive skills that best predict success in the program and as a future practicing professional. While aptitude tests generally measure cognitive skills, noncognitive constructs are more difficult to measure appropriately. The traditional method of measuring noncognitive constructs has been the panel interview. Panel interviews have been described as inconsistent in measuring noncognitive domains and consistently reported as unreliable and susceptible to bias. An alternate interview method used in many health professions schools is the multiple mini-interview (MMI) that was specifically designed to assess noncognitive domains in health professions education. This paper discusses the purpose of using the MMI, how the MMI is conducted, specific domains of focus for the MMI, and the feasibility of creating an MMI. Finally, the paper uses Messick's framework on validity to guide the consideration of the MMI.

Ensuring Basic Competence in Thoracentesis

BACKGROUND

Trocar pigtail catheter thoracentesis (TPCT) is a common procedure often performed by junior physicians. Simulation-based training may effectively train physicians in the procedure prior to performing it on patients. An assessment tool with solid validity evidence is necessary to ensure sufficient procedural competence.

OBJECTIVES

Our study objectives were (1) to collect evidence of validity for a newly developed pigtail catheter assessment tool (Thoracentesis Assessment Tool [ThorAT]) developed for the evaluation of TPCT performance and (2) to establish a pass/fail score for summative assessment.

METHODS

We assessed the validity evidence for the ThorAT using the recommended framework for validity by Messick. Thirty-four participants completed two consecutive procedures and their performance was assessed by two blinded, independent raters using the ThorAT. We compared performance scores to test whether the assessment tool was able to discern between the two groups, and a pass/fail score was established.

RESULTS

The assessment tool was able to discriminate between the two groups in terms of competence level. Experienced physicians received significantly higher test scores than novices in both the first and second procedure. A pass/fail score of 25.2 points was established, resulting in 4 (17%) passing novices and 1 (9%) failing experienced participant in the first procedure. In the second procedure 9 (39%) novices passed and 2 (18%) experienced participants failed.

CONCLUSIONS

This study provides a tool for summative assessment of competence in TPCT. Strong validity evidence was gathered from five sources of evidence. A simulation-based training program using the ThorAT could ensure competence before performing thoracentesis on patients.

Evaluating the validity evidence of an OSCE: results from a new medical school

BACKGROUND

To prevent the problems of traditional clinical evaluation, the "Objective Structured Clinical Examination (OSCE)" was presented by Harden as a more valid and reliable assessment instrument. However, an essential condition to guarantee a high-quality and effective OSCE is the assurance of evidence to support the validity of its scores. This study examines the psychometric properties of OSCE scores, with an emphasis on consequential and internal structure validity evidence.

METHODS

Fifty-three first year medical students took part in a summative OSCE at the Lebanese American University-School of Medicine. Evidence to support consequential validity was gathered by using criterion-based standard setting methods. Internal structure validity evidence was gathered by examining various psychometric measures both at the station level and across the complete OSCE.

RESULTS

Compared to our actual method of computing results, the introduction of standard setting resulted in lower students' average grades and a higher cut score. Across stations, Cronbach's alpha was moderately low.

CONCLUSION

Gathering consequential and internal structure validity evidence by multiple metrics provides support for or against the quality of an OSCE. It is critical that this analysis be performed routinely on local iterations of given tests, and the results used to enhance the quality of assessment.

Validity Evidence for Direct Observation of Procedural Skills in Paediatric Gastroscopy

OBJECTIVES

Direct observation of procedural skills (DOPS) are competence-assessment tools in endoscopy. Formative paediatric gastroscopy DOPS were implemented into the UK curriculum in 2016 but lack validity evidence; we aimed to assess validity evidence using a recognised contemporary validity framework.

METHODS

We performed a prospective UK-wide analysis of formative paediatric gastroscopy DOPS submitted to the e-Portfolio over 1 year. Internal structure validity was assessed using interitem correlations between DOPS items, average domain, and skillset scores and with the overall competency rating. Overall competence scores and mean DOPS scores were compared by trainee seniority and procedure count (discriminative validity). Receiver operating characteristic curve analysis was performed to explore if DOPS scores could be used to delineate procedural competency (consequential validity).

RESULTS

A total of 157 DOPS assessments were completed by 20 trainers for 17 trainees. Strengths of correlations varied between DOPS components, with overall competency correlating most with technical-predominant items, domains and skillsets. Both the overall assessor's rating and mean DOPS scores increased with trainee seniority (P < 0.001) and lifetime procedure count (P < 0.001). Overall competency could be delineated using mean DOPS scores (area under receiver operating characteristic curve 0.95, P < 0.001), with a threshold of 3.9 providing optimal sensitivity (94.4%) and specificity (89.7%).

CONCLUSIONS

Competencies in paediatric gastroscopy, as assessed using DOPS, vary in their correlation with overall competence and increase with trainee experience. Formative DOPS thresholds could be used to indicate readiness for summative assessment. Our study therefore provides evidence of internal structure, discriminative, and consequential validity in support of formative paediatric gastroscopy DOPS.

Effectiveness of discovery learning using a mobile otoscopy simulator on knowledge acquisition and retention in medical students: a randomized controlled trial

Background

Portable educational technologies, like simulators, afford students the opportunity to learn independently. A key question in education, is how to pair self-regulated learning (SRL) with direct instruction. A cloud-based portable otoscopy simulator was employed to compare two curricula involving SRL. Pre-clerkship medical students used a prototype smartphone application, a 3D ear attachment and an otoscope to complete either otoscopy curriculum.

Methods

Pre-clerkship medical students were recruited and randomized to two curriculum designs. The “Discovery then Instruction” group received the simulator one week before a traditional lecture, while the “Instruction then Discovery” group received it after the lecture. To assess participants’ ability to identify otoscopic pathology, we used a 100-item test at baseline, post-intervention and 2-week retention time points. Secondary outcomes included self-reported comfort, time spent using the device, and a survey on learning preferences.

Results

Thirty-four students completed the study. Analysis of knowledge acquisition and retention showed improvement in scores of both groups and no significant effects of group (F_1,31 = 0.53, p = 0.47). An analysis of participants’ self-reported comfort showed a significant group x test interaction (F_1,36 = 4.61, p = 0.04), where only the discovery then instruction group’s comfort improved significantly. Overall device usage was low, as the discovery then instruction group spent 21.47 ± 26.28 min, while the instruction then discovery group spent 13.84 ± 18.71 min. The discovery first group’s time spent with the simulator correlated moderately with their post-test score (r = 0.42, p = 0.07). After the intervention, most participants in both groups (63–68%) stated that they would prefer the instruction then discovery sequence.

Conclusions

Both curricular sequences led to improved knowledge scores with no statistically significant knowledge differences. When given minimal guidance, students engaged in discovery learning minimally. There is value in SRL in simulation education, and we plan to further improve our curricular design by considering learner behaviours identified in this study.

The Development and Validation of a Concise Instrument for Formative Assessment of Team Leader Performance During Simulated Pediatric Resuscitations

AIM

The aim of this study was to assess the validity of a formative feedback instrument for leaders of simulated resuscitations.

METHODS

This is a prospective validation study with a fully crossed (person × scenario × rater) study design. The Concise Assessment of Leader Management (CALM) instrument was designed by pediatric emergency medicine and graduate medical education experts to be used off the shelf to evaluate and provide formative feedback to resuscitation leaders. Four experts reviewed 16 videos of in situ simulated pediatric resuscitations and scored resuscitation leader performance using the CALM instrument. The videos consisted of 4 pediatric emergency department resuscitation teams each performing in 4 pediatric resuscitation scenarios (cardiac arrest, respiratory arrest, seizure, and sepsis). We report on content and internal structure (reliability) validity of the CALM instrument.

RESULTS

Content validity was supported by the instrument development process that involved professional experience, expert consensus, focused literature review, and pilot testing. Internal structure validity (reliability) was supported by the generalizability analysis. The main component that contributed to score variability was the person (33%), meaning that individual leaders performed differently. The rater component had almost zero (0%) contribution to variance, which implies that raters were in agreement and argues for high interrater reliability.

CONCLUSIONS

These results provide initial evidence to support the validity of the CALM instrument as a reliable assessment instrument that can facilitate formative feedback to leaders of pediatric simulated resuscitations.

Redeveloping a workplace-based assessment program for physicians using Kane's validity framework

This paper describes the use of Kane's validity framework to redevelop a workplace-based assessment program for practicing physicians administered by the College of Physicians and Surgeons of Ontario. The developmental process is presented according to the four inferences in Kane's model. Scoring was addressed through the creation of specialty-specific assessment criteria and global, narrative-focused reports. Generalization was addressed through standardized sampling protocols and assessor training and consensus-building. Extrapolation was addressed through the use of real-world performance data and an external review of the scoring tools by practicing physicians. Implications were theoretically supported through adherence to formative assessment principles and will be assessed through an evaluation accompanying the implementation of the redeveloped program. Kane's framework was valuable for guiding the redevelopment process and for systematically collecting validity evidence throughout to support the use of the assessment for its intended purpose. As the use of workplace-based assessment programs for physicians continues to increase, practical examples are needed of how to develop and evaluate these programs using established frameworks. The dissemination of comprehensive validity arguments is vital for sharing knowledge about the development and evaluation of WBA programs and for understanding the effects of these assessments on physician practice improvement.

Student learning outcomes, perceptions and beliefs in the context of strengthening research integration into the first year of medical school

Research integrated into undergraduate education is important in order for medical students to understand and value research for later clinical practice. Therefore, attempts are being made to strengthen the integration of research into teaching from the first year onwards. First-year students may interpret attempts made to strengthen research integration differently than intended by teachers. This might be explained by student beliefs about learning and research as well as student perceptions of the learning environment. In general, student perceptions of the learning environment play a pivotal role in fostering student learning outcomes. This study aims to determine whether a curriculum change intended to promote research integration fosters student learning outcomes and student perceptions of research integrated into teaching. To serve this purpose, three subsequent cohorts of first-year students were compared, one before and two after a curriculum change. Learning outcomes of these students were measured using scores on a national progress test of 921 students and assessments of a sample of 100 research reports of a first-year student research project. 746 Students filled out the Student Perceptions of Research Integration Questionnaire. The findings suggest that learning outcomes of these students, that is, scores on research related test items of the progress test and the quality of research reports, were better than those of students before the curriculum change.

Simulation-based Education to Ensure Provider Competency Within the Health Care System

The acquisition and maintenance of individual competency is a critical component of effective emergency care systems. This article summarizes consensus working group deliberations and recommendations focusing on the topic "Simulation-based education to ensure provider competency within the healthcare system." The authors presented this work for discussion and feedback at the 2017 Academic Emergency Medicine Consensus Conference on "Catalyzing System Change Through Healthcare Simulation: Systems, Competency, and Outcomes," held on May 16, 2017, in Orlando, Florida. Although simulation-based training is a quality and safety imperative in other high-reliability professions such as aviation, nuclear power, and the military, health care professions still lag behind in applying simulation more broadly. This is likely a result of a number of factors, including cost, assessment challenges, and resistance to change. This consensus subgroup focused on identifying current gaps in knowledge and process related to the use of simulation for developing, enhancing, and maintaining individual provider competency. The resulting product is a research agenda informed by expert consensus and literature review.

Simulation for Assessment of Milestones in Emergency Medicine Residents

OBJECTIVES

All residency programs in the United States are required to report their residents' progress on the milestones to the Accreditation Council for Graduate Medical Education (ACGME) biannually. Since the development and institution of this competency-based assessment framework, residency programs have been attempting to ascertain the best ways to assess resident performance on these metrics. Simulation was recommended by the ACGME as one method of assessment for many of the milestone subcompetencies. We developed three simulation scenarios with scenario-specific milestone-based assessment tools. We aimed to gather validity evidence for this tool.

METHODS

We conducted a prospective observational study to investigate the validity evidence for three mannequin-based simulation scenarios for assessing individual residents on emergency medicine (EM) milestones. The subcompetencies (i.e., patient care [PC]1, PC2, PC3) included were identified via a modified Delphi technique using a group of experienced EM simulationists. The scenario-specific checklist (CL) items were designed based on the individual milestone items within each EM subcompetency chosen for assessment and reviewed by experienced EM simulationists. Two independent live raters who were EM faculty at the respective study sites scored each scenario following brief rater training. The inter-rater reliability (IRR) of the assessment tool was determined by measuring intraclass correlation coefficient (ICC) for the sum of the CL items as well as the global rating scales (GRSs) for each scenario. Comparing GRS and CL scores between various postgraduate year (PGY) levels was performed with analysis of variance.

RESULTS

Eight subcompetencies were chosen to assess with three simulation cases, using 118 subjects. Evidence of test content, internal structure, response process, and relations with other variables were found. The ICCs for the sum of the CL items and the GRSs were >0.8 for all cases, with one exception (clinical management GRS = 0.74 in sepsis case). The sum of CL items and GRSs (p < 0.05) discriminated between PGY levels on all cases. However, when the specific CL items were mapped back to milestones in various proficiency levels, the milestones in the higher proficiency levels (level 3 [L3] and 4 [L4]) did not often discriminate between various PGY levels. L3 milestone items discriminated between PGY levels on five of 12 occasions they were assessed, and L4 items discriminated only two of 12 times they were assessed.

CONCLUSION

Three simulation cases with scenario-specific assessment tools allowed evaluation of EM residents on proficiency L1 to L4 within eight of the EM milestone subcompetencies. Evidence of test content, internal structure, response process, and relations with other variables were found. Good to excellent IRR and the ability to discriminate between various PGY levels was found for both the sum of CL items and the GRSs. However, there was a lack of a positive relationship between advancing PGY level and the completion of higher-level milestone items (L3 and L4).

Using Virtual Reality Simulation Environments to Assess Competence for Emergency Medicine Learners

Immersive learning environments that use virtual simulation (VS) technology are increasingly relevant as medical learners train in an environment of restricted clinical training hours and a heightened focus on patient safety. We conducted a consensus process with a breakout group of the 2017 Academic Emergency Medicine Consensus Conference "Catalyzing System Change Through Health Care Simulation: Systems, Competency, and Outcomes." This group examined the current uses of VS in training and assessment, including limitations and challenges in implementing VS into medical education curricula. We discuss the role of virtual environments in formative and summative assessment. Finally, we offer recommended areas of focus for future research examining VS technology for assessment, including high-stakes assessment in medical education. Specifically, we discuss needs for determination of areas of focus for VS training and assessment, development and exploration of virtual platforms, automated feedback within such platforms, and evaluation of effectiveness and validity of VS education.

Development and validation of a written credentialing examination for overseas-educated dietitians

AIM

Health professionals seeking employment in foreign countries are commonly required to undertake competency assessment in order to practice. The present study aims to outline the development and validation of a written examination for Dietetic Skills Recognition (DSR), to assess the knowledge, skills, capabilities and professional judgement of overseas-educated dietitians against the competency standards applied to dietetic graduates in Australia.

METHODS

The present study reviews the design, rationale, validation and outcomes of a multiple choice question (MCQ) written examination for overseas-educated dietitians based on 5 years of administration. The validity of the exam is evaluated using Messick's validity framework, which focuses on five potential sources of validity evidence-content, internal structure, relationships with other variables, response process and consequences. The reference point for the exam pass mark or "cutscore" is the minimum standard required for safe practice.

RESULTS

In total, 114 candidates have completed the MCQ examination at least once, with an overall pass rate of 52% on the first attempt. Pass rates are higher from countries where dietetic education more closely reflects the Australian model. While the pass rate for each exam tends to vary with each cohort, the cutscore has remained relatively stable over eight administrations.

CONCLUSIONS

The findings provide important data supporting the validity of the MCQ exam. A more complete evaluation of the validity of the exam must be sought within the context of the whole DSR program of assessment. The DSR written component may serve as a model for use of the MCQ format for dietetic and other professional credentialing organisations.

Focused Evidence-Based Medicine Curriculum for Trainees in Neonatal-Perinatal Medicine

INTRODUCTION

While evidence-based medicine (EBM) is an Accreditation Council for Graduate Medical Education core competency, EBM teaching in pediatric subspecialties is rarely reported. Therefore, we designed, implemented, and evaluated this focused EBM curriculum for trainees in neonatal-perinatal medicine.

METHODS

This EBM curriculum consists of seven weekly 1-hour sessions. Specific EBM skills taught in the sessions include formulating a structured clinical question, conducting an efficient literature search, critically appraising published literature in both intervention and diagnostic studies, and incorporating evidence into clinical decision-making. The course was evaluated by a neonatology-adapted Fresno test (NAFT) and neonatology case vignettes, which were administered to learners before and after the curriculum. This publication includes the needs assessment survey, PowerPoint slides for the seven sessions, the NAFT, and the scoring rubric for the test.

RESULTS

The NAFT was internally reliable, with a Cronbach's alpha of .74. The intraclass correlation coefficient of the three raters' variability in assessment of learners was excellent at .98. Mean test scores increased significantly (54 points, p < .001) in 14 learners after the EBM curriculum, indicating an increase in EBM-related knowledge and skills.

DISCUSSION

This focused EBM curriculum enhances trainees' knowledge and skills and fosters evidence-based practice. The curriculum can be easily adapted for learners in pediatrics, as well as family medicine, in order to enhance trainees' EBM skills and knowledge.

Conducting multicenter research in healthcare simulation: Lessons learned from the INSPIRE network

Simulation-based research has grown substantially over the past two decades; however, relatively few published simulation studies are multicenter in nature. Multicenter research confers many distinct advantages over single-center studies, including larger sample sizes for more generalizable findings, sharing resources amongst collaborative sites, and promoting networking. Well-executed multicenter studies are more likely to improve provider performance and/or have a positive impact on patient outcomes. In this manuscript, we offer a step-by-step guide to conducting multicenter, simulation-based research based upon our collective experience with the International Network for Simulation-based Pediatric Innovation, Research and Education (INSPIRE). Like multicenter clinical research, simulation-based multicenter research can be divided into four distinct phases. Each phase has specific differences when applied to simulation research: (1) Planning phase, to define the research question, systematically review the literature, identify outcome measures, and conduct pilot studies to ensure feasibility and estimate power; (2) Project Development phase, when the primary investigator identifies collaborators, develops the protocol and research operations manual, prepares grant applications, obtains ethical approval and executes subsite contracts, registers the study in a clinical trial registry, forms a manuscript oversight committee, and conducts feasibility testing and data validation at each site; (3) Study Execution phase, involving recruitment and enrollment of subjects, clear communication and decision-making, quality assurance measures and data abstraction, validation, and analysis; and (4) Dissemination phase, where the research team shares results via conference presentations, publications, traditional media, social media, and implements strategies for translating results to practice. With this manuscript, we provide a guide to conducting quantitative multicenter research with a focus on simulation-specific issues.