Competency assessment in simulation-based procedural education

Assessment of Chief Resident Practice Readiness in a Porcine Lab: A 4-Year Experience

OBJECTIVE

General surgery training prepares residents for the autonomous practice of surgery; however, assessment for readiness for independent practice presents several challenges. The simulation lab offers a safe and standardized environment for assessing the technical skills of a resident in the absence of numerous confounders of the real operating room. We describe our experience with evaluation and remediation of chief resident assessments in a porcine simulation lab.

DESIGN

Operative skill assessment of surgical residents was conducted using anesthetized porcine models. Procedure's representative of basic and complex operative skill was chosen for the assessment. Faculty assessed the residents using a checklist for the completion of all critical operative steps. A "failing" score or "critical fail" on a given procedure determined mandatory remediation. For remediation, faculty provided immediate post-procedure feedback on all errors, and residents were offered supervised practice. Residents were then retested to demonstrate competency.

SETTING

Large animal research center at Indiana University School of Medicine, Indianapolis, IN PARTICIPANTS: From 2017 to 2020, thirty-seven PGY5 residents participated in the porcine lab over a 4-year period. These general surgery residents were assessed at the beginning of their chief year.

RESULTS

There were a total of 6 residents that failed 1 or more procedures. There were no failures in the cholecystectomy, 3 failures for Nissen, 4 failures for Hand sewn anastomosis, and 1 failure for stapled anastomosis. Two residents failed 2 procedures. All residents received remediation with a faculty member and were subsequently able to perform the procedure competently.

CONCLUSIONS

A formal simulation-based assessment of procedural competence can identify technical performance deficiencies even at the chief resident level. Combined with a formal remediation program, such deficiencies can be addressed well in advance of residency graduation. Determining the relationship of such simulation-based assessments with operative performance is currently underway.

Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism

BACKGROUND

Assessment of emergent, rare or complex medical conditions in Endocrinology and Metabolism (E&M) is an integral component of training. However, data is lacking on how this could be best achieved. The purpose of this study was to develop and administer an Objective Structured Clinical Examination (OSCE) for E&M residents, and to gather validity evidence for its use.

METHODS

A needs assessment survey was distributed to all Canadian E&M Program Directors and recent graduates to determine which topics to include in the OSCE. The top 5 topics were selected using a modified Delphi technique. OSCE cases based on these topics were subsequently developed. Five E&M residents (PGY4-5) and five junior Internal Medicine (IM) residents participated in the OSCE. Performance of E&M and IM residents was compared and results were analyzed using a Generalizability study. Examiners and candidates completed a survey following the OSCE to evaluate their experiences.

RESULTS

The mean score of IM and E&M residents was 41.7 and 69.3 % (p < 0.001), respectively, with a large effect size (partial η² = 0.75). Overall reliability of the OSCE was 0.74. Standard setting using a borderline regression method resulted in a pass rate of 100 % of E&M residents and 0 % of IM residents. All residents felt the OSCE had high value for learning as a formative exam.

CONCLUSIONS

The E&M OSCE is a feasible method for assessing emergent, rare and complex medical conditions and this study provides validity evidence to support its use in a competency-based curriculum.

Establishing Validity of a Comprehensive Hand Surgical Training and Educational Platform (STEP)

PURPOSE

The purpose of this study was to develop and perform the initial validation for the Surgical Training and Educational Platform (STEP), a cost-effective psychomotor training and assessment instrument designed to teach and evaluate fundamental skills considered critical to competency in hand surgery.

METHODS

An American Society for Surgery of the Hand (ASSH) taskforce of 13 board-certified hand surgeons developed 8 skills considered fundamental to competency in hand surgery including: (1) lag screw fixation of an oblique fracture, (2) depth of plunge during bicortical drilling, (3) central axis scaphoid fixation, (4) phalangeal fracture pinning, (5) flexor tendon repair, (6) microsurgical suturing, (7) full-thickness skin graft harvest, and (8) wrist arthroscopy. The tasks were developed from commercially available, nonclinical supplies at low cost. The startup cost for the entire system was less than $600 USD, with a cost-per-trainee-assessment of approximately $25 USD. After the tasks were finalized, 2 examiners traveled to 8 sites around the country to evaluate 93 surgeons, including 57 board-certified hand surgeons and 36 first-year orthopedic and plastic surgical residents. Scoring criteria for each task were based on a combination of time and clinically relevant quality metrics.

RESULTS

Board-certified hand surgeons significantly outperformed surgical trainees on all 8 tasks, demonstrating that the STEP appropriately differentiates the skill level of expert and novice hand surgeons.

CONCLUSIONS

The STEP is an effective simulator that encompasses a wide range of fundamental psychomotor skills considered critical to competency in hand surgery. Although the STEP tasks are a valuable surgical training tool, further work is required to evaluate and refine the scoring system prior to using it as a critical evaluation of performance.

CLINICAL RELEVANCE

The STEP simulator is a valuable, validated instrument for resident and fellow education and evaluation in hand surgery outside of the operating room.

Bougie-assisted cricothyroidotomy: Delphi-derived essential steps for the novice learner

OBJECTIVE

A cricothyroidotomy is a life-saving procedure, performed as a final option to emergency airway algorithms, and is essential for all clinicians who perform emergency airway management. The bougie-assisted cricothyroidotomy (BAC) is a novel technique that may be performed faster and with fewer complications than other traditional approaches. There is no established standard set of steps to guide the instruction of BAC performance. This study sought to systematically develop a BAC checklist for novice instruction using a modified Delphi methodology and international airway experts.

METHODS

A literature review informed the creation of a preliminary BAC checklist. A three round, modified Delphi method was used to establish a BAC checklist intended for novice-level instruction. The consensus level for each step and the final checklist were predefined at 80%. Participants were international airway experts identified by study personnel and snowball sampling.

RESULTS

Fourteen international airway experts across six acute care specialities participated in the study. The checklist was refined using a seven-point rating scale for each item and participant comments. A 17-item checklist was developed with expert consensus achieved after three rounds. Internal consistency, measured with Cronbach's α, was 0.855 (95% confidence interval 0.73-0.94).

CONCLUSION

This modified Delphi-derived checklist is the first systematically developed list of essential steps for guiding BAC instruction for novice learners. This tool serves to standardize BAC skill instruction and provide learners with a structured and consistent set of steps for deliberate practice.

Cricothyroidotomy In Situ Simulation Curriculum (CRIC Study): Training Residents for Rare Procedures

INTRODUCTION

Technical skill acquisition for rare procedures can be challenging given the few real-life training opportunities. In situ simulation (ISS), a training technique that takes place in the actual workplace, is a promising method to promote environmental fidelity for rare procedures. This study evaluated a simulation-based technical skill curriculum for cricothyroidotomy using deliberate practice, followed by an ISS evaluation session.

METHODS

Twenty emergency medicine residents participated in a two-part curriculum to improve cricothryoidotomy performance. A pretest established participant baseline technical skill. The training session consisted of two parts, didactic teaching followed by deliberate practice using a task-training manikin. A posttest consisted of an unannounced, high-fidelity ISS, during an emergency department shift. The primary outcome was the mean performance time between the pretest and posttest sessions. Skill performance was also evaluated using a checklist scale and global rating scale.

RESULTS

Cricothyroidotomy performance time improved significantly from pretest to posttest sessions (mean difference, 59 seconds; P < 0.0001). Both checklist and global rating scales improved significantly from the pretest to the posttest with a mean difference of 1.82 (P = 0.002) and 6.87 (P = 0.0025), respectively. Postcourse survey responses were favorable for both the overall curriculum experience and the unannounced ISS.

CONCLUSIONS

This pilot study demonstrated that unannounced ISS is feasible and can be used to effectively measure cricothyroidotomy performance among EM residents. After a two-part training session consisting of didactic learning and deliberate practice, improved cricothyroidotomy skill performance was observed during an unannounced ISS in the emergency department. The integration of ISS in cricothyroidotomy training represents a promising approach; however, further study is needed to establish its role.

The skeletons in our closet: E-learning tools and what happens when one side does not fit all

In the anatomical sciences, e-learning tools have become a critical component of teaching anatomy when physical space and cadaveric resources are limited. However, studies that use empirical evidence to compare their efficacy to visual-kinesthetic learning modalities are scarce. The study examined how a visual-kinesthetic experience, involving a physical skeleton, impacts learning when compared with virtual manipulation of a simple two-dimensional (2D) e-learning tool, A.D.A.M. Interactive Anatomy. Students from The University of Western Ontario, Canada (n = 77) participated in a dual-task study to: (1) investigate if a dual-task paradigm is an effective tool for measuring cognitive load across these different learning modalities; and (2) to assess the impact of knowledge recall and spatial ability when using them. Students were assessed using knowledge scores, Stroop task reaction times, and mental rotation test scores. Results demonstrated that the dual-task paradigm was not an effective tool for measuring cognitive load across different learning modalities with respect to kinesthetic learning. However, our study highlighted that handing physical specimens yielded major, positive impacts on performance that a simple commercial e-learning tool failed to deliver (P < 0.001). Furthermore, students with low spatial ability were significantly disadvantaged when they studied the bony joint and were tested on contralateral images (P = 0.046, R = 0.326). This suggests that, despite limbs being mirror images, students should be taught the anatomy of, as well as procedures on, both sides of the human body, enhancing the ability of all students, regardless of spatial ability, to take anatomical knowledge into the clinic and perform successfully. Anat Sci Educ 10: 570-588. © 2017 American Association of Anatomists.

Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

OBJECTIVE

A method for rating surgical performance of a mastoidectomy procedure that is shown to apply universally across teaching institutions has not yet been devised. This work describes the development of a rating instrument created from a multi-institutional consortium.

DESIGN

Using a participatory design and a modified Delphi approach, a multi-institutional group of expert otologists constructed a 15-element task-based checklist for evaluating mastoidectomy performance. This instrument was further refined into a 14-element checklist focusing on the concept of safety after using it to rate a large and varied population of performances.

SETTING

Twelve otolaryngological surgical training programs in the United States.

PARTICIPANTS

A total of 14 surgeons from 12 different institutions took part in the construction of the instrument.

RESULTS

By using 14 experts from 12 different institutions and a literature review, individual metrics were identified, rated as to the level of importance and operationally defined to create a rating scale for mastoidectomy performance. Initial use of the rating scale showed modest rater agreement. The operational definitions of individual metrics were modified to emphasize "safe" as opposed to "proper" technique. A second rating instrument was developed based on this feedback.

CONCLUSIONS

Using a consensus-building approach with multiple rounds of communication between experts is a feasible way to construct a rating instrument for mastoidectomy. Expert opinion alone using a Delphi method provides face and content validity evidence, however, this is not sufficient to develop a universally acceptable rating instrument. A continued process of development and experimentation to demonstrate evidence for reliability and validity making use of a large population of raters and performances is necessary to achieve universal acceptance.

The practice of simulation-based assessment in respiratory therapy education

Clinical simulation has gained prominence as an educational approach in many Canadian respiratory therapy programs and is strongly associated with improved learning, clinical and nonclinical skill, future performance, and patient outcomes. Traditionally, the primary assessment approach employed in clinical simulation has been formative debriefing for learning. Contextual factors, such as limited opportunities for learning in clinical practice and technologically oriented perspectives on learning in clinical simulation, are converging to prompt a move from using formative debriefing sessions that support learning in simulation to employing high-stakes testing intended to measure entry-to-practice competencies. We adopt the perspective that these factors are intricately linked to the profession's regulatory environment, which may strongly influence how simulation practices become embedded with respiratory therapy educational programs. Through this discussion we challenge the profession to consider how environmental factors, including externally derived requirements, may ultimately impact the effectiveness of simulation-based learning environments.

Examination of Learning Trajectories for Simulated Lumbar Puncture Training Using Hand Motion Analysis

OBJECTIVES

A prospective cohort study was conducted using hand motion analysis (HMA) to assess the acquisition and retention of technical proficiency among first-year medical students learning the lumbar puncture (LP) skill in a simulated setting.

METHODS

Twenty-five subjects attended three or four simulation sessions at 6-week intervals. The initial session consisted of an introduction to the procedure and a baseline HMA assessment. This was followed by a session involving deliberate practice and evaluation using HMA. Subject HMA metrics were followed over time and compared to performance benchmarks to determine the volume of practice required to achieve and maintain technical proficiency in the simulated setting. Performance benchmarks were derived from the assessment of experts using HMA.

RESULTS

Subject baseline metrics were significantly different from expert (p < 0.01). At the outset of session 2, none of the subjects achieved the performance benchmarks. At the outset of sessions 3 and 4, 40 and 60% of subjects, respectively, demonstrated retention of technical proficiency. However, there was evidence of significant skill erosion between sessions (p < 0.01). The mean number of practice attempts required to achieve technical proficiency declined between sessions. Regression analysis indicated that there was a significant training effect for all students (overall negative slopes) over time. When examining the group as a whole, the speed at which students reached the expert benchmark was not significantly associated with number of practices in the first three sessions, although for some individuals these factors did appear associated. A total of 76% of subjects retained technical proficiency by session 4 and required a mean of 14 practices (range = 5 to 19).

CONCLUSIONS

These results show that the majority of students require three to four sessions of deliberate practice to achieve a sustainable level of proficiency in the LP procedure. There is considerable variation in learning progression and retention of technical proficiency. These results have important implications for the design and resource requirements of a competency-based medical education program targeting LP training.

The ABCs of DKA: Development and Validation of a Computer-Based Simulator and Scoring System

BACKGROUND

Clinical management of diabetic ketoacidosis (DKA) continues to be suboptimal; simulation-based training may bridge this gap and is particularly applicable to teaching DKA management skills given it enables learning of basic knowledge, as well as clinical reasoning and patient management skills.

OBJECTIVES

1) To develop, test, and refine a computer-based simulator of DKA management; 2) to collect validity evidence, according to National Standard's validity framework; and 3) to judge whether the simulator scoring system is an appropriate measure of DKA management skills of undergraduate and postgraduate medical trainees.

DESIGN

After developing the DKA simulator, we completed usability testing to optimize its functionality. We then conducted a preliminary validation of the scoring system for measuring trainees' DKA management skills.

PARTICIPANTS

We recruited year 1 and year 3 medical students, year 2 postgraduate trainees, and endocrinologists (n = 75); each completed a simulator run, and we collected their simulator-computed scores.

MAIN MEASURES

We collected validity evidence related to content, internal structure, relations with other variables, and consequences.

KEY RESULTS

Our simulator consists of six cases highlighting DKA management priorities. Real-time progression of each case includes interactive order entry, laboratory and clinical data, and individualised feedback. Usability assessment identified issues with clarity of system status, user control, efficiency of use, and error prevention. Regarding validity evidence, Cronbach's α was 0.795 for the seven subscales indicating favorable internal structure evidence. Participants' scores showed a significant effect of training level (p < 0.001). Scores also correlated with the number of DKA patients they reported treating, weeks on Medicine rotation, and comfort with managing DKA. A score on the simulation exercise of 75 % had a sensitivity and specificity of 94.7 % and 51.8%, respectively, for delineating between expert staff physicians and trainees.

CONCLUSIONS

We demonstrate how a simulator and scoring system can be developed, tested, and refined to determine its quality for use as an assessment modality. Our evidence suggests that it can be used for formative assessment of trainees' DKA management skills.

Do orthopaedic fracture skills courses improve resident performance?

INTRODUCTION

We hypothesized that resident participation in a hands-on fracture fixation course leads to significant improvement in their performance as assessed in a simulated fracture fixation model.

METHODS

Twenty-three junior orthopaedic surgery residents were tasked to treat radial shaft fractures with standard fixation techniques in a sawbones fracture fixation simulation twice during the year. Before the first simulation, 6 of the residents participated in a fraction fixation skills course. The simulation repeated 6 months later after all residents attended the course. Residents also completed a 15-question written examination. Assessment included evaluation of each step of the procedure, a score based on the objective structured assessment of technical skill (OSATS) system, and grade on the examination. Comparisons were made between the two cohorts and the two testing time points.

RESULTS

Significant improvements were present in the percentage of tasks completed correctly (64.1% vs 84.3%) the overall OSATS score (13.8 vs 17.1) and examination correct answers (8.6 vs 12.5) for the overall cohort between the two testing time points (p<0.001, p<0.03, p<0.04 respectively). Residents who had not participated in the surgical skills course at the time of their initial simulation demonstrated significant improvements in percentage of tasks completed correctly (61.3% vs 81.2%) and OSATS score (12.4 vs 17.0) (p<0.002, p<0.01 respectively). No significant difference was noted in performance for the cohort who had already participated in the course (p=0.87 and p=0.68). The cohort that had taken the course prior to the initial simulation showed significantly higher scores at initial evaluation (88.5% vs 58.5% percentage of tasks completed correctly, 17.3 vs 12.0 OSATS score, 12.5 vs 8.6 correct answers on the examination). At the second simulation, no significant difference was seen with task completion or examination grade, but a significant difference still existed with respect to the OSATS score (20.0 vs 17.0; p<0.03).

CONCLUSION

Participation in a formal surgical skills course significantly improved practical operative skills as assessed by the simulation. The benefits of the course were maintained to 6 months with residents who completed the training earlier continuing to demonstrate an advantage in skills. Such courses are a valuable training resource which directly impact resident performance.

Implementing a third-year emergency medicine medical student curriculum

BACKGROUND

Emergency medicine (EM) is commonly introduced in the fourth year of medical school because of a perceived need to have more experienced students in the complex and dynamic environment of the emergency department. However, there is no evidence supporting the optimal time or duration for an EM rotation, and a number of institutions offer third-year rotations.

OBJECTIVE

A recently published syllabus provides areas of knowledge, skills, and attitudes that third-year EM rotation directors can use to develop curricula. This article expands on that syllabus by providing a comprehensive curricular guide for the third-year medical student rotation with a focus on implementation.

DISCUSSION

Included are consensus-derived learning objectives, discussion of educational methods, considerations for implementation, and information on feedback and evaluation as proposed by the Clerkship Directors in Emergency Medicine Third-Year Curriculum Work Group. External validation results, derived from a survey of third-year rotation directors, are provided in the form of a content validity index for each content area.

CONCLUSIONS

This consensus-derived curricular guide can be used by faculty who are developing or revising a third-year EM medical student rotation and provide guidance for implementing this curriculum at their institution.

Curriculum development for basic gynaecological laparoscopy with comparison of expert trainee opinions; prospective cross-sectional observational study

OBJECTIVE

To develop content for a basic laparoscopic curriculum in gynaecology.

STUDY DESIGN

Prospective cross-sectional observational study. Modified Delphi method with three iterations undertaken by an invited group of national experts across the United Kingdom (UK). Two anonymous online surveys and a final physical group meeting were undertaken. Junior trainees in gynaecology undertook a parallel iteration of the Delphi process for external validation. Population included: expert panel - certified specialists in minimal-access gynaecological surgery, RCOG national senior trainee representatives, and medical educationalists, junior trainees group - regional trainees in gynaecology in first and second year of speciality training.

RESULTS

Experts (n=37) reached fair to almost complete significant agreement (κ=0.100-0.8159; p<0.05) on eight out of nine questions by the second iteration. Trainees (n=19) agreed with the experts on 89% (51/57) of categories to be included in the curriculum. Findings indicated that 39 categories should be included in the curriculum. Port placement, laparoscopic equipment and patient selection were ranked the most important theoretical categories. Hand-eye co-ordination, camera navigation and entry techniques were deemed the most valuable skills. Diagnostic laparoscopy, laparoscopic sterilisation, and laparoscopic salpingectomy were the operations agreed to be most important for inclusion. Simulation training was agreed as the method of skill development. The expert panel favoured box trainers, whereas the junior trainee group preferred virtual reality simulators. A basic simulation laparoscopic hand-eye co-ordination test was proposed as a final assessment of competence in the curriculum.

CONCLUSION

Consensus was achieved on the content of a basic laparoscopic curriculum in gynaecology, in a cost- and time-effective, scientific process. The Delphi method provided a simple, structured consumer approach to curriculum development that combined views of trainers and trainees that could be used to develop curricula in other areas of post-graduate education.

Evaluation of skill level between trainees and community orthopaedic surgeons using a virtual reality arthroscopic knee simulator

BACKGROUND

Several virtual reality simulators have been developed to assist orthopaedic surgeons in acquiring the skills necessary to perform arthroscopic surgery. The purpose of this study was to assess the construct validity of the ArthroSim virtual reality arthroscopy simulator by evaluating whether skills acquired through increased experience in the operating room lead to improved performance on the simulator.

METHODS

Using the simulator, six postgraduate year-1 orthopaedic residents were compared with six postgraduate year-5 residents and with six community-based orthopaedic surgeons when performing diagnostic arthroscopy. The time to perform the procedure was recorded. To ensure that subjects did not sacrifice the quality of the procedure to complete the task in a shorter time, the simulator was programmed to provide a completeness score that indicated whether the surgeon accurately performed all of the steps of diagnostic arthroscopy in the correct sequence.

RESULTS

The mean time to perform the procedure by each group was 610 seconds for community-based orthopaedic surgeons, 745 seconds for postgraduate year-5 residents, and 1028 seconds for postgraduate year-1 residents. Both the postgraduate year-5 residents and the community-based orthopaedic surgeons performed the procedure in significantly less time (p = 0.006) than the postgraduate year-1 residents. There was a trend toward significance (p = 0.055) in time to complete the procedure when the postgraduate year-5 residents were compared with the community-based orthopaedic surgeons. The mean level of completeness as assigned by the simulator for each group was 85% for the community-based orthopaedic surgeons, 79% for the postgraduate year-5 residents, and 71% for the postgraduate year-1 residents. As expected, these differences were not significant, indicating that the three groups had achieved an acceptable level of consistency in their performance of the procedure.

CONCLUSIONS

Higher levels of surgeon experience resulted in improved efficiency when performing diagnostic knee arthroscopy on the simulator. Further validation studies utilizing the simulator are currently under way and the additional simulated tasks of arthroscopic meniscectomy, meniscal repair, microfracture, and loose body removal are being developed.

Role and challenges of simulation in undergraduate curriculum

RATIONALE

Medical simulation is relatively a novel technology widely utilized for teaching and assessing students clinical skills. Students and faculty face many challenges when simulation sessions are introduced into undergraduate curriculum.

OBJECTIVES

The aim of this study is to obtain the opinion of undergraduate medical students and our faculty regarding the role of simulation in undergraduate curriculum, the simulation modalities used, and the perceived barriers in implementing simulation sessions.

METHODS

A self-administered pilot tested questionnaire with 18 items using a 5-point Likert scale was distributed to undergraduate male (n = 125) and female students (n = 70) as well as to the faculty members (n = 14) at King Fahad Medical City, King Saud Bin Abdul Aziz University of Health Sciences, Saudi Arabia, to respond. Survey elements addressed the role of simulation, simulation modalities used, and perceived challenges to implementation of simulation sessions.

RESULT

Various learning outcomes are achieved and improved through the technology enhanced simulation sessions such as communication skills, diagnostic skills, procedural skills, self-confidence, and integration of basic and clinical sciences. The use of high fidelity simulators, simulated patients and task trainers was more desirable by our students and faculty for teaching and learning as well as an evaluation tool. According to most of the students', institutional support in terms of resources, staff and duration of sessions was adequate. However, motivation to participate in the sessions and provision of adequate feedback by the staff was a constraint.

CONCLUSION

The use of simulation laboratory is of great benefit to the students and a great teaching tool for the staff to ensure students learn various skills.

Development and verification of a taxonomy of assessment metrics for surgical technical skills

PURPOSE

To create and empirically verify a taxonomy of metrics for assessing surgical technical skills, and to determine which types of metrics, skills, settings, learners, models, and instruments were most commonly reported in the technical skills assessment literature.

METHOD

In 2011-2012, the authors used a rational analysis of existing and emerging metrics to create the taxonomy, and used PubMed to conduct a systematic literature review (2001-2011) to test the taxonomy's comprehensiveness and verifiability. Using 202 articles identified from the review, the authors classified metrics according to the taxonomy and coded data concerning their context and use. Frequencies (counts, percentages) were calculated for all variables.

RESULTS

The taxonomy contained 12 objective and 4 subjective categories. Of 567 metrics identified in the literature, 520 (92%) were classified using the new taxonomy. Process metrics outnumbered outcome metrics by 8:1. The most frequent metrics were "time," "manual techniques" (objective and subjective), "errors," and "procedural steps." Only one new metric, "learning curve," emerged. Assessments of basic motor skills and skills germane to laparoscopic surgery dominated the literature. Novices, beginners, and intermediate learners were the most frequent subjects, and box trainers and virtual reality simulators were the most frequent models used for assessing performance.

CONCLUSIONS

Metrics convey what is valued in human performance. This taxonomy provides a common nomenclature. It may help educators and researchers in procedurally oriented disciplines to use metrics more precisely and consistently. Future assessments should focus more on bedside tasks and open surgical procedures and should include more outcome metrics.

Do the skills acquired by novice surgeons using anatomic dry models transfer effectively to the task of diagnostic knee arthroscopy performed on cadaveric specimens?

BACKGROUND

The use of surgical simulation in orthopaedic education is increasing; however, its ideal place within the training curriculum remains unknown. The purpose of this study was to determine the effectiveness of training novice surgeons on an anatomic dry model of the knee prior to training them to perform diagnostic arthroscopy on cadaveric specimens.

METHODS

Fourteen medical students were randomly assigned to two groups. The experimental group was trained to perform diagnostic arthroscopy of the knee on anatomic dry models prior to training on cadaveric specimens. The control group was trained only on cadaveric specimens. Proficiency was assessed with use of a modified version of a previously validated objective assessment of arthroscopic skill, the Basic Arthroscopic Knee Skill Scoring System (BAKSSS). The mean number of trials required to attain minimal proficiency when performing diagnostic knee arthroscopy was compared between the groups. The cumulative transfer effectiveness ratio (CTER) was calculated to measure the transfer of skills acquired by the experimental group.

RESULTS

The mean number of trials to demonstrate minimum proficiency was significantly lower in the experimental group (2.57) than in the control group (4.57) (p &lt; 0.01). The mean time to demonstrate proficiency was also significantly less in the experimental group (37.51 minutes) than in the control group (60.48 minutes) (p &lt; 0.01). The CTER of dry-model training for the task of performing diagnostic knee arthroscopy on cadaveric specimens was 0.2.

CONCLUSIONS

Previous training utilizing an anatomic dry knee model resulted in improved proficiency for novice surgeons learning to perform diagnostic knee arthroscopy on cadaveric specimens. A CTER of 0.2 suggests that dry models can serve as a useful adjunct to cadaveric training for diagnostic knee arthroscopy but cannot entirely replace it within the orthopaedic curriculum. Further work is necessary to determine the optimal amount of training on anatomic dry models that will maximize transfer effectiveness and to determine how well skills obtained in the simulated environment transfer to the operating room.

Impact of endovascular simulator training on vascular surgery as a career choice in medical students

OBJECTIVE

The primary goal of this study was to determine whether exposure to endovascular simulator training increases interest in vascular surgery among medical students. Secondary goals were to determine whether interest in vascular surgery is inversely related to the time after exposure, to identify factors associated with interest, and to identify students' characteristics that positively influence performance metrics.

METHODS

This was a prospective, randomized, crossover study comprising 80 medical students who were randomized into group A (n = 40) and group B (n = 40). Participants completed a survey of their interest in vascular surgery and attitudinal factors using Vascular Surgery Interest Form (VSIF) before exposure to the simulator (pretest). At 1 month after exposure of group A to the simulator, both groups were tested using VSIF (test). Upon completion of testing, group B was exposed to simulator training, whereas group A received no further training. At 2 months after exposure of group B to the simulator, both groups were posttested using VSIF, which asked the students' level of interest in vascular surgery using a 1 to 10 scale. Performance metrics were recorded during each exposure. Differences among cohort demographics were determined using Pearson χ(2) analysis. Differences in interest were determined with paired sample correlations. Linear regression and analysis of variance were used to correlate VSIF responses with interest and the performance metrics.

RESULTS

Both student cohorts had significant increases in interest after exposure to simulation. In group A, test interest (mean ± standard deviation) was significantly higher than pretest and posttest interests (5.51 ± 1.73 vs 4.00 ± 1.88 vs 4.18 ± 1.82; P < .05). In group B, posttest interest was significantly higher than pretest and test interests (5.62 ± 2.03 vs 3.96 ± 1.61 vs 4.08 ± 1.64; P < .05). The increase in interest was reciprocally related to the time passed since the initial exposure. Resident and attending lifestyle, length of training, radiation concerns, gender identification of a mentor, and personality fit with occupation were not correlated with interest. Sex, medical school year, comfort with endovascular procedures, willingness to work long hours, interest in performing percutaneous procedures, and commitment to surgical career did not affect impact performance metrics.

CONCLUSIONS

One exposure of students to endovascular simulator training is associated with an increase in vascular surgery interest. Acquired interest is reciprocally related to the time demonstrating the temporal importance of the exposure.

Training and simulation in otolaryngology

This article focuses on key issues surrounding the needs and application of simulation technologies for technical skills training in otolaryngology. The discussion includes an overview of key topics in training and learning, the application of these issues in simulation environments, and the subsequent applications of these simulation environments to otolaryngology. Examples of past applications are presented, with discussion of how the interplay of cultural changes in surgical training in general along with the rapid advancements in technology have shaped and influenced their adoption and adaptation. The authors conclude with emerging trends and potential influences advanced simulation and training will have on technical skills training in otolaryngology.

What is the cost associated with the implementation and maintenance of an ACS/APDS-based surgical skills curriculum?

PURPOSE

The purpose of this study was to evaluate the cost associated with the American College of Surgery (ACS)/Association of Program Directors in Surgery (APDS)-based surgical skills curriculum (SSC) within a general surgery residency program.

METHODS

The Penn Surgical Simulation Center (PSSC) of the University of Pennsylvania was established by the Department of Surgery during the 2006-2007 academic year and became a Level-I ACS Accredited Education Institute in 2008. Each academic year, 38 junior residents are assigned to a 4-week dedicated simulation rotation based on the ACS/APDS-based SSC. In conjunction with voluntary participation by faculty, a salaried educational fellow is responsible for maintaining the schedule and administering the surgical skills training modules. The costs associated with the ACS/APDS-based SSC were divided in initial implementation capital expenses and annual operational maintenance expenses.

RESULTS

The overall capital expenditures associated with the implementation of the curriculum were $4.204 million. These costs included the purchase of low and high-fidelity simulation equipment and initial construction costs to renovate a previous operating room (OR) and recovery suite into the Penn Medicine Clinical Simulation Center (PMCSC) which has housed the PSSC since 2008. The annual operational expenses are $476,000 and include the salary for the educational fellow, disposables, and other supplies, and the PMCSC average student fees. The annual cost per resident for the 4-week dedicated simulation rotation is $12,516. This figure does not include the average cost for teaching efforts including the simulation teaching per participating faculty member which is $30,000 in Relative Teaching Value Units per year.

CONCLUSIONS

The expenditures associated with the implementation and maintenance of the ACS/APDS-based surgical skills curriculum in a surgical residency program are significant. This center's experience might be useful to programs deciding on more cost-effective means of implementing the ACS/APDS-SSC into their training.

Deliberate practice of motor skills in nursing education: CPR as exemplar

Our study explored the effects of deliberate practice on the retention ofcardiopulmonary resuscitation (CPR) psychomotor skills among nursing students. The practice sessions were short, six minutes a session one time a month. Differences in performance between students who had deliberate practice and a control group, with no practice beyond the initial training, were compared every three months for one year. The intervention group performed better than the control over the 12 months. There is a need in nursing education for deliberate practice of relevant and high-use skills for students to improve their performance and gradually develop their expertise.

Orthopedic boot camp: examining the effectiveness of an intensive surgical skills course

BACKGROUND

Changes in health care across the globe have had a profound impact on the number of hands-on surgical training opportunities that are available to residents. In the current study, we examine whether an intensive laboratory-based skills course at the start of orthopedic surgical training is an effective mechanism for teaching core technical skills.

METHODS

First-year residents were divided into 3 groups (on-service, n = 8; off-service, n = 8; and a new, competency-based program that has as a major element of the curriculum a focused, intensive skills laboratory-based experience, n = 6). Baseline surgical skills were assessed prior to commencing training. The intensive skills laboratory group was then given an intensive surgical skills course, whereas the other 2 groups embarked on traditional residency. After the surgical skills course, all the residents were assessed for core surgical skills using an objective structured assessment of technical skills (OSATS) procedure.

RESULTS

Pretraining scores revealed no differences between the groups of residents using both checklist (F[2,19] = 0.852, P = .442) and global rating scores (F[2,19] = 0.704, P = .507). Post-training scores revealed a significant difference, with residents from the intensive skills laboratory group performing better on both the checklists (on-service = 78.9, off-service = 78.6, intensive skills laboratory = 92.3; F[2,19] = 6.914, P < .01) and global rating scores (on-service = 3.4, off-service = 3.4, intensive skills laboratory = 4.3; F[2,19] = 5.722, P < .01), than the other groups who showed no differences between them.

CONCLUSION

The intensive skills course used in this study was highly effective at teaching and developing targeted surgical skills in first-year orthopedic residents. We predict that allowing residents to acquire key technical skills at the start of their training will enhance learning opportunities at later stages of training.

Measuring learning gain during a one-day introductory bronchoscopy course

BACKGROUND

Rigorous assessment of medical knowledge and technical skill inspires learning, reinforces confidence, and reassures the public. Identifying curricular effectiveness using objective measures of learning is therefore crucial for competency-oriented program development in a learner-centric educational environment. The aim of this study was to determine whether various measures of learning, including class-average normalized gain, can be used to assess the effectiveness of a one-day introductory bronchoscopy course curriculum.

METHODS

We conducted a quasi-experimental one-group pre-test/post-test study at the University of California, Irvine. The group comprised 24 first-year pulmonary and critical care trainees from eight training institutions in southern California. Class-average normalized gain, single-student normalized gain, absolute gain, and relative gain were used as objective measures of cognitive knowledge and bronchoscopy technical skill learning. A class-average normalized gain of 30% was used to determine curricular effectiveness. Perceived educational value using Likert-scale surveys and post-course questionnaires was determined during and 3 months after course participation.

RESULTS

Mean test scores of cognitive knowledge improved significantly from 48 to 66% (p = 0.043). Absolute gain for the class was 18%, relative gain was 37%, class average normalized gain was 34%, and the average of the single-student normalized gains g(ave) was 29%. Mean test scores of technical skill improved significantly from 43 to 77% (p = 0.017). Absolute gain was 34%, relative gain was 78%, class average normalized gain was 60%, and the average of the single-student normalized gains g(ave) was 59%. Statistically significant improvements in absolute gain were noted in all five elements of technical skill (p < 0.05). Likert-scale surveys, questionnaires, and surveys demonstrated strong perceived educational value.

CONCLUSION

The effectiveness of a one-day introductory bronchoscopy curriculum was demonstrated using a pre-test/post-test model with calculation of normalized gain and related metrics.

A prospective comparison between written examination and either simulation-based or oral viva examination of intensive care trainees' procedural skills

We compared results of written assessment of intensive care trainees' procedural skills with results obtained from one of two live assessment formats for the purposes of assessing the concurrent validity of the different test methods. Forty-five Australasian senior trainees in intensive care medicine completed a written test relating to a procedural skill, as well as either a simulation format or oral viva assessment on the same procedural skill. We analysed correlation between written exam results and results obtained from simulation format or oral viva assessment. For those who completed the simulation format examination, we also maintained a narrative of actions and identified critical errors. There was limited correlation between written exam results and live (simulation or viva) procedure station results (r = 0.31). Correlation with written exam results was very low for simulation format assessments (r = 0.08) but moderate for oral viva format assessment (r = 0.58). Participants who passed a written exam based on management of a blocked tracheostomy scenario performed a number of dangerous errors when managing a simulated patient in that scenario. The lack of correlation between exam formats supports multi-modal assessment, as currently it is not known which format best represents workplace performance. Correlation between written and oral viva results may indicate redundancy between those test formats, whereas limited correlation between simulation and written exams may support the use of both formats as part of an integrated assessment strategy. We hypothesise that identification of critical candidate errors in a simulation format exam that were not exposed in a written exam may indicate better predictive validity for simulation format examination of procedural skills.