A systems approach for institutional CBME adoption at Queen's University

The Royal College of Physicians and Surgeons of Canada (RCPSC) has begun the transition to Competency by Design (CBD), a new curricular model for residency education that 'ensure[s] competence, but teaches for excellence'. By 2022, all Canadian specialty programs are anticipated to have completed the CBD cohort process which includes workshops facilitated by a Royal College Clinician Educator. Queen's University in Ontario, Canada, was granted approval by the RCPSC to embark upon an accelerated path to competency-based medical education (CBME) for all our postgraduate specialties. This accelerated path allowed us to take an institutional approach for CBME implementation and ensure that all specialities were part of a system-wide change. Our unique institution-wide approach to CBD is the first of its kind across Canada. From both a theoretical and practical perspective we undertook CBME using a systems approach that allowed us to build the foundations for CBME, implement the change, and plan for sustainability. This has created opportunities to bridge and connect the various programs involved in the implementation of CBME on Queen's campus. The systems approach was an essential part of our strategy to develop a community dedicated to ensuring a successful CBME implementation.

Delivering on the promise of competency based medical education - an institutional approach

The Royal College of Physicians and Surgeons of Canada (RCPSC) adopted a plan to transform, over a seven-year horizon (2014-2021), residency education across all specialties to competency-based medical education (CBME) curriculum models. The RCPSC plan recommended implementing a more responsive and accountable training model with four discrete stages of training, explicit, specialty specific entrustable professional activities, with associated milestones, and a programmatic approach to assessment across residency education. Embracing this vision, the leadership at Queen's University (in Kingston, Ontario, Canada) applied for and was granted special permission by the RCPSC to embark on an accelerated institutional path. Over a three-year period, Queen's took CBME from concept to reality through the development and implementation of a comprehensive strategic plan. This perspective paper describes Queen's University's approach of creating a shared institutional vision, outlines the process of developing a centralized CBME executive team and twenty-nine CBME program teams, and summarizes proactive measures to ensure program readiness for launch. In so doing, Queen's created a community of support and CBME expertise that reinforces shared values including fostering co-production, cultivating responsive leadership, emphasizing diffusion of innovation, and adopting a systems-based approach to transformative change.

Student Reflections on the Queen's Accelerated Route to Medical School Programme

CONTEXT

Since its inception more than 150 years ago, the School of Medicine at Queen's University has aspired 'to advance the tradition of preparing excellent physicians and leaders in health care by embracing a spirit of inquiry and innovation in education and research'. As part of this continuing commitment, Queen's School of Medicine developed the Queen's University Accelerated Route to Medical School (QuARMS). As Canada's only 2-year accelerated-entry premedical programme, QuARMS was designed to reduce training time, the associated expense of medical training, and to encourage a collaborative premedical experience. Students enter QuARMS directly from high school and then spend 2 years enrolled in an undergraduate degree programme. They then are eligible to enter the first-year MD curriculum. The 2-year QuARMS academic curriculum includes traditional undergraduate coursework, small group sessions, and independent activities. The QuARMS curriculum is built on 4 pillars: communication skills, critical thinking, the role of physician (including community service learning [CSL]), and scientific foundations. Self-regulated learning (SRL) is explicitly developed throughout all aspects of the curriculum. Medical educators have defined SRL as the cyclical control of academic and clinical performance through several key processes that include goal-directed behaviour, use of specific strategies to attain goals, and the adaptation and modification to behaviours or strategies that optimize learning and performance. Based on Zimmerman's social cognitive framework, this definition includes relationships among the individual, his or her behaviour, and the environment, with the expectation that individuals will monitor and adjust their behaviours to influence future outcomes.

OBJECTIVES

This study evaluated the students' learning as perceived by them at the conclusion of their first 2 academic years.

METHODS

At the end of the QuARMS learning stream, the first and second cohorts of students completed a 26-item, 4-point Likert-type instrument with space for optional narrative details for each question. A focus group with each group explored emergent issues. Consent was obtained from 9 out of 10 and 7 out of 8 participants to report the 2015 survey and focus group data, respectively, and from 10 out of 10 and 9 out of 10 participants to report the 2016 survey and focus group data, respectively. Thematic analysis and a constructivist interpretive paradigm were used. A distanced facilitator, standard protocols, and a dual approach assured consistency and trustworthiness of data.

RESULTS

Both analyses were congruent. Students described experiences consistent with curricular goals including critical thinking, communication, role of a physician, CSL, and SRL. Needs included additional mentorship, more structure for CSL, more feedback, explicit continuity between in-class sessions, and more clinical experience. Expectations of students towards engaging in independent learning led to some feelings of disconnectedness.

CONCLUSIONS

Participants described benefit from the sessions and an experience consistent with the curricular goals, which were intentionally focused on foundational skills. In contrast to the goal of SRL, students described a need for an explicit educational structure. Thus, scaffolding of the curriculum from more structured in year 1 to less structured in year 2 using additional mentorship and feedback is planned for subsequent years. Added clinical exposure may increase relevance but poses challenges for integration with the first-year medical class.

Eight-year outcomes of a competency-based residency training program in orthopedic surgery

Background: The Division of Orthopaedic Surgery at the University of Toronto implemented a pilot residency training program that used a competency-based framework in July of 2009. The competency-based curriculum (CBC) deployed an innovative, modularized approach that dramatically intensified both the structured learning elements and the assessment processes. Methods: This paper discusses the initial curriculum design of the CBC pilot program; the refinement of the curriculum using curriculum mapping that allowed for efficiencies in educational delivery; details of evaluating resident competence; feedback from external reviews by accrediting bodies; and trainee and program outcomes for the first eight years of the program's implementation. Results: Feedback from the residents, the faculty, and the postgraduate residency training accreditation bodies on the CBC has been positive and suggests that the essential framework of the program may provide a valuable tool to other programs that are contemplating embarking on transition to competency-based education. Conclusions: While the goal of the program was not to shorten training per se, efficiencies gained through a modular, competency-based program have resulted in shortened time to completion of residency training for some learners.

Comparison of faculty versus structured peer-feedback for acquisitions of basic and intermediate-level surgical skills

PURPOSE

Video feedback and faculty feedback has been shown to improve surgical performance; however, consistent access to faculty is challenging. We studied the utility of structured peer-feedback (PF) compared to faculty-feedback (FF) during acquisition of basic and intermediate surgical skills.

METHODOLOGY

Two randomized non-inferiority trials were conducted with 1st (n = 30) and 2nd year (n = 29) medical students learning skin-lesion excision and closure (S), and single-layer hand-sewn bowel anastomosis (B), respectively. Five attempts were performed. PF participants used an Objective Structured Assessment of Technical Skills tool to guide feedback. Blinded raters assessed video-recorded performance, time and Integrity of the completed task were also assessed.

RESULTS

For both tasks performance by PF was comparable to FF (P = 0.111). Both groups improved significantly: performance (B:P < 0.0001, S:P = 0.035), time (B:P = 0.043, S:P < 0.0001) and integrity (B:P < 0.0001, S:P < 0.032).

CONCLUSION

Structured peer-feedback is equivalent to faculty-feedback in the acquisition of basic and intermediate surgical skills, giving students freedom to practice independently.

Comparison of Canadian and Swiss Surgical Training Curricula: Moving on Toward Competency-Based Surgical Education

OBJECTIVE

Quality of surgical training in the era of resident duty-hour restrictions (RDHR) is part of an ongoing debate. Most training elements are provided during surgical service. As exposure to surgical procedures is important but time-consuming, RDHR may affect quality of surgical training. Providing structured training elements may help to compensate for this shortcoming.

DESIGN

This binational anonymous questionnaire-based study evaluates frequency, time, and structure of surgical training programs at 2 typical academic teaching hospitals with different RDHR.

SETTING

Departments of Surgery of University of Basel (Basel, Switzerland) and the Queen's University (Kingston, Ontario, Canada).

PARTICIPANTS

Surgical consultants and residents of the Queen's University Hospital (Kingston, Ontario, Canada) and the University Hospital Basel (Basel, Switzerland) were eligible for this study.

RESULTS

Questionnaire response rate was 37% (105/284). Queen's residents work 80 hours per week, receiving 7 hours of formal training (8.8% of workweek). Basel residents work 60 hours per week, including 1 hour of formal training (1.7% of working time). Queen's faculty and residents rated their program as "structured" or "rather structured" in contrast to Basel faculty and residents who rated their programs as "neutral" in structure or "unstructured." Respondents identified specific structured training elements more frequently at Queen's than in Basel. Two-thirds of residents responded that they seek out additional surgical experiences through voluntary extra work. Basel participants articulated a stronger need for improvement of current surgical training. Although Basel residents and consultants in both institutions fear negative influence of RDHR on the training program, this was not the case in Queen's residents.

CONCLUSIONS

Providing more structured surgical training elements may be advantageous in providing optimal-quality surgical education in an era of work-hour restrictions.

Gastrointestinal Endoscopy Competency Assessment Tool: reliability and validity evidence

BACKGROUND

Rigorously developed and validated direct observational assessment tools are required to support competency-based colonoscopy training to facilitate skill acquisition, optimize learning, and ensure readiness for unsupervised practice.

OBJECTIVE

To examine reliability and validity evidence of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) for colonoscopy for use within the clinical setting.

DESIGN

Prospective, observational, multicenter validation study. Sixty-one endoscopists performing 116 colonoscopies were assessed using the GiECAT, which consists of a 7-item global rating scale (GRS) and 19-item checklist (CL). A second rater assessed procedures to determine interrater reliability by using intraclass correlation coefficients (ICCs). Endoscopists' first and second procedure scores were compared to determine test-retest reliability by using ICCs. Discriminative validity was examined by comparing novice, intermediate, and experienced endoscopists' scores. Concurrent validity was measured by correlating scores with colonoscopy experience, cecal and terminal ileal intubation rates, and physician global assessment.

SETTING

A total of 116 colonoscopies performed by 33 novice (<50 previous procedures), 18 intermediate (50-500 previous procedures), and 10 experienced (>1000 previous procedures) endoscopists from 6 Canadian hospitals.

MAIN OUTCOME MEASUREMENTS

Interrater and test-retest reliability, discriminative, and concurrent validity.

RESULTS

Interrater reliability was high (total: ICC=0.85; GRS: ICC=0.85; CL: ICC=0.81). Test-retest reliability was excellent (total: ICC=0.91; GRS: ICC=0.93; CL: ICC=0.80). Significant differences in GiECAT scores among novice, intermediate, and experienced endoscopists were noted (P<.001). There was a significant positive correlation (P<.001) between scores and number of previous colonoscopies (total: ρ=0.78, GRS: ρ=0.80, CL: Spearman's ρ=0.71); cecal intubation rate (total: ρ=0.81, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.75); ileal intubation rate (total: Spearman's ρ=0.82, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.77); and physician global assessment (total: Spearman's ρ=0.90, GRS: Spearman's ρ=0.94, CL: Spearman's ρ=0.77).

LIMITATIONS

Nonblinded assessments.

CONCLUSION

This study provides evidence supporting the reliability and validity of the GiECAT for use in assessing the performance of live colonoscopies in the clinical setting.

Simulation for Teaching Orthopaedic Residents in a Competency-based Curriculum: Do the Benefits Justify the Increased Costs?

BACKGROUND

Although simulation-based training is becoming widespread in surgical education and research supports its use, one major limitation is cost. Until now, little has been published on the costs of simulation in residency training. At the University of Toronto, a novel competency-based curriculum in orthopaedic surgery has been implemented for training selected residents, which makes extensive use of simulation. Despite the benefits of this intensive approach to simulation, there is a need to consider its financial implications and demands on faculty time.

QUESTIONS/PURPOSES

This study presents a cost and faculty work-hours analysis of implementing simulation as a teaching and evaluation tool in the University of Toronto's novel competency-based curriculum program compared with the historic costs of using simulation in the residency training program.

METHODS

All invoices for simulation training were reviewed to determine the financial costs before and after implementation of the competency-based curriculum. Invoice items included costs for cadavers, artificial models, skills laboratory labor, associated materials, and standardized patients. Costs related to the surgical skills laboratory rental fees and orthopaedic implants were waived as a result of special arrangements with the skills laboratory and implant vendors. Although faculty time was not reimbursed, faculty hours dedicated to simulation were also evaluated. The academic year of 2008 to 2009 was chosen to represent an academic year that preceded the introduction of the competency-based curriculum. During this year, 12 residents used simulation for teaching. The academic year of 2010 to 2011 was chosen to represent an academic year when the competency-based curriculum training program was functioning parallel but separate from the regular stream of training. In this year, six residents used simulation for teaching and assessment. The academic year of 2012 to 2013 was chosen to represent an academic year when simulation was used equally among the competency-based curriculum and regular stream residents for teaching (60 residents) and among 14 competency-based curriculum residents and 21 regular stream residents for assessment.

RESULTS

The total costs of using simulation to teach and assess all residents in the competency-based curriculum and regular stream programs (academic year 2012-2013) (CDN 155,750, USD 158,050) were approximately 15 times higher than the cost of using simulation to teach residents before the implementation of the competency-based curriculum (academic year 2008-2009) (CDN 10,090, USD 11,140). The number of hours spent teaching and assessing trainees increased from 96 to 317 hours during this period, representing a threefold increase.

CONCLUSIONS

Although the financial costs and time demands on faculty in running the simulation program in the new competency-based curriculum at the University of Toronto have been substantial, augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more efficient learning outcomes.

CLINICAL RELEVANCE

The higher costs and demands on faculty time associated with implementing simulation for teaching and assessment must be considered when it is used to enhance surgical training.

Lessons learned in the pursuit of a dream

CONTEXT

The author describes a career in which he combined clinical surgery with the formal study of medical education. In the 1980s, when the author embarked on this career track, it was an uncommon pathway. Over the last 30 years there has been an exponential increase in the number of individuals who have made medical education their principal academic focus. This paper provides examples from the author's personal story and lessons derived from that experience.

PROCESS

The author outlines his experience of attaining formal training in education and concludes that this training was a foundational element in his pursuit of a career in health education research. The author describes his involvement in the transition from paper and pencil-based tests to performance-based testing in high-stakes examinations. He describes the development of a research centre in health professions education and the establishment of a simulation centre. The author's experiences in the development of an examination intended to measure technical skills, in the adoption of surgical safety checklists and in the elaboration of a programme in competency-based education are discussed.

DISCUSSION

The author describes several of the lessons learned in the course of his career in medical education. He argues that successful enterprises in scholarship in medicine are almost invariably the product of interdisciplinarity. He describes the power of a joint venture between a university and an academic hospital. He argues that the geographical footprint of an emerging centre is critical. He discusses the importance of graduate studentship in an emerging discipline and enterprise.

Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy

BACKGROUND

Ensuring competence remains a seminal objective of endoscopy training programs, professional organizations, and accreditation bodies; however, no widely accepted measure of endoscopic competence currently exists.

OBJECTIVE

By using Delphi methodology, we aimed to develop and establish the content validity of the Gastrointestinal Endoscopy Competency Assessment Tool for colonoscopy.

DESIGN

An international panel of endoscopy experts rated potential checklist and global rating items for their importance as indicators of the competence of trainees learning to perform colonoscopy. After each round, responses were analyzed and sent back to the experts for further ratings until consensus was reached.

MAIN OUTCOME MEASUREMENTS

Consensus was defined a priori as ≥80% of experts, in a given round, scoring ≥4 of 5 on all remaining items.

RESULTS

Fifty-five experts agreed to be part of the Delphi panel: 43 gastroenterologists, 10 surgeons, and 2 endoscopy managers. Seventy-three checklist and 34 global rating items were generated through a systematic literature review and survey of committee members. An additional 2 checklist and 4 global rating items were added by Delphi panelists. Five rounds of surveys were completed before consensus was achieved, with response rates ranging from 67% to 100%. Seven global ratings and 19 checklist items reached consensus as good indicators of the competence of clinicians performing colonoscopy.

LIMITATIONS

Further validation required.

CONCLUSION

Delphi methodology allowed for the rigorous development and content validation of a new measure of endoscopic competence, reflective of practice across institutions. Although further evaluation is required, it is a promising step toward the objective assessment of competency for use in colonoscopy training, practice, and research.

Reflections on current methods for evaluating skills during joint replacement surgery

Valid and reliable techniques for assessing performance are essential to surgical education, especially with the emergence of competency-based frameworks. Despite this, there is a paucity of adequate tools for the evaluation of skills required during joint replacement surgery. In this scoping review, we examine current methods for assessing surgeons’ competency in joint replacement procedures in both simulated and clinical environments. The ability of many of the tools currently in use to make valid, reliable and comprehensive assessments of performance is unclear. Furthermore, many simulation-based assessments have been criticised for a lack of transferability to the clinical setting. It is imperative that more effective methods of assessment are developed and implemented in order to improve our ability to evaluate the performance of skills relating to total joint replacement. This will enable educators to provide formative feedback to learners throughout the training process to ensure that they have attained core competencies upon completion of their training. This should help ensure positive patient outcomes as the surgical trainees enter independent practice.

Cite this article: Bone Joint J 2013;95-B:1445–9.

Training and simulation for patient safety

BACKGROUND

Simulation-based medical education enables knowledge, skills and attitudes to be acquired for all healthcare professionals in a safe, educationally orientated and efficient manner. Procedure-based skills, communication, leadership and team working can be learnt, be measured and have the potential to be used as a mode of certification to become an independent practitioner.

RESULTS

Simulation-based training initially began with life-like manikins and now encompasses an entire range of systems, from synthetic models through to high fidelity simulation suites. These models can also be used for training in new technologies, for the application of existing technologies to new environments and in prototype testing. The level of simulation must be appropriate to the learners' needs and can range from focused tuition to mass trauma scenarios. The development of simulation centres is a global phenomenon which should be encouraged, although the facilities should be used within appropriate curricula that are methodologically sound and cost-effective.

DISCUSSION

A review of current techniques reveals that simulation can successfully promote the competencies of medical expert, communicator and collaborator. Further work is required to develop the exact role of simulation as a training mechanism for scholarly skills, professionalism, management and health advocacy.

Addressing the severe shortage of health care providers in Ethiopia: bench model teaching of technical skills

CONTEXT

There is a severe shortage of health care workers in Ethiopia. This situation must be addressed by the efficient training of mass cohorts of students.

OBJECTIVES

This study aimed to demonstrate that bench model training is a feasible approach to teaching surgical skills in Ethiopia.

METHODS

A pre-test, simulation-based training intervention and post-test design was used. Two objective structured assessments of technical skills (OSATS) and a bench-top simulation training session were administered at the Black Lion Hospital, Addis Ababa, Ethiopia. Participants included 19 surgical residents who volunteered as trainees. Five surgical faculty members and one senior resident from the Black Lion Hospital, as well as two faculty members from the University of Toronto, participated as trainers and evaluators. The intervention consisted of OSATS tests comprising four stations, covering knot tying, closure of skin laceration, elliptical excision and bowel anastomosis. Tests were separated by 2-hour practice sessions. Main outcome measures included previously validated instruments comprising global rating scales (GRS) and skill-specific checklists (SSC).

RESULTS

The measures showed no improvement on knot tying (GRS: P = 0.14; SSC: P = 0.7), marginal improvement on closure of laceration (GRS: P = 0.48; SSC: P = 0.003), and improvements on excision (GRS: P = 0.012; SSC: P = 0.003) and bowel anastomosis (GRS: P < 0.001; SSC: P < 0.001).

CONCLUSIONS

The bench models and scoring schemes developed in Toronto, Canada were directly applicable in Addis Ababa, Ethiopia. This approach may prove a feasible, safe and cost-effective method for training a multitude of health care professionals in technical skills and may help to address the human resources deficit in Africa.

The use of virtual reality for training in carotid artery stenting: a construct validation study

BACKGROUND

Given that carotid artery stenosis (CAS) intervention is procedurally difficult, possesses an extensive learning curve, and involves a grave list of potential complications, construct validation of new non-clinical training devices is of increasing importance.

PURPOSE

To evaluate the construct validity of the Procedicus-Virtual Interventional Simulator Trainer (Procedicus-VIST) and its use as a training tool.

MATERIAL AND METHODS

Sixteen interventionalists (15 males, one female; mean interventional radiology [IR] experience >11 years) and 16 medical students (15 males, one female; no IR experience) received 1 hour of didactic instruction followed by an hour of familiarization training. Subjects then attempted to complete a carotid artery stenting procedure within 1 hour while their performance metrics were recorded. All participants completed a qualitative exit survey of subjective parameters using a visual analog scale.

RESULTS

Procedure and fluoroscopic time was 8.7 and 8.7 min greater in the novice group (P=0.0066 and P=0.0031), respectively. There were no significant differences in performances between the two groups in the remaining metrics of cine loops (number recorded), tool/vessel ratio, coverage percentage, and placement accuracy or residual stenosis. Contrast measurement metrics were found to be too imprecise for statistical analysis. Experienced and novice opinions differed significantly for six of 10 subjective parameters. No statistically significant difference in video-gaming habits was demonstrated.

CONCLUSION

With the exception of the metrics of performance time and fluoroscopic use, construct validity of the Procedicus-VIST carotid metrics were not confirmed. Virtual reality simulation as a training method was valued more by novices than by experienced interventionalists.

Porcine Transfer Study: Virtual Reality Simulator Training Compared with Porcine Training in Endovascular Novices

PURPOSE

To compare the learning of endovascular interventional skills by training on pig models versus virtual reality simulators.

METHODS

Twelve endovascular novices participated in a study consisting of a pig laboratory (P-Lab) and a virtual reality laboratory (VR-Lab). Subjects were stratified by experience and randomized into four training groups. Following 1 hr of didactic instruction, all attempted an iliac artery stenosis (IAS) revascularization in both laboratories. Onsite proctors evaluated performances using task-specific checklists and global rating scales, yielding a Total Score. Participants completed two training sessions of 3 hr each, using their group's assigned method (P-Lab x 2, P-Lab + VR-Lab, VR-Lab + P-Lab, or VR-Lab x 2) and were re-evaluated in both laboratories. A panel of two highly experienced interventional radiologists performed assessments from video recordings. ANCOVA analysis of Total Score against years of surgical, interventional radiology (IR) experience and cumulative number of P-Lab or VR-Lab sessions was conducted. Inter-rater reliability (IRR) was determined by comparing proctored scores with the video assessors in only the VR-Lab.

RESULTS

VR-Lab sessions improved the VR-Lab Total Score (beta = 3.029, p = 0.0015) and P-Lab Total Score (beta = 1.814, p = 0.0452). P-Lab sessions increased the P-Lab Total Score (beta = 4.074, p < 0.0001) but had no effect on the VR-Lab Total Score. In the general statistical model, both P-Lab sessions (beta = 2.552, p = 0.0010) and VR-Lab sessions (beta = 2.435, p = 0.0032) significantly improved Total Score. Neither previous surgical experience nor IR experience predicted Total Score. VR-Lab scores were consistently higher than the P-Lab scores (Delta = 6.659, p < 0.0001). VR-Lab IRR was substantial (r = 0.649, p < 0.0008).

CONCLUSIONS

Endovascular skills learned in the virtual environment may be transferable to the real catheterization laboratory as modeled in the P-Lab.

Teaching surgical skills: what kind of practice makes perfect?: a randomized, controlled trial

OBJECTIVE

Surgical skills laboratories have become an important venue for early skill acquisition. The principles that govern training in this novel educational environment remain largely unknown; the commonest method of training, especially for continuing medical education (CME), is a single multihour event. This study addresses the impact of an alternative method, where learning is distributed over a number of training sessions. The acquisition and transfer of a new skill to a life-like model is assessed.

METHODS

Thirty-eight junior surgical residents, randomly assigned to either massed (1 day) or distributed (weekly) practice regimens, were taught a new skill (microvascular anastomosis). Each group spent the same amount of time in practice. Performance was assessed pretraining, immediately post-training, and 1 month post-training. The ultimate test of anastomotic skill was assessed with a transfer test to a live, anesthetized rat. Previously validated computer-based and expert-based outcome measures were used. In addition, clinically relevant outcomes were assessed.

RESULTS

Both groups showed immediate improvement in performance, but the distributed group performed significantly better on the retention test in most outcome measures (time, number of hand movements, and expert global ratings; all P values <0.05). The distributed group also outperformed the massed group on the live rat anastomosis in all expert-based measures (global ratings, checklist score, final product analysis, competency for OR; all P values <0.05).

CONCLUSIONS

Our current model of training surgical skills using short courses (for both CME and structured residency curricula) may be suboptimal. Residents retain and transfer skills better if taught in a distributed manner. Despite the greater logistical challenge, we need to restructure training schedules to allow for distributed practice.

Assessment of a Virtual Interventional Simulator Trainer

PURPOSE

To assess the construct validity of the Procedicus Virtual Interventional Simulator Trainer (Procedicus-VIST) and its use as a training tool.

METHODS

Two groups comprised of 8 interventional radiologists (experts) and 8 medical students (novices) performed 6 renal artery procedures on the Procedicus-VIST. All participants received a 45-minute standardized didactic introduction before starting the simulations. The first 2-hour session was used for familiarization, whereas the second session constituted the testing period. During each procedure, objective performance data including procedure time, fluoroscopic time, contrast, cine loops, lesion coverage, tool:lesion ratio, placement accuracy, and residual stenosis were recorded by the Procedicus-VIST software. Exit surveys were completed to document demographic and subjective data. A visual analogue scale (VAS) from 0 to 100 was used to rate total, guidewire, catheter, balloon, stent, fluoroscopic, and joystick realism, as well as the simulator's pedagogic value.

RESULTS

There were no significant differences in performances between the 2 groups in residual stenosis, placement accuracy, procedure time, number of cine loops, lesion coverage, or tool:lesion ratio. The total fluoroscopic use was greater for the novice group (p < 0.01). Experts rated 6 of the 8 subjective parameters favorably, whereas the novice group approved of 7.

CONCLUSIONS

Using this study design, the quantitative metrics recorded by the Procedicus-VIST software failed to stratify performances based upon experience level, with the exception of fluoroscopic use. Investigation comparing standard training to virtual reality training should be performed to assess any differences in actual performance in the catheterization laboratory.

Getting teams to talk: development and pilot implementation of a checklist to promote interprofessional communication in the OR

BACKGROUND

Pilot studies of complex interventions such as a team checklist are an essential precursor to evaluating how these interventions affect quality and safety of care. We conducted a pilot implementation of a preoperative team communication checklist. The objectives of the study were to assess the feasibility of the checklist (that is, team members' willingness and ability to incorporate it into their work processes); to describe how the checklist tool was used by operating room (OR) teams; and to describe perceived functions of the checklist discussions.

METHODS

A checklist prototype was developed and OR team members were asked to implement it before 18 surgical procedures. A research assistant was present to prompt the participants, if necessary, to initiate each checklist discussion. Trained observers recorded ethnographic field notes and 11 brief feedback interviews were conducted. Observation and interview data were analyzed for trends.

RESULTS

The checklist was implemented by the OR team in all 18 study cases. The rate of team participation was 100% (33 vascular surgery team members). The checklist discussions lasted 1-6 minutes (mean 3.5) and most commonly took place in the OR before the patient's arrival. Perceived functions of the checklist discussions included provision of detailed case related information, confirmation of details, articulation of concerns or ambiguities, team building, education, and decision making. Participants consistently valued the checklist discussions. The most significant barrier to undertaking the team checklist was variability in team members' preoperative workflow patterns, which sometimes presented a challenge to bringing the entire team together.

CONCLUSIONS

The preoperative team checklist shows promise as a feasible and efficient tool that promotes information exchange and team cohesion. Further research is needed to determine the sustainability and generalizability of the checklist intervention, to fully integrate the checklist routine into workflow patterns, and to measure its impact on patient safety.

How surgeons disclose medical errors to patients: a study using standardized patients

BACKGROUND

Calls are increasing for physicians to disclose harmful medical errors to patients, but little is known about how physicians perform this challenging task. For surgeons, communication about errors is particularly important since surgical errors can have devastating consequences. Our objective was to explore how surgeons disclose medical errors using standardized patients.

METHODS

Thirty academic surgeons participated in the study. Each surgeon discussed 2 of 3 error scenarios (wrong-side lumpectomy, retained surgical sponge, and hyperkalemia-induced arrhythmia) with standardized patients, yielding a total of 60 encounters. Each encounter was scored by using a scale developed to rate 5 communication elements of effective error disclosure. Half of the encounters took place face-to-face; the remainder occurred by videoconference.

RESULTS

Surgeons were rated highest on their ability to explain the medical facts about the error (mean scores for the 3 scenarios ranged from 3.93 to 4.20; maximum possible score, 5). Surgeons used the word error or mistake in only 57% of disclosure conversations, took responsibility for the error in 65% of encounters, and offered a verbal apology in 47%. Surgeons acknowledged or validated patients' emotions in 55% of scenarios. Eight percent discussed how similar errors would be prevented, and 20% offered a second opinion or transfer of care to another surgeon.

CONCLUSIONS

The patient safety movement calls for disclosure of medical errors, but significant gaps exist between how surgeons disclose errors and patient preferences. Programs should be developed to teach surgeons how to communicate more effectively with patients about errors.

Perceptions of operating room tension across professions: building generalizable evidence and educational resources

BACKGROUND

Effective team communication is critical in health care, yet no curriculum exists to teach it. Naturalistic research has revealed systematic patterns of tension and profession-specific interpretation of operating room team communication. Replication of these naturalistic findings in a controlled, video-based format could provide a basis for formal curricula.

METHOD

Seventy-two surgeons, nurses, and anesthesiologists independently rated three video-based scenarios for the three professions' level of tension, responsibility for creating tension and responsibility for resolution. Data were analyzed using three-way, mixed-design analyses of variance.

RESULTS

The three professions rated tension levels of the various scenarios similarly (F=1.19, ns), but rated each profession's responsibility for creating (F=2.86, p<.05) and resolving (F=1.91, p<.01) tension differently, often rating their profession as having relatively less responsibility than the others.

CONCLUSIONS

These results provide an evidence base for team communications training about tension patterns, disparity of professional perspectives, and implications for team function.

Communication failures in the operating room: an observational classification of recurrent types and effects

BACKGROUND

Ineffective team communication is frequently at the root of medical error. The objective of this study was to describe the characteristics of communication failures in the operating room (OR) and to classify their effects. This study was part of a larger project to develop a team checklist to improve communication in the OR.

METHODS

Trained observers recorded 90 hours of observation during 48 surgical procedures. Ninety four team members participated from anesthesia (16 staff, 6 fellows, 3 residents), surgery (14 staff, 8 fellows, 13 residents, 3 clerks), and nursing (31 staff). Field notes recording procedurally relevant communication events were analysed using a framework which considered the content, audience, purpose, and occasion of a communication exchange. A communication failure was defined as an event that was flawed in one or more of these dimensions.

RESULTS

421 communication events were noted, of which 129 were categorized as communication failures. Failure types included "occasion" (45.7% of instances) where timing was poor; "content" (35.7%) where information was missing or inaccurate, "purpose" (24.0%) where issues were not resolved, and "audience" (20.9%) where key individuals were excluded. 36.4% of failures resulted in visible effects on system processes including inefficiency, team tension, resource waste, workaround, delay, patient inconvenience and procedural error.

CONCLUSION

Communication failures in the OR exhibited a common set of problems. They occurred in approximately 30% of team exchanges and a third of these resulted in effects which jeopardized patient safety by increasing cognitive load, interrupting routine, and increasing tension in the OR.

Communication failures in the operating room: an observational classification of recurrent types and effects

BACKGROUND

Ineffective team communication is frequently at the root of medical error. The objective of this study was to describe the characteristics of communication failures in the operating room (OR) and to classify their effects. This study was part of a larger project to develop a team checklist to improve communication in the OR.

METHODS

Trained observers recorded 90 hours of observation during 48 surgical procedures. Ninety four team members participated from anesthesia (16 staff, 6 fellows, 3 residents), surgery (14 staff, 8 fellows, 13 residents, 3 clerks), and nursing (31 staff). Field notes recording procedurally relevant communication events were analysed using a framework which considered the content, audience, purpose, and occasion of a communication exchange. A communication failure was defined as an event that was flawed in one or more of these dimensions.

RESULTS

421 communication events were noted, of which 129 were categorized as communication failures. Failure types included "occasion" (45.7% of instances) where timing was poor; "content" (35.7%) where information was missing or inaccurate, "purpose" (24.0%) where issues were not resolved, and "audience" (20.9%) where key individuals were excluded. 36.4% of failures resulted in visible effects on system processes including inefficiency, team tension, resource waste, workaround, delay, patient inconvenience and procedural error.

CONCLUSION

Communication failures in the OR exhibited a common set of problems. They occurred in approximately 30% of team exchanges and a third of these resulted in effects which jeopardized patient safety by increasing cognitive load, interrupting routine, and increasing tension in the OR.

Toward reliable operative assessment: the reliability and feasibility of videotaped assessment of laparoscopic technical skills

BACKGROUND

Decision making on the competency of surgical trainees to perform laparoscopic procedures has been hampered by the lack of reliable methods to evaluate operative performance. The goal of this study was to develop a feasible and reliable method of evaluation.

METHODS

Twenty-nine senior surgical residents were videotaped performing a low anterior resection and a Nissen fundoplication in a pig. Ten blinded laparoscopists rated the videos independently on two scales. Rating time was minimized by allowing raters to fast-forward through the tapes at their discretion. Interrater reliability and the time required to rate a procedure were assessed.

RESULTS

Rating time per procedure was a median of 15 min (range, 6-40). The mean interrater reliability for the two scales was 0.74.

CONCLUSIONS

The use of videotapes of operations enabled multiple raters to assess a performance reliably and shortened assessment times by 80%. This assessment technique shows potential as a means of evaluating the performance of advanced laparoscopic procedures by surgical trainees.

The effectiveness of video feedback in the acquisition of orthopedic technical skills

BACKGROUND

The addition of video feedback to bench model training offers residents the opportunity to see themselves perform a surgical task. Videotaped feedback therefore promotes self-evaluation, a critical learning skill, and also has the potential to influence how a resident executes a skill once they have had the opportunity to see themselves perform the task.

METHODS

Twenty-nine surgical residents were video recorded while performing three technical skills. They then were randomly assigned to receive either no feedback, video feedback alone, or video feedback with the help of an expert, an orthopedic surgeon. The surgical task was then repeated. Orthopedic surgeons evaluated the videotapes using the global rating scale and technical checklist form.

RESULTS

One-way between-subject analysis of variance comparing the pretest and post-test difference scores on three different measures for each of the three tasks revealed no statistically significant differences. After controlling for rater variance, the global rating scores across the three surgical tasks did not reveal any statistically significant differences.

CONCLUSIONS

This study failed to demonstrate an improvement in technical skills based on utilization of video feedback.

Resident self-assessment of operative performance

BACKGROUND

In medicine, the development of expertise requires the recognition of one's capabilities and limitations. This study aimed to verify the accuracy of self-assessment for the performance of a surgical task, and to determine whether self-assessment may be improved through self-observation or exposure to relevant standards of performance.

METHODS

Twenty-six senior surgical residents were videotaped performing a laparoscopic Nissen fundoplication in a pig. Experts rated the videos using two scoring systems. Subjects evaluated their performances after performance of the Nissen, after self-observation of their videotaped performance, and after review of four videotaped "benchmark" performances.

RESULTS

Expert interrater reliability was 0.66 (intraclass correlation coefficient). The correlation between experts' and residents' self-evaluations was initially moderate (r = 0.50, P <0.01), increasing significantly after the residents reviewed their own videotaped performance to r = 0.63 (Deltar = 0.13, P <0.01), yet did not change after review of the benchmarks.

CONCLUSIONS

Self-observation of videotaped performance improved the residents' ability to self-evaluate.

Forming professional identities on the health care team: discursive constructions of the 'other' in the operating room

BACKGROUND

Inter-professional health care teams represent the nucleus of both patient care and the clinical education of novices. Both activities depend upon the'talk' that team members use to interact with one another. This study explored team members' interpretations of tense team communications in the operating room (OR).

METHODS

The study was conducted using 52 team members divided into 14 focus groups. Team members comprised 13 surgeons, 19 nurses, nine anaesthetists and 11 trainees. Both uni-disciplinary (n = 11) and multi-disciplinary (n = 3) formats were employed. All groups discussed three communication scenarios, derived from prior ethnographic research. Discussions were audio-recorded and transcribed. Using a grounded theory approach, three researchers individually analysed sample transcripts, after which group discussions were held to resolve discrepancies and confirm a coding structure. Using the confirmed code, the complete data set was coded using the 'NVivo' qualitative data analysis software program.

RESULTS

There were substantial differences in surgeons', nurses', anaesthetists', and trainees' interpretations of the communication scenarios. Interpretations were accompanied by subjects' depictions of disciplinary roles on the team. Subjects' constructions of other professions' roles, values and motivations were often dissonant with those professions' constructions of themselves.

CONCLUSIONS

Team members, particularly novices, tend to simplify and distort others' roles and motivations as they interpret tense communication. We suggest that such simplifications may be rhetorical, reflecting professional rivalries on the OR team. In addition, we theorise that novices' echoing of role simplification has implications for their professional identity formation.

Teaching invasive perinatal procedures: assessment of a high fidelity simulator-based curriculum

OBJECTIVE

Learning curves pose a difficult problem in the teaching of technical skills: how do you teach procedural skills without compromising patients' health? A simulator-based curriculum has been designed to minimize the risks to patients undergoing amniocentesis by shifting the learning curve away from patients and into the laboratory. This study evaluated the effectiveness of a high-fidelity simulator-based curriculum in improving the performance of amniocentesis by obstetric trainees.

DESIGN

Thirty trainees received a course on the practice of amniocentesis. The curriculum consisted of a lecture, a syllabus, and a hands-on training session with the simulator. Pre- and post-training performance were evaluated with two rating scales. Training and performance evaluation were completed using the same simulator. The effectiveness of the simulator-based workshop and the effect of year of training were assessed using a two-way analysis of variance.

RESULTS

Performance scores improved from a mean score of 55% to 94% using checklist scoring and from 57% to 88% using global ratings. The two-way analysis of variance revealed a significant effect of training (F1,60 = 43.57; P < 0.001) accounting for 45% of the variance in scores, and a significant effect of experience level (F2,60 = 9.16; P < 0.001) accounting for 25% of the variance in scores.

CONCLUSIONS

A comprehensive curriculum based on a high-fidelity simulator was effective at improving skills demonstrated on the simulator. The challenge remains to establish that skills acquired on a simulator are transferable to the clinical setting.

Team communications in the operating room: talk patterns, sites of tension, and implications for novices

PURPOSE

Although the communication that occurs within health care teams is important to both team function and the socialization of novices, the nature of team communication and its educational influence are not well documented. This study explored the nature of communications among operating room (OR) team members from surgery, nursing, and anesthesia to identify common communicative patterns, sites of tension, and their impact on novices.

METHOD

Paired researchers observed 128 hours of OR interactions during 35 procedures from four surgical divisions at one teaching hospital. Brief, unstructured interviews were conducted following each observation. Field notes were independently read by each researcher and coded for emergent themes in the grounded theory tradition. Coding consensus was achieved via regular discussion. Findings were returned to insider "experts" for their assessment of authenticity and adequacy.

RESULTS

Patterns of communication were complex and socially motivated. Dominant themes were time, safety and sterility, resources, roles, and situation. Communicative tension arose regularly in relation to these themes. Each procedure had one to four "higher-tension" events, which often had a ripple effect, spreading tension to other participants and contexts. Surgical trainees responded to tension by withdrawing from the communication or mimicking the senior staff surgeon. Both responses had negative implications for their own team relations.

CONCLUSIONS

Team communications in the OR follow observable patterns and are influenced by recurrent themes that suggest sites of team tension. Tension in team communication affects novices, who respond with behaviors that may intensify rather than resolve interprofessional conflict.

Team communications in the operating room: talk patterns, sites of tension, and implications for novices

PURPOSE

Although the communication that occurs within health care teams is important to both team function and the socialization of novices, the nature of team communication and its educational influence are not well documented. This study explored the nature of communications among operating room (OR) team members from surgery, nursing, and anesthesia to identify common communicative patterns, sites of tension, and their impact on novices.

METHOD

Paired researchers observed 128 hours of OR interactions during 35 procedures from four surgical divisions at one teaching hospital. Brief, unstructured interviews were conducted following each observation. Field notes were independently read by each researcher and coded for emergent themes in the grounded theory tradition. Coding consensus was achieved via regular discussion. Findings were returned to insider "experts" for their assessment of authenticity and adequacy.

RESULTS

Patterns of communication were complex and socially motivated. Dominant themes were time, safety and sterility, resources, roles, and situation. Communicative tension arose regularly in relation to these themes. Each procedure had one to four "higher-tension" events, which often had a ripple effect, spreading tension to other participants and contexts. Surgical trainees responded to tension by withdrawing from the communication or mimicking the senior staff surgeon. Both responses had negative implications for their own team relations.

CONCLUSIONS

Team communications in the OR follow observable patterns and are influenced by recurrent themes that suggest sites of team tension. Tension in team communication affects novices, who respond with behaviors that may intensify rather than resolve interprofessional conflict.

Structured assessment of minor surgical skills (SAMSS) for family medicine residents

PURPOSE

To develop a valid and reliable examination to assess the technical proficiency of family medicine residents' performance of minor surgical office procedures.

METHOD

A multi-station OSCE-style examination using bench-model simulations of minor surgical procedures was developed. Participants were a randomly selected group of 33 family medicine residents (PGY-1 = 16, PGY-2 = 17) and 14 senior surgical residents who functioned as a validation group. Examiners were qualified surgeons and family physicians who used both checklists and global rating scales to score the participants' performances.

RESULTS

When family medicine residents were evaluated by family physicians, interstation reliabilities were .29 for checklists and .42 for global ratings. When family medicine residents were evaluated by surgeons, the reliabilities were .53 for checklists and .75 for global ratings. Interrater reliability, measured as a correlation for total examination scores, was .97. Mean scores on the examination were 60%, 64%, and 87% for PGY-1 family medicine, PGY-2 family medicine, and surgery residents, respectively. The difference in scores between family medicine and surgery residents was significant (p < .001), providing evidence of construct validity.

CONCLUSION

A new examination developed for assessing family medicine residents' skills with minor surgical office procedures is reliable and has evidence for construct validity. The examination has low reliability when family physicians serve as examiners, but moderate reliability when surgeons are the evaluators.

Exporting a technical skills evaluation technology to other sites

BACKGROUND

The Objective Structured Assessment of Technical Skill (OSATS) is a multistation performance-based examination that assesses the technical skills of surgery residents. This study explores the implementation issues involved in remote administration of the OSATS focusing on feasibility and the psychometric properties of the examination.

METHODS

An eight-station OSATS was administered to surgical residents in Los Angeles and Chicago. The University of Toronto and the local institutions shared responsibility for organization and administration of the examination.

RESULTS

There was good reliability for both the checklist (alpha = 0.68 for LA, 0.73 for Chicago) and global rating forms (alpha = 0.82 for both sites). Both iterations also showed evidence of construct validity, with a significant effect of training year for the checklist and global rating forms at both sites (analysis of variance: F = 8.66 to 19.93, P <0.01). Despite some challenges, the model of central organization and peripheral delivery was effective for the administration of the examinations.

CONCLUSIONS

Two iterations of the OSATS at remote sites demonstrated psychometric properties that are highly consistent with previously reported data suggesting that the examination is portable. Both faculty and residents indicated satisfaction with the examination experience. A model of central administration with peripheral delivery was feasible and effective.

Using operative outcome to assess technical skill

BACKGROUND

This study examined whether an operative product and time to completion could serve as measures of technical skill.

METHODS

Nine final-year (PGY5) and 11 penultimate-year (PGY4) general surgery residents participated in a 6-station bench model examination. Time to completion was recorded. Twelve faculty surgeons (2 per station) evaluated the quality of the final product using a 5-point scale.

RESULTS

The mean interrater reliability was 0. 59 for product quality. Interstation reliability was 0.59 for analysis of the final product and 0.72 for time to completion. There was 63% and 78% agreement between attendings' ratings and product quality and time scores respectively. PGY5s' mean product quality score was 4.14 +/- 0.26, compared with 3.82 +/- 0.33 for PGY4s (P < 0.05). PGY5s' mean time was 110 +/- 19 minutes compared with PGY4s' 132 +/- 15 (P < 0.05).

CONCLUSIONS

Analysis of the operative end product and time to completion offer efficient alternatives to on-line examiner scoring for bench model examinations of technical competence.

Internal sphincterotomy is superior to topical nitroglycerin in the treatment of chronic anal fissure: results of a randomized, controlled trial by the Canadian Colorectal Surgical Trials Group

PURPOSE

This was a multicenter, randomized, controlled trial to compare the effectiveness of topical nitroglycerin with internal sphincterotomy in the treatment of chronic anal fissure.

METHODS

Patients with symptomatic chronic anal fissures were randomly assigned to 0.25 percent nitroglycerin tid or internal sphincterotomy. Both groups received stool softeners and fiber supplements and were assessed at six weeks and six months.

RESULTS

Ninety patients were accrued, but 8 were excluded from the analysis because they refused internal sphincterotomy after randomization (6), the fissure healed before surgery (1), or a fissure was not observed at surgery (1). There were 38 patients in the internal sphincterotomy group (22 males; mean age, 40.3 years) and 44 patients in the nitroglycerin group (15 males; mean age, 38.7 years). At six weeks 34 patients (89.5 percent) in the internal sphincterotomy group compared with 13 patients (29.5 percent) in the nitroglycerin group had complete healing of the fissure (P = 5x10(-8)). Five of the 13 patients in the nitroglycerin group relapsed, whereas none in the internal sphincterotomy group did. At six months fissures in 35 (92.1 percent) patients in the internal sphincterotomy group compared with 12 (27.2 percent) patients in the nitroglycerin group had healed (P = 3x10(-9)). One (2.6 percent) patient in the internal sphincterotomy group required further surgery for a superficial fistula compared with 20 (45.4 percent) patients in the nitroglycerin group who required an internal sphincterotomy (P = 9x10(-6)). Eleven (28.9 percent) patients in the internal sphincterotomy group developed side effects compared with 37 (84 percent) patients in the nitroglycerin group (P<0.0001). Nine (20.5 percent) patients discontinued the nitroglycerin because of headaches (8) or a severe syncopal attack (1).

CONCLUSIONS

Internal sphincterotomy is superior to topical nitroglycerin 0.25 percent in the treatment of chronic anal fissure, with a high rate of healing, few side effects, and low risk of early incontinence. Thus, internal sphincterotomy remains the treatment of choice for chronic anal fissure.

The Effects of Modeling on Learning a Simple Surgical Procedure: See One, Do One or See Many, Do One?

The effect of modeling a simple surgical task on the subsequent performance of pre-clinical medical students was investigated. Groups of students read a verbal description of the excision of a skin lesion and closure of the resulting wound. Subsequently, groups watched zero, one, or four videotapes in which expert surgeons demonstrated the task. Finally, students had to perform the task themselves four times. During the performances, students were rated by operating room nurses using a checklist and a global rating scale of surgical performance. Time to perform the task was also recorded. In general, the results showed significant effects of experimental condition and trial number: Subjects who watched either one or four models demonstrated similar performance and performed better than subjects who did not watch any model. Later trials showed better accomplishments than earlier trials, both in terms of the quality of the surgery and speed. For some measures, significant interaction effects were found, suggesting that the advantages of watching a model are reinforced, rather than weakened, by practical experience with the task. The results are discussed with respect to the literature on modeling of motor skill tasks and the practical implications for surgical education.

A new assessment tool: the patient assessment and management examination

BACKGROUND

The major goal of certification is to assure the public that the candidate is competent in all facets required of the position. The patient assessment and management examination (PAME) was developed to enable a more comprehensive assessment of competence in the practice of surgery.

METHODS

A six-station, 3-hour, standardized-patient-based evaluation was developed. Each station was scored using a set of five-point global rating scales. PAME results were compared to the last two in training evaluation reports (ITER), the clinical knowledge component of the ITER (ITER-CK), an in-house oral examination (OE), and the Canadian Association of General Surgeons' multiple-choice examination (CAGS).

RESULTS

Eighteen senior general surgery residents were evaluated. Overall reliability was 0.70 (Cronbach's alpha). Fifth-year residents scored significantly better than fourth-year residents (t = 3.062; p = 0.0074), with 1 year of training accounting for 37% of the variance in scores. Correlations between the PAME and each of the other measures were ITER, 0.24; ITER-CK, 0.38; OE, -0.13; and CAGS, 0.061, with the PAME demonstrating better reliability and stronger evidence of validity than any other.

CONCLUSIONS

The PAME had better psychometric properties than other measures and assessed areas often not evaluated. This type of evaluation may be useful for feedback, remediation, or certification decisions.